BridgeData V2: A Dataset for Robot Learning at Scale
Authors: Homer Rich Walke
Kevin Black
Tony Z. Zhao
Quan Vuong
Chongyi Zheng
Philippe Hansen-Estruch
Andre Wang He
Vivek Myers
Moo Jin Kim
Max Du
Abraham Lee
Kuan Fang
Chelsea Finn
Sergey Levine
Venue: CoRL 2023 Poster (2023)
Keywords: Datasets
Manipulation
Imitation Learning
Offline Reinforcement Learning

Abstract: We introduce BridgeData V2, a large and diverse dataset of robotic manipulation behaviors designed to facilitate research in scalable robot learning. BridgeData V2 contains 53,896 trajectories collected across 24 environments on a publicly available low-cost robot. Unlike many existing robotic manipulation datasets, BridgeData V2 provides enough task and environment variability that skills learned from the data generalize across institutions, making the dataset a useful resource for a broad range of researchers. Additionally, the dataset is compatible with a wide variety of open-vocabulary, multi-task learning methods conditioned on goal images or natural language instructions. In our experiments,we apply 6 state-of-the-art imitation learning and offline reinforcement learning methods to the data and find that they succeed on a suite of tasks requiring varying amounts of generalization. We also demonstrate that the performance of these methods improves with more data and higher capacity models. By publicly sharing BridgeData V2 and our pre-trained models, we aim to accelerate research in scalable robot learning methods.

BridgeData V2: A Dataset for Robot Learning at Scale
Homer Walke'
Kevin Black' Abraham Lee'
Moo Jin Kim? Max Du
Chongyi Zheng* Vivek Myers'
Tony Zhao?
Philippe Hansen-Estruch Quan Vuong" Andre He
Kuan Fang'
Chelsea Finn²
Sergey Levine'
UCC Berkeley
Stannord GGoogle DeepMind +CMU
We introduce BridgeData V2, a large and diverse dataset of robotic manipulation behaviors designed to facilitate research on scalable robot learning. BridgeData V2 contains 60,096 trajectories collected across 24 environments on publicly available low-cost robot. BridgeData V2 provides extensive task and environment variability, leading to skills that can generalize across environments, domains, and institutions, making the dataset useful resource for a broad range of researchers. Additionally, the dataset is compatible with a wide variety of open- vocabulary, multi-task learning methods conditioned on goal images or natural language instructions. In our experiments, we train 6 state-of-theeart imitation learning and offline reinforcement learning methods on our dataset, and find that they succeed on suite of tasks requiring varying amounts of generalization. We also demonstrate that the performance and higher capacity models, and that training greater variety of skills leads improved generalization. By publicly sharing BridgeData V2 and our pre-trained models, aim to accelerate research in scalable robot learning methods. Project page: https: //tp///raill berkeeey gi github. io/bridggdaaa.
Abstract:
these methods improves with more data
Many Skills
24 Environments
left burner the righ: burnez
draser
Open Vocabulary
100 Objects
Figure 1 (BridgeData V2) We propose a large-scale robotic manipulation dataset containing 60,096 trajectories across 24 environments. The dataset includes skills such pick-and-place, pushing. sweeping, stacking, folding, and more. Portions of the data include multiple camera views and depth data, and all of the data includes natural language labels.
Corresponding email: homer waleeee.elkeeteekeeeeeyy.e
7th Conference on Robot Learning (CoRI. 2023), Atlanta, USA.

---

Introduction
A useful robotic system needs skills that generalize across the wide variety of conditions found in the real world. Recent results in computer vision and natural language processing have demonstrated that we can obtain broad generalization by training high-capacity models on large and diverse datasets [1. 2]. How can we apply this same recipe robotics? The general strategy seems straightforward. First, collect large dataset of demonstrations of robot behaviors. Then, train an expressive policy extract the desirable behaviors with behavioral cloning or offline reinforcement learning (RL). Given enough data in enough settings, we should learn a policy that can generalize across many tasks and environments.
However, in practice, assembling a dataset with the right features to accelerate research in large-scale robot learning presents a significant challenge. Since collecting large dataset of robot behaviors is time-consuming, the dataset should be reusable outside of the institution where it was collected. Ideally, any researcher could train on the dataset and obtaín a policy with reasonable performance on their tasks. To enable this, the dataset needs extensive coverage of tasks and environments so that policies learned the data will generalize to new lab settings. Many existing robot datasets contain only one or a few environments and tasks [3, 4, 5]. meaning researcher would need to exactly replicate a scene from the data to use them for robot learning. Additionally, the dataset should support flexible task conditioning, through goal images or natural language instructions, so that researchers can easily command policies trained on the data to perform new tasks. Importantly, the dataset should contain data for many feasible tasks in a given environment so that multi-task policy must learn to pay attention to the task specification rather than inferring the task from the initial observation.
In this paper, we propose a new dataset, which call BridgeData V2 (Figure 1) because it greatly expands on the previously released Bridge Dataset [6]. BridgeData V2 contains 50,365 demonstrations of 13 skills across 24 environments, more than 7 times as many demonstrations as the original Bridge Dataset. BridgeData V2 is not just a quantitative improvement: the new dataset specifically designed as a useful resource for wide range of robot learning research. The greater quantity and diversity of data enables better generalization to new lab settings and, unlike the first Bridge Dataset, the demonstrations were collected in scenes with many possible skills to support open-vocabulary task specification through either goal images or language instructions. We also augment the demonstrations with 9,731 trajectories collected from a heavily randomized pick-and- place policy to boost the robustness of the foundational objection repositioning skill.
These features make Data V2 well suited for use with variety of different robot learning methods, many of which make different assumptions. While a number of datasets have been proposed in the past, these have typically only been used with one type of method (c.g.. text- conditional behavioral cloning) [5. 7J. To illustrate the versatility and broad applicability of BridgeDatt V2, we include a comprehensive evaluation of 6 different state-of-the methods, which include algorithms for rext-conditioned imitation learning [7], goal-conditioned imitation learning, and coal-conditioned RL [8, 9]. These methods cover a range of key design decisions involving the policy architecture, the use of observation histories, action discretization, and action prediction horizon. We also analyze how the performance of some of these methods changes with dataset size and diversity, showing that large and diverse datasets like BridgeData V2 are an important ingredient enabling good performance.
robot learning
Our contributions are new dataset of robotic manipulation behaviors as well as the empirical study of state-of-the-arr offline learning methods using the introduced dataset. We find that BridgeData V2 is useful as an offline dataset for a wide variety of learning algorithms and across multiple labs. Additionally, our scaling analysis, we find that performance improves with both model size and dataset size and diversity. We believe BridgeData V2 is both useful resource for robot learning research and a demonstration of the promise of data-driven robot learning.
2 Related Work
Prior work on robot learning has often leveraged small datasets that cover single task in a single
domain to develop better learning methods [10, 11, 12, 13, 14, 15. 16, 17, 18). Several prior projects have assembled large robot datasets for a single behavior like grasping [19, 20, 21, 22, 23], poking [24], pushing [25, 26), or rope manipulation [27]. Others include multiple tasks [3, 4, 28, 29, 30, 31, 5, 32, 33). However, even the multi-task datasets are difficult for other researchers to use,
2

---

Dataset
# Traj.
Skills
# Env.
Lang.
Public Data
Public Robot
Collection
MIME [3]
8.30k
12
1
X
human
RoboTurk [4]
2.10k
2
1
X
/
human
RoboNet [23]
162k
n/a
X
J
scripted
MT-OOpt [30]
800k
I
X
X
scripted & learned
BridgeData [6]
7.20k
4
12
J
V
V
human
BC-Z
26.0k
3
I
J
J
X
human
RT-1 [7]
130k
2
3
X
X
human
RH20T [40]
110k
41
50
V
human
29% human, 71% scripted
RoboSet [32]
98.5k
11
/
84% human, 16% scripted
BridgeData V2
60.1k
24
Table BridgeData V2 is large and diverse publicly available robotic manipulation dataset suitable for wide variety of learning methods. See Section 3.3 for definitions of "skill" and "environment."
because they are collected in single environment, and therefore do not facilitate enviroenvir generalization. In order for another researcher to use such a dataset, they would need to make their own environment match perfectly. We instead aim to provide a dataset that covers many tasks and many domains, so that other researchers can use the data to experiment with robot learning large datasets in their own laboratories. Some prior work has proposed large, diverse, and multi-task datasets, but uses proprietary robots [7, 5]. We aim to facilitate academic research by using a low- cost publicly available robot that any lab can acquire. Other projects have tried using low-cost, handheld grasping devices collect large datasets [34, 35], but this technique requires inferring actions from video and then transferring behaviors to a robot.
Most closely related to this project is the original Bridge Dataset [6, 36]. While our dataset is quantitatively larger, our work also provides number of qualitative differences (see Table 1). We include both demonstration data and autonomously collected data, as well as provide language descriptions. The aim make the dataset useful not only for multi-task imitation learning, but for variety of learning methods that might have different assumptions: for example, reinforcement learning methods that require data with broader coverage and may benefit from noisy suboptimal imitation
trajectories [37], methods that require language descriptions to run anguage-conditione learning [5], or methods that use high-capacity models and require very large datasets before they can perform at their full potential [7]. To validate this, we specifically select wide variety of methods our evaluation to show that our work provides a single dataset that compatible with many approaches [7, 38, 39, 17]. This is highly nontrivial, virtually all prior robotic manipulation datasets have been evaluated with only one or few methods [23, 5, 7]. Additionally, unlike many prior datasets [23, 5]. our experiments isolate the effect of data diversity and show that greater diversity improves generalization, corroborating the result from Brohan et al. [7]. Our experiments extend Brohan et al. [7] by showing that skill diversity improves generalization, not just object diversity. Released earlier this year and collected concurrently with BridgeData V2, RH20T [40] proposes dataset with several robots and 10k demonstrations. While RH20T larger, we provide a comprehensive analysis of policies trained on BridgeData V2, and show that these policies can generalize across environments and institutions. Training on combination of the largest datasets released so far is an exciting and promising direction for future work.
3 BridgeData V2
Our goal is design dataset that facilitates research in large-scale robot learning. The dataset should support generalization to novel tasks, environments, and even institutions. The dataset should also support flexible task conditioning through either goal images or natural language instructions. In this section, we discuss how we collected the data as well as its composition.
3.1 System setup
The robot setup (Figure 2) costs approximately $4,000 in total and consists of parts that are all publicly available with turnaround time of less than two weeks.

---

Fixed Depth
Figure (System setup) A picture of our robot setup showing the WidowX 250 robot arm and various cam- eras. For sensing, we use an RGBD camera that is fixed in an over-the-shoulder view, two RGB cameras with poses that are randomized during data collection, and an RGB camera attached to the robot's wrist. The images are saved at a 640x480 resolution and the control frequeney is Hz. We collect demonstrations by teleop- erating the robot with a VR controller. See Appendix C for additional details.
WidowXX 250 Arm
Rundomized Cameras
3.2 Data collection
We designed data collection protocol to support learning multi-task, generalizable skills. To support broad generalization, we collected data for a wide range of tasks in many environments with suitable variations in objects, camera pose, and workspace positioning. To support the evaluation of multi-task learning methods, we collected data for many possible tasks simultaneously in each environment. This ensures that a policy must pay attention to the task specification, rather than inferring the task from its observations. Notably, this philosophy differs from the original Bridge Dataset, where data was collected for a specific (smaller) set of predefined tasks.
For example, we might set up a kitchen scene with several food items and utensils as well as a drawer. Then, the data collector collects demonstrations by performing any feasible task, such as opening the drawer or putting a utensil in the sink. To speed up data collection, we do not require the positions of the objects in the environment the robot be reset between trajectories. We also do not require the data collector to label trajectories with task names. Every 50 trajectories, the collector randomizes the poses of the cameras, switches the objects in the scene, and randomizes the position of the workspace relative to the robot.
Repositioning objects is widely applicable skill and thus we wanted to learn policies that can pick- and-place a large number of objects in many environments. During the data collection process, we realized we could partially automate the collection of pick-and-place data with heavily randomized scripted policy. While this policy fails frequently, we can run it autonomously to collect a large amount of pick-and-place data for wide range of objects more quickly than teleoperating the robot. Methods that benefit from suboptimal data, such as offline RI., can leverage this autonomous data to learn more robust behaviors. Note that users free to exclude the autonomous data from training. but our dataset offers the flexibility to explore both options.
Since we do not annotate trajectories with task names during data collection, we used a crowdsourc- ing platform to label the data post-hoc. Annotators were asked to describe the task being performed by the robot in each trajectory, with particular emphasis the final location of any moved objects.
3.3 Dataset composition
To effectively describe the composition of our dataset, we will first define "skill" and "task," two terms that have had many different meanings in prior work. We use "skill" to denote groups of trajectories that exhibit similar motions such as pick-and-place opening - but potentially with different objects or arrangements of objects. We use "task" to denote groups of trajectories that correspond to a similar language instruction, which in practice typically means that similar motions (e-g.. pick-and-place) with different objects (e.g., a fork or a bowl) correspond to the same skill, but to different tasks. This usage of the terms resembles some prior work on imnuage-cooditione robot learning [7, 5].
sweeping, folding, or door
BriageData V2 includes 13 skills that range in complexity. A large portion of the data consists of the foundational skills of pick-and-place, pushing, and reorienting objects, since these skills are applicable in wide variety of settings and mastery of these skills may transfer to more complicated behaviors. The more complex skills include opening and closing doors and drawers, wiping surfaces, folding cloths, stacking blocks, twisting knobs, flipping switches, turning faucets, zipping zippers, and using tools to sweep granular media. To ensure that we can learn generalizable skills, the data includes examples of each these skills applied to wide variety of objects and in a wide variety of environments. BridgeData V2 features environments, including kitchens, sinks, and tabletops, as well as more than 100 objects. In total, BridgeData V2 contains 50,365 expert demonstrations and

---

Mixed Objjec Environment Manipulation (3,119)
Mixed Object Cloth Manipulation (2,086)
Environment Manipuaation (3,343)
Tahletops (12,290)
Cloth Manipulation (2,960)
Block Stacking; (2,769)
Granular Sweeping (1,136)
Other (498)
Otber (1,917)
Sinks (3,102)
Object Manipolation
(34,509)
(33,111)
Figure 3 (Dataset composition) (L) The various types of tasks in BridgeData V2. The majority of the data comes from foundational object manipulation tasks, such as pick-and-place pushing, and sweeping. Additional data comes from environment manipulation, which includes opening and closing doors and drawers. The remaining data comes from more complex tasks, such as stacking blocks, folding cloths, and sweeping granular media. Some segments of the data contain mixtures (R) The 24 environments in BridgeData V2, grouped into 4 categories. The majority of the data comes from 7 distinct toy kitchens, which include some combination of sinks, stoves, and microwaves. The remaining environments come from diverse sources, including various tabletops, standalone toy sinks, a toy laundry machine, and more.
of these categories.
pick and and-alace place reorientation
exeeping
Lever
draves
Figure 4 (Skills Environments) Examples of skills and environments in the dataset. BridgeData V2 in- cludes wide variety of skills, vironments, and objects to support broad generalization.
laundry
9,731 trajectories from a scripted policy, Figure 3 shows a breakdown of the environments and skills in the demonstration data and Figure shows some examples. We provide additional statistics in Appendix A.
4 Offline Learning Methods
To demonstrate that BridgeData V2 is compatible with variety of learning methods with different assumptions, we evaluated several state-of-the-art offline learning methods using the data. We selected both goal-conditioned methods (in the form of goal images) and language-conditionee methods because our dataset is designed for open- open-vocabulary task specification. The observation space for all of the methods 128x128 RGB images, except for RT-1 which uses 320x256 RGB images. We use only the weerthheshouldee camera view. The 7D action space of the robot consists of continuous 6D Cartesian end-effector motion, corresponding as discrete dimension to control the opening and closing of the gripper. We provide the full implementation details and training data statistics in Appendix B.
relative changes pose, as well
4.1 Goal-conditioned methods
Behavioral cloning with goals (GCBC). As baseline, evaluate standard goal-conditioned behavior cloning method. We use a ResNet-34 to encode the observation and goal. The image encoding is then passed through several fully connected layers to produce a robot action.
Diffusion behavioral cloning (D-GCBC). Prior work has shown that highly expressive policies can better model multi-modal action distributions, leading improved performance variety of tasks [41, 18, 38). We modify our GCBC baseline to represent the policy using a diffusion process and train with the DDPM objective [42].
Action chunking with transformers (ACT), Previous work has also noted that predicting action sequences instead of single actions can benefit imitation learning [17, 18). ACT instantiates this with transformers and a conditional VAE [43] training objective, allowing the model to generate multi-moda action sequences.
5

---

Contrastive RL (CRL). Along with imitation learning methods, we also evaluate a goal- conditioned reinforcement learning method. CRL casts goal-conddtioned RL as representation learning problem, parameterizing goal-conditioned value functions as log-linear representations analogous to contrastive learning [39, 44, 45).
4.2 Language-conddtioned methods
Behavioral cloning with language (LCBC). As baseline, we evaluate standard language- conditioned behavior cloning method [46]. The natural language instruction is first encoded using the MUSE sentence embedding [47]. Then the image observation is encoded using a ResNet-34 with FiLM conditioning on the language encoding [48]. The final encoding is then passed into fully connected policy network to produce a robot action.
RT-1 [7] is a large transformer model trained via behavior cloning to predict low-level robot actions. The inputs first tokenized using a pre-trained EfficientNet [49] (for images) and pre-trained language model [50] (for the language instruction). These tokens are then passed to decoder- only transformer predict discretized robot actions. RT-1 is the only method to take a history of observations as input and uses a higher image resolution than the other methods.
5 Experiments
The goal of our experiments is evaluate the utility of our dataset for testing a variety of multi-task offline learning methods. Rather than attempting to rigorously analyze different design decisions in the learning methods, aim to determine the extent to which our dataset facilitates large-scale robot learning research. Our experiments are designed to answer the following questions:
1. Can we learn broad range of tasks with both goal and
methods?
2. Can we generalize the skills in the dataset to new objects and environments?
3. Can another institution use the dataset to obtain skills that work in their lab?
4. How does model size, dataset size, and dataset diversity affect the performance of our learned policies?
5.1 Can we learn broad range of skills that generalize
objects and environments?
We first evaluated the methods in Section 4 on tasks that are seen in the training data (see Table 2). Even though these tasks are seen in training, the methods must still generalize to novel object positions, distractor objects, and lighting. These tasks test range of abilities: opening the drawer requires precisely inserting the gripper into the handle, sweeping the beans involves using tool and predicting the motion of granular media, stacking block requires delicate balancing, grasping the corn cob requires correctly orienting the wrist joint, and flipping the pot requires coordinating all six degrees of freedom. To obtain success rates for each method, we collected 10 trials for each task, varying the positions of objects and distractors between trials. Most apparently, RT-1 is significantly better than our LCBC baseline, likely due to combination of design decisions such as larger images, action discretization, and observation historics. The goal-conditioned methods comparable each other in success rate. We also noticed qualitative differences in the robotic behaviors each method produced. For instance, D-GCBC, ACT, and RT-1 exhibit more pauses, because their expressive policy distributions can accurately model pauses in the demonstration data. However, since D-GCBC lacks observation histories like RT-1 or action chunking like ACT, sometimes produces jerky behaviors indicating oscillation between modes in the action distribution. These behaviors are best observed in the video on our website.
Next, we evaluated the methods on tasks that require generalizing skills in the data to novel objects and environments (see Table 3). For instance, in the rice sweeping task, both the rice and brush unseen. In the cloth folding task, the cloth is a different color and thicker than the cloths in the training data. In the task of putting a marker into a bowl, the entire environment is unseen, including the objects. The large and diverse training set enabled considerable degree of generalization, with most methods attaining non-zero success on most tasks. However, the methods particularly struggled tasks involving unseen objects since these object names not grounded in the dataset. Once again, RT-1 greatly outperformed the LCBC baseline.
5.2
Can another institution use the dataset?
To demonstrate that the dataset useful for other researchers, we had another institution set up the robot and evaluate the same methods in their lab. The tasks and environments in this evaluation

---

Task
GCBC
D-GCBC
ACT
CRL
LCBC
RT-1
Open drawer
0.4
0.6
0.5
0.4
0.5
1.0
Sweep beans into pile with bar
0.9
0.9
0.9
0.7
0.4
0.6
Fold blue cloth over object
0.4
0,9
Stack green block on yellow block
0.4
0.2
0.3
0.6
0.0
0.0
Put corn in pot
0.9
0.8
0.8
0.8
0.0
0.0
Put carrot plate
0.7
0.4
0.1
0.0
0.0
0.8
Flip pot upright
0.1
0.1
0.0
0.4
0.4
0.4
Put eggplant pot
0.0
0.2
Average
0.49
0.49
0.42
0.23
0.49
Table 2 (Evaluation on seen tasks) Overall, the goal-conditioned methods were comparable. However, RT-1 outperformed LCBC. Success rates are averaged over 10 trials.
Task
GCBC
D-GCBC
ACT
CRI.
LCBC
Sweep rice into pile with brush*
0.6
0.0
0.3
0.3
0.0
0.1
Fold thick gray cloth over object"
0.3
0.6
0.7
0.0
0.0
Put marker in bowl
0.6
0.6
0.2
0.7
0.0
0.0
Wipe the table with the cloth Put the mushroom in the pot Put the spoon on the cloth
0.6
0.5
0.4
0.6
0.4
0.9
0.7
0.9
0.1
0.7
0.1
0.6
0.8
0.7
0.0
0.8
0.0
1.0
Average
0.55
0.28
0.52
0.08
0.50
Unseen objects, seen environment Unseen objects, unseen environment
Seen objects, unseen environment
Table (Evaluation unseen tasks) Either the objects, environment, both objects and environment were unseen. We found that both language and goal conditioned methods exhibit non- zero success, demonstrating that BridgeData V2 supports broad generalization. Success rates averaged over 10 trials.
matched tasks seen the training data; however, there were differences robot setup, camera placement, lighting conditions, and object instances that led to significant domain shift. We evaluated tasks in our lab (Lab 1) and at the other institution (Lab 2) with all the methods. As seen in Table 4, performance success. RT-1 especially showed only a small degradation in performance. The performance of goal-conditioned methods degraded more, though particularly difficult for goal-conditioned method since the orientation the pot is difficult to distinguish. Additionally, the "put eggplant in pot" is very challenging task in both labs since the eggplant easily slips out of the gripper. Note that these evaluations were performed zero-shot, without any new data collected in Lab 2, and we expect fine-tuning on small amount of data in new lab to significantly improve performance, as discussed in prior work [6].
somewhat worse in Lab 2, but all methods attained non-zero
note that the "flip the pot upright" task is
5.3 How does varying the size of the model and the size and diversity of the dataset affect performance?
Lastly, we analyzed how the performance of our gonliconditooned BC method varies with model size and dataset size and diversity (see Figure 5). First, we tested GCBC with different sizes of image
Lab + Lab 2
Task
GCBC
D-GCBC
ACT
CRL
LCBC
RT-1
Put carrot on plate
0.7 0.3
0.4 0.0
0.1 0.0
0.3
0.0 0.0
0.8 0.4
Flip pot upright
0.1 - 0.0
0.1 0.2
0.0
0.4
0.4 0.1
0.4
Put eggplant in pot
0.1 - 0.1
0.2 0.2
0.0
0.0 300+00 0.1
0.0 100+00 0.0
0.2 - 0.2
Average
0.30 0.13
0.23 0.13
0.03 0.10
0.13 0.20 0.13 0.03
0.47 0.40
Table 4 (Cross-institution evaluation) We find that both goal-conditioned and language- conditioned methods achieve non-zero success at a new institution (Lab 2), demonstrating that our dataset is useful for other rescarchers. Success rates averaged over 10 trials.

---

1.2
1.0
Avenge Suooes R.00
Avera Suocoes
0.5
0-4
Train on 3 skills
Train on 13 skills
0.30
0.65
0.2
82
Scen
Unseen
Re 18
Redle 14 Image Encoder
RecKe
0.5
0/25 Proportion of Dutzset
Figure 5 (Scaling analysis) (L) Performance of goal-conditioned behavior cloning trained on BridgeData V2 improves with higher capacity models and larger training sets, suggesting that our dataset provides an effective foundation for studying scalable policy learning. (R) We compared the performance of GCBC trained on dataset with only 3 skills and dataset with all 13 skills, keeping the size of the datasets approximately the same. On an unseen task, the policy trained with greater skill diversity performed significantly better, indicating positive transfer between skills. We used 20 trials per policy in this experiment.
encoders on the task of moving a spoon. We found that higher capacity models perform strictly better at this task. Qualitatively, the smaller models often failed to correctly rotate the gripper to pick up the spoon when it was positioned horizontally. Next, we evaluated GCBC trained on datasets of different size. To create smaller datasets, we randomly subsampled the data stratified by task, preserving diversity while decreasing the number of trajectories. We found that as dataset size increases, performance improves for both seen and unseen task. To test how skill diversity affects performance, we trained GCBC on two datasets: one dataset with only 3 skills (pick-and- place, pushing, and wiping) and another dataset with all 13 skills. We kept the size of these datasets approximately the same (28k vs 27k trajectories). We found that performance on an unseen pick- and place task was significantly improved by training on data with greater skill diversity. This result indicates that data from other skills can improve the robustness of the pick-and-plas skill. Brohan al. [7] also isolate data diversity and find that greater diversity improves generalization, however they study diversity in objects. We analyze skill diversity and arrive at a similar conclusion. Taken together, the results in this section indicate that scaling the dataset and model capacity further is a promising path toward greater capabilities.
Discussion, Limitations, and Future Work
We presented BridgeData V2, a dataset with 60,096 trajectories of robotic manipulation behaviors designed to enable research on scalable robot learning methods. Our dataset covers a wide range of tasks and environments and supports methods with various assumptions, including goal and language conditioned imitation learning and reinforcement learning. BridgeData V2 was collected with low-cost and widely available robot arm, making it suitable for accessible research on robot learning. Our evaluation also demonstrates that a wide range of learning algorithms can use Bata V2 to produces policies that generalize across tasks, environments, and institutions. We believe this is strong validation of the utility of our dataset, as many previously proposed datasets have only been evaluated with one or a small number of methods, and typically at a single institution.
Our dataset does have a number of limitations. While we provide a variety of behaviors, including manipulation of deformable objects, tools, and furniture, the tasks generally low-precision and do not require complex manipulation of forces. This is reasonable for studying generalization but does not cover the challenges that might occur with more forceful manipulation more dynamic tasks, such as throwing, moving heavy objects, or low-tolerance industrial insertion. Pushing the frontier on more precise, dexterous, and dynamic tasks is an exciting avenue for futuree data collection efforts. Additionally, our dataset was collected entirely at a single institution, and although we demonstrate that by utilizing a variety of environments our data enables robots to perform tasks in other labs, an even broader coverage of environments would be a desirable property in future datasets. Finally, although we deliberately chose an accessible and low-cost robot arm, may be difficult for other researchers to standardize around the same robot. Therefore, a particularly exciting direction for future data collection work would be to assemble multi-robot datasets that enable some degree of generalization across robot morphology. Nonetheless, we hope that our dataset will serve as an important stepping stone toward making scalable robot learning research practical, accessible, and effective.
8

---

Acknowledgements
We thank Abraham Lee, Mia Galatis, Caroline Johnson, Christian Aviña, Samantha Huang, and Nicholas Lofrese for collecting data, as well as Microsoft Research for assistance in labeling parts of the data with language. This research was supported by the TPU Research Cloud. This research was partly supported by ONR N00014-20-1-2383 and NSF IIS-2150826.
References
[1] T. Brown, B. Mann, N. Ryder, M. Subbiah, J. D. Kaplan, P. Dhariwal, A. Neelakantan, Shyam, G. Sastry, A. Askell, et al. Language models are few-shot learners. Advances in neural information processing systems, 33: 33:1877-1901, 2020.
[2] A. Ramesh, M. Pavlov, G. Goh, S. Gray, C. Voss, A. Radford, M. Chen, and I. Sutskever. Zero-shot text-to-image generation. In International Conference on Machine Learning, pages 8821-8831. PMLR, 2021.
[3] Sharma, L. Mohan, Pinto, and A. Gupta. Multiple interactions made easy (mime): Large scale demonstrations data for imitation. In Conference on robot learning, pages 906-915. PMLR, 2018.
[4] A. Mandlekar, Zhu, A. Garg, Booher, M. Spero, A. Tung, Gao, Emmons, A. Gupta, E. Orbay, et al. Roboturk: A crowdsoureing platform for robotic skill learning through imitation. In Conference on Robot Learning, pages 879-893, PMLR, 2018.
[5] E. Jang, A. Irpan, M. Khansari, D. Kappler, F. Ebert, C. Lynch, S. Levine, and Finn. BC-z: Zero-shot task generalization with rohotic imitation learning. In 5th Annual Conference on Robot Learning, 2021. URL
[6] F. Ebert, Y. Yang. K. Schmeckpeper, B. Bucher, G. Georgakis, K. Daniilidis, C. Finn, and Levine. Bridge data: Boosting generalization of robotic skills with cross-domain datasets. arXiv preprint arXiv: 109. 13396, 2021.
[7] A. Brohan, N. Brown, J. Carbajal, Y. Chebotar, J. Dabis, C. Finn, K. Gopalakrishnan, K. Hausman, A. Herzog, J. Hsu, et al. Rt-1: Robotics transformer for real-world control at scale. arXiv preprint 2rXiv:2212.068177 2022.
[8] L. P. Kaelbling. Learning Achieve Goals. IJCAI, 1993.
[9] Schaul, Horgan, K. Gregor, and D. Silver. Universal Value Function Approximators. In International Conference on Machine Learning (ICML), pages 1312-1320, 2015. ISBN 7781510810581 URL http:/ //tp:///////eeeeenggegee mmr..mmr...... presss/33/sscaaul1s httpaautll......tpp////o. jmlr.
html.
[10] S. Levine, C. Finn, T. Darrell, and P. Abbeel. End-to-End Training of Deep Visuomotor Policies. Journal of Machine Learning Research (JMLR), 17(1):133-1333, 2016. ISSN 15337928. do:11010077413338-444---1-3----
URL
00702. pdf.
[11] J. Ho and S. Ermon. Generative Adversarial Imitation Learning. In Advances in Neural Information Processing Systems, 2016. URL https://arxiis http::s:/aariiioog//d//////////e /166. 03476. pdf.
[12] Ebert, Finn, S. Dasari, A. Xie, A. Lee, and S. Levine. Visual foresight: Model-based deep reinforcement learning for vision-bast based robotic control. arXiv preprint arXiv: 1rXiv:1812.00566. 2018.
[13] T. Zhang, 7. McCarthy, O. Jow, D. Lee, X. Chen, K. Goldberg, and P. Abbeel. Deep imitation learning for complex manipulation tasks from virtual reality teleoperation. In 2018 IEEE International Conference on Robotics and Automation (ICRA), pages 5628-5635. IEEE, 2018
[14] A. Singh, A. Yu, J. Yang, J. Zhang, A. Kumar, and S. Levine. COG: Connecting New Skills Past Experience with Offline Reinforcement Learning. Conference on Robot Learning (CoRI.), 2020.

---

[15] A. Mandlekar, D. Xu, J. Wong, S. Nasiriany, C. Wang. R. Kulkarni, L. Fei-Fei, S. Savarese, Y. Zhu, and R. Martin-Martie Martin. What matters learning from offline human demonstrations 2021.
for robot manipulation. In arXiv preprint
[16] A. X. Lee, Devin,
Springenberg, Y. Zhou, T. Lampe, A. Abdolmaleki, and K. Bousmalis. How to spend your robot time: Bridging kickstarting and offline reinforcement learning for vision-based robotic manipulation. 2022.
[17] T. 7. Zhao, V. Kumar, S. Levine, and C. Finn. Learning fine-grained bimanual manipulation with low-cost hardware. arXiv preprint arXiv:2304. 3...0041375555 13705, 2023.
[18] C. Chi, S. Feng, Y. Du, 7. Xu, E. Cousineau, B. Burchfiel, and S. Song. Diffusion policy: Visuomotor policy learning via action diffusion. arXiv preprint arXiv:2303.04137, 2023.
[19] I.. Pinto and A. Gupta. Supersizing Self-supervision: Learning to Grasp from 50K Tries and 700 Robot Hours. IEEE International Conference on Robotics and Automation (ICRA), 2016.
[20] S. Levine, P. Pastor, A. Krizhevsky, and D. Quillen. Learning Hand-Eye Coordination for Robotic Grasping with Deep Learning and Large-Scale Data Collection. International Journal of Robotics Research, 2017.
[21] D. Kalashnikov, A. Irpan. P. Pastor, J. Ibarz, A. Herzog, E. Jang, D. Quillen, E. Holly, M. Kalakrishnan, V. Vanhoucke, et al. Scalable Deep Reinforcement Learning for Vision- Based Robotic Manipulation. In Conference on Robot Learning, pages 651-673, 2018.
[22] A. Gupta, A. Murali, D. Gandhi, and L. Pinto. Robot learning in homes: Improving generalization and reducing dataset bias. 7 2018. URL http://arxiv.ortt http:/wwwwwg//bb//.....7.. 07049.
[23] S. Dasari, Ebert, S. Tian, S. Nair, B. Bucher, K. Schmeckpeper, S. Singh, S. Levine, and C. Finn. Robonet: Large-scale multi-robot learning. arXiv preprint arXiv: arXiv: 191.. 12/5, 2019.
[24] P. Agrawal, A. Nair, P. Abbeel, J. Malik, and S. Levine. Learning to Poke by Poking: Experiential Learning of Intuitive Physies. In Advances in Neural Information Processing Systems, 2016. URL
www.////g///.../////// 07419.
[25] L. Pinto and A. Gupta. Learning push by grasping: Using multiple tasks for effective learning. In 2017 IEEE international conference on robotics and automation (ICRA), pages 2161-2168. IEEE, 2017.
[26] C. Finn, I. Goodfellow, and S. Levine. Unsupervised learning for physical interaction through video prediction. Advances in neural information processing systems, 29, 2016
[27] A. Nair, D. Chen, P. Agrawal, P. Isola, P. Abbeel, J. Malik, and S. Levine. Combining self-supervised learning and imitation for vision-based rope manipulation. In 2017 IEEE international conference on robotics and automation (ICRA), pages 2146-21111 2153. IEEE, 2017.
[28] T. Yu, C. Finn, A. Xie, S. Dasari, T. Zhang, P. Abbeel, and Levine. One-shot imitation from observing humans via domain-adapaan meta- metannea.,,ng. arXiv preprint trXiv:1802.01557 2018.
[29] S. Cabi, S. G. Colmenarejo, A. Novikov, K. Konyushkova, S. Reed, R. Jeong, K. Zolna, Y. Aytar, D. Budden, M. Vecerik, et al. Scaling Data-Driven Roboties With Reward Sketching and Batch Reinforcement Learning, arXiv preprint arXiv: 1909. / /2200, 2019.
[30] D. Kalashnikov, J. Varley, Y. Chebotar, B. Swanson, R. Jonschkowski, C. Finn, S. Levine, and K. Hausman. MT-Opt: Continuous Multi-Task Robotic Reinforcement Learning at Scale. 2021.
[31] Y. Chebotar, K. Hausman, Y. Lu, T. Xiao, D. Kalashnikov, J. Varley, A. Irpan, B. Eysenbach, R. Julian, C. Finn, and Levine. Actionable Models: Unsupervised Offline Reinforcement Learning of Robotic Skills. 4 2021.
10

---

[32] H. Bharadhwaj. J. Vakil, M. Sharma, A. Gupta, S. Tulsiani, and V. Kumar. Roboagent: Towards sample efficient robot manipulation with semantic augmentations and action chunking. arxiv, 2023.
[33] K. Bousmalis, G. Vezzani, Rao, C. Devin, A. X. Lee, M. Bauza, T. Davehev, Y. Zhou, A. Gupta, A. Raju, et al. Robocat: A self-improving foundation agent for robotic manipulation. arXiv preprint arXiv:2306. / 1706, 2023.
[34] S. Song, A. Zeng, J. Lee. and T. Funkhouser. Grasping in the wild: Learning 6dof closed- loop grasping from low-cost demonstrations. IEEE Robotics and Automation Letters, 5(3): 4978 4985, 2020.
[35] S. Young, D. Gandhi, S. Tulsiani, A. Gupta, Abbeel, and L. Pinto. Visual imitation made easy. In Conference on Robot Learning, pages 1992-2005. PMLR, 2021.
[36] A. Kumar, A. Singh, F. Ebert, Y. Yang, Finn, and S. Levine. Pre-training for robots: Offline rl enables learning new tasks from a handful of trials. arXiv preprint arXiv:2210.05177. 2022.
[37] Kostrikov, A. Nair, and S. Levine. Offline Reinforcement Learning With Implicit Q-
Learning- arXiv preprint /0.1v:2/10006666 2021.
[38] Hansen-Estruch, I. Kostrikov, M. Janner, J. G. Kuba, and S. Levine. Idql: Implicit q-learning as an actor-critic method with diffusion policies. arXiv preprint arXiv:2304. 2023.
[39] B. Eysenbach, Zhang, S. Levine, and R. R. Salakhutdinov. Contrastive learning as goal- conditioned reinforcement learning. Advances in Neural Information P Processing Systems, 35: 35603-35620, 2022.
[40] H.-S. Fang. H. Fang, Z. Tang, J. Liu, Wang. H. Zhu, and C. Lu. Rh20t: A robotic dataset for learning diverse skills in one-shot. In RSS 2023 Workshop on Learning for Task and Motion Planning, 2023.
[41] N. M. M. Shafiullah, J. Cui, A. Altanzaya, and L. Pinto. Behavior transformers: Cloning modes with one stone. In Thirty-Sixth Conference on Neural Information Processing Systems, 2022. URL https: http:///paeerereeeeeee/oa/an/aaa
[42] J. Ho, A. Jain, and P. Abbeel. Denoising diffusion probabilistic models. Advances in Neural Information Processing Systems, 33:6840-6851, 2020.
[43] K. Sohn, X. Yan, and H. Lee. Learning Structured Output Representation using Deep Conditional Generative Models. In Advances in Neural Information Processing Systems, 2015.
[44] B. Eysenbach, R. Salakhurdinov, and S. Levine. C-learning: Learning to achieve goals via recursive classification. In International Conference on Learning Representations, 2021. URL.
[45] C. Zheng, B. Eysenbach, H. Walke, P. Yin, K. Fang, R. Salakhutdinov, and S. Levine. Stabilizing contrastive rl: Techniques for offline goal reaching, 2023.
[46] S. Stepputtis, Campbell, M. Phielipp, Lee, C. Baral, and H. B. Amor. Language- conditioned imitation learning for robot manipulation tasks. ArXiv, abs/2010. 12083, 2020.
[47] Y. Yang, Cer, A. Ahmad, M. Guo, J. Law, N. Constant, G. H. Abrego, S. Yuan, C. Tar, Y.-H. Sung, et al. Multilingual Universal Sentence Encoder for Semantic Retrieval, July 2019.
[48] E. Perez, F. Strub, H. de Vries, V. Dumoulin, and A. Courville, FiLM: Visual Reasoning with a General Conditioning Layer, Dec. 2017.
[49] M. Tan and Q. Le. Efficientnet: Rethinking model scaling for convolutional neural networks.
In International conference on machine learning, pages 6105-61 14. PMLR, 2019.
[50] D. Cer, Y. Yang, S.-y. Kong, N. Hua, N. Limtiaco, R. S. John, N. Constant, M. Guajardo- Cespedes, S. Yuan, C. Tar, et al. Universal sentence encoder for english. In Proceedings of the 2018 conference on empirical methods in natural language processing; system demonstrations, pages 169-174, 2018.
[51] D. Kingma and J. Ba. Adam: A method for stochastic optimization. International Conference on Learning Representations (ICLR), 2015.
11

---

Skill
Pick-and-plase (includes reorienting objects in place)
Pushing objects
Wiping (e.g., wiping the table with cloth)
Sweeping (e.g., sweeping beans into pile)
Stacking
Folding cloths
Opening and closing drawers
Opening and closing doors
Opening and closing cardboard box flaps Twisting knobs Flipping switches Zipping and unzipping Turning levers
Table The full set of skills in BridgeData V2. We define skill to group of trajectories that require similar motions from the robot but may include different objects and start/end positions.
Over-the-Shoalder Randomized (13,299)
Sver-the-Shoulder Randomized, Depth (5.819)
Over-the-Shoulder Randemized, Wrist (5,533)
Overthhe-hhadl Shoulder Only (33,245)
Over-the Shoulder,
Randomized, Depth, Wrist (2,315)
Figure 6 Breakdown of the entire dataset, including the autonomously collected data, by what camera views are included. "Over-the-shoulder" refers to the primary fixed camera, and "random- ized" refers to the two alternative camera views that are randomized by the data collectors every 50 trajectories. "Depth". when present, is from the same perspective as the primary fixed camera. "Wrist" refers to the wide-angle wrist-mounted camera. More cameras were added the hardware setup throughout data collection, so the majority of the data only includes the primary fixed camera view, and very little data currently includes all 4 views. However, now that the hardware is in place, more and more data will include all 4 views as the dataset continues to grow.
A Data Statistics
We provide a full list of the skills in our dataset (following our definition of skill in Section 3.3) in Table We provide breakdown which portions dataset include which sensors in Figure 6.
B Learning Method Implementation Details
Below list relevant implementation details for each method. The complete set of evaluation tasks shown in Figure 7.
All the goal-conditioned methods take in both observation and goal. During training, the goal associated with an observation is selected by uniformly sampling an observation from the future timesteps in the trajectory.
12

---

Seen Tasks
Unseen Tosks
The complete set of evaluation tasks. The seen tasks are (clockwise): sweeping beans into a pile, putting corn cob in a pot, opening a drawer, putting an eggplant block, flipping a pot upright, folding cloth, and putting a carrot a plate. The unseen tasks are (left to right): putting a marker in a bowl, sweeping rice into a pile, folding a thick cloth, putting a mushroom in a pot, putting a spoon cloth, and wiping the table with cloth.
Figure 7
pot, stacking a
B.1 Goal-conditioned behavior cloning
The observation and goal are stacked channel-wise before being passed into ResNet-34 image encoder. This image encoding is passed through 3 256-unit fully connected layers to predict the robot action. We augment the observation and goal with random crops, random resizing, and color jitter. We use the Adam optimizer [51] with a learning rate of 3e-4. We use linear warmup schedule with 2000 steps.
B.2 Diffusion goal-conditioned behavior cloning
In our implementation, we adopt the behavior cloning with diffusion strategy from the IDQL [38]. However, we not learn a value function. The observation and goal are stacked channel- before being passed into a ResNet-34 image encoder. This image encoding is used to condition a diffusion process that models the action distribution. We use the DDPM (Denoising Diffusion Probabilistic Models) style objective introduced by Ho et al. [42]. We augment the observation and goal with random crops, random resizing, and color jitter. We use the Adam optimizer [51] with learning rate of 3e-4. We use linear warmup schedule with 2000 steps.
B.3 Action Chunking with Transformers
We use the same ACT hyperparameters as the original paper [17] except for the chunk size. The original ACT has chunk size 100 to accommodate for high-frequency control (50Hz) and long trajectories (1000 steps), while in our case the control frequency is 5Hz and the trajectory is much shorter at around 50-100 steps. We therefore reduced the chunk size to 5 and noted better performance. In addition, ACT was originally proposed as a single task method; we modify ACT to make it goal-conditioned. The observation and goal are stacked channel-wise before being passed into a ResNet- 18 image encoder. The ACT policy was trained on a consumer grade workstation with 1x Nvidia 2080Ti GPU for days.
B.4 Contrastive RL
Our implementation of contrastive RL mostly follows the design decisions described in [45] except for the following changes. First, given the observation and goal images, feed them separately through ResNet-34 encoder instead of a 3-layer CNN image encoder to get output encodings. Those image encodings then pass through the goal. Second, we share the ResNet-34 encoder between the value function and the policy. Intuitively, this encourages sharing of the causal information learned by the representations and speeds up learning. This ResNet backbone helps the method to consume large amounts of training data. Third, we increase the GCBC regularization coefficient to 0.2 to avoid sampling out-of- distribution actions and use the same batch size other methods to make fair comparisons. Our contrastive RI. objective retains the temporal-differnnce (TD) style used in [44]. We use the Adam optimizer [51] with a learning rate of 3e-4. We use linear warmup schedule with 2000 steps.
MLPs to get representations of the observation and
13

---

Task
BridgeData VI PTR
BridgeData V2
Put marker bowl Put mushroom in pot
0.05
0.65
0.10
0.70
Average
0.08
0.70
Unseen objects, unseen environment Seen objects, unseen environment
Table 6 (Dataset size and diversity ablation) Comparison of the performance of GCBC trained on only Data VI PTR (13k trajectories, 15 environments, 11 skills) and GCBC trained on BridgeData V2 (60k trajectories, 24 environments, 13 skills). The greater size and diversity of BridgeData V2 enables significantly better generalization these unseen tasks.
B.5 Languagg--onditioned behavior cloning
Our implementation of LCBC uses a ResNet-34 with FiLM conditioning. The language instruction first encoded with frozen MUSE encoder and passed through fully connected layers. The image observation is then passed into the ResNet, which is conditioned on the language embedding using FiLM layers. FiLM layers are applied at the end of every ResNet block to condition on language throughout the network. Finally, the image encoding is passed into fully connected network to predict the action. We use the Adam optimizer [51] with a learning rate of 3e-4. We use linear warmup schedule with 2000 steps.
B.6 RT-1
We use the same length of the transformer from 6 to 15 to accommodate for the longer episode length. We scale each action dimension range -1 and and use vocabulary size of 256 to tokenize the actions.
as the original RT-1 paper [7], except for increasing the sequence
C Hardware Setup
We use an Intel RealSense D435 RGBD camera fixed over-the-shoulder camera view and two Logitech C920 webcams to capture alternative camera views that are randomized during data collection. For the wrist camera, designed custom 3D printed mount attach Raspberry Pi camera module to the gripper. We used Meta Quest VR headset to teleoperate the robot.
D Comparison to the Original Bridge Data
In Table 6, compared the performance of GCBC trained the combination of the original Bridge Data and data collected for PTR [36]. with the performance of GCBC trained on all of BridgeData V2. We find that the additional data released as part this paper significantly improves performance on unseen pick-and-place tasks.
14


Reviews:
[{"review_id": "zreVDbRM0Fe", "reviewer": "Reviewer_AXMo", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "This paper introduces a larger dataset of robotic manipulation named BridgeData V2, which is collected in the real world with natural language labels. BridgeData V2 is suitable for both IL and  offline RL methods. The system setup is low-cost, making it accessible for many researchers.", "summary_of_recommendation": "As stated in weaknesses and questions, my final rating is Weak Reject, but I am glad to hear feedbacks from the authors during the rebuttal.\n\nNew: -> Weak Accept", "technical_quality": "Good: Generally convincing, but some additional experiments or analyses needed.", "clarity_of_presentation": "Good: Minor revision needed to improve organization or clarity.", "potential_impact": "", "robotics_focus": "Sufficient demonstration on hardware", "quality_of_limitations": "Limitations are addressed clearly", "questions_for_rebuttal": "As addressed in Weaknesses, the authors should provide more details about:\n  1. What is the difference between these labs? I think the authors could provide a figure to visualize the difference between labs or a table to list what is different exactly.\n  2. The analysis of experimental results should focus on the datatset, rather than simply comparing the baselines, since the main contribution is the dataset. The authors should also explain why the methods perform much better in unseen environments than in seen environments in Table. 3.\n  3. The authors should do some experiments to study how the diversity of skills Impact the generalization.", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "Strengths:\n  1. Compared to other real-world datasets, BridgeData V2 includes more skills and environments.\n  2. The hardware setup is low-cost and friendly to many researchers.\n  3. The experiments across labs are interesting, demonstrating the diversity and generalization of the proposed dataset.\n\nWeaknesses:\n  1. The authors should provide more details about the experiments across labs. What is the difference between these labs? Are manipulated objects,  background, light, or hardware setup different? These details will make the experimental results more convincing.\n  2. The experimental results in Table. 2, Table. 3 and Table. 4 are confusing. Different types of methods are evaluated for different tasks in this paper. Additionally, in Table. 3, the methods perform much better in unseen environments than in seen environments. RT-1 achieves nearly 1.8x performance improvement, which is weird. The authors should analyze the results with more details.\n  3. The setup of camera views (number of views, provided depth or not) is not the same for all trajectories, which will be burdensome for future researchers in preprocessing data and designing methods.", "extra_scores": {"impact": "3: The work is incremental and unlikely to have much impact, even though it may be technically correct and well executed."}, "extra_text": {}}, {"review_id": "lXanl1YBE4", "reviewer": "Reviewer_SSre", "summary": "", "questions": "", "limitations": "", "rating": 2, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "This paper proposes an extension of the Bridge Dataset (Ebert et al. 2021), with over 50K trajectories from a variety of toy tasks performed by a WidowX 250 arm. The dataset, comprising mostly of human demonstrations and some heuristically collected pick-and-place data, includes image observations, robot actions taken in the environment and human-annotated descriptions of the tasks performed in each trajectory. In addition, the authors provide evaluations of offline learning methods trained on the collected data. ", "summary_of_recommendation": "While the proposed dataset could undoubtedly be useful to some researchers benchmarking offline learning or tabletop manipulation algorithms, I find its contribution to the robot learning community limited due to the choice of robot and tasks. A larger, more diverse dataset or an extended comparative study of applying the dataset in other labs may be a convincing addition which could greatly improve this paper. ", "technical_quality": "Good: Generally convincing, but some additional experiments or analyses needed.", "clarity_of_presentation": "Very Good:  Well organized and clear.", "potential_impact": "", "robotics_focus": "Sufficient demonstration on hardware", "quality_of_limitations": "Limitations are addressed clearly", "questions_for_rebuttal": "While it is true that most methods exhibit non-zero success on generalization tasks, it is somewhat hard to assess the quality of the dataset from the standalone performance of the given methods. How do the various baseline methods perform on other, previous datasets?\n\nWhat tuning and setup challenges, if any, did the other institution (Lab 2) encounter while replicating the environment and evaluating the methods to obtain the results reported in Table 5? \n\n(On a side note, the text only mentions two methods were evaluated in Lab 2, while Table 5 reports results for all methods).", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "Standard benchmarking of learning algorithms is a worthy goal, and this paper takes a step towards better robotic benchmarks. It is evident that much time and effort were poured into the creation of the described dataset, both in planning which tasks and environments to include and in designing the data collection protocol. \n\nThe paper is easy to read and follow, and its contributions are stated clearly and concisely. The inclusion of benchmarking experiments on the collected dataset is a nice addition, and helps with evaluating the usability of the dataset. \n\nIf the dataset is publicly released and continuously supported by the authors, it could be a useful addition to the toolbox of benchmarks available to robotic learning researchers.\n\nNonetheless, I am concerned that the claims of generality and diversity made by the authors of this paper may be overstated. \n\nIn terms of diversity, as mentioned in the limitations section, all tasks require relatively simple manipulation with no dynamic movements or contact-rich interactions between objects. While some of these manipulation tasks are still quite challenging, many of them have been solved individually by approaches that do not require such a large dataset of previous interactions. In addition, the toy kitchen, sink and tabletop environments which constitute the majority of the dataset are not as diverse and different from each other as the authors describe.\n\nRegrading generality, the entire dataset is collected using a single model of robot, which may not be useful or applicable to many tasks outside of the toy domains chosen for the dataset, due to its limited reach, degrees of freedom and carrying capacity. In addition, while this robot may be readily available in North America, it may not be as easy to assemble all the required components to replicate a setup similar to the one presented in this paper in other parts of the world.  I believe that datasets for generalizing robot learning should be applicable to whichever setup a research lab already has installed, or at the very least, utilize a more commonly used or more generally useful robot arm. \n\nFinally, it seems like the dataset is not yet complete - as described in the appendix (and evident in the choice of observations for the methods benchmarked), most trajectories do not have depth images attached, and only use an RGB observation from the shoulder camera. Moreover, as the dataset was collected in a single lab on a single setup, the diversity of lighting conditions and other environment variables may be too small to generalize well. A larger comparison study of the setup in other labs would be more convincing of the usefulness of the dataset.", "extra_scores": {"impact": "3: The work is incremental and unlikely to have much impact, even though it may be technically correct and well executed."}, "extra_text": {}}, {"review_id": "GvNa_rPEEj_", "reviewer": "Reviewer_xJRp", "summary": "", "questions": "", "limitations": "", "rating": 4, "confidence": 5, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "With the motivation of facilitating large-scale robot learning research, the paper introduced a real-world crowdsourced-language-labelled dataset, on an accessible hardware platform, with diverse environments, objects, and skills.\nThey demonstrated that a diverse set of robot policy learning formulations and a different lab can make use of this data.\nThey also provided some empirical evidence that performance scales with dataset size and compute, which motivates future cross-institutional collaboration on large scale robot dataset collection.", "summary_of_recommendation": "I think the project's motivation is spot on and its contributions could be extremely valuable to the community.\nInvestigating datasets are important, challenging to study deeply and broadly, and crucially understudied in the robot learning community, and the field would benefit a lot from more projects in this direction.\nHowever, I do not think the project's ambitious research questions were sufficiently empirically backed up by their experiments.\nFor a weak accept, I would like to see a restructuring of the evaluation section (addressing my **Unsystematic Evaluation** point above).\nFor a strong accept, I would also like to see my statistical confidence concerns due to **breath-over-depth** evaluation design addressed.\nI think the paper can be educational after addressing my concern regarding **usage of non-foundational robot skills**, but that alone will not be enough to change my recommendation.\n\n**Update**\nEvaluation and statistical significance concerns addressed. `Strong Accept`.", "technical_quality": "Excellent: Very convincing and thorough.", "clarity_of_presentation": "Very Good:  Well organized and clear.", "potential_impact": "", "robotics_focus": "Sufficient demonstration on hardware", "quality_of_limitations": "Limitations are addressed clearly", "questions_for_rebuttal": "See questions and concerns in weaknesses above.", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "## Strengths\n\n - **A Robot Dataset Paper**: Datasets are very important but remain understudied within robot learning. I think this project made an important contribution in investigating scaling laws of dataset size, diversity, and compute resources (e.g, parameter count) while keeping in mind challenges specific to robotics (hardware accessibility, diverse algorithm/policy/task specification formulations).\n - **Not just another rigid pick and place dataset**: Includes deformable and granular manipulation.\n - **Careful multi-task environment design**: with multiple possible tasks given the same visual observation.\n - A small point but every multi-task paper should define clearly what they mean when using overloaded terms (e.g., tasks and skills), like this one.\n\n\n## Weaknesses\n\n**Unsystematic Evaluation**: Different tasks have different caveats and challenges (e.g., as explained in L234-L237). This is fine, and explaining this to readers is important. However, each table chose a different set of tasks, while not containing consistent coverage of all skills/objects. This makes the evaluation section confusing, because each new table requires readers to consider that table's tasks' caveats while looking at the numbers.\nFurther, without knowing exactly the degree to which these caveats affect the numbers, it makes it difficult for readers to extract takeaways from the reported numbers.\n\n**Breath over Depth**:\nThe project's main motivation is to investigate how a diverse dataset could facility a diverse group of people with access to different resources to study scalable robot learning.\nGiven this motivation, I think their evaluation design choice of breath over depth (many algorithms under different generalization axes for environments, objects, tasks) is more aligned with the motivation than the reverse.\nBreath over depth is not a weakness, but my main concern is with respect to *confidence*. The number of tasks and the number of episodes per task (10 trials per task) evaluated do not lead to high confidence in reported numbers and, therefore, generalizable conclusions for others to learn from. For instance, I think the scaling law plot (Fig 5, left) is extremely exciting to see. However, this plot was generated using one task, which is to move a spoon. \n - **Selective Breath over Depth**: The authors demonstrated that multiple policy learning formulations could learn from the data. However, to investigate research questions 2, 3, and 4 (in the beginning of section 5), it may not have been necessary to run costly real world evaluations on all 6 approaches for all research questions, allowing more trials to be allocated to each approach (e.g., 20 trials in [18]).\n\n**Non-foundational robot skills**: \nCompared to the zipping/unzipping task, the paper and supplementary website contains many more visualizations and results for other tasks.\nThe dataset also contains many more trajectories for other tasks.\n  - **Visualizations**: For this non-standard task, it will be helpful to also include visualizations of the zipping/unzipping task. From the supplementary website, the `traj_links.csv` only include 100 examples of unzip, and only include the first and last frame, but no \"zip\" task.\n  - **Usage of Non-foundational skills**: While less foundational than object rearrangement, more unique skills like such as zipping and unzipped (I do like this task!) are interesting to study within the context of developing a foundation multi-task model for robotics (and not just a foundation pick and place models, for which we already have good data-efficient solutions).\n    -  Exactly how many zipping/unzipping trajectories are included in this dataset? Were they used for any experiments? Do such a small amount of trajectories for a task add value to the dataset? Including a discussion of these questions will be interesting for future work to understand how they can contribute to BridgeDataV2 (e.g., by adding less studied skills).", "extra_scores": {"impact": "4: The work contains interesting new ideas that will potentially have major impact in robotics or machine learning."}, "extra_text": {}}, {"review_id": "11Zbja-HNu", "reviewer": "Reviewer_dwgC", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "This paper presents a new dataset BridgeDataV2, by the name, is clearly an extended version of the original BridgeData. It contains significantly more trajectories with more environments all collected through human experts compared to the original dataset. As an aim for providing a good source for training multi-task foundational robot policy, the paper further evaluates the dataset with multiple popular multi-task offline RL and language-conditioned methods. Results show that the policies trained on this dataset can offer good generalisation on novel objects and new environments.", "summary_of_recommendation": "Proposes a useful dataset that would be useful for the robotics community; evaluations are clean and complete", "technical_quality": "Good: Generally convincing, but some additional experiments or analyses needed.", "clarity_of_presentation": "Very Good:  Well organized and clear.", "potential_impact": "", "robotics_focus": "Sufficient demonstration on hardware", "quality_of_limitations": "Limitations are addressed clearly", "questions_for_rebuttal": "See review.", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "This paper itself is clearly well written and no doubt would be useful for the robot learning community particularly for the ones focusing on large-scale pre-training methods. Here are some other concerns and questions which hopefully could further improve the paper.\n\n1.\tTransferability of different robot. All trajectories collected seem to be on the same WidowX 250 robot arm, which from my own experience, is not a common robot used in many labs. I am wondering how the policy learned through such robot arm can transfer to other robot arms such as Franka or xArm?\n2.\tDetails on language prompts. The quality and graduality of language prompts will heavily influence the generalisation of the dataset. I am wondering how detailed, or the length of language prompts are for each task? E.g. labelling a task as: make a coffee and first grasp a cup to the coffee machine, then press the button… would lead to very different results.\n", "extra_scores": {"impact": "4: The work contains interesting new ideas that will potentially have major impact in robotics or machine learning."}, "extra_text": {}}]

Author rebuttal:

Meta-review: Summary of the Paper:

The paper introduces BridgeData V2, an expanded robotic manipulation dataset containing real-world data with natural language labels. This dataset, suitable for imitation learning and offline reinforcement learning, promises a more diverse set of trajectories and environments than its predecessor. It is primarily collected using the WidowX 250 robot arm and covers a variety of toy tasks. The cost-effective hardware setup makes it appealing to a wide range of researchers. Furthermore, the paper evaluates the dataset with a variety of multi-task offline RL and language-conditioned methods, aiming to showcase its potential for training multi-task foundational robot policies.

Strengths:

- Rich and Accessible Dataset: BridgeData V2 offers more skills and environments compared to other datasets, making it a valuable resource for robotic research.
- Cost-Effective Hardware Setup: The low-cost hardware setup is lauded by reviewers for making the research more accessible to many.
- Diversity in Manipulation: The dataset includes not just rigid pick and place tasks but also deformable and granular manipulation which is a noted positive.
- Comprehensive Evaluation: The dataset's evaluation using multiple popular offline RL methods provides insight into its usability and potential.

Weaknesses:

- Lack of Detailed Information: Several reviewers pointed out the need for more detailed information on various aspects:
- Experimental setups across different labs.
- Inconsistencies and confusing results in tables (especially Tables 2, 3, and 4).
- Variability in camera setup for different trajectories.
- Limited Diversity and Generality: Concerns were raised about the dataset's diversity, as all tasks seem to be simple manipulations, lacking dynamic movements or interactions between objects.
- Specific Robot Used: The dataset's reliance on the WidowX 250 robot arm might limit its applicability, as this model is not common in many labs. Questions arose regarding the transferability of policies learned on this robot to other models.
- Inconsistency in Evaluation: The presentation of results across different tables made the evaluation section confusing for readers. Moreover, the choice to go breadth-over-depth in an evaluation led to concerns about the statistical confidence in reported numbers.
- Language Prompts Concern: The quality, graduality, and specificity of language prompts can influence the dataset's generalization, but details on this aspect were seen as lacking.

Rebuttal and Discussion:

The authors responded diligently to the questions raised by the reviewers and managed to address several of them. However, not all concerns have been fully resolved, and further revisions are still required.

Overall Recommendation:

Given the feedback from all reviewers, rebuttal, and discussion, the consensus leans towards "Weak Accept" with certain reservations. The dataset, no doubt, adds value to the community, but there are areas in the paper that need revision, clarification, or additional detail. Addressing the mentioned weaknesses, especially in terms of the detailed information and clarifying confusing results, could elevate the paper's impact and acceptance within the community.
Decision: Accept (Poster)