VoxAct-B: Voxel-Based Acting and Stabilizing Policy for Bimanual Manipulation
Authors: I-Chun Arthur Liu
Sicheng He
Daniel Seita
Gaurav S. Sukhatme
Venue: CoRL 2024 (2024)
Keywords: bimanual manipulation
voxel representation
vision language models
TL;DR: We propose VoxAct-B for bimanual manipulation, a language-conditioned, voxel-based method that leverages Vision Language Models to prioritize key regions within the scene and subsequently reconstruct a voxel grid with increased voxel resolution.

Abstract: Bimanual manipulation is critical to many robotics applications. In contrast to single-arm manipulation, bimanual manipulation tasks are challenging due to higher-dimensional action spaces. Prior works leverage large amounts of data and primitive actions to address this problem, but may suffer from sample inefficiency and limited generalization across various tasks. To this end, we propose VoxAct-B, a language-conditioned, voxel-based method that leverages Vision Language Models (VLMs) to prioritize key regions within the scene and reconstruct a voxel grid. We provide this voxel grid to our bimanual manipulation policy to learn acting and stabilizing actions. This approach enables more efficient policy learning from voxels and is generalizable to different tasks. In simulation, we show that VoxAct-B outperforms strong baselines on fine-grained bimanual manipulation tasks. Furthermore, we demonstrate VoxAct-B on real-world $\texttt{Open Drawer}$ and $\texttt{Open Jar}$ tasks using two UR5s. Code, data, and videos are available at https://voxact-b.github.io.

VoxAct-B: Voxel-Based Acting and Stabilizing Policy for Bimanual Manipulation
I-Chun Arthur Liu Sicheng He Daniel Seita' Gaurav S. Sukhatme
Department of Computer Science, University of Southern California
Abstract: Bimanual manipulation is critical to many robotics applications. contrast single-arm arm manipulation, bimanual manipulation tasks are challeng- ing due to higher-dimensional action spaces. Prior works leverage large amounts of data and primitive actions to address this problem, but may suffer from sam- ple inefficiency and limited generalization across various tasks. To this end, we propose VoxAct-B, a sunguae--onditionn Vision Language Models (VLMs) to prioritize key regions within the scene and reconstruct a voxel grid. We provide this voxel grid to our bimanual manipulation policy to learn acting and stabilizing actions. This approach enables more efficient policy learning from voxels and is generalizable different tasks. In simulation, we show that VoxAct-B outperforms strong baselines fine-grained bimanual manipulation tasks. Furthermore, we demonstrate VoxAct-B on real-wor col-world Open rawer and Open Jar tasks using two UR5s. Code, data, and videos are available at https://ooxact-b.iitbbbb
voxel-based method that leverages
Introduction
Bimanual manipulation is essential for robotics tasks, such as when objects too large to be controlled by one gripper or when one arm stabilizes an object of interest to make it simpler for the other arm to manipulate [1]. In this work, we focus on asymmetric bimanual manipulation. Here, "asymmetry" refers to the functions of the two arms, where one is a stabilizing arm, while the other is the acting arm. Asymmetric tasks are common in household and industrial settings, such as cutting food, opening bottles, and packaging boxes. They typically require two-hand coordination and high-precision, fine-grained manipulation, which systems. To tackle bimanual manipulation, some methods [2, 3] train policies large datasets, and some exploit primitive actions [4, 5, 6, 7, 8, 9, 10]. However, they are generally sample inefficient, and using primitives can hinder generalization to different tasks as they are not easily adaptable to other types of tasks.
challenging for current robotic manipulation
To this end, we propose VoxAct-B, novel voxel-based, language-conditionee method for bimanual manipulation. Voxel representations, when coupled with discretized action spaces, can increase sam- ple efficiency and generalization by introducing spatial equivariance into learned system, where transformations of the input lead to corresponding transformations of the output [11]. However, pro- cessing voxels is computationally demanding [12. 13]. To address this, we propose utilizing VLMs to focus on the most pertinent regions within the scene by cropping out less relevant regions. This substantially reduces the overall physical dimensions of the areas used to construct a voxel grid, en- abling an increase voxel resolution without incurring computational costs. To our knowledge, this the first study apply voxel representations in bimanual manipulation. Additionally, Aot-AttB does not rely action primitives, making it more general and applicable
wider range of tasks.
We also employ language instructions and VLMs to determine the roles of each arm: whether they are acting or stabilizing. For instance, in a drawer-opening task, the orientation of the drawer and
"Equal advising
1 &GSS holds concurrent appointments as Professor USC and as an Amazon Scholar, This paper describes work performed at USC and is not associated with Amazon.
8th Conference Robot Learning (CoRL. 2024), Munich, Germany.

---

"Hold the drawer with left hand and open the top/hottom drower with right hand"
Put item Drawer
Open Drawer
Open Drawer
"Open the top drower with right hand put top drower with laft hand"
Simulation
Real World
"Grasp the jor with right hand and grosp
Open Jar
the lid the jor with left hand
Hand Over Item
anti-olackkwse direction
"Grosp block with laft hand and hand over to the right hand"
removed from the jar
detect object crop voxel grld
zoomed-in vaxel grid
Viaion Language Modele
Bimannual Manipulation
Policy
Simulation
Real World
Figure 1: VoxAct-B uses voxel representations and language to perform bimanual manipulation with 6-DoF manipulation from both arms. We test four language-condttioned bimanual tasks in simulation and two (0pen Drawer and Open Jar) on real-world setup with UR5s.
the position of the handle affect which arm is more suitable for opening the drawer (acting) and which is better for holding it steady (stabilizing). We use VLMs to compute the pose of the object of interest relative to the front camera and to decide the roles of each arm. Then, we provide appropriate language instructions to the bimanual manipulation policy to control the acting and stabilizing arms.
We extend the RLBench [14] benchmark to support bimanual manipulation. We introduce a manual version of Open Drawer, Open Jar, Put Item in Drawer, and Hand Over Item tasks. VoxAct-B outperforms strong baselines, such ACT [3]. Diffusion Policy [15], VoxPoser [16], and PerAct [11], by large margin. We also validate approach real-world bimanual manipula- tion setup with two UR5s on Open Drawer and Open Jar. See Figure for overview.
The contributions of this paper include:
VoxAct-B, novel method for bimanual manipulation which uses VLMs reduce the size of
voxel grid for learning with modified, downstream voxel-based behavior cloning method [11].
A suite of rision-language bimanual manipulation tasks, extended from RLBench [14].
Simulation experiments indicating that Act-C achieves state-of-the-art results on these tasks.
Demonstrations of VoxAct-B on real-world bimanual manipulation setup with two UR5s.
2 Related Work
Bimanual Manipulation. There has been much prior work in bimanual manipulation for folding cloth [5, 8, 17, 18, 19, 20, 21], cable untangling [6], scooping [9], bagging [22, 23, 24), throw- ing [25], catching [26], and untwisting lids [27]. Other works study bimanual manipulation with dexterous manipulators [28, 29, 30, 31] mobile robots [32]. In contrast these works, our focus on a general approach to bimanual manipulation with parallel-jaw grippers on fixed-base ma- nipulators. Works that study general approaches for bimanual manipulation include [2, 7, 33], which use primitive actions or skills to reduce the search space across actions. Other general ap- proaches focus on orthogonal tools such as interaction primitives [34] or screw motions [35]. Re- cently, Zhou et al. [3] introduced another general approach, based on "action chunking" to learn high-frequenne controls with closed-loop feedback and applied their method on multiple asymmetric bimanual manipulation tasks using low-cost hardware. Other works extended this by either incorpo- rating novel imitation learning algorithms [36] enhancing the hardware itself [37, However, these works may still require substantial training data and lack spatial equivariance for generaliza- tion. In closely-relate work, Grannen et al. [39] decouple system into stabilizing and acting arms

---

to enable sample-efiicient bimanual manipulation with simplified data collection. While effective, this formulation predicts top-down keypoints and was not tested with 6-DoF manipulation. In con- trast, our method supports 6-DoF manipulation for bimanual manipulation tasks. In independent and concurrent work, Grotz et al. [40] propose PerAct", extending RLBench to bimanual manipula- traguugg--cnndtiion 6-DoF, behavior-cloning agent. This
tion with 13 new tasks and presenting work is complementary to ours, and future work could merge techniques from both works.
Action Space Representation. For 2D manipulation, prior works have shown the benefits of action representations based on spatial action maps [21, 41, 42, 43, 44, 45, 46), including in bimanual con- texts [10, 21], where neural networks directly predict 2D "images" that indicate desired locations for the action. Compared to directly regressing the action location, using spatial action maps better handles multimodality and has 2D equivariance, where translations and rotations of the input image map to similar transformations of the output action. Recent works have extended this idea to support 3D spatial action maps, which classify an action's 's location as 3D point in the robot's workspace, and thus maintain spatial equivariance. For example, PerAct [11] is language- -enggiiccoodd behav- ioral cloning agent that takes voxel grids as input and outputs 6-DoF actions. While PerAct achieved state-of-the performance on RLBench, it has a high computational cost due to processing voxels. Follow-up works, such as RVT [12] and Act3D [13], have reduced the computational cost of PerAct by avoiding voxel representations but often need multiple views of the scene to achieve optimal per- formance and may be less interpretable compared to voxel grid that contains 3D spatial action map. These prior works have not been applied to bimanual manipulation. In this work, we retain the spatial equivariance benefits of voxel representations but reduce the cost of processing voxels by "zooming" into part of the voxel grid. This is similar to the intent of C2F-ARM [47] and RVT-2 [48], but we use the knowledge in VLMs to determine the most relevant regions in the voxel grid.
LLMs and VLMS for Roboties. LLMs and VLMs, such as GPT-4 [49], Llama 2 [50], and Gem- ini [51], have revolutionized natural language processing, computer vision, and robotics due their strong reasoning and semantic understanding capabilities. Consequently, recent work has integrated them in robotics and embodied AI agents, typically as high-level planner [52, 53, 54, 55], which may also produce code for a robot to execute [56, 57, 58). We defer the reader to [59, 60, 61] for representative surveys. Among the most relevant prior works, Huang et al. [16] propose VoxPoser, which uses pre-trained L.MS and VLMS to compose 3D affordance maps and 3D constraint maps, which are then used with motion planning generate trajectories for robotic manipulation. By leveraging LLMs and VLMs, VoxPoser can generalize to open-set instructions and objects. How- ever, as we later demonstrate in experiments, VoxPoser can struggle with tasks that require high precision and contact. In this work, we demonstrate how to use VLMS to effectively process the input of PerAct for bimanual manipulation, obtaining the generalization benefits of VLMs with the precision capabilities of PerAct. In recent and cenercenttrtent work, Varley et al. [62] also uses VLMM for bimanual manipulation. Our work differs in that we do not fix the roles of each arm; we use an off-the-shelf VLM [63] without any fine-tuning, and we do not use a skills library.
3 Problem Statement
Given access to pre-trained VLM and expert demonstrations, the objective to produce a bimanual policy π for a variety of language-condiiioned manipulation tasks. We assume a flat workspace with two fixed-base robot manipulators, each with parallel-jaw gripper. policy π controls both arms by producing actions a (ai, at each time step t, where at and of follow [39] and refer to the stabilizing and acting arm actions, respectively. For simplicity, we suppress the time / when the distinction is unnecessary. We use the low-level action representation from PerAct [1 I] with a" = (afnser 4xpen: @collide) and a² = (afnsey Capen: ( These specify each arm's 's 6-DOF gripper pose, its gripper open state, and whether a motion planner for the arms used collision avoidance to reach an intermediate pose. We assume task-specific demonstrations D. = {Ci, $2....,G,, G.) and two common language commands fas and Exc. where as denotes the left arm acting and right arm as stabilizing, and vice versa for sa. Each demonstration consists of set of keyframes extracted from sequence of continuous actions paired with observations. We adapt the keyframe extraction function
3

---

RGa Image
Zoomned in Vovel Grid
Acting and Stobitting Pollo Au
next
Vocel Fronder
langunge Models
Per Percevee Transformer
Vosel Decoder
metora, vouds
besk yourd
Langunge Goal
Lorguoze
Separ
Live
Cassa 2i Coesse 2i
Text Query
"Huld the dicteor with right hend opinn
Enooder
Opper Cive
Figure 2: Overview of VoxAct-B. Given RGB-D images and language goal. we input an RGB image from frout camera and a text query extracted from the language goal into the Vision Language Models (VLMs). The VLMs output the pose of the object of interest with respect to the front camera. This information determines the language goal and the roles of cach arm (i.c., acting or stabilizzin), Additionally, we use the object's position with the RGB-D images reconstruct voxel grid that spans oz" meters of the workspace using V voxels. The zoomed-in voxel grid, the language goal, proprioception data both robot arms, and an arm are provided to an acting policy n. and a stabilizing policy T.. The policies predict the discretized pose of the next best voxel, gripper open action, collision avoidance flag, and arm ID for fine-graiee fine-grained bimanual manipulation.
from [11] by including keyframes that have action with near-zero joint velocities and unchanged gripper open state for acting and stabilizing arms. The observation at each time is the 3D voxel grid v of dimension (L x W x H), where we use v(zz y, 2] to denote an individual voxel at coordinates (I, y: 2). The voxel grid is reconstructed from RGB-D sensors. The robot also receives the language command {fas; which fixed for all time steps in an episode, where the robot interacts with the environment for up to T time steps. An episode terminates with task-dependent success
criteria or failure (if otherwise).
4 Method
4.1
Extending PerAct for Bimanual Manipulation
PerAct [11] was originally designed and tested for single-arm manipulation. We extend it to support bimanual manipulation. A natural way do this would be train separate policies for the two arms. However, exploit the discretized action space that predicts the next best voxel with spatial equivariance properties and formulate a system that uses acting and stabilizing policies. In contrast to a policy that operates in joint-space control, acting and stabilizing policies perform the same functions irrespective of whether it is a left arm or a right arm, assuming the next best voxel is kinematically feasible for both arms. This policy formulation enables more efficient learning from multi-modal demonstrations compared to a joint-space control policy. In the low-level action space, the arms execute one low-level action at and af (see Section 3) at each time t. In the following, similar notation as [11] but index components belonging to arm using the superscript: arm E (acting, stabilizing).
At each time step, the input to each arm is a voxel observation v. proprioception data of both robot arms p. a language goal I E {faz: }, and an arm ID g E {0, 1). and the task to predict an action. During training, the language goal given in the data, but during evaluation, we use VLMs to determine which language goal, las or l.a, to use based on the given task. If the language goal is las. we assign the left arm (E 0) to the acting policy and the right arm (E = 1) to the stabilizing policy, and conversely for lan. This allows our method to learn to map the appropriate acting or stabilizing actions to given arm during training. Note that the predicted arm ID is discarded during evaluation. PerAct uses value maps to represent different components of the action space, where predictions for each arm are Q-functions with state-action values. Formally, we have the following five value maps per arm, as the output of the arm's 's learned deep neural network, where:
softmax
Qam ((x, y, z) =) p, 1))
yarm = softmax Qurm (( A, ,)lv, 2.))
trans
softmax( Qam w. E))
collide softmax Qarn p,
open
Jurn
softmax Q (ulv, 1))

---

ROB Image
Output Visualization
ROB Image
Output Visualization
Segment Anything
OWL VIT
Predicted bounding box from OWL VIT
crop center [x yl
Text Query "Jor"
preclicted bounding box [ex, cy. h]
1.0
0.3
0.1
Figure 3: Top: VLMs usage as part of VoxAct-B, visualizing the Open Jar task in simulation, showing the role of OWL-ViT and Segment Anything. The RGB images from the front camera shown above are examples of actual (uncropped) images provided input to the models. Bottom: visualization of different values resulting in coarser grids (a 1.0) finer grids (a 0.1). We o). w. K, and represent, respectively. the 3D position, the discretized Euler angle rotations, the binary gripper opening state, the binary collision variable, and the binary arm ID. At test time, select each arm's action, perform "argmax" over all the input variables to the arm's 's five Q-value, to get the five components. We refer the reader to [11] for more details.
0.3 for Open Jar.
and where (z, 2). (v.0,00
The demonstrations provide labels for each arm's five action components, giving us the following nine label sources: tmins E L xH for translations, Your E (3660/8)x3 (with R = 5 or 5-degree bins) for discretized rotations, Yupen IR2 for the binary open variables, yollide E R2 for the binary collide variables, and Yid IR2 for the binary arm ID variables. The overall training loss for VoxAct-B is:
Liotal Cacting + Casbillizin
(1)
and where for both values of arm E {acting, stabilizing). we have
Carn - Eyam log 1
parm trans.
Eyan log Vist
Eyam log Vapenl
Eyam [og Vellliee collide! - -y [log Via
(2)
which consists set cross-entropy classifier-style losses for each component action.
4.2 VoxAct-B: Voxel Representations and PerAct for Bimanual Manipulation
When using voxel representations for fine-grained manipulation, high voxel resolution is essential. While
increase the number of voxels, this would consume more memory, slow down train- ing, and adversely affect learning the policy is optimizing over larger state space. Therefore, given a voxel grid observational input v of size (L x W x H) that spans x3 meters of the workspace, we keep the number of voxels the same but reduce the relevant workspace. We use VLMs to de- voxel
tect the object of interest the scene and "crop" the grid around this object, resulting fraction that determines the size of the crop. This allows zooming into the more important region of interest. The voxel resolution becomes L. W H. voxels/meters from the original resolution of W H voxels/meters.
grid that spans a. meters of the workspace, where
(o
To detect the object of interest reliably, we use two-stage approach similar to [16]. We input text query and a RGB image from the front camera to OWL-VIT [64], an open-vocabulary object detector, to detect the object. Then, we use Segment Anything [65], a foundational image segmen- tation model, obtain the segmentation mask of the object and use the mask's centroid along with point cloud data, obtained from the front camera's 's RGB-D image, retrieve the object's pose with respect the front camera. We use the pose of the object to determine the task task-specific roles of each arm and the language goal. This cropped voxel grid and language goal are the input our bimanual
5

---

manipulation policy. We call our method VoxAct-B: Voxel- Bree-BBaeddd- Acting and Stabilizing Policy. See Figures and for an overview. Additional implementation details can be found Appendix A.1.
Experiments
For simulation experiments, we build on top of RLBench [14], a popular robot manipulation bench- mark widely used prior work, including VoxPoser and PerAct. We extend it to support bimanual manipulation and introduce additional variations (see Appendix A.2 for details), making the tasks more challenging. We design the following four bimanual tasks:
jar with screw-on lid is randomly spawned and scaled from 90% to 100% of the original size within the robot's 's 0.43 x 0.48 meters of workspace. The robot must grasp the jar with one hand and use the other to unscrew the lid in an anti-clockwise direction until it is removed.
Open Jar:
Open Drawer: drawer is randomly spawned inside a workspace of 0.65 x 0.91 meters. It is randomly scaled from 90% to 100% of its original size, and its rotation is randomized between - and E radians. The robot needs to stabilize the top of the drawer with one hand and then open the bottom drawer with the other.
Put Item in Drawer: drawer (the same type from Open Drawer) is randomly spawned in workspace of 0.65 0.91 meters, and is randomly scaled and rotated using the same sampling ranges from Open Drawer. The robot needs to open the top drawer with one hand, grasp the item
placed on top of the drawer with the other hand, and place it in the top drawer.
Hand Over Item: a block is randomly spawned in a workspace of 0.43 x 0.48 meters. The robot needs to grasp a block with one hand and hand it over to the other.
See Figure an illustration. In the real world, we test Open Jar and Open Drawer using coffee jar with dimensions 3.35 2.85 4.8 inches and a drawer of dimensions 12 12 12 inches. Note that the real-world jar and drawer cannot be opened without the use of second arm.
5.1 Baselines and Ablations
In simulation, we compare against several strong baseline methods: Action Chunking with Trans- formers (ACT) [3], Diffusion Policy [15], and VoxPoser [16]. ACT is a state-of-the-art method for bimanual manipulation. Diffusion Policy represents the policy as a conditional denoising diffu- sion process and excels learning multimodal distributions. ACT and Diffusion Policy use joint positions for their action space instead of predicting end-effector poses as our method. We adapt the Mobile ALOHA repository for ACT and CNN-based Diffusion Policy, and tune their pa- rameters (e.g., chunk size and action horizon) to improve performance. For VoxPoser, write and tune their LLM prompts to work on our bimanual manipulation tasks using the VoxPoser repository. Additionally, we include Bimanual PerActs baseline, which trains separate PerAct policies for the left and right arms, to show how a straightforward bimanual adaptation of a single-arm, state- of-the-art voxel-based method performs. uses the same number of voxels, 1003, as the original PerAct. See the Appendix for further details. We also test the following ablations of VoxAct-B:
VoxAct-B w/o VLMs: does not use the VLMs detect the object of interest and crop the voxel grid. It uses the same number of voxels as method and the default workspace dimensions. VoxAct-B w/o Segment Anything: uses the bounding box obtained from OWL-ViT to compute the object's centroid.
VoxAct-B w/o acting and stabilizing formulation: trains left-armed policy for left arm actions and a right-armed policy for right arm actions. Otherwise, is the same as VoxAct-B. VoxAct-B w/o arm ID: disables the arm ID loss function.
5.2 Experiment Protocol and Evaluation
To generate demonstrations in simulation, follow the convention from RLBench and define sequence of waypoints to complete the task, and use motion planning to control the robot arms

---

to reach waypoints. We generate 10 and 100 demonstrations of training data. Half of this data consists of left-acting and right-stabilizing demonstrations, and the other half contains right-acting and left-stabilizing demonstrations. We generate 25 episodes of validation and test data using dif- ferent random seeds. We train and evaluate methods using five random seeds and average the results. We evaluate all methods on the same set of test demonstrations for fair comparison. See Appendix A.1 for details on how checkpoint selection is done in VoxAct-B.
In the real world, we use a dual-arm CB2 UR5 robot setup. Each arm has 6-DOFs and has Robotiq 2F-85 parallel-jaw gripper. We collect ten demonstrations for each task with the GELLO telcoperation interface [66] for policy training. We use a flat workspace with dimension 0.97 m by 0.79 m and mount an Intel RealSense D415 RGBD camera reduces occlusions of the object. For evaluation, perform 10 rollouts per task. In Open Drawer, the arms have fixed roles of right acting and left stabilizing, and the acting arm opens the top drawer. The drawer has variations of c0 in translations and 20 of rotations. In Open Jar, conducted two experiments: one has the roles of the arms reversed and fixed and the other has unfixed roles of each arm. The jar has variations of 12 cm in translations. See Appendix for more details.
height of 0.42 at a pose which
6 Results
6.1 Simulation Results
Comparisons with baselines. Table I reports the test success rates of baselines and VoxAct-B, When we train all methods using ten demonstrations, VoxAct-B outperforms all baselines by a large margin. In low-data regime, the discretized action space with spatial equivariance properties (as used in VoxAct-B and Bimanual PerActs) may be more sample-efficient and easier for learning-base based methods compared to methods that use joint space (ACT and Diffusion Policy). When we train all methods using more demonstrations (100), VoxAct-B still outperforms all baselines. Through ablations of ACT and Diffusion Policy, we found that removing environment variations greatly improved their performance. We attribute the tasks' difficulty to the following: high environment variation, difficult bimanual manipulation tasks with digh-dimensional action spaces and fine-graines manipulation, and the two types of training data that each method needs to learn (based on which arms are acting and stabilizing).
Open Jar
Open Drawer
Put Item in Drawer
Hand Over
Item
Method
10
100
10
100
10
100
10
100
Diffusion Policy
4.8
21.6
4.8
5.6
2.4
4.8
0.0
0.0
ACT w/ Trannformmers
4.0
30.4
12.8
28.0
8.8
44.8
1.6
7.2
VoxPoser Bimanual PerActs
8.0
8.0
32.0
32.0
4.0
4.0
0.0
0.0
8.0
36.8
5.6
0.0
VoxAct-B (ours)
40.0
59.2
73.6
72.8
39.2
49.6
19.2
14.4
Table 1: Performance of different methods on bimanual manipulation tasks in simulation, based on 10 or 100 (task-specific) training demonstrations. We use five training seeds for all methods, and evaluate the same 25 episodes of seen test data using the best checkpoints from validation (Section 5.2). The results are the average evaluation over five seeds. We only test Bimanual Peracts with ten demonstrations (not 100) due to computational constraints. VoxPoser does not have training, its 10 and 100 results are identical.
Qualitatively, baseline methods, especially Vox- Poser, typically struggle with precisely grasping objects such drawer handles and jars. The base- lines also struggle with correctly assigning the roles of each arm. For instance, a policy intended to execute acting actions may unpredictably produce stabilizing actions. Furthermore, they can generate kinematically infeasible actions or actions that lead erratic movements, as seen in ACT and Diffusion Policy, which may be caused by insufficient training data. In contrast, we observe fewer of these errors with VoxAct-B.
Open Drawer
Methad
Aox-ct-B w/o VLMs
19.2
VoxAct-B w/o Segment Anything
67.2
VoxAct-B w/o acting and stabilizing
64.8
VoxAct-B w/o arm ID
68.0
VoxAct-B (ours)
73.6
Table 2: Ablation experiment results in simulation.
7

---

Oben Drower
Open Ja
Figure Example successful rollouts (one per row) VoxAct-B on a real-world bimanual setup with UR5s.
Ablation experiments. Table reports results on Open Drawer in simulation, based on 10 training demonstrations and evaluated across five training seeds. We use the same training and evaluation protocols Table 1. VoxAct-B w/o VLMs performs poorly versus VoxAct-B because, without ing the VLMs reduce the physical space, the voxel resolution is lower due to the large workspace area for each individual voxel, which hinders fine-grained manipulation. VoxAct-B w/o Segment Anything performs worse than VoxAct-B because the predicted object centroid from Segment Any- thing is closer to the ground truth centroid obtained from the simulator, which is used for training VoxAct-B, than OWL-ViT's. We found that the closer the predicted object centroid truth, the better the policy performs. Moreover, VoxAct-B w/o acting and stabilizing and VoxAct-B w/o arm ID perform worse than VoxAct-B, and they struggle with the same issues as the baselines. See Appendix A for additional experiments and details.
the ground
6.2 Physical Results
Figure shows real-world examples of VoxAct-B. In Open Drawer, success when the stabilizing arm holds the drawer from the top while the acting arm pulls the top part. VoxAct-B succeeds in out of 10 trials; the failures include robot joints hitting their limits, imprecision in grasping the handle, and collisions with the drawer. Open Jar, a success is when the stabilizing arm grasps the jar while the acting arm unscrews the lid. For the experiment with fixed roles of each arm, VoxAct-B succeeds in 5 out of 10 trials. While the stabilizing arm performs well in grasping the jar (9 out of 10 successes), the acting arm struggles with unscrewing the lid, succeeding only 5 out of 10 times due to imprecise grasping of the lid. For the experiment with unfixed roles of each arm, we train VoxAct-B on 10 left-acting, right-stabilizing and 10 right-acting, left-stabilizing demonstrations. succeeds in out of 10 trials, demonstrating its ability to learn from
real- wolld data.
6.3 Limitations and Failure Cases
VoxAct-B implicitly assumes the object of interest does not encompass most of the workspace. If does, it will be difficult to crop the voxel grid without losing relevant information. Another limitation is that VoxAct-B depends on the quality of VLMs. We have observed that some failures come from poor detection and segmentation from VLMs. which causes VoxAct-B to output undesirable ac- tions. In addition common errors described Section 6.1, Put Item Drawer, Vox tends to struggle more with executing acting actions (e.g., drawer-opening and pibbeeppckin//lacinn actions) in contrast to stabilizing actions.
7 Conclusion
In this paper, we present VoxAct-B, voxel-based, language-conditioned method for bimanual ma- nipulation. We use VLMs to focus on the most important regions in the scene and reconstruct a voxel grid around them. This approach enables the policy to process the same number voxels within a reduced physical space, resulting in a higher voxel resolution necessary for accurate, fine- greigrained bimanual manipulation. VoxAct-B outperforms strong baselines, such as ACT, Diffusion Policy, and VoxPoser, by large margin difficult bimanual manipulation tasks. We also demonstrate VoxAct-B real-world Open Drawer and Open Jar tasks using dual-arm URS robot. We hope that this inspires future work in asymmetric bimanual manipulation tasks.

---

Acknowledgments
We thank our colleagues Gautam Salhotra, Kr Zentner, Yigit Korkmaz, Yunshuang Li, and Guangyao Shi for helpful writing feedback.
References
[1] F. Krebs and 'T. Asfour. A Bimanual Manipulation Taxonomy. In IEEE Robotics and Automa- fion Letters (RA-L), 2022.
[2] R. Chitnis, S. Tulsiani, S. Gupta, and A. Gupta. Efficient bimanual manipulation using learned task schemas. IEEE International Conference on Robotics and Automation (ICRA), 2020.
[3] T. Z. Zhao, V. Kumar, S. Levine, and C. Finn. Learning Fine-Grained Bimanual Manipulation with Low-Cost Hardware. Robotics: Science and Systems (RSS), 2023.
[4] G. Franzese, L. d. S. Rosa, T. Verburg, L. Peternel, and J. Kober. Interactive imitation learning of bimanual movement primitives. IEEE/ASME Transactions on Mechatronics, 28(1):1-13,
2023.
[5] Y. Avigal, L. Berscheid, T. Asfour, T. Kröger, and K. Goldberg. SpeedFolding: Learning Efficiint Bimanual Folding of Garments. In IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS), 2022.
[6] Grannen, P. Sundaresan, B. Thananjeyan, Ichnowski, A. Balakrishna, V. Viswanath,
M. Laskey, E. Gonzalez, and K. Goldberg. Untangling dense knots by learning task-relevant keypoints. In Conference Robot Leurning (CoRL), 2020.
[7] Xie, Chowdhury, M. C. De Paolis Kaluza, L. Zhao, L. L. Wong, and R. Yu. Deep imi- tation learning for bimanual robotic manipulation. In Neural Information Processing Systems (NeurIPS), 2020.
[8] C. Bersch, B. Pitzer, and Kammel, Bimanual robotic cloth manipulation for laundry folding. In IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS), 2011.
[9] J. Grannen, Y. Wu, S. Belkhale, and D. Sadigh. Learning Bimanual Scooping Policies for Food Acquisition. In Conference on Robot Learning (CoRL), 2022.
[10] H. Ha and S. Song. Flingbot: The unreasonable effectiveness of dynamic manipulation for cloth unfolding. In Conference on Robot Learning (CoRL), 2021.
[11] M. Shridhar, L. Manuelli, and D. Fox. Perceiver-actor: A multi-task transformer for robotic manipulation. In Conference on Robot Learning (CoRL), 2022.
[12] A. Goyal, Xu, Guo, V. Blukis, Y.-W. Chao, and D. Fox. RVT: Robotic View Transformer for 3D Object Manipulation. In Conference on Robot Learning (CoRL), 2023.
[13] T. Gervet, Z. Xian, N. Gkanatsios, and K. Fragkiadaki. Act3D: 3D Feature Field Transformers for Multi-Task Robotic Manipulation. Conference on Robot Learning (CoRL), 2023.
[14] James, Z. Ma, D. Rovick Arrojo, and J. Davison. Rlbench:
The robot learning benchmark & learning environment. IEEE Robotics and Automation Letters, 2020.
[15] C. Chi, S. Feng Y. Du, 7. Xu, E. Cousineau, B. Burchfiel, and S. Song. Diffusion Policy: Visuomotor Policy Learning via Action Diffusion. In Robotics: Science and Systems (RSS), 2023.
[16] W. Huang, C. Wang, R. Zhang, Y. Li, Wu, and Fei-Fei. Voxposer: Composable 3d value maps for robotic manipulation with language models. In Conference Robot Learning (CoRL), 2023.

---

[17] J. Maitin-Shepare M. C J. Lei, and P. Abbeel. Cloth Grasp Point Detection Based on Aultiple-View Geometric Cues with Application to Robotic Towel Folding. In IEEE International Conference on Robotics and Automation (ICRA), 2010.
[18) A. Canberk, C. Chi, H. Ha, B. Burchfiel, E. Cousineau, S. Feng, and S. Song. Cloth funnels: anoonicalizedallignmeee for multi-purpose garment manipulation. In IEEE International Con- ference on Robolics and Automation (ICRA), 2022.
[19] A. Colomé and C. Torras Dimensionality reduction for dynamic movement primitives and application to bimanual manipulation of clothes. In IEEE Transactions on Robotics, 2018.
[20] G. Salhotra, I.CC. A. Liu, and G. Sukhatme. Learning robot manipulation from cross- morphology demonstration, In Conference on Robot Learning (CoRI.), 2023.
[21] Weng, S. Bajracharya, Y. Wang, K. Agrawal, and Held, Fabricflownet: Bimanual cloth manipulation with flow-based policy. In Conference on Robot Learning (CoRL), 2021.
[22] I.. Y. Chen, B. Shi, D. Seita, R. Cheng, T. Kollar, D. Held, K. Goldberg, K. Goldberg, K. Gold- berg, K. Goldberg, K. Goldberg, and K. Goldberg. AutoBag: Learning to Open Plastic Bags and Insert Objects. IEEE International Conference on Robotics and Automation (ICRA), 2023.
[23] L. Y. Chen, B. Shi, R. Lin, D. Seita, A. Ahmad, R. Cheng, T. Kollar, D. Held, and K. Gold- berg. Bagging by Learning to Singulate Layers Using Interactive Perception. In JEEE/RSJ International Conference on Intelligent Robots and Systems (IROS), 2023.
[24] A. Bahety, S. Jain, H. Ha, N. Hager, B. Burchfiel, E. Cousineau, S. Feng, and S. Song, Bag All You Need: Learning a Generalizable Bagging Strategy for Heterogeneous Objects. In JEEE/RSJ International Conference on Intelligent Robots and Systems (IROS). 2023.
[25] B. Huang, Y. Chen, T. Wang, Y. Qin, Y. Yang. N. Atanasov, and X. Wang. Dynamic handover: Throw and catch with bimanual hands. In Conference on Robol Learning (CoRL), 2023.
[26] L. Yan, T. Stouraitis, J. Moura, W. Xu, M. Gienger, and S. Vijayakumar. Impact-Aware Bi- manual Catching of Large-Momentum Objects. In IEEE Transactions on Robotics, 2024.
[27] Lin, Z.-H. Yin, H. Qi. Abbeel, and J. Malik. Twisting Lids Off with Two Hands. arXiv preprint arXiv:2403.0253 023388 2024.
[28] Y. Chen, Yang. T. Wu, S. Wang, X. Feng, J. Jiang, S. M. McAleer, H. Dong, Z. Lu, and S.-C. Zhu, Towards Human-Leva Bimanual Dexterous Manipulation with Reinforcement Learning- In Neural Information Processing Systems (NeurIPS), 2022.
[29] T. Lin, Y. Zhang, Q. Li, H. Qi, B. Yi, S. Levine, and J. Malik. Learning Visuotactile Skills with Two Multifingered Hands. arXiv preprint arXiv:2404. 16823, 2024.
[30] K. Zakka, P. Wu, L. Smith, N. Gileadi, T. Howell, X. B. Peng, S. Singh, Y. Tassa, P. Flo- rence, A. Zeng, and P. Abbeel. Robopianist: Dexterous piano playing with deep reinforcement learning. In Conference on Robot Learning (CoRL), 2023.
[31] C. Wang, H. Shi, W. Wang, R. Zhang, L. Fei-Fei, and C. K. Liu. DexCap: Scalable and Portable Mocap Data Collection System for Dexterous Manipulation. arXiv preprint arXi::2403.07788 2024.
[32] J. Yang, C. Deng, J. Wu, R. Antonova, L. Guibas, and J. Bohg. EquivAct: SIM(3)-Equvvariant Visuomotor Policies beyond Rigid Object Manipulation. arXiv preprint 1oXiv:2310116050 2023.
[33] K. Chu, X. Zhao, C. Weber, M. Li, W. Lu, and S. Wermter. Large Language Models for Orchestrating Bimanual Robots. arXiv preprint arXiv:2404.02018, 2024.
10

---

[34] S. Stepputtis, M. Bandari, S. Schaal, and H. Ben Amor. A System for Imitation Learning of Contact-Rich Bimanual Manipulation Policies. In IEEF/SS International Conference on Intelligent Robots and Systems (IROS), 2022.
[35] A. Bahery, P. Mandikal, B. Abbatematteo, and R. Martin-Martin. ScrewMimic: Bimanual Imitation from Human Videos with Screw Space Projection. In Robotics: Science and Systems (RSS), 2024.
[36] L. X. Shi, A. Sharma, T. Z. Zhao, and C. Finn. Waypoint-based imitation learning for robotic manipulation. In Conference on Robot Learning (CoRI.), 2023.
[37] Z. Fu, T. Z. Zhao, and C. Finn. Mobile aloha: Learning bimanual mobile manipulation with low-cost whole-hody teleoperation. In arXiv preprint arXiv:2401.02117. 2024.
[38] A. Team. Aloha An enhanced low-cost hardware for bimanual teleoperation, 2024. URL https://aloha- 2. .ai.oooaaa....u.b.b....
[39] J. Grannen, Y. Wu, B. Vu, and D. Sadigh. Stabilize to act: Learning to coordinate for bimanual manipulation. In Conference on Robot Learning (CoRL), 2023.
[40] M. Grotz, M. Shridhar, T. Asfour, and D. Fox. PerAct2: Benchmarking and Learning for Robotic Bimanual Manipulation Tasks. arXiv preprint arXiv:2407.00274 2024.
[41] A. Zeng, S. Song, K.-T. Yu, E. Donlon, F. Hogan, M. Bauza, D. Ma, O. Taylor, M. Liu, E. Romo Grau, N. Fazeli, F. Alet, N. Dafle, R. Holladay, Morena, P. Nair, D. Green, I. Taylor, W. Liu, and A. Rodriguez. Robotic pick-and-place of novel objects in clutter with multi- affordance grasping and cross-domain image matching. In IEEE International Conference on Robotics and Automation (ICRA), 2018
[42] A. Zeng, S. Song. S. Welker, J. Lee, A. Rodriguez, and T. Funkhouser. Learning Synergies be- tween Pushing and Grasping with Self-supervised Deep Reinforcement Learning. In /EEE/RS) International Conference on Intelligent Robots and Systems (IROS), 2018.
[43] A. Zeng, P. Florence, J. Tompson, S. Welker, J. Chien, M. Attarian, T. Armstrong, I. Krasin, D. Duong, V. Sindhwani, and J. Lee. Transporter networks: Rearranging the visual world for robotic manipulation. In Conference on Robot Learning (CoRL), 2020.
[44] D. Seita, P. Florence, J. Tompson, E. Coumans, V. Sindhwani, K. Goldberg, and A. Zeng- Learning to Rearrange Deformable Cables, Fabrics, and Bags with Goal-Conditioned Trans- porter Networks. In IEEE International Conference on Robotics and Automation (ICRA), 2021.
[45] R. Lee, D. Ward, A. Cosgun, V. Dasagi, P. Corke, and Leitner Learning arbitrary-good fabric folding with one hour of real robot experience. In Conference Robot Learning (CoRL), 2020.
[46] M. Shridhar, L. Manuelli, and D. Fox. CLIPort: What and Where Pathways for Robotic Manipulation. In Conference Robot Learning (CoRL), 2021.
[47] James, Wada, Laidlow, and A. Davison. Coarse-to-fim q-attention: Efficient learning for visual robotic manipulation via discretisation. In IEEE Conference on Computer Vision and Pattern Recognition (CVPR), 2022.
[48] A. Goyal, V. Blukis, Xu, Y. Guo, Y.-W. Chao, and D. Fox. Ry12: Learning precise manipu- lation from few demonstrations. RSS, 2024.
[49] OpenAl. GPT-4 Technical Report. arXiv preprint arXiv:2303.08777, 2023.
[50] H. Touvron and Others. Llama 2: Open Foundation and Fine- Line-Tuned Chat Models. arXiv preprint arXiv:2307.00288 2023,

---

[51] G. T. Google. Gemini: A Family of Highly Capable Multimodal Models. arXiv preprint arXiv:2312. /8800, 2023.
[52] W. Huang. Abbeel, D. Pathak, and I. Mordatch. Language Models as Zero- Planners: Extracting Actionable Knowledge for Embodied Agents. In International Conference Ma- chine Learning (ICML), 2022.
[53] D. Driess, F. Xia, M. S. M. Sajjadi, C. Lynch, A. Chowdhery, B. Ichter, A. Wahid, J. Tompson, Q. Vuong, Yu, W. Huang, Y. Chebotar, Sermanet, D. Duckworth, Levine, Vanhoucke, K. Hausman, M. Toussaint, K. Greff, A. Zeng, Mordatch, and P. Florence. PaLM-E: An Embodied Multimodal Language Model. In International Conference Machine Learning (ICML), 2023.
[54] M. Ahn, A. Brohan, N. Brown, Y. Chebotar, O. Cortes, B. David, C. Finn, C. Fu, K. Gopalakr- ishnan, K. Hausman, A. Herzog, D. Ho, J. Hsu, J. Ibarz, Ichter, A. Irpan, E. Jang, R. J. Ruano, K. Jeffrey, S. Jesmonth, N. Joshi, R. Julian, D. Kalashnikov, Y. Kuang, K.-H. Lee, S. Levine, Y. Lu, L. Luu, C. Parada, P. Pastor, J. Quiambao. K. Rao, J. Rettinghouse, D. Reyes, P. Sermanet, N. Sievers, C. Tan, A. Toshev, V. Vanhoucke, F. Xia, T. Xiao, P. Xu, S. Xu, M. Yan, and A. Zeng. Do as can and not as say: Grounding language in robotic affordances. In Conference on Robot Learning (CoRL), 2022.
[55] C. H. Song, Wu, C. Washington, M. Sadler, W.-L. Chao, and Y. Su. LLM-Planner: Few- Shot Grounded Planning for Embodied Agents with Large Language Models. In IEEE/CVF International Conference Computer Vision (ICCV), 2023.
[56] J. Liang, W. Huang, Xia, P. Xu, K. Hausman, B. Ichter, Florence, and A. Zeng. Code as policies: Language model programs for embodied control. In IEEE International Conference on Robotics and Automation (ICRA), 2023.
[57] W. Huang, F. Xia, D. Shah, D. Driess, A. Zeng, Y. Lu, P. R. Florence, I. Mordatch, S. Levine, K. Hausman, and B. Ichter. Grounded decoding: Guiding text generation with grounded mod- els for robot control. arXiv preprint vrXiv:2303.0085. 2023.
[58) I. Singh, V. Blukis, A. Mousavian, A. Goyal, D. Xu, J. Tremblay, D. Fox, J. Thomason, and A. Garg. ProgPrompt: Program generation for situated robot task planning using large lan- guage models. Autonomous Robots (AURO),
[59] K. Kawaharazuka, Matsushima, Gambardella, J. Guo, C. Paxton, and A. Zeng. Real-World Robot Applications of Foundation Models: A Review. arXiv preprint arXiv:2402.0574: 2024.
[60] R. Firoozi, J. Tucker, S. Tian, A. Majumdar, J. Sun, W. Liu, Y. Zhu, S. Song, A. Kapoor, K. Hausman, B. Ichter, D. Driess, J. Wu, C. Lu, and M. Schwager. Foundation Models in Applications, Challenges, and the Future. arXiv preprint arXiv:2312.0784: 2023.
Robotics:
[61] Hu, Q. Xie, V. Francis, J. Patrikar, N. Keetha, S. Kim, Y. Xie, T. Zhang, S. Zhao, Y. Q. Chong, C. Wang, K. Sycara, M. Soahsson-Robesonn D. Batra, X. Wang. S. Scherer, Z. Kira, F. Xia, and Y. Bisk. Toward General-Purpose Robots via Foundation Models: A Survey and Anal-Sn. arXiv preprint arXiv:23/2.08772, 2023.
[62] J. Varley, S. Singh, D. Jain, K. Choromanski, A. Zeng, S. B. R. Chowdhury, A. Dubey, and V. Sindhwani. Embodied AI with Two Arms: Zero-shot Learning, Safety and Modularity. arXiv preprint arXiv:2404.035700 2024.
[63] A. Kamath, M. Singh, Y. LeCun, Synnaeve, I. Misra, and N. Carion. Mdetr modulated detection for end-to-end multi-modal understanding. In 2021 IEEE/CVF International Con- ference Computer Vision (ICCV), pages 1760-1770, 2021. doi:10.1 11-00..110//CCV499222200111 00180.
12

---

[64] M. Minderer, A. Gritsenko, A. Stone, M. Neumann, D. Weissenborn, A. Dosovitskiy, A. Ma- hendran, A. Arnab, M. Dehghani, Z. Shen, et al. Simple open-vocabulary object detection. In European Conference on Computer Vision, pages 728-755. Springer, 2022.
[65] A. Kirillov, E. Mintun, N. Ravi, H. Mao, C. Rolland, L. Gustafson, T. Xiao, S. Whitehead, A. C. Berg, W.-Y. Lo, P. Dollár, and R. B. Girshick. Segment anything. 2023 IEEE/CVF International Conference on Computer Vision (ICCV), pages 3992-4003, 2023. URL https:
// /aap..semattccschol.. wwt.://oog/ooooo:::./.. ott.e//antttttttto.ooooooooooooa 22570rp9 257952310.
[66] P. Wu, Y. Shentu, Z. Yi, X. Lin, and P. Abbeel. Gello: A general, low-cost, and intuitive teleoperation framework for robot manipulators, 2023.
[67] T.-W. Ke, N. Gkanatsios, and Fragkiadaki. 3D Diffuser Actor: Policy Diffusion with 3D Scene Representations. arXiv preprint arXiv:2402. 2024.
[68] S. Chen, R. Garcia, C. Schmid, and I. Laptev. PolarNet: 3D Point Clouds for Language- Guidd Robotic Manipulation. In Conference on Robot Learning (CoRL), 2023.
[69] P.-L. Guhur, S. Chen, R. Garcia, M. Tapaswi, I. Laptev, and C. Schmid. Instruction-driven History-aware Policies for Robotic Manipulations. In Conference on Robot Learning (CoRL), 2022.
[70] A. Jaegle, S. Borgeaud, J.-B. Alayrac, C. Doersch, C. Ionescu, D. Ding, S. Koppula, A. Brock, E. Shelhamer, O. J. H'enaff, M. M. Botvinick, A. Zisserman, O. Vinyals, and J. Carreira. Perceiver io: A general architecture for structured inputs & outputs. ArXiv, abs/2107.14795, 2021. URL https::///// /api. semanticscholar how.ooaar..rr/coppsS0022020000 :D:66333379.
[71] Y. You, J. Li, S. J. Reddi, J. Hseu, S. Kumar, S. Bhojanapalli, X. Song. J. Demmel, K. Keutzer, and C.-J. Hsich. Large batch optimization for deep learning: Training bert in 76 minutes. arXiv: Learning, 2019. URL https::///apppp
165163737.
[72] K. Mo, Zhu, A. X. Chang. L. Yi, S. Tripathi, L. J. Guibas, and H. Su. PartNet: large-scale benchmark for fine mne-grained and hierarchical part-level 3D object understanding. In The IEEE Conference Computer Vision and Pattern Recognition (CVPR), June 2019.
13

---

A Simulation Benchmark for Bimanual Manipulation
We chose RLBench [14] as our choice of simulator since it is well-maintained by the research com- munity and has been used in a number of prior works [13, 12, 67, 68, 69, 16], including PerAct [1 []]], which is a core component of VoxAct-B,
A.1 Additional Implementation Details
VoxAct-B uses voxel grid size of 50 that spans 2 meters. The proprioception data includes: the gripper opening state of both arms, the positions of the left arm left finger, left arm right finger, right arm left finger, right arm right finger, and timestep. Following PerAct [11], we apply data augmentations to the training data using SE(3) transformations: [=:0.125 m, 1..2 m, 10.125 m] in translations and =45 in the yaw axis. We use 2048 latents of dimension 512 in the Perceiver Transformer [70] and optimize the entire network using the LAMB [71] optimizer. We use = 0.3 for Open Jar and = 0.4 for the drawer tasks and Hand Over Item. We select these values by using a starting state of the environment with the largest scaling size factor for the object of interest and checking whether the object remains entirely contained in the voxel grid after cropping.
Deciding the best checkpoints for VoxAct-B and ablations is nontrivial since iterating over all possi- ble combinations is computationally expensive. For example, with 400,000 training steps, using the same 10,000 checkpoint interval means there are 40 x 40 1600 possible combinations. Therefore, with the validation data, we use the latest stabilizing checkpoint to evaluate all acting checkpoints; we use the best acting checkpoint to evaluate all stabilizing checkpoints. Then, we use the best- performing acting and stabilizing checkpoints to obtain the test success rate.
The policy is trained with batch size of 1 on an Nvidia 3080 GPU for two days. Note that the batch size is not optimized based GPU memory capacity. The batch size can be increased using an Nvidia RTX Titan with 24 GB of VRAM. Morcover, VoxAct-B is much more efficient than Bimanual PerActs, requiring only two days of training to reach one million training iterations, while it would take Bimanual PerActs ten days to reach the same number of training iterations.
During policy evaluation, the language goal, las or Euas is determined by VL.Ms based on the given task. Specifically, in Open Drawer and Put Item in Drawer, we use the drawer's pose relative to the front camera to determine which robot arm the drawer is facing. If faces the left robot arm, las is selected because the orientation gives the left (acting) arm better angle for opening the drawer, and vice versa for the right robot arm. In Open Jar, we use the jar's pose to determine which robot arm it is closer to. If it is closer to the left robot arm, las is selected because it provides the right arm with a better angle for grasping the jar based on our experience with this task. This also applics to the block in Hand Over Item.
A.2 Additional Simulation and Task Details
We extend RLBench to support bimanual manipulation by incorporating an additional Franka Panda arm into the RLBench's 's training and evaluation pipelines. Importantly, we do not modify the under- lying APIs of CoppeliaSim (the backend of RLBench) to control the additional arm; consequently, the robot arms cannot operate simultaneously, resulting in delay in their control. However, this limitation is acceptable as our tasks do not require real-time, dual-arm collaboration.
Moreover, modify Open Jar, Open Drawer, and Put Item in Drawer to support bimanual manipulation: (1) adding an additional Franka Panda arm with a wrist camera; (2) adding new waypoints for the additional arm; (3) adjusting the front camera's position to capture the entire workspace; (4) removing the left shoulder, right shoulder, and overhead cameras. The new tasks use a three-camera setup: front, left wrist, and right wrist. We also modify the data generation pipeline to use motion planning with the new waypoints, process RGB-D images and the new arm's proprioception data (joint position, joint velocities, gripper open state, gripper pose), and include the [z, y, 2] position (world coordinates) of the object of interest. These modifications also apply to the Hand Over Item task.
14

---

The success conditions of these tasks have also been modified: for Open Jar, we define proximity sensor in the jar bottle to detect whether an arm has firm grasp of the jar (the gripper's opening amount is between 0.5 and 0.93); for Open Drawer, we define a proximity sensor on the top of the drawer to detect whether an arm stabilizing the drawer. While robot arm could still "open" the drawer without the other arm's stabilization, we would not classify it success in Open Drawer. Similarly, for Open Jar, robot arm could "open" the jar lid without firm grasp the jar from the other arm, but this would not be classified as success.
For the additional task variations, each method must learn from multi-modal demonstrations. For example, given ten training data, half have left-acting and right-stabilizing demonstrations, and the other half have right-acting and stabilizii demonstrations. During evaluation, the drawer from the drawer tasks is randomly rotated to face the left arm in the first half of the episodes and the right arm in the last half. This variation can make it kinematically infeasible for the left right robot arm to open the drawer, requiring each method to determine the appropriate arm for acting and stabilizing. Additionally, we randomly scale objects (drawers and jars) from 90% to 100% of their original size.
A.3 Multi-Task Experiment
We train single policy of ACT and VoxAct- B on Open Jar, Open Drawer, and Put Item in Drawer, with 10 demonstrations for each task. For evaluation, both methods use the checkpoint with the best average validation suc- cess rate across all tasks. We use the same vali- dation and test data as the baseline comparison. Table 3 presents the multi-task, results, showing that Act-B outperforms ACT on all tasks.
Open Jar
Open
Put Item in Drawer
Drawer
ACT w/Transformers VoxAct-B (ours) 21.3
2.7
12.0
14.7
62.7
17.3
Table 3: Multi-task results of ACT and Vox Act-B trained on 10 demonstrations of each task and evalu- ated across three training seeds.
A.4 Automatic Selection of a
Instead of using predefined a, we use the VLMs to automatically determine the value of a by computing the largest dimension of the object of interest, with small padding added. The method using the estimated achieved success rate of 32% Open Drawer, compared to 73.6% when using the predefined evaluated across three training seeds. We suspect the performance drop is due to the additional voxel resolutions the method must learn. As a varies between demonstrations due to the rescaling of objects, voxel resolution (2) W H. more challenging.
ax ox also changes, making policy learning
A.5 Jars and Drawers of Different Appearances
We introduce two new jars and drawers of different appearances. In the new Open Jar experiment (i.e., different from the Open Jar experiment in the main paper), each method is trained on 10 demonstrations using 3 different jars. The two new jars are imported from the PartNet- ParbiittMooility Dataset [72]. Each method then evaluated on 25 episodes of validation and test data using these jars across three training seeds. ACT obtains a success rate of 0%, while our method achieves success rate of 41.3%. In the new Open Drawer experiment, each method is trained on 10 demon- strations using a drawer with 3 different textures, as were unable to find drawer with similar structures online. This experiment follows the same setup as the new Open Jar experiment. In summary, ACT achieves a success rate of 9.3%, while our method achieves a success rate of 33.3%.
A.6 Ablation: Single Camera Setup
Instead of using three-cament setup, (front) single-camera setup Open Drawer. In this setup, the end-effector approaches the drawer handle, the view may be obstructed by the robot
15

---

arm. However, with the addition of wrist camera, the handle remains visible, allowing the voxel grid to be constructed with a clear view of the drawer handle. This ablation tests how well Aot-ct-B handles occlusion with a limited number of camera views. With a single-camera setup, VoxAct-B achieves a success rate of 60%, while using the original camera setup (front, left wrist, and right wrist) yields a success rate of 73.3%, evaluated across three training seeds.
B Real-World Experimental Details
Hardware Setup. An overview of the hardware setup is described in Section 5.2. Our perception system utilizes the D415 camera to capture RGB and depth images at resolution of 1280 720 pixels, where the depth images contain values in meters. We apply zero-padding these images, resulting in resolution of 1280 1280 pixels. Hand-eye calibration is performed to determine the transformation matrices between the camera frame and the left robot base frame, as well as between the camera frame and the right robot base frame, using the Movelt Calibration package. We use the python-urx library to control the robot arms, Additionally, I/O programming is employed to control the Robotiq grippers, as CB2 URS robots do not support URCaps.
Data Collection. We utilize the GELLO teleoperation framework collect real-world demon- strations. Due to the lack of Real-Time Data Exchange (RTDE) protocol support in CB2 UR5s, a noticeable lag is present when operating the GELLO arms. For Open Jar, dedicated function controls the gripper's counterclockwise rotations for unscrewing the lid and lifting it into the air, mit- igating the instability caused by latency. This function is triggered when the operator activates the GELLO arm's 's trigger. Additionally, we found that fixing the stabilizing arm while the acting arm is in motion is erucial for effective policy learning, as eliminates noise introduced by unintentional, slight movements of the stabilizing arm. Observations are recorded at frequency of 2 Hz.
Training and Execution. For training, we use higher value for stopped wwweddduuffffeeeeeeeeeeeeeeeeeee tiifferrtimmstess, a hyper-parameter that determines how frequently keyframes are extracted from the continuous ac- tions based on how long the joint velocities have been near and the gripper state has not been changed, in PerAct's keyframe extraction function to account for the slower movements of the robot arms due to latency compared to simulation. We apply the inverse of the transformation matri- ces obtained from hand-eye calibration to project each arm's gripper position to the camera frame. Using the camera's 's intrinsics and an identity extrinsic matrix, construct the point cloud in the camera frame, allowing both arms' gripper positions and the voxel grid to reside in the same ref- erence frame. For evaluation, we multiply the transformation matrices from hand-eye calibration by the policy's predicted left and right gripper positions to obtain the tool center point positions in their respective robot base frames. We visualize each robot arm's predicted gripper position in Open3D before executing on the robot. Additionally, we conduct real-world experiments using a Ubuntu laptop without GPU, resulting in significantly slower policy inference and robot execution times from capturing image observation to moving the robot arms cams-compared to a GPU setup. Consequently, this results in longer pauses between each robot execution and extended real-world videos, as demonstrated on our website.
C Additional Implementation Details for the Baselines
We carefully tune the baselines and include the hyperparameters used in Table 4. We report the results the best-tuned baselines in Table 1. For our three tasks, we found that for ACT, chunk size of 100 worked well, consistent with the findings reported in [3]. The temporal aggregation technique did not improve performance our tasks, so we disabled this feature. For Diffusion Policy, lower values (c.g., 16) of the action prediction horizon were inadequate, leading to agents getting stuck at certain poses and failing to complete the tasks, so we used an action prediction horizon of 100. We found the Time-series Diffusion Transformer outperform the CNN-based Diffusion Policy on Open Drawer and Open Jar, while both of them achieved comparable success rates on Put Item in Drawer. We use batch size of 32 for both methods, and the observation resolution is 128 x 128
16

---

Hyperparameterr
ACT Value
Diffusion Policy Value
learning rate
3e-5
le-4
weight decay (for transformer only)
le-3
encoder layers
decoder layers
7
layers
8
feedforward dimension
3200
hidden dimension
512
embedding dimension
256
heads
8
4
chunk size
100
100
beta
10
dropout
0.1
attention dropout probability
0.3
train diffusion steps
100
test diffusion steps
100
ema power
0.75
Table 4: Combined hyperparameters of ACT and Diffusion Policy, A dash ("-") indicates the absence of a hyperparameter for a given method.
Open Jar
Open Drawer
Put Item in Drawer
Method
FAS
FAS+NSV
FAS
FAS+NSV
FAS
FAS+NSV
Diffusion Policy 20.8
40.8
24.0
46.4
14.4
19.2
ACT w/Transformers
31.2
56.0
28.8
35.2
34.4
75.2
Table 5: Ablation results of ACT and Diffusion Policy trained 100 demonstrations and evaluated across five training seeds. "FAS" refers to the demonstrations with fixed acting and stabilizing arms (i.c., right acting and left stabilizing), while "FAS+NSV" refers to fixed acting and stabilizing and without size variation in the environment. We use the same validation and test data as the baseline comparison.
(same as VoxAct-B). For Diffusion Policy, we use the same image augmentation techniques as in [3]. As shown in Table 5, the performance of ACT and Diffusion Policy progressively improves as more environment variations are removed. For VoxPoser, bimanual manipulation tasks. See our VoxPoser prompts for details. For Bimanual PerActs, deliberately chose to use 1003 voxels instead of the 50 voxels used in Vox The increased number of voxels provides higher voxel resolution, which is essential for fine-grained bimanual manipulation. This is demonstrated in the VoxAct-B w/o VLM ablation, which only utilizes 50 voxels and shows huge drop in performance compared to VoxAct-B.
modified the LLM prompts work with
17


Reviews:
[{"review_id": "zVKdAaIKEp", "reviewer": "Reviewer_nMKS", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 5, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": 3, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "This paper proposes VoxAct-B, a language-guided, voxel-based method that uses VLMs to perform precise bimanual robotic manipulation. Given RGB images of the scene and a text query, VoxAct-B uses OWL-ViT as an open-vocabulary object detection algorithm to predict a bounding box around the object of interest. Then, VoxAct-B uses SAM to predict a segmentation mask for the object given its predicted bounding box and uses the mask centroid along with the point cloud data to compute the object’s pose to the camera. This pose serves multiple purposes: 1) Determine the role of each arm (either acting on the object or stabilizing) and thus the correct language instruction, and 2) Crop the whole scene voxel grid into a fine-grained voxel grid around the object of interest to have better resolution, thus better precision, at a lower computational cost. The language instruction and fine-grained voxel grid are given to a learned bimanual manipulation policy to predict each arm's next pose, gripper openness, and whether the motion planner should check for collision. Finally, the approach is evaluated in an extended version of RLBench for bimanual manipulation and the real robot.", "summary_of_recommendation": "My recommendation is Weak Reject. I believe that studying bimanual manipulation is very interesting and that the robotics community could greatly benefit from an extension of the RLBench benchmark for bimanual manipulation. VoxAct-B is novel enough and the authors demonstrate it works well on the real robot. However, I have some concerns about the benchmark quality and if the paper's motivation is strong enough, and the pointed weaknesses should be addressed.", "technical_quality": 3, "clarity_of_presentation": 4, "potential_impact": 3, "robotics_focus": 4, "quality_of_limitations": 3, "questions_for_rebuttal": "Please, take a look at the weaknesses in the review. Questions should be addressed to make the paper stronger.", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "aIJ6I0Ks8f", "reviewer": "Reviewer_7y5C", "summary": "", "questions": "", "limitations": "", "rating": 4, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": 4, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "This paper introduces a language-conditioned, voxel-based method for efficient and generalizable bimanual manipulation using Vision Language Models (VLMs)", "summary_of_recommendation": "I like the way this paper has explained how VLMs can be leveraged for task planning and allow scalability.", "technical_quality": 5, "clarity_of_presentation": 4, "potential_impact": 4, "robotics_focus": 4, "quality_of_limitations": 2, "questions_for_rebuttal": "As mentioned above, it might be useful for more experiments in different environments and with various objects.", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "9nmC7AvbGN", "reviewer": "Reviewer_6SQW", "summary": "", "questions": "", "limitations": "", "rating": 2, "confidence": 5, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": 2, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "The paper proposes the use of VLMs as a means to reduce the state space to voxelize in order to learn bimanual policies.", "summary_of_recommendation": "Thank you to the authors for this work. The paper was well written and clear overall. Main reason for the recommendation is that the experiments show limited generalization. Each task needs to be trained as a separate policy and evaluation is done essentially in distribution. However, there is value for the community to read this paper. I’m very willing to increase my score if authors include an experiment that showcases some generalization e.g. training one multi-task policy for all tasks and showcasing that the policy can perform well not only on in distribution jars/drawers but also jars/drawyers of different appearance more than just color and minimal rescaling.", "technical_quality": 3, "clarity_of_presentation": 4, "potential_impact": 2, "robotics_focus": 4, "quality_of_limitations": 3, "questions_for_rebuttal": "1. No skills library? Method does mention the use of primitives. That is in a sense a skills library? \n\n2. Isn’t always assuming a stabilizing and an acting arm already hurting generalization?\n\n3. Ln. 115-116: how do you keep important keyframes of a demonstration such as interaction points? In those time steps, the gripper state does not remain unchanged often. \n\n4. What VLM are you using? How do you constrain it to output language in the form of e.g. “Hold the desk with the left arm and open the bottom drawer with the right” which is my assumption, otherwise it’s not clear how you map language to l_as or l_sa.\n\n5. Ln. 139-140: “Note that the predicted arm ID is discarded”, re: I’m assuming the 2 predicted next voxels for each arm (stabilizing and acting one) would differ depending on whether the left arm is the stabilizing and the right the acting or vice versa. If you are discarding the predicted arm ID, how do you discern between the two possible predicted modes at inference? \n\n6. Why R = 5 for rotation bins? Discretization error must be too large to perform precision tasks.\n\n7. Fig. 3: Why do you need Segment Anything at all here? If Owl-ViT gives reliable enough bounding boxes, then your crop center would be the center of your bounding box and that should be close enough? \n\n8. Ln. 165-166: Point cloud data access/usage has not been introduced so far. You get this from one RGB-D camera? How do you get complete enough point clouds to get a 3D object centroid (which I assume you need)? \n\n9. Ln. 234-236: Unless I’m misunderstanding, with this way of picking checkpoints, you may still be missing the best combination of stabilizing-acting checkpoints you oculd possibly have?\n\n10. In the real world setup, you train each task policy on 10 demonstrations for that task, correct?\n\n11. Nit: Does your Open Drawer setting actually require stabilization at all to succeed? I can’t really tell what the friction is like. Does the drawer furniture actually move if you don’t stabilize it at all and only pull the drawer?", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Meta-review: Strengths:
+ The paper introduces simulated bimanual tasks for reproducible benchmarking.
+ VoxAct-B is validated with real-robot experiments.
+ The approach uses VLMs in a new way to isolate regions of interest for 3D manipulation. 


Weaknesses:
- The number of bimanual tasks proposed in the benchmark is limited.
- Experiments do not highlight generalization capabilities of the approach. This includes multi-task and robustness (lighting and different backgrounds) tests.
- It’s unclear if some of the tasks (e.g. Put Item in Drawer) need the bimanual setting to solve the task at all.
- Some implementation details are missing, for e.g., how the VLM is used, and hyperparameters such as rotation discretization.

Post rebuttal:  
After an extensive rebuttal by the authors, some concerns regarding the quantity and quality of tasks remain. However, 2/3 reviewers feel that the work is beneficial to the community.
Decision: Accept