Open6DOR: Benchmarking Open-instruction 6-DoF Object Rearrangement and A VLM-based Approach

Venue: thecvf_com_CVPR_2024_Workshop_VLADR
Published: 2024-04-03 08:39:43

Abstract: The integration of large-scale Vision-Language Models (VLMs) with embodied AI can greatly enhance the generalizability and capacity to follow open instructions for robots. However, existing studies on object rearrangement are not up to full consideration of the 6-DoF requirements, let alone establishing a comprehensive benchmark. In this paper, we propel the pioneering construction of the benchmark and approach for table-top Open-instruction 6-DoF Object Rearrangement (Open6DOR). Specifically, we collect a synthetic dataset of 200+ objects and carefully design 2400+ Open6DOR tasks. These tasks are divided into the Position-track, Rotation-track, and 6-DoF-track for evaluating different embodied agents in predicting the positions and rotations of target objects. Besides, we also propose a VLM-based approach for Open6DOR, named Open6DOR-GPT, which empowers GPT-4V with 3D-awareness and simulation-assistance and exploits its strengths in generalizability and instruction-following for this task. We compare the existing embodied agents with our Open6DOR-GPT on the proposed Open6DOR benchmark and find that Open6DOR-GPT achieves state-of-the-art performance. We further show the impressive performance of Open6DOR-GPT in diverse real-world experiments. Our constructed benchmark and method will be released upon paper acceptance.

Main paper: 
 introduction: 
 Large-scale multimodal models [9, 25] pre-trained on webscale data have revolutionized numerous fields beyond what was previously imaginable, enabling open-vocabulary text understanding and 2D visual perception. The pursuit to bring general intelligence into the robotic realm and 3D physical world stands at an exciting yet nascent stage, calling for stronger capabilities in 3D-aware perception, robotic interaction and complex reasoning.The advent of embodied large-scale models, exemplified by the RT series [2, 3, 7, 15] and VoxPoser [17] , has demonstrated considerable progress in mobile or fixedstation pick-and-place operations. While these models are capable of rearranging the object positions following human instructions, they fall short of satisfying full 6-DoF object placement instructions that involve specified 3D rotations. This limitation renders them incompetent at many practical robotic applications, where both object position and orientation are essential. For instance, in our daily life we desire a water bottle to be placed upright, while on the shelves in retail stores, goods should face the same direction. Moreover, previous works [3, 15, 17] are often evaluated on their own robots in their own scenes with self-reported performance and nonstandard evaluation metrics. The absence of a standard evaluation protocal condone cherry-picking, obstruct comparative assessment, and thus, hinder the iterative enhancement of effective approaches.In this paper, we target the task of Open-instruction 6-DOF Object Rearrangement, referred to as Open6DOR, which requires embodied agents to move the target objects according to open instructions that specify its 6-DoF pose. Open6DOR represents a fundamental skill for robotic manipulation tasks, presenting significant challenges in integrating instruction comprehension, 3D visual perception, and motion planning capabilities. Specifically, we promote the envelope of Open6DOR from two perspectives: 1) Benchmark Construction: We construct a standardized benchmark, namely Open6DOR Benchmark, which comprises 2447 tasks designed with 200 objects across diverse categories in simulation environments. For comprehensive evaluation, we divide the Open6DOR benchmark into the position track, rotation track, and 6-DoF track, each providing manually configured tasks along with comprehensive and quantitative 3D annotations. These tracks enable independent or combined translational, rotational, and overall performance assessments.2) VLM-based Approach:We propose a VLMbased approach for Open6DOR tasks. Due to the challenges of Open6DOR analyzed before, all prior works, such as VoxPoser [17] and Dream2Real [20] , fail to fulfill Open6DOR's 6-DoF requirements adequately. Most of them [17] determine the final positions of target objects while neglecting the rotation dimension. Among these efforts, Dream2Real [20] has the potential to consider position and rotation dimensions in a coupled way by utilizing a VLM to directly select the instruction-aligned result from all rendered images of the imagined rearranged scenes. This leads to almost intolerable time costs resulted from numerous renderings and VLM inferences, as well as unsatisfactory results due to the VLM's limited 3D perception, which renders it an incompetent critic. In contrast, we propose Open6DOR-GPT, which explicitly integrates 3D information from the initial scene to GPT-4V with equipped auxiliary modules and decomposes the translational and rotational determinations. In this way, we augment GPT-4V with 3D understanding capabilities and improve efficiency by reducing the determination space with decoupled modeling and simulation-assistance. Open6DOR-GPT achieves state-of-the-art performance in both benchmark evaluation and real-world experiments. 
object rearrangement methods: 
 Object rearrangement [1] requires an embodied agent to manipulate objects to the desired pose based on specific instructions. Early works [11, 12] address this challenge by using task-and-motion-planning (TAMP) which relies on pre-defined action primitives and close-to-perfect scene knowledge for trajectory sampling. TAMP methods are computationally inefficient and unscalable for complicated scenarios. To enhance generalizability and efficiency, recent research has shifted towards Learningbased approaches [10, 14, 24, 28, 33] . These methods are trained on simulators, predicting either high-level planning [10, 24, 28] or low-level actions [14, 33] . Despite exhibiting satisfactory performance in simulators, they suffer from a severe sim-to-real gap [24] . Recent advanced methods leverage the open-world understanding capabili-ties of large language models (LLM) [6] or vision language models (VLM) [21, 26] for real-world deployment. A part of these approaches [16, 20] construct carefully designed prompts to describe the environments, and query off-theshelf LLMs or VLMs for placement guidance before execution. Other methods [31] train large language models from self-collected data and directly output low-level actions. Departing from existing works that predominantly focus on location, our method also emphasizes rotation, leading to a 6-DoF rearrangement method.Benchmarking object rearrangement is extremely challenging and requires extensive annotation of ground truth placement. Existing object pick-and-place benchmarks [10, 32, 34] leverage pre-built environments such as Repilca-CAD [27] or hand-crafted scenes, and high-quality object reconstructions [4] . Besides, these benchmarks contain annotations about the initial position and target position of each object, which are leveraged for evaluating correct placement. Aside from the simulator benchmarks, there are real-world benchmarks [23] which directly evaluate baselines in real-world robots. However, even though these benchmarks can directly assess real-world performance, they fail to accurately replicate the testing environments for all baselines and are not permanently available due to hardware limitations. Different from existing works, we propose a 6-DoF object rearrangement benchmark that comprehensively evaluates 6-DoF placement, providing both position and rotation annotations.Large models trained on internet-scale data have demonstrated great potential in high-level planning [13, 17? ]. The recent advent of VLMs further bridges the gap between visual perception and textual interpretation, empowering embodied agents with semantic understanding of scenes and instructions [19] to perform 6-DoF tasks. Some of the prior works [17] leverage VLMs to compose 3D value maps, planning robot trajectories that comply with the given instruction; while other methods such as Dream2Real [20] employ VLMs as evaluators, generating goal states in the form of images for VLM to assess. However, both approaches fail to consider complex tasks that strictly specify the rotation of an object. Moreover, Dream2Real suffers from excessive time-consumption and VLM's inaccurate judgment. In contrast, our method addresses the rotation and position aspect of the 6-DoF problem in a decoupled way, enhancing VLM's decision-making capabilities while expediting the inference process. 
task formulation and benchmark overview: 
 Open6DOR task formulation. Open-instruction object rearrangement refers to the process wherein an embodied agent repositions objects within a scene from an initial state, following specific instructions. In particular, a 6-DoF object rearrangement task focuses on repositioning objects in a 6-DoF space, including both orientational and translational movement. For a long-horizon rearrangement problem, we decompose the process into several independent pick-andplace tasks, during which objects are repositioned one at a time. We define each of these tasks as an Open6DOR task, in which a single target object is moved from its initial pose to a goal pose based on an open-vocabulary instruction. The input comprises a single-view RGB-D image of the initial scene captured by a camera fixed on the robotic arm, denoted as I rgbd , along with an arbitrary task instruction Ĩ, which describes the desired goal pose of an object in the scene. Based on these, the model is required to output the quantitative goal position P goal and goal rotation R goal of the target object.Open6DOR benchmark. The Open6DOR Benchmark is specifically designed for table-top Open6DOR tasks within a simulation environment. Our dataset encompasses 200+ high-quality objects, forming diverse scenes and totaling 2400+ diverse tasks, with statistics shown in Tab. 1. All tasks are carefully configured and accompanied by detailed annotations. To ensure comprehensive evaluation, we provide three specialized tracks of benchmark: the Rotationtrack Benchmark B r , the Position-track benchmark B p , and the 6-DoF-track Benchmark B 6DOR . B r encompasses tasks achievable through a singular rotational movement at a fixed point-for example, "place the cup upside down". B p concentrates on tasks requiring the repositioning of an object, like "put the cup between A and B", without specific regard to the object's orientation. Meanwhile, B 6DOR integrates both rotation and position requirements, involving tasks such as "place the mug in front of A with its handle pointing towards the left". Constructing the benchmark was a challenging and laborious task. The process involved four stages: a) data collection and processing, b) instruction design, c) task formulation, and d) pose annotation . It took the team over a month to complete the preliminary version, and we anticipate further investment to expand and refine the Open6DOR Benchmark.Data composition and annotation. The Position-track benchmark includes 1028 tasks, each set in a table-top scene that contains several objects. Our synthetic object dataset O s comprises 200+ items, covering a range of 70+ distinct categories. Originally derived from YCB [5] and Objaverse- Place the capybara on top of the paper labeled "Open6DOR", with the capybara facing frontwards. XL [8] , the objects are carefully filtered so as to ensure our selections are physically intact and semantically reasonable to be placed on a table. We normalize the scale of all the objects and use a uniform format of mesh representation. The objects are then classified into different categories for the convenience of future analysis.For scene configuration, we randomly select 2-6 objects from O s and position them on the table with random initial poses. We then generate an RGB-D image for each of the scenes and filter out low-quality ones (e.g. scenes that include unreasonably placed or heavily occluded objects), resulting in a single-view RGB-D image dataset V p . For position instructions I p , we design three levels that evaluate the understanding of: basic directions (Level 0) such as Left, Right, Top, Behind, Front; object relations (Level 1) such as Between, Center; and customized commands (Level 2) like Put A into B. The task instructions adhere to a uniform format, such as 'place A in front of B', where A and B are subsequently specified based on the context of individual scenes. Additionally, we annotate a position range for each task according to the given instruction.Evaluation metrics. We assess the predicted goal postion P goal according to the annotated position range. A position that falls into that range is considered as correct, otherwise wrong. For instance, in the Left task category, we verify whether the predicted position is to the left of the reference object (indicated by a smaller y-axis coordinate).Data composition and annotation. The Rotation-track Benchmark consists of 360 diverse tasks, each set in a scene containing a single object. We use the same object dataset O s as the Position-track Benchmark. For the single-view RGB-D input I rgbd , we provide a dataset V r , which comprises RGB-D images of all the objects in O s . For the instruction input Ĩ, we construct a dataset of 70+ rotationspecified instructions, denoted as I r . For each object category in O s , we label it with 1-5 instructions in I r based on its features. The instructions are categorized into 3 levels that progressively increase in difficulty. Level 0 includes basic instructions that are related to the geometric shape of the object, such as "upright" and "upside down". Level 1 generally requires a higher understanding of direction and orientation, such as "handle to the left". Level 2 contains harder instructions concerning semantics and textual information of the object, such as "label forth" and "characters right side up". Evaluation metrics. Due to the diversity and complexity of human commands in our instruction set I r , it is difficult to design a uniform metric to judge alignment between the numerical representation of the rotation and the initial instruction. To address this problem, we manually annotate each task with a rotation range that complies with the instructions. Rotation results that fall into this range are considered as correct, otherwise wrong.Data composition and annotation. The 6-DoF-Track Benchmark comprises 1059 tasks, providing a comprehensive evaluation that jointly assesses the rotation and position performance of an Open6DOR task. The formulation of the RGB-D scene image I rgbd aligns with that of the Positiontrack. For instruction Ĩ, we combine instructions from I p and I r , forming instructions that specify both the position and rotation of an object. Each instruction is paired with an RGB-D scene image as the task input, and we exclude the incompatible pairs to ensure that the tasks are well-defined and performable. Evaluation metrics. We evaluate the quality of a 6-DoF pose from two perspectives: rotation and position. Specifically, we manually annotate the desired rotation and position of the target object based on the instruction. We consider a task successful only when it satisfies both criteria. 
method overview: 
 As shown in Fig. 2 , we enhance GPT-4V [25] 's capabilities to address the challenges of the Open6DOR task in a decomposed way. Initially, the Task Preprocessing Module deciphers Ĩ based on the I rgbd and feeds the resulting images to the Position Module and Rotation Module respectively. Within the two modules, we empower GPT-4V with 3D awareness and simulation assistance, thereby effectively outputting the predicted goal position P goal and rotation R goal . Finally, the Simulation-assisted Planning Module identifies a suitable grasping pose and plans out an optimal action trajectory to accomplish the task. We will first introduce each module of our proposed system in paragraph B-E to explain how an Open6DOR task is accomplished. We then elaborate on how the system tackles long-horizon tasks with multiple rounds of operations.With the single-view RGB-D Image I rgbd and the task instruction Ĩ as input, this module leverages GPT-4V to interpret the instruction and identifies object names{O name i }, which in turn triggers GroundedSAM [18] to generate a set of labeled masks. Based on the masked Image I mask , the RGB image of the target object I object is extracted. These images are used in subsequent modules.Taking the masked RGB-D image I mask and task instruction Ĩ as input, the 3D-aware Position Module M p determines and outputs the goal position that complies with the requirements.To incorporate three-dimensional (3D) data into GPT-4V's understanding, our approach utilizes back-projection based on I mask to generate a 3D masked point cloud, symbolized as P C 3d i . This computation includes determining the centroid Center 3d i and bounding box Bbox 3d i of the point cloud associated with the queried object.EQUATIONCenter 3d i , Bbox 3d i = Mean(P C 3d i ), Max(P C 3d i )-Min(P C 3d i )These spatial attributes are then integrated back into the prompt for GPT-4V, facilitating the model to accurately ascertain the goal position for the target object P goal .With the single-view RGB image of the target object I object and the task instruction Ĩ as input, the rotation module would output the goal rotation R goal for the object. We first reconstruct the target object from I object using One-2-3-45++ [22] , and outputs a textured mesh, denoted as M . The reconstruction process is followed by four phases: (1) large-scale sampling (2) simulation-assisted filtering (3) rotation categorization (4) GPT-4V selection. Large-scale sampling. In Phase 1, we randomly sample a total amount of N rotations {R 0 i } N i=0 as initial inputs for subsequent phases. We set N = 3600 and use Uniform Sampling in SO(3) space (Special Orthogonal Group in 3D space) to ensure the diversity of our samples. Simulation-assisted filtering. Now that we have a large pool of rotation candidates {R 0 i }, the goal of Phase 2 is to filter out the unreasonable candidates and narrow down the sample pool efficiently. To accomplish this task, we first examine the stability of the rotation candidates by incorporating a physics simulator through which all the unstable poses are excluded. To be specific, each R 0 i is applied to a replica of M , denoted by M i , as its initial rotation in the simulator, amounting to N actors {M 0 i } N i=0 in total. Then, all the actors are dropped from a low height, landing on the ground with diverse ending poses. We record the relative rotation from the original mesh M to the ending pose of M i as R t i . By now, we have narrowed down the originally random Then, we employ large-scale sampling to obtain multiple rotation samples. This sample set is then narrowed down through a simulationassisted filtering process to derive several stable pose categories. Finally, we generate rendered images of the pose candidates, from which GPT-4V selects the optimal goal rotation.and irregular distribution of {R 0 i } N i=0 to a more condensed space {R t i } N i=0 of geometrically stable rotations. By simulating the dropping process within 10 seconds, we avoid the time-consuming inferences of vision language models while accurately extracting stable poses of M . Rotation categorization. From Phase 2, we've obtained a set of rotations {R t i } N i=0 that guarantee the stability of the object. However, these rotations are unevenly cluttered around several distinct centers, each representing a stable pose. In order to categorize these rotations and extract a representative for each stable pose, we propose a criterion by which we judge whether two rotations R a and R b could be classified into the same category. In brief, we regard R a and R b as identical if they meet any one of the following two criteria: (1) the relative rotation from R a to R b is small in magnitude (2) R a and R b represents symmetrical poses that are transferrable via a rotation along the z-axis(perpendicular to the table surface). We define a threshold for each of these criteria and calculate whether R a and R b could be considered as identical. Through this method, we classify the rotations into several clusters and represent each category with a single rotation, which largely reduces the total number of rotation candidates for the task. Therefore, we successfully narrow down {R t i } N i=0 to a small set: {R j } n i=0 , in which each rotation represents a distinct stable pose. GPT-4V selection. During the last phase of our rotation engine, we feed the filtered set of rotations {R j } n i=0 in the form of the 2D image along with the original instruction to GPT-4V and let it select a candidate as the goal rotation R goal . To transform {R j } into a modal that VLM could easily understand, we apply each R j to M and render the image of the object loaded on a table accordingly. The im-ages are then arranged together into a collage, with an index mark on the upper left of each grid. Empirically, we found that this strategic approach of numbering and segmenting the images boosts the performance of GPT-4V in selecting the right answer. To further enhance our method, we employ a two-stage strategy that resamples a set of rotation candidates based on the rotation GPT-4V has selected in Stage 1. After this second round of adjustment, the goal rotation R goal is determined and outputted.Utilizing the predicted goal position P goal and goal rotation R goal , the planning module formulates an effective execution strategy with simulation assistance. Firstly, the Grasp Detection Model, GSNet [30] , takes the refined point cloud P C refined as input and generates a series of scored grasping pose candidates {(G j , s j )}. From {G j }, GPT-4V selects valid grasping poses that rest on the target object by leveraging the object bounding box Bbox 3d derived from the 3Daware Position, resulting in a ranked set of { Gj }.EQUATIONEQUATIONNext, we use cuRobo [29] as the motion planner, which enumerates { Gj } within the simulator based on their score rankings {s j }, and identifies a trajectory that optimizes both grasping and placement, denoted as T . Finally, the robot employs its control system to accomplish execution according to T . Success Rate (%) Level 0 Level 1 Level 2 Overall GPT-4V [25] 45 We compared our approach against several benchmarks for positioning proposals. This includes: (1) GPT-4V [25] , utilizing pixel input to predict object placement and employing depth for 3D location. (2) A tailored Dream2Real [20] baseline for our task. (3, 4) VoxPoser [17] original and adapted versions, aligning with our goals. Our tests include GPT-4V's Large Language Model (LLM) and Vision-Language Model (VLM) setups, with an asterisk denoting groundtruth data usage as reference baselines.With the framework outlined in previous sections, our system is capable of managing individual Open6DOR tasks. For long-horizon rearrangement tasks, we first employ GPT-4V to evaluate whether multiple Open6DOR steps are required. If the task necessitates multiple steps, GPT-4V leverages high-level planning to divide the instruction Ĩ into several discrete execution steps { Ĩi }. Subsequently, for each individual step I i , we apply the previously described methodology to carry out the task. Upon the completion of each Open6DOR sequence, the overall task is considered complete. This approach ensures a systematic and efficient handling of complex rearrangement tasks, breaking them down into manageable steps that are executed with precision. 
results on position-track benchmark: 
 We evaluate the performance of our position module and several baselines on the Position-track Benchmark. As shown in Table 2 , both Dream2Real and GPT-4V demonstrate incompetence at precise position determination. VoxPoser [17] , another baseline, yields unsatisfactory performance due to reliance on Large Language Models (LLM) without visual inputs. But even when adapted to a VLM-assisted version and incorporating image data, Vox-Poser(VLM)* fails to gain significant improvements. Comparatively, our approach markedly surpasses all these baselines by over 30 percent, demonstrating superior and consistent performance on the Position-track Benchmark.Our Rotation Module comprises four phases aimed at enhancing GPT-4V [25] and-filter mechanism. To evaluate the effectiveness of each phase, we conduct ablation studies using the Rotation-track of Open6DOR Benchmark, with results detailed in Table 3 . We compare our approach with Dream2Real [20] , replacing their CLIP Model with GPT-4V to ensure fairness. As shown in the first row of the table, directly querying GPT-4V yields an unsatisfactory success rate. Substituting our module with Dream2Real's method also leads to a noticeable performance decline. However, upon incorporating the Simulation-assisted Filtering Phase, we observe a noticeable performance increase as GPT-4V is able to choose from a confined set of rotation candidates. Further enhancements are achieved by integrating a 2-stage VLM Selection method. Notably, the combined module outperforms Dream2Real* by 12 percent.We evaluate our entire pipeline using the 6DoF-track of Open6DOR Benchmark. The evaluation of rotational, positional, and joint performance are presented in Table 4. For baseline methods, we found a limited number of works addressing the 6DoF problem and chose to compare with Dream2Real [20] . However, the original Dream2Real method is excessively time-consuming, requiring over 10 minutes per task for completion. To fully assess Dream2Real on the 6DoF-track Benchmark (con-Put the capybara into the metal mug.Place the green ball at the center of the objects and to the right of the toy.Put the baseball into the frying pan.Rotate the bottle so that the bottom end is oriented towards right.Place the mug on the top of the green paper, so that its handle is pointing frontwards.Take a piece of paper and lay it over the screwdriver.Place the bottle onto the bowl so that it stands upright.Place the capybara on top of the paper labeled "6DOR", with the capybara facing frontwards. taining 1k+ tasks), we skip the scene-scanning and reconstruction section of their method, using ground-truth mesh and image instead. Despite the disadvantage of the mesh quality on our side(our method reconstructs a mesh from the original image), Open6DOR-GPT significantly outperforms Dream2Real* by about 30 percent. Our approach also demonstrates better efficiency compared to baseline approaches.In our real-world experiments, we leverage a Franka Panda arm with a parallel gripper and mount a Realsense D415 camera to its end for image capturing. To comprehensively demonstrate the performance of our approach, we design tasks of varying difficulty levels: (1) place objects to the target position (2) place objects to the target rotation (3) place objects to the target position and rotation. We employ diverse objects with different geometries, textures, and materials, including transparent and specular ones. As shown in Fig. 4 , our zero-shot method is able to tackle challenging Open6DOR scenarios and demonstrates strong potential in long-horizon tasks. 
conclusion: 
 In this paper, we pioneer the establishment of the Open6DOR benchmark and VLM-based approach, addressing the need for a comprehensive evaluation and a foregoing method exploration in open-instruction 6-DoF object rearrangement. Our synthetic benchmark, comprising over 200 objects and 2400 tasks, offers a standardized framework for evaluating the capabilities of embodied agents in simulation environments. Additionally, our Open6DOR-GPT approach achieves state-of-the-art performance, augmenting GPT-4V with 3D awareness and simulation assistance. As for the current limitations, while Open6DOR-GPT significantly improves position and rotation handling, it does not achieve real-time performance, and rotation understanding remains suboptimal. Future enhancements to our benchmark are anticipated, especially real-world extensions. 
 
 Appendix:  



Reviews: review: title: title: Official review
main_review: main_review: This paper systematically studies and benchmarks the performance of existing VLMs for Open-instruction 6-DoF object rearrangement. I tend to accept the paper because:

- This benchmark is timely as the community is paying more attention to language-guided robot manipulation tasks. 
- The proposed Open6DOR-GPT surpasses other baselines significantly.
- It is highly relevant to our workshop. 

However, authors should revise the paper carefully to avoid obvious flaws such as the template-named title.