TaMMa: Target-driven Multi-subscene Mobile Manipulation

Authors: jiawei  hou
tianyu  wang
tongying  pan
shouyan  wang
xiangyang  xue
yanwei  fu
Venue: robot-learning_org_CoRL_2024_Conference
Published: 2024-06-05 04:41:44

Abstract: For everyday service robotics, the ability to navigate back and forth based on tasks in multi-subscene environments and perform delicate manipulations is crucial and highly practical.
While existing robotics primarily focus on complex tasks within a single scene or simple tasks across scalable scenes individually, robots consisting of a mobile base with a robotic arm face the challenge of efficiently representing multiple subscenes, coordinating the collaboration between the mobile base and the robotic arm, and managing delicate tasks in scalable environments.
To address this issue, we propose Target-driven Multi-subscene Mobile Manipulation (\textit{TaMMa}), which efficiently handles mobile base movement and fine-grained manipulation across subscenes. Specifically, we obtain a reliable 3D Gaussian initialization of the whole scene using a sparse 3D point cloud with encoded semantics. Through querying the coarse Gaussians, we acquire the approximate pose of the target, navigate the mobile base to approach it, and reduce the scope of precise target pose estimation to the corresponding subscene. Optimizing while moving, we employ diffusion-based depth completion to optimize fine-grained Gaussians and estimate the target's refined pose. For target-driven manipulation, we adopt Gaussians inpainting to obtain precise poses for the origin and destination of the operation in a \textit{think before you do it} manner, enabling fine-grained manipulation. 
We conduct various experiments on a real robotic to demonstrate our method in effectively and efficiently achieving precise operation tasks across multiple tabletop subscenes.

Main paper: 
 introduction: 
 It is common and highly practical for robotics to perform everyday tasks with cross-subscene mobility and fine-grained manipulation in a scalable environment. Handling compositional tasks is especially challenging. For example, robotics may be required to pick up a cup from one table and place it on another, pour water from a bottle in one location into a cup in another, collect cups from various tabletops and organize them on a cup rack, or tidy up multiple surfaces. However, existing robotics mainly focus on complex tasks in a single scene [1, 2, 3, 4] or simple tasks in a scalable scene [5, 6, 7, 8, 9] individually. Previous successful works [10, 8, 9, 11, 12, 13] often relied on Large Multimodal Models (LMMs) for scene perception and action planning, laying the groundwork for accomplishing long-horizon tasks. Nevertheless, due to a lack of further refinement of 3D scenes and their action planning often being in the form of policies, their generalization ability to scenes is relatively weak, leading to low success rates in various compositional tasks. Works like [5, 14] have also employed iterative search through building 3D scene graphs to accomplish TaMMa Target InstructionScene PriorFigure 1 : Our proposed multi-subscene mobile manipulation approach. Coarse scene prior and refined pose estimation empower the mobile base movement and robotic arm manipulation effectively and efficiently on compositional challenging tasks such as pouring, stacking, and tidy-up. tasks across multiple subscenes. However, the computational resources and manpower required for this approach are substantial. It is essential to research a method that can construct unstructured 3D subscenes and execute precise manipulations to handle compositional tasks efficiently.For the sake of generality and simplicity, we assume the robotics structure is conducted with a robotic arm leveraging on a mobile base. To tackle cross-subscene compositional target-driven mobile manipulation tasks, robotics need the following properties: (1) Represent precise multiple subscenes in a unified space. In such multi-subscenes, it is impractical to achieve cross-subscene tasks with observation through a single camera, ignoring the environmental prior knowledge. (2) Collaborate the mobile base with the robotic arm. The end-to-end robotic learning pipeline is not ready to solve delicate cross-subscene tasks due to the challenge of generalizing to different scenes and hardware. (3) Obtain precise pose for target manipulation. To achieve compositional target-driven mobile manipulation in a scalable scene, estimating a fine-grained 6 Degree-of-Freedom(DoF) pose is equally significant besides representing the whole scene.To effectively represent 3D scenes and obtain the pose of target objects, the distilled feature fields(DFFs) like works [15, 16, 17, 18, 19, 20, 21] have recently been proposed to model 3D scenes with implicit representations by distilling 2D features to 3D, enabling scene content querying and interaction. However, these approaches suffer from (1) the requirement of sufficient views to train the feature field, (2) the disability to perform complex tasks such as pouring liquids or stacking, (3) the lack of generalizability for tasks across multiple sub-scenes, and (4) the disability to model environmental changes caused by manipulation. At the same time, 3D Gaussian Splatting(3DGS) [22] is proposed as a highlight approach with real-time processing capability that enables scene editing. The application of diffusion-based strategies has achieved promising results [23, 24, 25] , demonstrating its few-shot reconstruction and scene completion ability. Recently, InFusion [26] proposed using diffusion priors for 3D Gaussians editing. While the 6 DoF pose of targets serves as a powerful guide for precise manipulation, these approaches enable the possibility of representing and editing scenes efficiently in a scalable scene to get a refined target pose at the origin and destination.In this paper, we propose TaMMa to realize cross-subscene fine-grained target-driven mobile manipulation. Our approach leverages scene priors to facilitate mobile movement and simplify subsequent pose optimization. We employ a diffusion-based 3D Gaussian completion and inpainting technique to obtain the refined target pose for manipulation. Specifically, we first scan only a few posed RGB-D frames of each subscene and unproject them to a sparse 3D point cloud to initialize the 3D Gaussians. Grounded-Light-HQ-SAM [3] is introduced to extract queries and encode semantics following [27, 28] into Gaussians, which can be queried to provide a coarse pose of the manipulation target. This coarse pose serves as a prior, deciding which subscene the target belongs to. The robotic base is then navigated to approach the target, and the scope of fine-grained pose estimation is narrowed down to the related subscene. Simultaneously, the diffusion-based depth completion andSubscene_2 Subscene_3Queried SubsceneCamera Poses Grounded-Light-HQ-SAM [3] is employed to encode semantics for Gaussians and depth completion. The coarse Gaussians provide a scene prior to mobile base navigation; further, a fine-grained manipulation target pose is optimized from completed and inpainted Gaussians with an object mask.Gaussian refinement are processed, and a precise pose is optimized from the refined Gaussians and target inpainting for fine-grained manipulation. The method pipeline is shown in Fig. 2 .In summary, our contributions are as follows: (1) To facilitate robotics in performing everyday tasks, we propose TaMMa, showing showcases cross-subscene compositional target-driven mobile manipulation ability efficiently and effectively. ( 2) By fusing scene-level priors from roughly reconstructed 3D Gaussians and precisely optimized target pose from diffusion-powered Gaussians completion and inpainting, TaMMa realizes cross-subscene movement and manipulation with insufficient observations in a sophisticated environment. (3) We conduct various experiments on Franka Panda arm with a mobile base, evaluating the availability of the 3D Gaussians reconstruction and refinement, the scene inpainting, and the fine-grained cross-subscene manipulation. 
related works: 
 3D Scene Reconstruction. To obtain accurate object poses for robotic manipulation, it is necessary to model the 3D scene with sufficient geometric and semantic information. Neural radiance field(NeRF) [29] , as a highly prominent work in scene reconstruction, has attracted significant attention since its proposal, encouraging impressive works [30, 31, 32, 33, 34, 35] . Recently, DFF approaches [15, 16, 17] distilled features into a 3D implicit feature field, achieving a unified 3D feature representation. However, they fall short in the requirement of sufficient views, the disability to perform complex tasks, the lack of generalizability for multiple sub-scenes, and the disability to model environmental changes. On the other hand, 3D Gaussian Splatting [22] has gained widespread attention due to its promising reconstruction results and fast training process. However, the reconstruction performance of 3D Gaussians heavily relies on the initial points obtained from Structure from Motion(SFM) [36] . Inadequate initialization can lead to overfitting on training observations. InFusion [26] employed depth completion learned from diffusion models, demonstrating that with a learned depth inpainting model, the placement of initial Gaussian points can be refined obviously. These works showcase the possibility of completing and inpainting the scene representation with inadequate observations of a scalable scene.Depth-only Pose Estimation Preliminaries. Obtaining the 6-DoF pose of objects serves as a powerful guide for executing precise robotic manipulation. We have made some extensions to the existing pose estimation methods [37, 3, 38] based on grounded visual perception, which can handle common tabletop-level category objects. The extended method relies solely on depth map inputs and aligns shapes using the queried mask with template classes, enabling pose estimation adaptable to unstructured settings. In real-world robotic perception, the quality of point clouds obtained from typ- ical depth cameras is often unsatisfactory, making depth estimation challenging for semi-transparent and highly reflective metallic objects. This significantly limits depth-based pose estimation models.In this work, we further harness the ability of latent diffusion models (LDMs) [24, 39, 25] to complete depth and enhance real-world pose estimation by optimizing the depth map.Target-driven Mobile Manipulation. Target-driven mobile manipulation (TDMM) has emerged as a critical area in robotics, aiming to enhance the capabilities of robotics in complex environments. The primary goal of TDMM is to enable mobile robotics to navigate and interact with objects to achieve specific tasks or targets, often in unstructured settings. This requires a synthesis of advanced techniques in visual perception, navigation, and manipulation [40] . Such synthesis presents numerous challenges. First, target-driven manipulation requires a fine-grained perception of the interactive scene and continuous guidance of robotic manipulation through foreseeable goals. CLIP-Fields [41] proposed a neural field which can be queried for mobile movement. Previous work [5, 14, 6, 42] has integrated global and local perception information through 3D scene mapping, providing executable plans for mobile manipulator robotics. Although some existing real-world scene maps from these works can understand and represent the environment, they are often structured. Xiong et al. [43] proposed an adaptive learning framework to tackle realistic articulated object operation. These works mainly present two issues: 1. The mapping process of the task space is time-consuming; 2. It fails to meet the needs for fine-grained scene updates. Second, mobile manipulation tasks often require the integration of navigation and actions to provide target-oriented sequences. Benefiting from the emergent capabilities of large language models (LLMs) in long-horizon planning [44, 45, 46] , some works have decomposed mobile manipulation into a series of atomic tasks. However, they have not adequately considered the affordance of real-world objects, resulting in low accuracy in achieving the goals. On the other hand, imitation learning from human demonstrations has shown impressive performance in robotics. Works like MOMA-Force [47] , TeleMoMa [48] , Mobile-ALOHA [49] , and [50] employed behavior cloning on solving mobile manipulation tasks. However, data collection and demonstration selection which rely on complex human labor remain a problem. In this work, we address these challenges by integrating 3D Gaussians and category-level depth-only pose estimation algorithms and focusing on the environment and object representations. Additionally, we edit scenes via 3D Gaussian inpainting, providing operational targets that can be utilized across different scenes.3 Method 
formulation and overview: 
 Problem Formulation Given an unstructured environment with multiple subscenes S = {s 1 , s 2 , . . . , s m }, we aim to achieve a cross-subscene instruction T of the target task leveraging on a mobile base and robotic arm with a set of color imagesI = {I i } n i=1 , depth images D = {D i } n i=1and relative poses P = {P i } n i=1 as input. To be specific, first, a whole scene representation θ = F (I, D, P)is needed to serve as the scene knowledge memory for querying the target object because the multisubscene environment is too big to be perceived through the observation of a single view. Then, a pose estimation method is required to obtain the manipulation target object's poseEQUATION)where t is the extracted query from T . Furthermore, to model the changes in the scene, a scene editing method is needed to update the scene representationEQUATIONTo enable a robotic with the ability to reconstruct and inpaint the scene and apply fine-grained manipulation across subscenes, the following properties are required.Understanding of the whole environment and each subscene. For an unstructured environment with multiple subscenes, prior knowledge of the whole scene and a detailed understanding of each subscene is vital for tasks across subscenes. Given a set of images I with depths D and corresponding camera poses P, an entire 3D scene can be reconstructed using 3D Gaussians θ. Depth and pose input are utilized to get a reliable initialization of Gaussians θ init with semantics encoded following [3, 27] . Moreover, by incorporating diffusion priors into estimating the depth images, the missing observation regions and regions that are inaccurate from depth camera shots can be completed. The refined depth D ′ can be unprojected to a fine-grained point cloud and, together with θ init , can be fine-tuned to form precise Gaussians θ.Imagine before you do it. For precise manipulation across subscenes, the robotic needs accurate poses of the origin and destination of target objects. Scene inpainting allows for making desired changes to the scene in a manner similar to imagination. For example, in tasks like pick-and-place, by inpainting the scene to remove the object from its original position p ori and add it to the intended destination p dest , the robotic can acquire a precise pose for picking up and placing the object and obtain updated 3D Gaussians θ ′ .Moving while optimizing. Cross-subscene manipulation tasks involve both the mobile base movement and robotic arm manipulation. However, optimizing the precise pose of the target in the unstructured scalable scene often takes time. An efficient approach is to navigate the robotic to the vicinity of the target position using a roughly estimated coarse pose p c . During the robotic's movement, a fine-grained pose p f can be further optimized in a narrowed-down subscene region. This two-step process allows for a more time-efficient execution of the task, during which p c can be obtained by querying the above-mentioned initial Gaussians θ init and the fine-grained pose p f can be obtained from fine-tuned and inpainted Gaussians θ ′ .The effectiveness of using 3D Gaussians as a 3D scene representation heavily relies on the allocation of initial points [51] . To get a reliable initialization θ init , we unproject the depth maps D to 3D space with poses P. However, directly using depth captured by cameras may lead to inaccurate values, especially for objects at the visual edges and for reflective or translucent objects. What's more, it's often hard to get sufficient observation of a multi-subscene environment, which leads to missing views of some partials. To address this issue, we employ diffusion-based depth completion to obtain more accurate depth for fine-tuning the initial Gaussians. Fig. 3 shows the visualization results.The diffusion-based depth completion process takes a set of RGB image I, depth image D, and an object mask M of the target region as input. The mask M indicates regions that need to be refined. In our approach, Grounded-Light-HQ-SAM [3] is employed as the segmentor to produce M . A pre-trained Variational Auto-Encoder(VAE) E and a U-Net-like [52] denoiser U are employed The mask M is also down-sampled to match the encoded embeddings as M ′ . Consequently, the noised embeddings at step t can be denoted asEQUATIONThe loss function of the diffusion process can be denoted asEQUATIONwhere t ∈ {1, 2, . . . , T } is the diffusion steps. The process is optimized following the DDPM [39] strategy. After completion, the depth D ′ is unprojected to 3D, and together with θ init , is fine-tuned to get the refined Gaussians θ with several iterations' optimization.Precise manipulation in the multi-subscene environment needs fine-grained object pose on the origin and destination. Taking the pick-and-place task as an example, the pose of pick targets can be obtained by direct estimation. However, for the placement target, it's hard to get an accurate pose straightforwardly. To address this, we propose an "imagine and do it" approach. We inpaint the target object into the destination subscene and estimate the pose of the inpainted object as the placement pose. Simultaneously, we erase the object at the origin pose and update the related subscenes.Specifically, to inpaint the changes in the environment, we select a set of reference views v s in subscene s for inpainting. Firstly, the depth D vs can be obtained from the Gaussians θ. The color image I vs inpainting is processed using an SDXL-based [25] model Ĩvs = SDXL(I vs ⊙ M vs ), and the depth image is completed as mentioned above Dvs = C(D vs , Ĩvs , M vs ),where C is the depth completion process. The loss function is conducted asEQUATIONwhere I ′ vs is the rendered RGB and λ 1 and λ 2 is the loss weight.Scene Knowledge Prior. In environments with multiple subscenes, the mobile base needs to move back and forth within the scene to collaborate with the robotic arm in completing tasks. However, optimizing the precise pose for robotic arm operations in the whole scene can be time-consuming.It would be highly inefficient to wait for all optimizations done before taking any movement. As mentioned earlier, a rough pose can be obtained by querying the initial Gaussians θ init for the mobile base to approach the target within the reachable range of the robotic arm. Moreover, based on the retrieved pose, the range of optimization can be narrowed down to the relevant subscene.Depth Completion Empowered Fine-grained Pose Estimation. Depth values obtained from depth cameras can be inaccurate for pose estimation, especially at the camera's edges or in regions with translucent or reflective objects. Additionally, there may be missing or insufficient observations in certain partials during the entire scene capture process. The depth-completed Gaussians θ can be projected onto the queried views from coarse poses for more precise estimation, as shown in Fig. 3 . Evaluation is conducted on various challenging tasks. To enable F3rm [2] and HomeRobot [6] with cross-subscene ability, we fine-tuned them to reconstruct the subscenes in an aligned space for given tasks. 'Ours*' stands for using Gaussians not refined by completed depth.Experiments Setups We collect RGB-D images with a RealSense D435 camera mounted on a Franka Panda arm with a movement base. For each subscene tabletop, we scan 20 images with a resolution of 640 × 480, and we set three tables at a distance from each other as multiple subscenes in our environment. Additionally, we take 2 ∼ 3 images of the whole scene far from three tables to get a good visualization. The scans are set surrounding the tabletop without a fixed trajectory. Our models and real-world robotic experiments are trained and run on an Nvidia RTX 3090 GPU.We conduct experiments comparing Gaussian initialization, showing that reliable coarse Gaussians can be obtained with the unprojected 3D point cloud initialization. Fig. 5 shows the comparisons. On the other hand, the reconstruction of an unstructured scalable scene typically demands extensive observations. However, our coarse-to-fine approach, coupled with the diffusion policy, significantly reduces the required number of sample views. The comparative results depicted in Fig. 4 demonstrate that even with just a few input views, each subscene can be reconstructed well. Target Driven Scene Inpainting. For manipulation across subscenes, estimating the target object's origin and destination position is of great significance. To demonstrate the effectiveness of inpainting, we compare it with a straightforward approach that directly optimizes the 3D Gaussians and renders the resulting RGB image to compute a rendering loss. The results are shown in Tab. 1. It can be seen that with Gaussians inpainting in the 3D space, the pose deviation and optimization time consumption are less. Fig. 5 shows some inpainting examples on 3D Gaussians.Depth Completion Empowered Pose Estimation Depth completion is an effective approach to get a more dense and accurate pose estimation of the target object. To evaluate the depth completion, we compare the depth map as well as the success rates of challenging tasks before and after depth completion. Results in Fig. 3 and Tab. 2 show that depth completion plays a vital role when the raw depth input is inaccurate, especially on translucent or reflective objects.Fine-grained Manipulation Evaluation. To evaluate our proposed compositional target-driven mobile manipulation pipeline, we evaluate the manipulation results on tasks spanning across subscenes. As for the comparing methods, we realize the state-of-the-art manipulation method for scalable environments and regard the whole scene as the workspace. The success rate on various challenging tasks is evaluated as the metric. Results shown in Tab. 2 and Fig. 6 demonstrate the advantages of our proposed method. 
conclusion: 
 This paper proposes TaMMa, which leverages cross-subscene fine-grained mobile base movement and robotic arm manipulation efficiently and effectively. The 3D point cloud, unprojected from depth images and poses, serves as a reliable 3D Gaussians initialization to provide a rough pose estimation of the target and narrow down the further optimizing scope of the target pose. Depth completion is employed for precise pose optimization aiming to solve the inadequate observation and inaccurate depth estimation. The completed point cloud is employed to refine the 3D Gaussians and inpaint the scene changes for target manipulation. Moving while optimizing and imagine before you do it manners are proposed as efficient coarse-to-fine and accurate manipulation pose estimation approaches. Various experiments were conducted on a real robotic to evaluate our proposed method.Limitations and Future Work. Our work have raised questions about two limitations: I. Singleview depth completion is not so robust, which hinders optimization in depth-only pose estimation. We plan to explore multi-view knowledge integration to improve depth completion. II. Inpainting 3D Gaussians works well for flat objects but struggles with complex, three-dimensional ones. Extracting and re-editing complex objects may not yield satisfactory results. Future research will aim to enhance inpainting techniques for complex geometries.with an average distance of 1 m between the tables, as shown in the diagram. The objects on the tables can be categorized as (1) daily-used containers, such as cups, bottles, plates, baskets, pen holders, and a sink, (2) geometric objects like cubes, (3) challenging objects for grasping, such as rubber toys, irregular toy models, and transparent objects. The arrangement of objects on the tables is relatively random. 
 
 Appendix: As for task evaluation:1) Pick & Place. The task is thought to be completed when robotic successfully catch the target object from a subscene, move to another subscene and put down it on the destination point without falling on the first attempt.2) Stacking. The task is thought to be successful when robotic collects objects from different scenes and stacks them as instructed on the first attempt, during which no object falling happens and the stacked objects won't fall over without external forces.3) Tidy-up. The task is successful when robotic recognizes all the objects needed to be rearranged and carries them to the target destination without leftover.Pouring. We prepare 300ml water in a plastic bottle. We consider the task successful if robotic could carry the bottle from one subscene to another subscene without falling and pour the water into the target cup more than 150ml.Gaussian Reconstruction. For the Gaussian-Splatting-based scene reconstruction, we employ the widely-used open-source 3DGS[? ] code base. The initial point clouds used as the Gaussian initialization are downsampled by a factor of 5. The rendering process generates results as the origin image resolution of 640 × 480. Subsequently, Gaussians are optimized for 30,000 iterations across all scenes, utilizing the same loss function, Gaussian density, schedule, and hyperparameters as specified in the original implementation, which includes a learning rate dynamically adjusts based on the iteration number and a densification process that starts early in training and continues until the specified iteration threshold. The densification parameters are finely tuned, with densification performed every 100 iterations, an opacity reset interval of 1,000 iterations, and a gradient threshold for densification set to 0.01.Depth Completion. The depth inpainting process is powered by a diffusion model which is built upon Latent Diffusion Models (LDMs) [24, 39, 25] using a pre-trained Variational Auto-Encoder (VAE) and a U-Net-based [52] denoising architecture. Utilizing the frozen VAE, we encode both the color image and the depth map into a latent space, forming the basis for training our conditional denoiser. The depth map is repeated at channels to form a tri-channel input as an RGB image and is normalized. A composite feature map is constructed by concatenating the encoded depth and image elements. The denoising step is set to 20 at inference as default to trade off the time consumption and the effect. The U-Net-based denoising architecture iteratively refines the depth latent by predicting and removing noise at each timestep, which is managed by the DDIM [39] scheduler to ensure that noise is progressively reduced in control.Gaussian Merging and Fine-tuning. With the corresponding camera pose and obtained depth map of the inpainted image, the 2D inpainted data is unprojected into a 3D colored point cloud from image space. Then, features from the original and inpainted Gaussian point clouds are merged by concatenating their poses, features, and opacities. To remove floaters at the edges of the mask, the minimum number of points within a radius of 0.1 for a point to be considered not an outlier is set to 100 as default. The following fine-tuning process optimizes the model using a combination of L1 loss and D-SSIM (Differentiable Structural Similarity Index) to ensure that the final rendered results closely match the inpainted reference images. The weight parameters are set to 0.8 and 0.2 respectively, reflecting the emphasis on maintaining a balance between pixel-wise accuracy and perceptual similarity. The optimization is performed over 150 iterations to achieve the final Gaussian model.We provide videos illustrating our cross-subscene fine-grained manipulation ability on 4 tasks. Each of the interacting objects can be located in the arbitrary sub-scenes.Pick and Place. The input query is in the form of "Move the [A] to the [B]", where [A] and [B] are objects in the whole scene. For example, in the video, we set "Move the pink cup to the white plate", in which objects lie on 2 separate tables.Stacking. The input query is in the form of "Stack the [A] onto the [B]", where [A] are objects to be stacked and [B] is the target place. In the video, we show the result of applying "Stack the orange cubes onto the blue cube". In this case, orange cubes on different tables will be collected and placed on the blue one.Pouring. The query is in the form of "Pour the liquid in the [A] into the [B]", where [A] is the container that has liquid in it and [B] is the target container. We show the result of "Pour the liquid in the bottle into the white cup". In this example, the robotic carries the bottle smoothly to the cup and rotates the bottle to pour.Tidy-up. The query is in the form of "Tidy up the table with [A]", where [A] is the representative objects of a subscene. For example, the video shows the result of "Tidy up the table with toys and cups". In this example, the small objects on the table with toys and cups will be rearranged into a basket.Long term tasks in complex environment with obstacles and more than 3 subscenes. To demonstrate the manageability of our approach in a complex environment with multi-subscenes beyond toy-example tabletops, we conduct an experiment instructed by "Pour the water from the shelf bottle into the pot, discard the bottle, and place the cleaning sponge in the pot." The environment is conducted with 4 subscenes: shelf, sink, table1, and table2, including obstacles on the way. The subscene setting is just like the daily look as shown in Fig. B1 .Table1 & Table2 SinkFigure B1 : The complex environment setup with obstacles and more than 3 subscenes, which is conducted to demonstrate the manageability of our proposed method.Point Cloud Extraction To extract a scene-wide point cloud for manipulation, we reconstructed the entire scene based on 3D Gaussian Splatting [? ] and performed depth completion using diffu-Unmet Poses with Depth with depth. In these bottom rows, the left column presents the original depths acquired by the depth camera, while the right column shows the depths completed using our method.sions [39] . Starting from the completed depth map, we performed back-projection to obtain point clouds of several sub-scenes. Subsequently, we transformed the point clouds into the robot's working coordinate system , and executed specific navigation or manipulation tasks based on the transformation matrices of the movement base and end-effector relative to the working coordinate system.The navigation process is finished by employing the API provided by the SLAMTEC mobile base. Specifically, a 2D topdown occupancy map is built for the environment as the scene map. Based on the occupancy map, the trajectory planning and obstacles avoidance is completed by querying the map with the target position and the current position. By generating a set of waypoints, the mobile base is guided to the target position. As for manipulation, Franka Panda Arm is operated through the MoveIt! library [? ]. We provide the 6 DoF pose of the target and employ the movement API to approach and operate the gripper to close and open depending on the width of the estimated target objects. Motion Planning based on 6D Pose Some fine-grained robotic manipulations mainly rely on the 6D pose of the target for motion planning [? ]. In the motion planning process of TaMMa, we integrated a depth-only pose estimation method, as the state of objects in 3D space is crucial for determining meaningful manipulation points across various tasks. Specifically, since the poses of intra-category instances are pre-normalized [? ], it becomes highly advantageous to define task-relevant categorylevel grasping poses. These defined grasping poses are then transformed from object coordinates to camera coordinates using the estimated 6D object poses. The robot subsequently performs motion planning to move its gripper to the target grasping pose to accomplish each task. Based on the poses of different category-level objects and the tasks to be performed (e.g., pouring, stack), we utilized MoveIt! [? ] to set up target-driven action sequences. Therefore, the success rate of robotic manipulation heavily depends on the accuracy of pose estimation. Fig. C1 presents the qualitative results of pose estimation across different scenes. The top row of Fig. C1 demonstrates the excellent poses in most cases. However, there are instances where the results are suboptimal, often due to inaccurate depth data. This can be observed in the bottom two rows. When we attempt to acquire the pose of the "black pen," the original depth (left column) and the completed depth (right column) are both unsatisfactory, affecting the subsequent pose estimation process.Capture Failures. Accurate camera pose is crucial for scene reconstruction. During the experimentation process, we discovered that the jittering and offset errors of the mobile base and robotic arm could result in inaccurate camera poses used for scene reconstruction. Consequently, this leads to errors in the initialization and optimization processes of Gaussians, resulting in issues such as ghosting in the reconstructed scene. One possible solution is to reduce the movement speed of the base and robotic arm, capture data only when they are stable, and minimize frequent movements of the chassis as much as possible. The visualization of comparison is shown in Fig. D1 .Segmentation Failures. Our experiments have shown that when using a segmentation model based on SAM [3] as a mask for image inpainting and depth completion, the inaccuracy of the mask can result in incomplete object editing, leaving behind edges or blurry traces. A more precise mask can achieve better visual effects. The comparison inpainting results between using inaccurate masks and refined masks are shown in Fig. D2 .Gaussians Inpaiting Failure Gaussians Inpaiting Success Monocular Depth Estimation. In practical applications of robotic arm manipulation, it is common to use the depth values directly captured by a depth camera as the input for scene modeling and object pose estimation. This is because the depth values provided by a depth camera are absolute depth values, which differ from the relative depth typically obtained through monocular depth estimation. This absolute depth information is highly valuable for practical operations. However, depth camera captured results often exhibit errors and significant uncertainty, particularly at the edges of the camera's field of view and when dealing with transparent or reflective objects. These factors raise issues in performing operations in challenging scenarios. In this paper, we utilize a diffusion-based depth completion method to partially address the discontinuities and errors present in the depth camera results. However, the completed depth results still contain certain errors compared to the original absolute depth input. How to leverage depth maps from multiple viewpoints to supplement depth maps from a single viewpoint and obtain a dense and reliable point cloud remains an open question. What's more, the object occlusion may also cause depth completion failure. As mentioned above, even the diffusion-based method is also unable to solve the depth prediction problem for objects with severe occlusion. Additionally, for transparent objects, objects located behind them can also affect depth and pose estimation. To manage this issue in 3D space, amodal-based method will be our future work direction.3D Objects Inpainting. Calculating precise poses for robotic arm manipulation has always been an open and challenging problem, especially in scenarios lacking constraints. For example, it is difficult for a robotic arm to place a cup in the exact middle of the plate to achieve the same level of precision as a human. Similarly, stacking blocks together is challenging for a robotic arm to achieve a perfect appearance. These results stem from two main reasons: the difficulty of endowing the robotic arm with real-time adjustment capabilities and the inability of the arm to obtain accurate pose information of the target location. Through a comparison with existing successful robotic arm grasping tasks, we observed that while the estimation of graspable object poses can now yield fairly accurate results, it remains challenging to estimate the precise poses required for tasks such as placing a cup in the center of a plate or achieving perfect block stacking. Consequently, we propose using inpainting techniques, employing a "think before you do it" approach, to address this issue. We first edit the images and depth maps, followed by editing Gaussians to create a 3D representation of the desired task-completed scenario. We then estimate the poses of the target objects in this scene to obtain a more accurate target pose for interaction. However, implementing this approach still presents difficulties. The current Gaussians inpainting methods perform well for editing relatively planar objects but struggle with editing voluminous and complex-shaped objects. Therefore, our future research direction will focuse on how to edit objects in 3D space to obtain accurate results that can be used for interactive operations.More Complex and Larger-scale Scenes. We have successfully evaluated our approach in an environment at about 200m 2 with about 10 subscenes. The obstacles set on the way are boxes, chairs, and objects with regular geometry with widths larger than 5cm. Each obstacle is observed by about 3 frames from the camera when recording the environment. As for environments with a larger number of subscenes, the whole scene is represented with united Gaussians, and subscenes are divided by humans according to the distribution of optional interactive objects. We consider a part as a subscene when there are many objects that need careful perception. The number of subscenes doesn't affect the Gaussians optimization. However, more sampled images are needed for better results when there are more objects. As for the movement and manipulation, the target object localization takes a text query as input and outputs a coarse 3D position, while manipulation takes a task query as input and outputs manipulation strategy. The above steps are totally free from the number of subscenes. Consequently, the number of subscenes does not affect our approach. As for environments with larger scales and areas, this mainly affects the computation and memory that build the Gaussians of the environment. Taking advantage of our coarse-to-fine approach, the point cloud initialization of the Gaussians could be sub-sampled to an appropriate scale, and further optimization could be conducted on the subscene with more interests. Further to this end, lots of recent works have made great progress on expanding the Gaussians application to un-constrained and larger scenes. This would be one of our future directions to optimize the Gaussians for application in a larger-scale scene.Transparent, Reflective, and Complex Shape Objects. The challenging objects' appearance mainly affects the depth estimation and the inpainting process. Reflective and transparent objects hesitate in their optical characteristics, making it difficult for depth cameras to capture precise depths, thus affecting subsequent depth completion processes. Transparent objects are easily influenced by other objects when occluding or being occluded, causing depth estimation to be affected, resulting in phenomena like depth truncation. Similarly, in complex backgrounds, similar phenomena may occur. Reflective objects are prone to distortion and pseudo-shadow phenomena in depth.Objects with complex shapes affect the accuracy of depth estimation, with depth cameras showing differences in attention to detail, and small parts being easily overlooked during depth estimation. Furthermore, for object inpainting, errors in depth estimation and the complex relationship between shapes, textures, and backgrounds can lead to a decrease in inpainting effects. Fig. E4 and E3 show some examples of objects with transparent ones and complex shapes, demonstrating our model's ability to manage a number of challenging objects. However, such challenging situations can cause blurs, shadows, and noises. Therefore, one of our future research directions is to optimize the depth and pose estimation of challenging objects in 3D space under multi-view observations and introduce various shape priors for common shapes to further enhance the representation accuracy of objects.For the cross-subscene mobile manipulation task, only a few works have been open-sourced. In this paper, we choose F3rm [2] and HomeRobot [6] as the comparison methods. To enable them with the cross-subscene ability in our environment, we re-implement and fine-tune these methods for the aiming tasks. Regarding real-world robotic tasks, we conduct evaluations across ten distinct scenes per task, each featuring novel objects from various categories in arbitrary poses. We ensure that the scenes are reset to approximately the same configuration for each method being compared.F3rm. We implement F3rm to expand its workspace to the cross-subscene environment. For the data source, the same images from RealSense D435 are used as input, and to be fair, the depth images are also employed to train the implicit representation as an additional loss. The camera poses come from the calibrated camera on the robotic arm. As the environment expands, the need for VRAM increases obviously. Fairly, we maximize the VRAM usage of the RTX 3090 by employing a smaller feature resolution, as mentioned in F3rm, and ignoring the regions out of the tabletop workspace.As for the manipulation process, we first drive the mobile base to ensure the manipulation targets are reachable for the robotic arm and optimize the target pose for manipulation.HomeRobot. We implement HomeRobot in our environment for the cross-subscene task. The provided Detic and Grounded-SAM libraries are employed to get semantics. The exploration of the environment is replaced by feeding the recorded RGB-D sequences to the HomeRobot, and the navigation and planning of the mobile base is implemented in the same way as our proposed TaMMa.For mobile manipulation, the mobile base is first guided to approach the target, ensuring the targets are reachable, and the Contact Graspnet is employed to generate the 6 DoF grasp pose of the target. The pose of the receptacle, the location on which the object is placed, comes from adjusting from a reference object. For example, to put the cup on a plate, we optimize the pose of the plate and add a bias on the y-axis to get the target pose of the cup.Diffusion based Depth Completion. Algorithm 1 outlines the process for diffusion based depth completion, which provides a structured approach to filling in missing depth information with multiple views of RGB-D inputs, addressing common challenges in depth sensing tasks such as incomplete data or occlusions. Utilizing a diffusion process that leverages latent diffusion models (LDMs), the algorithm mainly consists of three steps: Initialize Diffusion Process, Perform Depth Diffusion, Refine Depth Maps. Finally, the algorithm outputs the refined and completed depth maps, which can be used for the following 6D Pose Estimation task to enhance the effect of its depth-based estimation method.Algorithm 1 Diffusion based Depth Completion 1: Input: Apply channel-wise averaging to smooth the depth map d enh for post-processing 25: end for 26: Return D enh Fine-grained 6D Pose Estimation based Manipulation. Algorithm 2 outlines the process for finegrained 6D pose estimation based manipulation, which aims to precisely estimate the pose and size of objects within a scene and subsequently generate appropriate action sequences for cross-subscene manipulation tasks. The algorithm mainly consists of four primary steps: Grounded Visual Perception, Depth-based Category-Level Pose Estimation, Coordinate Transformation, Generate Targetdriven Action Sequences. Finally, the algorithm returns the estimated 6D poses, 3D sizes, and the generated action sequences, which collectively enable precise and efficient manipulation across multi-subscenes. T : Specific types of manipulation tasks 6: Output: Apply Grounded-Light-HQSAM to obtain mask M I,q for query q from I Back-project the depth map D and mask M I,q into point clouds in the observed view Transform pose P i 6D from the object coordinates to the camera coordinates 24:Transform pose P i 6D from the camera coordinates to the robotic arm base coordinates 25: end for 26:Step 4: Generate Target-driven Action Sequences 27: for each P i 6D do 28:Based on manipulation task type T , integrate each target pose P i 6D 29:Invoke MoveIt! to generate the action sequence A i 30: end for 31: Return P 6D , S 3D , A 


Decision: The paper introduces a new method for robot mobile manipulation across multiple sub-scenes. The method uses 3D scene reconstruction based on 3D Gaussians, scene inpainting and depth completion to build a framework for mobile manipulation.

Strengths

- The method demonstrates how to leverage perception techniques including 3D Gaussian splatting, depth completion, and scene inpainting for mobile manipulation.
- The framework is evaluated on several real-world mobile manipulation tasks.

Weaknesses

The reviewers have raised several concerns about the paper.

- Reviewer pWYS mentioned the separation of the navigation and the manipulation of the proposed method.
- Missing citations of previous mobile manipulation works.
- Some missing details of the proposed method.

Post-Rebuttal
- The authors have successfully addressed the concerns from the reviewers.

Reviews: review: title: title: Separation of navigation and manipulation stands out as a critical weakness
paper_summary: paper_summary: The paper proposed a Coarse-to-fine framework for cross-subscene mobile manipulation applications.   The paper used a diffusion-based 3D Gaussian completion and inpainting technique to obtain the refined target pose for manipulation.  Real world mobile manipulation experiments are conducted.
main_review: main_review: The coarse-to-fine framework for cross-subscene mobile manipulation presented in this paper is a significant contribution, enhanced by its real-world mobile manipulation experiments. 

However, the separation of navigation and manipulation stands out as a critical weakness. Currently, the mobile aspect is just calling off-the-shelf APIs for moving between different tables, while the selling point of the paper is about mobile manipulation. Let's consider we are in a simulation, where perfect sensors (RGBD and camera poses) and a floating gripper are provided, what fundamentally distinguishes this setup from tabletop manipulation setups like F3rm? 

I would suggest: 
1. Consider navigation as an integral part of the tasks by introducing obstacles between tables. This would require the robot to predict base poses during navigation in addition to end-effector poses, thereby presenting a more comprehensive and realistic problem formulation.
2. Move beyond simplistic, toy-example tabletop setups and explore realistic scenarios where cross-subscene functionality is essential. For inspiration and practical examples, refer to the HomeRobot paper.
3. Also, the paper points out capture failure is the major limitation of the paper. I would suggest the authors try out iPhone cameras, which provide better camera pose tracking and depth than the current setup. Refer to CLIP-Fields, OK-robot, etc. Also, realsense D435 adding T265 is an option.

A lot missing citations in prior mobile manipulation papers need to be considered, including but not limited to:

CLIP-Fields: Weakly Supervised Semantic Fields for Robotic Memory https://arxiv.org/abs/2210.05663

Mobile ALOHA: Learning Bimanual Mobile Manipulation with Low-Cost Whole-Body Teleoperation https://arxiv.org/abs/2401.02117

Adaptive Mobile Manipulation for Articulated Objects In the Open World https://arxiv.org/abs/2401.02117

TeleMoMa: A Modular and Versatile Teleoperation System for Mobile Manipulation https://arxiv.org/abs/2403.07869
questions: questions: My major concerns are listed in the reviews above.

Some minor concerns:

“Grounded-Light-HQ-SAM is introduced to extract queries and encode semantics following into Gaussians, which can be queried to provide a coarse pose of the manipulation target. ” More details are needed to explain how you obtain the manipulation target pose.

“Unlike most mobile manipulation methods that only support simple pick-and-place tasks”
This is not true. A lot of prior mobile manipulation papers aer missing in the related work.

The main paper should include more comprehensive details about the task setup, specifically addressing elements such as the definition of task success, as well as the initial configurations of both the robot and the objects involved.
limitations: limitations: 2
review_summary: review_summary: I expect to see more realistic setup in the real-world experiments.
title: title: Review
paper_summary: paper_summary: The paper presents TaMMa (Target-driven Multi-subscene Mobile Manipulation), a method designed to enable a mobile base and a arm to efficiently navigate and perform single-arm manipulations across multiple similar scenes.
main_review: main_review: ### Summary:
The paper proposes TaMMa, Target-driven Multi-subscene Mobile Manipulation, a method for target-driven multi-subscene mobile manipulation using 3D Gaussian splatting, depth completion, and scene inpainting. It enables robots to perform manipulation tasks across multiple subscenes efficiently.

### Strengths:
- This paper presents a method to integrate of 3D Gaussian splatting, depth completion, and scene inpainting.
- The multi-subgoal strategy presented in the paper is a practical solution to improve target-driven navigation.
- The paper demonstrates that robots has the ability to handle complex tasks like pouring and stacking across scenes.
- The paper provides a relatively thorough experimental evaluation, comparing the proposed method with existing approaches.

### Weaknesses:
- Limited evaluation on only one robotic platform (Franka Panda arm with mobile base).
- Lack of comparison with some recent state-of-the-art mobile manipulation methods, though admittedly there are few that are open-sourced or easily reproducible
- The In-painting method is limited for not complex 3D objects, as noted by authors. The scalability of the proposed approach to more complex and larger environments is not very well discussed.
questions: questions: - How does the method scale to environments with more than 3 subscenes?
- How sensitive is the method to errors in initial scene reconstruction or depth estimation?
- If attempting to transfer to a different robot configuration, how will the approach be on different robot platforms or arm configurations?
limitations: limitations: 3
review_summary: review_summary: -
title: title: TaMMa Review
paper_summary: paper_summary: This work presents TaMMa (Target-driven multi-subscene mobile manipulation). A franka arm with a realsense camera on a mobile base accomplishes navigation + manipulation tasks across 3 different nearby tables. The approach leverages 1) Diffusion based depth completion, 2) 3D gaussians for scene representation, and 3) Scene inpainting to visualize what the effects of pick & place like actions on the environment.   TaMMa first scans only a few posed RGB-D frames of each subscene and unprojects them to a sparse 3D point cloud to initialize 3D Gaussians. Grounded-Light-HQ-SAM[3] extracts queries and encodes semantics into Gaussians, which can be queried to provide a coarse pose of the manipulation target.
main_review: main_review: Quality:

This method makes comparisons to F3rm and HomeRobot. It is challenging to find and compare to other methods for complex robotic system performance, I appreciate the effort that went into benchmarking this approach against prior works.


Clarity:

Overall the paper and video present clearly.

For this video it would be nice to include additional examples of image / depth inpainting rather than just videos of the robot moving & doing tasks.

More information about Diffusion based depth completion described in section 3.2 would be helpful. I am not confident that I understand how this is working from reading your paper. Adding some pseudo code to the appendix or an additional figure describing how this is formulated would be really helpful. My understanding is that you take an original depth image, add noise to it, and then train the diffusion-denoiser based on its ability to remove this added noise. If I am understanding this correctly, it isn’t clear to me that the training of this denoiser would actually work at removing noise artifacts common to realsense cameras (transparencies, reflective surfaces, object edges where stereo matches are challenging to obtain, etc). Some additional description around this section or in the appendix would be greatly appreciated. 

Alternatively additional clarity could be provided by releasing some of the associated code.

Originality:

This work leverages rough 3D gaussians for rough pose iestimation, that are then further refined. The presented work does a great job at demonstrating how 3D gaussians, SAM, and multi-view depth images can be integrated to enable a mobile manipulator.

Significance:

I think this work presents a strong baseline for mobile manipulation.
questions: questions: More information about Diffusion based depth completion described in section 3.2 would be very helpful.
limitations: limitations: 3
review_summary: review_summary: I think this work presents a strong baseline for mobile manipulation, and a nice synthesis of SOTA methods to accomplish robotic tasks on real hardware.