M2T2: Multi-Task Masked Transformer for Object-centric Pick and Place
Authors: Wentao Yuan
Adithyavairavan Murali
Arsalan Mousavian
Dieter Fox
Venue: CoRL 2023 Poster (2023)
Keywords: Object manipulation
Multi-task learning
Pick and place

Abstract: With the advent of large language models and large-scale robotic datasets, there has been tremendous progress in high-level decision-making for object manipulation. These generic models are able to interpret complex tasks using language commands, but they often have difficulties generalizing to out-of-distribution objects due to the inability of low-level action primitives. In contrast, existing task-specific models excel in low-level manipulation of unknown objects, but only work for a single type of action. To bridge this gap, we present M2T2, a single model that supplies different types of low-level actions that work robustly on arbitrary objects in cluttered scenes. M2T2 is a transformer model which reasons about contact points and predicts valid gripper poses for different action modes given a raw point cloud of the scene. Trained on a large-scale synthetic dataset with 128K scenes, M2T2 achieves zero-shot sim2real transfer on the real robot, outperforming the baseline system with state-of-the-art task-specific models by about 19% in overall performance and 37.5% in challenging scenes were the object needs to be re-oriented for collision-free placement. M2T2 also achieves state-of-the-art results on a subset of language conditioned tasks in RLBench. Videos of robot experiments on unseen objects in both real world and simulation are available at m2-t2.github.io.

M2T2: Multi-Task Masked Transformer for Object-centric Pick and Place
Wentao Yuan University of Washington
A Adithyavairavan Murali NVIDIA admurali @ @nvidia.com
Arsalan Mousavian NVIDIA amousavian mnviiaacoomiiiiiicmcmmmm
wentaoy @ss. washingoon.cdu
Dieter Fox University of Washington, NVIDIA fox mackess.wasd www.eesswsnnnnoned...
With the advent of large language models and large-scale robotic datasets, there has been tremendous progress in high-level decision-making for object manipulation [1, 2, 3, 4]. These generic models are able to interpret com- plex tasks using language commands, but they often have difficulties generalizing out-of-distribution objects due the inability of low-level action primitives. In contrast, existing task-specific models [5, 6] excel in low-level manipulation of unknown objects, but only work for a single type of action. To bridge this gap. we present M2T2, a single model that supplies different types of low-level actions that work robustly on arbitrary objects in cluttered scenes. M2T2 a transformer model which reasons about contact points and predicts valid gripper poses for dif- ferent action modes given raw point cloud of the scene. Trained on large-scale synthetic dataset with 128K scenes, M2T2 achieves zero-shot sim2real transfer the real robot, outperforming the baseline system with state-of- the-art task- specific models by about 19% in overall performance and 37.5% in challenging scenes were the object needs to be re-oriented for collision-free placement. M2T2 also achieves state-of-ther the-art results on a subset of language conditioned tasks RLBBench [7]. Videos of robot experiments on unseen objects both real world and simulation are available on our project website.
Abstract:
Keywords: Object Manipulation, Pick-and andeee Multi-task Learning
Introduction
The successful completion of many complex manipulation tasks such object rearrangement relies on robust action primitives that can handle large variety of objects. Recently, tremendous progress has been made in open-world object manipulation [1, 2, 3, 4] using language models for high-level planning. However, these methods are often restricted to scenes with a few fixed object shapes due to the limited capability low-level skills such as picking and placing. Meanwhile, there are task- specific models [5, 8, 9] that excel on a particular skill on large variety of objects. This leads us to the question: is it possible have a single model for different action primitives that works robustly on diverse objects?
We propose Multi-Tase Masked Transformer (M2T2), unified model for learning multiple action primitives. As shown in Fig. 1, given a point cloud of the scene, M2T2 predicts collision-free gripper poses for various types of actions including 6-DoF grasping and placing, eliminating the need to use different methods for different actions. M2T2 can generate a diverse set goal poses that provide sufficient options for low-level motion planners. can also generate more specific goal poses conditioned on language. Combining high-level task planners and the robust action primitives from M2T2 allows the robot to solve many complex tasks like the ones in RLBench [7]. Overall, our contributions are as follows:
7th Conference Robot Learning (CoRL. 2023), Atlanta, USA.

---

Scene Point Cloud
RGB (only for visualization)
Multi Task Masked
T ransformer
Object-centree 6-Dof Grasp
Orlentation-aware 4-DoF Placement
Figure 1: We propose M2T2, unified model for learning multiple action primitives. M2T2 takes raw 3D point cloud and predicts 6-DoF grasps per-object (lower left) and orientation-awar place- ments (lower right, where green means the object can fit any orientation and yellow means only a subset of orientations are possible). Colors on the point clouds are for visualization only.
1. We present M2T2, unified transformer model for grasping and placing, which outper- forms state-of-the-art methods [5, 6] in terms of success rate and output diversity.
2. A large-scale synthetic dataset for training M2T2, consisting of 130K cluttered scenes with 8.8K different objects, annotated with valid gripper poses for picking and placing.
3. We show that M2T2 achieves zero-shot sim2real transfer for picking and placing out-of- distribution objects, outperforming baseline by about 19%.
4. We show that M2T2 outperforms state-of-the-art art end-to-end method [10] on subset of RLBench [7], demonstrating its potential in solving complex tasks with language goals.
Related Work
Multi-Task Learning in Robotics With the advent of robotic datasets with diverse tasks [7, 11], many recent works have shown that learning multiple manipulation tasks with single model can improve sample-efficiency and performance. Some works learn common representation for mul- tiple tasks [12, 13], while other works [2, 10, 14] train end-to-end language-conditione policies via imitation learning. However, these end-to-end agents have hard time generalizing to out-of- distribution tasks and objects. In contrast, we take more modular approach. M2T2 supplies action primitives like "placing the mug" that work on unseen objects in the real world. By interfacing with other task planning modules [3, 4), M2T2 can be part a robust and flexible open-world manipulation system.
Object Grasping Grasping is the most fundamental skill of robot manipulator. Recently, 6-DoF, learning-bast bbased grasp pose generator becoming mainstream. These methods typically take a 3D point cloud [5, 8. 15. 16] or voxelization of the scene [17, 18] and predict 6-DoF gripper poses to stably grasp an object. There are also task-oriented grasp generators [19, 20] that predicts grasps for downstream tasks (e.g. handover). However, these works are geared toward single skill, grasping. While grasping is important skill, it cannot solve all the tasks in manipulation. M2T2 aim to build common formulation for different kinds of skills, including grasping.

---

Grasp Pases
Scone Point cloud
Encoder
Scene Embedding
Decoder
Grasp Contact Masks
Optional
Language Tokens
Language
Goal
CLIP
Grasp Tokens
Object #1, 0.93
Object #2, 0.84
Grasp
MLP
Objectness Score
Tokens
Attn
Attm
Attin
Place Tokens
Contact Decoder
Query
Object
Tokens
Place
Tokans
Object
Orientation
Orientetion #2
Embedding
Figure 2: M2T2 generates valid gripper poses for grasping and placing with single model. First, a 3D network (scene encoder) takes the scene point cloud and produces multi-scale feature maps. Then, the features cross-attended with learnable query tokens via transformer (contact de- coder). Finally, the output tokens are multiplied with per-point features and generate contact masks and gripper poses for each object (for grasping) and each orientation (for placing). For grasping, addition MLPs are applied to the output tokens and per-point features to predict objectness scores (to filter out non-object proposals) and grasp parameters (to reconstruct gripper poses). Optionally, the contact decoder can take a set of tokens encoding language goals to produce goal-conditioned grasping and placing poses.
Object Placement Placement is another important action mode for robot manipulator. Com- pared to grasping methods, placement is less studied until recently. [6] uses rejection sampling and learning-basd based collision detector [21] ] to find all possible placement positions. [22] generates place- ment configurations for set of objects based on language commands, but it does not consider the gripper. M2T2 predicts placement poses without the need for sampling, uses the same model for grasping and considers both the position and orientation of the object and the gripper.
3 Technical Approach
M2T2 predicts target gripper poses for multiple action primitives. Here, we consider two most fundamental action modes of robot manipulator, picking and placing.
Object-centric 6-DoF Grasping: The input is single 3D point cloud of the scene which can be extracted from commodity depth sensors. The output is set of object grasp proposals. Each object grasp proposal is set of 6-DoF grasp poses (3-DoF rotation + 3-DoF translation), indicating where the end effector robot arm needs to in order to pick up the object successfully.
The input is 3D point cloud of the scene plus a partial 3D point cloud of the object to be placed. The output is set of 6-DoF placement poses, indicating where the end effector needs to be so that when the object is released, it will be placed stably without collision. Note that M2T2 generates not only the location, but also the orientation of the object to be placed. This ensures that the object can be re-oriented before placed to into cluttered spaces.
Orientation-aware Placing:
The key idea of M2T2 is reason about contact points. We view picking as the robot making contact with target object using empty gripper and placing the robot using object in its gripper to make contact with a surface.

---

3.1 Architecture
The scene encoder encodes a 3D point cloud of the scene into multi-scale feature maps that serve as context for the contact decoder. Specifically, it produces four feature maps that are 1/64, 1/16, 1/4, times the input size respectively. Each feature vector in the feature maps is grounded to a point in the input point cloud. We adapt PointNet++ [23] designed for semantic segmentation as the scene encoder, but in principle, any network that produces multi-resolution feature maps from 3D point clouds can serve the scene encoder.
Scene Encoder:
The contact decoder is transformer that predicts where to make contact for both grasping and placing. We used the grasp representation of [5] for grasping where each grasp is anchored around the visible point on the object that makes contact with the gripper during grasping and the model predicts additional parameters specifying the relative transform of grasp with respect the contact point. We extend this representation for placing by defining contact point as the location where the center of object point cloud projects the table.
Contact Decoder:
As result, we can borrow the latest insight from image segmentation. In our case, we modify the masked transformer [24] to predict contact masks. The transformer takes a set of learnable query tokens through multiple attention layers. Feature maps of multiple resolutions from scene encoder are passed in via cross-attention at different layers. The output tokens of each layer are multiplied with the per-point feature map from scene encoder to generate interim masks. The interim masks are used to mask the cross attention in the next layer to guide the attention into relevant regions (thus the name "masked transformer"). After the last attention layer, the model produces G grasping masks and P placing masks, where G' is the maximum number of graspable objects and P is the number of placement orientations.
Objectness MLP: An MLP takes the grasp tokens and produces an objectness score for each token. This is filter out the non-object tokens since the number of graspable objects in the scene can vary (see Sec. 3.2 and Sec. 3.3 for how the score is used in training and inference).
Object Encoder: The object encoder is PointNet+ [23] which encodes 3D point cloud of the object to be placed to single feature vector that is added to the place query tokens.
The action decoder is a 3-layer MLP that takes the per-point feature map from scene encoder and predicts a 3D approach direction, a 3D contact direction and a 1D grasp width for each point, which are used to reconstruct grasp poses together with the contact points (see Sec. 3.3).
Action Decoder:
3.2 Training Objective
Grasping: The grasping objective consists of three terms: objectness loss Lobj. mask loss Lmask and ADD-S loss LADD-S-
Because the number of objects N in the scene is unknown, we set G, the number of grasp tokens, to a large number (see Sec. 4.2 for ablations). M2T2 outputs G scalar objectness scores o, and G per-point masks M.rrap, We use Hungarian matching to select N masks that best match with the ground truth. First, we compute the following cost for each prediction (o,, M3rwsp and ground truth mask M
(1) where BCE is binary cross entropy and DICE the DICE loss [25]. Now, we apply Hungarian matching the G N cost matrix C to obtain the set of indices M {m; that minimizes the total cost Σ Cmys- Then, we compute the objectness loss by labeling all matched tokens as positive and others as negative
Ciy= =1y=1- o, + BCE M13h) + DICE .M55)
Lobi
[[ E + (1 1(i E MM) o.)

---

We compute the mask loss between the matched masks and the ground truth as
T.mask
Σ
BC M + DICE(MP)) M
N
In practice, we find that computing the BCE only for the points with top k largest loss improves performance, where & 512 for grasping and k = 1024 for placing. This likely due to the large class imbalance (over 90% of the points are contact points). See Sec. 4.2 for ablations.
The ADD-S loss is introduced by [5] and is critical for good grasp confidence estimation (see Sec. 4.2 for ablations). To compute it, we first need to define 5 key points {vk} on the gripper. Then, for each pair of predicted grasp and ground truth grasp, we compute the total distance be- tween transformed key points
di Σ
(IReprrd Vu + tpprad) (129" ve 1
Next, we find the closest ground truth to each prediction n; = argmin,d,; and compute ADD-S as
LADD-S
5;den,
(5)
Cpred
xECpred
where Cpred is the set of contact points of predicted grasps and 8; is the grasp confidence, scalar between 0 and I taken from the contact masks before thresholding. Note that since the loss is weighted by predicted grasps that are far away from any ground truth grasp will get a larger penalty on confidence, which improves contact point estimation.
Placing: The placing objective is defined combination of BCE and DICE [25] between the predicted and ground truth placement masks
1
Σ BCE .MA) + DICE(M, .M
Lplacing = P ==1
(6)
This is the only loss for placing since no other learnable quantities are needed reconstruct the placement poses (see 3.3).
3.3 Model Inference
Grasp Pose Prediction: During inference, we first select the contact masks whose objectness score above 0.5. Then, for each point p within the contact mask, we take the corresponding grasp parameters from the action decoder to reconstruct a 6-DoF grasp pose (Rgrasps (grasp) E SE(() as
Egrasp
p+
+ da
2
R
Rgrasp
C
C a a
(8)
where c is the unit 3D contact direction, a the unit 3D approach direction, w is the 1D grasp width and is the constant distance from the gripper base to the grasp baseline (the line between two fingers). We refer readers to Contact- Gratacttrarsp-eett paper for more details on the formulation [5].
Placement Pose Prediction: The P placement contact masks represent valid placement locations when the object in the gripper is rotated by P discrete planar rotations Rplanar- To recover the placement poses, we first compute the bottom center b of the object point cloud which used as the reference point for contact. Then, we use forward kinematics to obtain the gripper's current pose (Ree, t.n). The 6-DoF placement pose (Rplaces &place) can be computed as
Egrasp p Rplanar (tue b)
(9)
R Rgraep Rploner Ree
(10)
5

---

3.5
1.0
1.3
0.9
0.9
Wrecsion
Precsion
0.6
64
0.4
M212
M222
KDT2
Cortact
GraspNet
866
8
022
A6 Ooverage
0.
08
1.0
0.
D.
0.
0.8
1.0
Coverage
Coverage
Coverage
(a) Grasping Seen
(b) Grasping Unseen
(c) Placing Seen
(d) Placing Unseen
Figure 3: M2T2 outperforms task-specific models Contact-GraspNerr [5] for grasping and CabiNet [6] for placing on objects from seen categories (a,c) and unseen categories (b,d).
1.0
1.0
1.0
0.8
0.6
Vreceson
Precision
0.6
9.3
0.4
0..4
quer
quee
Discrete
With Hard Neg Neg
ADD-S 10
200 queries
0.0
0.2
0.4
0.6
8.0
0.4
6.6
0.8
0
022
0A Ooverage
0.6
08
1.0
%1
0.2
2.4
0.8
1.0
Coverage
Coverage
Coverage
(a) ADD-S weight
(b) Number tokens
(c) Discrete rotation
(d) Hard negative mining
Figure 4: Ablation studies
3.4 Synthetic Data Generation
We build synthetic dataset with 130K cluttered scenes for training and evaluating 6-DoF picking and placing methods. There are 64K training scenes and IK test scenes for picking and placing each. There are 1 to 15 objects per scene scattered on a randomly sized table mounted with a Franka Emika robot arm. The objects are sampled from the ACRONYM [26] dataset, which contains 8.8K object models, each labeled with 2K grasps. The objects are from 252 different categories, 12 of which are excluded from training. Half of the test scenes contain only objects from the 12 unseen categories. For each scene, we render 512 512 depth image from random viewpoint above the table generate the scene point cloud. We include example images of the dataset in the appendix.
Experimental Evaluation
4.1 Evaluation in Simulation
We use the precision-coverage curve the metric for our evaluation in sim- ulation. To plot this curve, start with confidence threshold of and add grasps/placements to the set of predictions by incrementally lowering the confidence threshold until 0.5. In the mean time, we keep track of two numbers: precision, the percentage of successful grasps/placements, and coverage, the percentage of ground truth grasps/placements that are within 5 cm of any predicted pose. Finally, we plot the coverage on the x-axis and precision on the y-axis. In practice, we found that this curve good indicator model's performance in the real world.
Evaluation Metrie:
A grasp is considered successful if the gripper does not collide with the scene (including occluded parts) and is stable. We evaluate the stableness of grasp by shaking the grasped object for seconds in the Isaac gym simulator [27] with physx physics engine, identical to the evaluation in ACRONYM [26]. placement is considered successful both the gripper and the object are collision-free and the bottom of the object less than em from the correct placement surface.
M2T2 vs. Specialized Baseline Models: We compare M2T2 against two state-of-the-ard spe- cialized models: Contact-GraspNNt [5] for grasping and CabiNet (6] for placing. The results are summarized in Fig. 3. M2T2 outperforms both models by a significant margin, especially on the placement. This shows the advantage of orientation reasoning for placement. In many cases, it is not possible to find good placement pose without rotating the object in hand.

---

Table Real Robot Experiments Success Rates
Model
Pick
Place
Place re-orient
Overall
M2T2 (ours)
85.7%
72.2%
62.5% 25.0%
61.9%
Contact-GrasppNet [5] CabiNet [6]
76.2%
56.2%
42.9%
Figure 5: Our robot experimental setup. Left: Scenes where the target object (highlighted in red) needs to reoriented to be placed the placement region (shown in green). Right: A example of scene where objects are sequentially moved from the right to the initially empty region on the left.
4.2 Ablations
Choice of ADD-S: We find that setting a larger weight for the ADD-S loss has critical impact on the grasping performance. As shown in Fig. 4a, setting lower ADD-S increases grasp coverage the expense of precision which is not desirable.
Number of Grasp Queries: We experimented with different number of grasp query tokens and find 100 to be an appropriate number, as shown by the results in Fig. 4b.
We compared our model with variant where there is only single placement mask and the placement rotations regressed just like the grasp parameters. As shown by Fig. 4c, it is better to have set of placement masks corresponding to discrete rotations. Since multiple orientations of the object can be valid for given placement location, regressing to single rotation does not model the multi-modality placement orientations.
Discrete Continuous Rotation:
Importance of Hard Negative Mining: As mentioned in Sec. 3.2, we use hard negative mining (by applying the mask loss for 1024 points with the largest loss. Without this trick, the quality of most confident placements become significantly worse (see Fig. 4d).
4.3 Real Robot Experiments
We evaluated M2T2 tabletop manipulation setting with 7-DOF Franka Panda Robot and parallel jaw gripper. For perception, we used single Intel Realsense L515 RGB- D camera overlooking the scene. We use the motion planner from [6] for reactive path planning via model predictive control. The picking target and placement region are specified by the user with 3 clicks on the camera image. All inference run on a single NVIDIA Titan RTX GPU, which takes about 0.1 second per frame. After obtaining set of collision-free gripper poses from M2T2, the robot execute the that is closest to the current robot configuration in joint space and has feasible inverse kinematics solution for the robot arm.
Hardware Setup:

---

Table Success Rate Comparison on RLBench
Task
open drawer
turn tap
meat off grill
M2T2 (ours)
89.3 + 1.8% 80.0%
88.0 + 5.6%
86.7 + 1.8%
PerAct [10]
84.0%
84.0%
Results: Table shows the success rate of 21 pick-and-plan sequences in 7 different scenes. Each scene contains more than 5 objects and all objects are unseen during training. We do not provide 3D models for any of the object. The placing success rate is conditioned on picking success, i.e. overall success = picking success x placing success. We designed four scenes where the object has to be re-oriented before placing to fit in the target region, as shown on the left in Fig 5. We can see that M2T2 significantly outperforms the baseline system, which task-specific models. Notably, M2T2 9.5% higher for grasping than [5] and 37.5% higher than [6] for the more challenging re-orientation placement. 2/3 pick failures for M2T2 were for cup objects, which were out-of-distribution during training. 4/5 pick failures for the baseline [5] was because the model did not generate grasps at all or the generated ones were not reachable based on Inverse Kinematics (IK). On the other hand, M2T2 generates grasps with higher coverage and hence have a greater chance of being within the robot's s kinematic workspace. We only use 8 bins for the orientation discretization, and further increasing the discretization granularity could potentially reduce our placement error. The real robot executions can be found on the project website.
combination of state-of-the-art
4.4 Evaluataion on RLBench
RLBench [7] is commonly used benchmark to evaluate mulli-task robot manipulation methods. We found that many complex tasks in RLBench can be decomposed into sequence of primitive actions and solved by M2T2. We demonstrate this by training and evaluating our model on 3 RLBench tasks: open drawer, turn tap and meat off grill. M2T2 is able to outperform PerAct [10], a state-of- the-art multi-task model, on all tasks. The results are summarized in Table 2. We report average success rate over 25 random seeds. The standard deviation is computed with repeated trials for each seed. PerAct's results taken from the original paper. This demonstrates M2T2's ability to learn action primitives other than generic pick and place and to incorporate multi-modal inputs including language. More details of the experiments are in the appendix.
5 Conclusion
In this paper present Multi-Task Masked Transformer (M2T2), an object-centric transformer model for pick-and- lock-and-place -lace of unknown objects in clutter. We train M2T2 on a large-scale synthetic dataset of 130K scenes and deploy it on a real robot without any real-world training data. M2T2 outperforms state-of-the-art specialized models in 6-DoF grasping [5] and placing [6] by about 19% in overall success rate in the real world. M2T2 is especially proficient in re-orienting objects for precise collision-free placement. In future, we plan to integrate M2T2 with language-conditione task planners [3, 4] to build an open-world manipulation system that works everyday scenes with out-of-distribution objects.
Limitations: M2T2's s performance is bounded by the visibility of contact points. For example, it cannot predict grasps on the side of the box opposite to the camera. M2T2 is also not able to directly predict actions without contact points, such as lifting. During placing. M2T2 needs segmentation of the object in gripper in order to estimate how far the gripper needs to be from the contact point between the object and placement surface. The grasp predictions for each token can spread across multiple objects in close contact. Currently, M2T2 is trained and evaluated only tabletop scenes, but this could be improved by training with more diverse procedurally generated synthetic data such as in [6, 28].

---

Acknowledgments
This work is supported by NSF Award IIS-2024057. The project title is Collaborative Research: NRE: FND: Graph Neural Networks for Multi-Object Manipulation.
References
[1] A. Brohan, Y. Chebotar, C. Finn, K. Hausman, A. Herzog, D. Ho, J. Ibarz, A. Irpan, E. Jang, R. Julian, et al. Do as can, not as say: Grounding language in robotic affordances. In Conference on Robot Learning, pages 287-318. PMLR, 2023.
[2] Y. Jiang, A. Gupta, Z. Zhang, G. Wang, Y. Dou, Chen, L. Fei-Fei, A. Anandkumar, Y. Zhu, and L. Fan. Vima: General robot manipulation with multimodal prompts. arXiv preprint arXiv:2210.03099, 2022.
[3] Liang, W. Huang, Xia, P. Xu, K. Hausman, B. Ichter, Florence, and Zeng. Code as policies: Language model programs for embodied control. arXiv preprint arXiv:2209.07753, 2022.
[4] I. Singh, V. Blukis, A. Mousavian, A. Goyal, D. Xu, J. Tremblay, D. Fox, J. Thomason, and A. Garg. Progprompt: Generating situated robot task plans using large language models. arXiv preprint arXiv:2209. 11302, 2022.
[5] M. Sundermeyer, A. Mousavian, R. Triebel, and D. Fox. Contact-graspias Efficient 6-dof grasp generation in cluttered scenes. In 2021 IEEE International Conference on Robotics and Automation (ICRA), pages 13438-123 13444. IEEE, 2021.
[6] A. Murali, A. Mousavian, C. Eppner, A. Fishman, and D. Fox. Cabinet: Scaling neural col- lision detection for object rearrangement with procedural scene generation. arXiv preprint arXiv:2304.093300 00302,, 2023.
[7] James, Z. Ma, Arrojo, and A. Davison. Rlbench: The robot learning benchmark & learning environment. IEEE Robotics and Automation Letters, 5(2):3019-3026, 2020.
[8] A. Murali, A. Mousavian, C. Eppner, C. Paxton, and D. Fox. 6-dof grasping for target-driven object manipulation in clutter. In 2020 IEEE International Conference on Robotics and Au- tomation (ICRA), pages 6232-6238. IEEE, 2020.
[9] S. S.Snn, A. Zeng. J. Lee, and T. Funkhouser. Grasping in the wild: Learning 6dof closed-loop grasping from low-cost demonstrations. Robotics and Automation Letters, 2020.
[10] M. Shridhar, L.. Manuelli, and D. Fox. Perceiver-actor: A multi-task transformer for robotic manipulation. In Conference on Robot Learning, pages 785-799, PMLR, 2023.
[11] J. Duan, Y. R. Wang, M. Shridhar, D. Fox, and R. Krishna. Ar2-d2: Training a robot without robot. arXiv preprint arXiv:2306.13818 2023.
[12] L. Pinto and A. Gupta. Learning to push by grasping: Using multiple tasks for effective learning. In IEEE International Conference on Robotics and Automation (ICRA), pages 2161- 2168. IEEE, 2017.
[13] W. Yuan, C. Paxton, Desingh, and D. Fox. Sornet: Spatial object-centric representations for sequential manipulation. Conference on Robot Learning, pages 148-157. PMLR, 2022.
[14] A. Brohan, N. Brown, J. Carbajal, Y. Chebotar, J. Dabis, C. Finn, K. Gopalakrishnan, K. Haus- man, A. Herzog, J. Hsu, et al. Rt-1: Robotics transformer for real-world control at scale. arXiv preprint arXiv:2212.0688 068177 2022.

---

[15] A. Mousavian, C. Eppner, and D. Fox. 6-dof graspnet: Variational grasp generation for object manipulation. Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 2901-2910, 2019.
[16] H.-S. Fang, C. Wang, M. Gou, and C. Lu. craspnet-lbillion: A large-scale benchmark for general object grasping. In Proceedings of the IEEE/CVF Conference on Computer Vision and Patlern Recognition, pages 11444-1 1 1453, 2020.
[17] M. Breyer, J. J. Chung, Ott, S. Roland, and N. Juan. Volumetric grasping network: Real-time
6 dof grasp detection in clutter. In Conference Robot Learning. 2020.
[18] Z. Jiang, Zhu, M. Svetlik, Fang, and Zhu. Synergies between affordance and geometry: 6-dof grasp detection via implicit representations. Roboties: science and systems, 2021.
[19] K. Fang, Y. Zhu, A. Garg. A. Kurenkov, V. Mehta, L. Fei-Fei, and S. Savarese. Learning task- oriented grasping for tool manipulation from simulated self-supervision. The International Journal of Robotics Research, 39(2-3):022216,, 2020.
[20] A. Murali, W. Liu, K. Marino, S. Chernova, and A. Gupta. Same object, different grasps: Data and semantic knowledge for task-oriented grasping. In Conference on robot learning, pages 1550-1557. PMLR, 2021.
[21] M. Danielczuk, A. Mousavian, C. Eppner, and D. Fox. Object rearrangement using learned im- plicit collision functions. In 2021 IEEE International Conference Robotics and Automation IEEE, 2021.
(ICRA), pages
[22] W. Liu, C. Paxton, T. Hermans, and D. Fox. Structformer: Learning spatial structure for tonguage-guider semantie rearrangement of novel objects. In 2022 International Conference on Robotics and Automation (ICRA), pages 6322-6329 IEEE, 2022.
[23] C. R. Qi, L. Yi, H. Su, and J. Guibas. Pointnet++:: Deep hierarchical feature learning on point sets metric space, Advances neural information processing systems, 30, 2017.
[24] B. Cheng, A. Choudhuri, I. Misra, A. Kirillov, R. Girdhar, and A. G. Schwing. Mask2former for video instance segmentation. arXiv preprint arXiv:2112.10764 2021.
[25] F. Milletari, N. Navab, and S.-A. Ahmadi. V-net: Fully convolutional neural networks for volumetric medical image segmentation. In 2016 fourth international conference on 3D vision (3DV), pages 565-571. leee, 2016
[26] C. Eppner, A. Mousavian, and D. Fox. Acronym: A large-scale grasp dataset based on sim- ulation. In 2021 IEEE International Conference Robotics and Automation (ICRA), pages 2222-6227 IEEE, 2021.
[27] V. Makoviychuk, Wawrzyniak, Guo, M. Lu, Storey, M. Macklin, Hoeller, N. Rudin, A. Allshire, A. Handa, et al. Isaae gym: High performance gpu-based physics simulation for robot learning. arXiv preprint arXiv:2108. 10470, 2021.
[28] A. Fishman, A. Murali, C. Eppner, B. Peele, B. Boots, and D. Fox. Motion policy networks. In Conference on Robot Learning, pages 967-977. PMLR, 2023.
[29] A. Radford, J. W. Kim, C. Hallacy, A. Ramesh, G. Goh, S. Agarwal, G. Sastry, A. Askell, Mishkin, J. Clark, et al. Learning transferable visual models from natural language supervi- sion. In International conference on machine learning, pages 8748-8763, PMLR, 2021.
[30] I. Loshchilov and F. Hutter. Decoupled weight decay regularization. arXiv preprint 17X1v:1711.05101, 2017.
10

---

Appendix
A Further Architectural Details
Scene Encoder The scene encoder is PointNet++ [23] with 4 multi- multi-ressoluiion abstraction layers the encoder and feature propagation layers as the decoder. The input point cloud is sub- sampled to 16384 points. Each set abstraction layer will select N/4 seed points using furthest point sampling where N the size of the input pointwise feature map. Then, local features are computed around each seed point and propagated with an MLP. As a result, the scene encoder produces feature maps of decreasing resolution, with 16384, 4096, 1024 and 256 points respectively. We use the first per-point feature map for prediction and the remaining 3 for cross-attention.
Contact Decoder The contact decoder takes G 100 grasp tokens and | = 64 placement tokens as input. These input query tokens are randomly initialized and learned during training. The G + P query tokens are fed into a transformer network with blocks. Each block consists of a cross- attention layer, a self-attention layer and feedforward MLP layer. the cross-attention layer, the query tokens are cross attended with one of the feature maps produced by the scene encoder to incorporate scene context. the self-attention layer, the query tokens are attended with themselves propagate information among different queries. The width (i.e. dimension of each token) is set 256. The input tokens of each transformer block (including the initial onc) are also used to produce intermediate mask predictions. Specifically, the tokens are multiplied with the per-point feature map from the scene encoder, passed through sigmoid and thresholded to generate per-point mask for each query. These intermediate masks are not only supervised by the ground truth masks during training, but also subsampled and used as attention masks for the cross-attention layer. This forces the network focus on relevant regions in the scene.
While the contact decoder is inspired by [24]. is specially designed to handle 3D inputs instead of images. For example, since the context features are grounded to 3D points, we compute position encodings from their 3D locations during cross-attention.
Modifications for RLBench In RLBench, we break down tasks like "put meat off grill" into predicting single grasp or placement pose conditioned on the language goal. To make the output conditioned on language, as shown Fig. A, we can introduce additional language tokens as query tokens, where the language tokens come from the language goal embedded by frozen CLIP [29] encoder. Following PerAct [10], we trained M2T2 on 100 demos and evaluate on 25 demos with random seeds different from training.
Scene Point Cloud
Action Decoder
Scene Embedding
Language Takens
Language Goal
take the steak off the grill
Grasp Tokan
Pace Tuken
Contact Decoder
Objjct
Cliject Embedding
Point Cloud
Figure A: Network for language- conditioooo bick-and-place in RLBench. Compared to the network for generie pick-and- -lacee, there are only grasp and place query token. The predicted grasp and placement are conditioned on language commands encoded by frozen CLIP.
11

---

B Comparison Against Single-task Model
We have trained our model to only perform a single task. These sass--pecialized models are worse than our multi-task model (see Fig. Ba, Bb). This shows the importance formulate both picking and placing under the same framework.
1.0
1.00
Pick Place
Pick Place
0.9
Pick Only
Place Only
0.95
Precision
0.8
Precision
0.90
0.7
0.85
0.6
0.0.0
0.80.0
0.2
0.6
Coverage
Coverage
(a) Against pick-only
(b) Against place-only
Figure B: Multi-task vs Single-task Models
C Training
M2T2 is trained using the Adam-W [30] optimizer with a fixed learning rate of 0.0008 on 8 V100 GPUs for 160 epochs. The batch size is 16 on each GPU. The training takes about 2 days to finish.
D Data Generation
We procedurally generated a large-scale synthetic dataset for training M2T2, as shown Fig C. In each scene, we randomly place 1-15 objects from the ACRONYM dataset [26] on table. Each object ACRONYM are labeled with 2000 grasps. We transform these grasps by the object pose and filter out colliding ones. The camera pose randomized around the entire hemisphere above the table, making M2T2 very robust to viewpoint changes.
Figure C: Examples for large-scale synthetic dataset, for the grasping (top) and placing (bottom)
tasks respectively. Objects are randomly sampled from ACRONYM [26]. Each scene can contain up 15 objects, which creates many very cluttered scenes. We also include robot in the observation to simulate realistic occlusion by the robot arm.
12


Reviews:
[{"review_id": "uTyZuF-OVHe", "reviewer": "Reviewer_whqu", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "The authors proposed a transformer-based approach to learning multi-task pick and place. In order to learn such a model, the authors introduce a synthetic data generation pipeline to train the policies. \n", "summary_of_recommendation": "Overall the proposed approach makes sense, and experiments are provided. However, the video demonstration of placing one object specifically in a position seems to sell the model short. Since the proposed model is targeted at performing pick-and-place seamlessly, I think the authors need to demonstrate how the learned model is used to perform a long horizon task of setting the table. Many questions need to be clarified before this score remains or improves.", "technical_quality": "Good: Generally convincing, but some additional experiments or analyses needed.", "clarity_of_presentation": "Good: Minor revision needed to improve organization or clarity.", "potential_impact": "", "robotics_focus": "Sufficient demonstration on hardware", "quality_of_limitations": "Limitations are addressed clearly", "questions_for_rebuttal": "- Please address the points mentioned in the weakness above\n- How is the ability of the model to perform a very long horizon task of placing? What kind of complicated tasks can this model handle?\n- Is this model able to perform tasks where objects might be placed on planes of different heights (for example, a shelf)?  \n- What does “Masked Transformer” mean in the title? It is confusing which “masks” this title refer to.\n", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "Strengths:\n- A single model that predicts both pick and place skills.\n- Provides a scalable pipeline in simulation to train robot policies.\n\nWeakness:\n- The first paragraph in Related Works is confusing. How is it relevant to this paper? Why do the authors need to have a specific paragraph when there is also a paragraph on multi-task learning for robotics? No connection is made between this paragraph and the proposed work\n- The task definitions are not self-explanatory. The videos also only include limited motion or behaviors. Please provide more information/details in the store.\n- This paper requires more explanation on how the learned models would be used in task executions.\n- The experiments need more ablations on how each design choice impacts the policy performance. The current status totally misses such a part.\n- Additional comments: It was not very clear whether it uses RGB-D or Depth point clouds. Figure 1 is quite misleading, given that the models did not use any RGB information. I think it’s better to either display depth images in Figure 1 or add captions to clarify that only depth is used in the following model design.\n- The authors chose to use a discrete set of rotations and claim that continuous rotations will have the issue of regressing to multi-modality (Ln204 - Ln 208). But surely, a model can regress to multiple values.  ", "extra_scores": {"impact": "3: The work is incremental and unlikely to have much impact, even though it may be technically correct and well executed."}, "extra_text": {}}, {"review_id": "md8bRV4b9C", "reviewer": "Reviewer_bMyz", "summary": "", "questions": "", "limitations": "", "rating": 2, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "The paper argues that there are two recent bodies of work. The first is to leverage Large Language Models (LLMs) and large-scale robotic datasets for high-level decision-making for object manipulation. However, these generic models are limited in terms of low-level pick-and-place primitives. The second is specialized models excel in low-level manipulation of unknown objects but only work for a single task. The paper claims that there is a gap to bridge. It proposes Multi-Task Masked Transformer (M2T2) which intends to achieve the goal of pick and place arbitrary objects in cluttered scenes. M2T2 adopts a transformer model which predicts pick and place poses for all (unknown) objects given just a point cloud observation of the scene. M2T2 is trained with a large-scale synthetic dataset consisting over 160K scenes and evaluated on zero-shot sim2real transfer on the real robot.", "summary_of_recommendation": "The paper's motivation is questionable, does not seem to be true. The architecture is also not clearly motivated. The baseline methods chosen are not clearly explained.", "technical_quality": "Fair: Significant additional evidence required to support the claims.", "clarity_of_presentation": "Fair: Major revision needed to improve organization or clarity.", "potential_impact": "", "robotics_focus": "Sufficient demonstration on hardware", "quality_of_limitations": "Limitations are addressed clearly", "questions_for_rebuttal": "Please address the three key weaknesses. \n\nBesides, the paper has serious writing, organization, and positioning problems. The intro and related work does not give a clear picture of the proposed method in the context of related work. The architecture is not clearly motivated other than it takes inspiration from mask2former. Is the scene encoder and contact decoder from mask2former the key contribution to performance gains? \n\n=== Post rebuttal\nI am slightly more positive given the new results comparing with PerAct. I still think the paper is borderline. Please interpret my score after rebuttal to be in between weak accept and reject. I still think the architecture is very much specialized, and may not be relevant for a system that solves manipulation tasks in practice. Do you really need specific modules for pick only or place only? Why can not pick and place policies be learned without these specialized modules like RT-2? In particular, will these specialized modules hurt out of distribution generalization?", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "Strengthes\n\n1. The paper presents a unified transformer model for object-centric, 6-DoF grasping and placing. Experimental results demonstrate that M2T2 outperforms methods Contact-graspnet and CabiNet [7, 8] in terms of success rate and output diversity.\n\n2. The paper collects a large-scale dataset which consists of 164K cluttered scenes and 8.8K different objects. \n\n3. It shows that M2T2 achieves zero-shot sim2real transfer for pick-and-place in unknown scenes and objects, outperforming baseline methods by about 19%.\n\nWeaknesses\n\n1. The paper's characterization of the current gap does not seem to be true. We see general foundation models including Palm-E, RT-1 which are capable to multiple manipulation tasks includes picking, placing, opening and closing drawers, getting items in and out drawers, placing elongated items up-right, knocking them over, pulling napkins and opening jars. These models predict 6-DOF (translation and rotation) and grasping parameters. For example, RT-1 exhibits varying levels of generalization: L1 for generalization to the new counter-top layout and lighting conditions, L2 for additionally generalization to unseen distractor objects, L3 for additional generalization to drastically new task settings, new task objects or objects in unseen locations such as near a sink. It is not clear to the reviewer these models are limited by low-level manipulation primitives.\n\nPaLM-E: An Embodied Multimodal Language Model, https://palm-e.github.io/\nRT-1: Robotics Transformer for Real-World Control at Scale, https://robotics-transformer.github.io/\n\n2. The paper claims M2T2 is a unified architecture. However, it seems to be designed separately for picking and placing. Then the two architectures are merged in a straightforward way. This is evident that object encoder is only used for placement tasks and action decoder only applies to picking tasks. Please explain the advantage of M2T2 with respect to general foundation model architectures such as RT1, PerAct, etc. \n\n3.  The baseline methods chosen to compare is not clearly explained. Why combining Contact-GraspNet [7] + CabiNet [8] is the best SOTA? Why not PerAct and other models? It is not clear what contributes to M2T2's performance gain with respect to the baselines.", "extra_scores": {"impact": "3: The work is incremental and unlikely to have much impact, even though it may be technically correct and well executed."}, "extra_text": {}}, {"review_id": "ds9sRMxdlb", "reviewer": "Reviewer_Bsjg", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "The authors propose a multi-task transformer model that predicts contact masks for picking and placing from a 3D point cloud of the scene.\nThese contact masks are then used to compute grasp and placement poses.\nThe model is trained end-to-end using a synthetic dataset with ground-truth information.\nThe authors evaluate the model and several design choices in simulation as well as its zero-shot sim2real performance.\nTheir experimental results indicate that the proposed method outperforms state-of-the-art single-task baselines.", "summary_of_recommendation": "The proposed method is novel, relevant and technical sound. However the paper lacks clarity and there are questions that should be addressed. Therefore I am recommending a weak reject.", "technical_quality": "Good: Generally convincing, but some additional experiments or analyses needed.", "clarity_of_presentation": "Good: Minor revision needed to improve organization or clarity.", "potential_impact": "", "robotics_focus": "Sufficient demonstration on hardware", "quality_of_limitations": "Additional details required", "questions_for_rebuttal": "Technical Approach:\n- Why is the query embedding formed using a task embedding and grasp/place embedding? What is the Motivation for this, e.g. why does the task embedding exist?\n- How are the different losses combined/weighted?\n- How is the object point cloud computed? If some segmentation model is used, imo this should be addressed in the limitations section.\n- More details on architecture hyperparameters, e.g. #neurons in the MLPs, would be nice for reproducability (supplementary material)\n\nExperiments:\n- You ablate the importance of hard negative mining, however, in the technical approach section hard negative mining is never mentioned.\n- It is unclear to me whether the comparisons to the baselines is fair: Are the baselines retrained on the same dataset as your model? How do they compare in model capacity to your model? Is your model the first that predicts placement poses with planar rotation from point clouds and, if not, why not compare to a method that also considers planar rotation? \n- Would be interesting to investigate/ablate the effects of the multi-task setting, e.g. train the architecture only for grasping or placing. How much of the performance gain comes from the architecture (which differs from the compared baselines)? How much from the multi-task training?\n- Would be interesting to report inference time. How does inference time compare to the inference time of the baselines?\n- The model predicts multiple grasp and placement poses. For the real robot experiments, how is decided which of these grasp and placements are executed for one pick-and-place sequence?", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "The paper tackles an important problem in robotic object manipulation: Finding grasp and placement poses. It proposed a novel technique that learns to predict these poses using a single model, and by that clearly differs from previous work. However, it lacks clarity and the experiment section could be improved. \n\nStrength:\n+ learns joint representation for picking and placing\n+ generates a set of possible grasps/placements in one forward pass\n+ proposed method is easily scalable to large training datasets\n+ consider placement position and planar rotation\n+ experiments show generalization to unseen objects\n\nWeakness:\n- paper lacks clarity at some points (see questions)\n- unclear whether the comparison to the state-of-the-art methods is meaningful (see questions)\n- imo further comparisons to baselines would be nice (see questions)\n- imo further ablation studies would be interesting (see questions)", "extra_scores": {"impact": "3: The work is incremental and unlikely to have much impact, even though it may be technically correct and well executed."}, "extra_text": {}}, {"review_id": "SAzG-n_VpfN", "reviewer": "Reviewer_dMwG", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "The paper presents Multi-Task Masked Transformer (M2T2), a solution for picking and placing arbitrary objects in cluttered scenes (which might require careful reorientation of the grasped objects before placing them). The method is pretty general in the sense that it takes raw RGB-D point clouds as input with no known object models. It’s trained with a large scale synthetic dataset and achieves zero-shot sim2real transfer  on the real robot. It significantly outperforms the state-of-the-art pick and place methods, respectively, especially in challenging scenes where the objects need to be reoriented for collision-free placement with precision. \n", "summary_of_recommendation": "Based on the review above, I recommend Weak Accept. I am open to changing my mind during the rebuttal. \n", "technical_quality": "Very Good:  Convincing.", "clarity_of_presentation": "Very Good:  Well organized and clear.", "potential_impact": "", "robotics_focus": "Sufficient demonstration on hardware", "quality_of_limitations": "Limitations are addressed clearly", "questions_for_rebuttal": "- Why are the two task embeddings needed, if you already have grasp embeddings [GxD] and place embeddings [PxD]?\n- How is the grasp confidence s_{i} predicted? Is it the same as M_{i}? \n- During real-world model inference, for a given object, is the grasp with the highest confidence being executed? If it fails, does the robot retry? \n- During training, do the groundtruth grasps/placements consist of zeros and ones, or are the ground truth grasps/placements also annotated with confidence?\n- On a related note, how are the ground truth grasps for each object generated? Is clutter taken into account? For example, if a cup is placed closely next to another object on the right hand side, are the grasps from the right hand side annotated as invalid (because the gripper would collide with that other object)? Same question applies to the ground truth placements. \n- If the answer to the above question is Yes, which means the dataset takes clutter into account, what about the whole trajectory of the robot arm to reach the desired gripper pose? In other words, for each positive label of ground truth grasp in the dataset, is it guaranteed that there is an entire trajectory for the robot to accomplish that grasp without colliding with any object in the scenes. For example, in the supplementary video, the robot tries to place the bowl next to the cracker box. Even though the bowl and the gripper don’t collide with the cracker box in the end configuration, the robot arm actually collides with the cracker box during the execution. How would you handle cases like these? \n- In the limitation section, it’s mentioned that “Additionally, the grasp predictions for each token can spread across multiple nearby objects even though during training, we want each pick token to predict grasps corresponding to only one object.” Can you address this with a better design of loss function?\n", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "Overall the paper has quite a lot of noteworthy strengths. First, the method is fairly general: a unified transformer model for object-centric 6 DoF grasping and 4 DoF placing that is based on raw 3D RGB-D point clouds. The architecture and technical approach (as described in section 3) is intuitive, elegant, and well-explained. Essentially the paper leverages existing motion generation approaches and transforms the pick-and-place task from a “robotics” task to a “3D scene understanding” task. When trained with large-scale dataset and supervised learning (with well-designed losses, as described nicely in section 3.3), the model seems to perform pretty well.\n\nFurthermore, the experimental evaluation both in sim and in real are fairly solid. The proposed method is compared against the respective state-of-the-art models for picking and placing and achieves better results. It’s also noteworthy that the proposed method achieves sim2real transfer in a zero-shot manner. In fact, the real experiments are very challenging because each scene contains more than 5 objects and all objects are unseen during training. Although not perfect, the proposed method also beat baselines by a significant margin in the real experiments. Finally, the ablation studies in 4.2 are helpful for readers to understand which part of the algorithmic design matters.\n\nIn terms of rooms for improvement, the paper can benefit from including more failure case analysis in both sim and real. Also, it would be helpful if the authors can quantify the sim2real gap when performing zero-shot transfer. Right now the sim results and the real results are completely separate and it’s difficult for researchers to draw lessons from them. Figure 2 can probably be improved to be more self-contained. In other words, as for now, Figure 2 is very difficult to understand without reaching section 3.1 in detail. Figure 4 can also be improved (the images are dark and a bit distorted). Maybe also consider adding another figure that is analogous to Figure 4 for placing. Also, the supplementary video is faulty (a large portion of the video is static, e.g. stuck starting from 0:44 for instance). Finally, I have compiled a list of clarification questions for the paper authors below. It would be great if the authors could help clarify (at least some of) those points in the revised manuscript.\n\n", "extra_scores": {"impact": "3: The work is incremental and unlikely to have much impact, even though it may be technically correct and well executed."}, "extra_text": {}}]

Author rebuttal:

Meta-review: The paper proposes an unified transformer based model for predicting grasp poses and placements for unknown objects in cluttered scenes.
While overall the presented method and ideas – unified transformer based model , the large-scale training (and dataset) is appreciated, and real word experiments are provided, there are mixed feeling among the reviewers. The main critic points are related to the comparison (and definition) of the state-of-the-art comparisons, ablation studies, additional experiments, and missing details on the tasks, execution, model, etc. During the rebuttal, the authors tackled most of the raised concerns and provided additional requested evaluations. The general consensus tends to accept the paper, but the authors need to update the manuscript with the additional evaluations and clarifications as well (taking the page limit into account!).
Decision: Accept (Poster)