BiAssemble: Learning Collaborative Affordance for Bimanual Geometric Assembly
Authors: Yan Shen
Ruihai Wu
Yubin Ke
Xinyuan Song
Zeyi Li
Xiaoqi Li
Hongwei Fan
Haoran Lu
Hao Dong
Venue: ICML 2025 poster (2025)
Keywords: Bimanual Manipulation
Geometric Shape Assembly
Primary area: applications->robotics
TL;DR: We introduce BiAssemble, a framework that learns bimanual collaborative affordances for long-horizon geometric shape assembly task.

Abstract: Shape assembly, the process of combining parts into a complete whole, is a crucial skill for robots with broad real-world applications. Among the various assembly tasks, geometric assembly—where broken parts are reassembled into their original form (e.g., reconstructing a shattered bowl)—is particularly challenging. This requires the robot to recognize geometric cues for grasping, assembly, and subsequent bimanual collaborative manipulation on varied fragments. In this paper, we exploit the geometric generalization of point-level affordance, learning affordance aware of bimanual collaboration in geometric assembly with long-horizon action sequences. To address the evaluation ambiguity caused by geometry diversity  of broken parts, we introduce a real-world benchmark featuring geometric variety and global reproducibility. Extensive experiments demonstrate the superiority of our approach over both previous affordance-based and imitation-based methods.

BiAssemble: Learning Collaborative Affordance for Bimanual Geometric Assembly
Yan Shen 12 Ruihai Wu" Yubin Ke
Xinyuan Song' Zeyi Li
Xiaoqi Li 12 Hongwei Fan 12 Haoran Lu Hao Dong' 12
Abstract
into fully constructed piece, emphasizing both the func- tional role of each part and the overall structural design. In contrast, geometric assembly involves reconstructing broken objects, like piecing together parts shattered mug, to re- store their original form. While furniture assembly has been relatively well-studied-rranginn from computer vision tasks that predict part poses in the assembled object (Zhan et al., 2020) to robotic systems that assemble parts in both simula- tion (Ankile et al. [024 Yu et al. 2021 Wang et al. 2022a)
Shape assembly, the process of combining parts into a complete whole, is a crucial robotic skill with broad real-world applications. Among vari- ous assembly tasks, geometric assembly- broken parts reassembled into their original form (e.g., reconstructing a shattered bowl)-is particularly challenging. This requires the robot to recognize geometric cues for grasping, as- sembly, and subsequent bimanual collaborative manipulation on varied fragments. In this pa- per, We exploit the geometric generalization of point-level affordance, learning affordance aware of bimanual collaboration in geometric assem- bly with long- horizon action sequences. To ad- dress the evaluation ambiguity caused by geom- ctry diversity of broken parts, we introduce a real-world benchmark featuring geometric va- riety and global reproducibility. Extensive ex- periments demonstrate the superiority of approach over both previous affordance-based and imitation-based methods. Project page:
and real-world environments (Heo et al. 2023 Suárez-Ruiz
et al.) 2018 Xian et al. 2017 geommtrric assembly re- mains under-explored despite its significant potential for real-world applications (Sellán et al. 2022] Lu al. (024b))
such repairing broken household items, reconstructing archaeological artifacts (Papaioannou & Karabassi 2003
assembling irregularly shaped objects in industrial tasks, aligning bone fragments in surgery (Liu et al. 2014), and reconstructing fossils in palcontology (Clarke et al. 2005).
Previous works on geometric assembly primarily focused on generating physically plausible broken parts through precise physics simulations in the graphics domain (Sellán et al.) 2022, 2023), and estimating the target assembled part poses based on observations in the computer vision domain (Wu et al. 2023c Lu et al., 2024c| Qi et al.] (025). These studies only consider the geometries and ideal assembled poses of broken parts, dismissing the process of step-by-step assembling parts the complete shape. However, different from opening a door closing drawer, only with the ideal part poses, it is difficult for a robot to directly and successfully manipulate broken parts to the complete shape.
1. Introduction
Shape assembly, the task of assembling individual parts into complete whole, is a critical skill for robots with wide-ranging real-world applications. This task can be
broadly categorized into two main branches: furniture as-
The challenges of the above robotic geometric shape assem- bly task mainly come from the exceptionally large observa- tion and action spaces. For the observation space, the broken parts have arbitrary geometries, and the graspness the object surface should consider not only the local geometry itself, but also whether grasping on such point can afford the subsequent bimanual assembly actions. For the action space, as illustrated in Figure T it requires long- long-horizon action trajectories. Given the contact-rich nature of the task, where collisions among the two parts and two robots will easily exist, the actions should be fine- gra-gga and aware
sembly (Zhan et al. 2020; Heo al.) 2023 Lee al.) 202T and geometric assembly (Wu et al., 2023c] Sellán et al. 2022
Lu et al. 2024c). Furniture assembly focuses com- bining functional components, such as chair legs and arms,
Equal contribution *CFCS, School of Computer Science, Peking University *PKU-Agibot Lab. Correspondence to: Hao Dong
Proceedings of the nnnd International Conference Machine Learning. Vancouver, Canada. PMLR 267, 2025. Copyright 2025 by the author(s).

---

BiAssemble: Learning Collaborative Affordance for Bimanual Geometric Assembly
C. Real- World Experiment Learning Assembly from Disassembly A. Direct Learning
(al)
Initial Observation
(n2) Pick-up
(a3) Alignment
(a4) Assembly
(b1) Initial Observation
(b2) Pick-up
(b3) Alignment
(b4) Assembly
S
A
Transformation
Affordance
Figure 1. (A) Direct learning long-horizon horizon action trajectories geometric assembly may face many challenges: grasping ungraspable points, grasping points not suitable for assembly (e.g., seams of fragments), robot colliding with parts and the other robot. (B) We formulate this task into steps: pick-up, alignment and assembly. For assembly, we predict the direction that will not result in part collisions. For alignment, transformed any assembled poses poses easy for the robot manipulate from the initial poses without collisions. For up. we learn point-level affordance aware of graspness and the following 2 steps. (C) Real- World Evaluations with affordance predictions
mugs and the corresponding manipulation.
of bimanual collaboration. Consequently, the policy must account for geometry, contact-rich assembly processes, and bimanual coordination.
geometric collisions), with part poses denoted alignment poses. We then gradually move the fragments toward each other to fit them together precisely. The alignment poses of the two fragments can be obtained by disassembling as- sembled parts in opposite directions. With this information, it becomes straightforward to extend the geometry-awee aware affordance to further aware of whether the controller can move fragments into the alignment poses without collisions.
We propose our BiAssemble framework for this challenging task. For geometric awareness, we utilize point-level affor- dance, which is trained to focus on local geometry. This ap- proach has demonstrated strong geometric generalization diverse tasks (Wu et al.) 2022; 2023b), including short-term bimanual manipulation Zhao al. 2022), such pushing box lifting basket. To enhance the affordance model with an understanding of subsequent long-hone horizon bimanual assembly actions, we draw inspiration from how humans intuitively assemble fragments: after picking up two frag- ments, we align them at the seam, deliberately leaving a gap (since directly placing them in the target pose often causes
We develop simulation environment where robots can be controlled to assemble broken parts. This simulation environment bridges the gap between vision-based pose pre- diction for broken parts and the real-world robotic geometric assembly. Moreover, since broken parts exhibit varied ge- ometries (e.g., the same bowl falling from different heights breaking into different groups of fragments), is challeng-
2

---

BiAssemble: Learning Collaborative Affordance for Bimanual Geometric Assembly
ing fairly assess policy performance in real-world settings. To address this, we further introduce real-world bench- mark featuring globally available objects with reproducible broken parts, along with their corresponding 3D meshes, which can be integrated into simulation environment. This benchmark enables consistent and fair evaluation of robotic geometric assembly policies. Extensive experiments on di- verse categories demonstrate the superiority of our method both quantitatively and qualitatively.
space. Current research primarily focuses on planning and collaboration. ACT (Fu et al. 2024 Zhao et al. 2023) in- troduces transformer- encoder-decoder architecture that leverages semantic knowledge from image inputs to pre- dict bimanual actions. PerAct2 (Grotz et al. 2024) learns features at both voxel and language levels, utilizing shared and private transformer blocks to coordinate two robotic arms based on semantic instructions. However, in tasks rich in geometric complexity, where objects have limited se- mantic information but intricate geometric structures, these foccch on semantic understanding -may en- counter generalization limits, DualAfford (Zhao et al. 2022) learns point-level collaborative visual affordance, while only for short-term tasks like pushing or rotating. To address this, we leverage the geometric generalization capability of point-level affordance, and enhance with the awareness of subsequent long- mong-horizon assembly actions.
2. Related Work
2.1. 3D Shape Assembly
Shape assembly is a well-established problem in visual ma- nipulation, with many studies focusing on constructing a complete shape from given parts, typically involving the pose prediction of each part for accurate placement (Zhan
et al.
2020 Wang al. 2022a). Further work (Heo al.
2.3. Visual Affordance for Robotic Manipulation
Tian et al. 2022 Jones et al. 2021 Willis et al. 2022 Tic et al., 2025) studied assembly with robotic execution, requiring robots to carry out each step, with benchmarks spanning various applications from home furniture assem- bly (Lee et al., (202T) to factory-based nut-and-bolt inter- actions (Narang et al. 2022). We categorize these tasks into two types: furniture assembly and geometric assembly. This paper focuses on geometric assembly, where pieces are irregular and lack semantic definitions, such as in the case of a broken bowl, making categorization difficult. This contrasts with furniture assembly that involves parts like nuts, bolts, and screws, each with specific functions and clear categorization.
2023
Among various vision- -based approaches for robotic manipu- lation (ooyal et al. 2023 Brohan et al. [023 Ze et 2024 Ju et al. 2024). for objects with rich geometric information and tasks requiring geometric generalization, point-level affordance, which reflects the functionality of each point for downstream manipulation (Mo et al., 2021] [i et al., 2024a), is broadly leveraged and can easily generalize to novel shapes with similar local geometries. series of re- search have leverage this representation to a broad range of robotic manipulation tasks, such as deformable object manipulation (Wu et al. 2023b Lu et al.) 2024a Wu et al. 2024). object manipulation in complex environments (Ding et al. 2024 ei al. 2024b Wu et al. [2023a). object manip- ulation with efficient exploration (Ning et al.] 2024 Wang
Previous work on geometric assembly (Sellán et al., 2022 Chen et al.) 2022 Lu et al. 2024c), includes Wu et al. (2023c), which learns SE(3) equivvaria part representa- tions by capturing part correlations for assembly, and Lee ct al. 2024). which introduces a low-complexity, high-order feature transform layer that refines feature pair matching. However, these methods primarily focus on synthesizing parts into cohesive object based pose considerations without incorporating robotic execution, which is impracti- cal in world scenarios where collisions may occur if the assembly process ignores actions. To tackle this challenge, we introduce a robotic bimanual geometric assembly frame- work that leverages two robots to collaboratively assemble pieces, enhancing stability real-world execution.
et al. 20226), and short-term bimanual manipulation (Zhao
et al. 2022p. Leveraging the strengths of affordance repre- sentation, we design sophisticated approach that incorpo- rates this representation into bimanual geometric assembly task requiring long-horizon line-grained actions, enhancing generalization and enabling more effective collaboration in addressing long-horizon geometric assembly challenges.
3. Problem Formulation
The task is to use two grippers to assemble a pair of 3D fractured parts initialized in random poses on the table. A camera situated in front of the table captures a partially scanned point cloud O. Given O, current state-of-the-ar methods (Scarpellini et al., 2024 Lu et al., 2024c] Chen et al. 2022) can imagine the assembled part poses and the assembled object S in any pose. Thus we assume taking imaginary assembled shape S as the input.
2.2. Bimanual Manipulation
Bimanual manipulation (Chen et al.
2023
Grannen et al.
2023 Mu et al.
2021] Chitnis et al. 2020
Lee et al. 2015
Xie et al. 2020 Ren et al. 2024b Liu al. 2024
2022
Li al. 2023
Mu et al. 2024) offers several advantages,
For the policy т,, as illustrated in Figure I we can simplify this long-term process into 3 key steps:
particularly in tasks requiring stable control or wide action
3

---

BiAssemble: Learning Collaborative Affordance for Bimanual Geometric Assembly
Input
Disussembly Direction
Emender
Shupe Feuture fs
Captions
Disassembly Predictor
Matrix Multiple
v E
Imaginary Assembled Shape $
(in pose)
Concatenate
Disassembly Feature fr
Imagine
Transformation Predictor
SE(3) Transformation RR 4x4
PN++
Global Feature fo
DN++
Observed Point Cloud O
Per-point Feuture (pp
Global Feature fs'
Transformed Shape 5
Bi-Affordance Predictor
MLP
Dir Feature f
Transformed Disassembly
Assembly
Alignment
Pick-up
Direction
Affordance Network A
Actor Network Ur
Affordance Map
Actor Network
Affordance Network Al
Pick-up Action 2
Pick-up Action
Affordance Map
Figure 2. Framework Overview. With the point cloud observation and Imaginary Assembled Shape, the model predicts the disassembly direction which the disassembled part poses can be easily reached by manipulating the raw parts under the guidance Bi-Affordance.
Pick-up: the two grippers pick up the fractured parts with actions (g) pick pick
tor to learn the feasible disassembly directions in which the opposite assembly direction will result in no collisions, based on the fracture geometry of the imaginary assembled shape in any pose (4.2). Next, we design the Transformation Predictor, transform disaasembled parts to poses where the controller can successfully manipulate the initial parts to these aligmment poses (.3).. Based the predicted part alignment poses, we propose the BiAffordance Predictor, which not only predicts where to grasp the fractured parts, but also considers the subsequent collaborative alignment and assembly steps (4.4). Finally, we explain training strat- egy and loss functions (4.6).
.922 . );
Alignment: grippers carry parts to alignment poses with actions (g) aaignn 92 align), positioning part seams to face each other and ensuring precise alignment for perfect assembly:
grippers move forward complete the assembly with actions (gpam .0.m).
Assembly:
Here, 1 and 2 denote the left and the right grippers, respec- tively. Each gripper action g is formulated as an SF(() matrix, representing the gripper pose in 3D space.
4.2. Disassembly Prediction Based on Shape Geometry
4. Method
The set of feasible disassembly directions (in which the dis- assembly and opposite assembly processes will not result in collisions) is an inherent attribute of pair fractured parts, determined by fracture geometries. Therefore, predict the disassembly directions, from the object-centric perspective, on the imaginary assembled shape S in any
4.1. Overview
Our BiAssembly framework is designed to predict collabora- tive affordance and gripper actions for bimanual geometric shape assembly. First, to propose the assembly direction on two aligned parts, we develop the Disassembly Predic-

---

BiAssemble: Learning Collaborative Affordance for Bimanual Geometric Assembly
pose. Additionally, we observe that when fractured parts rotate, the feasible disassembly directions will rotate corre- spondingly, maintaining SO(3) equivariance relative to part poses. This SO(3) equivariance property is advantageous for disentangling shape geometry from shape poses, as demon- strated in previous works (Wu et al. 2023c Scarpellini et al. 2024). Therefore, we adopt VN-DGCNN (Wu et al. 2023c Deng et al. 202I) to encode the imaginary assembled shape parts S and acquire the -qquvvaaiiar shape feature fs-
presented in Fig. (bottom), during inference, the BiAffor- dance Predictor conditionally predicts two gripper actions. The first Affordance Network generates the affordance map for the first gripper, highlighting the actionable regions for the bimanual assembly task, and we select a contact point pi with high actionable value. Then, the Actor Network pre- dicts the gripper orientation 71 for interaction at pl. Based on the first action 91 = (pi, r1), we can then predict the second gripper action 92 (pz. r2) using the second Affor- dance Network and Actor Network.
Inputting the equivariant representation fe. we use the Disassembly Predictor to predict the distribution of dis- assembly directions. Concretely, the Disassembly Pre- dictor is implemented as a conditional variational autoen- coder (cVAF) (Sohn et al., 2015), where the cVAE encoder maps the input disassembly direction v into Gauissian noise E 3R and the eVAE decoder reconstructs the disassem- bly direction from with ʃa as the condition.
Different from DualAfford that only predicts affordance for short-term tasks, we use whether the manipulation points can satisfy the following alignment pose (by the robotic controller) and the assembly step as the training signal.
To encode input information, one PointNet++ encodes the initial point cloud O obtains per-point features {.fp). An- other PointNet++ encodes the transformed shape S" and derives the global feature fsr Additionally, an MLP en- codes the transformed disassembly direction o' into for-
4.3. Transformation Prediction For Alignment Pose
Given the object-centric disassembly direction resulting in no collisions in the last-step assembly, we want to predict the alignment poses, where the robot can manipulate two parts from the initial poses to the alignment poses with- out collisions, and then robot can execute the assembly step. This problem can be formulated as predicting an SE(3) trans- formation M R2x44 that is applied the combination of the imaginary assembled shape S and the disassembly di- rection To capture this, we adopt PointNet++ (Qi et al. 2017a|b b) to encode the initial point cloud observation O into the global feature fo. We also employ an MLP to encode disassembly direction v into feature fo- The transformation predictor, which is implemented as a e cVEE, takes in con- catenation of (fo. fo) to predict the SE(3) transformation M. The yellow and blue arrow in Fig. 2 illustrate the data flow in this process: by applying the transformation M to the imaginary assembly S and disassembly direction v, we obtain the transformed S' and v'. Therefore, we can get the target poses to which the objects should be moved during the alignment and assembly phases.
For Affordance and Actor Networks' designs, the first Af- fordance Network is implemented as an MLP that receives the concatenated features (Jps fs for) and predicts an af- fordance score in the range of [0, 1] for each point p. By aggregating the affordance scores of all points, we obtain the first affordance map, and from which we select pi. The first Actor Network is implemented as a cVAE that takes the concatenated features (frt fs- for) as condition, and outputs the gripper orientation r1. The design of the second A Affordance Network and Actor Network follows a similar structure, with the difference that they additionally incorpo- rate the first gripper action's feature (fp; Sri) along with (fpr fs', fv'. More details about the BiAffordance Predic- tor can be found in Appendix
4.5. Alignment and Assembly Actions
After successfully grasping part, we now predict the grip- per alignment poses g; alsgn and assembly poses g with E [1,2] denotes the gripper id. We assume the relative pose between the gripper and the object remains stable. For example, in the first pickup step and the third assembly step, the relative gripper-object pose remains consistent, as expressed in Equation] I
4.4. BiAffordance Predictor
We build the BiAffordance Predictor to propose actions in the Pick-up step, indicating where to grasp the two frac- tured parts that can facilitate the whole long-horizon robotic assembly task. The BiAffordance Predictor should both identify easy-to-grasy grasp regions fractured parts and con- sider the subsequent Alignment and Assembly steps. This means (1) avoiding grasping regions in the seam and (2) pre- venting each gripper from adopting poses that could collide with the other part or gripper during subsequent steps.
picc spick
9.4 E SE(3).
(I)
Here, and denote gripper and object poses, respectively.
Next, as we have the imaginary part shapes P with pose g.mit, we can utilize a pretrained pose estimation model (Wen et al. 2024) to predict the relative pose of gicc with respect to g. Besides, by applying the predicted transformation M to P. we obtain the target assembled part pick P' and its pose as g.am M intt The gripper pose g.
Following DualAfford (Zhao et al., (022). we disentangle the bimanual task into two conditional submodules. As
5

---

BiAssemble: Learning Collaborative Affordance for Bimanual Geometric Assembly
B. Real- World Benchmark A. Sean Pipeline
52
AB
8B
3
Repot
angles phane
results
Whole Slage
(b)
Shupe
(b)
Shape
(b) Broken
Figure 3. Part illustrates the pipeline for scanning and reconstructing real objects. Part presents examples of fractured parts from
various categories, showcasing diverse geometries.
can be acquired from the robot control interface. There- fore, the gripper's final pose for assembling the parts can be calculated using Equation
(1l2,,2, as D(x, f), and denote the con-
catenated features (Sa, Jw) as sau- The loss function is: Cr LT, Egcol R*) D(T", fso) + D(R", fou)-
g. gpick orick (gf M- 9.9 A SE(3). (2)
For the losses used Bi-Affordance Predictor, we pro- vide detailed explanation in Appendix
It is important to note that, as indicated in the above simpli- fied equation, we do not need to define a canonical pose or try to obtain the values g:mitt we only require the relative pink pose of of to git...
5. Benchmark
5.1. Simulation Benchmark
A similar relationship can be established between the first and the second intermediate steps, with the difference being that q;
Constructing a large-scale dataset with real objects is both time-consuming and costly. To address this challenge, we utilize the Breaking Bad Dataset (Sellán et al. 2022), which models the natural destruction of objects into fragments. This dataset features multiple categories, diverse objects, and varying fracture patterns. For physics simulation, employ the SAPIEN (Xiang al. 2020) platform along with two two-finger Franka Panda grippers as robot actuators.
align M
L. iiit +
4.6. Training and Losses
Disassembly Direction Loss. The Disassembly Predictor implemented as eVAE. We apply Cosine Similarity Loss measure the error between the reconstructed disassembly direction and ground-truth v", and KL Divergence to measure the difference between two distributions:
We randomly select a pair of 3D fractured parts from a ran- domly chosen shape within a random category. The initial part poses are also randomized. Given the considerable di- versity in fractured parts, collecting successful manipulation data for assembly can be quite challenging. To enhance data collection efficiency, we implement several heuristic strategies, with details in Appendix B
LD = ( +
1)). (3)
Transformation Loss. The predicted SE(3) transforma- tion matrix M consists of translation T' and rotation R. 3D-vector using
Our model predicts the translation L1 Loss. The rotation, represented as a SO(3) matrix, can be expressed 6D vector by using two 3D vectors that correspond to the directions of the two orthogonal axes. Consequently, ours model predicts the rotation as a 6D- vector and employs the geodesic loss. In summary, let T and R. denote the ground-trute and for simplicity, denote
5.2. Real-World Benchmark
A real-world benchmark is crucial not only for the evalua- tion of various methods but also for providing a standardized platform that enables researchers to reproduce and share so- lutions. Illustrated Figure 3 we build the real-world benchmark by scanning with smart phone camera. First,
6

---

BiAssemble: Learning Collaborative Affordance for Bimanual Geometric Assembly
Table 1. Quantitative results for novel instances within training categories and for unseen categories.
Novel Instances in Training Categories
Novel Categories
A
7
Method
AVG
AVG
ACT
2%
0%
0%
0%
1%
0%
0%
0%
0%
0%
0.30%
0%
1%
0%
0%
1%
0.4%
Heuristic
5%
8%
0%
3%
2%
4%
5%
10%
2%
4.20%
1%
5%
2%
0%
14%
4.4%
SE(3)-Equiv
0%
0%
4%
0%
1%
5%
0%
7%
2%
11%
3.00%
4%
0%
2%
0%
2%
1.6%
Dual fford
21%
17%
0%
2%
2%
4%
14%
8%
10%
6%
8.40%
5%
10%
4%
1 1%
16%
7.2%
w/o SE(3)
59%
29%
13%
14%
%
8%
15%
19%
24%
20%
21.20%
13%
24%
4%
9%
22%
14.4%
Ours
60%
38%
13%
13%
%
9%
26% %
18%
%
25%
24.10%
14%
31% %
10%
7%
25%
17.4%
w/ GT Target|
71%
28%
4%
9%
9%
13%
27%
19%
25%
19%
22.40%
14%
33%
12%
9%
27%
19%
we put the object on an automatic turntable with 6 aruco markers around for precise camera localization, and capture a RGB video from a top-down view to a level view, low- ering the height by one level for each 360-degree rotation. After capturing 4-5 levels, we uniformly sample around 300 frames, and feed them COLMAP (Schönberger & Frahm 2016 Schönberger et al. 2016) for estimating camera poses. Then, we use Grounded SAM 2 (Ren et al., 2024a; Ravi et al. 2024) to generate object masks and Depth Anything V2 Yang et al.
6.2. Evaluation Metric, Baselines and Ablation
Evaluation Metric. Our metric evaluates whether relative distance (measured in unit-length) and rotation angle (mea- sured in degrees) of two parts are within the threshold range at the end of assembly. These thresholds ensure the success of assembly can be measured consistently and meaningfully. To evaluate each method, we prepare 100 samples in each category. For each sample, all methods are presented with the same initial observation for fair comparison.
2024) to predict monocular depths, and use SDFStudio Yuu et al.
2022 Wang et al. 2021) with depth
Baselines and Ablations. and two ablated versions:
We compare with four baselines
ranking loss (Wang et al. 2023) to reconstruct object mesh. To annotate the ground-truth of scanned object assembly, we import the object slices to Blender (Community 2018
(1) ACT (Zhao al. 2023). transformer network with ac- tion chunking that imitates successful action sequences in closed-loop manner. We enhance this method by providing depth information, object pose, and an ad- ditional target image as inputs. Besides, this baseline is trained and tested on individual categories, whereas other learning- based methods are trained on all training categories and evaluated on both novel instances and unseen categories.
and edit the object transformations.
Our real-world benchmark encompasses a diverse range of object categories, including wine glass, plate, beer bottle, bowl, mug, and teapot. These objects have been primarily selected from well-known international brands, ensuring both durability and accessibility. To promote object diver- sity, our shapes vary in size, geometry, transparency, and texture, with different seam geometries.
(2) Heuristic, where we hand-engineer a set of heuristic strategies to improve manipulation success rate. These strategies are similar to the data collection heuristies (Appendix B).
6. Experiments
6.1. Simulation and Settings
(3) SE(3)-Equiv (Wu et al. 2023c that learns SE(3)- equivariant part representations to estimate target as- sembled part poses in the computer vision domain. Though originally designed for assembled part pose predictions without considering robotic execution, we adapted it as a baseline by generating the pick-up ac- tions of robots via heuristic strategies. Additionally, given the predicted pose g,thm for part i, we compute the corresponding gripper pose geam for assembly ac- tion, following Equation
The simulation environment is built on the SAPIEN (Xiang jet al. 2020) platform, utilizing the Franka Panda grippers as the robot actuator. We employ the EverydayColorPiecee dataset from the Breaking Bad Dataset (Sellán et al.) 2022). covering 15 categories, 445 shapes and 1 1,820 fragment pairs, with 10 categories for training and the remaining 5 for testing. Training categories are further divided into train- ing shapes and novel instances, allowing the evaluation on generalization capabilities at both the object and category levels. More details can be found in Appendix A For each method, we provide 7,000 positive and 7,000 negative sam- ples. Negative samples encompass failures occurring during grasping, alignment, and assembly steps.
(4) Afforrford (Zhao et al. 2022) that learns collaborative visual affordance for bimanual manipulation. While it focuses on short-term manipulation, we adapt it to determine where to grasp the two fractured parts, using heuristic methods for alignment and assembly steps.
7

---

BiAssemble: Learning Collaborative Affordance for Bimanual Geometric Assembly
Initial Observation
Pick-up
Aligament
Assembly
Initial Observation
Pick-up
Alignment
Assembly
Unseen Categorios Train Categories
Figure 4. Simulation Experiment. We present qualitative results of the predicted affordance maps and rohot actions from our method.
(5) w/o SE(3), an ablation that replaces SO(3) equiivarun VN-DGCNN encoder with PointNet++.
For DualAfford, it performs better than the heuristic policy, demonstrating superior ability learn geometric-awane pick-up poses compared heuristic sampling. However, only with designs focused on short-term manipulation, it still lacks awareness of the subsequent alignment and as- sembly steps.
(6) w/ GT target, where we provide additional ground- truth disassembly direction and transformation M. Ground truth sampled using heuristic method that ensures least one feasible assembly.
In comparison with ablation models, our method also shows consistent improvements. Compared to the ablation w/o SE(3), we observe that utilizing the -Eu3)-qquivarian representation enhances the performance across most cate- gories. Compared to the ablation w/ GT Target, our method performs better on novel instances but worse on novel cat- egories. This suggests that, for the training categories, our method learns to predict a more accurate distribution of disassembly and transformation, surpassing those sampled from the heuristic strategy. However, on novel unseen cate- gories, while method still demonstrates generalization capability, the ablated version with the ground-truth target remains more effective.
6.3. Quantitative Results and Analysis
Table shows the success rate comparisons across different methods on both the novel instance dataset and the unseen category dataset. Our method outperforms the baselines and ablation models in most cases, demonstrating its effective- ness and geometric generalization capabilities.
For ACT, though we provide additional input such as depth, object poses, and the goal image, it achieves lower scores on our task. Although ACT successfully picks up parts in approximately 40% of trials, often fails during the align- ment phase, with a misalignment of over 100 between the parts in many cases. Furthermore, ACT struggles to avoid grasping the fractured seam regions, leading to collisions during the assembly process. This is because the observa- tion and action spaces in robotic geometric assembly are exceptionally large, making it challenging to directly learn the appropriate fine-grained actions.
Appendix E presents additional experiments and analyses, including multi falti-fragment assembly, the use "imperfect" imaginary assembly shape inputs, and further ablations.
6.4. Qualitative Results and Analysis
In Figure 4 we present collaborative affordance maps and robot manipulations predicted by our methods across mul- tiple categories, including novel instances training cate- gories and unseen categories. Predicted affordance demon- strates an awareness of part geometry, highlighting gras- pable regions while avoiding areas near the table that could result in collisions between the gripper and surface. Addi- tionally, the affordance accounts for subsequent alignment and assembly steps, avoiding seam areas that may cause col- lisions during the approach phase. Based on predicted affor- dance map, model predicts appropriate gripper actions for assembling parts. Moreover, the results demonstrate the ability to generalize to unseen categories and shapes.
For heuristic, it achieves higher scores because we provide substantial ground-truth information in simulation. How- ever, due to the significant diversity in both inter- and intra- category shape geometries, it is unrealistic to expect hand- engineered rules to generalize effectively across all shapes.
For SE(3)-Eqiiv, the lower accuracy primarily due to its disregard for the robotic execution process, which common limitation of prior visual assembly estimation meth- ods. Specifically, these methods do not determine suitable grasping locations on the fragments, not only for successful pick-up but also for avoiding the seam region critical to sub- sequent assembly. They also lack the capability to align the fragments properly at the seam, significantly increasing the likelihood of collisions when the parts are brought together.
8

---

BiAssemble: Learning Collaborative Affordance for Bimanual Geometric Assembly
Initial Observation
Pick-up
Aligament
Assembly
Manipulation
Bowl
0:75
0.5
0.25
TeaCup
WineGlass
BeerBottle
Affordances
Figure 5.
Real-World Experiment. We present the results of model tested real-world seans.
6.5. Real-World Experiments
Impact Statement
We set up Franka Panda with fractures positioned be- tween them. An Azure Kinect camera is mounted in front of the robots, capturing partial 3D point cloud data inputs for our models. The robots are controlled via the Robot Operating System (ROS) (Quigley et al.) 2009), with con- trol and communication managed through the frankapy li- brary (Zhang et al. 2020). Communication with the Kinect Azure is facilitated by the pyk4a library 4pyk4a 2019).
This work contributes to the development shape assem- bly skills in future robots, specifically focusing on visual collaborative affordance predictions. These affordances can be visualized and analyzed prior to deployment, which may reduce potential risks and dangers. We do not anticipate any significant ethical concerns or societal consequences arising from our research.
Figure and the bottom of Figure present promising results by directly testing method in real-world scenarios. Detailed manipulation processes are shown in Appendix [ and videos are available in the supplementary materials. We observe that our model not only learns which regions of the fractured parts to grasp but also avoids manipulating areas near the fracture regions or close the table surface, reducing the likelihood of collisions during manipulation. The results from the real-war real-world experiments demonstrate our model's capacity for generalization to real-world scenarios.
Acknowledgements
This paper supported by National Natural Science Foun- dation of China (No. 62376006) and National Youth Talent Support Program (8200800081).
References
Ankile, L., Simeonov, A., Shenfeld, I., and Agrawal, P. Juicer: Data-efficient imitation learning for robotic as- sembly, arXiv preprint arXiv:2404.03722 2024.
7. Conclusion
Brohan, A., Brown, N., Carbajal, J., Chebotar, Y., Chen, X., Choromanski, K., Ding, T., Driess, D., Dubey, A., Finn, C., al. Rt-2: Vision-language-action models transfer web knowledge to robotic control. arXiv preprint arXiv:2307.1588 15818, 2023.
We leverage the geometric generalization capability of point- level affordance develop method that enables both generalization and collaboration in long-horizon geometric assembly tasks. To evaluate performance across diverse geometries, we introduced real-world benchmark that fea- tures significant geometric variety and global reproducibility. Extensive experiments have shown that our approach out- performs previous methods, demonstrating its effectiveness handling complex and Long-horizon assembly tasks.
Chen, Y., Geng, Y., Zhong, F., Ji, J., Jiang, J., Lu, Z., Dong, H., and Yang. Y. Bi-dexhands: Towards human-level bimanual dexterous manipulation. IEEE Transactions on Pattern Analysis and Machine Intelligence, 2023.
Chen, Y.C.., Li, H., Turpin, D., Jacobson, A., and Garg,

---

BiAssemble: Learning Collaborative Affordance for Bimanual Geometric Assembly
A. Neural shape mating: Self-supervised object assem- bly with adversarial shape priors. In Proceedings of the IEEE/CVF Conference Computer Vision and Pattern Recognition, pp. 12724-12732 2022.
Ju, Y., Hu, K., Zhang, G., Zhang, G., Jiang, M., and Xu, H. Robo-abe: Affordance generalization beyond cate- gories via semantic correspondence for robot manipula- tion. arXiv preprint arXivv2401.07497 2024.
Chitnis, R., Tulsiani, S., Gupta, S., and Gupta, A. Efficient bimanual manipulation using learned task schemas. In 2020 IEEE International Conference Robotics and Automation (ICRA), pp. 1149-1155, IEEE, 2020.
Lee, A. X., Lu, H., Gupta, A., Levine, S., and Abbeel, P. Learning force-based manipulation of deformable objects from multiple demonstrations. In 2015 IEEE Interna- tional Conference on Robotics and Automation (ICRA), pp. 177-184, IEEE, 2015.
Clarke, J. A., Tambussi, C. P., Noriega, L., Erickson, G. M., and Ketcham, R. A. Definitive fossil evidence for the extant avian radiation in the cretaceous. Nature, 433 (7023):305-34) 308, 2005.
Lee, N., Min, J., Lee, J., Kim, S., Lee, K., Park, J., and Cho, M. 3d geometric shape assembly via efficient point cloud matching. arXiv preprint (xXiv:2407.10544 2024.
Community, B. O. Blender a 3D modelling and ren- dering package. Blender Foundation, Stichting Blender Foundation, Amsterdam, 2018. URL http:///wwy
Lee, Y., Hu, E. S., and Lim, J. Ikea furniture assembly en- vironment for long- norizhoizon complex manipulation tasks. In 2021 ieee international conference on robotics and automation (icra), pp. 6343-6349, IEEE, 2021.
blender.cn .org
Deng, C.,, Litany, O., Duan, Y., Poulenard, A., Tagliasacchi,
Li, X., Zhang, M., Geng, Y., Geng, H., Long, Y., Shen, Y., Zhang. R., Liu, J., and Dong. H. Manipllm: Embod- ied multimodal large language model for object-centric robotic manipulation. In Proceedings of the IEEE/CVF Conference on Computer Vision and Puttern Recognition, pp. 18061-18070 2024a.
A., and Guibas, L. J. Vector neurons: A general frame- work for so (3)-euiivriiant networks. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pp. 12200-12209, 2021.
Ding, K., Chen, B., Wu, R., Li, Y., Zhang, Z., Gao, H.-a., Li, S., Zhu, Y., Zhou, G., Dong, H., et al. Preafford: Uni- versal affordance-based pre-grasping for diverse objects and environments. IROS, 2024.
Li, Y., Pan, C.,, Xu, H., Wang, X., and Wu, Y. Efficient bi- manual handover and rearrangement via symmetry- aware actor-critic learning. In 2023 IEEE International Confer- ence on Robotics and Automation (ICRA), pp. 3867-3874. IEEE, 2023.
Fu. Z., Zhao, T. Z., and Finn, C. Mobile aloha: Learning bimanual mobile manipulation with low-cost whole-body teleoperation. arXiv preprint MxXiv:2401.02111 2024.
Li, Y., Wu, R., Lu, H., Ning, C., Shen, Y., Zhan, G., and Dong, H. Broadcasting support relations recursively from local dynamics for object retrieval in elutters. RSS, 2024b.
Goyal, A., Xu, J., Guo, Y., Blukis, V., Chao, Y.-W., and Fox, D. Rvt: Robotic view transformer for 3d object manipulation. In Conference on Robot Learning, pp. 694 710. PMLR, 2023.
Liu, B., Luo, X., Huang, R., Wan, C., Zhang, B., Hu, W., and Yue, Z. Virtual plate pre-bending for the long bone fracture based on axis pre-aligmment. Computerized med- ical imaging and graphics, 38(4):233-244, 2014.
Grannen, J., Wu, Y., Vu, B., and Sadigh, D. Stabilize to act: Learning to coordinate for bimanual manipulation. In Conference on Robot Learning, pp. 563-576. PMLR, 2023.
Liu, J., Chen, Y., Dong, Z., Wang. S., Calinon, S., Li, M., and Chen, F. Robot cooking with stir-fry: Bimanual non- prehennee manipulation of semi-fluid objects. IEEE Robotics and Automation Letters, ((2):51595516) 2022.
Grotz, M., Shridhar, M., Asfour, T., and Fox, D. Peract2: perceiver actor framework for bimanual manipulation tasks. arXiv preprint arXiv:2407. 00278, 2024.
Liu, Y., Yang, H., Si, X., Liu, I.., Li, 7.,, Zhang, Y., Liu, Y., and Yi, L. Taco: Benchmarking generalizable bimanual col-actionnobbees understanding. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 21740-21751. 2024.
Heo, M., Lee, Y., Lee, D., and Lim, J. Furniturebench: Re- producible real-world benchmark for long-horizon com- plex manipulation. arXiv preprint arXiv:2305. 12821, 2023.
Jones, B., Hildreth, D., Chen, D., Baran, I., Kim, V. G., and Schulz, A. Automate: A dataset and learning approach for automatic mating of cad assemblies. ACM Transactions on Graphics (TOG), 1018):1-18, 2021.
Lu, Hi, Y., Wu, R., Ning, C., Shen, Y., and Dong, H. Unigarment: A unified simulation and benchmark for garment manipulation. In ICRA Workshop on Deformable Object Manipulation, 2024a.
10

---

BiAssemble: Learning Collaborative Affordance for Bimanual Geometric Assembly
Lu, J., Liang, Y., Han, H., Hua, J., Jiang, J., Li, X., and Huang, Q. A survey on computational solutions for recon- structing complete objects by reassembling their fractured parts. arXiv preprint arXiv:2410. 14Xi::4410014770, 14770, 2024b.
open source software, volume 3, pp. 5. Kobe, Japan, 2009.
Ravi, N., Gabeur, V., Hu, Y.-T., Hu, R., Ryali, C., Ma, T., Khedr, H., Rädle, R., Rolland, C., Gustafson, L., Mintun, E., Pan, J., Alwala, K. V., Carion, N., Wu, C.-Y,, Girshick, R., Dollár, P., and Feichtenhofer, C.
Lu, J., Sun, Y., and Huang, Q. Jigsaw: Learning
semble multiple fractured objects. Advances in Neural Information Processing Systems, 36, 2024c.
Sam 2: Segment anything in images and videos. arXiv preprint arXiv:2408 00714 2024. URL https:// wxxiv.org/ab//2400 00714
Mo, K., Guibas, L. J., Mukadam, M., Gupta, A., and Tul- siani, Where2act: From pixels to actions for articulated 3d objects. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pp. 6813-6823, 2021.
Ren, T., Liu, S., Zeng, A., Lin, J., Li, K., Cao, H., Chen, J., Huang, X., Chen, Y., Yan, F., Zeng, Z., Zhang, H., Li, F,, Yang, J., Li, H., Jiang, Q.. and Zhang. L. Grounded sam: Assembling open-worid models for diverse visual tasks, 2024a
Mu, T., Ling, Z., Xiang, F., Yang, D., Li, X., Tao, S., Huang, Z., Jia, Z., and Su, H. Maniskill: Learning- from-demmnsttratoons benchmark for generalizable ma- nipulation skills. CoRR, abs/2107. 14483, 2021b. URL https://axxiv org/abs/2107, 14483, 2021.
Ren, Y., Zhou, Z., Xu, Z., Yang, Y., Zhai, G., Leibold, M., Ni, F., Zhang. Z., Buss, M., and Zheng, Y. Enabling versatility and dexterity of the dual-arm manipulators: general framework toward universal cooperative manipu- lation. IEEE Transactions on Robotics, 2024b.
Mu, Y., Chen, T., Peng. S., Chen, Z., Gao, Z., Zou, Y., Lin, L.,, Xie, Z., and Luo, P. Robotwin: Dual-arm robot benchmark with generative digital twins (early version). arXiv preprint ar wxx::2409002920, 2024.
Scarpellini, G., Fiorini, S., Giuliari, F., Morerio, P., and Del Bue. A. Diffassemble: A unified graph-diffusion model for and 3d reassembly. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), June 2024.
Narang, Y., Storey, K., Akinola, I., Macklin, M., Reist, P., Wawrzyniak, L., Guo, Y., Moravanszky, A., State, G., Lu, M., et al. Factory: Fast contact for robotic assembly. 2022.
arXiv preprint arXiv:2205.088:
Schönberger, J. L. and Frahm, J.-M. Structure-from motion revisited. In Conference on Computer Vision and Pattern Recognition (CVPR), 2016.
Ning, C., Wu, R., Lu, H., Mo, K., and Dong, H. Where2explore: Few-shot affordance learning for un- seen novel categories of articulated objects. Advances in Neural Information Processing Systems, 36, 2024.
Schönberger, J. L., Zheng, E., Pollefeys, M., and Frahm, J.-M. Pixelwise view selection for unstructured multi- view stereo. In European Conference on Computer Vision (ECCV), 2016.
Papaioannou, G. and Karabassi, E.-A. On the automatic assemblage of arbitrary broken solid artefacts. Image and Vision Computing, 21(5):401-415 2003.
Sellán, S., Chen, Y.-C., Wu, Z., Garg, A., and Jacobson, A. Breaking bad: A dataset for geometric fracture and reassembly. In Thirty-sixth Conference on Neural Infor- mation Processing Systems Datasets and Benchmarks Track, 2022.
pyk4a. 4yk4a, 2019. URL https::/ggitie com/
etiennedub/pen 4
Qi, C. R., Su, H., Mo, K., and Guibas, L. J. Pointnet: Deep learning on point sets for 3d classification and segmen- tation. Proc. Computer Vision and Pattern Recognition (CVPR), IEEE, 2017a
Sellán, S., Luong, J., Mattos Da Silva, I.., Ramakrishnan, A., Yang, Y., and Jacobson, A. Breaking good: Fracture modes for realtime destruction. ACM Transactions on Graphics, 42(1):1-12, 2023.
Qi. R., Yi, L., Su, H., and Guibas, L. J. Pointnet++: Deep hierarchical feature learning on point sets in a met- ric space. Advances in neural information processing systems, 30, 2017b.
Sera, I., Yamanobe, N., Ramirez-Alpizar, I. G., Wang, Z., Wan, W., and Harada, K. Assembly planning by recog- nizing graphical instruction manual. In 2021 IEFF/RSJ International Conference Intelligent Robots and Sys- tems (IROS), pp. 3138-3145. IEEE, 2021.
Qi, Yu, Y., Wei, T., Chu, C., Wong, I.. I.,, and Xu, H. Two by two: Learning multi-task pairwise objects assembly for generalizable robot manipulation. CVPR 2025, 2025.
Sohn, K., Lee, H., and Yan, X. Learning structured output representation using deep conditional generative models. Advances in neural information processing systems, 28: 3483-3491, 2015.
Quigley, M., Conley, K., Gerkey, B., Faust, J., Foote, T., Leibs, J., Wheeler, R., Ng, A. Y., et al. Ros: an open- source robot operating system. In ICRA workshop on

---

BiAssemble: Learning Collaborative Affordance for Bimanual Geometric Assembly
Suárez-Ruiz, F., Zhou, X., and Pham, Q-C. Can robots as- semble an ikea chair? Science Robotics, 3(17):eaat6385, 2018.
Wu, R., Zhao, Y., Mo, K., Guo, Z., Wang, Y., Wu, T., Fan, Q., Chen, X., Guibas, L.., and Dong, H. Vat-mart: Learn- ing visual action trajectory proposals for manipulating 3d articulated objects. ICLR, 2022.
Tian, Y., Xu, J., Li, Y., Luo, J., Sueda, S., Li, H., Willis, K. D., and Matusik, W. Assemble them all: Physics- based planning for generalizable assembly by disassem- bly. ACM Transactions on Graphics (TOG), 41(6):1-11, 2022.
Wu, R., Cheng. K., Zhao, Y., Ning. C., Zhan, G., and Dong, H. Learning environment-aware affordance for 3d artic- ulated object manipulation under occlusions. In Thirty- seventh Conference Neural Information Processing Systems, 2023a. URL //tpr//////eeeeieee... net/ forum?11==e22NNY0255
Tie, C., Sun, S., Zhu, J., Liu, Y., Guo, J., Hu, Y., Chen, H., Chen, J., Wu, R., and Shao, I.. Manual2skill: Learning to read manuals and acquire robotic skills for furniture assembly using vision-language models. In Proceedings of Robotics: Science and Systems (RSS), 2025.
Wu, R., Ning, C., and Dong, H. Learning foresightful dense visual affordance for deformable object manipulation. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pp- 10947-10956, 2023b.
Tsesmelis, T., Palmieri, L., Khoroshiltseva, M., Islam, A.. Elkin, G., Shahar, O. I., Scarpellini, G., Fiorini, S., Ohayon, Y., Alali, N., et al. Re-assembling the past: The repair dataset and benchmark for real world 2d and 3d puzzle solving. arXiv preprint arXiv:2410.24011 2024.
Wu, R., Tie, C., Du, Y., Zhao, Y., and Dong, H. Leveraging se (3) equivariance for learning 3d geometric shape assem- bly. In Proceedings of the IEEE/CVF International Con- ference on Computer Vision. pp. 1431 1-3220,,,, 2023c.
Wan, Y., Zhou, K., Dong, H., et al. Scanet: Correcting lego assembly errors with self-correct assembly network. arXiv preprint arXivv2403.10 18195, 2024.
Wu, R., Lu, H., Wang, Y., Wang, Y., and Dong, H. Uni- garmentmanip: A unified framework for category-level garment manipulation via dense visual correspondence. In Proceedings of the IEEE/CVF Conference on Com- puter Vision and Pattern Recognition, pp. 16340-16350, 2024.
Wang, G., Chen, Z., Loy, C. C., and Liu, Z. Sparsenerf: Distilling depth ranking for few-shot novel view synthe- sis. In IEEF/CVF International Conference on Computer Vision (ICCV), 2023.
Xian, Z., Lertkultanon, P., and Pham, Q-C. Closed-chain manipulation of large objects by multi-arm robotic sys- tems. IEEE Robotics and Automation Letters, 1(4):1832- 1839, 2017.
Wang, P., Liu, L., Liu, Y., Theobalt, C., Komura, T., and Wang, W. Neus: Learning neural implicit surfaces by volume rendering for multi-view reconstruction. NeurIPS, 2021.
Xiang. F., Qin, Y., Mo, K., Xia, Y., Zhu, H., Liu, F., Liu, M., Jiang, H., Yuan, Y., Wang, H., et al. Sapien: A simulated part-based interactive environment. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 11097-11100 1107, 2020.
Wang, R., Zhang, Y., Mao, J., Zhang, R., Cheng, C.-Y., and
Wu, J. Ikea-manual: Seeing shape assembly step by step. Advances in Neural Information Processing Systems, 35: 28428 28440, 2022a.
Wang, Y., Wu, R., Mo, K., Ke. J., Fan, Q. Guibas, L. J., and Dong, H. Adaafford: Learning to adapt manipulation affordance for 3d articulated objects via few-shot inter- actions. In European conference on computer vision, pp. 90-107. Springer, 2022b.
Xie, F. Chowdhury, A., De Paolis Kaluza, M., Zhao L.,, Wong. L., and Yu, R. Deep imitation learning for biman- ual robotic manipulation. Advances in neural information processing systems, 33:2327-2337, 2020.
Wen, B. Yang, W., Kautz, J., and Birchfield, S. Foun- dationpose: Unified 6d pose estimation and tracking of novel objects. In Proceedings of the IEEE/CVF Confer- ence on Computer Vision and Pattern Recognition, pp. 17868-17879, 2024.
Yang, I.., Kang, B., Huang, Z., Zhao, 7., Xu, X., Feng. J., and Zhao, H. Depth anything v2. 2406:2406609411 2024.
Yu, M., Shao, L., Chen, Z., Wu, T., Fan, Q. Mo, K., and Dong, H. Roboassembly: Learning generaliz- able furniture assembly policy in a novel multi-robot contact-rich simulation environment. arXiv preprint 12Xiv:2112.10144 2021.
Willis, K. D., Jayaraman, P. K., Chu, H., Tian, Y., Li, Y., Grandi, D., Sanghi, A., Tran, L.,, Lambourne, J. G., Solar- Lezama, A., et al. Joinable: Learning bottom-up assembly of parametric cad joints. In Proceedings of the JEEE/CVF conference on computer vision and pattern recognition, pp. 15860, 2022.
Yu, Z., Chen, A., Antic B., Peng, S., Bhattacharyya, A., Niemeyer, M., Tang, S., Sattler, T., and Geiger,
12

---

BiAssemble: Learning Collaborative Affordance for Bimanual Geometric Assembly
A. Sdfstudio: A unified framework for surface construction, 2022. [ttp::/tss////ttitubbbb.... hwwp://wwwssssiooosssssssue
Ze, Y., Zhang, G., Zhang, K., Hu, C., Wang, M., and Xu, H. 3d diffusion policy, arXiv preprini arX1v:2403.03954 2024.
Zhan, G., Fan, Q. Mo, K., Shao, L., Chen, B., Guibas, L. J., Dong, H., al. Generative 3d part assembly via dynamic graph learning. Advances Neural Information Processing Systems, 33:6315-6326 2020.
Zhang, K., Sharma, M., Liang, J., and Kroemer, O. A modular robotic arm control stack for research: Franka- interface and frankapy. arXiv preprint arXiv:2011.02398, 2020.
Zhao, T. Z., Kumar, V., Levine, S., and Finn, C. Learn- ing fine-grained bimanual manipulation with low-cost hardware. arXiv preprint arXiv:2304.1377 2023.
Zhao. Y., Wu, R., Chen, Z., Zhang, Y., Fan, Q., Mo, K., and Dong, H. Dualafford: Learning collaborative visual affordance for gripprr manipulation. arXiv preprint arXiv:2207.01971, 2022.
13

---

BiAssemble: Learning Collaborative Affordance for Bimanual Geometric Assembly
Appendix
A. Details about Data Statistics
BeerBottle
Bottle
Bowl
Mug
PillBottle
BreakingBad Dataset
Statue
Teapot
Toy Figure
Vase
Wineglass
Cup
DrinkBottle
PrinkingUtensii
Teacup
WineBottle
Figure 6. Visualization of simulation data. We present example shape from each object category used in paper.
In Figure 6 we present representative example for each object category from the dataset used in our experiments.
In this paragraph, we detail the data split for our experiments. We randomly select 10 out of the 15 categories for training, reserving the remaining 5 categories exclusively for testing. Within the 10 training categories, 60% of the shapes are randomly chosen for the training set, while the remaining 40% serve as a test set to assess the models' performance on novel instances within the training categories (shape-level). For the reserved categories, all shapes are included the test set evaluate the methods' generalization capabilities unseen categories (category-level). In summary. the training consists of 10 categories, totaling 237 shapes and 6,403 pairs of fragments. The shape-level test set includes 10 categories, comprising 131 shapes and 3,638 pairs of fragments. The category-level test set encompasses categories, containing 77 shapes and 1,779 pairs of fragments. Detailed statistics for each category can be found in Table
B. Details about Data Collection in Simulation
In this section, we provide detailed information about data collection in the simulation.
Due to the complexity of bimanual geometric assembly tasks, which stems from the vast observation and action spaces, it is nearly impossible to directly acquire positive data by randomly manipulating the fractured parts. To address this, we apply several heuristic strategies to improve the efficiency of data collection. Specifically, our strategies focus on the following three key steps the process:
1. Sampling the grasping poses for the two grippers
2. Sampling the alignment poses for the two grippers
3. Sampling the assembly poses for the two grippers
14

---

BiAssemble: Learning Collaborative Affordance for Bimanual Geometric Assembly
Table Shape and Fracture Counts Across Categories. Numbers before the slash represent the training set, and numbers after the slash represent the testing set. The top 10 categories are the training categories, and the bottom 5 categories are the unseen categories.
Category
Shape (Train/Test)
Fracture (Train/Test)
BeerBottle
6/3
/ 61
Bottle
51 22
1296 559
Bowl
16/
446 / 801
Mug
32/15 15
876 / 545
PillBottle
7/3
217 60
Statue
2/0
57 35
Теapot
7/3
315 / 104
ToyFigure
36/ 16
1118 / 556
Vase
74/32
1842 / 872
WineGlass
6/3
136 45
Cup
0/31
66
DrinkBottle
0/7
230
DrinkingUtensii
0/ 14
0/ 343
Teacup
0/7
167
WineBottle
0/ 18
0/ 376
Total
237 / 208
6403/ 5417
Each of these steps is described in detail in the following subsections.
B.1. Sampling Grasping Poses
Different from furniture assembly task, where grasp points are easier to define, the objects in geometric shape assembly tasks have more diverse geometries, making it challenging establish consistent grasping policy. As result, our heuristic strategy for grasping primarily focuses on the orientation of the grippers rather than specific grasp points.
At initialization, the two parts are randomly placed on the table. From the simulation, we obtain the ground -ruuh depth map and normalization map of the two parts. The normal directions often closely align with feasible grasping directions (i.e., the z-axis the gripper). Consequently, we randomly select grasp point on the part, and then choose a grasping direction within a cone that deviates less than 30 degrees from the normal direction at that point. To avoid potential collisions between the grippers and the table during grasping, we discard any directions that point towards the upper hemisphere of the world coordinate system.
In addition to the gripper's z-axis, the x-axis also significantly impacts grasping accuracy. Therefore, we uniformly sample list of n x-axis candidates that are orthogonal to the gripper's z-axis. By combining each candidate x- x-axis with the z-axis, we determine the gripper pose. We test each of these gripper poses sequentially. If a grasp pose successfully grasps the object, we proceed to the next stage; otherwise, we reset the scene and move on to the next x-axis candidate. all grasp pose candidates fail, we record this as negative data for the grasping step. In our implementation, we empirically set n resulting in each x-axis candidate being spaced 60 degrees apart.
B.2. Sampling Aligmment Poses
To sample the grippers' alignment poses, we begin by sampling feasible part poses during the aligmment step. Our heuristic strategy also follows a reverse disassembly process. Specifically, we load the ground-truth assembled object into the simulation at height of 0.5 meters above the tabletop, and allow the assembled object to take any pose rather than being restricted to canonical pose. Next, we randomly explore feasible disassembly directions for the two parts, ensuring that these directions are collision-free. The resulting poses of the parts, after moving in their respective disassembly directions, represent the parts' alignment poses. is important to note that we will discard alignment poses that are too distant from the parts' initial poses (for example, if the initial left part has an alignment pose to the right, while the initial right part has an alignment pose to the left).
15

---

BiAssemble: Learning Collaborative Affordance for Bimanual Geometric Assembly
Once we have determined the parts' alignment poses, we can calculate the grippers' alignment poses using the functions described in Section 4.5
B.3. Sampling Assembly Directions
In the previous step, we identified the feasible disassembly directions for the ground-truth assembled parts. Consequently, we can obtain the assembly directions by simply inverting these disassembly directions, allowing the two grippers to assemble the parts accordingly. However, it is important to note that this assembly process may lead to failures. This is because, although the parts can be successfully aligned in an idealized scenario without grippers, the presence of grippers increases the risk of collisions. For instance, if one gripper is positioned too close to the seam area of a fractured part, it may collide with another part or the other gripper during the assembly process.
C. More Details about the BiAffordance Framework
In this section, we provide more details about the BiAffordance Predictor. Following DualAfford (Zhao et al. 2022), we decompose the bimanual cooperation task into two separate yet closely interconnected submodules, Mi and M2, which conditionally predict the first and second gripper actions, respectively.
During inference, the first module M1 predicts the first gripper action 91 = (pi,r)) followed by the second module M2, which predicts the second action f/2 = (f5, r2) conditioned on gl, as described Section 4.4 of the main paper.
During training, M2 still takes the first gripper action ØI input, and then generates a complementary second action g2. However, since M₁ lacks knowledge of how Ø2 will be predicted, it faces challenges in predicting collaborative action gn To address this issue, we aim to make Mi aware of the types of actions that can be easily collaborated on. We assess the quality of MM 's actions by evaluating whether M2 can generate cooperative actions, which encourages M1 to predict actions with high collaborative quality. Following this approach, M2 guides the training of Mi. Thus, we first train M2 and then use the trained M2 to train M1. ensuring cooperative predictions.
During training, each submodule M; consists of three components: (1) an Affordance Network A., which predicts an affordance map indicate where interaction should occur; (2) an Actor Network U4 which predicts manipulation orientations determine how to interact at the selected point; and (3) Critic Network C, which assesses the likelihood of the action's 's success.
To explain the training process, we begin with the more straightforward second module, M2, which is also the first to be trained.
The Actor Network 242 in M2 is implemented as a conditional Variational Autoencoder (cVAE). As detailed in Section 4.4 takes concatenated input features fin (Spr f's, fw) and the ground-truth feature of the first action fn = (Upp from the collected data. We apply geodesic distance loss measure the error between the reconstructed gripper orientation r2 and the ground-truth orientation f2, along with KL divergence to quantify the difference between the two distributions:
Cuz = Cgoo P2) +
f. 1)).
(4)
The Critic Network C2 in M2 is implemented as a multilayer perceptron (MLP) and evaluates how well the predicted second gripper action 92 = (P5, r2) collaborates with the first action 9I. Using the collected data along with the corresponding ground-truth interaction results (where indicates a positive interaction and indicates a negative one), we train C2 with the standard binary cross-entropy loss:
Cc, = for fra)) + (1 ry )log(1 C2(fin for fog fra)).
(5)
The Affordance Network A₂ in M2 is implemented as a multilayer perceptron (MLP). The predicted affordance score represents the expected success rate for executing action proposals generated by the Actor Network, which can be directly evaluated by the Critic Network. To obtain the ground-truth affordance score åp, on Pt. we use the Actor Network Uz to
16

---

BiAssemble: Learning Collaborative Affordance for Bimanual Geometric Assembly
sample gripper orientations the point Pt and calculate the average action scores assigned by the Critic Network Cg. We apply 1.1 loss to measure the error between the predicted and ground-truth affordance scores specific point pi:
ap. n Σ C2 fp. for 24)); CAs = for ap.l.
(6)
After training the expert model M₂ = (A,, we can utilize it generate collaborative actions 92 for given 9i predicted by MM1. Thus, we can assess the quality of M1 's actions by evaluating whether .2 can generate cooperative actions. Specifically, to evaluate the predicted !1, we use the trained A and (/2 to generate multiple second gripper action candidates { We then employ C2 to determine how well these second gripper candidates {92} collaborate with the proposed gt. The average critic score reflects how easily the second gripper can cooperate with the proposed first action 91- Consequently, this average score serves the ground truth for the first Critic Network C₁, and we apply L1 loss for supervision:
n
m
6
C2(fin fo, U2( faafffssar fp. 25x)); Cc, = (n ég.
To train the Affordance Network A₁ and Actor Network U in Mi, the loss functions are similar to those used for Ag and Uz. Therefore, with the trained Critic Network C1 the Affordance Network A, assigns high scores to points that can be casily manipulated collaboratively by the subsequent gripper action.
In this training pipeline, the two gripper modules can generate collaborative affordance maps and manipulation actions for bimanual tasks. Note that during inference, the use of the Critic Networks optional.
D. Details about Training and Computational Costs
During training, there are two main components: (1) the Disassembly Predictor and the Transformation Predictor are trained together in an end-to-end manner, and (2) all modules within the BiAssembly Predictor are also trained collectively in an end-to-end manner. These two training components can be conducted simultaneously on single GPU. Using a single NVIDIA V100 GPU, the total training time for our model is approximately 48 hours: the combination of the Disassembly Predictor and Transformation Predictor converges in about 20 hours, while the BiAffordance Predictor converges in about 48 hours.
During inference, our method utilizes only 1,600 MB of GPU memory and processes each data point in an average of 0.1 seconds.
E. More Experimental Results
E.1. Handling Multiple Broken Parts
Table 3. Quantitative Results for multi calti-fframent assembly.
Method
AVG
Ours
24%
22% %
16%
12%
9%
16.6%
Our method can be extended to handle multi-fragmant assembly tasks. In Table 3 we present the accuracy our method on three-fragmene assembly tasks. Additionally, Figure Tigure/Jshowceee shoocase visualizations that include the predicted affordance maps and collaborative actions. Both the quantitative and qualitative results demonstrate that our proposed method can be
17

---

BiAssemble: Learning Collaborative Affordance for Bimanual Geometric Assembly
Initial Observation
Affordance
Pick-up
Alignment
Assembly
Stage 1
0.55
0.5
Stage
Figure 7. We provide the visualization of the predicted affordance maps and actions for multii-prt assembly task.
effectively adapted to multi-fragmest assembly tasks. Below, we provide detailed explanation of how our method can be adapted for multi- fragm-t assembly.
The multiefrrammnn assembly task can be achieved by iteratively applying the wwo-fragment assembly process. First, at each iteration, we can identify which two fragments, prand Pj, should be assembled next. (If some parts have already been assembled in previous iterations, their combination is treated assembled shape S, we calculate the minimum distance,min llp, Pyll. between sampled points from every pair of arg min
new fragment.) Specifically, based on the imaginary
fragments, and the pair (p.. P,) with the minimum distance is chosen for assembly: (ps.Py) (PrPP))ES, XS2
llp: -ncl.Once
P: and Py are identified on S. we then map these fragments their corresponding parts in the observed point cloud O. This mapping is formulated classification task, where the similarity between parts in S and O is estimated.
Finally, using the imaginary assembled shape of the selected fragments, Sp, U Sp, and the corresponding observed point cloud Op. U.ppp our method predicts the actions to pick up and assemble the fragments. This process mirrors the steps of the standard two-fragment assembly method. By iteratively applying this two-fragment assembly process, the complete assembly of all fragments can be achieved.
To validate the feasibility of this multi multi-fragment assembly process, we fine-tune our pretrained BiAssembly model on multi-category three-piece assembly data and present the results on unseen objects. Based on the accuracy reported in Table and the visualizations in Figure we that even for multi-fragment assembly task, method can still generate reasonable results.
E.2. Utilizing the Imperfect Imaginary Assembled Shape Prediction
Predicting the imaginary assembled shape from multiple fractured parts is relatively well-studied vision problem (Sellán et al., 2022, Wu et al. 2023c Lu al.
2024c Tsesmelis et al.) 2024| Scarpellini et al. 2024). Previous works have demonstrated the ability to predict precise fragment poses that allow for an imaginary assembled shape, making it reasonable to assume the existence of such shapes in our framework. Additionally, in traditional furniture assembly tasks, several
studies (Wang et al. 2022a Sera et al. 2021 Wan et al. 2024) also assume the existence of an imaginary Therefore, for simplicity, we adopt the concept of the imaginary assembled shape, aligning with advancements in prior works.
While this simplification reasonable, we also conduct experiments to evaluate our BiAssembly Method on "imperfect" imaginary assembled shapes. Specifically, we first utilize the pretrained model from prior visual assembly method (Wul et al. 2023c) to generate the assembled point cloud prediction. We then replace the input "perfect" imaginary assembled shapes with these "imperfect" predictions. Importantly, the only difference is the change in the input imaginary shape, the subsequent BiAssembly framework remains unchanged, and we do not fine-tune mune the model on the new predictions. The average accuracy based on imperfect imaginary shapes is 20.80% on training categories and 17.20% on novel unseen categories, which comparable to the accuracy on perfect imaginary shapes (24. 10% on training categories and 17.40% on
18

---

BiAssemble: Learning Collaborative Affordance for Bimanual Geometric Assembly
novel unseen categories). These results demonstrate that method is capable of tolerating imperfect input shapes, even without any fine-tuning process, supporting the reasonableness of our assumption for simplification.
E.3. More Ablation Studies
In this section, we conduct three additional ablation studies, and provide the quantitative results in Table for novel objects within training categories and in Table 5 for novel unseen categories.
w/o Affordance Network: During inference, we do use the trained Affordance Network to highlight actionable regions. Instead, randomly sample contact point the part. The results show significant drop in the success rates, which decrease to 4.60% for training categories and 2.80% in unseen categories, This demonstrates that the Affordance Network plays crucial role in filtering out non-granee con-graspable points and points that are unsuitable for the subsequent assembly process.
w/o Transformation Predictor: In this ablation, we remove the Transformation Predictor during inference. This results in success rates of 7.40% on training categories and 4.80% on unseen categories, both substantially lower than our original method. These results show that the Transformation Predictor plays an essential role in predicting alignment poses, enabling the robot to manipulate parts from their initial alignment poses without collisions.
w/ heuristic disassembly direction v: In this case, we remove the Disassembly Predictor during inference. Instead, we compute the center of each part from the imaginary assembled shape S by averaging the part points, and then use the relative direction of the two parts' centers the disassembly direction This ablation achieves success rates of 19.70% training categories and 15.20% unseen categories, both of which are lower than those achieved by method. While this ablated version performs well on certain categories, suggesting that the calculated relative direction can approximate the relative positions of the two parts, it falls short in categories with complex geometries, In such cases, the heuristic method lacks the accuracy needed to replace the assembly direction required for our task. This highlights the critical role the Disassembly Predictor in achieving better performance.
Table More ablation studies: quantitative results in novel instances within training categories.
Novel Instances in Training Categories
Method
AVG
w/o Affordance
7%
11%
0%
0%
1%
8%
1%
4%
6%
8%
4.60%
w/o Transformation
29%
19%
0%
0%
8%
4%
9%
7.40%
w/ heuristic
54%
28%
0%
3%
10%
5%
28%
23%
21%
25
19.70%
Ours
60%
38 %
133 %
13%
12 %
9%
26%
18%
27%
25 %
24.10%
Table 5. More Ablation studies: quantitative results the novel unseen categories.
Unseen Categories
PP
Method
AVG
w/o Affordance
2%
6%
2%
0%
4%
2.8%
Transformation
4%
10%
1%
9%
4.8%
w/ heuristic e Ours
18%
22%
15%
9 9%
12%
15,20%
14%
31 %
10%
7%
25 %5%
17.4%
E.4. More Bimanual Tasks
While we demonstrate the effectiveness on the challenging geometric shape assembly task, our approach is also applicable to broader range of bimanual tasks that require coordinated manipulation. To illustrate this generalizability, we present an additional experiment on bottle cap closing task. This task can be formulated into three sequential steps: (1) The two arms pick up the bottle and cap from the table. (2) The two arms align the cap with the bottle opening. (3) The two arms place
19

---

BiAssemble: Learning Collaborative Affordance for Bimanual Geometric Assembly
and secure the cap the bottle. Our results show that, after training on few bottle shapes, method generalizes well novel bottle shapes, achieving average accuracy of 67%. Figure g provides visualizations of the predicted affordance maps and the manipulation process.
Initial Observation
Affordance
Pick-up
Alignment
Assembly
0.75
0.5
0.25
Figure We provide visualizations of the predicted affordance maps and the manipulation process for the bottle cap closing task.
F. More Visualizations
In Figure y
we present detailed visualization of the manipulation process in real-world experiments, serving as supplement to Tigure5iin in the main paper.
In Figure TO we present additional qualitative results from our method.
20

---

BiAssemble: Learning Collaborative Affordance for Bimanual Geometric Assembly
Initial Observation
Pick-up
Aligament
Assembly
0.75
0.5
0.25
Experiment. We present the results our model tested on real-world scans. each data, visualized the affordance map, and the bimanual actions for the pick-up, alignment, and assembly steps. Manipulation videos can be found in supplementary materials.
Figure
21

---

BiAssemble: Learning Collaborative Affordance for Bimanual Geometric Assembly
Initial Observation
Affordance
Pick-up
Alignment
Assembly
Train Categories
0.75
0.5
0.25
Unseen Categories
Figure 10. We visualize additional qualitative results that augment Figure 4 in the main paper. In each row, from left right, respectively present the input observation, the predicted affordance maps for the two fractured parts, and the bimanual actions for the pick-up, alignment, and assembly steps. the top part are novel shapes from the training categories, while the bottom part are shapes from unseen categories.
22

---

BiAssemble: Learning Collaborative Affordance for Bimanual Geometric Assembly
Initial Observation
Pick-up
Initial Observation
Pick-up
Initial Observation
Pick-up
(c)
Initial Observation
Assembly
Initial Observation
Assembly
Figure visualize some failure cases, which demonstrate challenges tasks some cases are difficult robots to to determine appropriate actions. The first row presents three cases where the fractured parts either too large or too flat to grasp. The second row includes two cases where the graspable region corresponds exactly to the seam areas; while the objects can be grasped, collisions may occur during the assembly of the parts.
G. Failure Cases
We provide detailed analysis of failure cases and illustrate the inherent difficulty of the task with scenarios that particularly challenging for robots figure out. Additionally, we provide insights into potential future improvements to address these complexities more effectively.
G.1. Hard to Grasp
Heavy or Smmooth-Surfacee Parts. For instance, as shown in Figure 11 (a), categories such as teapots and vases, which are relatively large and feature smooth curved surfaces, exhibit notably high failure rates during grasping.
Fractured parts that are heavy have smooth surfaces often result in grasping failures.
Flat Parts. Flat fractured parts, particularly some shapes categories like statues and mugs, are challenging to pick up due to the limited gripping area. For example, as shown in (b), the statue part on the left is too close to the desktop and has very small thickness, which prevent the gripper from grasping Similarly, in (c), the fragment on the right is too flat, making it impossible for the gripper to grasp it. A potential solution is incorporating pre-gr grasp operations, such moving the fractured part to the table edge, allowing the shape hang off slightly and thus become graspable.
G.2. Hard to Assemble
Graspable Regions Overlapping Seam Areas. When the graspable regions of a fractured part align with its areas, collisions during assembly become frequent. This issue is common in categories such as wineglasses, mugs, and bowls. For example, as shown in Figure 11 (d), the left gripper avoids collision-prone regions, but the right gripper must grasp the neck of the wine bottle. Similarly, in (c), while the left gripper avoids collisions, the right gripper ends grasping the handle of a mug. A potential solution is to perform a series pick-and-place operations to adjust the object's initial pose. This adjustment can reduce overlap between the object's graspable regions and seam areas, thereby minimizing collisions during the assembly process.
Complex Object Shapes. Objects with intricate shapes, like those in the statues category, pose challenges due irregular edges and complex curves. Such designs increase the difficulty of alignment and manipulation, leading to higher failure rates during assembly.
Relative Displacement During Operations. Relative displacement between the gripper and fractured parts often occurs due to small contact areas and insufficient support, which can cause sliding or tipping during manipulation. For example, wine bottles with narrow necks, which have unstable center gravity, making the gripper prone to sliding during movement and leading operational failures.
23


Reviews:
[{"review_id": "9VL55SGk33", "reviewer": "Reviewer_z3UB", "summary": "This paper addresses the challenges in the observation space and action space by proposing the BiAssemble framework to solve the collaborative problem of bimanual robots in geometric assembly tasks. Specifically, the task is decomposed into three steps: pick-up, alignment, and assembly, which are addressed by progressively predicting the affordance maps and the gripper actions of the two parts of the object. Additionally, this paper establishes a real-world benchmark for assembling broken objects and conducts extensive experiments in both simulation and real-world environments, demonstrating the superiority of the proposed algorithm and its ability to generalize to unseen object categories.", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We sincerely appreciate your valuable questions, and we have provided detailed responses below.\n\n> W1. The algorithm relies on predefined assembly shapes...\n\nThe assumption of \"imaginary assembled shape\" is justified based on two well-established research areas that together ensure both adaptability and autonomy in real-world scenarios: (1) the reconstruction of broken parts, and (2) the prediction of target assembled shapes from those broken parts:\n\n1. Reconstructing unknown objects (broken parts) with robotic manipulators is a well-studied problem [Paper 1-3]. A feasible approach involves using two robotic arms to capture images of the fractured part and applying the method in Sec. 5.2 for reconstruction. The process and results are available at project website (Rebuttal-Figure1) [https://sites.google.com/view/biassembly/ ]. Additionally, alternative approaches have been explored in [Paper 1-3].\n2. Predicting the imaginary assembled shape from multiple fractured parts is also a well-studied vision problem [Paper 4-8]. Prior works have demonstrated the ability to predict precise fragment poses and shown strong generalization capability to unknown parts and shapes, enabling the construction of an imaginary assembled shape. Furthermore, our experiments (Appendix E.2) show that our method is robust to imperfect imaginary assembled shapes, even without fine-tuning.\n\nThese supporting works and empirical results demonstrate the adaptability and autonomy of our framework in real-world scenarios.\n\n> W2. The relative attitude between gripper and object may change due to external perturbations. Does it affect the robustness of model?\n\nIf external perturbations cause the relative pose between the gripper and the object to change, a pretrained pose estimation model (we used FoundationPose) can track the updated object pose in real-time performance. Consequently, as shown in Equation 2 of our paper, we only need to update the previous gripper pose $ q_{i}^{pick} $ with the perturbed pose $ \\hat{q_{i}^{pick}} $, allowing us to compute the correct gripper pose $ \\hat{g_{i}^{asm}} $ for assembly. Furthermore, since Equation 2 holds at any time step, our approach remains robust throughout the manipulation process. Specifically, at each time step $ t $, we update $ q_{i,t}^{pick} $ (obtained from FoundationPose) and $ g_{i,t}^{pick} $ (obtained from the robot control interface) to compute the appropriate gripper pose $ g_{i,t+1}^{pick} $ for the next step. This ensures that our method can dynamically adapt to perturbations without compromising assembly accuracy. We appreciate this insightful question and will incorporate this explanation into our paper after rebuttal.\n\n> W3. How does the accumulation of iterative errors affect the results of multi-piece task? Will an end-to-end approach be considered for efficient completion in multi-piece tasks?\n\nTo evaluate the impact of iterative error accumulation in multi-piece assemblies, we conduct a comparative experiment with (a)(b) settings in the first iteration: (a) the two parts are assembled by the robot, (b) the two parts are perfectly assembled using ground-truth alignment. Then, we evaluate the accuracy of the second-iteration assembly ''to assemble the third parts to'' under (a)(b) conditions: 21.20% accuracy for setting (a), and 24.80% accuracy for setting (b). These results indicate that iterative errors affect assembly accuracy, as misalignments in earlier steps can propagate and influence the integration of new parts.\n\nWe appreciate the suggestion regarding an end-to-end approach. While our current method is effective for multi-piece assembly, we will explore an end-to-end approach in future work. This would not only consider the geometry of the parts being assembled in each iteration but also optimize the overall assembly sequence based on the geometry of all parts, leading to more efficient and accurate multi-piece assembly.\n\n\n\n**References**\n\n[1] Nicholas Pfaff, et al. Scalable Real2Sim: Physics-Aware Asset Generation Via Robotic Pick-and-Place Setups. 2025\n\n[2] Saptarshi Dasgupta, et al. Uncertainty-aware Active Learning of NeRF-based Object Models for Robot Manipulators using Visual and Re-orientation Actions. IROS, 2024.\n\n[3] Zhizhou Jia, et al. An Efficient Projection-Based Next-best-view Planning Framework for Reconstruction of Unknown Objects. 2025.\n\n[4] Silvia Sellán1, et al. Breaking bad: A dataset for geometric fracture and reassembly. Neurips, 2022.\n\n[5] Ruihai Wu, et al. Leveraging SE-(3) equivariance for learning 3d geometric shape assembly. ICCV, 2023.\n\n[6] Jiaxin Lu, et al. Jigsaw: Learning to Assemble Multiple Fractured Objects. Neurips, 2024.\n\n[7] Theodore Tsesmelis, et al. Re-assembling the past: The RePAIR dataset and benchmark for real world 2D and 3D puzzle solving. Neurips, 2024.\n\n[8] Gianluca Scarpellini, et al. DiffAssemble: A Unified Graph-Diffusion Model for 2D and 3D Reassembly. CVPR, 2024.", "claims_and_evidence": "clear", "methods_and_evaluation_criteria": "yes", "theoretical_claims": "yes", "experimental_designs_or_analyses": "yes", "relation_to_broader_scientific_literature": "none", "essential_references_not_discussed": "no", "other_strengths_and_weaknesses": "Strengths:\n•\tThe paper proposes a novel geometric assembly task focusing on bimanual robot collaboration to repair broken objects, addressing the research gap in the field of geometric assembly for complex shapes and long-horizon action sequences.\n•\tThe idea of introducing point-level affordance is both interesting and highly significant, as it not only predicts the feasibility of grasping points but also simultaneously considers the collaborative requirements of subsequent alignment and assembly steps.\n•\tThe construction of a real-world benchmark bridges the gap between simulation and real-world environments.\n•\tThe article is well-written, with a clear structure, well-defined motivations, and comprehensive experiments.\nWeaknesses:\n•\tThe algorithm relies on predefined assembly shapes, and I suspect that it is unable to autonomously infer the correct assembly of unknown objects, thus limiting its applicability and reducing its adaptability and autonomy in real-world scenarios.\n•\tThe paper assumes that the relative pose between the gripper and the object remains stable, but in real environments, the relative attitude between the gripper and the object may change due to external perturbations. Does this perturbation significantly affect the robustness of the model? If so, in what ways? If the perturbation significantly affects the model performance, in what ways are subsequent plans to address this issue?\n•\tFor the implementation of the methodology mentioned in the authors' supplementary material that can be extended to handle multi-piece assemblies, how does the accumulation of iterative errors affect the results, and will an end-to-end approach be considered for efficient completion in multi-piece tasks?", "other_comments_or_suggestions": "none", "questions_for_authors": "see Strengths And Weaknesses", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "mqIfVEIf1B", "reviewer": "Reviewer_64ad", "summary": "This work focuses the shape assembly task aimed at reconstructing broken objects. A multi-stage BiAssembly framework is put forward to carry out this task. Initially, the BiAssembly framework utilizes SOTA techniques to obtain an imagined assembled shape. Subsequently, it forecasts the disassembly direction, alignment pose transformation, pick-up affordance, and ultimately, the gripper alignment and assembly poses. Moreover, a real-world framework is introduced in this paper. The experimental outcomes demonstrate that the BiAssembly framework outperforms previous approaches.", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for your valuable questions. We've addressed each of your concerns below. \n\n**For detailed paper references, please refer to our reply to Reviewer1 (Reviewer z3UB).**\n\n> W1. The multi-stage framework has some assumptions. For example, it assumes the object has two broken parts, the imagined assembled shape can be got early, and the robot follows a set alignment and assembly process.\n\nFor **(A) multiple broken parts**. We have conducted experiments to demonstrate that our method can handle multiple broken parts in **Appendix E.1**. And both the quantitative and qualitative results demonstrate that our proposed method can be effectively adapted to multi-fragment assembly tasks. We also provide a detailed explanation of how our method can be adapted for multi-fragment assembly in this section.\n\nFor **(B) imagined assembled shape**. The assumption of \"imaginary assembled shape\" is justified based on two well-established research areas that together ensure both adaptability and autonomy in real-world scenarios: (1) the reconstruction of broken parts, and (2) the prediction of target assembled shapes from those broken parts:\n\n1. Reconstructing unknown objects (broken parts) with robotic manipulators is a well-studied problem [Paper 1-3]. A feasible approach involves using two robotic arms to capture images of the fractured part and applying the method in Sec. 5.2 for reconstruction. The process and results are available at project website (Rebuttal-Figure1) [ https://sites.google.com/view/biassembly/ ] . Additionally, alternative approaches have been explored in [Paper 1-3].\n\n2. Predicting the imaginary assembled shape from multiple fractured parts is also a well-studied vision problem [Paper 4-8]. Prior works have demonstrated the ability to predict precise fragment poses and shown strong generalization capability to unknown parts and shapes, enabling the construction of an imaginary assembled shape. Furthermore, our experiments (Appendix E.2) show that our method is robust to imperfect imaginary assembled shapes, even without fine-tuning.\n\nThese supporting works and empirical results demonstrate the adaptability and autonomy of our framework in real-world scenarios.\n\nFor **(C) alignment and assembly process**. The alignment and assembly process mirrors the natural approach humans take when assembling fragments. Humans typically align the fragments along the seams first and then gradually move them together for precise fitting. Furthermore, when decomposing the assembly process into multiple frames, there is usually a stage where the two fragments are aligned but separated by a small distance. This intermediate step is captured in our formulation as the alignment step, which generalizes well to most shape assembly scenarios.\n\nThanks for your valuable comments! We will add the above discussions in our paper and make it more clarified.\n\n\n\n> W2. Overall, this paper presents a viable framework for shape assembly, which is beneficial for this field. However, the framework in this paper is limited by its single-task-oriented design. This makes such methods less general and less likely to inspire a wider readers. I tend to accept this paper. Meanwhile, I hope that the author can strive for greater generality in the design of the model in the feature.\n\nThank you for this constructive comment. Our framework focuses on learning bimanual collaborative and geometry-aware affordances to generate long-horizon action sequences for robotic manipulation tasks. While we demonstrate the effectiveness of the proposed method in geometric shape assembly task,  which involves diverse object categories, complex geometries, and significant generalization challenges, the learned affordance is applicable to a broader range of bimanual tasks that require coordinated manipulation.\n\nFor other tasks requiring bimanual coordiniation, such as peg insertion, bottle cap closing, and furniture assembly (which are relatively easier than geometric shape assembly), our method can be easily adapted. For instance, in the bottle cap closing task, the process can be formulated into three steps: (1) The two arms pick up the bottle and cap from the table. (2) The two arms align the cap with the bottle opening. (3) The two arms place and secure the cap onto the bottle. We conducted experiments on this task, and the results show that after training on a few bottle shapes, our method generalizes to novel bottle shapes, achieving an average accuracy of 67%. Visualizations of the predicted affordance maps and the manipulation process are available on our project website (Rebuttal-Figure3)  [ https://sites.google.com/view/biassembly/ ] .", "claims_and_evidence": "Yes", "methods_and_evaluation_criteria": "Yes", "theoretical_claims": "Yes", "experimental_designs_or_analyses": "Yes", "relation_to_broader_scientific_literature": "This work is related to shape assemble perdition task. And it focus on the execution policy.", "essential_references_not_discussed": "Yes", "other_strengths_and_weaknesses": "Strengths\nOverall, the paper is written well. Technical details and experiments are clearly explained. The framework for shape assembly seems to work. According to the paper's results, it does better than old heuristic or policy - based methods.\n\nWeaknesses\nThe multi-stage framework has some assumptions. For example, it assumes the object has two broken parts, the imagined assembled shape can be got early, and the robot follows a set alignment and assembly process.", "other_comments_or_suggestions": "Overall, this paper presents a viable framework for shape assembly, which is beneficial for this field. However, the framework in this paper is limited by its single-task-oriented design. This makes such methods less general and less likely to inspire a wider readers. I tend to accept this paper. Meanwhile, I hope that the author can strive for greater generality in the design of the model in the feature.", "questions_for_authors": "None", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "1kjMfwkmkf", "reviewer": "Reviewer_KY79", "summary": "In this paper, the authors present a framework for bimanual geometri assembly. They formulate the task into 3 steps: pick-up, alignment and assembly. For pick-up, a point-level affordance prediction module is trained and used; For alignment, a SE(3) transformation is predicted; For assembly, a collision-free direction is predicted. \nThe authors also introduce a real-world benchmark featuring geometry variety and global reproducibility.\nThe authors evaluate their method in simulation and on real data. The results outperform previous affordance-based and imitation-based methods.", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We sincerely appreciate your thoughtful questions. We've addressed them in detail below.\n\n**For detailed paper references, please refer to our response to Reviewer1 (z3UB).**\n\n> W1. Thresholds of distance and rotation. There is a gap between two parts after assembly.\n\nThank you for the suggestion. The threshold for distance is set to 2 unit-length in the simulation, and threshold for rotation is 30 degrees. While we are not permitted to make revisions during rebuttal, we will include this information in the final paper. Geometric shape assembly is a particularly challenging task due to the diversity of object categories, complex geometries, and the significant generalization required. In fact, even humans find it difficult to assemble shapes based solely on visual information. In this paper, we have taken a successful first step toward addressing this challenge. Moving forward, we plan to incorporate tactile information into our method, as it is especially valuable in contact-rich tasks and may help achieve more precise assembly of fractured parts.\n\n> W2. Comparisons with existing works on geometric assembly. \n\nWe conducted a comparison with an existing geometric assembly method [Paper-5]. Although this method only considers the geometries and ideal assembled poses of fractured parts, without taking the robotic assembly process into account, we adapted it as a baseline by: (1) using a heuristic method to generate the robots' pick-up actions (detailed in Appendix B), (2) denoting the predicted SE(3) pose for part i as $q_{i}^{asm}$, and using Equation 1 in our paper, we calculate the gripper’s target pose $g_{i}^{asm}$ for assembly. The average accuracy of this baseline is 3.00% on the training categories, which is significantly lower than our method. The main reason for this performance gap is that prior visual assembly methods neglect the robotic execution process. Specifically, these methods do not determine where to grasp the fragments, not only for successful pickup but also to avoid the seam region for subsequent assembly. They also lack the capability to align the fragments properly at the seam, which is crucial for avoiding collisions when the two parts are brought together. On the contrary, our method integrates the considerations of part geometry, shape assembly with robotic coordination and execution in the proposed affordance learning framework.\n\n> W3. The contribution of this work is specific to this task.\n\nThank you for this constructive comment. Our framework focuses on learning bimanual collaborative and geometry-aware affordances to generate long-horizon action sequences for robotic manipulation tasks. While we demonstrate the effectiveness of our method in geometric shape assembly task, which involves diverse object categories, complex geometries, and significant generalization challenges, the learned affordance is applicable to a broader range of bimanual tasks that require coordinated manipulation.\n\nFor other tasks requiring bimanual coordiniation, such as peg insertion, bottle cap closing, and furniture assembly (which are relatively easier than geometric shape assembly), our method can be easily adapted. For instance, in the bottle cap closing task, the process can be formulated into three steps: (1) The two arms pick up the bottle and cap from the table. (2) The two arms align the cap with the bottle opening. (3) The two arms place and secure the cap onto the bottle. We conducted experiments on this task, and results show that after training on a few bottle shapes, our method generalizes to novel bottle shapes, achieving an average accuracy of 67%. Visualizations of the predicted affordance maps and manipulation process are available on project website (Rebuttal-Figure3) [https://sites.google.com/view/biassembly/ ] .\n\n> W4. Figure 2 is convoluted\n\nIn the paper, Figure 1 introduces the intuition behind our proposed framework, while Figure 2 provides a more detailed illustration. Based on your feedback, we have revised and simplified Figure 2 to make it clearer and easier to follow. The updated version is on our website (Rebuttal-Figure2).\n\n> W5. The method relies on reconstruction from multi-view images, which may be impractical for real-world applications.\n\nReconstructing unknown objects (such as fractured parts) with robotic manipulators is a well-established research area. A feasible approach involves using two robotic arms with wrist camera to capture the images of fractured part and applying the method in Sec. 5.2 for reconstruction. The process and results are available at project website (Rebuttal-Figure1). Additionally, alternative approaches have been explored in previous works [Paper 1-3], including allowing the robot to re-orient the object while collecting visual observations to facilitate the reconstruction of unknown objects. Therefore, object reconstruction in real-world scenarios is both feasible and not inherently difficult, given the available techniques and tools.", "claims_and_evidence": "The claims are well supported by the comparison and ablation studies results.", "methods_and_evaluation_criteria": "The authors use the assembly success rate as the evaluation criteria, which can be reasonable. But the thresholds of distance and rotation angels are not given. It is very important for real applications. And according to the figure and video, there are still a large gap between the two parts after assembly.", "theoretical_claims": "There are no theoretical claims.", "experimental_designs_or_analyses": "The authors compare their approach using three methods: ACT, a manually designed heuristic strategy, and a modified version of the existing affordance prediction method, DualAfford. However, they do not include comparisons with existing works on geometric assembly. The authors justify this omission by stating that these prior works do not account for robot execution. While this reasoning has merit, it would strengthen the study if the authors had adapted these geometric assembly methods to the current setting and included them in the comparison. Such an approach could provide a more comprehensive evaluation of their method’s performance relative to established baselines.\nAdditionally, the authors perform ablation studies to assess their design choices, which are reasonably constructed.", "relation_to_broader_scientific_literature": "The contribution of this work is very specific to this task, and may not contribute much to the broader community.", "essential_references_not_discussed": "NA", "other_strengths_and_weaknesses": "The proposed framework is designed specifically to the task. However, its technical novelty is quite limited and does not meet the standards expected for ICML. \nThe paper is well-written, and the style of the figures is visually appealing. That said, the logic of Figure 2 is convoluted and difficult to follow.\nAdditionally, the method relies on complete reconstruction from multi-view images,which may be impractical for real-world applications.", "other_comments_or_suggestions": "NA", "questions_for_authors": "NA", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "SlkrnkqzVD", "reviewer": "Reviewer_LSLY", "summary": "This paper proposes a framework for learning collaborative affordance in bimanual geometric assembly. The task is assembling fractured parts into complete objects, which is a long-horizon task requiring pick-up, alignment, and assembly. The paper tackles this task through predicting collaborative affordance and gripper actions for bimanual geometric shape assembly. A real-world benchmark for re-assembling broken parts is created. Evaluations demonstrate the effectiveness of the approach and shows generalizability to unseen object categories in both simulated and real-world environments.\n\n## update after rebuttal\nThe paper addresses a novel and useful task of bimanual geometric assembly. The authors also provide additional experimental results applying the method to other tasks (e.g., a bottle opening task). However, the concerns of relatively low real-world performance and many assumptions such as imagined assembly shapes, floating grippers, relying on pose trackers, etc. remain unaddressed. I will maintain my original score.", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for your thoughtful review and questions. We have carefully addressed them below.\n\n> W1. Low reported success rates\n\nThe relatively low scores across all models and baselines stem from the highly diverse and complex nature of geometric shape assembly task. As detailed in Appendix G, our dataset contains varied fracture patterns, including many flat or minimal parts that are extremely difficult to grasp and assembly. We intentionally chose this challenging dataset to set a benchmark for future improvements. For instance, future methods could involve pre-grasp operations like moving flat parts to the table edge to make them graspable.\n\nWhile we demonstrate the effectiveness of our method in this complex task, our approach is also applicable to a broader range of bimanual tasks. Due to space limitations, we kindly refer the reviewer to our response to Reviewer3 (KY79m), W3, where we provide a detailed clarification that our method can be adapted to tasks like bottle cap closing, and has good performance (67% accuracy).\n\n> W2. Assumption of imaginary assembled shape. Two-fragment task setup. Three-fragment task' success rate.\n\nFor **(A) imagined assembled shape**, we kindly refer the reviewer to our response to Reviewer1 (z3UB), W1, for a detailed clarification.\n\nFor **(B) multi-fragment assembly**, our experiments show that our method can handle multiple parts (Appendix E.1). The relatively low success rate is mainly due to the presence of more minimal parts that are nearly impossible to grasp or assemble. Rather than excluding these highly challenging cases from test dataset, we intentionally include them for an honest evaluation. We recognize that multi-fragment assembly introduces additional challenges, we will explore pre-grasp and pre-orientation operations to handle such complexities and improve multi-fragment assembly.\n\n> W3. Quantitative results for real-world experiments. Sim2real transfer analysis.\n\nFor (A) real-world experiments, we tested our model on each category with 10 trials, without any fine-tuning. The success rates are as follows: Bowl: 3/10, Mug: 2/10, BeerBottle: 3/10, WineGlass: 2/10. For mug, in some trials, we intentionally place it with the handle facing downward, making it ungraspable, so the gripper must grasp mug's top edge. This leads to collisions when both grippers grasp the top edges, due to mug's small diameter. For wineglass, its glasswork is prone to slipping, even when gripper successfully grasps it, it may tip during manipulation.\n\nFor (B) sim2real transfer analysis, we load the object meshes (acquired from our real-world benchmark) into simulation. We observe better results in simulation, primarily due to discrepancies in joint constraints. For instance, when picking up a flat fragment on a table, gripper in simulation can move parallel and close to the table surface, whereas real robot encounters joint limitations that restrict its movement. This comparison highlights the need to incorporate bimanual joint constraints into simulation framework to better reflect real-world scenarios and improve transferability.\n\n> W4. floating grippers\n\nIn this work, following [Paper 9-11], we focus on learning collaborative affordance, abstracting away the robot arm control. While our real-world experiments show the proposed actions work with real arms using motion planning (MoveIt!), we acknowledge that incorporating arm control would enhance the system’s realism. In future work, we will address these challenges including arm singularities and collision issues, integrating cuRobo for motion generation for bimanual manipulators. \n\n> W5. failure analysis and actionable insights\n\nIn our failure analysis, we provided potential solutions for future works, including (1) incorporating pre-grasp operations like moving flat part to the table edge to make it graspable, (2) performing a series of pick-and-place operations to adjust object' pose. More details are available in Appendix G.\n\n> W6. Which component is the most brittle part?\n\nWe have conducted additional ablation studies, with results and analysis in Table 4,5 of Appendix E.3.\n\n> W7. Is model robust to occlusions or noise?\n\nAs described in Equation 2 of our paper, the pose estimator does not need to precisely predict the absolute object pose. Instead, it only needs to estimate the relative pose between two frames, which significantly simplifies the task. Beisdes, our empirical results show that FoundationPose, the SOTA model we use, performs well in continuous manipulation scenarios, maintaining accuracy and robustness even with occlusions (e.g., gripper occlusion after grasping) or sensor noise.\n\n**References**\n\n[9] Eisner, et al. Flowbot3d: Learning 3d articulation flow to manipulate articulated objects. RSS, 2022.\n\n[10] Xu, et al. UMPNet: Universal manipulation policy network for articulated objects. RAL, 2022.\n\n[11] Zhao, et al. Dualafford: Learning collaborative visual affordance for dual-gripper manipulation. ICLR, 2023.", "claims_and_evidence": "The paper claims to provide an effective solution for bimanual geometric assembly, but the reported success rates in experiments are low (20-30%), which does not support the reliability and practicability of the approach.", "methods_and_evaluation_criteria": "The integration of collaborative affordance prediction with geometric reasoning demonstrates potential for advancing bimanual assembly tasks.\n\nThe method has many assumptions. It assumes the availability of an ideal \"imaginary assembled shape\". Also, the task setup mainly considers objects with two fragments, however, in reality there could be an arbitrary number of fragments with multimodal contacts involved. Three-fragment assembly task results are shown in the supplementary material, but the success rate is quite low and cannot fully validate the scalability of the approach to multi-fragment assembly.\n\nA real-world benchmark on geometric assembly is created, which paves way for future research on this direction.", "theoretical_claims": "The equations in the paper are correct.", "experimental_designs_or_analyses": "Thorough evaluations in both simulation and real-world environments are carried out to demonstrate the effectiveness of the approach. The model is generalizable to shapes from unseen categories.\n\nThe proposed ablations validate the role of individual components, such as disassembly prediction and SE(3)-equivariant representations in the obtained performance.\n\nFor real-world experiments, only qualitative results are presented, there is a lack of quantitative results on more object shapes and comparisons to other baselines. There is also a lack of more detailed sim2real transfer analysis, for example, side-by-side assembly comparison of an exact same set of shapes in simulation and the real world.\n\nThe evaluations in simulation are carried out with floating grippers. It would be more realistic to control grippers mounted on bi-manual arms, as there could be singularity and arm-table collision issues that are not being taken into account with the floating grippers.", "relation_to_broader_scientific_literature": "The paper addresses a useful task that has been under-explored in previous robotics works, and provides an effective approach to solve this challenging task.", "essential_references_not_discussed": "No.", "other_strengths_and_weaknesses": "The paper addresses an underexplored but important task in robotics and provides a novel solution to tackle it. However, it relies on many assumptions (e.g., the availability of an \"imaginary assembled shape\" and the restriction to mostly two-part assemblies) and has low success rates and robustness. It is also not thoroughly validated in real-world environments of its effectiveness and scalability.\n\nThe failure analysis is helpful in categorizing errors but did not provide actionable insights or detailed solutions to address the low success rates.", "other_comments_or_suggestions": "See above.", "questions_for_authors": "Why is the success rate so low? Which component is the most brittle part?\n\nHow would the accuracy of the pose estimator affect the performance? If the pose estimation is a bit off due to occlusions or sensor noises in the real world, would the model be robust to it and still manage to succeed?", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Meta-review: This paper received four weak accepts after rebuttal. All reviewers acknowledged its novel contribution to geometric assembly tasks, addressing key challenges in complex shape manipulation and long-horizon action planning. The proposed point-level affordance method was noted as both innovative and practically relevant. The paper is well-structured, with clear motivations and comprehensive experimental validation. Most concerns raised during review were resolved in the rebuttal; notably, one reviewer raised their score from 2 to 3, solidifying the consensus. The Area Chair concurs with the reviewers and recommends Accept.
Decision: Accept (poster)