Jigsaw: Learning to Assemble Multiple Fractured Objects
Authors: Jiaxin Lu
Yifan Sun
Qixing Huang
Venue: NeurIPS 2023 poster (2023)
Keywords: Shape Matching; Reassembly; Shape Segmentation;
TL;DR: We present Jigsaw, a novel framework for assembling physically broken 3D objects from multiple pieces.

Abstract: Automated assembly of 3D fractures is essential in orthopedics, archaeology, and our daily life. This paper presents Jigsaw, a novel framework for assembling physically broken 3D objects from multiple pieces. Our approach leverages hierarchical features of global and local geometry to match and align the fracture surfaces. Our framework consists of four components: (1) front-end point feature extractor with attention layers, (2) surface segmentation to separate fracture and original parts, (3) multi-parts matching to find correspondences among fracture surface points, and (4) robust global alignment to recover the global poses of the pieces. We show how to jointly learn segmentation and matching and seamlessly integrate feature matching and rigidity constraints. We evaluate Jigsaw on the Breaking Bad dataset and achieve superior performance compared to state-of-the-art methods. Our method also generalizes well to diverse fracture modes, objects, and unseen instances. To the best of our knowledge, this is the first learning-based method designed specifically for 3D fracture assembly over multiple pieces. Our code is available at https://jiaxin-lu.github.io/Jigsaw/.

Jigsaw: Learning to Assemble Multiple Fractured Objects
Jiaxin Lu
Yifan Sun Qixing Huang
Department of Computer Science
University of Texas Austin
(lujiaxin, yifansun12)@uterxx edu
huangqx0cs. utw.sssuueexeaee..
Abstract
Automated assembly of 3D fractures is essential in orthopedics, archaeology, and daily life. This paper presents Jigsaw, novel framework for assembling physi- cally broken 3D objects from multiple pieces. Our approach leverages hierarchical features of global and local geometry match and align the fracture surfaces. Our framework consists of four components: (1) front-end point feature extractor with attention layers, (2) surface segmentation to separate fracture and original parts, (3) multi-parts matching to find correspondences among fracture surface points, and (4) robust global alignment to recover the global poses of the pieces. We show how to jointly learn segmentation and matching and scamlessly integrate feature matching and rigidity constraints. We evaluate Jigsaw the Breaking Bad dataset and achieve superior performance compared to statc-of-the ohethh--trrr art methods. Our method also generalizes well to diverse fracture modes, objects, and unseen instances. To the best of our knowledge, this is the first learning- based method designed specifically for 3D fracture assembly over multiple pieces. Our code is available https://jiaxin-D githtu.ii//igsseee
Introduction
The task assembling 3D fractures has extensive applications across numerous fields. For instance, orthopedic doctors need to realign dislocated bone fragments, and subsequently create bone plates and screws to heal compound fractures. Archaeologists, on the other hand, need to recreate the original shape and funetionality of unearthed artifacts by assembling the fractures. These procedures demand significant expertise, are prone to errors, and can be tedious. Even in the context of daily life, furniture assembly can be challenging and exhausting task that requires an understanding of mechanical structures and component matching. In the past two decades, many efforts attempted to address the challenge of automatic assembly. Traditional methods apply hand-crafted geometric features to detect the fracture surfaces and optimize pairwise matching among these surfaces [1, 2]. Recently the availability of large scale 3D datasets [3, 4, 5, 6] have boosted learning based frameworks for solving 3D assembly tasks. Semantic-aw aware methods [7, 8, 9, 10. 11, 12] target at assembly from semantically segmented parts and predict semantic labels as matching priors. Geometry based methods [13, 14, 15] leverage fracture shapes and continuity of textures in the procedure of piece matching.
However, in the context of fracture assembly for restoring broken objects, there is no guarantee that individual pieces will retain semantic meanings. Texture information may also be either non- accessible lacking, such on glass bottles worn artifacts. This calls for general 3D assembly solver that utilizes hierarchical features of both the global piece surfaces and local geometry shapes of fractures. In this work, introduce Joint Learning of Segmentation and Alignment Framework
Equual Contribution
37th Conference on Neural Information Processing Systems (NeurIPS 2023).

---

(Jigsaw), for assembling objects damaged or shattered due to physical impact or force. Given a set of fractured pieces represented as point clouds without texture, our method recovers the global pose of each piece to restore the underlying object.
Our Jigsaw framework consists of four parts: (1) A front-end feature extractor with self-attention and cross-attention layers for local geometric feature extraction. (2) Categorize the surface of each fractured piece into two segments: the indiscernible fracture surface and the visible original surface. (3) A novel formulation for multi-piece assembly that learns a bipartite matching among points on the fracture surface from all pieces. (4) Recover pairwise pose based on the learned correspondences and perform robust global alignment to compute the global poses of all pieces. Key features of Jigsaw are that 1) it learns two correlated tasks, i.e., segmentation and matching among all involving pieces, jointly, and 2) it seamlessly integrates feature matching and the rigidity constraint among consistent features, We evaluate the effectiveness of our framework on Breaking Bad [4]. Experimental results demonstrate the effectiveness of our method in surface segmentation, fracture point matching, and significantly outperforming state-of-the-an methods[8, 16]. The main contribution includes the following:
We propose Jigsaw, a novel joint learning framework tailored for multi-part fracture assembly. Our approach embodies an attention-based feature extractor network to accurately capture local geometry features of each point. Additionally, we introduce primal-dual descriptor that effectively captures viewpoint-depeddent characteristics for surface matching.
Jigsaw incorporates fracture point segmentation to capture intrinsic features, employs novel multi-part matching formulation to establish automatic piece positioning within one object, and utilizes global alignment for accurate global pose alignment.
Experimental evaluation on the multi-part assembly Breaking Bad dataset demonstrates the superior performance of Jigsaw compared to baseline models, showcasing its strong generalization ability to unseen objects. We further highlight the limitations of baseline models that rely on global features, which we argue are too abstract for this task and lack generalizability.
To the best of our knowledge, Jigsaw is the first learning-basee based method specifically designed for the assembly of multiple pieces from physically broken 3D objects.
2 Related Works
Feature matching. Early works apply hand-crafted features over fracture surfaces for fracture matching between different pieces [17, 1, 2, 18). These features are also used to identify the fractured part from the entire surface [17, 1, 2]. However, hand-crafted features are in lack of robustness for assembly tasks over large datasets due to different materials and fracture patterns of objects. In recent years, deep learning methods have gained significant traction in matching problems. Methods utilizing CNN, GNN, and attentions have found successful applications in various domains, such as image registration [19, 20] and graph or multi-grap -gaphh matching on images [21, 22, 23, 24, 25). While their efficacy in these scenarios is noteworthy, the majority of these methods predominantly focus on simpler settings and have not adequately addressed the challenges in multi-par 3D fracture assembly.
Part assembly. Semantic- aware learning methods have been highly successful in the task of part assembly. [9. 11] are designed for assembling specific CAD mechanics. For categorical everyday objects. [26, 10, 27, 12, 28] generate the missing parts based on the accumulated shape prior to completing the entire object, which can result in shape distortion from the input parts. [8, 7] apply graph learning to predict part labels and assembly orders. All of these methods require the input objects decomposed in a semantically consistent way and need specific training for each category of objects. Fracture assembly poses unique challenges due to the variety of objects and the lack of semantic meanings associated with individual pieces, adding to the complexity of the task.
Geometry based learning methods. Recent approaches have aimed to capture geometry information using deep features for piece matching. [13, 15] combine local geometry with textures for feature generation, but may suffer when texture information is not accessible, as in most point cloud representations. [14] apply a transformer for local shape encoding and an adversarial network to help generate a plausible assembly of two pieces. However, for multiple fracture assembly, objects can break into 5 pieces or more, with the largest piece smaller than half of the original object. Under this setting, training an adversarial network to evaluate the quality of assembly becomes less effective.

---

FFooneed Feature Extractor
Multi-part Matching
R(Rx
Po0k
Ar--Attttion
Cross Attention
Primal-dual Descriptor
Affinity Mabric
Fro x
Surface
Global Fracture A Alignment
Segmentation
RANSACIP, A. RANGACEP, &
Hungarian
Pairwise Pose Recovery
Pointhlet++
Aligmment
Figure 1: Overall pipeline for Jigsaw (mesh used only for visualization). The method consists of four parts: front-end feature extractor, surface segmentation, multi-part matching, and global fracture alignment. In the front-end feature extraction, self-attention layer followed by one cross-attention layer to extract features for each point. Surface segmentation is used to locate all fracture points in broken object, and multi-part matching finds correspondences of fracture points among multiple pieces. The matching results will be used for pairwise pose recovery and joint alignment to retrieve assembled object. More detail about each part will be discussed in Section 3.
multi-scale grouping PointNet++ [31], and one
Low-overlap 3D registration. Another relevant research field low-overlap 3D registration. Recent works have shown the potential data-driven methods in registration tasks with approximately 30% overlap [29, 16, 30]. [16, 30] learn classifier to determine the overlapped sections and predict inter-piece point matching inside the sections, while [29] using object semantic to guide registration. The fracture assembly task be viewed even more extreme case of registration, where the overlap between pieces can fall below 4%.
Joint learning framework for 3D fracture assembly
Given a set of fractured pieces P (Pi, Pa.-. ... PP. represented as point clouds uniformly distributed the surface, our goal is to recover the 6-DoF pose (Ti, . Ta in SE(3) for each piece and restore the underlying object 0 (P1) (PP)))) (P.), where T;(.,, S n is the operator recover P; to its original position by transformation T, In the fracture assembly setting, the object O is a rigid body, and the pieces are the result of physical cracking or breakage without any deformation. Also, no pieces are lost, which ensures that the restored pieces can approximately reconstruct the entire object O.
To handle this challenge, we propose Jigsaw, a learning-based framework that jointly optimizes surface segmentation and fracture matching among all pieces of the object. The entire assembly network consists segmentation module and a matching module that shares a front-end feature extractor. The segmentation module uses only the intrinsic shape information of each piece to separate the fracture surface from the original surface. The matching module exploits the primal dual descriptor of the affinity metric propagate mutual information among all pieces and establish the matching between fracture points (that is, points on the fracture surface) of different pieces via Sinkhorm [32]. With the fracture point matching predicted by the network. recover the pairwise transformations and perform global alignment with standard approaches. The complete pipeline of framework is shown in Fig. Note that this design ensures that segmentation and matching are performed jointly. In the subsequent part of this section, we will first introduce the front-end extractor, then delve into the details of the segmentation module, followed by the multi-pan part matching module and post-processing for pose recovery.
3.1 Front-end Feature Extractor
In contrast previous methods [33, 4] that utilize global piece-wise descriptors for assembly, focus on geometric features. As illustrated in Figure 7 of the PointNet paper [34], global descriptor remains the same long the piece lies between the learned critical point set and the upper-bound

---

point set. This indicates that global descriptors are coarse and insufficient to represent the intricate geometry of the fracture surface, which crucial for our task. To address this limitation, we employ multi-scale grouping +ointNet++ [31] as the backbone of our feature extractor to capture local geometry features, denoted f, I2" from each point p P.-
Furthermore, We emphasize the significance of relative positional information between points and pieces for identifying the non-smooth fracture surfaces, enforcing rigidity during matching, and accurately placing the pieces. To facilitate the integration of such intra-picce and inter-piece information, we leverage transformer layers. Specifically, we introduce both self-attention and cross-attention layers as tools to reason about the relative information between points.
For the self attention within piece, we employ single point transformer layer [35]. The self- attention mechanism is defined follows:
Σ F.EN(())
Doftmax(MMP_s(((M
- Wif,, + +D")))) e (WFf, + +
(1)
p
Here, W.: W We E RDDD are weights for query, key, value in the attention layer, pete represents the positional encoding for point pt, obtained using a Multi-Layer Perceptron (MLP). Unlike the standard dot-product attention layer, it employs vector weights: MLP s is mapping function that produces the weight vector, sof tmax(-) normalizes the weights, and @ denotes element-wise multiplication. (p)) E P. represents the neighborhood of point p. calculated using k-nearest neighbors for local feature aggregation.
For cross-attention, we use standard multi-head attention with position-win -iite feed forward over the entire object to facilitate the communication of local features among different pieces. Let F E R pack all the point features of one object after the self-attention, and F E xDD denote the point features produced by cross-attention,
F (Ca(W""cathheerl (eenheead....,hadd)))) where head, Attention (W F², W F", W K
QKT
and Attention Q, K,KK V) = softmax
Vdg V
(2)
Here, W 5: W K WV, E DDdx, W E Ad, x denote the weights for projection. The function FFN(-) consists of two linear functions with ReLU activation in between, followed by a LayerNorm as normalization.
3.2 Surface segmentation
The fracture assembly problem can be viewed as special case of the 3D registration problem, where the overlapping between adjacent pieces only lies in the fracture surface between them. Let P/ C P, be the subset of points fracture surfaces of P; and let P.ş C Pif denote the subset of points on the fracture surface between P. and P; for arbitrary two pieces P,, P,. For pieces P. and P3 adjacent in their original pose (P:)) and (P;), we have P.j * 0, Pąe # Ó. Under the assumption that the point clouds are uniformly distributed, we can approximate the relative pose Ti = T 'T, between P,, P3 with
T
argmin d (iy(()) T.JES((3)
(3)
where d(.,-) is some distance function between two point clouds. (3) becomes the standard formula- tion for point cloud registration with perfect prior for the overlapping. Before we can optimize (3) directly, need figure out the overlapping part P.j and Pj= for each pair (P., Pý). Although Psş
dependent on both pieces, previous studies [17, 1, 2 have demonstrated that the segmentation of fracture surfaces P. of P. is an intrinsic property that can be inferred from the local shapes of each P; individually. However, these methods require continuous smooth surface to compute hand-crafted shape features for accurate classification. To capture local geometric properties under the discrete point cloud setting where surface normal and curvature are more available, we introduce deep surface segmentation module to handle the surface segmentation task.
The surface segmentation module can be viewed indicator function that determines whether a point p of a point cloud P E P is a fracture point. For each 3D model uniformly sampled N = >n i== N; points, the surface segmentation module
binary classifier of each point. Let cp be the

---

takes the position of the points P, E RN:x3 of a piece P. E P and predicts confidence score c₁ to segment point p E P. into the fracture surface.
Our front-end feature extractor introduced in Section 3.1 has extracted local geometry feature E IR_ for each point. Next, use two MLP layers to reduce the number of channels to 1, followed by a sigmoid function to predict the confidence ep for labeling as a fracture point. We set the segmentation loss Lug to be negative log- log-likelihood loss, which is supervised by the ground truth label Cp for every point as
Σ
cp log + cp) log(1 Cp)).
N
PÊO
The ground truth label cp for each point E P; is constructed by examining its distance to its nearest neighbor in other parts:
C {
Dis(p, (N(,, P\).)) ?
(5)
otherwise.
where (-s(-) is distance function and NN(:.) is to find the nearest neighbour of one point in point set.
Although the fraction of the fracture surface area may differ between objects, and the positive and negative samples in (4) may be unbalanced, we have found that adding two MLP layers after feature encoding help to yield good performance for the segmentation task. The segmentation module is able accurately predict fracture points even in cases with large number of pieces very small fracture surfaces, and we leave the evaluation details of this module in the appendix A.
3.3 Multi-part Matching
One key aspect of Jigsaw is its ability to combine multiple pieces. Traditional approaches to assembling pieces rely on pairwise matching, which involves assessing the compatibility of two parts, identifying corresponding points within them, and aligning those points. However, this method prone cumulative errors, and single mistake can ruin the entire assembly task. In addition, matching small pieces with limited geometry information solely based on pairwise information is extremely challenging. Consequently, the pairwise information in the multipart fracture assembly dataset may not be comprehensive enough to capture the true matching possibilities between all the pieces involved in the assembly process. Even advanced low-overlap 3D registration methods like PREDATOR [16], which share a similar structure, have failed to handle assembly tasks effectively, as shown in Section 4.2. Thus, alternative approaches needed address the limitations of pairwise matching, and we introduce our multipart matching module.
Facilitated by the global fracture points of seg- mented object in Section 3.2, we present a global view to match all its pieces simultaneously. We observe that each fracture surface should have one precise match within a broken object, which is a rule that applies specifically to the fracture piece assembly task. Therefore, we learn the matching among all pieces without specifying their pairwise relationship. This allows each fracture point to automatically find a similar counterpart on a dif- ferent piece solely based on local geometry. More- over, the correspondences highlight the assembly regions, avoiding the need for costly predictions of fine-grained pairwise overlapping areas.
M||
1il
Prinal Descrpar
Descripor
Primal-dual Descriptor
Fracture Frace
Figure 2: Illustration of Primal-dual descriptor: The red region and points represent the target fragment piece and the points, respectively. The purple feature (primal descriptor) on the left learns the local convex shape of the piece, while the blue feature (dual descriptor) on the right captures the concavity observed from the out- side. The primary and dual descriptors on both sides will be matched in the matching module.
Primal-dual Descriptor. Intuitively, when an ob- ject has been broken into pieces by physical force, the two pieces should exhibit complemen- tary geometry. Conventional feature representa- tion is used to find similar geometry and, there- fore, would match two identical surfaces, which becomes undesirable in the context of fracture assembly. With this concern, we propose the primal- dual descriptor, designed to capture the essence of complementary geometry (see Fig. 2).
5

---

Let [f be the selected features of the fracture points extracted from the backbone network. These features encode the local geometry of the fracture surfaces. We apply MLP layers and normalization layer f and get the primal descriptor f." and dual descriptor f.
f f Morm(MMP_aff((?)
(6)
The primal-dual descriptor learns surface features that capture the characteristics of local surface from both directions. It promotes the robustness and reliability of surface matching and avoids learning biased features based on a single viewing direction. We leave more detailed analysis the appendix B.
Affinity Metric. Let F2, Fd E R collect all the fracture points features extracted from descrip- tors, compute the global affinity matrix M & and the respective doubly-stochastic matrix X [0, 1 as follows:
FP AFd
M exp
X Sinkhorn(M).
Here, [Rd consists of learnable affinity weights, and 7 denotes the temperature parameter [36]. We then apply the Sinkhorn layer [32], which is differentiable operation, to obtain doubly stochastic matrix X [0, 1| This matrix represents the soft matching predicted by the multipart matching module.
Matching Loss. Given the ground truth position of each part, construct the ground truth matching matrix X E [0,1] Nx Here, = if and only if the j-th fracture point is the nearest neighbor of the i-th fracture point and both points belong to different pieces; otherwise, zij = 0. To compute the matching loss, employ the cross entropy between X and Xsc;
mat
1 Ñ
Σ
of
log Tig + - 2gt log(1 k,j)) by
ISJJN
With the ground truth matrix x generated based on finding the nearest neighbor each point, the matching loss will also enforce global rigidity guidance that each node should be matched to its neighbor.
Rigidity Loss. As 3D fracture assembly is applied to rigid objects, we further enforce the rigidity loss over the pairs of matched pieces. Let X,j R. R.N.N,, N, be the submatrix of X that indicates the likelihood between the fracture points on the piece P; and Py. For fracture point p in P., Xiz matches it to point p' computed as the weighted average of all the matchable points in Pj:
(D)
p'
(p)(
where (x) E RIXN, is the row in Xi that contains matching likelihood from p to Pj. We optimize the transformation Tiy from P, to P, by minimizing the weighted mean squared error between the matched fracture points
Raj. Ey = argmin
Σ
(1A1(()))11 +t -11122
(10)
R,t
PEP.
where Reg, bi are the rotation and translation part of Tip and R(-) is the rotation operator. (10) has closed-form solution:
EpEr, [1xx)())) (R(p) -v)
(U, SVD
Σ
Rog UV,, E.
(11)
Eper, (Xis(p))l
PEP:
The rigidity loss is computed as:
Crs
Σ
Rij, where Rij
Σ (1 + Eoj -p'll2-
(12)
PEP:

---

To avoid numerical instability and improve efficiency in (11) during training, we compute Tiy in each iteration without involving back- prckpppppaattion and only set X1, the learnable variable. The loss for training composed as:
C = Cohe + 8L + 7Lrgg.
(13)
Multi-part Matching With the doubly-stochastic matrix X, computing multi-part matching results becomes straightforward. Given the fact that the primal and dual feature represents the surface looked from a different viewpoint, they should be distinct and not assigned good confidence for matching.
Therefore, we can directly pass X to Hungarian layer [37]. This yields a binary permutation matrix X, representing bipartite matching among all fracture points:
X Hungarian(X)
(14)
As mentioned, the resulting matching matrix X, along with the affinity matrix M and soft matching matrix X, captures the local geometry similarity across different pieces. Additionally, they quantify the confidence level of how well the pieces fit together. This information enables accurate positioning and alignment of the fractured components.
3.4 Global Fracture Alignment
To efficiently and robustly recover the global poses of each fractured piece of object O we adopt two-step pipeline based on the bipartite matching X {0, 1 predicted by the multipart matching module.
In the first step, we compute pairwise transformations between each pair of pieces (P;: Py) to remove outlier matches. Let X;, E N denote the submatrix of X, which establishes the correspon- dences between the fracture points from P; and Py. We apply the RANSAC algorithm [38] to compute the transformation T.j from P., P3 and Xij. This step ensures reliable pairwise transformations for subsequent alignment.
In the second step. we perform robust global alignment using the computed pairwise transformations. We model the global alignment configuration as a factor graph [39], denoted as G (V.E), where the global poses 7; are optimized on the vertices V and the pairwise transformations Fi are set as constraints on the edges E. The information matrix [e) over the edge e = (i,j) is set to |X,, 1-2 16. with le being the 6 x 6 identity matrix. We employ Shonan averaging [40]. state-of-the-art 6-DoF global alignment method, to optimize global poses over G. The global alignment method takes pairwise transformations between potentially adjacent pieces input and outputs pose for each piece up to a global transformation over the coordinate system. To avoid such an ambiguity, we anchor the coordinate system to the canonical one of the largest pieces in our experiment and measure the errors over all other pieces of the fractured object. Further technical details of this module can be found in the references [41, 42, 43]. Please refer to those sources for more information, as it is beyond the scope of our main contribution.
4 Experiments
We demonstrate the effectiveness of our 3D fracture assembly framework through experimental evaluations on a large-scale fracture assembly dataset. Our results show that Jigsaw significantly outperforms the baseline methods both quantitatively and qualitatively. Additionally, we conduct ablation studies analyze the contributions of each module in our framework. All experiments are conducted on a Linux workstation with 4 Tesla V V100-SXM2-32GB GPUs, Intel(R) Xcon(R) CPU ES-2698 v4 @ 2.20GHz CPUs, and 480GB Memory. Table lists the configurations of the parameters in our experiments. A comparison of the training/testing time is included in Appendix D.
4.1 Protocols
Dataset. We leverage the Breaking Bad dataset (Sellan al., 2022). novel data set of multiple fracture assemblies featuring synthetic physical breaking patterns. Our training was on everyday subset and the testing was on both everyday and artifact subsets for a fair comparison with
7

---

Table 1: The detailed experiment parameters. We follow the parameters provided in [16, 33, 4] to reproduce their results. Training, model, and dataset parameters are included.
Jigsaw
Predatod[16]
DGL [33]
LSTM [33]
Global [33]
description
epoch
250
200
200
200
200
training epochs
bs
16
32
batch
Ir
0.001
0.01
0.001
0.001
0.001
learning rate
optimizer
Adam
SGI>
Adam
Adam
optimizer during training
scheduler
Cosine
Exponential
Cosine le-5
Cosine le-5
Cosine
learning rate scheduler
min_Ir Ir
le-5
le-5
minimum learning rate for Cosine scheduler
segmentation loss ratio
Bie
1.0/ 10
matching loss ratio change at epoch rigidity loss ratio change at epoch temperature paramter for affinity
e/
1.0 1,0/200 200
0.05
sampling by points (N)
object
piece
piece 1000/p
piece 1000/p
piece 1000/p
sampling strategies for point clouds points sampled per object (/o) or piece (fp) segmentation ground truth label threshold
5000/o
800/p 0.025
1
0.025
baselines. Everyday consists of 498 models and 41,754 fracture patterns, and is split into a training set (34,075 fracture patterns from 407 objects) and a test set (7,679 fracture patterns from 91 objects). Artifact consists of 3651 fracture patterns from 40 uncategorized objects. The average diameter of the objects in the training and testing dataset was 0.8. Categorical information was concealed during all the experiments.
The input uniformly sampled N points on the surface of the object as a point cloud. In composing the point cloud, two sampling strategies are considered, sampling by piece and object. Sampling by piece, as used in the Breaking Bad benchmark [4], equally samples points in each fragment, which would cause point density imbalance. To better reflect real-world scanning. we opt for "sampling by object", sampling a fixed number of points in each object, while sampling points for each fragment based on its surface area. We ensure at least 30 points per fragment for multi-part matching. Additional processing details can be found in the Appendix C.I.
Baseline Approaches. As learning to assemble multiple fractured objects is novel task, there is a dearth of existing methods for direct comparison. Therefore, we have selected a set of established methods that primarily address similar challenges 3D alignment and assembly, and employ them as baselines for comparative evaluation. Global [26, 10] extracts per-piece features, which are combined with global shape descriptors regress the pose of each piece in one shot. LSTM applies a bidirectional LSTM similar to [27] and estimates the pose of each piece in sequential style. DGL [8] a state-of-the-ar approach for the assembly task of parts and can also be adapted the assembly task of fractures. It leverages an iterative graph neural network to reason about the relationships among pieces. Additionally, we take PREDATOR [16] a minor competitor, which is state-of- the-art approach for 3D registration with low overlapping rates. It employs a multitask transformer estimate the overlap between pairs of pieces in conjunction with their relative pose. All baseline approaches use textureless point clouds as input and are trained under the everyday object subset of the Breaking Bad dataset.
Evaluation Metrics. We adopt the same evaluation schemes for 3D assembly tasks used in [4, 14, 10]. We report the mean absolute error (MAF) and the root mean square error (RMSE) for the rotations and translations of the estimated global poses. Additionally, we include the part accuracy (PA) metric proposed in [33], which measures the ratio of perfectly assembled pieces based on an average Chamfer distance of less than 0.01 for each point between the assembly results and the ground truth.
4.2 Performance
Overall Performance. We report the performance of Jigsaw and all the baseline methods over the 3D fracture assembly task on both everyday and artifact object subsets. An overall quantitative comparison of the evaluation metrics is presented in Table 2. For PREDATOR we apply the evaluation metrics on pairwise transformations since it only supports overlapped pairs of pieces input.
Jigsaw significantly outperformed all the baselines across all the evaluation metrics. On everyday, we achieved an average rotation error of 36.3" over all pieces, which was a 46% reduction compared to the top-pertee hop-performing baseline method, DGL. Our transformation error was also outstanding, with an average of 8.7 x 10 surpassing the best baseline method with an error of 11.8 x 10 We
8

---

Table Quantitative results of baseline methods and Jigsaw on the Breaking Bad dataset.
Method
Original Task
RMSE (R)
MAE degree
RMSF(T)
1 X10
PA
degroc
x10
%
Results everyday abjecct
Global [26. [0]
assembly
82.4
69.7
14.8
11.8
21.8
[27)
assembly
72.7
DGL [8]
ussembly
80.6
67.8
15.8
12.5
23.9
(GL//5000))
assembly
15.4
PREDATOR [16]
registration
82.8
71.2
12.5
10.2
1.3
Jigsaw (Ours)
assembly
42.3
36.3
10.7
8.7
Resulis the artifact abject subsel.
Global [26, 10]
assembly
86.9
75.3
17.5
14.5
5.6
LSTM [27)
assembly
85.6
74.1
18.6
15.2
4.5
DGL [8]
ussembbl
86.3
74.3
18.0
14.9
9.6
PREDATOR
registration
86.0
74.8
13.4
10.9
Jigsaw (Ous)
assembly
52.4
45.4
22.2
19.3
45.6
Everyday
Artifact
Grond Truth
Global
LSTM
DGL
Jigsav
Figure 3: Qualitative results of baseline methods and Jigsaw on the Breaking Bad dataset (mesh used only for visualization). The coordinate system of the green piece is the global coordinate system. Better viewing with color and zooming in.
successfully restored over 57% of the pieces to their original pose while baseline methods restored only half of us. We also show that change in sampling strategy has had no impact on the outcome of baseline methods through sampling by object version of DGL. Our method exhibited remarkable generalizability on artifact, showing a slight increase of 10.1 in average rotation error and achieving part accuracy of 44%. In contrast, all baseline methods completely failed to handle new categories of models that were not present the training dataset. The PREDATOR model failed to accurately identify the pairwise overlap region, resulting in erroncous predictions of complete overlap between the two pieces. While this may seem to yield a small translation error, it failed to perform adequately in the assembly scenario. Hence, exclude it from further discussion.
Detailed distribution of each metric. Fig. presents a comprehensive analysis of each metric for all assembly models. Regarding rotation and translation metrics, larger surface area under the curve indicates better performance. Jigsaw exhibits significantly larger number of samples with small rotation and translation errors, as illustrated in the figure. Although Jigsaw reports a slightly larger translation error than the baseline models on the artifact object subset, the distribution results reveal that this is primarily due to few extreme cases, leading to an undesirable accumulation of the translation error. terms of part accuracy, a larger surface area above the curve signifies superior results. Notably, Jigsaw demonstrates higher proportion of correctly positioned pieces.
Visualization. The qualitative comparison of different models is shown in Fig. 3. We plot the recovered object from different models. It demonstrates that our model could find the position of the fracture pieces and recover good pose of each piece. Visualization of objects from different categories also implics our model has good generalization ability and doesn't 't require object-level information assembly guidance.

---

15
Figure 4: The discretized distributions of metrics evaluated over the test data of everyday (up) and artifact (bottom) object subset the Breaking Bad dataset.
Ablation Study. We conducted ablation study to assess the effectiveness of each component in our Jigsaw model. The baseline model consists of segmentation module and multi-par part matching module, trained separately using separate backbones and without incorporating transformers. The plain joint training model was trained using the same loss function as the Jigsaw model, but without using two transformer layers. The effectiveness of the rigidity loss regularization was evaluated over the complete network with both joint training and transformer layers.
As presented in Table 3, the joint training model outperformed the baseline in transformation met- rics. This is due to a more favorable initializa- tion for the matching module achieved through pretraining the backbone with the segmentation module. Joint training also aided the matching module's 's convergence to better optimum. Atten-
Table 3: Ablation study of Jigsaw.
Components
RMSE (8]
MAE (K)
RMSE (T)
MAE (T)
PA
Joint Training
Attention
Rigidity
degree
degree
x10-2
x10-2
%
$1.9
44.6
32.4
27.2
45.7
51.7
44.4
12.7
10.2
47.0
42,3
36.3
10.9
8.9
57.2
tion layers significantly reduced rotation errors:
42.3
36.3
10.7
8.7
57.3
the self-attention layer embedded geometric information into point features, while the cross-attention layer improved multi-pan matching by incorporating information from other parts. Applying rigidity loss as a refinement additionally brought an improvement of x 10-3 in translation. Even the baseline method outperformed previous works, highlighting the effectiveness of our segmentation and multi-pan part matching modules.
Conclusion and Limitations
In this study, we propose a novel geometry-aware framework to address the 3D fracture assembly task. Our approach involves joint learning model that effectively extracts local geometry features from the backbone and enables the simultaneous learning of piecewise fracture surface segmentation and global fracture point matching. Experimental evaluations and thorough analyses substantiate the exceptional performance of our method in terms of accurately recovering piece poses and restoring the original object. Furthermore, we demonstrate the generalizability of our approach across objects of diverse categories. One limitation of our approach lies in that as the number fractured pieces increases, the accuracy of pose estimation tends to decrease due to potential ambiguities in fracture matching caused by local geometric features. Future research directions may include exploring matching pruning techniques for pairwise pose estimation to prevent penetration between fractured pieces and enforcing smoothness constraints to enhance pose accuracy.
Acknowledgements
We would like to acknowledge NSF IIS 204767777 HDR- 193499,,, and CCF-2019844.
10

---

References
[1] Georgios Papaioannou and Evaggelia-Aggelkki Karabassi. On the automatic assemblage of arbitrary broken solid artefacts. Image and Vision Computing, 21(5):401-422 2003.
[2] Qi-Xing Huang. Simon Flöry, Natasha Gelfand, Michael Hofer, and Helmut Pottmann. Reassembling fractured objects by geometric matching. In ACM SIGGRAPH 2006 Papers, SIGGRAPH '06, pages 569-578, New York, NY, USA, 2006. ACM.
[3] Andy Zeng, Shuran Song, Matthias Nicßner, Matthew Fisher, Jianxiong Xiao, and Funkhouser. 3dmatch: Learning the matching of local 3d geometry in range seans. In CVPR, volume 1, page 4, 2017.
[4] Silvia Sellán, Yun-Chun Chen, Ziyi Wu, Animesh Garg, and Alec Jacobson. Breaking bad: dataset for geometric fracture and reassembly. In Thirty-sixth Conference on Neural Information Processing Systems Datasels and Benchmarks Truck, 2022.
[5] Kaichun Mo, Shilin Zhu, Angel Chang. Li Yi, Subarna Tripathi, Leonidas J Guibas, and Hao Su. PartNet: A Large-scale Benchmark for Fine-grained and Hierarchical Part-level 3D Object Understanding. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pages 909-918, 2019.
[6] Li Yi, Vladimir G Kim, Duygu Ceylan, I-Chao Shen, Mengyan Yan, Hao Su, Cewu Lu, Qixing Huang, Alla Sheffer, and Leonidas Guibas. A scalable active framework for region annotation in 3d shape collections. ACM Transactions Graphics (ToG), 35(6): 12, 2016.
[7] Abhinav Narayan Harish, Rajendra Nagar, and Shanmuganathan Raman. Rgl-net: A recurrent graph learning framework for progressive part assembly. In 2022 IEEE/CVF Winter Conference on Applications of Computer Vision (WACV), pages 647-656, IEEE, 2022.
[8] Guanqi Zhan, Qingnan Fan, Kaichun Mo, Lin Shao, Baoquan Chen, Leonidas J Guibas, Hao Dong, al. Generative 3d part assembly via dynamic graph learning. Advances Neural Information Processing Systems, 33:6315 15-6266, 2020.
[9] Benjamin Jones, Dalton Hildreth, Duowen Chen, Ilya Baran, Vladimir G Kim, and Adriana Schulz. Automate: A dataset and learning approach for automatic mating of cad assemblies. ACM Transactions on Graphics (TOG), 40(6):1-118 18, 2021.
[10] Yiehen Li, Kaichun Mo, Lin Shao, Minhyuk Sung, and Leonidas Guibas. Learning 3d part assembly from single image. In Computer Vision-ECCV 2020: 16th European Conference, Glaxgow, UK, August 23-28, 2020, Proceedings, Part VI 16, pages 664-6666666 682. Springer, 2020.
[11] Karl DD Willis, Pradeep Kumar Jayaraman, Hang Chu, Yunsheng Tian, Yifei Li, Daniele Grandi Aditya Sanghi. Linh Tran. Joseph G Lambourne. Armando Solar-Lezama, et al. Joinable: Learning bottom-up assembly of parametric cad joints. In Proceedings of the IEEE/CVF Conference Computer Vision and Pattern Recognition, pages 15849-15860, 2022.
[12] Kangxue Yin, Zhiqin Chen, Siddhartha Chaudhuri. Matthew Fisher, Vladimir G Kim, and Hao Zhang Coalesce: Component assembly by learning synthesize connections. In 2020 International Conference on 3D Vision (3DV), pages 61-70. JEEE, 2020.
[13] Corey Frlcr-Franklln, Benedict Brown, Tim Weyrich, Thomas Funkhouser, and Szymon Rusinkiewicz. Multi-feature matching of fresco fragments. ACM Transactions on Graphics (TOG), 19 2010.
[14] Yun-Chun Chen, Haoda Li, Dylan Turpin, Alec Jacobson, and Animesh Garg. Neural shape mating: Self- supervised object assembly with adversarial shape priors. Proceedings of the IEEE/CVF Conference on Computer Vision and Puttern Recognition, pages 12724-12733, 2022.
[15] Thomas Funkhouser, Hijung Shin, Corey Toler-Franklin, Antonio Gareía Castañeda, Benedict Brown, David Dobkin, Szymon Rusinkiewicz, and Tim Weyrich. Learning how to match fresco fragments. Journal on Computing and Cultural Heritage (JOCCH), 4(2):1-13, 2011.
[16] Shengyu Huang, Zan Gojeic, Mikhail Usvyatsov, Andreas Wieser, and Konrad Schindler. Predator: Registration of 3d point clouds with low overlap. In Proceedings of the IEEE/CVF Conference on computer vision and pattern recognition, pages
2021.
[17] Georgios Papaioannou. E-A Karabassi. and Theoharis Theoharis. Virtual archaeologist: Assembling the past. IEEE Computer Graphics and Applications, 21(2):53-59 2001.
[18] David Koller and Mare Levoy. Computer aided reconstruction and new matches in the forma urbis romae. Computer-aides aided Reconstruction and new Matches in The Forma Urbis Romae, pages 103- 2006.
11

---

[19] Paul-Edouard Sarlin, Daniel DeTone, Tomasz Malisiewicz, and Andrew Rabinovich. SuperGlue: Learning feature matching with graph neural networks. In CVPR, 2020.
[20] Jiaming Sun, Zehong Shen, Yuang Wang, Hujun Bao, and Xiaowei Zhou. LoFTR: Detector-free local feature matching with transformers. CVPR, 2021.
[21] Andrei Zanfir and Cristian Sminchisescu. Deep learning of graph matching. In CVPR, pages 2684-2693, 2018.
[22] Runzhong Wang. Junchi Yan. and Xiaokang Yang.
Learning combinatorial embedding networks for deep graph matching. In IEEE International Conference on Computer Vision, pages 3056-3065, 2019.
[23] Michal Rolínek, Paul Swoboda, Dominik Zietlow, Anselm Paulus, Vít Musil, and Georg Martius. Deep graph matching via blackbox differentiation of combinatorial solvers. In European Conference on Computer Vision, pages 407 424. Springer, 2020.
[24] Runzhong Wang, Junchi Yan, and Xiaokang Yang. Neural graph matching network: Learning lawler's quadratic assignment problem with extension to hypergraph and multiple- graph matching. IEEE Transac- tions on Pattern Analysis and Machine Intelligence, 2021.
[25] Zetian Jiang, Jiaxin La, Tianzhe Wang, and Junchi Yan. Learning universe model for partial matching networks over multiple graphs. arXiv preprint arXiv:2210. 100.14444 2022.
[26] Nadav Schor, Oren Katzir, Hao Zhang, and Daniel Cohen Or. Componet: Learning to generate the unseen by part synthesis and composition. In Proceedings of the IEEF/CVF International Conference on Computer Vision (ICCV), October 2019.
[27] Rundi Wu, Yixin Zhuang, Kai Xu, Hao Zhang, and Baoquan Chen. Pq-net: A generative part seq2seq network for 3d shapes. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 829-838, 2020.
[28] Anastasia Dubrovina, Fei Xia, Panos Achlioptas, Mira Shalah, Raphael Groscot, and Leonidas J. Guibas. Composite shape modeling via latent space factorization. In Proceedings of the IEEE/CVF International Conference on Computer Vision (ICCV), October 2019.
[29] Lifa Zhu, Dongrui Liu, Changwei Lin, Rui Yan. Francisco Gómez-Fernánde Fernández. Ninghua Yang, and Ziyong Feng. Point cloud registration using representative overlapping points, arXiv preprint arXiv:2107.02583 2021.
[30] Zheng Qin, Hao Yu, Changjian Wang, Yulan Guo, Yuxing Peng, and Kai Xu. Geometric transformer for fast and robust point cloud registration. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 11143-11152, 2022.
[31] Charles Ruizhongtai Qi, Li Yi, Hao Su, and Lconidas J Guibas. Pointnct++: Deep hicrarchical fcature learning on point sets a metric space. In Advances in neural information procexsing systems, pages 5099-5108, 2017.
[32] Marco Cuturi. Sinkhorn distances: Lightspeed computation of optimal transport. Advances in neural information processing systems, 26, 2013.
[33] Jialei Huang. Guanqi Zhan, Qingnan Fan, Kaichun Mo, Lin Shao, Baoquan Chen, Leonidas Guibas, and Hao Dong. Generative 3d part assembly via dynamic graph learning. Advances in Neural Information Processing Systems (NeurIPS), 2020.
[34] Charles R Qi. Hao Su, Kaichun Mo, and Leonidas J Guibas. Pointnet: classification and segmentation. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pages 652-660. 6660, 2017.
Deep learning on point sets for 3d
[35] Hengshuang Zhao, Li Jiang, Jiaya Jia, Philip HS Torr. and Vladlen Koltun. Point transformer. In Proceedings of the IEFF/CVF International Conference on Computer Vision, pages 16259-16268, 2021.
[36] Andrei Zanfir and Cristian Sminchisescu. Deep learning of graph matching, In Proceedings of the IEEE conference computer vision and pattern recognition, pages 2684 2204-2663,22 2018
[37] Martin A. Fischler and Robert C. Bolles. Random sample consensus: A paradigm for model fitting with applications to image analysis and automated cartography, Commun. ACM, 24(6))381-38) jun 1981.
[38] Sungjoon Choi, Qian- Yi Zhou, and Vladlen Koltun. Robust reconstruction of indoor scenes. In CVPR, pages 5556 5565, IEEE Computer Society, 2015.
12

---

[39] Daphne Koller and Nir Friedman. Probubilistic graphical models: principles and techniques. MIT press. 2009.
[40] Frank Dellaert, David M Rosen, Jing Wu, Robert Mahony, and Luca Carlone, Shonan rotation averaging: Global optimality by surfing so(n). In European Conference Computer Vision, pages 292-308. Springer, 2020.
[41] Qian-Yi Yi Zhou, Jaesik Park, and Vladlen Koltun. Open3d: A modern library for 3d data processing. arXiv preprint arXiv: (0t.0///00.994 2018.
[42] Frank Dellaert and GTSAM Contributors, borglab/gtsam, May 2022.
[43] Frank Dellaert and Michael Kaess. Factor Graphs for Robot Perception. Foundations and Trends in Robotics, Vol. 6, 2017.
13

---

Appendix
A Details of Network Architecture
We provide additional information about our network ar- chitecture. The backbone of our network is based on standard PyTorch implementation of multi-scale grouping PointNet++. It is followed by single MLP layer that ex- tracts a D-dimensional feature for each point. In our imple- mentation, we set D = 128.
D
desentian
The detailed structure of the transformer module can be seen in Fig. 5. The self-attention layer follows the official code of the point transformer layer, while the cross-attention layer employs a multi-head attention mechanism with position- wise feed-forward networks. In configuration, we set the number of attention heads to h 8, the head dimen- sion to d4 16, and use k-nearest neighbor sampling with 16. The inner layer of the forward network has dimensionality of d; 256.
IR's
Vil Appregation
Pouitoon Konsort
(6) Attention
Atteriion
Figure 5: Detailed structure of the at- tention layers.
Regarding the primal-dual descriptor, set its feature dimension 256.
B Analysis on Primal-dual Descriptor
201
Figure 6: Comparison between the doubly-stochastic soft matching matrix X computed by different types of descriptors.
We conducted analysis to compare the results obtained using the primal-dual descriptor and the standard single-view descriptor. Before passing the affinity metric computed from the single-view descriptor the Sinkhorn layer, common practice is mask the diagonal sub-matrix to prevent self-self alignment. For clarity, we refer to the masked and unmasked versions the primal-dual and single-view descriptors denote whether this masking operation has been applied
In Fig. 6, we provide visualization of the doubly-stochastic matrix X computed by the Sinkhorn algorithm using four types of descriptors: (1) the unmasked primal-dual descriptor, (2) the masked primal-dual descriptor, (3) the unmasked single-view descriptor, and (4) the masked single-view descriptor. The visualization depicts an example object with 3 pieces and 58 fracture points. The X obtained from the primal-dual descriptor clearly demonstrates its ability differentiate between the two viewpoints and avoid aligning a point to itself. In contrast, the single-view descriptor exhibits diagonal peak, resulting in self-self alignment. Even when the diagonal sub-matrix is masked to prevent self-self alignment, the soft matching computed from the single-view descriptor is less distinct compared that computed from the primal-dual descriptor.
Furthermore, we conducted evaluation on the everyday object subset of Breaking Bad dataset, using the unmasked primal-dual descriptor and the masked primal-duse descriptor. The results indicate that the unmasked primal-dual descriptor achieves comparable performance to the masked one, with only slight difference: decrease of 0..2% in rotation error (MAE(R)) and improvement of 0.7 x 10-2 in translation error (MAE(T)). We attribute this minor difference to the fine characteristics of the primal-due descriptor that prevent self-self alignment, thus resulting in consistent performance.
14

---

C Experiment Details
C.1 Dataset
We leverage Breaking Bad dataset [4] to evaluate our method and all baseline methods, As we have stated in section 4, the training of all methods were on the training subset of everyday, and testing were on the testing subset of both everyday and artifact object. Each object within the dataset has been fragmented into pieces, represented by triangle meshes, and all the pieces are in their original poses. By assembling these pieces together directly, the surface of the original object is seamlessly restored. The triangle meshes of the pieces solely consist of exterior faces that are visible from the outside.
In generating the point cloud for our experiments, we employ two sampling strategies for the compared methods: "sampling by piece" and "sampling by object". The "sampling by piece" strategy, originally utilized in the Breaking Bad benchmark [4], involves sampling equal number of points within each fragment. However, this approach leads to excessively dense sampling on small fragments, while larger fragments suffer from sparser point distributions, resulting in an imbalance in the representation of point density across fragments. To better mirror real-world scanning technology, we opt for the "sampling by object" strategy. With this approach, we sample fixed number of points within each object, and the number of sampled points for each fragment determined based on its surface area. In essence, smaller fragments receive fewer points, whereas larger ones receive more, ensuring a more realistic representation. Additionally, we ensure that each fragment sampled with minimum of 30 points to include even the tiniest fragments in multi-part matching. Detailed parameter settings can be found in Table Our analysis of the average fragment numbers and experiments conducted using DGL [8] indicate that the choice between the two sampling strategies has negligible impact on the baseline performance.
For each sampled point p on P., its fracture label cp is determined by the distance from p to its nearest neighbor q among points from all other pieces:
min ||p 0112 <
(15)
9E0%P
where 17 set to 0.02 for all the objects. The ground-truth matching point q of fracture point ÿ P; set to
q = argmin || 1'112-
(16)
Figure 7: An example of surface segmen- tation over the point cloud of bottle bro- ken into 7 major pieces. Fracture points are marked with green in the predicted result of our method (left) and red in the ground truth (right).
where Ô, P. denotes the set of fracture points in O and P.. During the training process, Cp was applied to segment the fracture points and during testing the predicted ēp was used instead. An example shown in Fig. 7.
After the ground truth labeling and matching were com- puted based on the pieces at their original pose, all pieces were recentered to the origin and random rotation was applied to each piece.
To ensure fair comparison with bascline methods [26, 10, 27, 8], we adopted the same implemen- tation as of the benchmark code of [4], which sampled the same number of point each piece. For PREDATOR, we applied the same sample routine as the baseline methods, and we paired the adjacent pieces for training and testing.
C.2 Evaluation Metrics
The mean absolute error MAE(R) and square-rooted mean squared error RMSE(R) of rotation were computed as
MAE(R) = 3.1 111 - R99 112,
(17)
RMSE(R) =
v3 11 R
15

---

Table 4: Comparison of training and influence time with Tesla V100 GPUs. For Jigsaw, the time used in the forward is 1.30s/batch and the rest of the time is used on Hungarian and global alignment.
Jigsaw
11e
DGL [33]
LSTM [33]
Global [33]
Training Time GPUs Influence Speed (s/batch) / batch size
120H 4
96H / 1.28 / 28
11H/1 2,46/ 32
16H
21H
7.67(1.30) 8
1.63 /
1.65 32
Table 5: Detailed quantitative results of Jigsaw on Breaking Bad dataset (mean and STD by 3 runs).
Method
RMSE (R)
MAE (R)
RMSH(T)
MAE
PA f
degree
degree
x10
x10
%
Results on the everyday object subset.
Jigsaw
42.3 +: 0.03
36.3 + 0.06
10.7 + 0.02
8.7 0.02
57.3 0.012
Results on the artifact object subset.
Jigsaw
52.4 + 0.09
45.4 - 0.14
22.2 - 0.05
19.3 + 0.04
45.6 = 0.015
for each piece, where i and Rs were predicted rotation and ground-truth rotation represented in Euler angles. The error over each object was computed as the mean error of all the pieces and the total error was the mean error of all the object. For translation, MAE(T) and RMSE(T) were computed in same way
MAE(T) 111 t 19'111
(18)
RMSE(T) = v3
11 t 1|2
where i and got were the predicted translation and ground-truth translation.
D Implementation Details
D.1 Additional Configuration of parameters
For PREDATOR [16], we carefully follow the parameters presented in its open sourced code, and the threshold for overlap recall add saliency loss set be 0.3. For Jigsaw, we training with only the segmentation loss. We add matching loss after first 10 epochs, and rigidity loss after 200 epochs.
D.2 Running time
Our framework is implemented in Pytorch. All methods are trained over Tesla V100-SXM2-32GB GPUs and distributed data parallel strategy is applied for multi-GPU training. The training and influence time for Jigsaw and baseline methods are shown in Table 4.
E
Additional Results
E.1
Detailed Quantitative Results
Fig. 8 gives detailed distribution of each error metric with respect the number of fractured pieces. Our approach exhibits a clear advantage in each of the metrics evaluated, and this advantage becomes more pronounced as the number of pieces decreases.
In addition, we present the mean and standard deviation (STD) of three runs of our method in Table 5. These values demonstrate the stability of our method across different random seeds and variations in the sampled point cloud, The consistent performance and low standard deviation affirm the robustness of our proposed approach.
16

---

LSTH
14
PICTAE
DO. VODP
Places
LSTH
DGL
=
=
11
14
16
16
13
20
DO.
=
Peccss
LSTT4
CAS.
13 14 18 17 17
Figure 8: Detailed analysis of the quantitative results obtained by each method on the everyday object subset. We provide results for each metric, categorized by the number of pieces involved. The shadowed area in the figures represents the distribution of each metric across the corresponding number of pieces. The horizontal line indicates the mean value of the metric for that specific number of pieces. Different hues represent different methods.
E.2 More Qualitative Results
We collect additional visualizations of the assembled objects in Fig, 9. Even with large number of pieces assemble, our method remains robust in restoring major pieces their original pose.
E.3 Additional Registration Baseline
We have received advice to consider GeoTransformer [30] as statc-of-the-arr baseline for low overlap registration. However, training GroTransfrrmer proves to be exceedingly slow (about 15 days over 4 V100 GPUs), primarily due to the significant scale Breaking- -Bad in comparison to the 3D registration datasets on which it was originally tested. On pairwise transformations Transforrre achieves MAE(R)=72.Adegree)
2), RMSE(R)=84. -:deee.), RMMS(()=14.3(x1))
1.E(T)=11.6(X 10 3), PA=3.1% %.. These findings align with the claims we' ve put forth in our paper, and we believe that presenting the full results of PREDATOR adequately reflects how registration baselines perform on the assembly task.
17

---

Everyday
Ground Truth
Global
LSTM
DGL
Jigsaw
Artifact
Ground Truth
Global
LSTM
DGL.
Jigsaw
Figure 9: More visualization of assembly results.
18


Reviews:
[{"review_id": "HXeKu3FuyN", "reviewer": "Reviewer_LoVE", "summary": "This work provides a Joint Learning of Segmentation and Alignment Framework(Jigsaw) framework for 3D fracture assembly task. Experiments are evaluted on Breaking Bad dataset and show better performance than other existing methods.", "questions": "The running time analysis of the proposed method or the comparison to other methods is missing. How about this metric?", "limitations": "In the experiments, evaluation on extra dataset rather than the Breaking Bad dataset is much desired.", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "1. The proposed approach applies a joint learning framework specific for multi-part fracture assembly, which appends an attention-based \nnetwork module to capture local geometric information. To further capture viewpoint-dependent features for surface matching, the proposed method applies the primal-dual descriptor to achieve better matching.\n\n2. The authors incorporate fracture point segmentation to capture intrinsic features and introduce a multi-part matching approach\nfor internal automatic piece positioning, with global pose alignment capability. \n\n3. The proposed method shows better results than baseline methods evaluated on the multi-part assembly Breaking Bad dataset. The ablation analysis also validates the efficacy of the proposed network structure.", "weaknesses": "1.  The motivation of using primal-dual descriptor seems not sufficient, it would be better to append a more description of it. \n2.  The ablation study part is considered a bit insufficient. e.g. The authors already mention matching loss and rigidity loss. The related ablation analysis about the impact of segmentation loss, matching loss and rigidity loss is missing.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thanks for your time and valuable comments. Below we respond to your comments.\n\n> The motivation of using primal-dual descriptor\n\n**A**: The motivation is to capture the essence of the complementary geometry of two broken pieces. We provide two examples (Fig. 2 in the paper, Fig. 1 of the authors’ response document) where a conventional feature extractor falls short while the primal-dual descriptor excels. Please refer to the general response (II) for a more detailed explanation. We will add a more detailed explanation of the motivation of using primal-dual descriptors to the manuscript.\n\n> The ablation study part is considered a bit insufficient. e.g. The authors already mention matching loss and rigidity loss. The related ablation analysis about the impact of segmentation loss, matching loss and rigidity loss is missing.\n\n**A**: Both the segmentation loss and matching loss play essential roles in ensuring the assembly task's success. Omitting either of these components would render the method ineffective. Therefore, we compared the outcomes of applying these two loss functions jointly versus independently. We additionally investigate the integration of the rigidity loss. The results reveal a enhancement of $0.2 (\\times 10^{-2})$ in the RMSE(T) metric.\n\n> The running time analysis of the proposed method or the comparison to other methods is missing.\n\n**A**: Please refer to the general response (I) for the running time of all the methods."}, {"review_id": "hDbj1bt3zp", "reviewer": "Reviewer_Pfa7", "summary": "This paper proposes Jigsaw, a novel framework to assemble 3D fragments of a shape to complete the whole 3D shape. \nJigsaw first segments the fracture surface, since only the points on the fracture surface are involved in the physical assemble between fragments. Jigsaw then performs multi-part matching to find correspondences among the fracture surface points, which is finally used to recover the global pose of each fragment via global alignment. \nJigsaw exhibits state-of-the-art performance on the Breaking Bad dataset.\nThe authors propose Jigsaw to be the first learning-based method to assemble 3D fractures over **multiple** pieces.", "questions": "Please refer to the above weaknesses section. It would be appreciated if the authors could include additional details to improve the self-containedness of the paper, and to clear any potential misunderstandings. ", "limitations": "The limitations of Jigsaw have been discussed by the authors. ", "rating": 5, "confidence": 5, "soundness": 4, "presentation": 3, "contribution": 4, "strengths": "* First learning based method to tackle the task of multi-part assembly of 3D fragments.\n\n* Well-motivated, novel learning pipeline to facilitate multi-part assembly of 3D fragments. \n\n* Strong performances on the Breaking Bad dataset. Considering that a two-fragment assembly problem can also be seen as a zero-overlap (or surface-overlap) point cloud registration, the baseline methods in the table are also appropriate.\n\n", "weaknesses": "* Lacks self-containedness. While it is understandable that Shonan-averaging was used as the off-the-shelf global alignment method, but the paper does not go on to explain how this global alignment functions at the minimum, nor how the global pose values from Shonan-averaging is used for evaluation against the given ground-truth global pose values.\n\n* Jigsaw samples a fixed number of points across all the fragments of a single shape. In that case, Jigsaw is expected to fail given a particularly small fragment, from which a very small number of fragments would be sampled. This may result in not a single point on the fracture surface being sampled, and this lacks emphasis in the manuscript.\n\n* The manuscript lacks explanation on how the primal-dual descriptor works. The authors state that their 'system learns surface features that capture the characteristics of a local surface from both directions' - which lacks the evidence the substantiate this claim. Is it the **aim** of primal-dual descriptors to function as mentioned, or is there a theoretical evidence to prove that the primal-dual descriptors actually captures such characteristics?\n\n* While the idea of surface segmentation to narrow down the points to consider is viable, it is expected that the fracture surface is rather easy to segment due to its irregularity in comparison to other parts of the shape. How would the proposed method perform if the fracture is not irregular, but of regular cuts (e.g., straight lines, smooth curves) such as in Neural Shape Mating? [1]\n\n[1] YC Chen et al., Neural Shape Mating: Self-Supervised Object Assembly with Adversarial Shape Priors, CVPR 2022", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thanks for your time and valuable comments. Below we respond to your comments and hope we can clear the potential misunderstandings.\n\n\n> Lacks self-containedness. While it is understandable that Shonan-averaging was used as the off-the-shelf global alignment method, but the paper does not go on to explain how this global alignment functions at the minimum, nor how the global pose values from Shonan-averaging is used for evaluation against the given ground-truth global pose values.\n\n**A:** Shonan-averaging computes a global alignment based on pairwise transformations and outputs a transformation matrix that implies a global pose within a specific global coordinate system. The input organization for Shonan-averaging is detailed in Section 3.4. For assessing against the ground truth, we adopt the canonical coordinate system of the largest piece, termed the anchor piece, as the global frame of reference (i.e., the global pose of the largest piece is consistently set to identity), and then align all other pieces relatively. The error is quantified by contrasting the predicted and ground-truth relative transformations of all other pieces with respect to the anchor. We will incorporate this comparison procedure into section 3.4 for clarification.\n\n>  ... may result in not a single point on the fracture surface being sampled.\n\n**A:** We assure that each fragment is sampled with a minimum of 30 points to ensure compactness. We will add this explanation to our manuscript.\n\n> The manuscript lacks explanation on how the primal-dual descriptor works. The authors state that their 'system learns surface features that capture the characteristics of a local surface from both directions' - which lacks the evidence to substantiate this claim. Is it the aim of primal-dual descriptors to function as mentioned, or is there theoretical evidence to prove that the primal-dual descriptors actually capture such characteristics?\n\n**A:** In Figure 6, we show a comparative analysis between the primal-dual descriptor and the\nconventional single-view descriptor. Experimental results show that the primal-dual descriptor highly favored matches between fracture points with complementary local geometries compared to identical local geometries. Please refer to the general response for a more detailed explanation. \n\n\n> While the idea of surface segmentation to narrow down the points to consider is viable, it is expected that the fracture surface is rather easy to segment due to its irregularity in comparison to other parts of the shape. How would the proposed method perform if the fracture is not irregular, but of regular cuts (e.g., straight lines, smooth curves) such as in Neural Shape Mating? [1]\n\n**A:** Our framework is explicitly designed for fracture assembly scenarios involving objects fractured due to physical forces. The Breaking Bad dataset is generated through physical simulation where the typical fracture pattern is irregular and rugged. In instances where a smoother fracture pattern is introduced, we believe that prior knowledge of the object’s shape is necessary. A hypothetical scenario could involve cutting a pyramid into two tetrahedrons. In the absence of shape priors, it would be reasonable to match the base or sides of the two fragments together.\n"}, {"review_id": "lhOVkd3ljR", "reviewer": "Reviewer_qeDQ", "summary": "The authors present a learning framework for assembling physically broken 3D objects with point cloud observations. There are three components in the framework, surface segmentation to identify the fracture points, multi-parts matching to find correspondences among the fracture points, and the global alignment to recover the global poses of the pieces. The main contributions are the joint learning of segmentation and multi-parts matching and the design of the multi-parts matching including the primal-dual descriptor and the losses. The contributions are validated by superior performance on the Breaking Bad dataset compared with other methods.", "questions": "Does the generation of the segmentation labels require additional knowledge about the data compared to other methods? Although I don't think so, a clarification from the authors would be helpful.\n\nIs only one precise match at line 193 an assumption? The assumption of only one precise match may not be satisfied as the points sampled on neighboring pieces may be quite far and no matching can be done simply based on point clouds.\n\nAddressing the weaknesses listed above could change my opinion.", "limitations": "The authors deals with the accuracy drop of pose estimation when the number of fractured pieces increases.", "rating": 6, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "The task is new with just few methods already developed. This work may serve as a baseline for future work in this line.\n\nThe overall framework is clearly clarified and organised. The adding of self-attention and cross-attention is well justified and illustrated for the front-end feature extractor. The primal-dual descriptor is introduced in the multi-part matching part, which is new for this task to my knowledge. A further analysis of the primal-dual descriptor is also provided in the appendix to provide better understanding of the effect. \n\nThe experiments and results are also solid.", "weaknesses": "The absence of the statistics of the number of pieces in the dataset may lead to unfair comparison against the baseline methods as points are sampled at different level as shown in Table 3 in the appendix.\n\nA further analysis of the multi-parts matching w.r.t. the number of pieces could better show the effect of multi-part matching against pairwise matching.\n\nThe notations are sometimes a bit confused, especially for the tilde and hat. For example, the hat_P_i at line 258 seems to be not defined.\n\nThe experimental settings are not clear for all the involved methods. A table to list and compare the conditions and data/label requirement for each method would be helpful. \n\nThe introduction of the rigidity loss is not well justified and analyzed, especially it is only applied to the model training after 200 epochs.\n\nTexts in the figures are too small to recognize.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thanks for your time and valuable comments. Below we respond to your concerns and hope we can clarify some questions.\n\n> The absence of the statistics of the number of pieces in the dataset may lead to unfair comparison against the baseline methods as points are sampled at different levels as shown in Table 3 in the appendix.\n\n**A:** We include the distribution of the number of pieces in Figure 2 of the authors' response document. The median number of fractured pieces is $4$ and the mean is $6.1$. Given the different backbone structure of different methods, it is infeasible to use the same sampling strategy among all the methods. To ensure equitable assessment, we therefore uniformly sampled 5000 points from each object in our method, akin to a 5-piece scenario for baseline methods. We additionally provide a detailed error distribution of our method across all potential piece numbers in Fig. 8 of appendix. This visual clearly represents our method's superiority across all subset variations.\n\n> A further analysis of the multi-parts matching w.r.t. the number of pieces could better show the effect of multi-part matching against pairwise matching.\n\n**A:**\nFigure 8 in the appendix provides an in-depth comparison of performance concerning the number of pieces in objects.\n\n> The notations are sometimes a bit confused, especially for the tilde and hat. For example, the hat_P_i at line 258 seems to be not defined.\n\n**A:**\nOur notation convention assigns a hat to variables associated with fractures and a tilde to variables denoting predictions or approximations. Specifically, $\\hat{P_i}$ represents the fracture points on piece $i$, and we will introduce a clarification for this notation.\n\n> The experimental settings are not clear for all the involved methods. A table to list and compare the conditions and data/label requirement for each method would be helpful.\n\n**A:** We include the experimental settings in Table 3 in the appendix. For all baseline methods and our approach, only the ground-truth poses are required for supervision. \n\n> The introduction of the rigidity loss is not well justified and analyzed, especially it is only applied to the model training after 200 epochs.\n\n**A:**\nThe rigidity loss increased the performance by $0.2 (\\times 10^{-2})$ in translation. It serves as a regularization to encourage rigid dense point matching, and its application requires a well-informed initialization. To meet this requirement, we train 200 epochs before introducing the rigidity loss.\n\n> Does the generation of the segmentation labels require additional knowledge about the data compared to other methods?\n\n**A:**\nNo additional knowledge about the data is required. Segmentation labels are computed based on the original Breaking Bad dataset's ground truth position of each piece (lines 171-173). This process doesn't involve any additional manual labor or introduce new information into the dataset.\n\n> Is only one precise match at line 193 an assumption? The assumption of only one precise match may not be satisfied as the points sampled on neighboring pieces may be quite far and no matching can be done simply based on point clouds.\n\n**A:** This is an observation that aligns with the real-world perspective when we treat surfaces as continuous. While sampled point clouds might not perfectly adhere to this rule due to the approximations introduced, the underlying observation guides our algorithm design of multi-part matching.\n"}, {"review_id": "cnwQDSBS0j", "reviewer": "Reviewer_jmgU", "summary": "This paper proposes a novel framework for the task of 3D fracture assembly. The proposed approach leverages hierarchical features of global and local geometry to match and align the fracture surfaces. Experiments are conducted on the Breaking Bad dataset to verify the effectiveness of the proposed method.", "questions": "1. As in the cross-attention, all the points from all the parts are leveraged for the attention computation. Therefore, the efficiency heavily relies on the size of point clouds. So how many points are included in each part and the whole shape? Is the proposed method still applicable if the number of points raised to >10k?\n\n2. What is the main difference between the assembly task and the task of pairwise/multi-frame point cloud registration?\n\n3. What is the overlap ratio (probably in average) between different parts?", "limitations": "Limitations have been discussed in the main paper.", "rating": 4, "confidence": 3, "soundness": 2, "presentation": 3, "contribution": 2, "strengths": "1. This paper is well-written and easy to follow;", "weaknesses": "1. I am not the guy from the same area and therefore I cannot judge the novelty of the proposed mehtod well. However, from my experiences in my research field, this task is quite close to pairwise and multi-frame point cloud registration, and the major difference is the utilization of surface segmentation and the significantly lower overlap ratio between parts. Based on that, some techniques, especially the geometric learning and the matching, are well-explored and widely-adopted in the aforementioned registration tasks, and therefore I consider the proposed solution trivial.\n\n2. For the experiments, I think there are some problems:\n  \n   2.1 The experiments are conducted on a single dataset, and I am not sure whether it is a synthetic or real one. To this end, I think it is necessary to also evaluate the proposed method on a second dataset. If the leveraged dataset is synthetic, it would be better to include experiments on real data;\n \n    2.2 In Tab. 1, the latest baselines are from 2020 which is 3 years ago. I have no idea why some latest ones are not included. Does it mean that there have been no work in this topic for 3 years? I would like to see the comparisons to some more recent works during rebuttal;\n\n    2.3 In Tab.1, the only point cloud registration baseline is Predator which is a work in 2021. Some more recent works, for example, CoFiNet (NeurIPS 2021), GeoTransformer (CVPR 2022), Lepard (CVPR 2022), RegTr (CVPR 2022) should be considered as the baselines from the field of point cloud registration (GeoTransformer is the state-of-the-art).  \n\n    2.4 As the task is similar to multi-frame point cloud registration, I also think it reasonable and necessary to include baselines from that area, like (1) and (2).\n\n3. Fig. 1 carries overly much information and is hard to focus on each specific part. I would suggest the authors modifying it to make it clear. Also, in Related Work (line 90), [30] is not based on the overlap prediction.\n\n\n---------------------------------------------------------------------------------------------------------\n\n(1) Gojcic et al. Learning Multiview 3D Point Cloud Registration, CVPR 2020\n\n(2) Yew et al. Learning Iterative Robust Transformation Synchronization, 3DV 2021", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thanks for your time and valuable comments. Below we respond to your questions/concerns point-by-point and hope we can resolve some confusions.\n\n> 2.1 The experiments are conducted on a single dataset, and I am not sure whether it is a synthetic or real one. To this end, I think it is necessary to also evaluate the proposed method on a second dataset. If the leveraged dataset is synthetic, it would be better to include experiments on real data;\n\n**A:** Breaking Bad is a synthetic dataset introduced in NeurIPS 2022, and is the only available option for multi-part fracture assembly assessment. It already exhibits considerable size and diversity, particularly regarding its fracture modes and object categories. As fracture assembly remains a relatively new field, we believe efforts towards introducing a distinct dataset for evaluation are underway.\n\n> 2.2 In Tab. 1, the latest baselines are from 2020 which is 3 years ago. I have no idea why some latest ones are not included. Does it mean that there have been no work in this topic for 3 years? I would like to see the comparisons to some more recent works during rebuttal;\n\n**A:** Fracture assembly remains a relatively new field, progressing at a gradual pace. Based on the Breaking Bad Dataset publication in NeurIPS 2022, the latest relevant baseline listed is DGL. We have not encountered any new publications pertaining to this topic in the recent half-year period.\n\n> 2.3 In Tab.1, the only point cloud registration baseline is Predator which is a work in 2021. Some more recent works, for example, CoFiNet (NeurIPS 2021), GeoTransformer (CVPR 2022), Lepard (CVPR 2022), RegTr (CVPR 2022) should be considered as the baselines from the field of point cloud registration (GeoTransformer is the state-of-the-art).\n\n**A:** GeoTransformer was indeed considered as a registration baseline. However, after preliminary experiments, its performance aligned with that of Predator, rendering it unsuitable for providing meaningful assembly results. Given the unaffordable time investment required for GeoTransformer's training (estimated 15+ days), it was not included in our results.\n\n> 2.4 As the task is similar to multi-frame point cloud registration, I also think it reasonable and necessary to include baselines from that area, like (1) and (2).\n\n**A**: Our task is different in the sense that besides we have to segment the fractured surfaces out, each point of the fractured surface has at most one potential match with fractured regions of other fragments. Our approach explicitly leverages this property. This is very different from multi-frame point cloud registration, in which one point may be matched with multiple points (each point from each overlapping scan). Therefore, we do not think it is necessary to compare against multi-frame point cloud registration baselines. \n\n> Q1. As in the cross-attention, all the points from all the parts are leveraged for the attention computation. Therefore, the efficiency heavily relies on the size of point clouds. So how many points are included in each part and the whole shape? Is the proposed method still applicable if the number of points raised to >10k?\n\n**A:** The overall shape contains 5K points, while individual parts’ point numbers are sampled based on the part surface area. We tested that our method is runnable with 10K points sampled in a single object on V100. We acknowledge the possibility of designing downsampling schemes for cross-attention, while this lies beyond the scope for this paper and merits exploration in future endeavors.\n\n> Q2. What is the main difference between the assembly task and the task of pairwise/multi-frame point cloud registration?\n\n**A:** The essential difference is that the assembly task focuses on the thin surfaces and edges of the object and relies on the complementary geometry between pieces. Though it may be perceived as an extreme case of pairwise/multi-frame point cloud registration, key differences include: 1) The overlapping area appears at most in 2 parts (leading to our multi-part matching strategy); 2) The overlap lacks “volume” and should based on complementary geometry instead of finding identical geometry, which requires distinct representation from objects/scenes registration (as demonstrated in our primal-dual descriptor design) 3) Constraints exist for maintaining smoothness in the assembled object's original surface, a matter addressed within our limitations and reserved for future exploration.\n\n> Q3. What is the overlap ratio (probably in average) between different parts?\n\n**A:** On average, the fracture points account for 20.0% of all points. For extreme cases (e.g., a fractured plate) the ratio can fall to 4%. Since each fracture point belongs to a unique match, this ratio indicates the average overlap between pieces.\n"}, {"review_id": "HnR2LMc2cV", "reviewer": "Reviewer_nZZN", "summary": "The authors address the problem of reassembling fractured 3D objects from its pieces, where each piece is represented as a point cloud. The main challenge lies in the fact that one has to simultaneously solve a segmentation problem (which part of an object is fracture surface and which part is the original surface) and a multi-matching problem. The authors propose a learning-based approach that nicely take into account these different aspects. Their approach is supervised, which sounds reasonable given that it is relatively straightforward to simulate object fractures from complete objects.", "questions": "1) The intro also discusses furniture assembly - however, this is quite a different problem since furniture does not have fractures. I suggest to smoothen this.\n2) In related work it is mentioned that most graph matching methods focus on pairwise matchings. There are some recent multi-matching methods which should be discussed (see e.g. Universe Points Representation Learning for Partial Multi-Graph Matching, AAAI'23 and references therein).\n3) Please make the font and visuals in all figures larger (ideally the fonts match the font size of the caption).\n4) How many V100 GPUs are in one machine? Please comment on the training and inference time.\n5) I think there is some ambiguity in the notation around lines 161-165: $N$ is already introduced before as the number of object pieces; also, the index $i$ is used both for the object piece index, and as point index.\n6) How do you ensure that the primal-dual feature pair is really complementary/consistent? Is this enforced by construction or does the learning take care of that? Also, the formulation \"the surface looks different from a different viewpoint\" is not ideal, since this suggests there is some camera (with projection) involved (at least that was my first thought). I suggest to rephrase this.\n7) Is matrix $A$ in eq. (6) positive semidefinite? It looks a bit like the Mahalanobis distance in a Gaussian kernel, but then we would also need a negative sign. Please comment on this.\n8) Eqn. (9) misses that (R,t) are elements of SE(3). Similarly, the solution below finds (R,t) in E(3) - one needs to ensure the determinant of the rotation is +1.\n9) How do you ensure that the blocks of the matrix X in eqn. (13) are consistent? In 3.4 it is explained that pairwise transformations are computed first (representing blocks of X), and then RANSAC is applied. Does this guarantee global consistency? Do you apply some variant of permutation synchronisation, or is consistency only guaranteed for the rigid-body transformations via the Shonan-averaging?\n10) What is the scale/unit of the point clouds? Absolute values of transformation error are not very meaningful.", "limitations": "Limitations are adequately addressed.", "rating": 7, "confidence": 5, "soundness": 3, "presentation": 4, "contribution": 3, "strengths": "+ The authors address a difficult, relevant and relatively novel problem and set the new state of the art\n+ The proposed solution is a neat combination of learning and traditional optimisation techniques\n+ All pieces are matched simultaneously which really is key to address this difficult problem\n+ The idea of using a primal-dual feature descriptor pair to account for the complementary geometric properties of features is novel and useful ", "weaknesses": "- Although the presented results look appealing, they only show a small subset of all results. Given the relatively large angular errors (up to 52.4 degrees), it is unclear whether results are cherry-picked or representative. Given the overall improvement over previous methods I do not see this as critical for acceptance, yet, it is important to understand this.\n- There a few things that leave some room for clarification (see below).", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thanks for your time and valuable comments. Thank you for your suggestions regarding the introduction, multi-graph matching references, and figure enhancements. We will make adjustments in the final version. \n\nBelow we respond to your questions and hope we can clarify the potential misunderstandings.\n\n> Although the presented results look appealing, they only show a small subset of all results. Given the relatively large angular errors (up to 52.4 degrees), it is unclear whether results are cherry-picked or representative. Given the overall improvement over previous methods I do not see this as critical for acceptance, yet, it is important to understand this.\n\n**A:** In Figure 8 we showed the error distribution with respect to the number of pieces. While we achieved general superiority, there remains space for improvement especially for over 5 pieces. We issued this limitation in Section 5 and left it as our future work.\n\n> Q4. How many V100 GPUs are in one machine? Please comment on the training and inference time.\n\n**A:** 4 V100 GPUs are used in one machine. Please refer to the general response for detailed information on time and resource allocation.\n\n> Q5. I think there is some ambiguity in the notation around lines 161-165: \n is already introduced before as the number of object pieces; also, the index \n is used both for the object piece index, and as point index.\n\n**A:** $n$ should be used to represent the number of object pieces and $N$ represents the number of points. We will adjust them accordingly in the final version.\n\n> Q6. How do you ensure that the primal-dual feature pair is really complementary/consistent? Is this enforced by construction or does the learning take care of that? Also, the formulation \"the surface looks different from a different viewpoint\" is not ideal, since this suggests there is some camera (with projection) involved (at least that was my first thought). I suggest to rephrase this.\n\n**A:** The affinity metric's design and the learning process, guided by the matching loss, jointly ensure that the primal-dual feature pair exhibits a preference for matching points with complementary geometry rather than similar geometry. A more detailed explanation is presented in the general response and we will further clarify this aspect in the final version.\n\n\n> Q7. Is matrix  in eq. (6) positive semidefinite? It looks a bit like the Mahalanobis distance in a Gaussian kernel, but then we would also need a negative sign. Please comment on this.\n\n**A:** In practice, Matrix A contains learnable parameters to weight point features and no PSD constraints are explicitly applied. This is a common practice in matching tasks since the paper \"Deep Learning of Graph Matching\", A Zanfir et. al, CVPR 2018.\n\n> Q8. Eqn. (9) misses that (R,t) are elements of SE(3). Similarly, the solution below finds (R,t) in E(3) - one needs to ensure the determinant of the rotation is +1.\n\n**A:** If the determinant of R is -1, we flip the singular vector with the smallest singular value in V.\n\n> Q9.1. How do you ensure that the blocks of the matrix X in eqn. (13) are consistent? \n\n**A:** All surface points are matched collectively and then organized to create pairwise matching. Thus, there's no consistency concern regarding the matrix X computed from Eq (13).\n\n> Q9.2. In 3.4 it is explained that pairwise transformations are computed first (representing blocks of X), and then RANSAC is applied. Does this guarantee global consistency? \n\n**A:** RANSAC was applied for computing pairwise transformation from point correspondences. For consistent global pose estimation we applied Shonan-averaging, which was a state-of-the-art robust alignment method.\n\n> Q9.3 Do you apply some variant of permutation synchronisation, or is consistency only guaranteed for the rigid-body transformations via the Shonan-averaging?\n\n**A:** In the fracture assembly task each fracture point is supposed to have one unique corresponding point among all other pieces. Therefore the permutation synchronization issue is not involved. Moreover, this global one-to-one matching was a critical insight for our multi-part matching design.\n\n> Q10. What is the scale/unit of the point clouds? Absolute values of transformation error are not very meaningful.\n\n**A:** Each object has a scale of ~0.8 on average in each dimension, while each piece is approximately ~0.3 in each dimension. It's important to note that translation errors can accumulate for matching-based methods if a piece is incorrectly assembled to a wrong position.\n"}]

Author rebuttal:
Dear area chair and reviewers,

We thank all the reviewers for the valuable comments. We appreciate that reviewers acknowledge that this paper is well-written, well motivated, and experimentally solid. Our proposed joint learning pipeline with the attention-based feature extractor, the fracture point segmentation, the primal-dual descriptor and multi-part matching reaches state-of-the-art performance for the multi-fracture assembly task, and are all recognized by the reviewers.

We begin with addressing some questions shared by reviewers.

**(I) The running time of Jigsaw compared to baseline methods:**

| Model     | Training Time / GPUs   | Speed (s/batch) / batch size       |
|-----------|:------------------------:|:----------------------------:|
| Global    | 21H           / 1 V100 |  1.65    /   32          |
| LSTM      | 16H           / 1 V100 | 1.63   / 32              |
| DGL       | 11H           / 1 V100 | 2.46    / 32             |
| Predator  | 96H           / 4 V100 | 1.28    / 28             |
| Jigsaw    | 120H          / 4 V100 | 7.67 / 8 * |

\* The time used in the forward is 1.3s/batch, the rest of the time is used on Hungarian and global alignment.


**(II) The motivation of primal-dual descriptor:**

When one object has been cracked into two pieces by physical force, the two pieces should exhibit complementary geometry. The conventional feature representation is used to find similar geometry and therefore would match two identical surfaces, which is undesirable in the context of fracture assembly. Our proposed primal-dual descriptor, however, is used to capture the essence of complementary geometry. It helps to foster correspondences between two parts that possess complementary attributes.

One example (Fig. 2 in the paper) is that a convex shape would be matched to another convex shape if only the conventional approach is used. Instead, our primal-dual descriptor could match it to a concave shape.
Another example (Fig. 1 of the authors' response document) showcases a situation that point $A$ would be matched to point $b$, and point $B$ would be matched to point $a$ with conventional approach. It causes two pieces to completely overlap. Though in the context of fracture assembly, with the proposed primal-dual descriptor, point $A$ could be correctly matched to point a and point $B$ to point $b$.

The learning of two descriptors should be viewed in conjunction with the construction of the affinity metric (Eq. (6)) and the learning procedure guided by the matching loss (Eq. (7)). By providing ground truth matching for complementary points, the system would favor the similarity between complementary attributes over the identical geometries. As demonstrated in Fig. 6 of the appendix, our primal-dual descriptor refrains from self-matching and finds matching across disparate fragments. It is an experimental evidence showing the primal-dual system can capture distinct features for one fracture point.

For other issues that require clarification, we provide detailed answers in the individual point-by-point response.


Meta-review: This paper introduces intriguing work for assembling fractured parts. Reviewers favor the proposed approach and the demonstrated results. The authors provided comprehensive feedback regarding specific questions, and the authors provided a supplement pdf file. In particular, there were concerns from reviewer jmgU on the practical aspects, but AC notes that the authors adequately defend it. AC agrees that assembling fractured parts has very small overlaps between parts, and it is distinct from pair-wise point cloud registration problems. AC also notes that the proposed primal-dual descriptor is an interesting attempt. After the rebuttal phase, most of the reviewers voted for the positive scores. As a result, AC recommends paper acceptance. It is strongly advised to apply the valuable discussions and additional materials in the revision.
Decision: Accept (poster)