Title: Jigsaw: Learning to Assemble Multiple Fractured Objects
Venue: NeurIPS 2023
Accepted: yes
Keywords: Shape Matching; Reassembly; Shape Segmentation;
Abstract: Automated assembly of 3D fractures is essential in orthopedics, archaeology, and our daily life. This paper presents Jigsaw, a novel framework for assembling physically broken 3D objects from multiple pieces. Our approach leverages hierarchical features of global and local geometry to match and align the fracture surfaces. Our framework consists of four components: (1) front-end point feature extractor with attention layers, (2) surface segmentation to separate fracture and original parts, (3) multi-parts matching to find correspondences among fracture surface points, and (4) robust global alignment to recover the global poses of the pieces. We show how to jointly learn segmentation and matching and seamlessly integrate feature matching and rigidity constraints. We evaluate Jigsaw on the Breaking Bad dataset and achieve superior performance compared to state-of-the-art methods. Our method also generalizes well to diverse fracture modes, objects, and unseen instances. To the best of our knowledge, this is the first learning-based method designed specifically for 3D fracture assembly over multiple pieces. Our code is available at https://jiaxin-lu.github.io/Jigsaw/.

Jigsaw: Learning to Assemble Multiple Fractured Objects

Abstract

Automated assembly of 3D fractures is essential in orthopedics, archaeology, and our daily life. This paper presents Jigsaw, a novel framework for assembling physically broken 3D objects from multiple pieces. Our approach leverages hierarchical features of global and local geometry to match and align the fracture surfaces. Our framework consists of four components: (1) front-end point feature extractor with attention layers, (2) surface segmentation to separate fracture and original parts, (3) multi-parts matching to find correspondences among fracture surface points, and (4) robust global alignment to recover the global poses of the pieces. We show how to jointly learn segmentation and matching and seamlessly integrate feature matching and rigidity constraints. We evaluate Jigsaw on the Breaking Bad dataset and achieve superior performance compared to state-of-the-art methods. Our method also generalizes well to diverse fracture modes, objects, and unseen instances. To the best of our knowledge, this is the first learning-based method designed specifically for 3D fracture assembly over multiple pieces. Our code is available at https://jiaxin-lu.github.io/Jigsaw/.

1 Introduction

The task of assembling 3D fractures has extensive applications across numerous fields. For instance, orthopedic doctors need to realign dislocated bone fragments, and subsequently create bone plates and screws to heal compound fractures. Archaeologists, on the other hand, need to recreate the original shape and functionality of unearthed artifacts by assembling the fractures. These procedures demand significant expertise, are prone to errors, and can be tedious. Even in the context of daily life, furniture assembly can be a challenging and exhausting task that requires an understanding of mechanical structures and component matching. In the past two decades, many efforts attempted to address the challenge of automatic assembly. Traditional methods apply hand-crafted geometric features to detect the fracture surfaces and optimize pairwise matching among these surfaces [1, 2]. Recently the availability of large scale 3D datasets [3, 4, 5, 6] have boosted learning based frameworks for solving 3D assembly tasks. Semantic-aware methods [7, 8, 9, 10, 11, 12] target at assembly from semantically segmented parts and predict semantic labels as matching priors. Geometry based methods [13, 14, 15] leverage fracture shapes and continuity of textures in the procedure of piece matching.

However, in the context of fracture assembly for restoring broken objects, there is no guarantee that individual pieces will retain semantic meanings. Texture information may also be either nonaccessible or lacking, such as on glass bottles or worn artifacts. This calls for a general 3D assembly solver that utilizes hierarchical features of both the global piece surfaces and local geometry shapes of fractures. In this work, we introduce a J o i nt Learnin g of S egmentation and A lignment Frame w ork

( Jigsaw ), for assembling objects damaged or shattered due to physical impact or force. Given a set of fractured pieces represented as point clouds without texture, our method recovers the global pose of each piece to restore the underlying object.

Our Jigsaw framework consists of four parts: (1) A front-end feature extractor with self-attention and cross-attention layers for local geometric feature extraction. (2) Categorize the surface of each fractured piece into two segments: the indiscernible fracture surface and the visible original surface. (3) A novel formulation for multi-piece assembly that learns a bipartite matching among points on the fracture surface from all pieces. (4) Recover pairwise pose based on the learned correspondences and perform robust global alignment to compute the global poses of all pieces. Key features of Jigsaw are that 1) it learns two correlated tasks, i.e., segmentation and matching among all involving pieces, jointly, and 2) it seamlessly integrates feature matching and the rigidity constraint among consistent features, We evaluate the effectiveness of our framework on Breaking Bad [4]. Experimental results demonstrate the effectiveness of our method in surface segmentation, fracture point matching, and significantly outperforming state-of-the-art methods[8, 16]. The main contribution includes the following:

We propose Jigsaw, a novel joint learning framework tailored for multi-part fracture assembly. Our approach embodies an attention-based feature extractor network to accurately capture local geometry features of each point. Additionally, we introduce a primal-dual descriptor that effectively captures viewpoint-dependent characteristics for surface matching.

Jigsaw incorporates fracture point segmentation to capture intrinsic features, employs a novel multi-part matching formulation to establish automatic piece positioning within one object, and utilizes global alignment for accurate global pose alignment.

Experimental evaluation on the multi-part assembly Breaking Bad dataset demonstrates the superior performance of Jigsaw compared to baseline models, showcasing its strong generalization ability to unseen objects. We further highlight the limitations of baseline models that rely on global features, which we argue are too abstract for this task and lack generalizability.

To the best of our knowledge, Jigsaw is the first learning-based method specifically designed for the assembly of multiple pieces from physically broken 3D objects.

2 Related Works

Feature matching . Early works apply hand-crafted features over fracture surfaces for fracture matching between different pieces [17, 1, 2, 18]. These features are also used to identify the fractured part from the entire surface [17, 1, 2]. However, hand-crafted features are in lack of robustness for assembly tasks over large datasets due to different materials and fracture patterns of objects. In recent years, deep learning methods have gained significant traction in matching problems. Methods utilizing CNN, GNN, and attentions have found successful applications in various domains, such as image registration [19, 20] and graph or multi-graph matching on images [21, 22, 23, 24, 25]. While their efficacy in these scenarios is noteworthy, the majority of these methods predominantly focus on simpler settings and have not adequately addressed the challenges in multi-part 3D fracture assembly.

Part assembly . Semantic-aware learning methods have been highly successful in the task of part assembly. [9, 11] are designed for assembling specific CAD mechanics. For categorical everyday objects, [26, 10, 27, 12, 28] generate the missing parts based on the accumulated shape prior to completing the entire object, which can result in shape distortion from the input parts. [8, 7] apply graph learning to predict part labels and assembly orders. All of these methods require the input objects decomposed in a semantically consistent way and need specific training for each category of objects. Fracture assembly poses unique challenges due to the variety of objects and the lack of semantic meanings associated with individual pieces, adding to the complexity of the task.

Geometry based learning methods . Recent approaches have aimed to capture geometry information using deep features for piece matching. [13, 15] combine local geometry with textures for feature generation, but may suffer when texture information is not accessible, as in most point cloud representations. [14] apply a transformer for local shape encoding and an adversarial network to help generate a plausible assembly of two pieces. However, for multiple fracture assembly, objects can break into 5 pieces or more, with the largest piece smaller than half of the original object. Under this setting, training an adversarial network to evaluate the quality of assembly becomes less effective.

Surface

Segmentation

𝑷𝟏(N1 × 3)

𝑷𝟐(𝑁2 × 3)

𝑷𝒏(𝑁𝑛 × 3)

⋯

𝐅𝟏(𝑁 1 ×𝐷)

𝐅𝟐(𝑁2 × 𝐷)

𝐅𝐧(𝑁𝑛 × 𝐷)

⋯

PointNet++

⋯

MLP

෠

𝐅( ෡

𝑁 × 𝐷)

MLP

෠

𝐅 𝐝

( ෡

𝑁 × 𝑑)

෠

𝐅 𝐩

( ෡

𝑁 × 𝑑)

⋅

⋅

exp

NN

𝑷𝟏

𝑷𝒏 ⋯ 𝑷𝟐

⋯

ℒmat

⋯

𝐷𝑖𝑠

≤ 𝜂

𝑷𝟏

𝑷𝟐

𝑷𝒏 ⋯

,

NN

𝑷𝟏

𝑷𝟐

𝑷𝒏 ⋯

Affinity Metric

Primal-dual Descriptor

Sinkhorn

𝑋 𝑔𝑡

෠ 𝑋

ℒseg

𝑋

Hungarian

RANSAC

(𝑃 1

, 𝑃

2 ,

)

RANSAC

(𝑃 1

, 𝑃

3 ,

)

Pairwise Pose

Recovery

Joint Alignment

Multi-part Matching

Global Fracture Alignment

Self-Attention Layer

Cross-Attention Layer

𝐅 𝐜 (𝑁 × 𝐷)

Front-end Feature Extractor

෠ 𝑋

ℒrig

Low-overlap 3D registration . Another relevant research field is low-overlap 3D registration. Recent works have shown the potential of data-driven methods in registration tasks with approximately 30% overlap [29, 16, 30]. [16, 30] learn a classifier to determine the overlapped sections and predict inter-piece point matching inside the sections, while [29] using object semantic to guide registration. The fracture assembly task can be viewed as an even more extreme case of registration, where the overlap between pieces can fall below 4%.

3 Joint learning framework for 3D fracture assembly

Given a set of fractured pieces P = { P 1 , P 2 , · · · , P n } represented as point clouds uniformly distributed on the surface, our goal is to recover the 6-DoF pose { T 1 , T 2 , · · · T n } in SE (3) for each piece and restore the underlying object O = T 1 ( P 1 ) ∪ T 2 ( P 2 ) ∪·· ·∪ T n ( P n ) , where T i ( · ) , 1 ≤ i ≤ n is the operator to recover P i to its original position by transformation T i . In the fracture assembly setting, the object O is a rigid body, and the pieces are the result of physical cracking or breakage without any deformation. Also, no pieces are lost, which ensures that the restored pieces can approximately reconstruct the entire object O .

To handle this challenge, we propose Jigsaw, a learning-based framework that jointly optimizes surface segmentation and fracture matching among all pieces of the object. The entire assembly network consists of a segmentation module and a matching module that shares a front-end feature extractor. The segmentation module uses only the intrinsic shape information of each piece to separate the fracture surface from the original surface. The matching module exploits the primal dual descriptor of the affinity metric to propagate mutual information among all pieces and establish the matching between fracture points (that is, points on the fracture surface) of different pieces via Sinkhorn [32]. With the fracture point matching predicted by the network, we recover the pairwise transformations and perform global alignment with standard approaches. The complete pipeline of our framework is shown in Fig. 1. Note that this design ensures that segmentation and matching are performed jointly. In the subsequent part of this section, we will first introduce the front-end extractor, then delve into the details of the segmentation module, followed by the multi-part matching module and post-processing for pose recovery.

3.1 Front-end Feature Extractor

In contrast to previous methods [33, 4] that utilize global piece-wise descriptors for assembly, we focus on geometric features. As illustrated in Figure 7 of the PointNet paper [34], a global descriptor remains the same as long as the piece lies between the learned critical point set and the upper-bound

point set. This indicates that global descriptors are coarse and insufficient to represent the intricate geometry of the fracture surface, which is crucial for our task. To address this limitation, we employ a multi-scale grouping PointNet++ [31] as the backbone of our feature extractor to capture local geometry features, denoted as f p ∈ R D from each point p ∈ P i .

Furthermore, We emphasize the significance of relative positional information between points and pieces for identifying the non-smooth fracture surfaces, enforcing rigidity during matching, and accurately placing the pieces. To facilitate the integration of such intra-piece and inter-piece information, we leverage transformer layers. Specifically, we introduce both self-attention and cross-attention layers as tools to reason about the relative information between points.

For the self-attention within a piece, we employ a single point transformer layer [35]. The selfattention mechanism is defined as follows:

Here, W s Q , W s K , W s V ∈ R D × D are weights for query, key, value in the attention layer, p enc represents the positional encoding for point p i , obtained using a Multi-Layer Perceptron (MLP). Unlike the standard dot-product attention layer, it employs vector weights: MLP_s is a mapping function that produces the weight vector, softmax ( · ) normalizes the weights, and glyph[circledot] denotes element-wise multiplication. N ( p ) ∈ P i represents the neighborhood of point p , calculated using k -nearest neighbors for local feature aggregation.

For cross-attention, we use standard multi-head attention with position-wise feed forward over the entire object to facilitate the communication of local features among different pieces. Let F s ∈ R N × D pack all the point features of one object after the self-attention, and F c ∈ R N × D denote the point features produced by cross-attention,

Here, W m Q i , W m K i , W m V i ∈ R D × d h , W m O ∈ R hd h × D denote the weights for projection. The function FFN ( · ) consists of two linear functions with ReLU activation in between, followed by a LayerNorm as normalization.

3.2 Surface segmentation

The fracture assembly problem can be viewed as a special case of the 3D registration problem, where the overlapping between adjacent pieces only lies in the fracture surface between them. Let P f i ⊂ P i be the subset of points on fracture surfaces of P i and let P ij ⊂ P f i denote the subset of points on the fracture surface between P i and P j for arbitrary two pieces P i , P j . For pieces P i and P j adjacent in their original pose T i ( P i ) and T j ( p j ) , we have P ij = ∅ , P ji = ∅ . Under the assumption that the point clouds are uniformly distributed, we can approximate the relative pose T glyph[star] ij = T -1 j T i between P i , P j with glyph[negationslash]

glyph[negationslash]

where d ( · , · ) is some distance function between two point clouds. (3) becomes the standard formulation for point cloud registration with perfect prior for the overlapping. Before we can optimize (3) directly, we need to figure out the overlapping part P ij and P ji for each pair ( P i , P j ) . Although P ij is dependent on both pieces, previous studies [17, 1, 2] have demonstrated that the segmentation of fracture surfaces P f i of P i is an intrinsic property that can be inferred from the local shapes of each P i individually. However, these methods require a continuous smooth surface to compute hand-crafted shape features for accurate classification. To capture local geometric properties under the discrete point cloud setting where surface normal and curvature are no more available, we introduce our deep surface segmentation module to handle the surface segmentation task.

The surface segmentation module can be viewed as a binary classifier of each point. Let c p be the indicator function that determines whether a point p of a point cloud P ∈ P is a fracture point. For each 3D model uniformly sampled as N = ∑ n i =1 N i points, the surface segmentation module

takes the position of the points P i ∈ R N i × 3 of a piece P i ∈ P and predicts a confidence score c i to segment a point p ∈ P i into the fracture surface.

Our front-end feature extractor introduced in Section 3.1 has extracted local geometry feature f p ∈ R D for each point. Next, we use two MLP layers to reduce the number of channels to 1, followed by a sigmoid function to predict the confidence c p for labeling p as a fracture point. We set the segmentation loss L seg to be a negative log-likelihood loss, which is supervised by the ground truth label c p for every point p as

The ground truth label c p for each point p ∈ P i is constructed by examining its distance to its nearest neighbor in other parts:

where Dis ( · ) is a distance function and NN ( · , · ) is to find the nearest neighbour of one point in a point set.

Although the fraction of the fracture surface area may differ between objects, and the positive and negative samples in (4) may be unbalanced, we have found that adding two MLP layers after feature encoding help to yield good performance for the segmentation task. The segmentation module is able to accurately predict fracture points even in cases with a large number of pieces or very small fracture surfaces, and we leave the evaluation details of this module in the appendix A.

3.3 Multi-part Matching

One key aspect of Jigsaw is its ability to combine multiple pieces. Traditional approaches to assembling pieces rely on pairwise matching, which involves assessing the compatibility of two parts, identifying corresponding points within them, and aligning those points. However, this method is prone to cumulative errors, and a single mistake can ruin the entire assembly task. In addition, matching small pieces with limited geometry information solely based on pairwise information is extremely challenging. Consequently, the pairwise information in the multipart fracture assembly dataset may not be comprehensive enough to capture the true matching possibilities between all the pieces involved in the assembly process. Even advanced low-overlap 3D registration methods like PREDATOR [16], which share a similar structure, have failed to handle assembly tasks effectively, as shown in Section 4.2. Thus, alternative approaches are needed to address the limitations of pairwise matching, and we introduce our multipart matching module.

Facilitated by the global fracture points of a segmented object in Section 3.2, we present a global view to match all its pieces simultaneously. We observe that each fracture surface should have one precise match within a broken object, which is a rule that applies specifically to the fracture piece assembly task. Therefore, we learn the matching among all pieces without specifying their pairwise relationship. This allows each fracture point to automatically find a similar counterpart on a different piece solely based on local geometry. Moreover, the correspondences highlight the assembly regions, avoiding the need for costly predictions of fine-grained pairwise overlapping areas.

Primal-dual Descriptor. Intuitively, when an object has been broken into two pieces by physical force, the two pieces should exhibit complementary geometry. Conventional feature representation is used to find similar geometry and, therefore, would match two identical surfaces, which

Primal-dual Descriptor

Primal Descriptor

Dual Descriptor

Fracture Piece

becomes undesirable in the context of fracture assembly. With this concern, we propose the primaldual descriptor, designed to capture the essence of complementary geometry (see Fig. 2).

Let { ˆ f c i } n c i =1 be the selected features of the fracture points extracted from the backbone network. These features encode the local geometry of the fracture surfaces. We apply MLP layers and a normalization layer to ˆ f c i and get the primal descriptor ˆ f p i and dual descriptor ˆ f d i :

The primal-dual descriptor learns surface features that capture the characteristics of a local surface from both directions. It promotes the robustness and reliability of surface matching and avoids learning biased features based on a single viewing direction. We leave more detailed analysis to the appendix B.

Affinity Metric. Let ˆ F p , ˆ F d ∈ R ˆ N × d collect all the fracture points features extracted from descriptors, we compute the global affinity matrix M ∈ R ˆ N × ˆ N and the respective doubly-stochastic matrix ˜ X ∈ [0 , 1] ˆ N × ˆ N as follows:

Here, A ∈ R d × d consists of learnable affinity weights, and τ denotes the temperature parameter [36]. We then apply the Sinkhorn layer [32], which is a differentiable operation, to obtain a doubly stochastic matrix ˜ X ∈ [0 , 1] ˆ N × ˆ N . This matrix represents the soft matching predicted by the multipart matching module.

Matching Loss. Given the ground truth position of each part, we construct the ground truth matching matrix X gt ∈ { 0 , 1 } ˆ N × ˆ N . Here, x gt ij = 1 if and only if the j -th fracture point is the nearest neighbor of the i -th fracture point and both points belong to different pieces; otherwise, x gt ij = 0 . To compute the matching loss, we employ the cross entropy between ˜ X and X gt :

With the ground truth matrix X gt generated based on finding the nearest neighbor of each point, the matching loss will also enforce a global rigidity guidance that each node should be matched to its neighbor.

Rigidity Loss. As 3D fracture assembly is applied to rigid objects, we further enforce the rigidity loss over the pairs of matched pieces. Let ˜ X ij ∈ R ˆ N i × ˆ N j be the submatrix of ˆ X that indicates the likelihood between the fracture points on the piece P i and P j . For a fracture point p in P i , ˜ X ij matches it to a point p ′ computed as the weighted average of all the matchable points in P j :

where ˜ X ij ( p ) ∈ R 1 × ˆ N j is the row in ˜ X ij that contains matching likelihood from p to P j . We optimize the transformation ˜ T ij from P i to P j by minimizing the weighted mean squared error between the matched fracture points

where ˜ R ij , ˜ t ij are the rotation and translation part of ˜ T ij and R ( · ) is the rotation operator. (10) has a closed-form solution:

The rigidity loss is computed as:

To avoid numerical instability and improve efficiency in (11) during training, we compute ˜ T ij in each iteration without involving back-propagation and only set ˜ X ij as the learnable variable.

The loss for training is composed as:

Multi-part Matching With the doubly-stochastic matrix ˜ X , computing multi-part matching results becomes straightforward. Given the fact that the primal and dual feature represents the surface looked from a different viewpoint, they should be distinct and not assigned good confidence for matching. Therefore, we can directly pass ˜ X to a Hungarian layer [37]. This yields a binary permutation matrix X , representing a bipartite matching among all fracture points:

As mentioned, the resulting matching matrix X , along with the affinity matrix M and soft matching matrix ˜ X , captures the local geometry similarity across different pieces. Additionally, they quantify the confidence level of how well the pieces fit together. This information enables an accurate positioning and alignment of the fractured components.

3.4 Global Fracture Alignment

To efficiently and robustly recover the global poses of each fractured piece of object O , we adopt a two-step pipeline based on the bipartite matching X ∈ { 0 , 1 } ˆ N × ˆ N predicted by the multipart matching module.

In the first step, we compute pairwise transformations between each pair of pieces ( P i , P j ) to remove outlier matches. Let X ij ∈ R ˆ N i × ˆ N j denote the submatrix of X , which establishes the correspondences between the fracture points from P i and P j . Weapply the RANSAC algorithm [38] to compute the transformation ˜ T ij from ˆ P i , ˆ P j and X ij . This step ensures reliable pairwise transformations for subsequent alignment.

In the second step, we perform robust global alignment using the computed pairwise transformations. We model the global alignment configuration as a factor graph [39], denoted as G = ( V , E ) , where the global poses ˜ T i are optimized on the vertices V and the pairwise transformations ˜ T ij are set as constraints on the edges E . The information matrix I ( e ) over the edge e = ( i, j ) is set to | X ij | -2 Fro · I 6 , with I 6 being the 6 × 6 identity matrix. We employ Shonan averaging [40], a state-of-the-art 6-DoF global alignment method, to optimize global poses over G . The global alignment method takes pairwise transformations between potentially adjacent pieces as input and outputs a pose for each piece up to a global transformation over the coordinate system. To avoid such an ambiguity, we anchor the coordinate system to the canonical one of the largest pieces in our experiment and measure the errors over all other pieces of the fractured object. Further technical details of this module can be found in the references [41, 42, 43]. Please refer to those sources for more information, as it is beyond the scope of our main contribution.

4 Experiments

We demonstrate the effectiveness of our 3D fracture assembly framework through experimental evaluations on a large-scale fracture assembly dataset. Our results show that Jigsaw significantly outperforms the baseline methods both quantitatively and qualitatively. Additionally, we conduct ablation studies to analyze the contributions of each module in our framework. All experiments are conducted on a Linux workstation with 4 Tesla V100-SXM2-32GB GPUs, Intel(R) Xeon(R) CPU E5-2698 v4 @ 2.20GHz CPUs, and 480GB Memory. Table 1 lists the configurations of the parameters in our experiments. A comparison of the training/testing time is included in Appendix D.

4.1 Protocols

Dataset. We leverage the Breaking Bad dataset (Sellan et al., 2022), a novel data set of multiple fracture assemblies featuring synthetic physical breaking patterns. Our training was on everyday subset and the testing was on both everyday and artifact subsets for a fair comparison with

baselines. Everyday consists of 498 models and 41,754 fracture patterns, and is split into a training set (34,075 fracture patterns from 407 objects) and a test set (7,679 fracture patterns from 91 objects). Artifact consists of 3651 fracture patterns from 40 uncategorized objects. The average diameter of the objects in the training and testing dataset was 0.8. Categorical information was concealed during all the experiments.

The input was uniformly sampled N points on the surface of the object as a point cloud. In composing the point cloud, two sampling strategies are considered, sampling by piece and object. Sampling by piece, as used in the Breaking Bad benchmark [4], equally samples points in each fragment, which would cause point density imbalance. To better reflect real-world scanning, we opt for 'sampling by object', sampling a fixed number of points in each object, while sampling points for each fragment based on its surface area. We ensure at least 30 points per fragment for multi-part matching. Additional processing details can be found in the Appendix C.1.

Baseline Approaches. As learning to assemble multiple fractured objects is a novel task, there is a dearth of existing methods for direct comparison. Therefore, we have selected a set of established methods that primarily address similar challenges in 3D alignment and assembly, and employ them as baselines for comparative evaluation. Global [26, 10] extracts per-piece features, which are combined with global shape descriptors to regress the pose of each piece in one shot. LSTM applies a bidirectional LSTM similar to [27] and estimates the pose of each piece in sequential style. DGL [8] is a state-of-the-art approach for the assembly task of parts and can also be adapted to the assembly task of fractures. It leverages an iterative graph neural network to reason about the relationships among pieces. Additionally, we take PREDATOR [16] as a minor competitor, which is a state-ofthe-art approach for 3D registration with low overlapping rates. It employs a multitask transformer to estimate the overlap between pairs of pieces in conjunction with their relative pose. All baseline approaches use textureless point clouds as input and are trained under the everyday object subset of the Breaking Bad dataset.

Evaluation Metrics. Weadopt the same evaluation schemes for 3D assembly tasks used in [4, 14, 10]. We report the mean absolute error (MAE) and the root mean square error (RMSE) for the rotations and translations of the estimated global poses. Additionally, we include the part accuracy (PA) metric proposed in [33], which measures the ratio of perfectly assembled pieces based on an average Chamfer distance of less than 0.01 for each point between the assembly results and the ground truth.

4.2 Performance

Overall Performance. We report the performance of Jigsaw and all the baseline methods over the 3D fracture assembly task on both everyday and artifact object subsets. An overall quantitative comparison of the evaluation metrics is presented in Table 2. For PREDATOR we apply the evaluation metrics on pairwise transformations since it only supports overlapped pairs of pieces as input.

Jigsaw significantly outperformed all the baselines across all the evaluation metrics. On everyday , we achieved an average rotation error of 36 . 3 ◦ over all pieces, which was a 46% reduction compared to the top-performing baseline method, DGL. Our transformation error was also outstanding, with an average of 8 . 7 × 10 -2 , surpassing the best baseline method with an error of 11 . 8 × 10 -3 . We

Ground

Truth

LSTM

Global

DGL

Jigsaw

Everyday

Artifact

successfully restored over 57% of the pieces to their original pose while baseline methods restored only half of us. We also show that change in sampling strategy has had no impact on the outcome of baseline methods through a sampling by object version of DGL. Our method exhibited remarkable generalizability on artifact , showing a slight increase of 10 . 1 ◦ in average rotation error and achieving a part accuracy of 44% . In contrast, all baseline methods completely failed to handle new categories of models that were not present in the training dataset. The PREDATOR model failed to accurately identify the pairwise overlap region, resulting in erroneous predictions of complete overlap between the two pieces. While this may seem to yield a small translation error, it failed to perform adequately in the assembly scenario. Hence, we exclude it from further discussion.

Detailed distribution of each metric. Fig. 4 presents a comprehensive analysis of each metric for all assembly models. Regarding rotation and translation metrics, a larger surface area under the curve indicates better performance. Jigsaw exhibits a significantly larger number of samples with small rotation and translation errors, as illustrated in the figure. Although Jigsaw reports a slightly larger translation error than the baseline models on the artifact object subset, the distribution results reveal that this is primarily due to a few extreme cases, leading to an undesirable accumulation of the translation error. In terms of part accuracy, a larger surface area above the curve signifies superior results. Notably, Jigsaw demonstrates a higher proportion of correctly positioned pieces.

Visualization. The qualitative comparison of different models is shown in Fig. 3. We plot the recovered object from different models. It demonstrates that our model could find the position of the fracture pieces and recover a good pose of each piece. Visualization of objects from different categories also implies our model has good generalization ability and it doesn't require object-level information as assembly guidance.

0

50

100

150

0

20

40

60

80

100

Percent

0

25

50

75

100

125

150

0.0

0.1

0.2

0.3

0.4

0.0

0.1

0.2

0.3

0.4

0.0

0.2

0.4

0.6

0.8

1.0

0

50

100

150

RMSE(R)

0

20

40

60

80

100

Percent

0

25

50

75

100

125

150

MAE(R)

0.0

0.1

0.2

0.3

0.4

RMSE(T)

0.0

0.1

0.2

0.3

0.4

MAE(T)

0.0

0.2

0.4

0.6

0.8

1.0

Part Accuracy

method

Global

LSTM

DGL

Jigsaw

Ablation Study. We conducted an ablation study to assess the effectiveness of each component in our Jigsaw model. The baseline model consists of a segmentation module and a multi-part matching module, trained separately using separate backbones and without incorporating transformers. The plain joint training model was trained using the same loss function as the Jigsaw model, but without using two transformer layers. The effectiveness of the rigidity loss regularization was evaluated over the complete network with both joint training and transformer layers.

As presented in Table 3, the joint training model outperformed the baseline in transformation metrics. This is due to a more favorable initialization for the matching module achieved through pretraining the backbone with the segmentation module. Joint training also aided the matching module's convergence to better optimum. Attention layers significantly reduced rotation errors:

the self-attention layer embedded geometric information into point features, while the cross-attention layer improved multi-part matching by incorporating information from other parts. Applying rigidity loss as a refinement additionally brought an improvement of 2 × 10 -3 in translation. Even the baseline method outperformed previous works, highlighting the effectiveness of our segmentation and multi-part matching modules.

5 Conclusion and Limitations

In this study, we propose a novel geometry-aware framework to address the 3D fracture assembly task. Our approach involves a joint learning model that effectively extracts local geometry features from the backbone and enables the simultaneous learning of piecewise fracture surface segmentation and global fracture point matching. Experimental evaluations and thorough analyses substantiate the exceptional performance of our method in terms of accurately recovering piece poses and restoring the original object. Furthermore, we demonstrate the generalizability of our approach across objects of diverse categories. One limitation of our approach lies in that as the number of fractured pieces increases, the accuracy of pose estimation tends to decrease due to potential ambiguities in fracture matching caused by local geometric features. Future research directions may include exploring matching pruning techniques for pairwise pose estimation to prevent penetration between fractured pieces and enforcing smoothness constraints to enhance pose accuracy.

References

Georgios Papaioannou and Evaggelia-Aggeliki Karabassi. On the automatic assemblage of arbitrary broken solid artefacts. Image and Vision Computing , 21(5):401-412, 2003.

Qi-Xing Huang, Simon Flöry, Natasha Gelfand, Michael Hofer, and Helmut Pottmann. Reassembling fractured objects by geometric matching. In ACM SIGGRAPH 2006 Papers , SIGGRAPH '06, pages 569-578, New York, NY, USA, 2006. ACM.

Andy Zeng, Shuran Song, Matthias Nießner, Matthew Fisher, Jianxiong Xiao, and T Funkhouser. 3dmatch: Learning the matching of local 3d geometry in range scans. In CVPR , volume 1, page 4, 2017.

Silvia Sellán, Yun-Chun Chen, Ziyi Wu, Animesh Garg, and Alec Jacobson. Breaking bad: A dataset for geometric fracture and reassembly. In Thirty-sixth Conference on Neural Information Processing Systems Datasets and Benchmarks Track , 2022.

Kaichun Mo, Shilin Zhu, Angel X Chang, Li Yi, Subarna Tripathi, Leonidas J Guibas, and Hao Su. PartNet: A Large-scale Benchmark for Fine-grained and Hierarchical Part-level 3D Object Understanding. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition , pages 909-918, 2019.

Li Yi, Vladimir G Kim, Duygu Ceylan, I-Chao Shen, Mengyan Yan, Hao Su, Cewu Lu, Qixing Huang, Alla Sheffer, and Leonidas Guibas. A scalable active framework for region annotation in 3d shape collections. ACM Transactions on Graphics (ToG) , 35(6):1-12, 2016.

Abhinav Narayan Harish, Rajendra Nagar, and Shanmuganathan Raman. Rgl-net: A recurrent graph learning framework for progressive part assembly. In 2022 IEEE/CVF Winter Conference on Applications of Computer Vision (WACV) , pages 647-656. IEEE, 2022.

Guanqi Zhan, Qingnan Fan, Kaichun Mo, Lin Shao, Baoquan Chen, Leonidas J Guibas, Hao Dong, et al. Generative 3d part assembly via dynamic graph learning. Advances in Neural Information Processing Systems , 33:6315-6326, 2020.

Benjamin Jones, Dalton Hildreth, Duowen Chen, Ilya Baran, Vladimir G Kim, and Adriana Schulz. Automate: A dataset and learning approach for automatic mating of cad assemblies. ACM Transactions on Graphics (TOG) , 40(6):1-18, 2021.

Yichen Li, Kaichun Mo, Lin Shao, Minhyuk Sung, and Leonidas Guibas. Learning 3d part assembly from a single image. In Computer Vision-ECCV 2020: 16th European Conference, Glasgow, UK, August 23-28, 2020, Proceedings, Part VI 16 , pages 664-682. Springer, 2020.

Karl DD Willis, Pradeep Kumar Jayaraman, Hang Chu, Yunsheng Tian, Yifei Li, Daniele Grandi, Aditya Sanghi, Linh Tran, Joseph G Lambourne, Armando Solar-Lezama, et al. Joinable: Learning bottom-up assembly of parametric cad joints. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition , pages 15849-15860, 2022.

Kangxue Yin, Zhiqin Chen, Siddhartha Chaudhuri, Matthew Fisher, Vladimir G Kim, and Hao Zhang. Coalesce: Component assembly by learning to synthesize connections. In 2020 International Conference on 3D Vision (3DV) , pages 61-70. IEEE, 2020.

Corey Toler-Franklin, Benedict Brown, Tim Weyrich, Thomas Funkhouser, and Szymon Rusinkiewicz. Multi-feature matching of fresco fragments. ACM Transactions on Graphics (TOG) , 29(6):1-12, 2010.

Yun-Chun Chen, Haoda Li, Dylan Turpin, Alec Jacobson, and Animesh Garg. Neural shape mating: Selfsupervised object assembly with adversarial shape priors. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition , pages 12724-12733, 2022.

Thomas Funkhouser, Hijung Shin, Corey Toler-Franklin, Antonio García Castañeda, Benedict Brown, David Dobkin, Szymon Rusinkiewicz, and Tim Weyrich. Learning how to match fresco fragments. Journal on Computing and Cultural Heritage (JOCCH) , 4(2):1-13, 2011.

Shengyu Huang, Zan Gojcic, Mikhail Usvyatsov, Andreas Wieser, and Konrad Schindler. Predator: Registration of 3d point clouds with low overlap. In Proceedings of the IEEE/CVF Conference on computer vision and pattern recognition , pages 4267-4276, 2021.

Georgios Papaioannou, E-A Karabassi, and Theoharis Theoharis. Virtual archaeologist: Assembling the past. IEEE Computer Graphics and Applications , 21(2):53-59, 2001.

David Koller and Marc Levoy. Computer-aided reconstruction and new matches in the forma urbis romae. Computer-aided Reconstruction and new Matches in The Forma Urbis Romae , pages 103-125, 2006.

Paul-Edouard Sarlin, Daniel DeTone, Tomasz Malisiewicz, and Andrew Rabinovich. SuperGlue: Learning feature matching with graph neural networks. In CVPR , 2020.

Jiaming Sun, Zehong Shen, Yuang Wang, Hujun Bao, and Xiaowei Zhou. LoFTR: Detector-free local feature matching with transformers. CVPR , 2021.

Andrei Zanfir and Cristian Sminchisescu. Deep learning of graph matching. In CVPR , pages 2684-2693, 2018.

Runzhong Wang, Junchi Yan, and Xiaokang Yang. Learning combinatorial embedding networks for deep graph matching. In IEEE International Conference on Computer Vision , pages 3056-3065, 2019.

Michal Rolínek, Paul Swoboda, Dominik Zietlow, Anselm Paulus, Vít Musil, and Georg Martius. Deep graph matching via blackbox differentiation of combinatorial solvers. In European Conference on Computer Vision , pages 407-424. Springer, 2020.

Runzhong Wang, Junchi Yan, and Xiaokang Yang. Neural graph matching network: Learning lawler's quadratic assignment problem with extension to hypergraph and multiple-graph matching. IEEE Transactions on Pattern Analysis and Machine Intelligence , 2021.

Zetian Jiang, Jiaxin Lu, Tianzhe Wang, and Junchi Yan. Learning universe model for partial matching networks over multiple graphs. arXiv preprint arXiv:2210.10374 , 2022.

Nadav Schor, Oren Katzir, Hao Zhang, and Daniel Cohen-Or. Componet: Learning to generate the unseen by part synthesis and composition. In Proceedings of the IEEE/CVF International Conference on Computer Vision (ICCV) , October 2019.

Rundi Wu, Yixin Zhuang, Kai Xu, Hao Zhang, and Baoquan Chen. Pq-net: A generative part seq2seq network for 3d shapes. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition , pages 829-838, 2020.

Anastasia Dubrovina, Fei Xia, Panos Achlioptas, Mira Shalah, Raphael Groscot, and Leonidas J. Guibas. Composite shape modeling via latent space factorization. In Proceedings of the IEEE/CVF International Conference on Computer Vision (ICCV) , October 2019.

Lifa Zhu, Dongrui Liu, Changwei Lin, Rui Yan, Francisco Gómez-Fernández, Ninghua Yang, and Ziyong Feng. Point cloud registration using representative overlapping points. arXiv preprint arXiv:2107.02583 , 2021.

Zheng Qin, Hao Yu, Changjian Wang, Yulan Guo, Yuxing Peng, and Kai Xu. Geometric transformer for fast and robust point cloud registration. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition , pages 11143-11152, 2022.

Charles Ruizhongtai Qi, Li Yi, Hao Su, and Leonidas J Guibas. Pointnet++: Deep hierarchical feature learning on point sets in a metric space. In Advances in neural information processing systems , pages 5099-5108, 2017.

Marco Cuturi. Sinkhorn distances: Lightspeed computation of optimal transport. Advances in neural information processing systems , 26, 2013.

Jialei Huang, Guanqi Zhan, Qingnan Fan, Kaichun Mo, Lin Shao, Baoquan Chen, Leonidas Guibas, and Hao Dong. Generative 3d part assembly via dynamic graph learning. Advances in Neural Information Processing Systems (NeurIPS) , 2020.

Charles R Qi, Hao Su, Kaichun Mo, and Leonidas J Guibas. Pointnet: Deep learning on point sets for 3d classification and segmentation. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition , pages 652-660, 2017.

Hengshuang Zhao, Li Jiang, Jiaya Jia, Philip HS Torr, and Vladlen Koltun. Point transformer. In Proceedings of the IEEE/CVF International Conference on Computer Vision , pages 16259-16268, 2021.

Andrei Zanfir and Cristian Sminchisescu. Deep learning of graph matching. In Proceedings of the IEEE conference on computer vision and pattern recognition , pages 2684-2693, 2018.

Martin A. Fischler and Robert C. Bolles. Random sample consensus: A paradigm for model fitting with applications to image analysis and automated cartography. Commun. ACM , 24(6):381-395, jun 1981.

Sungjoon Choi, Qian-Yi Zhou, and Vladlen Koltun. Robust reconstruction of indoor scenes. In CVPR , pages 5556-5565. IEEE Computer Society, 2015.

Daphne Koller and Nir Friedman. Probabilistic graphical models: principles and techniques . MIT press, 2009.

Frank Dellaert, David M Rosen, Jing Wu, Robert Mahony, and Luca Carlone. Shonan rotation averaging: Global optimality by surfing so ( n ) . In European Conference on Computer Vision , pages 292-308. Springer, 2020.

Qian-Yi Zhou, Jaesik Park, and Vladlen Koltun. Open3d: A modern library for 3d data processing. arXiv preprint arXiv:1801.09847 , 2018.

Frank Dellaert and GTSAM Contributors. borglab/gtsam, May 2022.

Frank Dellaert and Michael Kaess. Factor Graphs for Robot Perception . Foundations and Trends in Robotics, Vol. 6, 2017.

Appendix

A Details of Network Architecture

We provide additional information about our network architecture. The backbone of our network is based on a standard PyTorch implementation of multi-scale grouping PointNet++. It is followed by a single MLP layer that extracts a D -dimensional feature for each point. In our implementation, we set D = 128 .

The detailed structure of the transformer module can be seen in Fig. 5. The self-attention layer follows the official code of the point transformer layer, while the cross-attention layer employs a multi-head attention mechanism with positionwise feed-forward networks. In our configuration, we set the number of attention heads to h = 8 , the head dimension to d h = 16 , and use k -nearest neighbor sampling with k = 16 . The inner layer of the feed-forward network has a dimensionality of d i = 256 .

𝐟 𝑝

𝒑

𝑾𝑸

𝒔

𝑾𝑲

𝒔

𝑾𝑽

𝒔

MLP

⨁

⨁

MLP

KNN Aggregation

𝐟 𝑝

s

𝑾𝑸

c

𝟏

𝑾𝑲

c

𝟏

𝑾𝑽

𝐜

𝟏

Scaled Dot-Product

Attention

𝐅 s

cat

𝑾𝑶

c

Position-Wise Feed

Forward

𝐅 𝒄

(a) Self-Attention Layer

(b) Cross-Attention Layer

Regarding the primal-dual descriptor, we set its feature dimension to d = 256 .

B Analysis on Primal-dual Descriptor

X

from unmasked primal-dual descriptor

X

from masked primal-dual descriptor

X

from unmasked single-view descriptor

X

from masked single-view descriptor

0.00

0.01

0.02

0.03

0.04

0.05

0.00

0.01

0.02

0.03

0.04

0.05

0.01

0.02

0.03

0.04

0.05

0.06

0.07

0.08

0.000

0.005

0.010

0.015

0.020

0.025

0.030

0.035

We conducted an analysis to compare the results obtained using the primal-dual descriptor and the standard single-view descriptor. Before passing the affinity metric computed from the single-view descriptor to the Sinkhorn layer, a common practice is to mask out the diagonal sub-matrix to prevent self-self alignment. For clarity, we refer to the masked and unmasked versions of the primal-dual and single-view descriptors to denote whether this masking operation has been applied or not.

In Fig. 6, we provide a visualization of the doubly-stochastic matrix ˜ X computed by the Sinkhorn algorithm using four types of descriptors: (1) the unmasked primal-dual descriptor, (2) the masked primal-dual descriptor, (3) the unmasked single-view descriptor, and (4) the masked single-view descriptor. The visualization depicts an example object with 3 pieces and 58 fracture points. The ˜ X obtained from the primal-dual descriptor clearly demonstrates its ability to differentiate between the two viewpoints and avoid aligning a point to itself. In contrast, the single-view descriptor exhibits a diagonal peak, resulting in self-self alignment. Even when the diagonal sub-matrix is masked to prevent self-self alignment, the soft matching computed from the single-view descriptor is less distinct compared to that computed from the primal-dual descriptor.

Furthermore, we conducted evaluation on the everyday object subset of Breaking Bad dataset, using the unmasked primal-dual descriptor and the masked primal-dual descriptor. The results indicate that the unmasked primal-dual descriptor achieves comparable performance to the masked one, with only a slight difference: a decrease of 0 . 2 ◦ in rotation error (MAE(R)) and an improvement of 0 . 7 × 10 -2 in translation error (MAE(T)). We attribute this minor difference to the fine characteristics of the primal-dual descriptor that prevent self-self alignment, thus resulting in consistent performance.

C Experiment Details

C.1 Dataset

We leverage Breaking Bad dataset [4] to evaluate our method and all baseline methods. As we have stated in section 4, the training of all methods were on the training subset of everyday , and testing were on the testing subset of both everyday and artifact object. Each object within the dataset has been fragmented into pieces, represented by triangle meshes, and all the pieces are in their original poses. By assembling these pieces together directly, the surface of the original object is seamlessly restored. The triangle meshes of the pieces solely consist of exterior faces that are visible from the outside.

In generating the point cloud for our experiments, we employ two sampling strategies for the compared methods: 'sampling by piece' and 'sampling by object'. The 'sampling by piece' strategy, originally utilized in the Breaking Bad benchmark [4], involves sampling an equal number of points within each fragment. However, this approach leads to excessively dense sampling on small fragments, while larger fragments suffer from sparser point distributions, resulting in an imbalance in the representation of point density across fragments. To better mirror real-world scanning technology, we opt for the 'sampling by object' strategy. With this approach, we sample a fixed number of points within each object, and the number of sampled points for each fragment is determined based on its surface area. In essence, smaller fragments receive fewer points, whereas larger ones receive more, ensuring a more realistic representation. Additionally, we ensure that each fragment is sampled with a minimum of 30 points to include even the tiniest fragments in multi-part matching. Detailed parameter settings can be found in Table 1. Our analysis of the average fragment numbers and experiments conducted using DGL [8] indicate that the choice between the two sampling strategies has negligible impact on the baseline performance.

For each sampled point p on P i , its fracture label c p is determined by the distance from p to its nearest neighbor q among points from all other pieces:

where η is set to 0.02 for all the objects. The ground-truth matching point ˆ q of a fracture point ˆ p ∈ ˆ P i is set to

where ˆ O, ˆ P i denotes the set of fracture points in O and P i . During the training process, c p was applied to segment the fracture points and during testing the predicted ˜ c p was used instead. An example is shown in Fig. 7.

After the ground-truth labeling and matching were com- puted based on the pieces at their original pose, all pieces were recentered to the origin and a random rotation was applied to each piece.

To ensure a fair comparison with baseline methods [26, 10, 27, 8], we adopted the same implementation as of the benchmark code of [4], which sampled the same number of point each piece. For PREDATOR, we applied the same sample routine as the baseline methods, and we paired the adjacent pieces for training and testing.

C.2 Evaluation Metrics

The mean absolute error MAE(R) and square-rooted mean squared error RMSE(R) of rotation were computed as

for each piece, where ˜ R and R gt were predicted rotation and ground-truth rotation represented in Euler angles. The error over each object was computed as the mean error of all the pieces and the total error was the mean error of all the object. For translation, MAE(T) and RMSE(T) were computed in the same way

where ˜ t and t gt were the predicted translation and ground-truth translation.

D Implementation Details

D.1 Additional Configuration of parameters

For PREDATOR [16], we carefully follow the parameters presented in its open sourced code, and the threshold for overlap recall to add saliency loss is set to be 0 . 3 . For Jigsaw, we start training with only the segmentation loss. We add matching loss after first 10 epochs, and rigidity loss after 200 epochs.

D.2 Running time

Our framework is implemented in Pytorch. All methods are trained over Tesla V100-SXM2-32GB GPUs and distributed data parallel strategy is applied for multi-GPU training. The training and influence time for Jigsaw and baseline methods are shown in Table 4.

E Additional Results

E.1 Detailed Quantitative Results

Fig. 8 gives a detailed distribution of each error metric with respect to the number of fractured pieces. Our approach exhibits a clear advantage in each of the metrics evaluated, and this advantage becomes more pronounced as the number of pieces decreases.

In addition, we present the mean and standard deviation (STD) of three runs of our method in Table 5. These values demonstrate the stability of our method across different random seeds and variations in the sampled point cloud. The consistent performance and low standard deviation affirm the robustness of our proposed approach.

2

3

4

5

6

7

8

9

10

11

12

13

14

15

16

17

18

19

20

Pieces

0

20

40

60

80

100

120

140

RMSE(R)

method

Global

LSTM

DGL

Jigsaw

2

3

4

5

6

7

8

9

10

11

12

13

14

15

16

17

18

19

20

Pieces

0

20

40

60

80

100

120

140

MAE(R)

method

Global

LSTM

DGL

Jigsaw

2

3

4

5

6

7

8

9

10

11

12

13

14

15

16

17

18

19

20

Pieces

0.0

0.1

0.2

0.3

0.4

0.5

RMSE(T)

method

Global

LSTM

DGL

Jigsaw

2

3

4

5

6

7

8

9

10

11

12

13

14

15

16

17

18

19

20

Pieces

0.0

0.1

0.2

0.3

0.4

0.5

MAE(T)

method

Global

LSTM

DGL

Jigsaw

2

3

4

5

6

7

8

9

10

11

12

13

14

15

16

17

18

19

20

0.0

0.2

0.4

0.6

0.8

1.0

Part Accuracy

method

Global

LSTM

DGL

Jigsaw

Pieces

E.2 More Qualitative Results

We collect additional visualizations of the assembled objects in Fig. 9. Even with a large number of pieces to assemble, our method remains robust in restoring major pieces to their original pose.

E.3 Additional Registration Baseline

We have received advice to consider GeoTransformer [30] as a state-of-the-art baseline for low overlap registration. However, training GeoTransformer proves to be exceedingly slow (about 15 days over 4 V100 GPUs), primarily due to the significant scale of Breaking-Bad in comparison to the 3D registration datasets on which it was originally tested. On pairwise transformations GeoTransformer achieves MAE(R)=72.4(degree), RMSE(R)=84.8(degree), RMSE(T)=14.3( × 10 -2 ), MAE(T)=11.6( × 10 -2 ), PA=3.1 % . These findings align with the claims we've put forth in our paper, and we believe that presenting the full results of PREDATOR adequately reflects how registration baselines perform on the assembly task.

Ground

Truth

LSTM

Global

DGL

Jigsaw

Artifact

Ground

Truth

LSTM

Global

DGL

Jigsaw

Everyday

Peer reviews:
{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe authors address the problem of reassembling fractured 3D objects from its pieces, where each piece is represented as a point cloud. The main challenge lies in the fact that one has to simultaneously solve a segmentation problem (which part of an object is fracture surface and which part is the original surface) and a multi-matching problem. The authors propose a learning-based approach that nicely take into account these different aspects. Their approach is supervised, which sounds reasonable given that it is relatively straightforward to simulate object fractures from complete objects.\n\nSTRENGTHS:\n+ The authors address a difficult, relevant and relatively novel problem and set the new state of the art\n+ The proposed solution is a neat combination of learning and traditional optimisation techniques\n+ All pieces are matched simultaneously which really is key to address this difficult problem\n+ The idea of using a primal-dual feature descriptor pair to account for the complementary geometric properties of features is novel and useful\n\nWEAKNESSES:\n- Although the presented results look appealing, they only show a small subset of all results. Given the relatively large angular errors (up to 52.4 degrees), it is unclear whether results are cherry-picked or representative. Given the overall improvement over previous methods I do not see this as critical for acceptance, yet, it is important to understand this.\n- There a few things that leave some room for clarification (see below).\n\nQUESTIONS:\n1) The intro also discusses furniture assembly - however, this is quite a different problem since furniture does not have fractures. I suggest to smoothen this.\n2) In related work it is mentioned that most graph matching methods focus on pairwise matchings. There are some recent multi-matching methods which should be discussed (see e.g. Universe Points Representation Learning for Partial Multi-Graph Matching, AAAI'23 and references therein).\n3) Please make the font and visuals in all figures larger (ideally the fonts match the font size of the caption).\n4) How many V100 GPUs are in one machine? Please comment on the training and inference time.\n5) I think there is some ambiguity in the notation around lines 161-165: $N$ is already introduced before as the number of object pieces; also, the index $i$ is used both for the object piece index, and as point index.\n6) How do you ensure that the primal-dual feature pair is really complementary/consistent? Is this enforced by construction or does the learning take care of that? Also, the formulation \"the surface looks different from a different viewpoint\" is not ideal, since this suggests there is some camera (with projection) involved (at least that was my first thought). I suggest to rephrase this.\n7) Is matrix $A$ in eq. (6) positive semidefinite? It looks a bit like the Mahalanobis distance in a Gaussian kernel, but then we would also need a negative sign. Please comment on this.\n8) Eqn. (9) misses that (R,t) are elements of SE(3). Similarly, the solution below finds (R,t) in E(3) - one needs to ensure the determinant of the rotation is +1.\n9) How do you ensure that the blocks of the matrix X in eqn. (13) are consistent? In 3.4 it is explained that pairwise transformations are computed first (representing blocks of X), and then RANSAC is applied. Does this guarantee global consistency? Do you apply some variant of permutation synchronisation, or is consistency only guaranteed for the rigid-body transformations via the Shonan-averaging?\n10) What is the scale/unit of the point clouds? Absolute values of transformation error are not very meaningful.\n\nLIMITATIONS:\nLimitations are adequately addressed.", "TITLE": "Official Review", "RECOMMENDATION": 7, "REVIEWER_CONFIDENCE": 5, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2023-06-26"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper proposes a novel framework for the task of 3D fracture assembly. The proposed approach leverages hierarchical features of global and local geometry to match and align the fracture surfaces. Experiments are conducted on the Breaking Bad dataset to verify the effectiveness of the proposed method.\n\nSTRENGTHS:\n1. This paper is well-written and easy to follow;\n\nWEAKNESSES:\n1. I am not the guy from the same area and therefore I cannot judge the novelty of the proposed mehtod well. However, from my experiences in my research field, this task is quite close to pairwise and multi-frame point cloud registration, and the major difference is the utilization of surface segmentation and the significantly lower overlap ratio between parts. Based on that, some techniques, especially the geometric learning and the matching, are well-explored and widely-adopted in the aforementioned registration tasks, and therefore I consider the proposed solution trivial.\n\n2. For the experiments, I think there are some problems:\n  \n   2.1 The experiments are conducted on a single dataset, and I am not sure whether it is a synthetic or real one. To this end, I think it is necessary to also evaluate the proposed method on a second dataset. If the leveraged dataset is synthetic, it would be better to include experiments on real data;\n \n    2.2 In Tab. 1, the latest baselines are from 2020 which is 3 years ago. I have no idea why some latest ones are not included. Does it mean that there have been no work in this topic for 3 years? I would like to see the comparisons to some more recent works during rebuttal;\n\n    2.3 In Tab.1, the only point cloud registration baseline is Predator which is a work in 2021. Some more recent works, for example, CoFiNet (NeurIPS 2021), GeoTransformer (CVPR 2022), Lepard (CVPR 2022), RegTr (CVPR 2022) should be considered as the baselines from the field of point cloud registration (GeoTransformer is the state-of-the-art).  \n\n    2.4 As the task is similar to multi-frame point cloud registration, I also think it reasonable and necessary to include baselines from that area, like (1) and (2).\n\n3. Fig. 1 carries overly much information and is hard to focus on each specific part. I would suggest the authors modifying it to make it clear. Also, in Related Work (line 90), [30] is not based on the overlap prediction.\n\n\n---------------------------------------------------------------------------------------------------------\n\n(1) Gojcic et al. Learning Multiview 3D Point Cloud Registration, CVPR 2020\n\n(2) Yew et al. Learning Iterative Robust Transformation Synchronization, 3DV 2021\n\nQUESTIONS:\n1. As in the cross-attention, all the points from all the parts are leveraged for the attention computation. Therefore, the efficiency heavily relies on the size of point clouds. So how many points are included in each part and the whole shape? Is the proposed method still applicable if the number of points raised to >10k?\n\n2. What is the main difference between the assembly task and the task of pairwise/multi-frame point cloud registration?\n\n3. What is the overlap ratio (probably in average) between different parts?\n\nLIMITATIONS:\nLimitations have been discussed in the main paper.", "TITLE": "Official Review", "RECOMMENDATION": 4, "REVIEWER_CONFIDENCE": 3, "SOUNDNESS_CORRECTNESS": 2, "DATE": "2023-07-05"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe authors present a learning framework for assembling physically broken 3D objects with point cloud observations. There are three components in the framework, surface segmentation to identify the fracture points, multi-parts matching to find correspondences among the fracture points, and the global alignment to recover the global poses of the pieces. The main contributions are the joint learning of segmentation and multi-parts matching and the design of the multi-parts matching including the primal-dual descriptor and the losses. The contributions are validated by superior performance on the Breaking Bad dataset compared with other methods.\n\nSTRENGTHS:\nThe task is new with just few methods already developed. This work may serve as a baseline for future work in this line.\n\nThe overall framework is clearly clarified and organised. The adding of self-attention and cross-attention is well justified and illustrated for the front-end feature extractor. The primal-dual descriptor is introduced in the multi-part matching part, which is new for this task to my knowledge. A further analysis of the primal-dual descriptor is also provided in the appendix to provide better understanding of the effect. \n\nThe experiments and results are also solid.\n\nWEAKNESSES:\nThe absence of the statistics of the number of pieces in the dataset may lead to unfair comparison against the baseline methods as points are sampled at different level as shown in Table 3 in the appendix.\n\nA further analysis of the multi-parts matching w.r.t. the number of pieces could better show the effect of multi-part matching against pairwise matching.\n\nThe notations are sometimes a bit confused, especially for the tilde and hat. For example, the hat_P_i at line 258 seems to be not defined.\n\nThe experimental settings are not clear for all the involved methods. A table to list and compare the conditions and data/label requirement for each method would be helpful. \n\nThe introduction of the rigidity loss is not well justified and analyzed, especially it is only applied to the model training after 200 epochs.\n\nTexts in the figures are too small to recognize.\n\nQUESTIONS:\nDoes the generation of the segmentation labels require additional knowledge about the data compared to other methods? Although I don't think so, a clarification from the authors would be helpful.\n\nIs only one precise match at line 193 an assumption? The assumption of only one precise match may not be satisfied as the points sampled on neighboring pieces may be quite far and no matching can be done simply based on point clouds.\n\nAddressing the weaknesses listed above could change my opinion.\n\nLIMITATIONS:\nThe authors deals with the accuracy drop of pose estimation when the number of fractured pieces increases.", "TITLE": "Official Review", "RECOMMENDATION": 6, "REVIEWER_CONFIDENCE": 3, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2023-07-06"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper proposes Jigsaw, a novel framework to assemble 3D fragments of a shape to complete the whole 3D shape. \nJigsaw first segments the fracture surface, since only the points on the fracture surface are involved in the physical assemble between fragments. Jigsaw then performs multi-part matching to find correspondences among the fracture surface points, which is finally used to recover the global pose of each fragment via global alignment. \nJigsaw exhibits state-of-the-art performance on the Breaking Bad dataset.\nThe authors propose Jigsaw to be the first learning-based method to assemble 3D fractures over **multiple** pieces.\n\nSTRENGTHS:\n* First learning based method to tackle the task of multi-part assembly of 3D fragments.\n\n* Well-motivated, novel learning pipeline to facilitate multi-part assembly of 3D fragments. \n\n* Strong performances on the Breaking Bad dataset. Considering that a two-fragment assembly problem can also be seen as a zero-overlap (or surface-overlap) point cloud registration, the baseline methods in the table are also appropriate.\n\nWEAKNESSES:\n* Lacks self-containedness. While it is understandable that Shonan-averaging was used as the off-the-shelf global alignment method, but the paper does not go on to explain how this global alignment functions at the minimum, nor how the global pose values from Shonan-averaging is used for evaluation against the given ground-truth global pose values.\n\n* Jigsaw samples a fixed number of points across all the fragments of a single shape. In that case, Jigsaw is expected to fail given a particularly small fragment, from which a very small number of fragments would be sampled. This may result in not a single point on the fracture surface being sampled, and this lacks emphasis in the manuscript.\n\n* The manuscript lacks explanation on how the primal-dual descriptor works. The authors state that their 'system learns surface features that capture the characteristics of a local surface from both directions' - which lacks the evidence the substantiate this claim. Is it the **aim** of primal-dual descriptors to function as mentioned, or is there a theoretical evidence to prove that the primal-dual descriptors actually captures such characteristics?\n\n* While the idea of surface segmentation to narrow down the points to consider is viable, it is expected that the fracture surface is rather easy to segment due to its irregularity in comparison to other parts of the shape. How would the proposed method perform if the fracture is not irregular, but of regular cuts (e.g., straight lines, smooth curves) such as in Neural Shape Mating? [1]\n\n[1] YC Chen et al., Neural Shape Mating: Self-Supervised Object Assembly with Adversarial Shape Priors, CVPR 2022\n\nQUESTIONS:\nPlease refer to the above weaknesses section. It would be appreciated if the authors could include additional details to improve the self-containedness of the paper, and to clear any potential misunderstandings.\n\nLIMITATIONS:\nThe limitations of Jigsaw have been discussed by the authors.", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 5, "SOUNDNESS_CORRECTNESS": 4, "DATE": "2023-07-07"}

comments: SUMMARY:
This work provides a Joint Learning of Segmentation and Alignment Framework(Jigsaw) framework for 3D fracture assembly task. Experiments are evaluted on Breaking Bad dataset and show better performance than other existing methods.

STRENGTHS:
1. The proposed approach applies a joint learning framework specific for multi-part fracture assembly, which appends an attention-based 
network module to capture local geometric information. To further capture viewpoint-dependent features for surface matching, the proposed method applies the primal-dual descriptor to achieve better matching.

2. The authors incorporate fracture point segmentation to capture intrinsic features and introduce a multi-part matching approach
for internal automatic piece positioning, with global pose alignment capability. 

3. The proposed method shows better results than baseline methods evaluated on the multi-part assembly Breaking Bad dataset. The ablation analysis also validates the efficacy of the proposed network structure.

WEAKNESSES:
1.  The motivation of using primal-dual descriptor seems not sufficient, it would be better to append a more description of it. 
2.  The ablation study part is considered a bit insufficient. e.g. The authors already mention matching loss and rigidity loss. The related ablation analysis about the impact of segmentation loss, matching loss and rigidity loss is missing.

QUESTIONS:
The running time analysis of the proposed method or the comparison to other methods is missing. How about this metric?

LIMITATIONS:
In the experiments, evaluation on extra dataset rather than the Breaking Bad dataset is much desired.
TITLE: Official Review
SOUNDNESS_CORRECTNESS: 3
DATE: 2023-07-09