PuzzleFusion++: Auto-agglomerative 3D Fracture Assembly by Denoise and Verify

## Abstract

Abstract This paper proposes a novel “auto-agglomerative” 3D fracture assembly method, PuzzleFusion++, resembling how humans solve challenging spatial puzzles. Starting from individual fragments, the approach 1) aligns and merges fragments into larger groups akin to agglomerative clustering and 2) repeats the process iteratively in completing the assembly akin to auto-regressive methods.
Concretely, a diffusion model denoises the 6-DoF alignment parameters of the fragments simultaneously, and a transformer model verifies and merges pairwise alignments into larger ones, whose process repeats iteratively.
Extensive experiments on the Breaking Bad dataset show that PuzzleFusion++ outperforms all other state-of-the-art techniques by significant margins across all metrics In particular by over 10% in part accuracy and 50% in Chamfer distance.
The code will be available on our project page: https://puzzlefusion-plusplus.github.io .

## 1 Introduction

Humans have an innate proficiency in solving complex spatial puzzles.
Starting with scattered pieces, we assess connections based on their shapes and fits, gradually merge them into larger components, and repeat the process through trial and error until forming a single assembly.
The task requires a combination of sophisticated skills on shape analysis, spatial reasoning, and trial and error process of navigating the massive combinatorial solution space.

A computational system with such capabilities would have significant impacts across broad domains. For example, archaeologists could reassemble fragments and uncover ancient artifacts.
Forensic specialists could reconstruct broken objects from accident scenes to deduce causative events. Biochemistry scientists could discover effective drugs by analyzing compatible protein 3D structures.

Deep neural networks (DNNs) have brought dramatic progress in developing such an intelligent computational method. DNN-based encoder enables robust shape matching by encoding raw 3D scans (i.e., point clouds) into latent embeddings, where a combinatorial optimization technique searches for an optimal set of pairwise alignments []. PuzzleFusion [] proposed an innovative fully neural system based on Diffusion Models that simultaneously aligns all the pieces in solving 2D jigsaw puzzles. This paper pushes the frontier of spatial puzzle solving by proposing a fully neural system that simulates how humans tackle this problem.

Starting from individual fracture fragments, the approach 1) simultaneously aligns and merges fragments into larger groups, akin to agglomerative clustering, and 2) repeats this process iteratively in completing the assembly akin to auto-regressive methods.
We call our approach “auto-agglomerative”,
mimicking the way humans tackle challenging spatial puzzles.
Concretely, after using PointNet++ [] and VQVAE [] to encode each fragment into latent embeddings, the approach uses a diffusion model to solve for the 6-DoF alignments of all the fragments. A transformer model verifies the inferred pairwise alignments and merges verified pairs into larger groups. The process repeats until forming a single assembly.

Extensive experiments on the Breaking Bad dataset [] show that PuzzleFusion++ outperforms all existing methods by significant margins in all metrics, specifically by over 10% in part accuracy and over 50% in the Chamfer distance metric,
pushing the frontier of computational methods for challenging spatial puzzle tasks.

## 2 Related Work

Diffusion models.
Diffusion models [, , ] have gained prominence for their exceptional performance in generating images [, , ], videos [, ], and 3D assets [, , , , ]. The great capacity to capture complex data distribution makes diffusion-based models suitable for many tasks beyond generation. Recent works have extended diffusion models as general solvers for deterministic tasks, including object detection [], camera pose estimation [], shape reconstruction [, ], and puzzle solving [, ]. Our method exploits diffusion models within an agglomerative puzzle-solving framework to deal with the complex geometric reasoning for the fracture assembly tasks.

Auto-regressive methods.
Auto-regressive methods recursively predict the next values in a sequence based on the previously observed or predicted elements. Auto-regressive approaches like GPT [] and its follow-ups have revolutionized natural language processing (NLP). Similar approaches also generate images as pixel sequences [], audio [], or 3D shapes [, ], etc.
The auto-regressive agglomeration process in our method draws inspiration from these successes. By iteratively aligning and merging fractured fragments into larger ones for future iterations, we gradually assemble all fragments into a single object. The process mimics human cognitive puzzle-solving strategies, aiming for more precise and efficient assembly.

3D shape assembly.
3D fracture assembly has been a challenge for machine intelligence []. PartNet [] presents a large 3D object segmentation database (*e.g*., the legs or seat of a chair), fueling research on learning-based methods [, , ].
Recent transformer-based approaches [, , ] capture the global semantic and geometric contexts well to enhance the alignment accuracy. Chen et al. [] proposes a shape-mating dataset where objects are randomly fractured into two fragments, and an adversarial learning framework reassembles the two fragments. Lamb et al. [] collects a dataset with real-world broken objects.
The Breaking Bad dataset [] introduces a more challenging benchmark, where diverse objects are fractured into multiple (*i.e*., 2 to 100) fragments via physical simulations. Jigsaw [] utilizes deep networks for fracture surface segmentation and matching, followed by a global alignment step to derive the poses.
SE(3)-equiv [] extracts global equivariant and invariant features to represent each fractured fragment and directly regress the 6-DoF alignment parameters. DiffAssemble [] further applies a diffusion model to this framework. This paper introduces an auto-agglomerative framework to mimic how humans solve spatial puzzles, where a diffusion model with local geometric features serves as a spatial solver in each iteration, where a transformer verifies the correctness of aligned fragments and merges them into larger groups.

## 3 PuzzleFusion++: An auto-agglomerative approach for fracture assembly

PuzzleFusion++ innovates computational methods for fracture assembly via an “auto-agglomerative” approach, where the task is to take a set of rigid fractured fragments as input and estimate their 6-DoF alignments. In our case, an input fragment and an output alignment are a point-cloud and a 3D rigid transformation, respectively.
PuzzleFusion++ consists of two modules, the denoiser and the verifier. The denoiser aligns individual fragments (§3.2), where the verifier classifies the correctness of pairwise alignments and merges verified pairs into larger fragments (§3.3), akin to agglomerative clustering.
The approach repeats the process based on previous predictions until forming a single 3D shape,
akin to auto-regressive methods (§3.4).

Figure: Figure 2: The architecture overview of PuzzleFusion++. Left: An illustration of the auto-agglomerative fracture assembly process with the first two iterations. Right: Close-ups of the SE3 denoise transformer and the pairwise alignment verifier transformer. Please refer to Figure 3 for the details of the architectures.
Refer to caption: x2.png

### 3.1 Preliminaries

Input representation. An input fragment comes as point clouds.
We train PointNet++ [] and VQ-VAE [] to encode the point clouds into latent vectors at the points
(details in Section A.1). PointNet++ employs
furthest point sampling to select 25 points, creating 25 corresponding point latent vectors for each fragment.
This approach retains intricate surface details,
offering an advantage over the global feature extraction used in previous studies [, ].
Note that the encoding process normalizes the 3D coordinate frame so that the center of mass is at the origin, and the longest dimension of the axis-aligned bounding box becomes a unit length. The encoding is still
sensitive to rotation, where latent vectors are re-calculated as we optimize fragment alignments.

Output representation.
The output alignment is represented as a 7D vector for each fragment, composed of a 4D quaternion and a 3D translation vector.

Anchor fragments.
Fracture assembly has an inherent ambiguity in the 3D rigid transformation. We introduce the concept of anchor fragments, whose alignments are fixed to the identity matrix for rotation and the zero vector for translation. During inference, the largest fragment (based on the longest dimension of its axis-aligned bounding box) becomes an anchor. Any fragment that merges with an anchor also becomes an anchor.
During training, we set anchors to be the largest fragment (with the same definition as above) plus its neighboring fragments with a 50% chance each to simulate the merging process at the inference time.

### 3.2 SE3 denoiser

Figure: Figure 3: Architecture specifications. The denoiser (blue) is a diffusion model with Transformer architecture at its core. The verifier (orange) is a Transformer.
Refer to caption: x3.png

Forward process. Let $x^{f}_{t}$ denote the 7D alignment vector of fragment $f$ at timestep $t$. $x^{f}_{0}$ is the ground-truth alignment before noise injection. A piece-wise quadratic scheduler adds a Gaussian Noise $\delta^{f}_{t}$ to $x^{f}_{0}$, using $m$=700 and $T$=1000 (refer to Figure 3). No noise is added to anchor fragments.

Reverse process.
Following PuzzleFusion [], each sampled point of each fragment keeps track of the alignment estimate in the denoise transformer architecture.
The denoising architecture is standard (See Figures 2 and 3),
where the main components are 1) Intra-fragment self-attention (Local SA) among 25 sampled points within a fragment; 2) Inter-fragment self-attention (Global SA) among all sampled points across all fragments; and 3) Average pooling over the 25 sampled points of fragment $f$ to predict the residual $\tilde{\delta}^{f}_{t}$.

Both intra-fragment and inter-fragment SA modules utilize adaptive layer norm [] to inject the timestep $t$. Each module comprises 6 self-attention layers, each with 8 multi-heads and a feature dimension of 512.

A feature embedding $\hat{x}^{f}_{p,t}\in\mathbb{R}^{512}$ of a sampled point $p$ of a fragment $f$ is
the concatenation of the four features: 1) The alignment estimate $x^{f}_{t}\in\mathbb{R}^{7}$ with a positional encoding ($\gamma()\in\mathbb{R}^{147}$; 2) The point latent $z^{f}_{p,t}\in\mathbb{R}^{64}$; 3) The 3D coordinate $c^{f}_{p,t}\in\mathbb{R}^{3}$ with a positional encoding ($\gamma()\in\mathbb{R}^{63}$); and 4) The scale normalization factor $n^{f}$ of the point-cloud (§3.1) with a positional encoding ($\gamma()\in\mathbb{R}^{21}$). MLP maps the concatenated feature into a 512D latent embedding.

Note that anchor fragments are processed exactly the same way, while their alignment should be the identity rotation and zero-vector translation. Therefore, alignment estimation is discarded at every denoising step during inference, and no gradients are injected during training for anchor fragments.
The standard DDPM [] loss is imposed on non-anchor fragments:
$E_{t,x_{0}^{f},\delta_{t}^{f}}\left\lVert\delta^{f}_{t}-\tilde{\delta}^{f}_{t}
\right\rVert^{2}$.

### 3.3 Pairwise alignment verifier

Given the alignment of $F$ fragments, the verifier employs a Transformer architecture with $\binom{F}{2}$ nodes to perform binary classification on the correctness of $\binom{F}{2}$ pairwise alignments simultaneously. Note that verifying the correctness of a given alignment is a lot easier than searching for an optimal alignment of many fragments, thus a straightforward Transformer architecture with the following node embeddings suffices for this task.

An input node embedding for a pair of fragments ($f$ and $g$) is the sum of two vectors. The first vector is the concatenation of the sinusoidal positional encodings of the fragment indices, $[PE(f),PE(g)]\in\mathbb{R}^{256}$. The second vector represents the alignment quality of $f$ and $g$, utilizing a point matching module from Jigsaw by Lu et al. [].
Specifically, the module takes the point clouds of all the fragments (without alignment) and identifies a set of matching points.
For point matches between $f$ and $g$, we calculate the Euclidean distances of points and construct a normalized histogram with six bins. We append the total number of matches as the seventh value, which is fed into an MLP to construct the second vector.
The distance thresholds of these bins are set at (0, 1e-3, 5e-3, 1e-2, 5e-2, 1e-1, $\infty$).
The verifier is optimized using the standard binary cross-entropy loss.

### 3.4 Auto-agglomerative inference

The denoiser and the verifier iteratively align and merge fracture fragments into larger groups. This process repeats six iterations or until all the fragments are merged into a single component. Fragment pairs are verified for merging if their classification score exceeds a threshold of 0.9.
One challenge in fragment merging is the potential inclusion of points from inner surfaces when taking the union of the point-clouds. We employ simple heuristics to detect and remove such inner points before merging. Specifically, we compute a surface normal for each point of a point-cloud using estimate_pointcloud_normals in the pytorch3d.ops module. A point is identified as an inner surface point if another point in the opposing point-cloud is within a distance of 0.001 and their surface normals yield a negative dot product. A point-cloud of a fragment contains 1,000 points. After removing inner surface points, we resample to maintain 1,000 points using the farthest point sampling method (§4). For anchor fragments, we preserve them as separate and freeze their alignments rather than merging to retain the high-resolution information crucial for large (e.g., anchor) fragments.

## 4 Experiments

We use a server with four NVIDIA RTX A6000 GPUs for experiments.
The denoiser is trained for 2000 epochs with a batch size of 64. The initial learning rate is 2e-4 and decays by a factor of 10 at 1200 and 1700 epochs. The AdamW optimizer is used with a decay factor of 1e-6.
The verifier is trained
for 100 epochs using the same training settings as the denoiser.
The training of the denoiser and the verifier takes approximately 75 hours and 10 minutes.

Datasets.
Following recent works [, ], we use the Breaking Bad dataset []. Specifically, 34,075 assemblies from 407 objects in the everyday subset are for training. 7,679 assemblies from 91 objects in the everyday subset and 3,651 assemblies from 40 uncategorized objects in the artifact subset are for testing. We limit assemblies with at most 20 fragments and do not use the assembly category labels (e.g., bottles, vases, and mirrors). Note that the training/testing split is exactly the same as Breaking Bad dataset [] for fair comparisons.

Evaluation metrics.
The Breaking Bad dataset offers three evaluation metrics, where the anchor fragment aligns the predicted assembly and the ground truth before the metric calculation:

$\bullet$ Root mean square error (RMSE)
of both the rotation and the translation parameters.

$\bullet$ Part accuracy (PA) is the ratio of fragments whose per-fragment Chamfer distance is less than 0.01.

$\bullet$ Chamfer distance (CD)
is calculated per assembly.

Pre-processing. At test time, we apply random rotations to all the fragments and move the center-of-mass of each fragment to the coordinate frame origin to hide all the ground-truth information. At training time, we prepare ground-truth assemblies and their alignment parameters by applying a random rotation to each whole assembly and moving the center-of-mass of its anchor fragment to the coordinate frame origin.

Competing methods.
We include three baselines from the Breaking Bad dataset and three other existing approaches designed for the fracture assembly task:

$\bullet$ Global, LSTM, and DGL are the initial baselines provided by the Breaking Bad dataset.
Global combines the global shape features with per-fragment features and directly regresses the pose.
LSTM focuses on learning cross-fragment relationships, using a bi-directional LSTM to predict pose.
DGL utilizes the graph neural network to find the relationship between fragments.
These methods are originally designed for PartNet [] assembly tasks.

$\bullet$ SE(3)-Equiv [] integrates equivariant and invariant features to model multi-part correlations. The released code only trains and tests on samples with at most 8 fragments. We train and test their system for assemblies with at most 20 fragments for fair comparison.

$\bullet$ DiffAssemble [] uses the equivariant encoder to extract per-fragment features similar to SE(3) and uses a diffusion model to predict the pose.

$\bullet$ Jigsaw [] is the state-of-the-art fracture assembly method, leveraging hierarchical features of global and local geometry to do fracture surface point matching and recover the poses.

**Table 1: Quantitative evaluations on the Breaking Bad dataset. The numbers for the five baselines (marked as $*$) are copied from their papers, except for the CD metric of Jigsaw which we calculated by running their official codebase. SE(3)-Equiv reported numbers for assemblies with at most 8 fragments in their paper, where we modified their codebase to calculate the metrics under our setting (i.e., for assemblies with at most 20 fragments). The running speed is measured on a single RTX NVIDIA 4090 GPU by running the official codebase if available. The cyan and the orange texts denote the best and the second best results, respectively.**
| Method | RMSE(R) $\downarrow$ | RMSE(T) $\downarrow$ | PA $\uparrow$ | CD $\downarrow$ | Speed |
| --- | --- | --- | --- | --- | --- |
|  | degree | $\times 10^{-2}$ | $\%$ | $\times 10^{-3}$ | ms/sample |
| Trained and tested on the everyday subset |  |  |  |  |  |
| Global[, ]^∗ | 80.7 | 15.1 | 24.6 | 14.6 | 23.7 |
| LSTM[]^∗ | 84.2 | 16.2 | 22.7 | 15.8 | 29.9 |
| DGL[]^∗ | 79.4 | 15.0 | 31.0 | 14.3 | 28.7 |
| SE(3)-Equiv[] | 79.3 | 16.9 | 8.41 | 28.5 | 129.9 |
| DiffAssemble[]^∗ | 73.3 | 14.8 | 27.5 | - | - |
| Jigsaw[]^∗ | 42.3 | 10.7 | 57.3 | 13.3 | 1063.5 |
| PuzzleFusion++ (Ours) | 38.1 | 8.04 | 70.6 | 6.03 | 928.5 |
| Trained on the everyday subset, tested on the artifact subset |  |  |  |  |  |
| Jigsaw[]^∗ | 52.4 | 22.2 | 45.6 | 14.3 | - |
| PuzzleFusion++ (Ours) | 52.1 | 13.9 | 49.6 | 14.5 | - |

### 4.1 Quantitative evaluations

Table 1 demonstrates that PuzzleFusion++ outperforms all the six baselines across all the metrics with significant margins, except one metric in one case to be discussed below.
Among the six baselines, Jigsaw is a clear winner that boasts a learning-based point matcher, leveraging global and local geometry.
However, Jigsaw is not a fully neural system, relying on classical optimization in searching for a compatible set of point matches. PuzzleFusion++ is fully neural, directly searches for a compatible alignment of fragments with a powerful diffusion model, and repeats the process many times as confident partial alignments are merged into bigger fragments, which is precisely how humans would solve the task.

The lower section of Table 1 evaluates the generalization capabilities of Jigsaw and PuzzleFusion++. Both models are trained on the everyday subset and tested on the artifact subset.
Jigsaw marginally surpasses PuzzleFusion++ in the CD metric (14.5 vs 14.3), where PuzzleFusion++ suffers from a major performance decline across all metrics compared to Jigsaw.
Despite apparent superior generalization by Jigsaw,
we argue that this is a side-effect of powerful global geometry learning by PuzzleFusion++, which learns what “everyday” objects are at training and fail on “artifact” objects at test time. Jigsaw focuses more on local geometry learning which is less affected by the change of an object category.

Figure: Figure 4: Qualitative comparisons on the Breaking Bad dataset. We normalize the assembled objects for cleaner visualization so the same fragment can have different sizes in each row. For each result, we show the number of successfully assembled fragments versus the total number of fragments (*i.e*., the part accuracy metric). Please see the Appendix for additional results.
Refer to caption: x4.png

Table: Table 2: Ablation study on the number of iterations.

Figure: Figure 5: Visualization of the assembly process in the first three auto-agglomerative iterations. We show two challenging examples with more than 10 fragments. The number of successfully assembled fragments increases as the system runs for more iterations.
Refer to caption: x5.png

### 4.2 Qualitative evaluations

Figure 4 visualizes the assembly results of PuzzleFusion++ and baselines for representative examples. The appendix provides many more results, organized by the number of fragments in each assembly without cherry-picking to provide a broad and unbiased selection.
For each result, we put the number of correctly assembled fragments and the number of total fragments.
We show two easy examples with less than 6 fragments in the first two rows, where Jigsaw obtains similar results to ours. The rest of the figure presents challenging examples with more than 10 fragments. As the number of fragments increases, all baselines have trouble making reasonable assembly while PuzzleFusion++ produces much more accurate and reasonable results. Specifically, the potential ambiguity in local geometric features could deteriorate the performance of Jigsaw when there are too many small fragments, while our approach stays robust by exploiting high-level geometric reasoning in the auto-agglomerative process.
Please refer to Appendix B for direct comparisons between Jigsaw and our PuzzleFusion++, and the supplementary video for the animations.

### 4.3 Ablation study

Iterations of the auto-agglomerative process. Table 2 presents the ablation study on the number of auto-agglomerative iterations (#$ite$):
The second row (#$ite$=1) demonstrates that
our denoiser alone performs better than the other baselines (Table 1).
DiffAssemble[] also uses diffusion models but performs much worse, indicating the effectiveness of our feature embeddings (§3.2) and system-level designs (Figure 3).
The rest of the table shows that our performance consistently improves with more auto-agglomerative iterations
(*i.e*., iterating the denoiser and the verifier). The iterations are particularly important for complex assemblies with many fragments, as shown by two hard examples
in Figure 5. The verifier finds a few small fragments correctly aligned in the early iterations, which simplifies the problem for the denoiser and helps complete the assembly in the later iterations.

Sampling steps of the denoiser.
Table 3 shows how performance varies with different numbers of sampling steps when only the denoiser is used during testing. The optimal performance is achieved with 20 sampling steps, yet performance remains robust even with fewer steps. Accelerated sampling in diffusion models[] is an active area of research, leaving good potential for our approach also to achieve a boost in speed.

### 4.4 Failure cases and limitations

Figure 6 shows two failure modes due to
1) local geometric ambiguity and 2) small fracture surfaces.

$\bullet$ Local geometric ambiguity (left four examples) result in the misplacement of fragments
such as the red and yellow fragments on a mug handle in the first column, the green and yellow bowl fragments in the second column, and an upside-down bottle in the fourth column. The third column presents an exceptionally difficult scenario with 20 fragments. As the number of fragments increases, the likelihood of encountering fragments with similar local geometries also rises. Nonetheless, our method uses global shape priors to assemble all fragments into a coherent global shape.

$\bullet$ Small fracture surfaces (right two examples) make it hard to assess the fitness of fragments.
In the right examples, the bases of two objects are successfully reassembled but fail to connect to the main bodies due to the thin structures serving as connections. Specifically, the right-most column illustrates a 180-degree rotation error in the base of the wine glass.

Figure: Figure 6: Two popular failure modes are local geometric ambiguity (left four examples) and small fracture surfaces (right two examples).
Refer to caption: x6.png

## 5 Conclusion

This paper introduces PuzzleFusion++, an advanced framework for 3D fracture assembly. Our key contributions are a fully neural auto-agglomerative design that simulates human cognitive strategies for puzzle solving and a diffusion model enhanced with feature embedding designs that directly estimate 6-DoF alignment parameters.
Extensive quantitative and qualitative experiments show that PuzzleFusion++ outperforms existing methods with significant margins on the Breaking Bad dataset. Future work will focus on increasing inference speed and scaling the approach to more complex assembly tasks involving up to 100 fragments.

Social impacts.
PuzzleFusion++ brings positive social impacts. In archaeology, it can aid in reconstructing ancient artifacts and preserving cultural heritage. In medicine, it can enhance preoperative planning by accurately reconstructing fractured bones from CT scans. Forensic science can benefit from more precise reconstruction of evidence from accident scenes, aiding in criminal investigations. Additionally, in scientific research, PuzzleFusion++ can improve the structural analysis of proteins, potentially accelerating drug discovery. On the negative side, automation of such complex tasks, traditionally involving human cognitive and manual skills, could lead to the displacement of jobs.

## Appendix A Additional Implementation Details

This section presents the remaining implementation details that are omitted in the main paper.

### A.1 Details of the Fragment Autoencoder

We employ PointNet++ for self-supervised pre-training of the autoencoder, using a Single-Scale Grouping PointNet++ encoder $E$ to aggregate features.
Each fragment is represented by a point cloud with 1000 uniformly sampled points. The point cloud is normalized so
the center of mass is at the origin and the longest
dimension of the axis-aligned bounding box becomes a unit length.
The point cloud is initially encoded into 25 distinct point latent vectors representing local features with dimension 64. Each point latent $z_{p}$ is associated with the local center coordinates $c_{p}$.

We apply vector quantization (VQ) for regularization. We use a codebook containing 1024 embeddings, each with 16 dimensions. The encoder outputs point latents mapped to 4 codebook entries. Then, these 4 codes, each with 16 dimensions reshaped back to 64 dimensions, become the regularized local point embedding $z_{p}$.

The decoder $D$ takes the point latent $z_{p}$ and its corresponding local center coordinates $c_{p}$. Then, the MLP layer reconstructs each local latent vector to a local point cloud. We then shift all local reconstructed point clouds based on the corresponding local center positions. The reconstructed points are the union of all local reconstructions:

$$ $P^{\text{rec}}\leftarrow\{D(z_{p})+c_{p}\mid p\}.$ $$

We follow the training objectives of VQ-VAE, while using bidirectional Chamfer Distance for the reconstruction loss between the original fragment point cloud $P$ and the reconstructed $P^{\text{rec}}$:

$$ $\text{Loss}=\text{CD}(P_{p},P_{p}^{\text{rec}})+\|\text{sg}[E(c_{p})]-z_{p}\|^ {2}_{2}+\beta\|E(c_{p})-\text{sg}[z_{p}]\|^{2}_{2},\forall(p)$ $$

The parameter $\beta$ is set to $0.25$.
After training the autoencoder, we keep only the encoder and codebook for our SE(3) denoiser to encode the fragment shape. During denoiser training, the weights of the encoder and codebook are frozen.

### A.2 Details of noise scheduler

We have presented our noise scheduler in Figure 3 of the main paper. Similar to humans solving jigsaw puzzles, accurately aligning local fracture surfaces is usually more challenging than knowing the rough location of a fragment.
Therefore, we allocate more denoising budgets to getting precise alignments than moving fragments to the rough locations by designing the piecewise function.
Figure 7 shows the comparison between the commonly used linear scheduler, cosine scheduler, and our customized scheduler.

Figure: Figure 7: Our customized noise scheduler.
Refer to caption: x7.png

## Appendix B Additional Experimental Results and Analyses

We provide additional qualitative results based on the number of fragments: Figure 8 for 2-5 fragments, Figure 9 for 6-10 fragments, Figure 10 for 11-15 fragments, and Figure 11 for 16-20 fragments.
In the main paper, we have demonstrated that Jigsaw is the primary competing method to our method, so we exclude other baselines to save space.

The additional qualitative results show that Jigsaw yields good results on simple objects with fewer than 6 fragments. When the number of fragments increases, PuzzleFusion++ consistently surpasses Jigsaw. For both approaches, the performance declines as the number of fragments increases, potentially attributed to the local geometric ambiguity discussed in §4.4 of the main paper.

Figure: Figure 8: Additional qualitative results for 2 to 5 fragments.
Refer to caption: x8.png

Figure: Figure 9: Additional qualitative results for 6 to 10 fragments.
Refer to caption: x9.png

Figure: Figure 10: Additional qualitative results for 11 to 15 fragments.
Refer to caption: x10.png

Figure: Figure 11: Additional qualitative results for 16 to 20 fragments.
Refer to caption: x11.png

## Peer Reviews

confidence: 4
contribution: 3
initial_rating: 6
presentation: 3
questions: - Why was the number of iterations set to 6? Table 2 shows that the results are best at # iterations = 6. At what number of iteration does PuzzleFusion++ achieve the best results, without further increase in performance with increasing number of iterations?

- In Table 4, what are the results when the scheduler is x (i.e., not the proposed scheduler), while the Autoencoder and Anchor fragment are being used? It would be helpful to include this result in the ablation, for improved clarity of the ablation results.

- Figure 9 visualizes the difference between 3 noise schedulers - could the authors include a quantitative/qualitative comparison of all 3 noise schedulers? The proposed rationale of "locating more denoising budgets to getting precise alignments than moving fragments to the rough locations" sounds intuitive, and it would certainly help to have additional results to validate this claim, beyond the results in Table 4. 

The idea of the proposed PuzzleFusion++ is interesting and novel, and also shows strong empirical results. While the paper can be still improved by including more recent baselines and providing more comprehensive results, I believe that the strengths of the paper outweigh its weaknesses as-is. I am leaning towards accept, and am willing to improve my score if my questions / weaknesses are addressed.
soundness: 3
strengths: - Novel approach ideated as 'auto-agglomerative', which has been implemented through the integration of a diffusion model (SE3 denoiser) and a transformer model (pairwise alignment verifier). 

- The manuscript is overall well-written and easy to follow.

- Comprehensive analysis experiments, which validates the design choices of PuzzleFusion++ and clarifies how each design choice affects the performance.

- Strong performance on the Breaking Bad benchmark, outperforming existing methods by a large margin.
summary: This paper proposes PuzzleFusion++, an "auto-agglomerative" method for the task of 3D fracture assembly.
Specifically, the proposed pipeline undergoes a iterative process of using a diffusion model to predict a 6-DoF alignment for each fragment, followed by a transformer model which verifies and merges pairwise alignments into larger ones. 
This is much alike how humans assemble fragments - hypothesizing how two fragments fit together, and checking if it the alignment is indeed ture. 
To encode fragments into latent vectors suitable for diffusion training, the authors integrate PointNet++ and VQVAE.
PuzzleFusion++ achieves state of the art on the Breaking Bad dataset by a large margin, and comprehensive analysis clarifies the significance of each introduced module.
weaknesses: - While Figure 3 provides valuable details into how the SE3-denoiser and Pairwise Alignment Verifier work, it is not very visual, and not straightforward to understand as-is. I believe providing Figure 3 as a form of an algorithm may improve its readability.

- The authors mention that "different anchor initialization does not affect the quality of the assembly results (L482)"; however, it can be seen that the results in Table 7 is closer to the results for Ours(#ite=1) in Table 2. I believe it is an overstatement to mention that anchor initialization 'does not affect' the quality of the assembly results - unless a single iteration (#iteration=1) was used for the results in Table 7. It would be informative to provide the results for varying number of iterations for random initializations of the anchor fragment.

- The paper is missing a recent baseline for 3D assembly, which seem to show strong performances on the Breaking Bad benchmark as well: 
Nahyuk Lee et al, 3D Geometric Shape Assembly via Efficient Point Cloud Matching, ICML 2024.


- Minor writing mistakes: 
1) Table 4: Autoncoder -> Autoencoder 
2) L 454: Denioser -> Denoiser

---

confidence: 5
contribution: 2
initial_rating: 6
presentation: 3
questions: - L150-L154 what guarantees that the input representation are equivariant (aka "sensitive to rotation"). Normalizing wrt center mass achieves invariance, but not necessarily equivariance.
soundness: 2
strengths: - Authors' proposal is easy to follow and well-thought
- Noise scheduler improves performances by a surprising margin. Since different noise schedulers do not change training dynamics and effects only MC estimate of the loss (Kingma et al. 2024), my guess it's that this scheduler makes training more efficient. I'd like to see more comments on that, either in main paper or supplementary. Right now it's relegated to a small section in supplementary and not discussed in main paper.
- Good ablations section
- Method is somewhat novel. There's no technical novelty since it adopts existing methods and approaches (iterative refinement has been adopted by Ken et al, diffusion model formulation was in Scarpellini et al). Still the combination of these modules make outperform the proposed baselines so I would not consider this a weakness

Kim, Jinhyeok, Inha Lee, and Kyungdon Joo. "Fracture Assembly with Segmentation And Iterative Registration." ICASSP 2024-2024 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP). IEEE, 2024.

Kingma, Diederik, and Ruiqi Gao. "Understanding diffusion objectives as the elbo with simple data augmentation." Advances in Neural Information Processing Systems 36 (2024).
summary: Authors improve over existing work on 3d fracture assembly by proposing an iterative approach. The proposed method iteratively denoised poses for current fragments and merge aligned fragments into clusters until all fragments are merged into a single 3D object. They build over existing work (Scarpellini et. al, Wu et al.) regarding the diffusion model formulation for alignment. The novelty regards adopting a transformer for clustering fragments.
weaknesses: - L158  anchor fragments were introduced in Jigsaw (Lu eta al) and are not adopted in original BreakingBad. Comparing anchor-based methods to methods that do not adopt anchors is a bit unfair. A fair comparison would entail re-training those baselines with anchors (e.g., L212-214 regarding alignment could be adopted in all other baselines). I believe this point makes the experimental section weaker and does not reflect the actual performances of the baselines.
- Missing baselines: authors should also compare to other existing methods that were published before ICLR deadline: PHFormer: Multi-Fragment Assembly Using Proxy-Level Hybrid Transformer (Cui et al, AAAI2024), Fracture Assembly with Segmentation And Iterative Registration (Kim et al., ICASSP2024). Both achieve impressive results and are not cited by the authors--especially the latter achieves RMSE R 23.8, RMSE T 7.30, PA 74.50.

---

confidence: 4
contribution: 3
initial_rating: 6
presentation: 3
questions: 1.	The main difference between PuzzleFusion and PuzzleFusion++ should be clarified in the paper, since it is a future work from PuzzleFusion.
2.	Are 25 points enough to represent a fragment in Sec.3.1?
3.	Does the pairwise alignment verifier work well for all objects?
4.	In auto-agglomerative inference, are 6 iterations enough for merging? Have you try more iterations?
5.	Please discuss could reinforcement learning whether is help for this task.
6.	As mentioned in Weakness, could you provide more complicate objects during rebuttal?
7.	Please discuss the possible solution for your limitations.
soundness: 3
strengths: 1.	This paper is well-written.
2.	The motivation is clear enough.
3.	The organization of this paper is great.
4.	The alignment verifier is well designed.
summary: This paper proposes PuzzleFusion++, a framework for 3D fracture assembly. A fully neural auto-agglomerative design is proposed that simulates human cognitive strategies for puzzle solving. Moreover, a diffusion model enhanced with feature embedding designs is devised that directly estimate 6-DoF alignment parameters.
weaknesses: 1.	The Figure 3 is a little puzzled. A better format is recommended.
2.	The showed 3D objects seems relatively simple. More complicate objects from Objaverse are commended.

---

confidence: 5
contribution: 2
initial_rating: 6
presentation: 3
questions: 1. With the name of this paper, it’s hard not to compare it with PuzzleFusion that have a similar motivation in tackling jigsaw puzzle related problem. Could PuzzleFusion be adapted to handle 3D tasks as a baseline for comparison? What distinguishes PuzzleFusion++ from PuzzleFusion, particularly in features that enhance its suitability for 3D tasks?

2. The matching from Jigsaw is not perfect, is there a scheme to handle such situation? Meanwhile, the provide upper bound based on perfect matching is only 34 degree error. This is still very high. If Jigsaw can have such perfect matching, the error can be reduced to less than 10 degrees. This raises questions about the suitability of using a diffusion model for this type of problem.

3. Even for comparison among diffusion models, why PuzzleFusion++ can significantly outperforms DiffAssemble? Can the authors describe the key difference that make your model works better?
soundness: 2
strengths: The paper demonstrates performance improvements over previous work. I like the idea of using a verifier to assess the quality of the assembly results.
summary: This paper introduces PuzzleFusion++, which addresses the 3D fracture assembly problem. It employs a diffusion model enhanced by a proposed autoencoder, optimized denoise scheduling, denoiser structure, and an auto-agglomerative verifier. These advancements collectively contribute to performance that surpasses that of previous work.
weaknesses: Despite significant efforts on ablation studies of the proposed method, the paper does not clearly differentiate from prior works. A valuable contribution would be for the authors to provide (component level) comparisons with similar methods, such as PuzzleFusion and DiffAssemble. It would be beneficial to highlight that what limits the existing approaches and what contributes to the success of the proposed one. For now, the uniqueness of this paper is not clear.

---

confidence: 5
contribution: 3
initial_rating: 8
presentation: 3
questions: good paper, but i still have quesitons. please see the weaknesses above 

minor question: which software, library is used to render images? the rendering is very professional
soundness: 3
strengths: 1. the proposed method makes sense and I believe this is the way to solve this problem as in doing some clustering first and then merging them (unlike prior work which tries to predict a pose for each fragment in one single pass).

2. performance is good compared to state-of-the-art methods. this means that the proposed method is effective

3. presentation is good and the paper is easy to follow
summary: This paper looks at the task of object reassembly in which a set of fractures are given and the goal is to assemble them into an object. This is a challenging problem and has applications in computer vision, computer graphics and robotics. The paper proposes a method called puzzlefusion++ which solves the problem by aligning and merging the fragments into larger groups (think of this as a clustering processing) and then iteratively completing the assembly. Results show that the proposed method make some improvements over the competing methods.
weaknesses: 1. speed is significantly slower than some methods (see table 1)

2. can the proposed method solve the reassembly task when the number of fractures is big (eg 100)

3. how well can one expect the proposed method to work on unseen objects?

Meta-review: This paper proposes a 3D fracture assembly method that learns a diffusion model to predict a 6-DoF alignment for each fragment iteratively, followed by a transformer model that verifies and merges pairwise alignments into larger ones. Unlike previous diffusion-based models, it simulates how humans assemble fragments gradually and check the validity of the alignment. To encode fragments into latent vectors suitable for diffusion training, it integrates PointNet++ and VQVAE. The proposed method achieves state-of-the-art on the Breaking Bad dataset by a large margin, and comprehensive analysis clarifies the significance of each introduced module.
All reviewers appreciated the novelty of the proposed auto-agglomerative approach and its comprehensive analyses. The main concerns raised by reviewers were unclear exposition, unfair experimental setups, and missing comparisons. The authors’ detailed rebuttal addressed most of them, resulting in unanimous acceptance at the end of the discussion. AC thus recommends acceptance.

Decision: Accept (Poster)