Jigsaw++: Imagining Complete Shape Priors for Object Reassembly

## Abstract

Abstract The automatic assembly problem has attracted increasing interest due to its complex challenges that involve 3D representation. This paper introduces Jigsaw++, a novel generative method designed to tackle the multifaceted challenges of reconstruction for the reassembly problem. Existing approach focusing primarily on piecewise information for both part and fracture assembly, often overlooking the integration of complete object prior. Jigsaw++ distinguishes itself by learning a category-agnostic shape prior of complete objects. It employs the proposed “retargeting” strategy that effectively leverages the output of any existing assembly method to generate complete shape reconstructions. This capability allows it to function orthogonally to the current methods. Through extensive evaluations on Breaking Bad dataset and PartNet, Jigsaw++ has demonstrated its effectiveness, reducing reconstruction errors and enhancing the precision of shape reconstruction, which sets a new direction for future reassembly model developments.

## 1 Introduction

The challenge of object reassembly spans numerous applications from digital archaeology to robotic furniture assembly, and even to the medical field with fractured bone restoration. Object reassembly problems are classified into part assembly, which deals with semantically significant parts (Zhan et al., ; Schor et al., ; Li et al., ; Wu et al., ; Dubrovina et al., ), and fractured assembly, which handles pieces broken by substantial forces (Huang et al., ; Lu et al., ; Wu et al., ). However, existing methods, which primarily focus on performing piece-wise matching, typically suffer from incomplete information about the pieces of the object and do not capture the complete original object. In particular, in real settings, we usually observe a very limited number of pieces of the original complete object. Reconstruction is heavily based on prior knowledge of the complete object, c.f. (Thuswaldner et al., ; Papaioannou et al., ). These limitations underscore the need for a new approach that could address these gaps and provide a complete shape prior to future research.

Our proposed solution, Jigsaw++, explores how to leverage knowledge of the complete object to “imagine” its full structure when presented with a partial and/or inaccurate assembled input. Although some existing fractured assembly methodologies (Yin et al., ; Zhang et al., ; Deng et al., ) utilize a complete shape prior, they are mainly confined to specific categories or assume that the complete shape is known beforehand. Such constraints are too restrictive for addressing a broader spectrum of assembly problems.

Our approach draws inspiration from the recent success of 3D shape generators employing diffusion models, which map Gaussian noise to instances on the data manifold. Based on this principle, we propose to learn a complete shape prior through the generative model, then optimize the mapping from the partially assembled input towards this complete shape space. Ideally, this method will provide a realistic representation of what the complete object would look like based on the given input. Among many 3D representations, we focus on the point-cloud representation, due to its tight connection to acquisition devices and training data, e.g., Break Bad (Sellán et al., ) and PartNet (Mo et al., ).

Learning a point cloud generative model for fractured object reassembly is difficult. Most approaches require a fixed number of points and are also restricted to specific categories or need class conditioning. Another challenge is the scale of training data for learning shape priors. We overcome these challenges by adopting the LEAP image-to-3D reconstruction model (Jiang et al., ) under the point cloud representation. Our goal is to leverage its training on broad 2D datasets (Oquab et al., ). by developing a suitable mapping between raw point clouds and RGB images.

Figure: Figure 1: Overview of the problem setting. The input consists of a partially assembled object represented as a point cloud. The task requires the method to reconstruct a complete object from this input. We identify several representative challenges: (a) When the object is nearly fully assembled, the output should maintain the overall shape. (b) Although all parts are visible and present, their positions are misaligned. The algorithm needs to adjust their positions correctly. (c, d) In cases where parts are incomplete or significantly misplaced, the method should not only complete the object but also correct the displacements.
Refer to caption: /html/2410.11816/assets/x1.png

Drawing insights from contemporaty image editing approaches (Song et al., ; Mokady et al., ; Meng et al., ), our model interprets the partially assembled object as user input, with the target being the complete object. This setup helps to utilize the learned shape generative model to predict the complete object from partial inputs. However, the difference in our setting is that the input is inaccurate and incomplete. Naively conditioning the output on the input still leads to inaccurate output. To address this issue, we introduce a “retargeting” phase which fine-tunes the mapping from the encoding of inaccurate input to the complete object output. This fine-tuning step significantly improves reconstruction quality.

In summary, our main contributions are as follows.

- •
We introduce Jigsaw++, a novel method that imagine the complete shape prior through retargeted rectified flow for addressing the object reassembly problem.
- •
We develop an object-level point cloud generation module capable of adapting to a large or arbitrary size of input and output point numbers. This model leverages the image-to-3D model and encompasses a joint generation of global embeddings and reconstruction latent via the rectified flow technique.
- •
The proposal of a “retargeting” strategy that links the reconstruction challenges in reassembly tasks with guided generation processes. This strategy facilitates the reconstruction of complete objects from partially assembled inputs and takes advantage of the straightness provided by rectified flow, resulting in lower tuning costs and higher flexibility.
- •
Jigsaw++ works orthogonally to the existing object reassemble methods. Our experiments on both the Breaking Bad dataset and PartNet demonstrate its adaptability to various assembly challenges and its ability to achieve significant improvements over baseline inputs.

## 2 Related Work

### 2.1 Object Reassembly

Object reassembly problem falls into two primary categories: part assembly and fractured assembly. In part assembly, semantic-aware learning methods have emerged in recent years. Specific tools designed for the assembly of CAD mechanics have been developed (Jones et al., ; Willis et al., ). For the assembly of categorical everyday objects, research efforts (Schor et al., ; Li et al., ; Wu et al., ; Dubrovina et al., ) have concentrated on generating missing parts based on an accumulated shape prior to completing the entire object, although this approach can lead to shape distortions relative to the input parts. More recent works (Zhan et al., ; Harish et al., ; Li et al., ; Du et al., ) learns the part positions directly through regression or generative methods. However, these methods require the input objects to be semantically decomposed in a consistent manner and necessitate specific training for each object category.

The fractured assembly problem specifically addresses objects broken by extreme external forces. Previous research in this area typically falls into two categories: assembly based on fracture surface features or complete shape template. The former approach focuses on detecting fractured surfaces and extracting robust descriptors, with early work (Ruiz-Correa et al., ; Gelfand et al., ; Salti et al., ; Huang et al., ) employing hand-crafted features for assembly. More recent learning-based techniques have introduced methods (Chen et al., ; Wu et al., ; Lu et al., ; Scarpellini et al., ) using learned features for matching local geometries, or predicting or generating piece positions. Another significant limitation of existing approaches is that they require that most of the fragments be available as input. However, this assumption is violated in real settings where a significant potion of fragments is missing (Thuswaldner et al., ; Papaioannou et al., ), in which prior knowledge of the complete object is critical.

Existing approaches that use information of complete shapes are template-based methods (Yin et al., ; Zhang et al., ; Deng et al., ). However, they often assume a specific complete shape for assembly, but are typically constrained by specific categories or challenges in generating accurate shape priors. Such settings do not apply to general-purpose fracture object reassembly.

### 2.2 3D Object Generation

The field of 3D shape generation has witnessed significant progress, driven by the application of various generative models that produce high-quality point clouds and meshes.
Techniques such as variational autoencoders (Yang et al., ; Gadelha et al., ; Kim et al., ) and generative adversarial networks (GANs) (Valsesia et al., ; Achlioptas et al., ) have been widely implemented to process 3D data.
Further enhancements have been achieved through the integration of normalizing flows and diffusion models, which have spurred the development of state-of-the-art approaches (Yang et al., ; Kim et al., ; Zhou et al., ; Luo & Hu, ; Zeng et al., ; Lyu et al., ; Wu et al., ; Mo et al., ; Zhang et al., ; Gao et al., ). People also studied using 2D images and implicit neural fields to create text-guided 3D shapes (Xu et al., ; Ruiz et al., ; Lin et al., ; Cheng et al., ). Some approaches (Zhou et al., ; Lyu et al., ) also explored the generative shape completion which is highly relative to our task. These techniques strive to generate point clouds, SDFs, and meshes with both high fidelity and diversity, with some employing latent-based generation to even support multimodal 3D generation.

Our approach adopts comparable results in this space and addresses two fundamental challenges in point cloud generation. The first challenge is limited paired 3D data we have for learning a shape prior. Our approach develops a mapping between point clouds and RGB images, allowing us to use pretrained models that take 2D images as the input. The second challenge is point clouds with varying number of points. We again address this issue using the mapping between RGB images and point clouds, which enable us to generate 3D point clouds with many more points than prior approaches.

### 2.3 Diffusion Model and Rectified Flow

Our approach uses state-of-the-art diffusion-based techniques for learning the shape prior and the mapping from inaccurate input to complete object output. Diffusion models (Ho et al., ; Song et al., ; Dhariwal & Nichol, ; Zhang et al., ; Podell et al., ; Song et al., ) have demonstrated their versatility and effectiveness in a variety of generative tasks, including image, audio, and video generation (Saharia et al., ; Kong et al., ; Ho et al., ). These models operate via a forward process that incrementally adds Gaussian noise, coupled with a reverse process that gradually restores the original data, thus achieving high fidelity in the generated outputs. Beyond stochastic differential equation (SDE)-based approaches (Song et al., ; ), recent efforts have emerged (Liu et al., ; Liu, ; Lipman et al., ; Albergo et al., ) focusing on directly learning probability flow ordinary differential equations (ODEs) between two distributions. This shift has led to improvements in generative efficiency and quality. Specifically, the introduction of Rectified Flow (Liu et al., ; Liu, ) implements a reflow process that significantly speeds up the generation process, which is effective in large-scale image generation (Esser et al., ; Liu et al., ). These collective advances highlight the transformative impact of diffusion models in various generative modeling tasks. This work focuses on developing a fractured object reassembly approach that uses these generative models under novel 2D-3D representations.

## 3 Problem Statement and Approach Overview

We begin with the problem statement of Jigsaw++ in Section [3.1](#S3.SS1). Section [3.2](#S3.SS2) then presents an overview of Jigsaw++.

### 3.1 Problem Statement

Denote a collection of $n$ pieces as $\mathcal{P}=\left\{P_{1},P_{2},\cdots,P_{n}\right\}$, represented as point clouds of the surface of each piece. An assembly algorithm (e.g., Zhan et al. (); Lu et al. ()) produces a set of 6-DoF poses $\left\{T_{1},T_{2},\cdots,T_{n}\right\}$. These poses, derived from existing methods, partially restore the underlying object $\hat{O}=T_{1}(P_{1})\cup T_{2}(P_{2})\cup\cdots\cup T_{n}(P_{n})$, where $T_{i}(\cdot),1\leq i\leq n$ is an operator that applies the transformation $T_{i}$ to piece $P_{i}$.
The objective of the proposed task is to infer a possible set of complete 3D shapes $\mathcal{S}=\left\{S_{1},S_{2},\cdots,S_{k}\right\}$ based on $\hat{O}$ that share a similar outer shape with the original object $O$. Importantly, we aim for a data-driven approach where the complete restorations may contain geometries not present in the input. Fig. [1](#S1.F1.fig1) provides a comprehensive overview of this problem.

Figure: Figure 2: Intuitive methods, including point cloud completion method AdaPoinTr (Yu et al., ), LION (Zeng et al., ) VAE’s reconstruction, and editing method SDEdit (Meng et al., ), fails in providing shape prior when given partially assembled object.
Refer to caption: /html/2410.11816/assets/x2.png

To clearly establish the scope of this problem, we elucidate the following key aspects: (1) The input is the partially assembled objects from a prior algorithm, represented as point clouds. The state of this partially assembled object is not provided. There is no quantification of whether a piece is correctly assembled or how accurate the assembling is. (2) The output is a complete shape prior in point cloud form. This prior is not required to exactly replicate the geometric details of the input pieces, aligning with the template shape used in previous works (Yin et al., ; Zhang et al., ; Deng et al., ). However, a more accurate representation of the outer shape is preferred, as reflected in our evaluation metrics. (3) The purpose of this method is not to design a reassembly algorithm, but rather an additional layer of information to improve the reassembly algorithm. (4) Given the absence of prior work addressing this specific problem, we demonstrate how intuitive solutions fail in Fig. [2](#S3.F2.1), highlighting the problem’s difficulty and uniqueness. A detailed analysis of these results is presented in Appendix [A](#A1).

### 3.2 Approach Overview

Jigsaws proceeds in two stages. The first stage learns a generative model to capture the shape space of complete objects. The second stage focuses on“regargeting” which reconstructs the complete shape from partially assembled inputs. Below we highlight the main characteristics of each stage.

#### Learning Complete Shape Priors.

The first stage learns a generative model of point clouds that capture shape prior of the underlying objects. There are many available point cloud generative models (Zhou et al., ; Zeng et al., ; Lyu et al., ). However, there are two fundamental challenges in adopting them for our setting. First, most point cloud generative models are category specific and use a fixed number of points. Therefore, it is difficult to adopt them to learn a category agnostic model that requires different numbers of points capture geometric details of different categories of objects. Second, 3D data is sparse, which is insufficient to learn a category agnostic model to encode the shape space of objects in diverse categories.

Jigsaw++ adopts LEAP (Jiang et al., ), a pretrained multi-image-2-3D model to learn shape priors. LEAP uses DINOv2 features, which are trained from massive image data. In doing so, our generative model uses not only 3D data, but also 2D large-scale data. We introduce a bidirectional mapping between uncolored point clouds and RGB images. This mapping addresses the domain gap between raw 3D geometry and colored inputs to LEAP (as well as many other image-based 3D reconstruction model). It also nicely addresses the issue of having a limited number of 3D points. We will discuss details in Sec. [4](#S4).

#### Reconstruction through Retargeting.

The second stage learns the reconstruction model that takes the assembly result of an off-the-shelf method as input and outputs a complete 3D model. A standard approach is to formulate this procedure as condition generation (Song et al., ; Mokady et al., ; Meng et al., ; Liu et al., ). In the image generation setting, the output is conditioned on user-guided or domain-mixed RGB inputs.

The difference in our setting is that the inputs are biased partially assembled objects, and we do not have quantification of which part of the input is correct and which is not. In contrast, image-based conditions in existing approaches are unbiased complete objects. Due to this distribution shift, if we naively condition the learned generative model on the biased inputs, the resulting 3D shape is also biased. This is because not all latent codes in standard latent spaces correspond to valid 3D shapes. Addressing this issue requires a “retargeting” phase where the model is fine-tuned to understand the disparities between the partially assembled and complete objects.

In addition to fine-tuning, the typical approach for guidance-based generation in diffusion models involves performing reverse sampling, mixing the latent representation with a certain level of noise, and then executing forward sampling (standard generation). As diffusion-based models often require extensive sampling steps, we opt for the rectified flow (Liu et al., ) formulation, which allows for skip-over of steps during inverse sampling, thereby accelerating the fine-tuning process. This necessitates the use of rectified flow as the formulation for our generative model in the first stage. We will discuss details in Sec. [5](#S5).

## 4 Generation on Images-to-3D

This section presents details on how to build a rectified flow based generation model for point cloud generation using an image-2-3D mapping. The generation pipeline is presented in Fig. [3](#S4.F3).

#### Mapping Point Clouds to Images.

The key to our generative mode is a novel bi-directional mapping between point clouds and 2D images. Specifically, consider a normalized point cloud represented as ${\bm{o}}\in[0,1]^{N\times 3}$. Each point ${\bm{o}}_{i}\in[0,1]^{3}$ within this cloud, is associated with a function $f:[0,1]^{3}\rightarrow[0,255]_{{\mathbb{Z}}}^{3}$. This function maps each point ${\bm{o}}_{i}$ to a color vector ${\bm{c}}_{i}\in[0,255]_{{\mathbb{Z}}}^{3}$ in the RGB space, where the mapping process is described by ${\bm{c}}_{i}=f({\bm{o}}_{i})=\lfloor 255{\bm{c}}_{i}\rfloor$. Conversely, an inverse mapping is straightforwardly defined as ${\bm{o}}_{i}=f^{\prime}({\bm{c}}_{i})=\frac{1}{255}{\bm{c}}_{i}$. Please note that, although the color space is treated with integer values in this context, for applications involving image-to-3D reconstruction models, the color values can be maintained as fractional, thereby preserving accuracy throughout the transformation process.

Given a camera pose, the function $f$ defined above maps a point cloud to a colored image by rasterization. Likely, given a colored image under the same color encoding scheme, we obtain a corresponding partial colored point cloud that corresponds to the 2D pixels. The computed point cloud will be further refined through a camera-pixel alignment operation that projects the decoded 3D point onto the ray that connects the pixel and the camera center.

Figure: Figure 3: Generation on image-to-3D. The point cloud (or mesh if presented) is first rendered under specific camera parameters by mapping positions to RGB space. The image-to-3D reconstruction model then encodes these rendered images into both a reconstruction latent and a global latent. A rectified flow model is trained to jointly generate these latents. Subsequently, the generated latents are decoded, rendered, and mapped back to a point cloud.
Refer to caption: /html/2410.11816/assets/x3.png

With an appropriate set of camera poses, we can produce a sequence of images which are fed into the image-to-3D encoder. The resulting latent codes are subsequently decoded and rendered. The rendered colors are then mapped back onto the 3D point cloud using the procedure described above.

#### A category agnostic image encoder.

The point could to image map described above opens the door to employ rich results in multi-view to 3D reconstruction models. Such models are trained from massive datasets. Some of them, including LEAP (Jiang et al., ), use the pretrained DINOv2 (Caron et al., ; Oquab et al., ) feature extractor, which boosts generalizability to novel categories. Jigsaw++ uses LEAP as the image encoder backbone. It provides a global embedding ${\bm{g}}$ from the input images and a reconstruction latent ${\bm{r}}$ for 3D reconstruction, we harness these global embeddings as the desired global latent for our generation model, aiming to simultaneously generate both the global and the reconstruction latents. Although only the latent reconstruction is directly utilized in the decoding phase, the global latent is generated throughout to help the model grasp global information of the input, which is vital for complete object reconstruction for object reassembly.

#### Rectified Flow Generation.

Rectified Flow, as outlined in (Liu et al., ; Lipman et al., ), presents a unified ODE-based framework for generative modeling, facilitating the learning of transport mappings $T$ between two distributions, $\pi_{0}$ and $\pi_{1}$. In our images-to-3D model, $\pi_{0}$ typically represents a standard Gaussian distribution, while $\pi_{1}$ corresponds to the latent output of the image encoder.

The method involves an ordinary differential equation (ODE) to transform $\pi_{0}$ to $\pi_{1}$:

$$ $\frac{dZ_{t}}{dt}=v(Z_{t},t),\text{ initialized from }Z_{0}\sim\pi_{0}\text{ to final state }Z_{1}\sim\pi_{1},$ (1) $$

where $v:{\mathbb{R}}^{d}\times[0,1]\to{\mathbb{R}}^{d}$ represents the velocity field. This field is learned by minimizing the objective:

$$ $\mathbb{E}_{(X_{0},X_{1})\sim\pi_{0}\times\pi_{1}}\left[\int_{0}^{1}\left\|\frac{d}{dt}X_{t}-v(X_{t},t)\right\|dt\right],$ (2) $$

where $X_{t}=\phi(X_{0},X_{1},t)$ is an arbitrary time-differentiable interpolation between $X_{0}$ and $X_{1}$. The rectified flow specifically suggests a simplified setting where

$$ $X_{t}=(1-t)X_{0}+tX_{1}\Longrightarrow\frac{d}{dt}X_{t}=X_{1}-X_{0},$ (3) $$

and the solver

$$ $Z_{t+\frac{1}{N}}=Z_{t}+\frac{1}{N}v(Z_{t},t),\forall t\in\left\{0,\ldots,N-1\right\}/N.$ (4) $$

This linear interpolation facilitates straight trajectories, promoting fast generation, as discussed in (Liu et al., ).

Rectified Flow offers two significant advantages: (1) it avoids assuming a fixed distribution for $\pi_{1}$, thus providing more flexibility in integrating the reconstruction encoder’s learned distribution; (2) the model’s ability to learn linear trajectories expedites both the forward and reverse sampling processes, benefiting the fine-tuning phase outlined in Sec. [5](#S5).

#### Pipeline.

Given a set of 3D objects, our generator learns to generate objects that match the data space of the provided shapes through a three-stage process. In the encode stage, the colored 3D objects are rendered into images following camera settings from Kubric-ShapeNet (Greff et al., ). These images are then fed into DINOv2 (Oquab et al., ) and passed through a 2D-3D mapping layer both pre-trained using LEAP (Jiang et al., ), resulting in two types of latents: a voxel-based reconstruction latent ${\bm{r}}$ and a global latent ${\bm{g}}$ containing categorical information.
The generation stage follows, where a joint latent rectified flow model is trained on the encoded latents. During inference, two latents are jointly generated as described in Eq. [4](#S4.E4). The final stage, decode, involves converting the generated reconstruction latent ${\bm{r}}$ into a neural volume. This neural volume is then rendered and converted into a point cloud, which represents the output of the entire pipeline.

To effectively handle the joint generation of the global and reconstruction latents, we employ the U-ViT (Bao et al., ) framework as our generative backbone. This structure has proven its efficacy in image generation tasks (Bao et al., ; Esser et al., ), affirms its suitability for our application.

## 5 Complete Object Reconstruction

Figure: Figure 4: Reconstruction and retargeting. The reconstruction involves a reverse sampling stage to convert input to a latent. The latent will be perturbed to generate a complete shape. The retargeting is to provide guidance for those latent of low likelihood in $\mathcal{N}(0,I)$.
Refer to caption: /html/2410.11816/assets/x4.png

This section presents the details of the Jigsaw++ reconstruction module. We take inspiration from relevant approaches in image generation which transform user guidance into realistic outputs (Song et al., ; Meng et al., ; Mokady et al., ; Liu et al., ). A common theme begins with inverse sampling based on given guidance, followed by forward sampling (generation) to produce the desired image in the target space.

In the context of the reassembly problem, the partially assembled pieces using an off-the-shelf approach serve as the user-provided guidance
Challenges, however, arise as previously discussed in Sec. [3](#S3). Unlike the 2D case where inputs are assumed accurate, our scenario demands larger adaptations, such as positional adjustments or the handling of non-observable overlapping pieces. These extensive modifications necessitate a targeted fine-tuning stage, which we term “retargeting”.

Given the partially assembled object $\hat{O}$ and its associated latent $\hat{{\bm{x}}}_{1}=(\hat{{\bm{g}}}_{1},\hat{{\bm{r}}}_{1})$ (representing a set of global and reconstruction latents), we can employ a reverse ODE solver to determine the latent $\hat{{\bm{x}}}_{0}$. Since the input is not a naturally assembled complete object, $\hat{{\bm{x}}}_{0}$ is likely to have low likelihood under $\pi_{0}=\mathcal{N}(0,I)$. To adjust this, we apply Langevin dynamics:

$$ ${\bm{x}}_{0}=\alpha\hat{{\bm{x}}}_{0}+\sqrt{1-\alpha^{2}}\xi,\ \xi\sim\mathcal{N}(0,I),$ (5) $$

which moves it to a region of higher likelihood.

Ideally, a subsequent forward sampling from ${\bm{x}}_{0}$ should yield a ${\bm{x}}_{1}$ that accurately represents the learned complete shape space. However, given the significant discrepancies between the input partially assembled object and the target, we find that fine-tuning with data pairs $({\bm{x}}_{0},{\bm{x}}_{1})$ is necessary to more effectively guide our generative model. The objective for this stage is,

$$ $\mathbb{E}_{{\bm{x}}_{0},{\bm{x}}_{1}}\lVert({\bm{x}}_{0}-{\bm{x}}_{1})-v({\bm{x}}_{t},t)\rVert^{2},$ (6) $$

where ${\bm{x}}_{0}$ is computed as Eq. [5](#S5.E5) and ${\bm{x}}_{1}$ corresponds to the ground truth of the complete object.

We again use rectified flow (Liu et al., ; Liu, ) to train this reconstruction module. The efficiency and straightness of the rectified flow is critical; they enable a substantial reduction in the number of steps required during the reverse sampling phase - to just $1/25$ of the original steps - while preserving a faithful latent representation. This efficiency is key to decreasing the fine-tuning cost.

## 6 Experiment and Evaluation

### 6.1 Experiment Setup

#### Dataset.

We use the Breaking Bad dataset (Sellán et al., ) for the fracture assembly problem. The Breaking Bad Dataset encompasses a diverse array of synthetic physically broken patterns for the task of fracture assembly problem. Our experiments were conducted on the everyday subset of this dataset, consisting of 498 models with 41,754 distinct fracture patterns. This subset is segmented into a training set with 34,075 fracture patterns from 407 objects, and a testing set containing 7,679 fracture patterns from 91 objects. The average diameter of the objects in both the training and testing sets is 0.8. The generative model is trained only on the training set to ensure a fair comparison. Categorical information is not provided during the experiments.

For the part assembly problem, we employed PartNet (Mo et al., ), following the approach of previous work DGL (Zhan et al., ) for training and evaluation. PartNet offers a large collection of daily objects with detailed and hierarchical part information. We selected the same three categories as prior work: 6,323 chairs, 8,218 tables, and 2,207 lamps, adhering to the standard train/validation/test splits with the finest level of segmentation used. We independently trained the model on three subsets, ensuring that the validation/test sets were not included in the training set of the generation model.

#### Metrics.

We adopted two types of evaluation metrics to evaluate the performance of our proposed methods. (1) Shape difference. The chamfer distance defined by
$CD(S1,S2)=\frac{1}{|S_{1}|}\sum_{x\in S_{1}}\min_{y\in S_{2}}\lVert x-y\rVert^{2}_{2}+\frac{1}{|S_{2}|}\sum_{y\in S_{2}}\min_{x\in S_{2}}\lVert x-y\rVert^{2}_{2}$,
is used to assess the differences between the ground truth shape, the partially assembled shape, and the reconstructed global shape. (2) Shape accuracy. We follow a similar idea of F-score to define the precision and recall metric as
$\textup{precision}=\frac{1}{|S_{gt}|}\sum_{x\in S_{gt}}\mathbf{1}_{\textsf{Dis}(x,\textsf{NN}(x,S))\leq\eta}$, and $\textup{recall}=\frac{1}{|S|}\sum_{x\in S}\mathbf{1}_{\textsf{Dis}(x,\textsf{NN}(x,S_{gt}))\leq\eta}$,
to evaluate how closely the reconstructed shape matches the ground truth. Here, $\textsf{Dis}(\cdot)$ is a distance function, and $\textsf{NN}(\cdot,\cdot)$ is to find the nearest neighbor of one point in another shape.

#### Baseline Methods.

We compare our methods with state-of-the-art assembly algorithms for the fracture and part assembly problem: SE(3) (Wu et al., ), Jigsaw (Lu et al., ) and DGL (Zhan et al., ). Jigsaw employs 3D feature matching to position pieces based on fractured geometry, while DGL enhances piece poses through graph-based learning. Both methods are open-source with available model checkpoints, which we used to generate the partially assembled inputs for our model and comparison.
Since our algorithm works orthogonally to existing methods, it is sufficient to demonstrate its superiority by demonstrating improvements over these methods.

**Table 1: Quantitative results of baseline methods and Jigsaw++ on the Breaking Bad dataset and ParNet. Jigsaw++ consistently improves performance of the baseline method across all settings.**
| Breaking Bad Dataset |  |  |  |  |  |  |  |  |  |
| --- | --- | --- | --- | --- | --- | --- | --- | --- | --- |
| Method | CD ($\times 10^{-2}$) $\downarrow$ | Precision (%) $\uparrow$ | Recall (%) $\uparrow$ |  |  |  |  |  |  |
| SE(3) (Wu et al., ) | 22.4 | 20.2 | 22.5 |  |  |  |  |  |  |
| w/ Jigsaw++ | 14.3 | 37.8 | 36.6 |  |  |  |  |  |  |
| Difference | -8.1 | +17.6 | +14.1 |  |  |  |  |  |  |
| Jigsaw (Lu et al., ) | 10.5 $\pm$ 0.1 | 45.6 $\pm$ 0.1 | 42.7 $\pm$ 0.1 |  |  |  |  |  |  |
| w/ Jigsaw++ | 4.5 $\pm$ 0.3 | 48.7 $\pm$ 0.2 | 49.5 $\pm 0.3$ |  |  |  |  |  |  |
| Difference | -6.0 | +3.1 | +6.8 |  |  |  |  |  |  |
| PartNet |  |  |  |  |  |  |  |  |  |
|  | Chair | Table | Lamp |  |  |  |  |  |  |
| Method | CD | Pre. | Rec. | CD | Pre. | Rec. | CD | Pre. | Rec. |
| DGL (Zhan et al., ) | 47.8 | 21.5 | 20.0 | 53.6 | 16.6 | 15.4 | 68.8 | 18.6 | 17.9 |
| w/ Jigsaw++ | 41.0 | 52.0 | 33.6 | 42.6 | 53.6 | 31.0 | 46.3 | 42.3 | 28.5 |
| Difference | -6.8 | +30.5 | +13.6 | -11.0 | +37.0 | +15.6 | -22.5 | +23.7 | +10.6 |

### 6.2 Performance

#### Overall Performance.

We evaluated the performance of baseline methods with our proposed Jigsaw++ on both Breaking Bad dataset (Sellán et al., ) for the fracture assembly problem and PartNet (Mo et al., ) for the part assembly problem. A quantitative analysis is detailed in Table [1](#S6.T1).

Jigsaw++ consistently outperformed the baseline methods, demonstrating its capability to reconstruct a meaningful underlying complete shape that corresponds closely to the input partially assembled objects. Even with a less favorable initialization algorithm SE(3) (Wu et al., ), our algorithm can give a large improvement on their results. Specifically, Jigsaw++achieves significantly better results in terms of reconstruction error in the fracture assembly problem. We draw three insights: (1) The original size of the objects in the Breaking Bad Dataset is considerably smaller compared to those in PartNet (please refer to Sec. [6.3](#S6.SS3) for a failed reconstruction case on PartNet). This small size discrepancy enables the mapping between point clouds and images to pose minimal impacts on the representation of the complete shape. (2) The diversity of complete shapes in the Breaking Bad Dataset is less varied than in PartNet, simplifying the modeling of the complete shape space.

Despite less favorable initialization in part assembly, Jigsaw++ significantly improves the precision and recall metrics to depict complete shapes on PartNet. Since the assembled object from DGL could be significantly displaced or reordered, Jigsaw++ offers valuable insights into the likely overall shape. Such insight on the complete shape is essential for the general object reassembly problem, and provides a new possibility for developing better algorithms for the object reassembly problem.

**Table 2: Left: Reconstruction performance of Jigsaw++ when presented with input with missing pieces. The model are tested on the Bottle category of the Breaking Bad dataset. Right: Fracture assembly performance with original-shape matching with the shape prior generated by Jigsaw++.**
| Breaking Bad - Bottle | Breaking Bad |  |  |  |  |  |  |  |  |
| --- | --- | --- | --- | --- | --- | --- | --- | --- | --- |
| Method | Input | CD$\downarrow$ | Precision$\uparrow$ | Recall$\uparrow$ | Method | Matching Type | MAE(R)$\downarrow$ | MAE(T)$\downarrow$ | PA$\uparrow$ |
| $\times 10^{-2}$ | $\%$ | $\%$ | degree | $\times 10^{-2}$ | $\%$ |  |  |  |  |
| Jigsaw | complete | 3.4 | 52.8 | 49.9 | Jigsaw | fracture | 36.3 | 8.7 | 57.3 |
| Jigsaw++ | complete | 1.8 | 61.0 | 59.4 | Jigsaw++ | + GT shape prior | 17.8 | 3.6 | 73.1 |
| Jigsaw++ | 20% missing | 2.0 | 59.5 | 59.4 | Jigsaw++ | + 20% noise shape prior | 18.2 | 3.7 | 72.6 |

#### Performance with Missing Pieces.

To demonstrate the effectiveness and the robustness of the proposed method, we conduct a test using the Bottle category from the Breaking Bad dataset. Each piece will have 20% probability of been removed and we ensure at least one piece is presented in one object. We input the Jigsaw’s result with pieces removed to the Jigsaw++ model.

As shown in Table [2](#S6.T2) left, the resilience of Jigsaw++ is evidenced when processing inputs with 20% missing pieces. Under these conditions, the model maintained a low CD of $2.0\times 10^{-2}$, with precision and recall approximately at 59.4%. This performance closely aligns with that seen in fully intact inputs, highlighting Jigsaw++’s robustness in dealing with data incompleteness.

#### Performance for Fracture Reassembly Algorithm.

While our primary interest lies in applying the generated shapes to reassembly algorithms, we encountered challenges in finding an algorithm that effectively utilizes the complete shape prior. To demonstrate the potential of our approach in assembly problems, we present an alternative evaluation method.

We augment the Jigsaw algorithm (Lu et al., ) by providing a matching between the original object surface and our generated shape prior during its global alignment stage. This matching is computed by finding the closest point from the ground truth position of each point to the generated shape. It is important to note that the fractured surface matching and global alignment algorithm remain unchanged from Jigsaw and may contain errors.

Table [2](#S6.T2) right shows that when using the closest point matching with ground truth, we can reduce Jigsaw’s error by $50\%$.
Even with the introduction of $20\%$ noise to this “ground truth” matching, performance remains significantly improved over the baseline Jigsaw algorithm.
These results demonstrate that our generated shape can indeed assist assembly algorithms. This suggests that future research efforts to develop algorithms that can fully utilize these complete shape priors could yield significant advancements in reassembly tasks.

#### Ablation Study on varying Parameters.

We now show how different parameter settings influence the performance during the “retargeting” phase of Jigsaw++. We first examine the effect of the rectified flow formulation under varying reverse sampling steps. As discussed in Sec. [5](#S5), this formulation significantly reduces the required number of reverse sampling steps. Letting $N$ denote the forward sampling steps, and $N_{r}=kN$ the reverse sampling steps, we explore the effects of altering $k$ on reconstruction outcomes. The results, illustrated in the upper row of Fig. [5](#S6.F5), show that the model performs best when $k=1/10$. A full reverse sampling phase tends to overly mimic the input, which is suboptimal for reconstruction. Moreover, setting $k$ too low can cause the latent to deviate excessively, leading to a different output.

Further, we explored the impact of modifying the latent composition ${\bm{x}}_{0}=\alpha\hat{{\bm{x}}}_{0}+\sqrt{1-\alpha^{2}}\xi,\xi\sim\mathcal{N}(0,I)$ on reconstruction quality. Research in image generation, such as those by (Liu et al., ; Meng et al., ), indicates that a larger $\alpha$ generally replicates the input more closely, while a smaller $\alpha$ pushes the generation towards the data domain. We observed a similar trend in our generative model as in Fig. [5](#S6.F5) lower row. At $\alpha=1$, the output is very similar to the input, whereas decreasing $\alpha$ makes the result progressively diverge towards representing a complete object. Interestingly, although the precise shape might not be replicated, the reconstructed form invariably aligns visually with the ground truth category.

Figure: Figure 5: Ablation study of Jigsaw++ with varying parameters on the Breaking Bad dataset. Top: Varies the reverse sampling steps to $N_{r}=kN$ to assess how well the rectified flow model accommodates step reductions. Bottom: Alter the $\alpha$ parameter in the Langevin dynamics to explore how changes in latent resampling during the retargeting phase affect model performance.
Refer to caption: /html/2410.11816/assets/x5.png

### 6.3 Limitation and Failure Cases

Figure: Figure 6: Three types of failure cases of Jigsaw++. (a) Size limitation in color mapping. (b) Limitation on unseen objects. (c) Topology constraints.
Refer to caption: /html/2410.11816/assets/x6.png

While we have investigated various strategies to enhance the robustness of point cloud generation, our model still struggles to generalize to unseen object types or significantly varied objects. We identify three main types of failure cases as in Fig. [6](#S6.F6): (a) Size limitation in color mapping. Converting object point clouds into color spaces imposes significant size constraints. Objects like tall street lights might not be adequately visible in the rendered images, causing the reconstruction process to fail. Conversely, the model tends to perform better with smaller objects. (b) Dataset limitations. Given that our model is trained on selected datasets, it struggles to recognize and reconstruct rarely encountered or unseen object types. Specific details cannot be accurately reconstructed using the current methodology. (c) Topological Accuracy. The model’s capability to delineate object topology is insufficient, particularly for objects with complex structures. For instance, in the mug category, if the handle does not distinctly appear in the input, the model might reconstruct a decent representation of the mug’s body but falter in accurately forming the handle.
While our approach improves upon existing methods, the outlined limitations underscore the necessity for employing larger and better models, as well as richer datasets, in future research efforts to address these challenges.

## 7 Conclusions and Future Work

In this study, we present Jigsaw++, a novel framework developed to tackle the challenge of complete shape reconstruction in object reassembly tasks. Jigsaw++ utilizes a novel point cloud generative model that reimagines the complete object shape from partially assembled inputs. By incorporating image-to-3D reconstruction techniques, Jigsaw++ adeptly navigates the challenges of scale and diversity in training data. Additionally, we show the rectified flow formulation enhances our proposed “retargeting” phase, establishing a more robust connection between the latent space and the complete object space. Experimental results demonstrate Jigsaw++’s superior reconstruction performance, marking a significant improvement over existing methods. Although we have achieved successful reconstructions, we have yet to devise methods to effectively leverage our outputs as guidance for further reconstructions. This limitation opens up new avenues for research in the field of object reassembly.

## Appendix

## Appendix A Potential Solutions and How They Fails

Figure: Figure 7: Qualitative demonstration of potential solutions.
Refer to caption: /html/2410.11816/assets/x7.png

Generating a complete shape prior based on a partially assembled object is a relatively new problem that is often underestimated in its complexity. We explored several intuitive solutions during the development of our method to demonstrate the challenges involved. While it is impossible to enumerate all potential solutions, we have selected representative approaches to highlight the uniqueness of our task.
The difficulty in providing an accurate shape prior stems from two main challenges: (1) Lack of quantification of assembly errors: We do not know which pieces are correctly assembled and which are not.
(2) Balancing shape alteration: The algorithm must adapt to varying degrees of assembly accuracy, from minor adjustments for nearly perfect assemblies to significant corrections for misplaced or incomplete pieces.

We tested three representative algorithms using state-of-the-art models and evaluated their performance on four test cases. Figure [7](#A1.F7) illustrates the results of these experiments.

#### Point Cloud Completion

We adopted AdaPoinTr (Yu et al., ) using their open-sourced code and model trained on the ShapeNet dataset. We provided the algorithm with a subset of correctly placed pieces from Jigsaw’s result. The algorithm exhibited the following limitations: (a) It interpreted the subset of parts as a complete shape, resulting in no additional completion as in the first bottle. (b) With more parts in the second bottle, it completed the top slightly, but the was sparse and limited in range. (c) It produce a resonable result for the plate which most closely resembled a typical completion task, while for the vase, it over-correct the given input.

#### Point Cloud Generative Model with Editing

We employed LION (Zeng et al., ) with the SDEdit (Meng et al., ) model using their open-sourced code. The results showed that (a) the generated shapes with similar overall forms (e.g., thin long shape for bottle input, flat shape for plate), but (b) unable to to consistently maintain the correct object category.

#### Point Cloud Auto-encoders

We utilized LION’s (Zeng et al., ) VAE to assess the effectiveness of reconstruction. Results showed that the output was mostly identical to the input, with only minor changes towards the desired shape. This behavior is consistent with the VAE’s objective of accurate shape reconstruction.

While these methods excel in their designed tasks, they fall short in addressing the specific challenges of inferring complete shape prior for the reassembly problems.

## Appendix B Implementation Details

### B.1 Used Codebases and Datasets

For baseline comparison, the following codes are used:

- •
DGL (Zhan et al., ): [https://github.com/hyperplane-lab/Generative-3D-Part-Assembly](https://github.com/hyperplane-lab/Generative-3D-Part-Assembly).
- •
SE(3) (Wu et al., ): [https://github.com/crtie/Leveraging-SE-3-Equivariance-for-Learning-3D-Geometric-Shape-Assembly/tree/main](https://github.com/crtie/Leveraging-SE-3-Equivariance-for-Learning-3D-Geometric-Shape-Assembly).
- •
Jigsaw (Lu et al., ): [https://github.com/Jiaxin-Lu/Jigsaw](https://github.com/Jiaxin-Lu/Jigsaw), (MIT License).
- •
PoinTr and AdaPoinTr (Yu et al., ; ): [https://github.com/yuxumin/PoinTr](https://github.com/yuxumin/PoinTr), (MIT License).
- •
LION (Zeng et al., ): [https://github.com/nv-tlabs/LION](https://github.com/nv-tlabs/LION), (NVIDIA Source Code License).
- •
SDEdit (Meng et al., ): [https://github.com/ermongroup/SDEdit](https://github.com/ermongroup/SDEdit), (MIT License).

For building our methods, the following codes are referenced:

- •
LEAP (Jiang et al., ): [https://github.com/hwjiang1510/LEAP](https://github.com/hwjiang1510/LEAP).
- •
UViT (Bao et al., ): [https://github.com/baofff/U-ViT](https://github.com/baofff/U-ViT), (MIT License).
- •
Rectified Flow (Liu et al., ): [https://github.com/gnobitab/RectifiedFlow](https://github.com/gnobitab/RectifiedFlow).

The following datasets are used:

- •
Breaking Bad Dataset (Sellán et al., ): [doi:10.5683/SP3/LZNPKB](https://borealisdata.ca/dataset.xhtml?persistentId=doi:10.5683/SP3/LZNPKB) (License as listed in the link).
- •
PartNet (Mo et al., ): The [Pre-release v0] version at [https://partnet.cs.stanford.edu/](https://partnet.cs.stanford.edu/) for mesh, and the version presented with [DGL](https://github.com/hyperplane-lab/Generative-3D-Part-Assembly) (Zhan et al., ).
- •
Kubric-ShapeNet (Greff et al., ): The version with [LEAP](https://github.com/hwjiang1510/LEAP) for camera parameters.

### B.2 Parameters

We provide a detailed model parameters in Table. [3](#A2.T3).

**Table 3: The detailed experiment parameters.**
|  |  | Breaking Bad Dataset | PartNet |  |  |  |
| --- | --- | --- | --- | --- | --- | --- |
|  | Parameter | base | retargeting | base | regargeting | description |
| Training | epoch | 500 | 100 | 1000 | 400 | training epochs |
| bs | 32 | 32 | 16 | 16 | batch size |  |
| lr | 0.0001 | 0.00002 | 0.0001 | 0.00002 | learning rate |  |
| optimizer | Adam | Adam | Adam | Adam | optimizer during training |  |
| scheduler | Cosine | - | Cosine | - | learning rate scheduler |  |
| min_lr | 1e-6 | - | 1e-6 | - | minimum learning rate for Cosine scheduler |  |
| frames | 5 | 5 | 5 | 5 | input frames to the image encoder |  |
| Model | $N$ | 100 | 100 | 100 | 100 | sample steps in Rectified Flow |
| $N_{r}$ | - | 4 | - | 4 | reverse sampling steps in retargting |  |
| $\alpha$ | - | 0.5 | - | 0.5 | scaling factor for latents during retargeting |  |
| depth | 12 | 768 | depth of UViT |  |  |  |
| $d$ | 768 | 768 | token dimension in UViT |  |  |  |

## Appendix C Training Details

### C.1 Training resources and Inference Time

Our experiments utilized a setup featuring eight NVIDIA Tesla A100 GPUs, with all running times based on this specific GPU configuration.

The finetuning of LEAP reconstruction model takes 232 GPU hours. In the training phase for the base generative model, different datasets required varying amounts of GPU time: the Breaking Bad dataset needed 480 GPU hours, while the PartNet categories required 40 GPU hours for Lamp, 216 GPU hours for Chair, and 240 GPU hours for Table. Additionally, the ”retargeting” stage fine-tuning took 480 GPU hours for the Breaking Bad dataset and half of base model for each PartNet category.

For inference, reverse sampling of a single instance on one GPU took 0.2 seconds, and forward generation took 5 seconds. The complete processing time for one instance, includes rendering and reconstruction, was approximately 7.5 seconds on average.

### C.2 Trained Models

On the Breaking Bad dataset of the fracture assembly problem, LEAP (Jiang et al., ) is first finetuned using rendered mesh data. One generation model is trained for the entire subset without categorical information. This model is finetuned and “retargeted” for the reconstruction task.

On the PartNet of the part assembly problem, LEAP is first finetuned using rendered mesh data for all three categories. For each category, one generation model is trained, which results to three base generative models. These models are finetuned independently for the reconstruction task.

### C.3 Data Visualization

Figure [8](#A3.F8) provides a visualization of the rendered data utilized in our experiments. In the top row, the partially assembled object is shown in point cloud format, which is used as input during the “retargeting” phase and for testing. The bottom row features the rendered complete objects, which are based on the mesh data from the dataset. Due to the superior continuity of this mesh data, it is selected as the ground truth for guiding the training of the generative model and the image-to-3D model, LEAP.

Figure: Figure 8: Visualization of one instance from the Breaking Bad Dataset. Top: The input partially assembled object is presented in point cloud format, which is employed both in the ”retargeting” phase and for testing purposes. Bottom: Input complete objects rendered from meshes. Those data are used to create ground truth data for the training phases of the generative model and the image-to-3D model LEAP.
Refer to caption: /html/2410.11816/assets/x8.png

## Appendix D Visualization

We present detailed visualization of results on the Breaking Bad Dataset (Fig. [9](#A4.F9)) and PartNet (Fig. [10](#A4.F10)). Each instances are organized in the order of “partial input - Jigsaw++ - Ground Truth” vertically.

Figure: Figure 9: Detailed visualization of results on the Breaking Bad Dataset.
Refer to caption: /html/2410.11816/assets/x9.png

Figure: Figure 10: Detailed visualization of results on the PartNet.
Refer to caption: /html/2410.11816/assets/x10.png

## Appendix E Broader Impacts

This paper tackles object reassembly problem, which has no known negative impact on society as whole. On the contrary, its application in archaeology and medication would benefits research in other areas. Our method utilizes 3D generative model, which we hope could address several hard problems overlook by the current researches. The data we use are all objects datasets. Although we see no immediate negative use cases or content from this model, we acknowledge the necessity of handling the generative model with care to prevent any potential harm.

## Peer Reviews

confidence: 4
contribution: 2
initial_rating: 3
presentation: 3
questions: - The first contribution, the authors claim the proposed Jigsaw++ is a novel method for addressing the object reassembly problem. At the end of Sec. 3.1, the authors say the purpose of the proposed method is not to design a reassembly algorithm, but rather an additional layer of information to improve the reassembly algorithm. They are very different concepts, and the contribution summarized in the Intro section is very misleading.
- The shape before and after "retargeting" may vary, and probably the worst case, it diverges to a totally different shape. It would be necessary to quantify how much shape change occurs between input and output.
- Would the proposed method generate parts that were not originally from the fractured pieces since it is based on a 3D generation model?
- Explain how the proposed approach could be adapted or extended to provide actionable assembly instructions from scanned point clouds.
- It would be greatly appreciated if the authors could provide several real-life examples, instead of simple samples from datasets, to prove the proposed method is useful for assembly purposes as the title suggested.

I suggest the authors could detailedly address the concerns and I will provide a final rating based on the feedback and other reviewers' opinions.
soundness: 2
strengths: - Jigsaw++ learns a shape prior that is not limited to specific object categories, making it more versatile and adaptable to a wide range of objects.
- The proposed retargeting makes the method complementary to other approaches.
summary: The paper introduces Jigsaw++, a generative method to improve automatic assembly tasks involving complex 3D reconstruction challenges. Unlike existing methods that focus on assembling individual pieces or fractures, often neglecting the overall object structure, Jigsaw++ learns a category-agnostic shape prior for complete objects. A key innovation is its "retargeting" strategy, which utilizes the output of any existing assembly method to generate more accurate complete shape reconstructions. This makes Jigsaw++ complementary to current approaches. The authors provide evaluations on the Breaking Bad dataset and PartNet demonstrates its effectiveness in reducing reconstruction errors and improving shape precision, offering a promising new direction for future reassembly models.
weaknesses: - Initial guidance may harm the performance of the proposed method, though this is a strength claimed by the authors' proposed "retargeting" tech.
- Although category-agnostic, the performance on highly intricate or unconventional objects is not extensively discussed. Only simple results are shown in the paper. 
- Utilizing the generative models is a good idea, but how to ensure the generated results comply with the topic of "fractured assembly"? In the shown results, the shape may change, and the SE(3) transformation is also difficult to obtain, making the proposed method hard to use in real-life applications, for example, providing a few scanned point clouds and then understanding how to assemble them to form a complete shape.

---

confidence: 4
contribution: 3
initial_rating: 6
presentation: 2
questions: - How many shape pairs were used for training during the fine-tuning (or retargeting) of the rectified model? Was the model trained on the assembly results of one model directly applied to others? (e.g., Jigsaw → SE(3))
- The discussion of existing works in L207-L211 is somewhat inaccurate:
    - Regarding category-agnostic models and geometric details: many existing geometric generation models can produce detailed geometry (such as CLAY, X-Cube, etc., whose generated results can also be converted to point clouds). Whether training a category-agnostic model is essential might be more a matter of training and experimental details.
- Referring to works like (Song et al., 2021a; Mokady et al., 2022; Meng et al., 2022; Liu et al., 2023) as “conditional generation” in L219-L223 could be misleading. Alternative terms like “inversion-based methods” might be more accurate.
- It is necessary to discuss the relationship between mapping point clouds to images and NOCS[1] as well as X-NOCS[2].

[1] Normalized Object Coordinate Space for Category-Level 6D Object Pose and Size Estimation

[2] Multiview Aggregation for Learning Category-Specific Shape Reconstruction
soundness: 3
strengths: - Learning robust latent representations and generative models based on limited intact geometry and initial assemble results is a challenging problem. The authors use multi-view images as a bridge, employing a pre-trained images-to-3D sparse-view reconstruction model to provide robust encoders and decoders. By mapping point clouds to multi-view images and utilizing these pre-trained robust encoders and decoders, they cleverly overcome this issue.
- Experimental results show that the generative model proposed by the authors can effectively enhance the initial assembly results of multiple models in both fracture assembly and part assembly tasks.
summary: This paper aims to provide a category-agnostic global shape prior for shape assembly (part/fracture assembly) methods, improving upon the shortcomings of existing approaches that rely solely on local feature information. In modeling the generative model, to learn robust feature encoders and decoders based on limited initial assembly results (both intact and damaged), the proposed method maps 3D point clouds to multi-view images. This allows the use of robust pre-trained encoders and decoders trained on large-scale data. Based on this, the authors designed a training process based on rectified flow to implement a generative model that encodes damaged initial assembly results into latent representations and generate the intact ones. The effectiveness of the proposed pipeline across different assembly models has been validated on part and fracture assembly tasks.
weaknesses: My primary concern is the necessity of the model design. In the problem setting, paired data is available, and paired data is directly used for fine-tuning the rectified flow model in the final stage, which makes the necessity of the entire pipeline uncertain. Unlike methods by Song et al., 2021a; Mokady et al., 2022; Meng et al., 2022; Liu et al., 2023, which perform distribution transformations like image editing, restoration, or image-to-image translation based on inversion, the process proposed in this paper requires fine-tuning based on paired data (L-355~L-359). This makes the overall process more complex compared to training-free approaches and directly training conditional generation models either from scratch or fine-tuning a pre-trained conditional model for example with ControlNet on paired data, without demonstrating advantage of the proposed method over these two types of approaches in the experiments. More specifically, the necessity of the pipeline lacks the following experimental clarifications:
- Given the problem formulation in the paper, paired data is available, which allows for directly training a conditional rectified flow. Compared to this naive solution, does performing inversion and “retargeting” before training the rectified flow on paired data offer advantage, like better robustness or generalization?
- Regarding the rectified flow model trained in Sec-4, how effective is the inversion-then-generate process without the “retargeting” step? Alternatively, what is the effect of skipping fine-tuning on the rectified flow model? Although the paper mentions that the results are suboptimal, there is no quantitative or qualitative evidence provided to support this claim.

---

confidence: 5
contribution: 3
initial_rating: 8
presentation: 3
questions: good paper, but i still have questions. please see the weaknesses above
soundness: 4
strengths: 1. A method that is able to "imagine" the complete shape prior which can be useful for the object reassembly problem.

2. The proposed retargeting method is able to facilitate the reconstruction of complete objects from partially assembled input. 

3. The proposed method can be used with other existing methods to further improve performance.
summary: This paper studies the problem of object reassembly. the paper proposes a method called Jigsaw++, which learns a category agnostic prior of complete objects. The proposed method is able to generate complete shape reconstructions and can be used in conjunction with existing methods. Results on Breaking Bad and PartNet show the effectiveness of the proposed method.
weaknesses: 1. the first two columns of figure 6a are very difficult to see what they are trying to show (they look all black). maybe consider changing the style of presentation.

2. if other researchers want to apply this method to assemble a set of fractures from an unseen object, how confident can they be? should they expect the proposed method to give reasonably good assembly results? if not, why is that and how could they make improvements?


--- New ---

I read the comments from the other reviewers. I am convinced that the comments mentioned by reviewer kWoJ are indeed concerns for the current method, especially the issue regarding practical usability. Therefore, I would like to downgrade my score to 6.

---

confidence: 3
contribution: 3
initial_rating: 6
presentation: 2
questions: Please see the parts of “Weaknesses” that end in a question mark for questions addressed at the authors. On top of those,

- What exactly is the function “f” in L248? Are the point cloud visualizations throughout the paper actual rasterizations as described in these lines?
- What do these two sentences mean? “Likely, given a colored image under the same color encoding scheme, we obtain a corresponding partial colored point cloud that corresponds to the 2D pixels. The computed point cloud will be further refined through a camera-pixel alignment operation that projects the decoded 3D point onto the ray that connects the pixel and the camera center.” Could the authors elaborate, or explain this in a different way? Unfortunately, I cannot follow this reasoning or understand what I should take from it.
- In figure 3, which parts of this are didactic and which parts aren’t? For example, the voxel grid shows geometric information which then converges to the bottle shape. Is this what the latent r looks like? Or is a decoding of r what is being displayed?
soundness: 3
strengths: I congratulate the authors for their work and thank them for their submission. As far as I know (I could be wrong though), this is the only work that proposes separating the “global” first-attempt-at-reassembly problem from the “local” fine-tuning-the-placement-of-the-pieces problem and proposes to solve the reassembly problem in two stages. Given the struggles of existing reassembly works, I believe this may very well be a strategy worth pursuing. The work seems technically sound and exhaustively evaluated, enough to convince me that the combination of this work and DGL/Jigsaw is the new state of the art for the reassembly task.
summary: The authors propose an algorithm to fine-tune a partially reassembled object into a fully reassembled one, potentially with missing pieces or imperfections. They represent the objects as point clouds, which can be rendered into images via differentiable rasterization. A point cloud of a partially reassembled object is then rendered and encoded using a pre-trained diffusion model. The algorithm relies on the fact that partially reassembled shapes are likely underrepresented in the pre-trained training data; thus, they propose flowing the encoding to a higher probability region and decoding it to obtain the reassembled shape. The authors test their algorithm exhaustively on existing baselines.
weaknesses: From a technical perspective, this work is fundamentally reliant on the completed object being given higher likelihood by the diffusion model than the partially reassembled one, which probably translates to similar objects being present in the data that was used to train said models. This is in opposition to purely geometric or feature-matching-type works like the ones mentioned in Sec. 2.1 that assemble pieces based on geometric features in the fracture faults. This reliance is never mentioned explicitly as a weakness, nor explored experimentally, which I found disappointing. For example, what happens if one tests the proposed method on procedurally generated shapes with procedurally generated fractures?  

From a less technical perspective, I believe the authors could scope the proposed method better and quicker in the introduction. By reading the abstract and the first four paragraph, a reader will believe that they are being presented with a full reassembly method, not just one to fine-tune an already-good, partial reassembly. As said above (see “Strengths”), this is a perfectly valid problem to tackle, but it should be very clear immediately.

The presentation of the qualitative results could also be improved, and I would really appreciate if the authors did so in their revision. While the authors use a point cloud in their network, both datasets they are using are actually meshes, and it is really hard to visually distingush the geometric matching of two shapes by looking at unstructured points. Could the authors use the learned transformations of the points to obtain a per-part mesh transformation, apply it to the input and show that as the result? At the very least, show that in addition to the point cloud. Now that I think of it, how would one extract a per-part transformation (i.e., in the form of T_i) from the existing pipeline, not just an output point cloud? Isn’t this transformation what one needs for the suggested applications in robotics, archeology and medicine?

Meta-review: This paper introduces Jigsaw++, which handles object reassembly problems. The key idea is to propose a novel generative model, which is different from prior work focusing on piecewise part assembly that may overlook overall assembled shape. The authors demonstrated the effectiveness of the proposed approach with the Breaking Bad and ParNet datasets.

In summary, the strengths of the proposed approach are summarized as follows:
- Jigsaw++ learns shape prior that is not limited to specific object categories
- Clever idea that learns multi-view images as a bridge
- Superior accuracy in both fracture assembly and part assembly tasks

The weaknesses of the proposed approach are summarized as follows:
- Inconsistency or misleading points in the paper writing 
- Problem configuration
- Possibility of converging completely different shapes after the assembly 
- Requests on the real, non-pairwise fractured shapes
- Misleading visualization of point clouds 

In general, AC values the idea of the proposed approach and good demonstration. However, the common concerns raised by reviewers are more convincing, and AC also noted that even the reviewers pGbo and r9Py downgraded the scores after the rebuttal phase. Therefore, AC does not recommend the acceptance of the paper.

Decision: Reject