POC-SLT: Partial Object Completion with SDF Latent Transformers

## Abstract

Abstract 3D geometric shape completion hinges on representation learning and a deep understanding of geometric data.
Without profound insights into the three-dimensional nature of the data, this task remains unattainable.
Our work addresses this challenge of 3D shape completion given partial observations
by proposing a transformer operating on the latent space representing Signed Distance Fields (SDFs).
Instead of a monolithic volume, the SDF of an object is partitioned into smaller high-resolution patches leading to a sequence of latent codes.
The approach relies on a smooth latent space encoding learned via a variational autoencoder (VAE), trained on millions of 3D patches.
We employ an efficient masked autoencoder transformer to complete partial sequences into comprehensive shapes in latent space.
Our approach is extensively evaluated on partial observations from ShapeNet and the ABC dataset where only fractions of the objects are given.
The proposed POC-SLT architecture compares favorably with several baseline state-of-the-art methods,
demonstrating a significant improvement in 3D shape completion, both qualitatively and quantitatively.

## 1 Introduction

The 3D measurements from scanners or sensors can be used to assemble virtual objects or entire worlds, either for visualizations, but also for reasoning about the world for example in self-driving cars.
In general, with optical sensors, it is only possible to measure 3D surfaces that are visible to the sensor from its current point of view.
3D scans are therefore always composed of many partial scans from different angles.
The more unique views are combined, the more complete the reconstructed object surface will become.
This creates a trade-off between speed and completeness.
Shape completion takes an incomplete shape and tries to estimate the complete geometry based on some learned shape prior.

In this work, we present a new method for shape completion using a transformer in the style of Masked Auto Encoders (MAE) (He et al., ) operating in latent SDF space. For an unknown or masked region, it is supposed to fill in a plausible shape that connects well with the given geometry.
The method consumes and produces shapes represented as Signed Distance Fields (SDF) on volumetric tiles.
Similar to Stable Diffusion (Rombach et al., ), the incomplete shapes are first encoded into a smooth, compressed latent space representation using our dedicated *Patch Variational Autoencoder (P-VAE)*. For high-resolution models, the entire shape is represented as a sequence of latent codes generated by the P-VAE on volumetric patches.
Completion is performed in latent space and the completed shape is decoded back to an SDF using the decoder of the P-VAE on each patch.
Note that our *SDF-Latent-Transformer* is a Masked Auto Encoder.
It therefore solves the task directly in a single inference step without a transformer decoder.

Shape completion requires a strong prior for real-world shapes in order to estimate the most likely shapes for incomplete inputs.
We trained our P-VAE and SDF-Latent-Transformer on ShapeNetCoreV1 (Chang et al., ) with all 55 categories.
We further fine-tuned the transformer on a subset of the ABC dataset (Koch et al., ) with 100K meshes.

By thoroughly evaluating our methods on shape completion tasks,
we demonstrate superior quality compared to state of the art.
Our key contributions to the field are:
i) A comprehensive Patch Variational Autoencoder (P-VAE) to compress SDF shapes into sequences of codes in a smooth latent space.
ii) A SDF-Latent-Transformer trained as Masked Auto Encoder completes input shapes within milliseconds in a single inference step.
iii) Upon acceptance, we will release code and model checkpoints and make the training data sets available.
For the P-VAE, it consists of 2.6 million $32^{3}$ patches and for the SDF-Latent-Transformer of $128^{3}$ SDFs for all objects in ShapeNetCoreV1 which were severely cleaned.

Ours is the first SDF-based method trained and evaluated
on all of ShapeNet (around 50k meshes) and a 100k subset of ABC.
This is an order of magnitude more than previous SDF-based approaches and
at higher resolution, while using consumer-grade GPUs.
While MAEs are used in many domains by now, they usually are a means to pretrain a powerful encoder,
whose strong representation can be used for downstream tasks.
We however directly use the training objective as a shortcut to get instant completion.
This only works because of the robustness and strong geometric prior in our latent space.

## 2 Related Work

We review related point-based, occupancy-based, and SDF-based 3D completion methods and methods inspiring our deep-learning architecture.

#### Point-based Shape Completion.

3D Shape completion tackles the task of reconstructing 3D objects from partial observations in the form of 2D images,
sometimes with depth information, or 3D point clouds.
Areas occluded during capture need to be filled in to form a complete object.

PointNet and its variations (Qi et al., ; ) have pioneered the direct processing of 3D coordinates with neural networks, catalyzing research in numerous downstream tasks, including point cloud completion, as for example PCN (Yuan et al., ).
PCN uses PointNet in an encoder-decoder framework and integrates a FoldingNet (Yang et al., ) to transpose 2D points onto a 3D surface by simulating the deformation of a 2D plane.
Following PCN, a plethora of other methods have emerged (Tchapmi et al., ; Huang et al., ; Xie et al., ; Liu et al., ), aiming to enhance point cloud completion with higher resolution and increased robustness.

PointTR (Yu et al., ) was one of the first approaches
which used transformers (Vaswani et al., ) for point cloud completion.
They derive the point proxies from a fixed count of furthest point sampled representatives,
which are turned into a feature vector using a DGCNN (Wang et al., ).
Completion is done by a geometry-aware transformer, which is queried for missing point proxies.
The missing points are extracted from the predicted proxies using a
FoldingNet (Yang et al., ). The completed points are finally
merged with the input point cloud.
AdaPointTR (Yu et al., ) addresses discontinuity issues and is more robust to noisy input.
SeedFormer (Zhou et al., ) completes a sparse set of patch seeds
from an incomplete input and upsamples the seeds to a complete point cloud.
The seeds are sparse 3D positions enriched by a transformer with semantic information about the local neighborhood.
SnowFlakeNet (Xiang et al., ) solves this decoding task with a SkipTransformer-based architecture, which repeatedly splits and refines low-resolution points.
LAKe-Net (Tang et al., ) first localizes aligned keypoints to generate a surface skeleton mesh which aids in producing a complete point cloud.
AnchorFormer (Chen et al., ) extracts global features
and predicts key ”anchor” points which are combined with a subset of input points and then upsampled into a dense mesh.
VRPCN (Pan et al., ) models the shape from the partial input as
probability distributions, which are sampled and then refined by a hierarchical
encoder-decoder network. This is related to our approach as we
also use a Variational Auto Encoder (VAE) to compress the shape in latent space.
LION (Zeng et al., ) introduced a flexible latent diffusion model for point clouds also using a VAE to map point cloud features into latent space. The features and diffusion model are constructed from Point-Voxel CNNs.

#### Occupancy-based Shape Completion.

Occupancy Networks (Mescheder et al., ) introduced a functional, implicit representation of 3D shapes via their occupancy.
These methods can be queried for arbitrary 3D locations and will return whether the queried position is inside or outside of the represented object.
Meshes can quickly be recovered from this representation using the MISE algorithm (Mescheder et al., ).
Peng et al. () extend this to a convolutional method and represent entire scenes by storing and interpolating latent representations in 2D/3D grids.
Chibane et al. () combine local and global information using multi-scale feature grids in IF-Nets.

These methods focus on identifying and representing 3D objects seen in images
and adding detail to uniformly low-resolution point cloud and voxel input.

ShapeFormer (Yan et al., ) compactly encodes shapes into a sparse grid of vector-quantized deep implicit functions (VQDIF).
Partial inputs are completed by an autoregessive transformer operating on a sequence of location and patch index pairs.
3DILG (Zhang et al., ) uses an irregular grid of latent codes for shape representation and completes partial inputs with an autoregressive transformer.
3DShape2VecSet (Zhang et al., ) derives multiple global vectors for shape representation using cross-attention between input points and sampled query points.
The (variationally) auto-encoded shape is then completed using latent-space diffusion.
The output occupancy is queried by cross-attention between a dense point cloud and the global vectors.

The input to these methods is typically a point cloud generated from a single camera view.
Occluded parts need to be inferred from the partial point cloud.
In many cases, the input still covers large parts of the object,
exposing global information about the shape,
while our method completes objects from a small subset of complete patches.

#### SDF-based Shape Completion.

Signed Distance Fields (SDFs) represent 3D shapes implicitly as continuous 3D scalar functions, which describe the distance of any 3D point to the closest surface.
The surface is given by the zero-level-set while in- or outside points are identified by the sign.
This ensures that objects represented by SDFs are always watertight.
In contrast, meshes are often non-manifold since they are commonly created to ”look right” on screen,
rather than to represent the volume of solid geometry.
While SDFs could be modeled in any functional basis, we use regular 3D grids.

There are only few approaches related to shape completion on SDFs in the literature.
For SDF representation and class-related shape generation, DeepSDF (Park et al., ) proposes to
use an
AutoDecoder (Tan & Mavrovouniotis, ) to jointly optimize a compressed
latent representation of an SDF and the decoder to extract the full SDF from the latent representations.
Due to the fully connected layer, the approach has limited resolution. Our approach instead splits the SDF into patches, which are
encoded with a rigorously trained VAE (Kingma & Welling, ).
Those patches can be assembled to very highly detailed shapes.
We experimented with using a similar AutoDecoder optimization scheme to refine latent codes for our frozen decoder
but found that the optimized latent codes complicate subsequent shape completion tasks (Appendix A.1).

AutoSDF and SDFusion (Mittal et al., ; Cheng et al., ) focus on multi-modal shape generation and completion in latent space, e.g. from an input image, point clouds, or depth maps.
Both approaches use Vector Quantized Variational Auto Encoders (VQ-VAEs) (van den Oord et al., ; Razavi et al., ) to patchwise compress SDF shapes to a compact latent space. SDFusion then uses a latent diffusion model, while AutoSDF employs an autoregressive transformer to generate or complete shapes sequentially.
AutoSDF (Mittal et al., ) is in many regards similar to our
approach. The main differences are: i) the choice of a VAE over a VQ-VAE to avoid
quantization errors.
ii) Larger patches with $32^{3}$ grid points over the $8^{3}$
grid points used by AutoSDF.
iii) Our transformer is trained as Masked Auto Encoder
(MAE) (He et al., ) while the AutoSDF Transformer is trained
autoregressively. This drastically changes how the model can be used as our approach does not require a sequential generation.
iv) Our approach is trained on SDF grids with a resolution of $128^{3}$ and trained on the full ShapeNetCoreV1 (Chang et al., ) and refined on the ABC datasets (Koch et al., ) instead of the SDF resolution of $64^{3}$ on a 13-category subset of SapeNetCoreV1 for AutoSDF.

PatchComplete (Rao et al., ) and DiffComplete (Chu et al., ) both complete $32^{3}$ Truncated SDF input using multi-scale features.
While PatchComplete uses an attention mechanism, DiffComplete uses diffusion
and also allows combination of multiple inputs.
Both methods learn strong priors that generalize well to new inputs, albeit at comparably low resolution.
While our method works on a subset of clean SDF patches to extend partial geometry,
these methods work on the 3D-EPN (Dai et al., ) dataset, containing noisy, low-resolution partial SDFs generated from a single camera view.

#### Variational Autoencoders (VAE)

(Kingma & Welling, ) are networks that learn latent representations of input data.
Unlike standard autoencoders, they learn a probability distribution over the latent space.
This generates a smooth latent space which allows manipulation and computations in latent space.
Inspired by Stable Diffusion (Rombach et al., ), we train a VAE to encode the $32^{3}$ SDF patches into a smooth latent space.
We call this network Patch Variational Autoencoder (P-VAE).
A full $128^{3}$ SDF grid is therefore represented by only 64 latent tokens.
This makes any further processing very efficient and scalable.

#### Transformers

(Vaswani et al., ) are a network architecture for sequence-to-sequence tasks.
It uses attention as the core mechanism to identify pairwise relationships between elements in a sequence.
They have revolutionized Natural Language Processing and are the foundation of all current Large Language Models.
Transformers can however process any sequence of data.
In Dosovitskiy et al. (), images are split into a sequence of patches, which is then processed by transformers. The approach in
He et al. () masks out some of those patches and train to fill in the masked-out patches.
Similarly, we split the SDF grid into patches and encode them into a sequence of latent codes.
We mask out parts of the latent codes that contain unknown geometry and use our SDF-Latent-Transformer to complete the missing patches.

Figure: Figure 1: Architecture Overview. The SDF is partitioned into tiles of $32^{3}$ samples. For each tile, a latent code is generated by a variational autoencoder (P-VAE) resulting in the potentially partial input stream to our SDF-Latent-Transformer. It is trained as a Masked Autoencoder to generate a completed series of tokens which are finally translated back to SDF tiles using the P-VAE decoder.
Refer to caption: extracted/5982697/figures/sdf_nonmasked.png

## 3 Method

Our POC-SLT shape completion pipeline consists of four main steps which are depicted in Figure 1:
First, the input SDF grid is split into patches of $32^{3}$ 1.
Each of those patches is encoded into a latent vector using the P-VAE 2.
The latent codes $z$ that form the input shape are then assembled in a sequence,
which is masked and processed by the SDF-Latent-Transformer (SLT) 3.
Finally, the resulting sequence is decoded patch-by-patch with the decoder from the P-VAE into a completed SDF 4.
We will now elaborate on these four steps.

### 3.1 P-VAE

The task for the P-VAE is to encode $32^{3}$-SDF grid patches $(p_{i})_{i=0,\dots,n}$ into latent code vectors $(z_{i})_{i=0,...,n}$.
For details about the architecture and implementation of our P-VAE, refer to Appendix D.
The P-VAE consists of an encoder $E_{\mathrm{VAE}}$ and a decoder $D_{\mathrm{VAE}}$, which are both based on 3D-convolutional layers.
A patch $p_{i}$ is encoded by the encoder into mean $\mu_{i}$ and variance $\sigma^{2}_{i}$ of a Gaussian distribution.
During training, this distribution is sampled to obtain the latent code $z_{i}\sim\mathcal{N}(\mu_{i},\sigma_{i}^{2})$,
which is then decoded by the Decoder $D_{\mathrm{VAE}}$ into an SDF patch $\tilde{p}_{i}=D_{\mathrm{VAE}}(z_{i})$.
We use the mean absolute error between $p_{i}$ and $\tilde{p}_{i}$ as reconstruction loss.
To regularize the latent space of the VAE, we follow the *KL-regularized* VAE implementation from Stable Diffusion (Rombach et al., ).
During inference, we directly use the means $z_{i}=\mu_{i}$ without additional sampling.

The P-VAE is trained on SDF patches extracted randomly from parts of the meshes of ShapeNetCoreV1 (Chang et al., ).
Each patch has a fixed SDF resolution of $32^{3}$.
We however strongly vary the side length of each extracted patch while ensuring that they remain close to a surface. Specifically, we uniformly sample surface points and side lengths $d\sim\mathcal{U}_{[0.1,1]}$ as well as offsets $x,y,z\sim\mathcal{N}(0,d/3)$.
The P-VAE is therefore exposed to a large variety of detail levels, scales, and surface types and has to learn the full variety of how patches of natural shapes can potentially look like.
This yields a strong, generalizable shape prior which allows the P-VAE to generalize even to patches of shapes from completely different datasets.

Figure: Figure 2: P-VAE (left): On millions of patches, we train a smooth embedding space for SDFs with a variational autoencoder (P-VAE) that samples the noise according to the predicted variance before passing the estimated latent to the decoder. SDF-Latent-Transformer (SLT) (right): Performs shape completion on input sequences consisting of SDF-Patches (cyan) in latent space. During training, some of the input patches are masked and substituted with a trainable shared vector (blue). The utilized masking schemes *Random*, *Half*, *Octant* and *Slice* are visualized on the right. 3D positional encoding is added before a TransformerEncoder propagates the information from the remaining patches to all the masked patches completing the 3D shape.
Refer to caption: extracted/5982697/figures/sdf_nonmasked.png

### 3.2 SDF-Latent-Transformer

The actual shape completion happens completely in latent space.
Our approach is visualized on the right in Figure 2.
The incomplete input shape is provided as a sequence of latent space vectors.
We employ a standard transformer encoder as a Masked Auto Encoder to fill in missing latent vectors.
The missing input parts are marked by a special trainable *mask token* (blue in the figure).
The positions of all vectors, including masked ones, are provided to the transformer
as 3D positional encoding (Sitzmann et al., ).
Linear layers before and after concatenating the positional embedding have
empirically shown to improve convergence. Finally, the transformer completes the
sequence.

#### Outside SDF Patches.

For many, especially anisotropic shapes, some of the encoded SDF-Patches will not contain any geometry either because they are completely inside or outside an object.
Note however, that those patches still contain valid distance values towards surfaces that lie outside
those patches. We therefore do not differentiate between patches that explicitly contain surfaces and those that lie outside.

#### Masking during Training.

The training strategy of the SDF-Latent-Transformer is visualized in Figure 2.
For creating the mask for a given training example, we randomly choose from one of the following masking strategies:
1. *Random Masking*, 2. *Octant*, 3. *Half*, 4. *Slice*.
With *Random Masking*, each patch is masked out with a $40\%$ probability.
With *Octant*, all but one of the octants of the patch-grid are masked out.
With *Half*, one half of the patch-grid is masked out.
With Slice, everything but one axis-aligned slice through the patch-grid is masked out.
The four different masking schemes are visualized in the same order from top to bottom on the right side of Figure 2.
We use *Random Masking* with a probability of $35\%$ and all the other strategies with a probability of $21.7\%$ each.

#### Loss Functions.

We supervise our training with ground truth patch encodings $z=E_{\mathrm{VAE}}$ for the complete object.
The entire training of the SDF-Latent-Transformer happens in latent space.
The full loss function $\mathcal{L}_{\mathrm{comp}}$ consists of two terms:

$$ $\mathcal{L}_{\text{comp}}=\alpha\mathcal{L}_{\text{masked}}+\beta\mathcal{L}_{ \text{non-masked}}$ (1) $$

Both $\mathcal{L}_{\mathrm{masked}}$ and $\mathcal{L}_{\mathrm{non-masked}}$ are simple $L1$-losses on the latent codes
evaluated on results with masked and non-masked inputs respectively.
The weights $\alpha$ and $\beta$ are chosen, such that the contribution of
every patch to the total loss is equal, no matter how many patches were masked.
With $N$ patches in total from which $M$ are masked out we get:

$$ $\displaystyle\alpha$ $\displaystyle=\frac{M}{N}$ $\displaystyle\beta$ $\displaystyle=\frac{N-M}{N}$ (2) $$

#### Shape Completion.

For shape completion, one can use the SDF-Latent-Transformer in the same mode as during training.
We assume the partial shape is given in the form of a high-resolution SDF volume.
For each given patch, a latent code is generated with the P-VAE encoder.
The known SDF patches (including known empty patches)
are handed over as input to the transformer while marking all others as masked.
The transformer will then predict proper latent codes for all masked tokens, which includes predicting latent codes for empty patches.

The resulting grid of latent codes can then be decoded back into an $128^{3}$ SDF
which can be converted into a mesh, e.g. using Marching Cubes (Lorensen & Cline, ).

## 4 Evaluation

POC-SLT is an efficient and fast solution for patch-wise SDF
completion in latent space. It consists of the P-VAE which encodes SDF patches
into latent space and the SDF-Latent-Transformer which fills in missing patches in latent space.
To demonstrate the effectiveness of our method, we measure its performance on various completion
tasks and compare it to related work below.
Additional tasks are demonstrated in Appendix A.
All metrics used in the experiments are defined in Appendix B.
Data preparation for turning meshes into SDFs is described in Appendix C.

#### Completion:

First, we provide quantitative and qualitative results for three SDF shape
completion tasks on the full ShapeNetCoreV1 (Chang et al., )
dataset and ABC (Koch et al., ) test sets.
The tasks are to complete an SDF based on different types of partial inputs.
1. Only the bottom half of the SDF is given as input (Half).
2. Only the bottom right octant is given as input (Oct).
3. Patches are removed randomly
with 25%, 50%, and 75% of the SDF remaining (R25, R50, R75).

A simpler version of those tasks on a subset of ShapeNet has been suggested
by Wu et al. () and used by (Wu et al., ; Yu et al., ; Mittal et al., ; Cheng et al., ) for completion comparisons.
Here, only 13 categories of
ShapeNet are used for training and evaluation and the Unidirectional Haushorff
Distance (UHD) is reported. Note that this only measures how faithfully the
geometry given in the input was preserved during the completion.

In addition, we also compare to AnchorFormer (Chen et al., ) on the (Half) task, by
using point cloud inputs with only points in the bottom half of the bounding box (Half).

#### P-VAE:

The P-VAE is evaluated by measuring the deviation between
the input of the encoder and the output of the decoder.
We furthermore compare the reconstructions
from our P-VAE latent space to the reconstructions of an expensively optimized latent space
using Park et al. (); Tan & Mavrovouniotis ().

#### Timing:

Our latent space completion consists of a single forward step of the SLT.
This makes the shape completion inference very fast. We demonstrate this
advantage over previous latent shape completion
works (Mittal et al., ; Yan et al., ) in
Table 1.

**Table 1: Comparing shape completion inference time.**
| Method | Inference Time | Output Resolution |
| --- | --- | --- |
| POC-SLT (Ours) | 8.6 milliseconds | $128^{3}$ |
| AutoSDF (Mittal et al., ) | 4.3 seconds | $64^{3}$ |
| ShapeFormer (Yan et al., ) | 20 seconds | $128^{3}$ |

#### Hardware:

The P-VAE and SDF-Latent-Transformer variations were trained on a machine with 3 Nvidia RTX 4090 GPUs and 512GB of main memory.
The P-VAE was trained for approximately two weeks, while the transformers were trained for about two days each.
Additional computation was required for preprocessing (Appendix C) and evaluation,
which was carried out over several machines in our cluster, using mostly Nvidia RTX 2080 Ti GPUs and several CPUs.

### 4.1 Completion

The SDF-Latent-Transformer is the core component of our completion pipeline.
It receives an incomplete sequence of SDF patches in latent space and completes it to a full sequence of latent codes in a single forward step.
The latent codes of the completed sequence are independently decoded into SDF patches with the P-VAE decoder.
The patches are trivially assembled into a high-resolution shape volume from
which we extract surface meshes using Marching Cubes (Lorensen & Cline, ).
We trained two slightly different variants of the SDF-Latent-Transformer:
SLT: trained on ShapeNetCoreV1
with latent codes $z$ from P-VAE as ground truth
and SLT ABC: SLT fine-tuned on a subset of the ABC dataset.
The quality of our shape completion approach is evaluated on unseen objects from
ShapeNetCoreV1 (Chang et al., ) and ABC (Koch et al., ).

#### Qualitative Results (ShapeNet).

Examples for the previously
described bottom half (Half) completion task on ShapeNetCoreV1 (Chang et al., )
are shown in Figure 3. Here, we compare our
SLT against AutoSDF (Mittal et al., ) and
AnchorFormer (Chen et al., ) as recent state-of-the-art methods.

Our POC-SLT pipeline generates highly plausible shapes.
Both AutoSDF (Mittal et al., ) and AnchorFormer (Chen et al., ) struggle to fill in the missing parts.
Compared to AutoSDF, our method works at a significantly higher resolution and can handle all ShapeNet classes.
The higher resolution is for example important for the fine structures on the bench table in column four or the faucet in column two.
AutoSDF (Mittal et al., ) completely fails to reconstruct the car in column one or the armchair in
column three, while our SLT on the same input produces correct results.
AnchorFormer (Chen et al., ) often produces sparse and simplistic completions for inputs with unknown top halves which is especially
visible for the faucet in column two.
However, the sparseness and therefore lack of detail are also visible in the car in the first column,
the armchair in the second, and the cupboard in the last column.

Figure: Figure 3: Completion of ShapeNet (Chang et al., ) objects from bottom half. Comparison to AutoSDF (Mittal et al., ) and AnchorFormer Chen et al. (). Our SLT completes these objects more plausibly than AutoSDF. The density of completed points by AnchorFromer drastically varies in the completed regions.
Refer to caption: extracted/5982697/figures/Shapenet/Input/bottom_half/Masked_non_optimized_latent_codes-ObjID=1279_.obj.png

#### Qualitative Results (ABC).

Shape completion results from halves (Half) or octants (Oct) on
ABC (Koch et al., ) are visualized in Figure 4.
For this task, we use SLT ABC, which was fine-tuned on the ABC dataset.
The ABC dataset contains many planar and rotationally symmetric objects. These symmetries are picked up by the SDF-Latent-Transformer to complete missing parts with high detail and high plausibility, such as the object in column one or column four.

On the other hand, the provided input does not always constrain the output sufficiently. This can lead to deviations when compared to ground truth while still generating plausible objects, such as the upper spokes in the third column, the unsymmetric object in the sixth column, or the missing hole in
the last column of the figure.

Figure: Figure 4: Completion of ABC (Koch et al., ) objects from the bottom half (left) and octant (right). The SLT learned to complete partially symmetric objects quite successfully.
Refer to caption: extracted/5982697/figures/ABC/bottom_half/Input/Masked_non_optimized_latent_codes-ObjID=109_.obj.png

Figure: Figure 5: Completion of out-of-distribution objects from Objaverse (Deitke et al., ) using the SLT trained on ShapeNet (Chang et al., ). The last three columns show completions of scanned real-world objects.
Refer to caption: extracted/5982697/figures/Objaverse/Input/ev0/Masked_non_optimized_latent_codes-ObjID=35_.obj.png

#### Generalization (Objaverse).

To demonstrate the generalizability of our approach we evaluate the SLT, trained only on ShapeNet (Chang et al., ),
on the (Half) task with objects from Objaverse (Deitke et al., ).
The type of objects shown in Figure 5 have never been seen by the SLT.
As expected, the results show clear deviations from the ground truth,
especially for objects which are far away from the kind of objects that are found in ShapeNet,
such as the animals in column four and five.
For simpler objects, such as the pillow in column three or the mushroom in column one,
our approach produces good completions.
There is a chair category in ShapeNet, which allows our model to plausibly complete the rocking chair in column two.

#### Quantitative Results.

We numerically evaluate the previously described tasks in Table 2.

**Table 2: Shape Completion. Evaluation on completion tasks (Half), (Oct), (R75), (R50) and (R25) with the ShapeNetCoreV1 (Chang et al., ) dataset with all 55 categories and on (Half) and (Oct) with the ABC (Koch et al., ) dataset and . The results report the mean over all categories. Details on the metrics can be found in Appendix B.**
| Model | Dataset | Task | IoU$\uparrow$ | $F_{1}$$\uparrow$ | CD$\downarrow$ | HD$\downarrow$ | NC$\uparrow$ | IN$\downarrow$ | CMP$\uparrow$ |
| --- | --- | --- | --- | --- | --- | --- | --- | --- | --- |
| SLT | SN | Half | 0.7466 | 0.8468 | 1.0221 | 0.0765 | 0.9200 | 0.4196 | 0.9067 |
| SLT | SN | Oct | 0.5884 | 0.7336 | 1.2467 | 0.0966 | 0.8589 | 0.6034 | 0.8404 |
| SLT | SN | R75 | 0.9153 | 0.9792 | 0.2258 | 0.0452 | 0.9677 | 0.2862 | 0.9905 |
| SLT | SN | R50 | 0.8650 | 0.9495 | 0.4829 | 0.0595 | 0.9504 | 0.3512 | 0.9751 |
| SLT | SN | R25 | 0.7645 | 0.8677 | 0.8567 | 0.0789 | 0.9183 | 0.4512 | 0.9329 |
| SLT ABC | ABC | Half | 0.8617 | 0.9159 | 0.8703 | 0.0575 | 0.9435 | 0.2551 | 0.9466 |
| SLT ABC | ABC | Oct | 0.7144 | 0.7744 | 2.9247 | 0.1077 | 0.8779 | 0.3986 | 0.8391 |

The results on the aforementioned benchmark from
Wu et al. () are reported in Table 3.
Numbers from previous works are taken from Mittal et al. ().
The numbers for the *chair* category
are given explicitly to compare to previous work. Our SLT outperforms all
other methods. Even when evaluated on all 13 categories, there remains a
significant improvement.

**Table 3: Quantitative comparison on shape completion on the *chair* category of the ShapeNetCoreV1-subset with 13 categories from Wu et al. (). Completion tasks are (Half) and (Oct). In compliance with previous work, we report the Unidirectional Hausdorff Distance from input to completed result.**
| Method/UHD$\downarrow$ | Half | Oct |
| --- | --- | --- |
| MPC (Wu et al., ) | 0.0627 | 0.0579 |
| PoinTR (Yu et al., ) | 0.0572 | 0.0536 |
| AutoSDF (Mittal et al., ) | 0.0567 | 0.0599 |
| SDFusion (Cheng et al., ) | 0.0557 | - |
| SLT (ours) | 0.0445 | 0.0467 |
| SLT (ours) all 13 categories | 0.0379 | 0.0399 |

### 4.2 P-VAE & AutoDecoder

The P-VAE accepts SDF patches of size $32^{3}$ as input, encodes them into compressed latent
representations and decodes latent representations back to SDF patches.
It was trained and evaluated on a training split of ShapeNet. We evaluate the P-VAE
on unseen data in Table 4. The numbers demonstrate the high
quality of our latent space. Note that the P-VAE was not fine-tuned for the
following experiments on ABC (Koch et al., ) or
Objaverse (Deitke et al., ) which shows the strong generalizability
resulting from our training scheme.

**Table 4: P-VAE encode-decode performance evaluated on ShapeNet (Chang et al., ).**
| IoU$\uparrow$ | $F_{1}$$\uparrow$ | CD$\downarrow$ | HD$\downarrow$ |
| --- | --- | --- | --- |
| 0.9482 | 0.9931 | 0.0001 | 0.0264 |

Park et al. () suggested to use an AutoDecoder
(AD) (Tan & Mavrovouniotis, ) to optimize the latent codes for a known
SDF such that the frozen decoder $D_{\mathrm{VAE}}$ produces the best possible
result. The optimized latent codes should yield more accurate surface
reconstruction than the codes directly produced by the encoder $E_{\mathrm{VAE}}$
in a single forward pass. However, the time spent on refinement is orders of
magnitude longer, e.g. around $0.04$ sec for the P-VAE encoder vs. 25 sec for
the AutoDecoder per object.

We compare our P-VAE to the *AutoDecoder* approach by encoding a
given SDF patch $p_{gt}$ to a latent code $z=E_{\mathrm{VAE}}(p_{gt})$.
We then add random noise to $z$ to get an initial version $z^{\prime}$, which
is then decoded into a SDF $p_{pred}=D_{\mathrm{VAE}}(z^{\prime})$.
We compute a loss $|p_{gt}-p_{pred}|$ to update $z^{\prime}$ in a loop till convergence.

Both P-VAE and AD produce high-quality results with only minor differences to ground truth.
Across the dataset, we get a tiny improvement over the Hausdorff Distance from 0.026 to 0.024 from the optimization.
These improvements are mostly visible for objects with fine detail as shown in Figure 6.
One could use these optimized latent codes $z^{\prime}$ to train the SLT towards producing such output tokens at no extra cost.
We experimented on this idea and show results in Appendix A.1.

Figure: Figure 6: Comparison of latent codes generated by P-VAE encoding and refined via the AutoDecoder. While overall subtle, these extreme examples demonstrate the benefit of using the AutoDecoder to refine the patch embeddings.
Refer to caption: x1.png

## 5 Limitations

While the presented results are of high quality, some downstream tasks for the POC-SLT pipeline might need additional work.
At the moment, it is trained assuming a fixed bounding box. This is a valid assumption for most objects but might be too limiting for open scenes.
In principle, the transformer should be able to cope with sequences of basically arbitrary length. However, it still needs to be investigated if the current model can deal with higher-resolution spatial encodings to make use of additional tokens.

The completion is performed for masked patches. Implicitly, the approach always assumes either completely given SDF patches or completely unknown patches.
In completion applications for images, depth maps, or partial 3D scans,
one might additionally need to indicate that the patch information itself might be incomplete, e.g. a missing occluded surface in the same cell.

We do not consider the single-view 3D reconstruction task.
Previous work (Tatarchenko et al., ) has shown that predicting object-centered results heavily relies on
identifying (and augmenting) similar objects in the training set and can even be outperformed by object retrieval.
Lacking a single global object representation, our method is not well-suited for such a classification task.
Instead, one could first estimate a canonical pose, monocular depth, and then compute 3D SDF patches from the then recovered partial geometry
in order to meaningfully perform geometric 3D reconstruction with our method.
Evaluation of such an approach would highly depend on the models employed for pose estimation and depth estimation in addition to our model,
preventing meaningful comparisons.

## 6 Conclusion

With POC-SLT, we proposed an accurate and efficient new method for SDF shape
completion in latent space. POC-SLT processes SDFs in patches of fixed size.
Two main components are used to refine and fill in missing patches in a shape.
Firstly, an extensively trained Patch Variational Autoencoder (P-VAE) for
accurately compressing the patches into a sequence of latent codes and back.
Secondly, an SDF-Latent-Transformer (SLT) which completes and refines the latent
sequence of an incomplete shape in a single inference step. We demonstrate that
our approach produces highly accurate and plausible 3D shape completions,
outperforming prior works. POC-SLT is trained on ShapeNetCoreV1 (Chang et al., ), is class agnostic,
and can easily be adapted to new datasets, which we demonstrated with the ABC (Koch et al., )
dataset. We show that the P-VAE works across different datasets, even without
additional training, and hope that it will be a helpful tool for future
research. Extending the approach to deal with partial information within patches
and developing applications like real-time scan completions are left open for
future work.

## Acknowledgements

This work has been supported by the Deutsche Forschungsgemeinschaft (DFG) – EXC number 2064/1 – Project number 390727645 and SFB 1233, TP 2, Project number 276693517, and by the IMPRS-IS.

## Appendix A Additional Experiments

In this section we show several additional experiments which did not fit into
the main paper. We evaluate training the SLT on auto-decoded latent codes
(Section A.1), inspect how the SLT trained on
ShapeNet (Chang et al., ) can generalize to the ABC dataset
(Section A.2), show visual results of shape completion
on the completion tasks (R25), (R50) and (R75) with randomly masked inputs
(Section A.3), compare our latent space representation
with DeepSDF (Park et al., ) and
3DShape2VecSet (Zhang et al., )
(Section A.4), provide evaluation on
3D-EPN Dai et al. () (Section A.5),
and, finally, present further comparisons with AnchorFormer (Chen et al., ) in
Section A.6.

### A.1 SDF-Latent-Transformer on Refined latent codes

The comparison in Figure 6 suggests that the latent codes for
some objects might be improved by explicitly optimizing the latent codes of the
patches using the AutoDecoder technique (Park et al., ; Tan & Mavrovouniotis, ). To test if the SLT might improve performance when
trained on regular partial latent codes $z$ as input but with optimized $z^{\prime}$ as
ground truth, we optimized the latent codes for all objects in the training
dataset of ShapeNet (Chang et al., ). We trained a separate SDF Latent Transformer on this dataset called SLT-AD, expecting the SLT-AD to also learn to
do the expensive AutoDecoder optimization for free in its forward pass. We also
tested to run the regular SLT and, on the resulting SDF, the SLT-AD without
masking just for refinement. This model is called SLT + SLT-AD. Finally, we tried
running the SLT-AD twice: Once on the incomplete input and then again on the
completed output, which is called SLT-AD + SLT-AD.

The quantitative results are shown in Table 5 and qualitative results are shown in Figure 7.
The numbers for the bottom half (Half) experiment clearly suggest, that the expected
improvement did not happen. The results for the octant (Oct) experiment are
inconclusive at best.

While the AutoDecoder technique (Park et al., ; Tan & Mavrovouniotis, )
produces latent codes that decode to a more accurate representation of the given input,
we believe that the latent codes generated this way are more likely to be outliers
and thus not as easily understood and utilized by the SLT-AD,
resulting in worse performance.

Figure: Figure 7: Completion from halves and octants by our SLT and then refined by SLT-AD on ShapeNet.
Refer to caption: extracted/5982697/figures/Refinement/SLT_through_AD_SLT/octant/Input/Masked_non_optimized_latent_codes-ObjID=1052_.obj.png

**Table 5: Shape completion on ShapeNet (Chang et al., ) with several versions of the SLT. SLT is the default version, SLT-AD hs been trained on auto-decoded ground-truth latent codes, and the ”+ SLT-AD” variants feed the previous output through SLT-AD again without masking for refinement.**
| Model | Task | IoU$\uparrow$ | $F_{1}$$\uparrow$ | CD$\downarrow$ | HD$\downarrow$ | NC$\uparrow$ | IN$\downarrow$ | CMP$\uparrow$ |
| --- | --- | --- | --- | --- | --- | --- | --- | --- |
| SLT | Half | 0.7466 | 0.8468 | 1.0221 | 0.0765 | 0.9200 | 0.4196 | 0.9067 |
| SLT-AD | Half | 0.7237 | 0.8344 | 1.2065 | 0.0846 | 0.9119 | 0.4767 | 0.9086 |
| SLT + SLT-AD | Half | 0.6867 | 0.8101 | 1.0039 | 0.0811 | 0.8976 | 0.5268 | 0.8696 |
| SLT-AD + SLT-AD | Half | 0.6780 | 0.7969 | 0.9111 | 0.0880 | 0.8923 | 0.5528 | 0.8658 |
| SLT | Oct | 0.5884 | 0.7336 | 1.2467 | 0.0966 | 0.8589 | 0.6034 | 0.8404 |
| SLT-AD | Oct | 0.6127 | 0.7121 | 1.8994 | 0.1010 | 0.8694 | 0.5704 | 0.8279 |
| SLT + SLT-AD | Oct | 0.5878 | 0.6848 | 1.6883 | 0.0970 | 0.8557 | 0.6052 | 0.7729 |
| SLT-AD + SLT-AD | Oct | 0.5912 | 0.6795 | 1.4225 | 0.0940 | 0.8560 | 0.6167 | 0.7767 |

### A.2 ABC completion using SLT trained on ShapeNet

To test the generalizability of our SLT trained on ShapeNet (Chang et al., ),
we evaluate it on the ABC (Koch et al., ) dataset without fine-tuning.
We report the metrics for the tasks (Half) and (Oct) in Table 6.
To compare to the fine-tuned SLT ABC, we duplicate its numbers
reported in Table 2 here for convenience.
As expected, the reconstruction metrics of the fine-tuned model are much better.
Figure 8 shows, however, that the STL still does a
reasonably good job at completing many of the objects from ABC (Koch et al., ).
Instead of producing complete mechanical parts,
it often creates chairs, tables, and vases from their partial inputs
and it picks up surprisingly well on some of the inherent symmetries.
With the (Oct) input, the SLT has more freedom to complete the input into objects found in ShapeNet (Chang et al., ), such as chairs, e.g., column four.

**Table 6: Evaluation on completion tasks (Half), (Oct) on the ABC (Koch et al., ) dataset using SLT and comparing to the SLT ABC results copied from Table 2. The results report the mean over all categories. Details on the metrics can be found in Appendix B.**
| Model | Dataset | Task | IoU$\uparrow$ | $F_{1}$$\uparrow$ | CD$\downarrow$ | HD$\downarrow$ | NC$\uparrow$ | IN$\downarrow$ | CMP$\uparrow$ |
| --- | --- | --- | --- | --- | --- | --- | --- | --- | --- |
| SLT | ABC | Half | 0.7478 | 0.8123 | 2.0658 | 0.1053 | 0.9134 | 0.3730 | 0.8869 |
| SLT ABC | ABC | Half | 0.8617 | 0.9159 | 0.8703 | 0.0575 | 0.9435 | 0.2551 | 0.9466 |
| SLT | ABC | Oct | 0.5710 | 0.6304 | 5.1872 | 0.1532 | 0.8208 | 0.5433 | 0.7234 |
| SLT ABC | ABC | Oct | 0.7144 | 0.7744 | 2.9247 | 0.1077 | 0.8779 | 0.3986 | 0.8391 |

Figure: Figure 8: Testing generalizability via completion of ABC (Koch et al., ) objects using the SLT trained exclusively on ShapeNet (Chang et al., ).
Refer to caption: extracted/5982697/figures/ABC_through_Shapenet/octant/Input/Masked_non_optimized_latent_codes-ObjID=3490_.obj.png

### A.3 Completion on Randomly Masked ShapeNet

One of the masking strategies during training is to randomly mask out inputs as outlined in Section 3.2. We also numerically evaluated the random masking
completion tasks (R75), (R50) and (R25) in Table 2.
Now, in Figure 9 we show visual results on ShapeNet (Chang et al., ), for the same three completion tasks.
Even when only small parts of the input are given, e.g. 25% in the (R25) task,
the information in neighboring tokens is used to produce near-perfect completions.
This is in contrast to the significantly more challenging (Half) and (Oct) tasks we chose to evaluate our method where completion needs to happen most often for non-neighbor patches.

Figure: Figure 9: Completion of randomly masked ShapeNet (Chang et al., ) objects using the SLT. This is often a slightly simpler task as small holes are relatively easily filled by propagating information from neighboring patches.
Refer to caption: extracted/5982697/figures/RandomMasking/0.75/Input/Masked_non_optimized_latent_codes-ObjID=7249_.obj.png

### A.4 Latent Quality Comparison with DeepSDF and 3DShape2VecSet

We compare the auto-encoding quality of our P-VAE (Section 3.1)
against DeepSDF (Park et al., ) in
Table 7 and against
3DShape2VecSet (Zhang et al., ) in
Table 8. For
3DShape2VecSet (Zhang et al., ), we follow their evaluation code
with regards to the size of the object and with regards to the L1 Chamfer
Distance. Like in their code, we omit the
division by 2 which would be part of the original definition from the
supplemental of Mescheder et al. ().
In addition to the numbers on our regular P-VAE, we also report the results when
decoding optimized latent codes $z^{\prime}$ using an AutoDecoder as described in
Section A.1 under the label P-VAE-AD.

The results in Table 7 demonstrate that,
on auto encoding quality,
our P-VAE outperforms DeepSDF on all categories with and without optimized latent codes.
The numbers in Table 8 suggest that
the auto-encoding performance of our P-VAE is significantly better in $F_{1}$ and CDL1
and on par with the cross-attention based auto encoding strategy from 3DShape2VecSet.

**Table 7: Comparing mean and median (L2) Chamfer Distance in a P-VAE auto-encoding quality comparison with DeepSDF (Park et al., ).**
| CD$\downarrow$ (mean) | chair | plane | table | lamp | sofa |
| --- | --- | --- | --- | --- | --- |
| P-VAE | 0.0759 | 0.0236 | 0.0914 | 0.415 | 0.0269 |
| P-VAE-AD | 0.1427 | 0.0177 | 0.0743 | 0.4041 | 0.0263 |
| DeepSDF | 0.204 | 0.143 | 0.553 | 0.832 | 0.132 |
| CD$\downarrow$ (median) | chair | plane | table | lamp | sofa |
| P-VAE | 0.0203 | 0.0075 | 0.023 | 0.0167 | 0.0264 |
| P-VAE-AD | 0.0199 | 0.0067 | 0.0225 | 0.0138 | 0.0259 |
| DeepSDF | 0.072 | 0.036 | 0.068 | 0.219 | 0.088 |

**Table 8: P-VAE auto-encoding quality comparison with 3DShape2VecSet (Zhang et al., ).**
| Method | IoU$\uparrow$ | $F_{1}$$\uparrow$ | CD${}_{\mathrm{L1}}\downarrow$ | HD$\downarrow$ |
| --- | --- | --- | --- | --- |
| P-VAE | 0.9483 | 0.9931 | 0.0094 | 0.0658 |
| 3DShape2VecSet | 0.965 | 0.970 | 0.038 | - |

### A.5 Shape Completion on 3D-EPN

While our model was never intended for and never trained to perform shape
completion from incomplete SDF patches, it might still be interesting to see how
it performs on such a task.
Incomplete SDF patches mean that, within the same patch, parts of the geometry are already missing.

In Figure 10, we show the result of trying to complete the
partial $32^{3}$ SDFs in the 3D-EPN (Dai et al., ) dataset with our SLT
trained on complete, high-resolution ShapeNet (Chang et al., ) patches.
The partial SDFs from 3D-EPN (Dai et al., ) are generated from a
single camera view and, thus, everything behind the first visible surface is
considered inside, often producing incorrect SDF patches where outside
areas are encoded to be deep inside an object.
We upsample the low resolution SDFs to our $128^{3}$ resolution in order to get multiple patches. This leads to staircase artifacts which have not been seen in this way by the SLT, even for complete patches.

Our SLT assumes that every given, non-masked $32^{3}$ input patch contains
complete information.
Therefore, it does not attempt to repair their partially corrupt information
and – for the most part – just feeds the given input through to the output.
While some masked patches on the side of the objects can be filled in,
as expected, this generally leads to poor results.

While it could be possible to train or fine-tune the existing SLT architecture on
this task, we think that further research would be required to best adapt our
method to this setting.

Figure: Figure 10: Completion of partial $32^{3}$ SDFs from 3D-EPN (Dai et al., ) using our SLT which is trained only on complete patches from $128^{3}$ SDFs from ShapeNet (Chang et al., ). While not trained on this kind of input, the SLT still demonstrates its geometric understanding and produces reasonable shape completions.
Refer to caption: extracted/5982697/figures/3D-EPN/Input/Masked_non_optimized_latent_codes_40a7ed7ce8ba70fa4ddda47ee90d6a21__4_table-ObjID=175_.obj.png

### A.6 Comparison with AnchorFormer

Since our method works on SDF patches rather than point clouds,
performing one-to-one comparisons with point cloud methods is somewhat problematic.

In Table 9, we show the performance of AnchorFormer (Chen et al., )
when evaluated on the regular PCN (Yuan et al., ) test split
and when presented with bottom-half inputs from the same objects,
uniformly sampled from our preprocessed (Appendix C) version of ShapeNet (Chang et al., ).
We removed some of the objects from the PCN (Yuan et al., ) test split, mostly cars,
for which we did not have a matching mesh available to generate bottom-half inputs.

For evaluating AnchorFormer (Chen et al., ),
we use their only published pre-trained checkpoint which was trained on PCN (Yuan et al., ).
Completion from just the bottom half can be a much harder task,
since the output on the top half is completely unconstrained.
Typically, PCN (Yuan et al., ) and other partial point cloud settings provide
at least some sparse global information which guides and constrains the completion process.

As a second experiment (also shown in Table 9),
we use the same AnchorFormer (Chen et al., ) model and checkpoint
to complete our test set of ShapeNet (Chang et al., ) meshes from bottom halves,
as shown in Figure 3 and evaluated for our method in Table 2.

The numbers on the PCN (Yuan et al., ) data show that AnchorFormer (Chen et al., ) is not well suited to solve
the (Half) completion task. Both metrics drop significantly.
Furthermore, on our ShapeNet (Chang et al., ) data, the (Half) task produces slightly better numbers than on the PCN (Yuan et al., ) data.
However, our approach achieves better completion quality on the (Half) shape completion task.

**Table 9: Comparison with AnchorFormer (Chen et al., ) on completing partial inputs from ShapeNet (Chang et al., ). Details on the compared configurations are explained in Section A.6. Some qualitative results can be seen as part of Figure 3.**
| Method | Data Source | Split | Task | $F_{1}$ $\uparrow$ | CD(L2) $\downarrow$ |
| --- | --- | --- | --- | --- | --- |
| AnchorFormer | PCN | PCN | PCN | 0.8379 | 0.1986 |
| AnchorFormer | ShapeNet | PCN | Half | 0.6953 | 1.4791 |
| AnchorFormer | ShapeNet | Ours | Half | 0.7164 | 4.2816 |
| Ours (see Table 2) | ShapeNet | Ours | Half | 0.8468 | 1.0221 |

## Appendix B Metrics

In order to evaluate any of the following metrics,
we use Marching Cubes (Lorensen & Cline, ) to generate a mesh from our generated SDF
and then uniformly sample 1M points $X$ and $Y$ from the resulting meshes.

We use the following point-based metrics:
Hausdorff Distance (HD), to measure the largest gap between the original and the reconstructed geometry:

$$ $\operatorname{HD}(X,Y)=\max\left\{\max_{x\in X}\left\{\min_{y\in Y}d(x,y) \right\},\max_{y\in Y}\left\{\min_{x\in X}d(x,y)\right\}\right\}.$ (3) $$

Similarly, the Unidirectional Hausdorff Distance (UHD) measures the maximum distance between objects, but here, in line with previous work (Mittal et al., ),
it is measured from partial input $X$ to completion $Y$ to measure the ”fidelity” of the completed output:

$$ $\operatorname{UHD}(X,Y)=\max_{x\in X}\left\{\min_{y\in Y}d(x,y)\right\}.$ (4) $$

We use the ($L2$) Chamfer Distance (Fan et al., ), multiplied by 1000, to measure the accuracy of the reconstruction via the mean squared distance between the original and reconstructed geometry:

$$ $\operatorname{CD}(X,Y)=\frac{1}{|X|}\sum_{x\in X}\min_{y\in Y}\left\{d^{2}(x,y )\right\}+\frac{1}{|Y|}\sum_{y\in Y}\min_{x\in X}\left\{d^{2}(y,x)\right\}.$ (5) $$

In both cases, $d(x,y)$ measures the $L2$ distance between two points $x,y\in\mathbb{R}^{3}$.

For consistency with prior work, these metrics are evaluated at a normalized scale where the bounding box diameter is 1.
This corresponds to the normalization of the ShapeNet (Chang et al., ) dataset.

Following the evaluation of AutoSDF (Mittal et al., ),
we separately measure objectionable reconstruction artifacts using F-score @1% ($F_{1}$), normal consistency (NC) and inaccurate normals (IN).
The F-score @1% ($F_{1}$) sets a threshold at 1% of the side length of the reconstructed volume (Tatarchenko et al., ),
within which a neighboring point on the ground truth mesh needs to be found from a reconstruction sample for computing precision and vice-versa for recall.
From these, the $F_{1}$ score is computed as usual.

Completeness (CMP) measures the recall within a threshold of $1.5\%$ of the side length, based on the implementation of Wu et al. ().

Normal Consistency (NC) measures the mean absolute dot-product between normals on the reconstructed surface and the closest ground-truth surface point:

$$ $NC(X,Y)=\frac{1}{|X|}\sum_{x\in X}\left\{|n_{x}\cdot n_{y}|~{}:~{}y= \operatorname{\arg\min}_{y\in Y}d(x,y)\right\}$ (6) $$

The absolute value is taken to allow for flipped normals in the ground truth data.
For the normals $n_{x}$ and $n_{y}$, we use the geometric face normals of the faces that generated the sampled points $x$ and $y$.
Inaccurate Normals (IN) complements the NC measure by counting the percentage of normals which are outside of a 5-degree threshold of the normal of the closest ground-truth point.

We also measure Intersection over Union (IoU) based on the sign of the $128^{3}$ SDFs.

## Appendix C Data Preparation

In order to compute SDFs from meshes,
we translate them such that their bounding box is centered at the origin and then uniformly scale them into $[-1,1]^{3}$.
Then, for ShapeNetCoreV1, we remeshed the mesh in Blender via voxelization using a voxel size of $0.008$.
This was necessary to create manifold meshes from which SDFs could be computed.
Admittedly, this resulted in the loss of some geometry which does not have any volumetric counterpart.
Nevertheless, we consider this remeshed version our ground truth data for SDF-based shape completion training.
For evaluation, we excluded objects which lost all geometry in the conversion.
The meshes in the ABC dataset are of much higher quality.
Here, we only normalized the meshes to $[-1,1]^{3}$ and then densely sampled signed distances in a regular grid of size $128^{3}$.

## Appendix D Architecture Details

We implemented all our models in PyTorch (Paszke et al., ) and trained using PyTorch Lightning (William Falcon, ).

The P-VAE is a variational autoencoder built with 3D convolutions.
The encoder converts an SDF-patch of shape $[B,1,32,32,32]$ into a mean
$\mu$ and variance $\sigma^{2}$ vector of shape $[B,8192]$.
To be precise,
the encoder returns $\log(\sigma^{2})$, from which we compute $\sigma^{2}$. During
training, the latent representation of the SDF-patch is sampled from the normal
distribution $z\sim\mathcal{N}(\mu,\sigma^{2})$. During inference we use $z=\mu$.
The architectures of the encoder and the decoder are given in Table 12 and Table 13 respectively.
The ConvBlock specified in Table 10 and the DecoderLayer in Table 11 are reoccurring architectural structures in both the encoder and the decoder.
Note that each layer specifies its predecessor in the *parent* column. Skip connections are joined by adding the output of two parents together.

**Table 10: Architecture of a ConvBlock element as used in the P-VAE Encoder in Table 12**
| Layer Name | Parent Name | Layer Parameters | Output Shape |
| --- | --- | --- | --- |
| Conv3D | - | kernel 3, stride 1, pad 1 | $[B,c_{out},d,d,d]$ |
| BatchNorm3d | Conv3D | - | $[B,c_{out},d,d,d]$ |
| ReLU | BatchNorm3d | - | $[B,c_{out},d,d,d]$ |

**Table 11: Architecture of a DecoderLayer as used in Table 13. If not otherwise noted, all layers use a kernel size of 3 ($[3,3,3]$), stride 1 and padding 1 ($[1,1,1]$). The input has shape $[B,c_{in},d,d,d]$.**
| Layer Name | Parent Name | Layer Parameters | Output Shape |
| --- | --- | --- | --- |
| Conv3D | layer input | - | $[B,c_{out},d,d,d]$ |
| Conv3DTrans | layer input | stride 2, output padding 2 | $[B,c_{out},2d,2d,2d]$ |
| BatchNorm3D | Conv3D | - | $[B,c_{out},d,d,d]$ |
| Upsample | BatchNorm3D | factor 2, trilinear | $[B,c_{out},2d,2d,2d]$ |
| Add | Conv3DTrans, Upsample | - | $[B,c_{out},2d,2d,2d]$ |
| ReLU | Add | - | $[B,c_{out},2d,2d,2d]$ |

**Table 12: Architecture of the P-VAE Encoder. If not otherwise noted, all layers use a kernel size of 3 ($[3,3,3]$) stride 1 and padding 1 ($[1,1,1]$). The input patch has shape $[B,1,32,32,32]$.**
| Layer Name | Parent Name | Layer Parameters | Output Shape |
| --- | --- | --- | --- |
| ConvBlock-1 | SDF patch | - | $[B,32,32,3232]$ |
| ConvBlovk-2 | ConvBlock-1 | - | $[B,32,32,32,32]$ |
| ConvBlovk-3 | ConvBlock-2 | - | $[B,32,32,32,32]$ |
| ConvBlovk-4 | ConvBlock-3 | - | $[B,32,32,32,32]$ |
| Add-1 | ConvBlock-1, SDF patch | broadcasting | $[B,32,32,32,32]$ |
| MaxPool-1 | Add-1 | stride 2 | $[B,32,16,16,16]$ |
| Conv3D-1 | Add-1 | stride 2 | $[B,32,16,16,16]$ |
| Add-2 | MaxPool-1, Conv3D-1 | - | $[B,32,16,16,16]$ |
| Conv3D-2 | Add-2 | kernel 1, padding 0 | $[B,32,16,16,16]$ |
| ConvBlock-5 | Add-2 | - | $[B,64,16,16,16]$ |
| ConvBlock-6 | ConvBlock-5 | - | $[B,64,16,16,16]$ |
| ConvBlock-7 | ConvBlock-6 | - | $[B,64,16,16,16]$ |
| ConvBlock-8 | ConvBlock-7 | - | $[B,64,16,16,16]$ |
| Add-3 | Conv3D-1, ConvBlock-7 | - | $[B,64,16,16,16]$ |
| MaxPool-2 | Add-3 | stride 2 | $[B,64,8,8,8]$ |
| Conv3D-3 | Add-3 | stride 2 | $[B,64,8,8,8]$ |
| Add-4 | MaxPool-2, Conv3D-3 | - | $[B,64,8,8,8]$ |
| Conv3D-4 | Add-4 | kernel 1, padding 0 | $[B,128,8,8,8]$ |
| ConvBlock-9 | Add-4 | - | $[B,128,8,8,8]$ |
| ConvBlock-10 | ConvBlock-9 | - | $[B,128,8,8,8]$ |
| ConvBlock-11 | ConvBlock-10 | - | $[B,128,8,8,8]$ |
| ConvBlock-12 | ConvBlock-11 | - | $[B,128,8,8,8]$ |
| Add-5 | Conv3D-4, ConvBlock-12 | - | $[B,128,8,8,8]$ |
| MaxPool-3 | Add-5 | stride 2 | $[B,128,4,4,4]$ |
| Conv3D-5 | Add-5 | stride 2 | $[B,128,4,4,4]$ |
| Add-6 | MaxPool-3, Conv3D-5 | - | $[B,128,4,4,4]$ |
| Conv3D-6 | Add-4 | kernel 1, padding 0 | $[B,256,4,4,4]$ |
| ConvBlock-13 | Add-6 | - | $[B,256,4,4,4]$ |
| ConvBlock-14 | ConvBlock-13 | - | $[B,256,4,4,4]$ |
| ConvBlock-15 | ConvBlock-14 | - | $[B,256,4,4,4]$ |
| ConvBlock-16 | ConvBlock-15 | - | $[B,256,4,4,4]$ |
| Add-7 | Conv3D-4, ConvBlock-16 | - | $[B,128,8,8,8]$ |
| MaxPool-4 | Add-7 | stride 2 | $[B,128,2,2,2]$ |
| Conv3D-7 | Add-7 | stride 2 | $[B,128,2,2,2]$ |
| Add-8 | MaxPool-4, Conv3D-7 | - | $[B,128,2,2,2]$ |
| Conv3D-8 | Add-8 | kernel 1, padding 0 | $[B,512,2,2,2]$ |
| ConvBlock-17 | Add-8 | - | $[B,512,2,2,2]$ |
| ConvBlock-18 | ConvBlock-17 | - | $[B,512,2,2,2]$ |
| ConvBlock-19 | ConvBlock-18 | - | $[B,512,2,2,2]$ |
| ConvBlock-20 | ConvBlock-19 | - | $[B,512,2,2,2]$ |
| Add-9 | Conv3D-8, ConvBlock-20 | - | $[B,512,2,2,2]$ |
| Conv3D-9 | Add-9 | kernel 1, padding 0 | $[B,1024,2,2,2]$ |

**Table 13: Architecture of the P-VAE Decoder. The decoder converts a input latent code with shape $[B,8192]\rightarrow[B,512,2,2,2]$ into an SDF-patch of shape $[B,1,32,32,32]$.**
| Layer Name | Parent Name | Layer Parameters | Output Shape |
| --- | --- | --- | --- |
| DecoderLayer-1 | latent code | $c_{in}=512$, $c_{out}=512$ | $[B,512,4,4,4]$ |
| DecoderLayer-2 | DecoderLayer-1 | $c_{in}=512$, $c_{out}=256$ | $[B,256,8,8,8]$ |
| DecoderLayer-3 | DecoderLayer-2 | $c_{in}=256$, $c_{out}=256$ | $[B,256,16,16,16]$ |
| DecoderLayer-4 | DecoderLayer-3 | $c_{in}=256$, $c_{out}=128$ | $[B,128,32,32,32]$ |
| Conv3D-1 | DecoderLayer-4 | kernel 1, stride 1, padding 0 | $[B,1,32,32,32]$ |

## Appendix E Training Details

### E.1 Patch-Variational Autoencoder

We trained the P-VAE for 250 epochs with early stopping at epoch 193 with initial learning rate of 1e-4 and CosineAnnealing scheduler.
The batch size for the training was 128 per GPU.

### E.2 AutoDecoder

The refined codes $z^{\prime}$ were optimized with Adam for 200 steps per patch. The
gaussian noise that was added on the initial $z$ from the P-VAE was sampled from a gaussian distribution
$\mathcal{N}(0,1e^{-2})$.

### E.3 SDF Latent Transformer

The masking ratio used for training is $0.4$.
Our learning rate was $1e^{-5}$ and we use a cosine scheduler with 1400 warm-up steps for the transformer and trained for 120k steps. The batch size was 64 per GPU.
All SLT configurations used the same training configuration and setup.

## Peer Reviews

confidence: 5
contribution: 1
initial_rating: 1
presentation: 3
questions: There is no questions, as the method proposed in this paper combines different existing models and therefore straightforward.
soundness: 2
strengths: 1. The paper is easy to understand.
2. The experimental results are better than the compared methods.
summary: This paper proposes a sdf latent transformer for partial object completion. Specifically, it first divides high-resolution voxels into multiple smaller voxels and learns latent representations $z$ of these smaller voxels using a vae. Then, these latent representations are arranged as a sequence and processed by a masked transformer, which shares similarities with masked autoencoder. The experiments are conducted on three datasets for the completion task. The experimental results are better than those of the compared approaches.
weaknesses: The novelty of this paper is very limited.

1. The division of high-resolution voxels into smaller patches is reasonable but is straightforward and thus of limited innovation.

2. The p-VAE is almost identical to the original vae without any adaptation or improvements for this task.

3. The SDF-Latent-Transformer idea is very similar to the masked autoencoder [1], so it lacks novelty.

In short, the method proposed in this paper is somewhat like a combination of different well-known models, therefore, the novelty is the main concern.

[1] He K, Chen X, Xie S, Li Y, Dollár P, Girshick R. Masked autoencoders are scalable vision learners. InProceedings of the IEEE/CVF conference on computer vision and pattern recognition 2022 (pp. 16000-16009).

---

confidence: 5
contribution: 2
initial_rating: 5
presentation: 2
questions: 1. Can authors elaborate on the design choice for the patch size and size of the SDF volume and how changes to those hyper-parameters will affect the proposed method's performance?
2. The SDF volume contains much more information in the shape completion task than point cloud input because it hinted the location of the possible surface. How did the authors address this possible information leak in the evaluation benchmarks? Real-world sensors typically cannot provide SDF volume as an input because the location of the real surface is yet to be known; how could this method be applied to real-world 3D reconstruction applications?
3. Can authors provide the number of parameters in Table 1 to better understand differences compared to the previous method?
4. Can authors provide more information on the quantitative comparison to demonstrate the proposed method is comparable to more recent publications?
soundness: 2
strengths: 1. The authors proposed an efficient architecture that runs much faster than previous diffusion-based methods and auto-regressive-based methods and still demonstrated decent quantitative and qualitative results on shape completion benchmarks. The efficient design and fast running speed are appreciated for possible real-world applications. 
2. This paper addressed an interesting problem of 3D shape completion, which could lead to possible applications in 3D reconstruction and robotics. 
3. The authors compared the proposed method with the previous method quantitatively and qualitatively with high-quality renderings.
summary: The paper works on the problem of partial object shape completion. Given a partial SDF volume, the proposed method first encodes the SDF volume into a latent code for each patch using the proposed Patch Variational Autoencoder(P-VAE), which encodes  SDF patches of resolution 32^3 into a latent code. 4^3 latent code encoded by the P-VAE can be used to train a masked autoencoder for the shape completion task. The proposed method achieved a significant reduction in the running speed thanks to its MAE architecture and short context length (64) for the transformer. The authors tested the proposed method on ShapeNet and ABC datasets and achieved better performance on metrics for reconstruction quality compared to previous methods. The qualitative study also demonstrated the results of reconstruction.
weaknesses: 1. Lack of ablative study on the resolutions of the patch size and number of patches. The 32^3 SDF volume seems to be a relatively large SDF with lots of information. It will be good to have an ablative study with smaller patch sizes, such as 16^3 or 8^3.
2. The presentation of the proposed method is vague and confusing. It will be much easier for the reader to understand if the authors point out they only use a transformer with 8x8x8 context length, and each token encodes the information of a 32^3 SDF volume, and that's why the whole model is lightweight. It would be good if the pipeline figures and method section made this point clear. 
3. Multiple publications have followed AutoSDF, such as DiffComplete, but the authors did not quantitatively compare to those newer publications, which makes it difficult for the reviewer to judge the technical improvements over that method. 
4. Previous shape completion, like AutoSDF and Shapeformer, also demonstrated its power to generate multiple hypotheses when the partial observation is ambiguous. However, the proposed method did not discuss nor benchmark the task of multi-modal shape completion. Would it be possible to also evaluate total mutual difference(TMD)?
5. The authors provided autoencoding results in Table 4 but did not compare them against any baseline methods such as shapeformer, providing limited information on understanding the effectiveness of the power of presentation. 

Minor issues:
- Please use the word autoencoder instead of "Auto Encoder"

---

confidence: 3
contribution: 2
initial_rating: 3
presentation: 2
questions: * Referring to the first weakness, it is hard to understand the function of masking in inference, perhaps should elaborate / clarify.
* Is using a VAE really necessary if the model is non-probabilistic?
* Is composing SDF from patch SDFs that simple? Intuitively it appears that this might cause issues near patch boundaries
soundness: 2
strengths: * Fast inference time: a key advantage over other sequential (eg autoregressive) approaches is utilizing the MAE decoder. 
* Modular Patch-VAE architecture enables generalization by pre-training on small-scale patches, an effective component, even if previously used in related works.  
* High-quality shape completion, especially in capturing fine-grained details.  
* Simple yet effective approach, that avoids unnecessary complexity with a straightforward architecture and objective.
summary: This paper introduces a shape completion method for high-resolution signed distance functions (SDFs), leveraging shared weights to efficiently represent small SDF patches.

The model encodes the entire shape as a sequence of latent codes generated by a patch-based variational autoencoder (P-VAE). Each patch is labeled as either "masked" (incomplete) or "non-masked" (complete), and shape completion is performed in latent space, with the result decoded back into an SDF using the P-VAE decoder. This patch-based approach allows for efficient inference and achieves high-quality results.
weaknesses: - Potential "leakage" issue: Non-masked voxels adjacent to masked patches may encode distances to missing parts, indirectly leaking information about regions to be completed. Discussing this limitation and potential remedies (e.g., using TSDF instead of SDF) would strengthen the work. Beyond conducting an ablation study of usage of SDF vs TSDF, first it should be qualitatively checked how much information is in fact encapsulated in non-masked patches, which regards the masked patches.
Masking in inference seems to require predefined regions marked as "known" or "unknown" potentially limiting the method. If so, this should be clarified as it diverges from traditional shape completion tasks, which typically address irregular partial shapes. You can also suggest possible modifications and future directions which allow general completion patterns.
- The approach is similar to pre-training on (T)SDF patches as done in [1]. This related work should be cited and discussed. Additionally, [1] demonstrated generalization to large scenes; it would be valuable to discuss whether the proposed method can achieve similar scalability.
- Limited model comparisons: The evaluation mainly compares against older methods like DeepSDF, omitting more recent approaches. To represent an implicit method such as DeepSDF, consider utilizing more recent architectures such as Neural Fields as Learnable Kernels for 3D Reconstruction.
- Comparison fairness: Competing models like SDFusion and PoinTR, unlike the proposed model, do not appear to receive explicit masking information on areas to be completed. If true, this gives the proposed model an advantage and should be addressed.
- Metrics: The evaluation relies heavily on UHD, which I believe is secondary in shape completion contexts, mainly measuring the fidelity of preserving the known region. Including additional metrics as is done in baselines would better reveal the model's performance. Those metrics can be all or at least some of the metrics you have used for your inner comparisons. Give precedence to important metrics regarding shape completion, such as Chamfer distance which is elementary as a main metric.
- The ordering of grid coordinates in the latent code (due to positional encoding in the transformer) suggests that the model expects canonically oriented shapes, which may be difficult to achieve in practice given partial input.

**Minor Points:**
- Please clarify the distinction between "transformer encoder" and "transformer decoder" and explain why the encoder allows single-step inference.

**Missing References:**  
[1] _Local Implicit Grid Representations for 3D Scenes_  
[2] _Neural Fields as Learnable Kernels for 3D Reconstruction_

---

confidence: 3
contribution: 3
initial_rating: 6
presentation: 3
questions: - See the weaknesses above.
soundness: 3
strengths: 1. This paper introduces VAE-like encoder-decoder architecture and MAE-like transformer architecture to efficiently complete partial SDF.
2. P-VAE and SLT have shown superior performance on ShapeNet and ABC dataset.
summary: This paper introduces a framework for efficient completion of partial SDF. Its contributions are:
1. A novel framework for efficient completion of partial SDF using VAE which is described as Patch Variational Autoencoder (P-VAE) and MAE-like transformer architecture which is described as SDF-Latent-Transformer (SLT).
2. Evaluation of P-VAE and SLT on ShapeNet and ABC dataset that indicates that P-VAE is superior to some of the state-of-the-art methods.

This paper indicates the effectiveness of P-VAE and SLT with a lot of qualitative and quantitative results. But this method lacks some special designs for SDF and there are still some questions that need to be clarified.
weaknesses: 1. SDF represents the surface of an object with a signed distance function, and the value reflects the distance to the surface. However, P-VAE has no unique design for addressing the conflicts caused by inconsistent values in different SDF patches, which may harm the performance of shape completion based on SDF. The authors should discuss how to handle or mitigate potential inconsistencies between SDF patches or provide an analysis of how much this affects the completion results in practice.
2. In lines 250-251, the authors mention that position encoding is helpful for shape completion, but the paper lacks details like how to design the position embedding and some results that indicate it impacts shape completion. The authors should provide the exact formulation of the positional encoding and the ablation study showing the impact of including against excluding the positional encoding on completion quality.
3. In lines 306-307, measuring the completion performance by measuring how faithfully the geometry given in the input was preserved during the completion may not be enough. The metrics that evaluate the plausibility or realism of the completed regions should also be considered.
4. There are three types of completion, but there is no qualitative result about randomly removing the patches. The authors should include qualitative examples of the random patch removal completion task in their results section or supplementary materials.
5. The paper compares the quantitative results with some state-of-the-art methods only on chairs, which is insufficient to prove the effectiveness of P-VAE and SLT. More object categories should be considered.

Meta-review: The paper describes an autoencoder for 3D shape patches that allows to train a masked autoencoder-like transformer for shape completion.
The reviews of this work are negatively leaning, being concerned about technical novelty, missing applicability to real data and insufficient comparisons.
After taking a closer look at paper and discussions, I agree with the general sentiment of the reviewers and would follow with a reject recommendation.

Decision: Reject