POC-SLT: Partial Object Completion with SDF Latent Transformers

Authors: Faezeh S Zakeri, Raphael Braun, Lukas Ruppert, Hendrik Lensch
Primary area: applications to computer vision, audio, language, and other modalities
Keywords: SDF Completion, Masked Encoder Transformer, Variation Auto Encoder, Scale Invariant, Million-scale Point Processing, Repairing Latent Codes

Abstract:
3D geometric shape completion hinges on representation learning and a deep understanding of geometric data.
Without profound insights into the three-dimensional nature of the data, this task remains unattainable.
Our work addresses this challenge of 3D shape completion given partial observations
by proposing a transformer operating on the latent space representing Signed Distance Fields (SDFs).
Instead of a monolithic volume, the SDF of an object is partitioned into smaller high-resolution patches leading to a sequence of latent codes.
The approach relies on a smooth latent space encoding learned via a variational autoencoder (VAE), trained on millions of 3D patches.
We employ an efficient masked autoencoder transformer to complete partial sequences into comprehensive shapes in latent space.
Our approach is extensively evaluated on partial observations from ShapeNet and the ABC dataset where only fractions of the objects are given.
The proposed POC-SLT architecture compares favorably with several baseline state-of-the-art methods,
demonstrating a significant improvement in 3D shape completion, both qualitatively and quantitatively.

Decision: reject
Reviewer scores: 5, 5, 3, 6

Reviews and discussion:
<thread>
creation date:
1730551120138

thread_type:
official_review

<thread_object>
creator: reviewer_jb2s

<content>
summary:
This paper introduces a framework for efficient completion of partial SDF. Its contributions are:
1. A novel framework for efficient completion of partial SDF using VAE which is described as Patch Variational Autoencoder (P-VAE) and MAE-like transformer architecture which is described as SDF-Latent-Transformer (SLT).
2. Evaluation of P-VAE and SLT on ShapeNet and ABC dataset that indicates that P-VAE is superior to some of the state-of-the-art methods.

This paper indicates the effectiveness of P-VAE and SLT with a lot of qualitative and quantitative results. But this method lacks some special designs for SDF and there are still some questions that need to be clarified.

strengths:
1. This paper introduces VAE-like encoder-decoder architecture and MAE-like transformer architecture to efficiently complete partial SDF.
2. P-VAE and SLT have shown superior performance on ShapeNet and ABC dataset.

weaknesses:
1. SDF represents the surface of an object with a signed distance function, and the value reflects the distance to the surface. However, P-VAE has no unique design for addressing the conflicts caused by inconsistent values in different SDF patches, which may harm the performance of shape completion based on SDF. The authors should discuss how to handle or mitigate potential inconsistencies between SDF patches or provide an analysis of how much this affects the completion results in practice.
2. In lines 250-251, the authors mention that position encoding is helpful for shape completion, but the paper lacks details like how to design the position embedding and some results that indicate it impacts shape completion. The authors should provide the exact formulation of the positional encoding and the ablation study showing the impact of including against excluding the positional encoding on completion quality.
3. In lines 306-307, measuring the completion performance by measuring how faithfully the geometry given in the input was preserved during the completion may not be enough. The metrics that evaluate the plausibility or realism of the completed regions should also be considered.
4. There are three types of completion, but there is no qualitative result about randomly removing the patches. The authors should include qualitative examples of the random patch removal completion task in their results section or supplementary materials.
5. The paper compares the quantitative results with some state-of-the-art methods only on chairs, which is insufficient to prove the effectiveness of P-VAE and SLT. More object categories should be considered.

questions:
- See the weaknesses above.
</content>
</thread_object>

</thread>
<thread>
creation date:
1730661824332

thread_type:
official_review

<thread_object>
creator: reviewer_1tgp

<content>
summary:
This paper introduces a shape completion method for high-resolution signed distance functions (SDFs), leveraging shared weights to efficiently represent small SDF patches.

The model encodes the entire shape as a sequence of latent codes generated by a patch-based variational autoencoder (P-VAE). Each patch is labeled as either "masked" (incomplete) or "non-masked" (complete), and shape completion is performed in latent space, with the result decoded back into an SDF using the P-VAE decoder. This patch-based approach allows for efficient inference and achieves high-quality results.

strengths:
* Fast inference time: a key advantage over other sequential (eg autoregressive) approaches is utilizing the MAE decoder. 
* Modular Patch-VAE architecture enables generalization by pre-training on small-scale patches, an effective component, even if previously used in related works.  
* High-quality shape completion, especially in capturing fine-grained details.  
* Simple yet effective approach, that avoids unnecessary complexity with a straightforward architecture and objective.

weaknesses:
- Potential "leakage" issue: Non-masked voxels adjacent to masked patches may encode distances to missing parts, indirectly leaking information about regions to be completed. Discussing this limitation and potential remedies (e.g., using TSDF instead of SDF) would strengthen the work. Beyond conducting an ablation study of usage of SDF vs TSDF, first it should be qualitatively checked how much information is in fact encapsulated in non-masked patches, which regards the masked patches.
Masking in inference seems to require predefined regions marked as "known" or "unknown" potentially limiting the method. If so, this should be clarified as it diverges from traditional shape completion tasks, which typically address irregular partial shapes. You can also suggest possible modifications and future directions which allow general completion patterns.
- The approach is similar to pre-training on (T)SDF patches as done in [1]. This related work should be cited and discussed. Additionally, [1] demonstrated generalization to large scenes; it would be valuable to discuss whether the proposed method can achieve similar scalability.
- Limited model comparisons: The evaluation mainly compares against older methods like DeepSDF, omitting more recent approaches. To represent an implicit method such as DeepSDF, consider utilizing more recent architectures such as Neural Fields as Learnable Kernels for 3D Reconstruction.
- Comparison fairness: Competing models like SDFusion and PoinTR, unlike the proposed model, do not appear to receive explicit masking information on areas to be completed. If true, this gives the proposed model an advantage and should be addressed.
- Metrics: The evaluation relies heavily on UHD, which I believe is secondary in shape completion contexts, mainly measuring the fidelity of preserving the known region. Including additional metrics as is done in baselines would better reveal the model's performance. Those metrics can be all or at least some of the metrics you have used for your inner comparisons. Give precedence to important metrics regarding shape completion, such as Chamfer distance which is elementary as a main metric.
- The ordering of grid coordinates in the latent code (due to positional encoding in the transformer) suggests that the model expects canonically oriented shapes, which may be difficult to achieve in practice given partial input.

**Minor Points:**
- Please clarify the distinction between "transformer encoder" and "transformer decoder" and explain why the encoder allows single-step inference.

**Missing References:**  
[1] _Local Implicit Grid Representations for 3D Scenes_  
[2] _Neural Fields as Learnable Kernels for 3D Reconstruction_

questions:
* Referring to the first weakness, it is hard to understand the function of masking in inference, perhaps should elaborate / clarify.
* Is using a VAE really necessary if the model is non-probabilistic?
* Is composing SDF from patch SDFs that simple? Intuitively it appears that this might cause issues near patch boundaries
</content>
</thread_object>

</thread>
<thread>
creation date:
1730679285691

thread_type:
official_review

<thread_object>
creator: reviewer_mpjj

<content>
summary:
The paper works on the problem of partial object shape completion. Given a partial SDF volume, the proposed method first encodes the SDF volume into a latent code for each patch using the proposed Patch Variational Autoencoder(P-VAE), which encodes  SDF patches of resolution 32^3 into a latent code. 4^3 latent code encoded by the P-VAE can be used to train a masked autoencoder for the shape completion task. The proposed method achieved a significant reduction in the running speed thanks to its MAE architecture and short context length (64) for the transformer. The authors tested the proposed method on ShapeNet and ABC datasets and achieved better performance on metrics for reconstruction quality compared to previous methods. The qualitative study also demonstrated the results of reconstruction.

strengths:
1. The authors proposed an efficient architecture that runs much faster than previous diffusion-based methods and auto-regressive-based methods and still demonstrated decent quantitative and qualitative results on shape completion benchmarks. The efficient design and fast running speed are appreciated for possible real-world applications. 
2. This paper addressed an interesting problem of 3D shape completion, which could lead to possible applications in 3D reconstruction and robotics. 
3. The authors compared the proposed method with the previous method quantitatively and qualitatively with high-quality renderings.

weaknesses:
1. Lack of ablative study on the resolutions of the patch size and number of patches. The 32^3 SDF volume seems to be a relatively large SDF with lots of information. It will be good to have an ablative study with smaller patch sizes, such as 16^3 or 8^3.
2. The presentation of the proposed method is vague and confusing. It will be much easier for the reader to understand if the authors point out they only use a transformer with 8x8x8 context length, and each token encodes the information of a 32^3 SDF volume, and that's why the whole model is lightweight. It would be good if the pipeline figures and method section made this point clear. 
3. Multiple publications have followed AutoSDF, such as DiffComplete, but the authors did not quantitatively compare to those newer publications, which makes it difficult for the reviewer to judge the technical improvements over that method. 
4. Previous shape completion, like AutoSDF and Shapeformer, also demonstrated its power to generate multiple hypotheses when the partial observation is ambiguous. However, the proposed method did not discuss nor benchmark the task of multi-modal shape completion. Would it be possible to also evaluate total mutual difference(TMD)?
5. The authors provided autoencoding results in Table 4 but did not compare them against any baseline methods such as shapeformer, providing limited information on understanding the effectiveness of the power of presentation. 

Minor issues:
- Please use the word autoencoder instead of "Auto Encoder"

questions:
1. Can authors elaborate on the design choice for the patch size and size of the SDF volume and how changes to those hyper-parameters will affect the proposed method's performance?
2. The SDF volume contains much more information in the shape completion task than point cloud input because it hinted the location of the possible surface. How did the authors address this possible information leak in the evaluation benchmarks? Real-world sensors typically cannot provide SDF volume as an input because the location of the real surface is yet to be known; how could this method be applied to real-world 3D reconstruction applications?
3. Can authors provide the number of parameters in Table 1 to better understand differences compared to the previous method?
4. Can authors provide more information on the quantitative comparison to demonstrate the proposed method is comparable to more recent publications?
</content>
</thread_object>

</thread>
<thread>
creation date:
1730811163556

thread_type:
official_review

<thread_object>
creator: reviewer_aop9

<content>
summary:
This paper proposes a sdf latent transformer for partial object completion. Specifically, it first divides high-resolution voxels into multiple smaller voxels and learns latent representations $z$ of these smaller voxels using a vae. Then, these latent representations are arranged as a sequence and processed by a masked transformer, which shares similarities with masked autoencoder. The experiments are conducted on three datasets for the completion task. The experimental results are better than those of the compared approaches.

strengths:
1. The paper is easy to understand.
2. The experimental results are better than the compared methods.

weaknesses:
The novelty of this paper is very limited.

1. The division of high-resolution voxels into smaller patches is reasonable but is straightforward and thus of limited innovation.

2. The p-VAE is almost identical to the original vae without any adaptation or improvements for this task.

3. The SDF-Latent-Transformer idea is very similar to the masked autoencoder [1], so it lacks novelty.

In short, the method proposed in this paper is somewhat like a combination of different well-known models, therefore, the novelty is the main concern.

[1] He K, Chen X, Xie S, Li Y, Dollár P, Girshick R. Masked autoencoders are scalable vision learners. InProceedings of the IEEE/CVF conference on computer vision and pattern recognition 2022 (pp. 16000-16009).

questions:
There is no questions, as the method proposed in this paper combines different existing models and therefore straightforward.
</content>
</thread_object>

</thread>


Metareview:
metareview:
The paper describes an autoencoder for 3D shape patches that allows to train a masked autoencoder-like transformer for shape completion.
The reviews of this work are negatively leaning, being concerned about technical novelty, missing applicability to real data and insufficient comparisons.
After taking a closer look at paper and discussions, I agree with the general sentiment of the reviewers and would follow with a reject recommendation.