VP-LLM: Text-Driven 3D Volume Completion with Large Language Models through Patchification

## Abstract

Abstract Recent conditional 3D completion works have mainly relied on CLIP or BERT to encode textual information, which cannot support complex instruction. Meanwhile, large language models (LLMs) have shown great potential in multi-modal understanding and generation tasks. Inspired by the recent advancements of LLM, we present Volume Patch LLM (VP-LLM), which leverages LLMs to perform conditional 3D completion in a single-forward pass. To integrate a 3D model into the LLM tokenization configuration, the incomplete 3D object is first divided into small patches that can be encoded independently. These encoded patches are then fed into an LLM along with the text prompt, instructing the LLM to capture the relations between these patches as well as injecting semantic meanings into the 3D object. Our results demonstrate a strong ability of LLMs to interpret complex text instructions and understand 3D objects, surpassing state-of-the-art diffusion-based 3D completion models in generation quality.

## 1 Introduction

3D modeling serves as a pivotal component in a multitude of 3D vision applications including robotics and virtual reality, where the quality of 3D data critically influences model performance. Despite the advancement in 3D scanning technology, the raw data acquired are often noisy, clustered, and may contain large portions of missing data due to occlusion, complex real-world scenes and restricted camera angles, resulting in incomplete 3D acquisition.
This necessitates robust pre-processing to recover or complete the 3D objects, which can enhance the efficiency of subsequent 3D vision tasks.

Current approaches for 3D shape completion typically operate on a depth map or partial point cloud, converting it into a voxel representation or sampling points to restore the original 3D objects. While [, ] showcase advancements, they are confined to specific categories and lack the ability of cross-object generation. Although efforts have been made [, , , ] to create a unified model that handles multi-category 3D completion, these models often overlook the inclusion of textual input in guiding the completion process, leading to uncertainty when the input is ambiguous, as well as degradation of feasibility when given captions deviate from the training set.
Consequently, methods are needed to generate a completed shape aligning precisely with the provided text description. Some attempts mimic the 2D diffusion method [] or score distillation sampling (SDS) [] to incorporate text guidance in the 3D completion tasks, but they cannot be precisely controlled when the description is complicated, and are very time-consuming to generate the results.

To this end, we propose Volume Patch Large Language Model (VP-LLM), which achieves 3D completion with precise textual control. Inspired by the recent progress in 3D multi-modality models [, , ], we believe that Large Language Models (LLMs) can underpin our approach by decoding the complex associations between 3D structures and textual descriptions.
LLMs, pretrained on large-scale text datasets, have the capability to process long sequences and comprehend complex human languages, while 3D models represented by voxel grids can be straightforwardly converted into a one-dimensional format through flattening. Therefore, we investigate how to enable LLMs to understand a 3D model by decoding complex correlations between 3D structures and textual descriptions, or “translating” it into a “sentence”.

For seamless incorporation of 3D data into the LLM tokenization framework, 3D models are initially segmented into smaller patches, facilitating independent encoding and decoding. Different from most previous methods that manage the 3D object as a unified, this idea of patchification is more scalable and extendable.
The patchified 3D voxel volume can be processed as a sequence and fed into the LLM with the textual description after alignment. The LLM can fuse the 3D and textural features into its hidden latents, which are finally decoded into complete 3D models.

Our whole pipeline is presented in Fig. 1. The 3D volumes are first patchified into individual patches and processed by a patch-wise Variational Autoencoder (VAE) to encode individually. The encoded patches are then projected and concatenated with user-specified text conditions to the LLM. Finally, the output projection layer extracts the features generated by the LLM and lets the VAE decode back each patch individually.

In summary, the major contributions of our papers are

- 1.
VP-LLM is the first work leveraging LLM to achieve text-guided 3D completion, outperforming existing state-of-the-art text-conditioned 3D completion works.
- 2.
The patchification method we proposed enables a scalable integration of 3D volumes into the LLM, solving the difficulty in handling high-resolution voxel grids faced by existing works.
- 3.
Our work can perform 3D completion and denoising for multiple categories using detailed text control within one unified model.

Thus, this experimental paper offers insights and lessons learned, providing the first LLM solution to text-guided 3D object completion. Codes and data will be made public upon the paper’s acceptance.

## 2 Related Works

### 2.1 Multimodality Large Language Models

The advent of Large Language Models (LLMs) has significantly accelerated advancements in natural language processing. Several studies [, , ] have demonstrated the capabilities of LLM in comprehending long contexts, ensuring scalability and adaptability, and facilitating the understanding and generation of natural language. Benefiting from these advantages of LLM, many works have already employed LLM across different modalities, including images [, , ], motion [], and video [, ]. Recently, several works combined LLM with 3D data. For example, [] leverages a 3D-aware VQ-VAE [] and integrates its codebook into the LLM’s vocabulary, enabling the LLM to generate and understand 3D objects. But the codebook size may bottleneck the capability of LLM to tackle 3D objects with more complex and various structures.
LLM-Grounder [] carefully designs LLM prompt to translate the instructions into regular sub-tasks and instructs some pre-trained 3D grounders [, ] for 3D reasoning, where 3D models are not integrated into the LLM.
Octavius [] adopts the object detector to first discover candidate regions, followed by the application of pre-trained point cloud encoders for extracting features at the instance level. These features are then aggregated and mapped into an LLM for diverse 3D understanding tasks. However, this process reduces the entire 3D model to a single feature, thereby omitting crucial detailed information.
In contrast, VP-LLMemploys a VAE [] combined with projection layers, capable of effectively aligning the 3D latent space with the LLM’s text space, thus enhancing the model’s generalizability, especially for out-of-distribution data.

### 2.2 Text-to-3D Generation

Prior to the era of machine learning, primitive works attempted to retrieve 3D assets from large databases, such as [, ]. With the rise of GANs [], attempts such as Text2Shape [] started to dominate the 3D generation field.
Recently, due to promising advancements in text-to-image generation, research focus on text-control 3D generation has shifted to diffusion model [].
Some works [, , ] adopt CLIP [] to align the rendered images with the input text, thus ensuring the semantic meaning of the 3D model, while others like [, , , , , ] leverage pre-trained 2D diffusion models to provide text control and score distillation sampling to improve the 3D consistency. Although text-to-3D generation serves as an inspiration for text-guided 3D completion, none of the existing methods employ large language models for 3D-text interaction to guide the completion results.

### 2.3 3D Completion

3D completion is a crucial process in various industries, enabling accurate and efficient design and production, and enhancing the overall quality of products and projects. Early works such as [, , , , , , ] that use 3D convolutions with structured representation require high memory usage and compute. Followed by [], many works [, , , ] that adopt point clouds as 3D representation for shape completion were proposed. For example, [, , ] generate the final shape in an auto-regressive manner, while [] uses GANs [] to complete the model. But none of these offer satisfactory user control. With the introduction of DDPM [], works such as  [, , , , , , , ] have advanced the 3D shape completion pipelines. Notably, [] adopts a 3D diffusion model with VAE to achieve 3D completion with multi-type control, and [] uses score distillation to perform test-time optimization. However, neither of them can perform completion tasks at a larger scale where model details are required, nor even generate satisfactory results without tedious denoising steps. Thus, scalability, speed and level of control are still issues to be addressed, providing strong motivation for our work.

## 3 Methodology

Given an incomplete 3D model and user-supplied textual description of the target 3D model, our model aims to recover the underlying 3D model aligned with the input text. First, the incomplete 3D model undergoes patchification, where it is split into small patches, and each patch is independently encoded by our Variational Autoencoder (VAE). Next, a shared-weight linear layer maps the patch features to the embedding space of the LLM, which are then combined with the textual description and input into the LLM. The LLM, concatenated with our specially-designed output projection layer, generates the features of patches at all positions, allowing for the separate decoding and subsequent assembly, or de-patchification, to the underlying complete 3D model.

Figure: Figure 2: Patchification: given a 3D object, we first fit it into a voxel grid and then divide it into a sequence of small patches. Next, we utilize a patch-wise Variational Autoencoder (VAE) to extract the features of each patch individually and then reconstruct it back. It is important to note that only one VAE is trained for all the patches throughout the entire dataset, making our method a scalable approach.
Refer to caption: x2.png

### 3.1 Patchification

The first step of our method is to divide the 3D models into small patches, dubbed patchification. Figure 2 demonstrates the process of patchification. For a 3D object represented in voxel $V\in\{0,1\}^{H\times W\times D}$, $V(x,y,z)=1$ if the position $x,y,z$ is occupied and 0 otherwise. Patchification uniformly partitions the 3D voxel volume into $p$ small patches of the same size, each containing a local region of the entire object. For each patch $P_{i,j,k}\in\{0,1\}^{h\times w\times d}$, the coordinate for position $(x,y,z),0\leq x\leq h,0\leq y\leq w,0\leq z\leq d$ is:

$$ $\displaystyle P_{i,j,k}(x,y,z)=V(i\cdot h+x,\;j\cdot w+y,\;k\cdot d+z).$ (1) $$

Thus, $p=\lfloor H/h\rfloor\cdot\lfloor W/w\rfloor\cdot\lfloor D/d\rfloor$. In our experiments, we set $H=W=D=64$ and $h=w=d=8$.

After patchification, a patch Variational Autoencoder (VAE) is adopted to extract the feature for each patch independently. Our patch VAE consists of an encoder $\mathbf{E}$ and a decoder $\mathbf{D}$, where $\mathbf{E}$ encodes a patch into a Gaussian distribution $\mathcal{N}({\mu},{\sigma})$ where $\mu$ and $\sigma$ are mean and variance, respectively, and $\mathbf{D}$ recovers the original patch from this distribution. The VAE training loss for a single patch $P$ is defined as

$$ $\displaystyle\mathcal{L}_{\mathit{VAE}}(P)=\mathcal{L}_{\mathit{BCE}}\left(P, \;\mathbf{D}(\mathbf{E}(P))\right)+\beta\mathcal{L}_{kld}(\mathbf{E}(P)),$ (2) $$

where $\mathcal{L}_{\mathit{BCE}}$ is the binary cross entropy loss, $\mathcal{L}_{kld}$ is the KL-divergence and $\beta$ is a hyperparameter.

Benefiting from this patchification structure that allows for independent encoding and decoding of each patch, VP-LLM ensures that when the completions of certain patches are undesired, they do not affect the performance of other well-performed patches, solving the problems faced by previous works that encode or decode the entire scene collectively.

Figure: Figure 3: The training process of the input projection (left) and output projection (right). During the input projection training, a single share-weighted MLP maps the masked or noisy 3D tokens encoded by our patch-wise VAE to the embedding space of the LLM. After wrapping the prompt with the 3D tokens as input and feeding them to the LLM, we back-propagate the loss calculated between the ground-truth caption and the LLM’s prediction, enabling the LLM to learn to generate captions that accurately describe the 3D object from the input patches. For the output projection, we freeze the input projection layer and train the output projection layer, while also fine-tuning the LLM with LoRA. The output projection layer comprises a Transformer and a cluster of MLPs, such that after passing the Transformer, every 3D token is processed independently with an MLP.
Refer to caption: x3.png

### 3.2 Mask Strategy

To enhance the understanding of incomplete 3D models, we designed three different strategies to mask out different parts of the original 3D input, aiming to mimic the possible user inputs during the inference stage. Specifically, the following three strategies will be applied randomly with the same possibility:

- 1.
Random Mask: Given the input 3D model in $p$ patches, we randomly set $m_{r}\cdot p$ patches to 0 0 (unoccupied), where $m_{r}$ is a mask ratio sampled within a pre-defined range;
- 2.
Plane Mask: Given the input 3D model represented in voxel $V(x,y,z)$, we first project the model onto $x$-axis and find the first and last occupied voxels, denoted as $x_{1}$ and $x_{2}$, along $x$-axis. Next, a plane parallel to $yOz$ is sampled with $x$-coordinate between $[x_{1},x_{2}]$. Intuitively, such a plane cuts the 3D model into two parts, and we then discard one of them by setting all voxels to be 0 0 (unoccupied) to simulate large portions of missing data in real capture;
- 3.
Random Noise: Since real-world 3D models usually contain noises and artifacts, we mimic this situation by randomly inverting voxel occupancy (setting occupied voxels to be unoccupied, and vice versa). The noise level is also sampled in a pre-defined range indicating how many voxels to invert.

### 3.3 Input Projection Layer Training

The input project layer, which can map the VAE latent space into the LLM input embedding space, is a single linear model operated on each VAE latent patch.
After patchifying and encoding the incomplete 3D model, each patch, represented by its respective $\mu$ and $\sigma$, is first reparameterized into a single feature $f$ by sampling from the Gaussian distribution $\mathcal{N}({\mu},{\sigma})$. The feature (or each encoded patch) going through the input projection layer becomes a 3D token, which then can be understood by the LLM. To train the input projection layer, the LLM is instructed to predict the caption of the incomplete 3D model, given the prompt “Given the incomplete 3D input, describe the 3D model by text.” We use the training loss in [] which is the negative log-likelihood on the caption. We use the training loss from the LLM to supervise this stage of training.

### 3.4 Output Projection Layer Training

The LLM, with the output projection layer appended, takes as input the tokenized sequence of the underlying incomplete 3D model, the user-supplied caption of the complete model as well as the instructions for completion, and outputs the complete 3D model aligned with the textual description. To be specific, we formulate the prompt for LLM as ‘‘Given the caption, recover the incomplete 3D model, <tokenized incomplete 3D sequence>, <Caption>’’. The output projection layer architecture employs a transformer consisting of a 2-layer encoder and a 2-layer decoder, translating the LLM hidden states into a sequence of separable latent codes. We observe that to sufficiently explore the highly fused information in the LLM, more layers of hidden states are necessary. Thus, we select $5$ layers in our experiments, which balances the amount of information with computational complexity. To ensure the length of the generated sequence, we set an additional learnable token sequence as the target. We then utilize Multi-layer Perceptrons (MLPs) to individually map each token to the desired 3D token. The final result is obtained through the concatenation of the mapped tokens.

During training, the input projection layer is frozen, while the LLM is finetuned with LoRA [], while the output projection layer is trained from scratch. We use mean-squared error (MSE) loss between the output projection layer output and the VAE latent of ground-truth 3D model patches to update our model.

## 4 Experiments

### 4.1 Dataset

We train our model on a subset of ShapeNet [] dataset, comprising over 3000 objects. To obtain the detailed textual description of 3D models in human languages, we adopt Cap3D [], which leverages BLIP to predict and GPT-4 to refine captions for 3D models. In our experiments, the resolutions of the 3D voxels and patches are respectively set as $64\times 64\times 64$ and $8\times 8\times 8$, while we explore the capability of the model to handle higher resolution formats in Sec. 5.2.

To improve the robustness of our model, we apply data augmentation during training. For the 3D models, we rotate them along one random axis with an arbitrary angle making the order of sequence different, while for the captions of the 3D model, we adjust the GPT configurations in Cap3D. Only 3D data augmentation is used in input projection training, while both 3D data and caption augmentation are used in output projection training. More details of the data augmentation can be found in Appendix C.

### 4.2 Comparison

**Table 1: Quantitative results compared with SDFusion and 3DQD. We can observe that our method consistently hits the lowest (best) Chamfer Distance (CD) and the highest (best) CLIP-s score compared with SDFusion (text-conditioned completion) and 3DQD (label-conditioned completion). Moreover, our method is capable of denoising extremely noisy 3D inputs, while the baselines cannot accomplish the task.**
| Methods | Seg 20% | Seg 50% | Seg 80% | Noise 1% | Noise 2% |  |  |  |  |  |
| --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- |
| CD.$\downarrow$ | CLIP-s.$\uparrow$ | CD.$\downarrow$ | CLIP-s.$\uparrow$ | CD.$\downarrow$ | CLIP-s.$\uparrow$ | CD.$\downarrow$ | CLIP-s.$\uparrow$ | CD.$\downarrow$ | CLIP-s.$\uparrow$ |  |
| Ours | 10.96 | 27.80% | 11.37 | 27.71% | 17.42 | 25.17% | 16.03 | 23.92% | 34.44 | 23.07% |
| SDFusion | 95.44 | 26.66% | 137.31 | 26.20% | 235.98 | 22.22% | – | – |  |  |
| 3DQD | 172.89 | 22.63% | 170.20 | 22.62% | 196.12 | 22.62% | – | – |  |  |

**Table 2: Comparison of our method with SDFusion and 3DQD, on Airplane dataset. We can clearly see our method outperforms their methods when the input is segmented by a plane and performs reasonably well when the input is added with noises. Since the two baselines cannot work on noisy inputs, “N/A” is placed instead.**
|  | Ground Truth | Masked/Noisy | Ours | SDFusion | 3DQD |
| --- | --- | --- | --- | --- | --- |
| Seg 20% |  |  |  |  |  |
|  | “3D model of a Boeing 747-400 featuring a spherical fuselage shell, truncated oblate wings, |  |  |  |  |
|  | and made of aluminum and steel.” |  |  |  |  |
| Seg 50% |  |  |  |  |  |
|  | “3D model of a toy airplane featuring a wing, fuselage, tail, rudder, and propeller, |  |  |  |  |
|  | available in 3ds Max, OBJ, FBX, and C4D formats.” |  |  |  |  |
| Seg 80% |  |  |  |  |  |
|  | “Royalty-free 3D model of a stealth fighter jet, featuring a delta wing |  |  |  |  |
|  | with horizontal and vertical stabilizers” |  |  |  |  |
| Noise 1% |  |  |  | N/A | N/A |
|  | “3D model of a Saber fighter jet featuring detailed wings, fuselage, and multiple landing gears, |  |  |  |  |
|  | compatible with 3ds Max, Maya, Blender, and other 3D software.” |  |  |  |  |
| Noise 2% |  |  |  | N/A | N/A |
|  | “Royalty-free 3D model of a Boeing 747-400 jumbo jet.” |  |  |  |  |

**Table 3: Comparison of our method with SDFusion and 3DQD, on Car dataset. We can clearly see our method outperforms their methods when the input is segmented by a plane and performs reasonably well when the input is added with noises. Since the two baselines cannot work on noisy inputs, “N/A” is placed instead.**
|  | Ground Truth | Masked/Noisy | Ours | SDFusion | 3DQD |
| --- | --- | --- | --- | --- | --- |
| Seg 20% |  |  |  |  |  |
|  | “3D model of a Chevrolet Tahoe pickup truck in black and red, available in multiple formats including OBJ and FBX.” |  |  |  |  |
| Seg 50% |  |  |  |  |  |
|  | “Royalty-free 3D model of a Mercedes-Benz SLK sports car” |  |  |  |  |
| Seg 80% |  |  |  |  |  |
|  | “3D model of a muscle car with a hood, fenders, and a hood scoop.” |  |  |  |  |
| Noise 1% |  |  |  | N/A | N/A |
|  | “3D model of a yellow Dodge Viper SRT10 sports car.” |  |  |  |  |
| Noise 2% |  |  |  | N/A | N/A |
|  | “A 3D model of a police car, available royalty-free, with previews from different angles.” |  |  |  |  |

We compare our model with SDFusion [] and 3DQD [], which are two state-of-the-art diffusion-based methods on conditional 3D completion tasks. SDFusion accepts multi-modality input for shape completion, so in our case, we only enable text-conditioned completion and enforce no image condition.

The quantitative comparison results are shown in Tab. 1. Following [, ], we use Chamfer Distance (CD) and CLIP-s score for evaluation. For Chamfer Distance, we transform our volume representation into point clouds and use the coordinates in the voxel grids. For the CLIP-s score, we render 20 different-view 2D images around each volume and take the maximums among the CLIP feature scores between the images and the textual description. We test the performance of the models under various circumstances, by segmenting the different fractions of the object and adding random noise to the objects.
We also provide visualization results in Tab. 2 and Tab. 3.

It is worth noticing that 3DQD is a label-conditional completion model that does not include detailed text control during inference, leading to large variance in the prediction results.
Though SDFusion contains text control in completion, it adopts BERT [] for text encoding, thereby suffering from complex text understanding.

More qualitative results
are presented in Appendix E.

## 5 Ablation Study

**Table 4: Comparison using different LLM structures. We utilize two LLMs, namely Mistral-7B and Gemma-2B, to train our whole pipeline with the same dataset as before. For comparison purposes, we demonstrate the Chamfer Distance (CD) and CLIP-s score on data containing 1% noise and 20% mask.**
| Methods | Seg 20% | Noise 1% |  |  |
| --- | --- | --- | --- | --- |
| CD.$\downarrow$ | CLIP-s.$\uparrow$ | CD.$\downarrow$ | CLIP-s.$\uparrow$ |  |
| Mistral-7B | 10.96 | 27.80% | 16.03 | 23.92% |
| Gemma-2B | 11.19 | 28.08% | 10.64 | 26.34% |

### 5.1 Comparison of different LLM architectures

LLM is one of the most important components in our model, whose ability to capture multi-modal semantic information and token relations may greatly affect the performance of the entire pipeline. To investigate the effect of different LLMs on the performance of our model, we re-trained our method on Mistral-7B [] and Gemma-2B []. To ensure a fair comparison, we keep the LoRA ranks for the two models to be the same. From Tab. 5.1, we find that under this setting, the two LLMs demonstrate similar results on mask completion, while the performance of Gemma-2B is better than Mistral-7B on denoising tasks while slightly worse in completion from masked inputs. Higher performance can be expected if larger LLM models or higher LoRA ranks are deployed.

### 5.2 Scalability on Higher Resolution Voxel Volumes

Our VAE encodes and decodes each patch of the 3D model individually, thus enabling our model to scale to higher voxel resolutions.
To demonstrate the scalability, we have expanded upon our previous experiments by setting $H=W=D=64$, and further increasing the voxel resolution to $H=W=D=72$, while maintaining the patch size at $8$. By fixing the patch size, we can ensure that the fine details of the data remain consistent as we increase the input scale. After this operation, the sequence length of each 3D object will increase from $512$ to $729$. We can see from Tab. 5 that the two resolutions have comparable results, indicating our method can successfully generalize to higher voxel volume resolutions.

## 6 Limitation

Our method has been proven effective on small LLMs such as Mistral-7B. However, due to the limitation of computational resources, LLMs with stronger capability to understand long sequences, such as 70B or larger scale models, are not employed in our model. Thus, huge potential of our method is still left to probe. Additionally, though we verify the effectiveness of our method by patchification on voxel volumes, which can be intuitively extended to point clouds and SDFs, it is still very hard to employ it on those nascent 3D representations like NeRF [] and 3D Gaussian Splatting (3DGS []) that encoded 3D in an implicit (MLP weights for NeRF and Gaussians for 3DGS). More investigations on how to patchify such representations are left in future works.

## 7 Conclusion

In this paper, we present VP-LLM, which combines text and 3D through LLMs to achieve text-guided 3D completion. We introduce a novel approach called patchification to incorporate 3D models into LLMs, and adopt a two-stage training process that allows LLMs to understand input incomplete 3D models and generate entire 3D models. Such an approach also allows an independent encoding and decoding process, thereby ensuring the scalability of our method. Experiments on the ShapeNet dataset validate that our method surpasses the state-of-the-art methods in the 3D completion task. Moreover, our model can achieve satisfactory results from noisy 3D inputs, a practical issue in real 3D data capture.

## Appendix A Implementation Details

#### Dataset preparation

We randomly selected 3130 data from ShapeNetCore [], and randomly split data of each category into 90% training data and 10% testing data.

#### Model structure

We present the hyperparameters in Tab. 6. Those values can determine the detailed structures of each component in our pipeline.

**Table 6: Hyperparameters used to configure model structure.**
| Hyperparameter | Value |
| --- | --- |
| VAE Encoder Convolution Layer Num | 2 |
| VAE Decoder Convolution Layer Num | 2 |
| VAE Hidden Dimension | 64 |
| VAE Latent Size | 128 |
| LoRA Rank | 32 |
| LoRA Alpha | 32 |
| LoRA Dropout | 0.05 |
| LLM Type | Mistral-7B |
| Output Projection Transformer Encoder Layer Num | 2 |
| Output Projection Transformer Decoder Layer Num | 2 |
| Output Projection Transformer Feedforward Dimension | 2048 |
| Output Projection Transformer Num Heads | 4 |

#### Training configuration

We train our patch VAE on 2 RTX 4090 cards for 100 epochs and batch size using ShapeNet voxels volumes with resolution $64^{3}$ and adopt the same VAE for voxels with different resolutions. This training takes approximately 2 hours. The model is trained with AdamW [] optimizer with a learning rate $3e^{-4}$.

Our input projection and output projection models are trained on 8 RTX 6000 Ada GPU cards until converge, for around 100 and 500 epochs respectively. For Mistral-7B, these processes take around 1 hour and 18 hours, respectively, while for Gemma-2B, the numbers go down to 20 minutes and 8 hours. Both stages are trained with AdamW optimizer as well, with input projection training using a learning rate of $3e^{-4}$ and output projection training using $5e^{-4}$ or $5e^{-5}$, depending on the LLM size.

## Appendix B Examples of Ground-Truth and Predicted Captions

Here we present some examples of ground-truth captions and the captions predicted by our LLM model during input projection layer training. We notice that the captions are not perfect, while they provide adequate semantic meanings.

Predicted Caption 1: 3AD model of a ", featuring a exterior such as wings, fuselage, and, andinglets, and, and, andilerons, and flaps" with for 3 and7-400 and 747-800 variants.",

Ground-Truth Caption 1: "3D model of Boeing aircraft, featuring detailed components such as wings, fuselage, tail, winglets, rudder, elevators, ailerons, and flaps, available in both 747-400 and 737-800 variants."

Predicted Caption 2: 3D model of a rectangular 737- featuring a fuselage body with a wings, and tail tail tail, and, and, andilerons, and a gear.,

Ground-Truth Caption 2: 3D model of a Boeing 747, featuring a cylindrical fuselage, elliptical wings, a truncated cone tail, rudder, elevators, ailerons, and landing gear.

Predicted Caption 3: 3D model of of a guitars Ghostcar aircraft jets, including a-A-18E F-14., with for download3ds Max. OBJ.,

Ground-Truth Caption 3: 3D model collection of various Phantom and Super Hornet fighter jets, including F/A-18 and F-16 variants, available for 3ds Max and Maya.

Predicted Caption 4: 3D model of a rectangular 737-800 aircraft a fuselage dome, with a conelate spher, and a- a. steel.,

Ground-Truth Caption 4: 3D model of a Boeing 747-400 featuring a spherical fuselage shell, truncated oblate wings, and made of aluminum and steel.

Predicted Caption 5: 3 with a cylindrical, a, and, and, and,, and landing landing gear.,

Ground-Truth Caption 5: A spaceship featuring a wing, fuselage, tail, propeller, rotor blade, and retractable landing gear.

Predicted Caption 6: 3D model of a electric with a, a, and, and, and, andilerons, and gear, and a.,

Ground-Truth Caption 6: 3D model of an aircraft featuring wings, fuselage, tail, rudder, elevators, ailerons, landing gear, and propeller.

Predicted Caption 7: 3Aalty-free 3D model of a female 737-400 aircraft featuring a exterior such as a fuselage, fuselage, and tail.",

Ground-Truth Caption 7: "Royalty-free 3D model of a Boeing 747-400, featuring detailed components such as a wing, fuselage, and tail."

Predicted Caption 8: 3D model of a rectangularliner with a fuselage wing, a fuselage, and, and, andders, and, and gear, and landing.,

Ground-Truth Caption 8: 3D model of a jet plane featuring a delta wing, triangular fuselage, tail, fin, rudders, propeller, landing gear, and hull.

## Appendix C Illustrations of Data Augmentation

#### 3D Data Augmentation

During training, we performed 3D augmentation by randomly rotating each 3D object with a different angle with respect to either $x$, $y$, or $z$ axis. Figure 4 visualizes this result.

Figure: Figure 4: 3D data augmentation example result of an airplane.
Refer to caption: x4.png

#### Caption Augmentation

During training, we leveraged Cap3D [] to generate ground-truth captions for every 3D model. To perform caption augmentation, we run Cap3D for three times, using GPT-4-Turbo, GPT-4-Turbo with another seed, and ChatGPT (GPT-3.5). Here we present the captions generated by them.

Figure: Figure 5: From left to right: Object 1, 2, 3, 4.
Refer to caption: extracted/5653073/figs/supp-augmentation/8855c5531c093275146f724acb952fba_airplane_gt.png

Captions for Object 1:

- GPT-4 Seed 1: Royalty-free 3D model of a Boeing 747-400 featuring detailed components including a cylindrical fuselage, delta wings, tail, rudder, elevators, and ailerons.
- GPT-4 Seed 2: Royalty-free 3D model of a Boeing 747-400 featuring a cylindrical fuselage, delta wings, a tail, rudder, elevators, and ailerons, representing a four-engine jet airliner.
- ChatGPT: Boeing 747-400 3D model featuring a fuselage, wings, and tail, a jumbo jet with a delta wing design and four engines.

Captions for Object 2:

- GPT-4 Seed 1: A 3D model of a two-seater, single-engine RC airplane featuring a four-bladed, fixed-pitch propeller, retractable tricycle landing gear, and control surfaces including ailerons, rudder, and elevator.
- GPT-4 Seed 2: A 3D model of a small, two-seater, single-engine RC airplane featuring a retractable tricycle landing gear, a fixed-pitch, four-bladed propeller, and control surfaces including wings, fuselage, tail, rudder, elevator, and ailerons.
- ChatGPT: 3D model of a two-seater single-engine airplane with retractable landing gear and a four-bladed propeller.

Captions for Object 3:

- GPT-4 Seed 1: Royalty-free 3D model of a blue McLaren MP4-12C sports car with polygonal geometry.
- GPT-4 Seed 2: Royalty-free 3D model of a McLaren MP4-12C sports car with polygonal geometry.
- ChatGPT: 3D model of a McLaren MP4-12C sports car.

Captions for Object 4:

- GPT-4 Seed 1: 3D model of a police car, available royalty-free, featuring detailed polygonal geometry.
- GPT-4 Seed 2: 3D model of a police car, featuring detailed polygonal vertices and edges, available royalty-free.
- ChatGPT: A detailed 3D model of a police car.

## Appendix D Unlock Better Quality via Iterative Completion

We also found that our model is able to refine the results without further training, by directly passing the output from the last step and the caption into our model. We show this using the denoising example as the changes are more obvious. After applying the denoising twice, we observe that the quality in Fig. 6 has significantly increased.

Figure: Figure 6: Result on Iterative Denoising
Refer to caption: extracted/5653073/figs/iteratively/supp_iter.png

## Appendix E More Results

Figure 7 to 16
present more results of more categories, notice that all these results are inferenced from the same checkpoint as used in the experiment session.

Figure: Figure 7: Results Seg$20\%$
Refer to caption: x5.jpeg

Figure: Figure 8: Results Seg$20\%$
Refer to caption: x21.jpeg

Figure: Figure 9: Results Seg$50\%$
Refer to caption: x37.jpeg

Figure: Figure 10: Results Seg$50\%$
Refer to caption: x53.jpeg

Figure: Figure 11: Results Seg$80\%$
Refer to caption: x69.jpeg

Figure: Figure 12: Results Seg$80\%$
Refer to caption: x85.jpeg

Figure: Figure 13: Results Noise$1\%$
Refer to caption: x101.jpeg

Figure: Figure 14: Results Noise$1\%$
Refer to caption: x115.jpeg

Figure: Figure 15: Results Noise$2\%$
Refer to caption: x131.jpeg

Figure: Figure 16: Results Noise$2\%$
Refer to caption: x147.jpeg

## Peer Reviews

confidence: 3
contribution: 3
initial_rating: 6
presentation: 3
questions: 1. What about the performance of VP-LLM if the text description is quite different from the training set generated by Cap3D?
2. Since the authors claim the model can perform completion tasks at a larger scale, why not show the 3D completion results for real-world scanned objects?
3. Why choose separate MLPs to process and generate each patch? How to deal with the huge memory cost if the number of patches increases?
4. Why not use the MLLM to support multi-modal conditions?
soundness: 3
strengths: 1. The paper introduces an innovative approach by leveraging Large Language Models (LLMs) for 3D volume completion guided by textual prompts. This novel integration of patchification with LLMs addresses the scalability issues seen in traditional voxel-based approaches, enabling finer control through text guidance.
2. The methodology is well-grounded, with detailed explanations of the VP-LLM model architecture, including the patchification process, VAE integration, and LLM fine-tuning steps. The use of quantitative metrics (Chamfer Distance and CLIP-s score) and comparisons against state-of-the-art models like SDFusion and 3DQD lend credibility to the results. 
3. This work is significant for 3D computer vision, offering a scalable alternative to diffusion-based methods with increased flexibility through text control.
summary: The paper presents VP-LLM, which leverages Large Language Models (LLMs) to achieve text-guided 3D volume completion. Traditional methods for 3D completion, typically reliant on diffusion models, face scalability limitations, particularly in handling high-resolution voxel grids and understanding complex textual descriptions. VP-LLM addresses these issues by introducing a "patchification" technique, which segments the 3D volume into small patches. These patches are encoded independently using a Variational Autoencoder (VAE) and then mapped to the LLM’s embedding space, allowing for precise text-guided completion.
weaknesses: 1. The comparison is only conducted on the Airplane and Car dataset, which is not aligned with the baseline SDFusion. It is highly recommended to provide more comparison results on the chair, table, or house for a fair comparison.
2. The model only supports text description as the condition. However, the baseline SDFuion enables conditional generation with multiple modalities like text description and RGB image. The image guidance is a more user-friendly way so missing this condition will limit the application capability of the model.
3. The experiments primarily use ShapeNet data and Cap3D-generated captions, which may limit the generalizability of results. Evaluating additional datasets or real-world 3D captures could strengthen the argument for VP-LLM’s applicability to diverse scenarios.
4. Compared with diffusion-based methods, how is the generation diversity of VP-LLM?

---

confidence: 3
contribution: 3
initial_rating: 5
presentation: 3
questions: - The performance of the compared baselines appears significantly poorer than reported in their original papers (e.g., SDFusion results mostly show disconnected points, whereas smoother volumes would be expected). It is unclear what causes this discrepancy. Converting the SDF to an isosurface mesh, rather than visualizing voxels directly, might provide a more accurate comparison.
- Although the author acknowledges the low resolution of the volumes, the current results remain suboptimal. Many studies have focused on better compression and generation of SDF/occupancy-based volumes, such as Michelangelo [1] and 3DShape2VecSet [2]. These works propose encoding high-resolution 3D volumes into compact sequential codes using a VAE, which could be better alternatives.
- It is unclear why the projection layers utilize independent MLP clusters instead of a more commonly used shared MLP head.

[1] Conditional 3D Shape Generation based on Shape-Image-Text Aligned Latent Representation  
[2] 3DShape2VecSet: A 3D Shape Representation for Neural Fields and Generative Diffusion Models
soundness: 3
strengths: * Overall, the idea of leveraging pretrained LLMs for the 3D completion task is quite novel. Previous studies typically concentrate on integrating LLMs for 3D perception tasks, whereas this work is the first to utilize LLMs for 3D generative tasks. 

* The paper is well-organized and clearly presented, enhancing its readability and comprehension.
summary: The paper introduces a novel framework that enables large language models (LLMs) to perform 3D shape completion. The framework processes incomplete or noisy 3D volumes along with textual instructions to interactively generate complete or refined models. In this approach, the 3D volume is divided into patches and sequentialized for input into a transformer backbone, which can remain frozen during training. This work is the first to leverage pretrained LLMs for 3D shape completion, with experimental results demonstrating the potential of this design.
weaknesses: The primary concern is that the demonstrated task is overly simplified, and the experimental results are not sufficiently convincing:
- The resolution of 64 is too low to capture detailed representations. Such low-resolution volumes fail to accurately reflect the textual descriptions (e.g., "perpendicular wings" and "solar plane" in Table 4), thereby diminishing the motivation for instruction-based generation.
- All presented examples appear to involve simple shapes, such as airplanes, cars, and lamps. ShapeNet includes more complex shapes, such as tables and chairs, which would better showcase the framework's performance and robustness.

---

confidence: 3
contribution: 3
initial_rating: 5
presentation: 3
questions: + Does the method have an advantage in 3D generation? I did not see any obstacle in extending it to 3D generation. Why not test in this task? 
 
+ The author can compare the denoising part with some other methods, such as Point-DAE, and Point-E. The comparison method mentioned in the paper does not support it. Comparing with some other models that can reduce noise can further illustrate the effectiveness of the method.
soundness: 3
strengths: + The overall pipeline seems reasonable by combining existing techniques. 
+ Leveraging LLM to complete 3D shapes is interesting. It would be interesting to investigate how large pre-trained models can benefit other modal tasks. 
+ VP-LLM outperforms existing methods such as SDFusion and 3DQD, showing its effectiveness in 3D completion and denoising tasks. Compared with the baseline, VP-LLM can reduce more severe noise, while the baseline cannot do this.
summary: This paper presents VP-LLM for text-driven 3D completion. The authors divide a shape into smaller patches and encode each patch with a patch-wise variational autoencoder (VAE), allowing the LLM to process 3D data alongside textual instructions effectively. VP-LLM uses the ability of LLMs to complete 3D models and demonstrates good denoising capabilities. Experimental results show that VP-LLM outperforms existing models in 3D completion and denoising tasks.
weaknesses: - The shape rendering is of very low quality, making it hard to judge shape details. 
- Although the paper emphasizes the advantages of LLM in understanding complex text instructions and long contexts, there is a lack of comparative analysis. The author did not explain the shortcomings of traditional methods (such as CLIP or BERT) in specific tasks, and in what specific aspects VP-LLM surpasses these methods. There is a lack of detailed discussion on why LLM is inherently superior to other multimodal models designed specifically for 3D tasks.
- The generalization ability of the method needs further evaluation. The data distribution of ShapeNet is different from the real world. It is interesting to see results on more complex datasets, e.g., Objaverse.
- Only two baseline methods were selected in the comparative experiment. The author can try to compare with point cloud-based methods， There are relatively more alternatives. 
- The method requires the use of a decoder for completion, which may limit its generalization ability on more complex or different types of 3D data. However, the paper did not conduct an in-depth analysis of whether the output part would limit the model's capabilities.
- This paper conducted an ablation study on different LLM architectures and voxel resolutions but did not study the impact of other key hyperparameters (such as patch size and number of patches). Testing how these factors affect performance will help evaluate the robustness and flexibility of VP-LLM.

---

confidence: 4
contribution: 1
initial_rating: 3
presentation: 3
questions: Please see the weakness section.
soundness: 3
strengths: The proposed approach shows it is possible to use LLM with 3D modalities through LORA. Furthermore, the proposed approach shows a way of interactively completing shapes, which has been underexplored so far. The denoising and completion tasks are quite important results of this manuscript.
summary: This paper introduces an LLM-based solution to shape completion. Given a 3D model and a text prompt, first, the incomplete model is projected to a sequence of latent vectors through a VAE. Then it is projected to the LLM latent space and concatenated with the encoded prompt. The LLM (+LORA) parses it and decodes a patched version of the complete model. Finally, a decoder reconstructs the final 3D model. The different components are trained, or fine-tuned, independently.
The capabilities of the model are shown on ShapeNet and compared with different alternative approaches.
weaknesses: Different points in the paper concern me. I have listed them below and I hope the authors can address these points in the rebuttal.

1. ShapeNet is known to have axis-aligned models, what happens when this is not the case?
2. The authors should present results on other datasets.
3. The shape completion and editing tasks are quite similar to shape editing when encompassing a text prompt. Such an application would make the paper much stronger. Has the author tested this application? In particular, the input shape would be a complete one, rather than incomplete.

These 3 points are those that drive my final score. I hope the authors can address them appropriately during the rebuttal period.

Meta-review: The paper introduces the VP-LLM, which leverages large language models (LLMs) for text-guided 3D volume completion. Several notable limitations are raised. The results were primarily based on ShapeNet with limited comparisons to baseline methods. The low-resolution outputs weaken the claims of scalability and practical use cases. The lack of multi-modal conditioning (e.g., image guidance) and design choices further limits its novelty. I’d recommend the authors to consider all these and further improve the work.

Decision: Reject