MiniGPT-3D: Efficiently Aligning 3D Point Clouds with Large Language Models using 2D Priors

Venue: acmmm_org_ACMMM_2024_Conference
Published: 2024-04-07 10:05:57

Abstract: Large 2D vision-language models (2D-LLMs) have gained significant attention by bridging Large Language Models (LLMs) with images using a simple projector. Inspired by their success, large 3D point cloud-language models (3D-LLMs) also integrate point clouds into LLMs. However, directly aligning point clouds with LLM requires expensive training costs, typically in hundreds of GPU-hours on A100, which hinders the development of 3D-LLMs. In this paper, we introduce MiniGPT-3D, an efficient and powerful 3D-LLM that achieves multiple SOTA results while training for only 27 hours on one RTX 3090. Specifically, we propose to align 3D point clouds with LLMs using 2D priors from 2D-LLMs, which can leverage the similarity between 2D and 3D visual information. We introduce a novel four-stage training strategy for modality alignment in a cascaded way, and a mixture of query experts module to adaptively aggregate features with high efficiency. Moreover, we utilize parameter-efficient fine-tuning methods LoRA and Norm fine-tuning, resulting in only 47.8M learnable parameters, which is up to 260x fewer than existing methods. Extensive experiments show that MiniGPT-3D achieves SOTA on 3D object classification and captioning tasks, with significantly cheaper training costs. Notably, MiniGPT-3D gains an 8.12 increase on GPT-4 evaluation score for the challenging object captioning task compared to ShapeLLM-13B, while the latter costs 160 total GPU-hours on 8 A800. We are the first to explore the efficient 3D-LLM, offering new insights to the community. We will release the code and weights after review.

Main paper: 
 introduction: 
 Large Language Models (LLMs) have recently driven advancements in multiple fields [15, 35, 45, 46] , benefiting from their world knowledge. Built on LLMs, large 2D vision-language models (2D-LLMs) [4, 27, 62] can align image features with text through an image feature projector, enabling 2D-LLMs to understand visual content. Inspired by 2D-LLMs, large 3D point cloud-language models (3D-LLMs) [39, 40, 51] aim to incorporate 3D point cloud features into LLMs, equipping LLMs with the ability to perceive and reason in 3D space. These 3D-LLMs hold promise for widespread applications in fields like robotics [44, 48] and autonomous driving [10, 15] . However, 3D-LLMs are expensive to train. For example, training PointLLM-13B [51] takes 213 total GPU-hours on 8 A100 GPU, making research and applications extremely challenging. Here, we aim to find a more efficient way to connect 3D point clouds with LLMs.We observe that existing 3D-LLMs directly align point cloud encoders with LLMs. Although these encoders can produce somewhat unified features through multimodal pre-training, there is still a significant modality gap between 3D points with LLMs, requiring substantial resources for alignment. Besides, in contrast to resourceintensive alignment between vision and language, 3D point clouds and 2D images are both visual modalities, which makes it easier to align their representations. Thus, we pose a question: Can we use 2D-LLMs as a strong prior to connect LLMs and 3D data, making alignment more efficient? In other words, as shown in Figure 2 , leveraging pre-trained 2D-LLMs directly allows for cutting down the cost of vision-language alignment, leaving only the 2D-3D vision alignment, which is significantly cheaper. Following this intuition, we propose MiniGPT-3D, an efficient 3D-LLM that connects 3D point clouds and LLMs using 2D-LLMs as priors. Our MiniGPT-3D achieves multiple state-of-the-art (SOTA) results, requiring only 27 hours of training on a single RTX 3090 GPU. Specifically, we propose an efficient four-stage training strategy in a cascaded way, gradually allowing the model to learn unified visual-textual representations. This process achieves the smooth transfer of priors from 2D-LLM to the 3D space, thus efficiently constructing a bridge from 3D to LLM. Moreover, we introduce the Mixture of Query Experts (MQE), which comprises multiple query experts and an expert router, enabling the adaptive aggregation of features from multiple experts with only 0.4M parameters. MQE dynamically adjusts the cooperation relationship between experts, thereby aggregating 3D features from multiple perspectives into the semantic space of 2D-LLM. Meanwhile, we employ various parameter-efficient fine-tuning (PEFT) technologies like LoRA [21] and Norm fine-tuning, and utilize an efficient LLM, further reducing the model's training overhead.As shown in Figure 1 , MiniGPT-3D achieves new SOTA performance on generative 3D object classification and object captioning tasks. Specifically, compared to the powerful baseline ShapeLLM-13B [39] , MiniGPT-3D achieves a 6.77% increase in classification average accuracy and an 8.12 increase in GPT-4 evaluation score. Notably, MiniGPT-3D utilizes extremely cheaper training resources (1× RTX 3090 vs. 8× A800), with up to 6× acceleration (26.8h on RTX 3090 vs. 160h on A800). Furthermore, our model has significantly fewer trainable parameters, reduced by up to 260×, with 2.95B model parameters in total, which is decreased by up to 4.6×.MiniGPT-3D takes the first step in efficient 3D-LLM, we hope that MiniGPT-3D can bring new insights to this community. In summary, our contributions are as follows:• We present MiniGPT-3D, an efficient and powerful 3D-LLM that aligns 3D points with LLMs using 2D  
related work 2.1 large 2d vision-language models: 
 The exceptional instruction-following and generalization capabilities of LLMs [46, 49, 53, 55] have been integrated into vision, leading to the emergence of large 2D vision-language models (2D-LLMs). Early works such as Flamingo [1] and BLIP-2 [27] successfully use projectors to align vision information to LLMs. More recently, most works mainly focus on improving model capabilities through expanding the instruction-tuning dataset [5, 30, 61] , increasing resolution of image [2, 31] , enhancing image encoders [7, 59] . Meanwhile, some methods [8, 9, 57, 60] have also begun to explore efficient 2D-LLM. Models like TinyLlama [60] and TinyGPT-V [57] use Phi-2 [33] , an efficient LLM, to achieve easily deployable 2D-LLMs. Among them, TinyGPT-V leverages LoRA [21] technology and pre-trained modules to achieve extremely efficient fine-tuning. However, TinyGPT-V can only handle 2D images, efficient 3D-LLM remains unexplored, and we aim to fill this gap.Large 3D point cloud-language models (3D-LLMs) introduce LLM into the point cloud modality [6, 20, 23, 29, 36, 39, 40, 51, 54] . Early attempt [20] renders 3D objects into 2D multi-view images, then utilizes 2D-LLM to understand 3D. However, the absence of direct perception of raw point cloud data limits its comprehension of 3D geometry. To address this issue, recent works [6, 23, 36, 40] propose to discard the "rendering" and encode point cloud directly, followed by modal alignment to fixed LLMs via trainable projectors. PointLLM [51] and ShapeLLM [39] show that models can be enhanced after fully fine-tuning. However, the training of 3D-LLMs is expensive. For instance, PointLLM-13B requires training on 8 A100 GPUs for up to 216 total GPU-hours. We observe that with 2D-LLM as visual prior, we can not only bypass the "point cloud rendering", but also make this hierarchical alignment extremely efficient. Therefore, we propose MiniGPT-3D, different from existing 3D-LLMs which aligns 3D points directly to LLMs, our MiniGPT-3D leverages the powerful priors from 2D-LLM as a linkage between LLM and 3D points, using only a RTX 3090 to train for 27 hours.Mixture of Experts (MoE) [22, 24] is an ensemble learning technique that adaptively activates selected modules, referred to as experts, based on input. MoE is widely used in various fields [14, 25, 26, 42, 43] . Shazeer et al. introduce MoE into NLP for the first time, where each intervening layer between LSTM layers serves as an expert. Gshard [26] further expands the MoE to Transformer [47] , treating each Feed-Forward Neural Network (FNN) as an expert. Recently, with the emergence of LoRA, several works [13, 16, 58 ] design FFN's LoRA network as an expert to efficiently fine-tune LLM. Moreover, OneLLM [19] introduces MoE to the learned projector of 2D-LLM, with each projector serving as an expert. In our work, we integrate the MoE concept into the queries of Q-Former [27] , treating each set of queries as an expert. These experts adaptively aggregate point cloud features across diverse extraction perspectives. 
method: 
 In this section, we first introduce the architecture of MiniGPT-3D (Sec. Figure 3 depicts the architecture of MiniGPT-3D, which consists of the six main components: a point cloud encoder, a point cloud projection layer (MLP), a Q-Former, a mixture of query expert (MQE), a modality projector, and a large language model. The MiniGPT-3D framework introduces a two-step projection process, transforming the point cloud from 3D to 2D and then to 1D. Specifically, the point cloud is passed to the point cloud encoder to extract 3D features. Subsequently, features are then projected into a 2D semantic space using the point cloud projection layer. Finally, leveraging the 2D-LLM modules including the Q-Former, modality projector, Norm of LLM, and LoRA of LLM, features in 2D-LLM space are transduced into the 1D-text space of LLM, enabling efficient alignment between 3D and LLM. Additionally, MQE enhances MiniGPT-3D's comprehensive and accurate perception of 3D objects. Details are presented in the following sections. 𝑃 ∈ R 𝑛×𝑑 is input to 𝑓 𝑝𝑐 ,where 𝑛 is the number of points and 𝑑 denotes the feature dimension of each point. Then, 𝑓 𝑝𝑐 outputs a point feature sequence 𝑋 ∈ R 𝑚×𝑏 , comprising 𝑚 features, each with a dimension of 𝑏. In our experiments, we employ the Point-BERT [56] model, pre-trained on ULIP-2 [52] using the Objaverse [12] dataset, as the point cloud encoder. To maintain pre-training knowledge, we freeze the encoder's parameters on all training stages.Point Cloud Projection Layer. The point cloud projection layer 𝑓 𝑀𝐿𝑃 is an MLP with two linear layers, which embeds point features 𝑋 into the semantic space of the pre-trained 2D Q-Former [27] , aligning their dimensions. Concisely, 𝑌 = 𝑓 𝑀𝐿𝑃 (𝑋 ), where 𝑌 ∈ R 𝑚×𝑏 ′ and 𝑏 ′ is the hidden space dimension of Q-Former.Features in 2D-LLM space to LLM. This part transduces the point cloud representation in the 2D semantic space of 2D-LLM to the 1D text space of LLM.Q-Former. The Q-Former 𝑓 𝑄𝐹 , with a decoder-based Transformer structure, transforms point features 𝑌 into point queries 𝑄. This process not only enhances the information extracted from point cloud features but also reduces input size for subsequent LLM, accelerating training and inference. Concisely, 𝑄 = 𝑓 𝑄𝐹 (𝑌, 𝑄),where 𝑄 ∈ R 𝑜 ×𝑏 ′ , 𝑄 ∈ R 𝑜 ×𝑏 ′ .𝑄 is the queries of Q-former and 𝑜 is the number of query. In experiments, we initialize Q-Former with BLIP-2 [27] pre-trained weights. Given Q-Former's extensive 105M parameters, we employ PEFT technologies to fine-tune its Query, Key, and Value layers, and normalization layers, thus enhancing adaptability to point clouds while preserving 2D knowledge.Mixture of Query Experts. Inspired by multi-view image rendering for 3D-to-2D projection, we propose the Mixture of Query Experts (MQE) to achieve a similar effect. In the process of MQE, multiple sets of queries (query expert) are used to transform point features into the semantic space of 2D Q-Former. MQE is the first EQUATIONEQUATIONTo enable query experts to learn knowledge within a stable 3D-LLM semantic context, MQE is only utilized in the final training stage, by which time other modules have completed training.Modality Projector. We use an MLP as the modality projector to bridge the modality gap between point cloud and text, while transforming point queries 𝑄 ∈ R 𝑜 ×𝑏 ′ into point tokens𝑇 𝑝𝑐 ∈ R 𝑜 ×𝑐 , where 𝑐 denotes the shared dimension of both point and text tokens.Large Lanuguage Model Backbone. To minimize GPU memory usage during training, we utilize Phi-2 [33] with 2.7 billion parameters as the large language model backbone of MiniGPT-3D.In MiniGPT-3D, the LLM backbone 𝑓 𝑙𝑙𝑚 processes a sequence of tokens 𝑇 = 𝑡 1 , 𝑡 2 , . . . , 𝑡 𝑗 ∈ R 𝑗 ×𝑐 , where 𝑗 is the number of EQUATIONwhere T = t1 , t2 , . . . , t𝑗 ∈ R 𝑗 ×𝑐 , and t𝑖 denotes the predicted 𝑖-th token, based on the semantics of all previous tokens {𝑡 <𝑖 }. Subsequently, t𝑖 is passed through a linear layer 𝑓 𝑙𝑙𝑚→𝑣𝑜𝑐𝑎𝑏 to be mapped into the vocabulary space. A softmax operation is then applied to compute a probability distribution across the vocabulary, with the word of highest probability designated as the prediction 𝑧 𝑖 for t𝑖 . The process can be formulated as:EQUATIONEQUATIONAs LLMs are primarily trained on text, a perception gap arises when processing non-textual information. Therefore, we adapt PEFT technology LoRA [21] to the LLM backbone, and also further fine-tune the normalization layers, preserving learned knowledge and reducing computational overhead.To gradually transfer the priors of 2D-LLM to point cloud modality and enhance the nascent 3D-LLM's comprehension, our training process includes four stages, each focusing on a distinct task, as shown in Figure 3 . The following subsections will describe them.As shown in Figure 3 (a), the first stage aims to bridge the knowledge gap between the 3D point cloud encoder and 2D-LLM modules, facilitating a seamless transition from 3D to 2D. We solely train the point cloud projection layer (MLP), with other modules frozen. Initialization is sourced from ULIP-2 [52] for the encoder, BLIP-2 [27] for Q-Former, and TinyGPT-V [57] for normalization layers of LLM, LoRA of LLM, and the modality projector. Since the frozen Q-Former from BLIP-2 is also used in TinyGPT-V, MiniGPT-3D only owns two knowledge domains from 3D of ULIP-2 and 2D-LLM of TinyGPT-V before training. To build a robust bridge between domains, we train the projection layer using 660k caption-point cloud pairs, involving 1.4M parameters, as detailed in Table 1 .In the second stage, our objective is to transfer the vision-language knowledge domain to 3D space, establishing the 3Dlanguage knowledge domain. As shown in Figure 3 (b), we fine-tune four parts: the point cloud projection layer (MLP), the Q-Former, the modality projector, and the LLM. Utilizing the 3D-2D bridge of the first stage, 2D-LLM modules, via fine-tuning, gain comprehension of 3D point clouds and gradually transfer the powerful priors to be the 3D-language knowledge. During this process, to minimize the impact of the 3D-2D bridge, we employ the identical dataset from the first stage to train 47.4M parameters, as outlined in Table 1 .To gain better 3D-language knowledge, we further fine-tune the modules trained in the second stage and utilize a more challenging dataset, including detailed caption-point cloud pairs and conversations, to empower MiniGPT-3D with the capabilities to comprehend and respond to complex instructions.During the prior stages, using a single set of queries restricts 3D perception perspective, leading to incomplete cognition. To refine MiniGPT-3D's perception, we introduce MQE to adaptively activate suitable multiple query experts for Q-Former, as shown in Figure 3(d) . Distinct from the preceding three stages focusing on rapidly establishing 3D-language knowledge, this stage presents a stable semantic context for query experts to learn knowledge efficiently. Specifically, we only fine-tune 0.4M MQE-related parameters, reusing the dataset from the third stage to minimize the impact of data distribution changes, as outlined in Table 1 .The training objective of MiniGPT-3D aims to minimize the discrepancy between predicted and true probability distributions at each token position. Given a point cloud and corresponding text instruction, MiniGPT-3D outputs a sequence T . Next, T is processed by 𝑓 𝑙𝑙𝑚→𝑣𝑜𝑐𝑎𝑏 and then a softmax operation is applied to obtain the probability distribution over the vocabulary for each output token, denoted as 𝑇 . The training loss is formulated as follows:EQUATIONwhere the ℎ(•) represents the LLM's tokenizer. 𝐺 is the ground truth text. The 𝐶𝑟𝑜𝑠𝑠𝐸𝑛𝑡𝑟𝑜𝑝𝑦 (•) refers to the cross-entropy loss function. Notably, we only compute the loss for the generated text. 
experiments 4.1 experimental settings: 
 Utilizing one RTX 3090 GPU with 24GB of RAM, we train MiniGPT-3D with only 47.8M trainable parameters in 26.8 hours. We adopt the AdamW optimizer with a weight decay of 0.05 and a cosine decay with linear warm up learning rate schedule. The initial learning rate decreases gradually as the training stage advances, as shown in Table 1 . We use the point-text instruction dataset [51] , including 660K brief-description instructions and 70K complex instructions. 200 objects are splited as test data, following PointLLM [51] and ShapeLLM [39] . For each input point cloud 𝑃 ∈ R 𝑛×𝑑 , the number of point 𝑛 is 8192, and the dimension 𝑑 is 6. We default point clouds without color to black. For a fair comparison, we adopt the identical versions models of GPT-4 [35] ("gpt-4-0613") and ChatGPT [34] ("gpt-3.5-turbo-0613") as our evaluation tools, like prior works [39, 51] . We choose multiple SOTA 3D-LLMs [18, 20, 39, 51] and two popular open-source 2D-LLMs [11, 32] as our baselines.We conduct the generative 3D object classification tasks [51] on ModelNet40 [50] and Objaverse [12] datasets to assess MiniGPT-3D's categorical cognitive ability.Settings. For a fair comparison, we utilize the classification evaluation settings similar to prior works [39, 51] . We employ identical prompts: the Instruction-typed (I) prompt "What is this?" and the Completion-type (C) prompt "This is an object of". Point clouds and these prompts are fed into our MiniGPT-3D, outputting textual responses. For close-set zero-shot classification on ModelNet40, ChatGPT processes the text responses of MiniGPT-3D to select predicted categories from 40 ModelNet40 classes. For open-vocabulary classification on Objaverse, GPT-4 is employed as an evaluator to determine whether MiniGPT-3D's text response refers to the same category as the ground-truth caption.Results. Experimental results are shown in Table 2 . We achieve SOTA performance on all classification benchmarks using only one RTX 3090. Specifically, compared to the best baseline, ShapeLLM [39] , we achieve significant improvements of 7.78% and 5.75% in average accuracy on ModelNet40 and Objaverse datasets, respectively. Unlike other methods using LLM (7B or 13B) that require fine-tuning on 8 A100 or 8 A800 for hundreds of total GPU-hours, our MiniGPT-3D only utilizes a 2.7B LLM and trains 47.8M parameters on a single RTX 3090 GPU in 27 hours. These demonstrate the superiority and efficiency of our MiniGPT-3D, which leverages the powerful priors from 2D-LLMs to build 3D-LLM. Additionally, we observe that MiniGPT-3D exhibits the best performance in recognizing 3D objects of the ModelNet40 dataset unused during training, indicating its stronger generalization ability compared to other methods. Furthermore, even if using different prompts ( I and C) on the Objaverse dataset, MiniGPT-3D demonstrates highly consistent classification performance compared to other 3D-LLMs that have a 4% accuracy gap, showcasing its robustness for diverse prompts.To assess the model's understanding of 3D object details, we perform the 3D object captioning task.Settings. For a fair comparison, we follow the evaluation settings of prior works [39, 51] . We use the prompt "Caption this 3D model in detail". We adopt three distinct evaluation methods: human evaluation, GPT-4 [35] evaluation, and traditional metric evaluation. In human evaluation, volunteers evaluate the model using standardized processes from PointLLM [51] . Specifically, focusing on object Table 4 : Qualitative comparisons. The classification and caption results of models on ModelNet40 and Objaverse are presented. Our MiniGPT-3D generates more detailed and insightful responses compared to other baselines.What is this? Briefly caption this 3D model. Ground Truth Laptop A cartoon black monster like a dragon InstructBLIP-13B [11] symbol letter l a black lizard with a sharp tooth in a dark room LLaVA-13B [32] A small, grainy, black and white letter j. A 3D model of a dark, menacing dragon. 3D-LLM [20] -A black and white tiger with long legs, standing on its hind leg. Point-Bind LLM [18] This is a laptop computer. The 3D model features a large, ornate gargoyle with a horned helmet, sitting on top of a building. PointLLM-13B [51] The 3D model represents a notebook computer, typically a laptop. The 3D model depicts a menacing black dragon, with its mouth opened wide revealing a row of sharp teeth.This is a 3D model of a sleek, black laptop. The laptop is rectangular in shape and has a glossy finish, suggesting a high-quality build. It has a flat screen, which is a common feature in modern laptops, and it's equipped with a keyboard and a touchpad for user interaction.The laptop appears to be in good condition, suggesting it could be used for various tasks such as work, entertainment, or education.This is a 3D model of a fearsome, black-colored dinosaur. The dinosaur possesses an aggressive stance, with sharp claws extending from its arms and legs. It has long black fangs that seem ready to attack. attributes (such as type, color, material, etc.), volunteers visually assess objects and assign correctness scores and hallucination scores to captions. Correctness measures model accuracy in describing attributes, while hallucination evaluates fabricated details' severity. Each attribute, correct or hallucinated, receives a point. Precision is calculated as the ratio of correct information in model-generated content. The Inter-Annotator Agreement score is 0.89 on ICC1k, indicating volunteers' high consistency in cognitive understanding and scoring criteria. GPT-4 evaluates semantic similarity between our model's output and manually annotated captions. In traditional metric evaluation, like prior works [39, 51] , we use data-driven metrics like Sentence-BERT [41] and SimCSE [17] , instead of BLEU-1 [37] , ROUGEL [28] , and METEOR [3] , because the latter lack sufficient capabilities in semantic evaluation.Results. As shown in Table 3 , our MiniGPT-3D achieves SOTA performance on multiple metrics. Specifically, MiniGPT-3D outperforms ShapeLLM-13B [39] , by a large margin of 8.12 on the GPT-4 evaluation score, setting new SOTA with only 2.7B LLM, indicating robust 3D detail comprehension. Also, compared to ShapeLLM-13B, MiniGPT-3D surpasses 1.02 and 1.41 on Sentence-BERT and Sim-CSE metrics, respectively, achieving new SOTA with its remarkable ability to generate accurate captions matching ground truth. Human evaluation further reveals MiniGPT-3D's superior correctness and precision scores compared to baselines. Notably, even with a 2.7B LLM, MiniGPT-3D exhibits a hallucination score comparable to SOTA, surpassing larger 13B LLM-based methods. These outstanding results showcase MiniGPT-3D's fine-grained understanding of 3D objects, inheriting the cognitive capabilities of 2D-LLM.Figure 1 (e) qualitatively shows the MiniGPT-3D's powerful ability to perceive 3D object details. Our MiniGPT-3D precisely extracts information from 3D objects, encompassing categories, colors, shapes, materials, and internal component relationships. Additionally, MiniGPT-3D can perform reasonable reasoning based on object cues, such as potential occurrence periods and locations. Figure 1 (f) further demonstrates MiniGPT-3D's comprehension of 3D object information in open-ended dialogues. MiniGPT-3D accurately outputs 3D object-related world knowledge, showcasing its extensive textual knowledge inherited from LLMs.In sample 1 of Figure 4 , our MiniGPT-3D successfully recognizes the shape, screen, and keyboard of a laptop, compared to other methods. Furthermore, it can deduce the potential usage of this 3D object. In the more complex sample 2 of Figure 4 , our MiniGPT-3D demonstrates superior understanding capabilities of 3D objects by recognizing additional features like the dinosaur's sharp claws and inferring its potential action intentions, compared to other methods.In this section, we conduct ablation studies to investigate various model design options. Herein, we report the total average accuracy of MiniGPT-3D on the generative classification benchmark.Training process . We conduct ablation study to validate the efficacy of our four-stage training strategy. The results in Table 5 highlight the optimal performance achieved by our approach. Specifically, comparing Row #4 vs. #6, we observe that the first 10 , our experiments demonstrate that two MLP layers offer the optimal setup, as excessive or insufficient layers can result in information loss, compromising overall performance.Router type of MQE. The routing mechanism in MQE regulates the cooperation among query experts. The constant router [25] assigns static average weights, while the soft router [38] dynamically assigns weights during training. The sparse router [43] selects the top two experts based on the dynamic weights provided by the soft router. We explore these router types in Table 11 , finding that the sparse router, which dynamically assigns weights and selects the most promising experts, maximizes the capabilities of MQE. Discussion. MiniGPT-3D's limitations lie in its training on object-level datasets, preventing it from understanding large-scale point clouds. Moreover, like existing 3D-LLMs, our MiniGPT-3D solely focuses on comprehending static 3D objects, lacking the capacity to recognize the actions of dynamic objects. We will extend our 3D-LLM building approach to autonomous driving scenarios. 
surface. original mesh (only for reference) point cloud input: 
 The 3D model represents a traditional windmill, characterized by its circular base and tall vertical blades. The mill is painted in shades of brown to mimic the color of wood or stone used in real-life mills. It's positioned on top of a green field, suggesting it could be part of an agricultural setting or historical site.Elaborate on the details of this point cloud, please.This is a 3D model of an ornate, antique-style globe. The globe features intricate designs and patterns that suggest it might be from the Renaissance period or earlier. It's mounted on a sturdy wooden base with four legs for stability. This type of globe could serve as both a decorative piece in homes or offices due to its aesthetic appeal, while also being educational tool for learning about geography.I am interested in this, can you explain? We present MiniGPT-3D, an efficient and powerful 3D-LLM that aligns 3D point clouds with large language models using 2D priors from large 2D vision-language models. This figure demonstrates MiniGPT-3D's superior performance and efficient training compared to existing 3D-LLMs. We also show some prediction examples in 3D recognition, captioning, and question-answering tasks, with the correct and fine-grained answers highlighted in green.Unpublished working draft. Not for distribution. 
 
 Appendix:  


Decision: The paper introduces a 3D-language model that leverages pre-trained 2D vision-language models (2D-LLMs) as a prior for more efficient training. All of the reviewers agree that it is a nice paper. The AC also agrees with their comments and recommends the paper be accepted.

Reviews: review: paper_summary: paper_summary: The architecture consists of several components: a 3D encoder, a 3D projection layer, a Query Transformer (Q-Former), a mixture of query experts, a modality projector, and a large language model (LLM). The training process is strategically divided into four stages to progressively align 3D embeddings with language embeddings. Despite its minimal number of trainable parameters, this model achieves efficient training and superior performance compared to existing methods in 3D object classification and captioning.
strength_weakness: strength_weakness: 1. The paper presents a complex technical design featuring multiple modules and sequential training stages, illustrating a sophisticated approach to aligning 3D point clouds with language.
2. The experiment and ablation sections are comprehensive. Discussions and comparisons with related works are thorough.
3. The paper is well-written and easy to follow.
limitations: limitations: 1. The modules introduced in this paper, such as the modality projector, Mixture of Experts (MoE), and Q-Former, are well-established in the field of VLMs. Only reusing these familiar techniques may lead to perceptions of limited novelty at the idea level within this paper.
2. I am curious about the network’s capabilities in open-world settings or its understanding at the scene level. Does the proposed model demonstrate any advancements in these aspects?
3. In the object captioning experiments, the MiniGPT-3D achieves a notably higher GPT-4 score compared to other methods, yet the improvement in sentence semantic metrics (such as Sentence-BERT and SimCSE) is marginal. Could you provide an analysis or discussion on this phenomenon? What might explain the significant disparity in performance across these metrics?
paper_summary: paper_summary: The paper introduces MiniGPT-3D, an innovative 3D point cloud-language model (3D-LLM) that leverages pre-trained 2D vision-language models (2D-LLMs) as a prior for more efficient training. Unlike traditional 3D-LLMs that require extensive resources, MiniGPT-3D achieves state-of-the-art (SOTA) results on 3D object classification and captioning tasks with significantly reduced resource usage and training time. The model uses a novel four-stage training strategy and a Mixture of Query Experts (MQE) module to adaptively aggregate features, employing parameter-efficient fine-tuning methods such as LoRA and Norm fine-tuning.
strength_weakness: strength_weakness: 1. Efficiency in Training: MiniGPT-3D drastically reduces the training resources and time required compared with previous models like ShapeLLM13B.
2. Well-designed Training Strategy: The paper introduces a cascaded four-stage training strategy that effectively aligns 3D point cloud data with LLMs using 2D priors. This method facilitates a more efficient transfer and integration of visual-textual representations from 2D to 3D.
limitations: limitations: 1. The 2D priors are embedded within 2D-LLM parameters without a clear method for aligning 2D and 3D representations, such as using multi-view images of point clouds. This lack of detailed alignment strategy brings into question how well the 3D model utilizes the 2D information.
paper_summary: paper_summary: The paper presents MiniGPT-3D, an efficient 3D point cloud-language model (3D-LLM) that leverages 2D vision-language models (2D-LLMs) as priors to align 3D point clouds with LLMs. The authors propose a four-stage training strategy that progressively transfers knowledge from 2D-LLMs to the 3D space, requiring only 27 hours of training on a single RTX 3090 GPU. MiniGPT-3D introduces a Mixture of Query Experts (MQE) module for feature aggregation and uses parameter-efficient fine-tuning methods, resulting in a model with significantly fewer trainable parameters than existing methods. The model achieves state-of-the-art (SOTA) results on 3D object classification and captioning tasks with substantially reduced training costs.
strength_weakness: strength_weakness: Efficiency: MiniGPT-3D demonstrates a highly efficient training process, requiring minimal GPU hours and leveraging a consumer-grade GPU (RTX 3090), making it accessible for more researchers.
Performance: The model achieves SOTA results on generative 3D object classification and captioning tasks, outperforming other methods with less computational overhead.
limitations: limitations: 1. Integrating 2D modality to connect point cloud and text modalities isn't a novel approach. ULIP[1]/CG3D[2]/OpenScene[3] has utilized similar methodologies to synchronize point cloud attributes with the features of the CLIP foundation model. A thorough comparative analysis would be beneficial to understand the advancements or improvements over previous methods.

2. The absence of testing on more intricate datasets, such as ScanNet, is noticeable. OCTAVIUS[4] has endeavored to develop a multimodal Large Language Model (LLM) to interpret complex scenes and perform Captioning/Visual Question Answering (VQA) tasks on the ScanNet dataset. This suggests a need for experimentation on such complex datasets to validate the robustness and versatility of the model.

3. The introduction of LoRA-MoE by Octavius marks an interesting development. Discussing the relationship between MQE and LoRA-MoE requires a multifaceted approach, considering aspects like model efficiency, knowledge retention, and adaptability to various tasks. 

4. The four-stage training process appears intricate, and providing more detailed information about each stage, such as the duration of training and specific processes involved, would be essential for a comprehensive understanding of the model's learning efficiency. 

[1] Xue L, Gao M, Xing C, et al. Ulip: Learning a unified representation of language, images, and point clouds for 3d understanding[C]//Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. 2023: 1179-1189.

[2] Hegde D, Valanarasu J M J, Patel V. Clip goes 3d: Leveraging prompt tuning for language grounded 3d recognition[C]//Proceedings of the IEEE/CVF International Conference on Computer Vision. 2023: 2028-2038.

[3] Peng S, Genova K, Jiang C, et al. Openscene: 3d scene understanding with open vocabularies[C]//Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. 2023: 815-824.

[4] Chen Z, Wang Z, Wang Z, et al. Octavius: Mitigating task interference in mllms via moe[C] ICLR2024
paper_summary: paper_summary: This paper addresses the challenge of integrating 3D point clouds into Large Language Models (LLMs) in a cost-effective manner. While Large 2D vision-language models (2D-LLMs) have demonstrated success in combining text and images, aligning 3D data with LLMs incurs substantial training costs. The authors propose MiniGPT-3D, a novel approach that achieves state-of-the-art (SOTA) results while significantly reducing training time and computational resources.

MiniGPT-3D leverages 2D priors from 2D-LLMs to align 3D point clouds with language models efficiently. The paper introduces a four-stage training strategy for modality alignment and a mixture of query experts module to aggregate features adaptively. Additionally, parameter-efficient fine-tuning methods such as LoRA and Norm fine-tuning are employed, resulting in a model with significantly fewer learnable parameters compared to existing approaches.

Experimental results demonstrate that MiniGPT-3D outperforms previous methods on 3D object classification and captioning tasks, achieving superior performance at a fraction of the training cost.
strength_weakness: strength_weakness: 1. Novelty in Approach: The paper introduces MiniGPT-3D, a novel approach for efficiently aligning 3D point clouds with Large Language Models (LLMs) by leveraging 2D priors from 2D-LLMs. This novel approach significantly reduces the training costs associated with integrating 3D data into language models, addressing a critical challenge in the field.

2. Efficiency and Cost-Effectiveness: MiniGPT-3D achieves state-of-the-art results while requiring substantially fewer computational resources compared to existing methods. By training for only 27 hours on a single RTX 3090 and employing parameter-efficient fine-tuning methods, the model demonstrates high efficiency and cost-effectiveness, making it accessible for researchers with limited computational resources.

3. Comprehensive Evaluation: The paper conducts extensive experiments to evaluate MiniGPT-3D's performance on 3D object classification and captioning tasks. The results demonstrate significant improvements over existing methods, showcasing the effectiveness of the proposed approach across different applications.
limitations: limitations: Limited Scope to 3D Objects: While the proposed MiniGPT-3D model demonstrates impressive performance in aligning 3D point clouds with language models, its applicability is restricted to 3D object-related tasks. The paper does not address the broader challenge of integrating 3D scenes or complex environments into language models.
However, it's important to note that this limitation is not unique to the proposed approach but rather a common challenge faced by current 3D Multi-Modal Language Models (MLLMs).