Exploring the Potential of Encoder-free Architectures in 3D LMMs
Authors: Yiwen Tang
Ziyu Guo
Zhuhao Wang
Renrui Zhang
Qizhi Chen
Junli Liu
Delin Qu
Dong Wang
Bin Zhao
Xuelong Li
Venue: ICLR 2026 Poster (2026)
Keywords: Multi-modal Large Language Model
3D Understanding
Large Language Model
Primary area: foundation or frontier models, including LLMs

Abstract: Encoder-free architectures have been preliminarily explored in the 2D Large Multimodal Models (LMMs), yet it remains an open question whether they can be effectively applied to 3D understanding scenarios. In this paper, we present the first comprehensive investigation into the potential of encoder-free architectures to alleviate the challenges of encoder-based 3D LMMs. These long-standing challenges include the failure to adapt to varying point cloud resolutions during inference and the point features from the encoder not meeting the semantic needs of Large Language Models (LLMs). We identify key aspects for 3D LMMs to remove the pre-trained encoder and enable the LLM to assume the role of the 3D encoder: 1) We propose the LLM-embedded Semantic Encoding strategy in the pre-training stage, exploring the effects of various point cloud self-supervised losses. And we present the Hybrid Semantic Loss to extract high-level semantics. 2) We introduce the Hierarchical Geometry Aggregation strategy in the instruction tuning stage. This incorporates inductive bias into the LLM layers to focus on the local details of the point clouds. To the end, we present the first Encoder-free 3D LMM, **ENEL**. Our 7B model rivals the state-of-the-art model, PointLLM-PiSA-13B, achieving 57.91%, 61.0%, and 55.20% on the classification, captioning, and VQA tasks, respectively. Our results show that the encoder-free architecture is highly promising for replacing encoder-based architectures in the field of 3D understanding. The code is released at https://github.com/Ivan-Tang-3D/ENEL.

Published as a conference paper at ICLR 2026
EXPLORING THE POTENTIAL OF ENCODER-FREE AR- CHITECTURES IN 3D LMMS
Yiwen Tang 1.2 Zoey G Zhuhao Wang**, Ray Zhang?; Qizhi Chen', Junli Liu Delin Qu', Dong Wang', Bin Zhao 1.2 f Xuelong Li
Shannhaai AI Laboratory "Northwestern Polytechnical University The Chinese University of Hong Kong "Tsinghua University TTele AI
ABSTRACT
Encoder-free architectures have been preliminarily explored in the 2D Large Mul- timodal Models (LMMs), yet it remains open question whether they be effectively applicd to 3D understanding scenarios. In this paper, we present the first comprehensive investigation into the potential of encoder-free architectures alle- viate the challenges encoder-basee based 3D LMMs. These long-standing challenges include the failure adapt varying point cloud resolutions during inference and the point features from the encoder not meeting the semantic needs of Large Language Models (LLMs). We identify key aspects for 3D LMMs to remove the pre-trained encoder and enable the LLM to assume the role of the 3D encoder: 1) We propose the .emeeeebedded Semantic Encoding strategy the pre-training stage, exploring the effects of various point cloud self-supervised losses. And we present the Hybrid Semantic Loss to extract high-level semantics. 2) We introduce the Hierarchical Geometry Aggregation strategy in the instruction tuning stage. This incorporates inductive bias into the LLM layers to focus on the local details of the point clouds. To the end, present the first Encoder-fiee 3D LMM. ENEL. Our 7B model rivals the state-of-the-er model, PointLLM-PPin -MMM--PPA--II 13B, achieving 57.91%, % 61.0%. and 55. 55.20% the classification, captioning, and VQA tasks, spectively. Our results show that the encoder-fres architecture is highly promising for replacing encoder-based architectures in the field of 3D understanding. The code is released at https: / http//gttttt //ggtttt.bb....g.gggggggg....e.t com/ Ivan T Tang 3D/ENEL
INTRODUCTION
Large Language Models (LLMs) Touvron et al. y; Bai et al. (003) have gained unprecedented attention for their proficiency understanding and generating complex language scenarios. Building upon these advances, many recent efforts have been made to develop Large Multimodal Models (LMMss, empowering LLMs with the capability to interpret multimodal information, such as 2D) images Li et al. (2024): Ding al. (2025), Tang al. (2025b); Yang al. (2025). 3D point clouds Guo ct al. (2033a) Xu et al. (2025): Wang et al. (2025): Guo et al, (2023b); Tang et al. (2024a b 2025a
Chen et al. (2024b) and visual generation Tong et al. 22025 Diang et al. (025 Guo et al. (2025d
Mainstream LMMs are typically encoder-based, relying on heavyweight yet powerful pre-trained encoders (c.g., CLIP Radford et al. (202I) for 2D and I2P-MAE Zhang et al. (2023a) for 3D). While these pre-trained encoders offer robust multimodal embeddings enriched with pre-existing knowledge. they also introduce challenges that could limit the future advancement of multimodal understanding. To mitigate the limitations introduced by visual encoders in VLMs such as resolution, aspect ratio, and semantic priors priors-many encoder-free free LMM studies [i al. (2025a); |Dioo al. (2024a 2025); Lei et al. (2025); Luo et al. (2025) have explored the possibility of training without pre-trained encoders.
"Equal Contribution.
1 "oorrsponndig Author.

---

Published as a conference paper at ICLR 2026
PointLLA
(Ours)
Low
High
Attention Scores
Objoverse Captioning
PoinfLLM
Encoder-based
51.0
50.0
41,4
14.0
41.7
46.3
ENEL (Ours)
30-
33.7
32.8
Encoder-free
P.C.500
12K
PT.
(a) Point Cloud Resolution Limitation
(b) Embedding Semantic Discrepancy
Figure I: Issues of encoder-based 3D LMMs. (a) Point Cloud Resolution Limitation. During training, the point cloud size (P.C. Size) and point token size (P.T. Size) are fixed at 8192 and 512, respectively. And we adjust these two sizes during inference. point cloud size from 2K to 16K and the corresponding point token size from 128 to 2048. We evaluate them on the captioning task of the Objaverse benchmark using GPT-4 score as the evaluation metric. (b) Embedding Semantic Discrepancy. We visualize the attention scores of the average text token to the point tokens, where red indicates higher values. The point tokens in the encoder-free architecture exhibit stronger textual semantic relevance needed for the LLM.
Specifically for 3D LMMs, the encoder-based architecture has the following potential drawbacks: (1) Point Cloud Resolution Limitation. 3D encoders are often pre-trained on point cloud data at fixed resolution, such as 8,19, 191 points for Point-BERT Yu et al. (2022) in PointLLM Xu et al. (2025). However, during inference, the resolution of point clouds may vary (e.g.. 12,000 or 4,000 points). This difference between training and inference resolutions can result in the loss of spatial information when extracting 3D embeddings, leading to difficulties for LLMs to comprehend, as showcased in Figure (a). (2) Embedding Semantic Discrepancy. 3D encoders are typically pre-trained using self- supervised methods like MAF(Pang al. (2022); (Tng et al. 2024a b) and contrastive learning Xie et al.] (2020): Qi et (2023). but these training objectives may not align with the specific semantic needs of LLMs. In other words, they may not capture the most relevant semantics for LLMs to understand 3D objects, as visualized in Figure (b). (b). Even when projection layer is used to connect 3D encoders with LLMs, simple MLPs are often insufficient for complete semantic transformation. Given these issues, we ask: Is it possible to explore encoder-free architecture for 3D LMMs, eliminating the 3D encoder and instead integrating its functionality directly within the LLM itself?
In this paper, we present the first systematic investigation into the potential of an encoder-free architecture for 3D LMMs. To minimize external influences and ensure clarity, we use the pioneering and sufficiently concise PointLL.M Xu et al. (2025) as our encoder-based baseline, which consists of two progressive training stages: pre-training and instruction tuning. We evaluate the performance on 3D classification Deitke et al. )202)), 3D captioning Deitke et al. (2023) and 3D VQA Deitke et al. (2023) tasks. Specifically, to remove the encoder while mitigating any performance degradation, we explore solutions the following two key questions:
for (1) How can we compensate for the high-level 3D semantics originally extracted by the 3D encoder?
In 3D LMMs, the raw point cloud input is first passed through a token embedding module for low- level tokenization, before being processed by the main 3D encoder, usually a Transformer Vaswani (0177). generate high-level embeddings. Skipping the encoder entirely poses challenge in capturing the complex spatial structures of 3D point clouds. To address this, we propose a strategy called LLM-embedded Semantic Encoding in the pre-training stage. First, we adopt simple yet effective token embedding module that captures as much informative semantic content as possible. These 3D tokens are then directly fed into the LLM. Next, we aim shift the responsibility of capturing high-level 3D semantics to the L.LM itself. To guide this process, we explore various 3D self-supervised loss functions, such as masked modeling loss and distillation loss, and ultimately propose the Hybrid Semantic Loss as the most effective choice. Further, we make the carly layers of the LLM to be learnable, allowing them to specialize in multimodal alignment.
(2) How can we integrate inductive bias into LLMs for better perception of 3D geometric struc- tures? Pre-trained 3D encoders typically embed explicit inductive bias into their architectures to progressively capture multi-level 3D geometries. For instance, models like Point-M2AE 2aa222//naaggggg Zhang et al.) (2022) use a local-to-globe hierarchy, which is a concept also common in convolutional layers for 2D image processing He al. (2016). In contrast, L.MSs employ standard Transformer architectures,
2

---

Published as a conference paper at ICLR 2026
where each layer processes the same number of tokens, representing the same semantic level across the network. In the absence of the encoder, we introduce the approach of Hierarchical Geometry Aggregation during the fine-tuning stage. In the early layers of the LLM, we aggregate 3D tokens based on their geometric distribution using Dynamic Grid Sampling. This approach enables the LLM to gradually integrate detailed 3D semantics and develop more holistic understanding of the 31) object. In the later layers, we reverse this aggregation, propagating the tokens back to their original distribution to maintain the fine- tnneggraieed representation necessary for complex tasks.
Through a series of experimental investigations, we have uncovered the strong potential of applying encoder-free architecture to the 3D LMM domain. Building on our insights, we introduce ENEL, an ENcoder- freE 3D LMM evolved from Vicuna-7he 7 Chiang et al. (2023) using the same training dataset from PointI.LM. Notably, without any 3D encoders, ENEL 7B achieves comparable performance to the oftte-ofthhe-art Point...M-PiSA-1333 Guo et al. (2025a). We hope ENEL may provide the community with an effective path for adapting the encoder- architecture to 3D scenarios.
Our main contributions are summarized as follows:
We present the first comprehensive empirical study of applying encoder-free architectures to the 3D LMM domain, offering valuable insights for the field.
We aim to transfer the original roles of 3D encoders to the LLM itself, and propose the L.LM- embedded Semantic Encoding and Hierarchical Geometry Aggregation strategy, both of which have been validated as effective.
We further introduce ENEL, concise and well-performed encoder-free 3D LMM, which, at the 7B parameter scale, achieves 57.91%, 11%,, 61.0%, and 55.20% on 3D captioning, classification, and 3D VQA tasks, respectively, on par with existing encoder- based models.
RELATED WORK
3D LMM. Recent advancements in integrating large language models (LLMs) with 3D data have led to significant progress in both object-level and scene-level understanding. At the object level, early approaches like Hong et al. (2024) utilize 2D rendering to leverage 2D LLMs, but this sacrifices geometric details. More recent models, including Point-Bind LLM Guo et al. (023a)), PointLLM Xu et al. (2023b) and L Qi et al. (2024), directly encode point clouds and align them with LLMs. by combining the 3D encoder with a powerful language model, effectively fusing geometric, appearance, and linguistic information. MiniGPT-3D Tang et al. (022cc is introduced, which efficiently aligns 3D point clouds with LLMS by leveraging 2D priors from 2D-LLMMs. employes a four-stage cascaded training strategy along with a Mixture of Query Experts (MoQE) module. Zeng al. propose GreenPLM [Tan al.] (2025c). an energy- erfegy-efficient framework that directly translates monolingual pre-trained language models into other languages using bilingual lexicons. At the scene level, models like Chat-3D Wang et al. (203) and Scene-LLM et al. (2024) focus on understanding complex spatial relationships through dialogue and tasks like captioning. Scene-LLM Fu al. (024) enhances embodied agents' abilitics in interactive 3D indoor environments by integrating both scene- level and egocentric 3D information. Grounded 3D-LLMM MD-eLLMMChen et al. (2024d) utilizes referent tokens to reference specific objects within 3D scenes, enabling tasks such as object detection and language grounding. However, conventional encoder-based 3D LMMs commonly suffer from limitations, specifically Point Cloud Resolution Limitation and Embedding Semantic Discrepancy, which stem from the inductive bias inherent in the 3D pre-trained encoder. Our ENEL alleviates these restrictions by removing the encoder and utilizes lightweight architecture to significantly boost performance.
Encoder-free Vision- Lisiua-Language Models. Traditional vision- languaee models (VLMs) often rely on vision encoders to extract visual features before processing them with language models, integrating image encoders like CLIP Radford et al. a (2021) and DINO V2 V2[Quubb et al. (2023). However, recent efforts have explored encoder-free VLMs for their simplicity. Approaches like Chameleon Team (2024); Xie et al. (2024) use VQ tokenizers Esser et al. (2021) or linear projection layers Diao et al. (2024a) Chen al. (202cc) to represent images. Fuyu-SB Bavishi et al. (2023), pure decoder-only model, directly processes image patches through linear projections, handling high-resolution images but showing only average performance. The EVE series Diao et al. (2024b) 2025) eliminates the need for a separate vision encoder by bridging Vision-language representation within a unified decoder and
enhancing visual recognition capabilities through additional supervision. Mono-Intern' VI. series Luo
3

---

Published as a conference paper at ICLR 2026
Table I: Token Embedding. Performance on Objaverse with Loin as the base- line. 'Cls'/Cap': classifcattion/aptiinnnn tasks. 'Avg': accuracy under prompts "What is this?" and "This is an object of." "S-BERT': Centence-BER1. "T...: our designed token embedding module.
Table 2: Learnable Layers. We set the LLM early
layers to be learnable. 'LR' represents the learning rate during the pre-training stage, with the original learning rate set to 2e-3.
Cls (Avg)
Cap
Method
LR
GPT-4
GPT-4
S-BERT
Pointl M-7BB
2e-3
53.00
44.85
47.47
(Avg)
Cap
Method
2c-3
40.00
40.20
44.82
learnable layers
GPT4
GPT-4
S-BERT
4e-4
42.62
46.30
Poi7
53.00
44.85
47.47
learnable layers
2c:3
43.75
40.13
45.76
4c-4
47.90
43.50
46.70
Encoder
35,50
33.37
41.19
2e-3
42.35
37,91
41.28
learnahle layers
2-layer T.E.
40.60
38.85
43.25
4c-4
46,70
42.80
46.14
3-layer T.E.
45.55
41.36
44.82
2e-3
41.55
40.05
41.40
4-layer T.F.
43.00
40.47
43.50
12 leurnable layers
42.39
46.00
et al. (2024; 2025)
leverage visual experts and progressive visual pre-training (EViP/EViP++) to achieve stable optimization and competitive performance. SAIL series [ei et al. (2025) directly encode raw pixels and decodes language within single architecture, achieving competitive vision- language performance without pre-trained vision encoders. The key idea behind ENEL is enabling the LLM to assume the functionality of the encoder by effective and efficient methods. This approach diverges from 2D encoder-fiee free LMMs, which tend to focus on larger datasets and more complex structures for better results.
INVESTIGATION OF ENCODER-FREE 3D LMM
3.1 PRELIMINARY
2025a is an encoder-free Video-LLM that directly models nuanced lideo-language interactions without relying vision encoder. EVE Diao et al. (2024)) and its successor EVEv2 Diao et al. (2025) are designed as efficient encoder-free vision- language models. SAIL Lei al. (2025) serves a unified transformer for vision and language, while Mono- Intro-Intennvve VL Luo et al. (2025) represents monolithic multimodal LLM. In parallel, Fuyu- 8B Bavishi et al. (2023), decoder-only transformer developed by Adept AI, has gained substantial community adoption. A common characteristic across these works is the adoption of lightweight, randomly initialized token embedding layer to convert inputs into tokens for the I.M.M. This design eliminates the need for dedicated vision encoder and enables end-to-end training and inference.
Encoder-free in 2D LMMs. ELVA Li et al.
Pre-trained Encoders 3D LMMs. Traditionally. 3D pre- pre-trained encoders are characterized by two properties: (1) independent pretraining point cloud tasks (e.g.. reconstruction), and (2) structural decoupling, where they are connected to the language model through projection layers. 3D LMMs, commonly adopted encoders refer to pre-trained models such as PointMAE Pang al. (2022). PointBERT Yu et al. (2022), and Uni3D Zhou et al. (2023).
Overall Architecture. We select PointLLM as the baseline model for the exploration and evaluate the performance of different strategies on the Objaverse dataset Deitke et al. 2203), using GPT-4 scores combined with traditional metrics
evaluation metries. Point Embedding Layer. As shown in Figure 2 we first remove the encoder of PointLLM and adopt the original token embedding Yu et al.) (2022) However, the coarse structural design results in a significant performance degradation, as observed in T where the GPT-4 scores for the classification and captioning tasks decrease by 17.5% and 10.48%. respectively. To mitigate excessive information loss and provide refined local features to the LLM, we adopt a small network with a limited number of parameters, which is lightweight variant of Point-PN Zhang al. (023b)). Specifically, for the input {P:).V we apply Farthest Point Sampling (FPS) for downsampling the number of points, k- k-Nearest Neighbors (k-NN) with group size k for local aggregation, and learnable linear layers for feature encoding. After a series of repetitive operations and the projection layer, we transform the point clouds into high-dimensional vectors {F, ) E IR x D., In Table 1 we experiment with token embedding at different depths and find that three layers yield the best performance. 3D Encoding & Alignment. We discover that the absence of the encoder results in a lack of context modeling in point cloud feature processing. Therefore, we attempt have the early layers of the L.LM take the encoder's role in capturing
4

---

Published as a conference paper at ICLR 2026
Stage
Stage 2
"Duttigpiiee by is unseventionl that houss fan."
Cross Enntroprtpppp Loss
Frozen
Learnable
3D model of coumenii
airplane engins
Hibrid Semontic
Cross-Entrapy Loss
Transformer Lagar
(1-1)
Geumefric Propagation
Tronsformer Layer
Transformer Loyer
Transformer Layer
Layer
Transformer Layer
Geometric Aggregetion
Transformer Layer
Tronsformer Lager 1
Point Embedding Loyer
Text Tokenizer
Point Embedding Layer
Text Tobenizer
"Caprion this madd desil."
"Whet distingatiies the nowe of sirphems
Figure 2: Overall Pipeline of ENEL. The training is divided into two stages: the pre-training stage and the instruction tuning stage. In the first stage, we set the first layers be learnable and apply the proposed Hybrid Semantic Loss to embed high-level semantics into the LLM. In the second stage, we adopt the Hierarchical Geometric Aggregation strategy to capture local structures point clouds.
global interactions of features, further encoding the point cloud features. In the pre-training stage, set the first K layers of the frozen LLM to be learnable. Within the shared semantic space, 31) tokens and text tokens interact and align naturally. Early Fusion provides a more practical way to achieve modality alignment between 3D and textual semantic spaces. Meanwhile, we experiment with different learning rates. As shown in Table 2 a smaller learning rate yields better results by stabilizing early layer optimization. Based on the designed token embedding module, setting the first four layers to be learnable yields the best results.
3.2 LLM-EMBEDDED SEMANTIC ENCODING
The lack of the 3D encoder results in insufficient encoding of point cloud semantic information, which greatly hinders the LLM to understand the structural details of point clouds. Most existing 3D encoders use sulfvvuee losses embed the high-level semantics of point clouds into the transformer, primarily categorized into four types: Masked Modeling Loss Pang et al. (202)). Reconstruction Loss Qi al. (023),, Contrastive Loss Loss|Khost Khosla al.] (2020); Qu et al. (025), and Knowledge Distillation Loss Zhang et al. (2023a). Based on the proposed token embedding module and LLM learnable early layers, we implement and evaluate the effects of these losses on the encoder- free 3D LMM in the pre-training stage, as described in Figure 3 Finally, we propose the Hybrid Semantic Loss, which assists the LLM to learn the relationship between local spatial information in the point clouds and grasp the high-level 3D semantics.
Masked Modeling Loss. In the pre-training stage, we apply the Masked Modeling Loss to the point tokens processed by the LLM, shown in Figure 3 (a). Through the token embedding module, the point clouds [P. are divided into point patches (M E R Mxx3 and the corresponding point tokens { We randomly mask the point tokens with masking ratio r, and replace them with learnable tokens. The masked feature tokens can be denoted as (Fai, ) MMr. which serve as the ground truth for the loss computation. After the masked tokens are replaced with learnable tokens and processed by the LLM, a linear layer is applied to predict the point tokens [Frr,, J E Mr and the Mean Squared Error (MSE) is computed between Ppee and Fg- The optimization is:
Mar Σ
Cmask
(1
Fg. 13).
(1)
M
1=1
The specific process of applying Masked Modeling point patches G is detailed in Appendix A.2.
5

---

Published as a conference paper at ICLR 2026
frewiiin form Pasked Tolan
surpet Token
UMt Pwitsce
Par
Morting Lom
Large Language Medel
Lange Language Model
Large Language Medel
Large Longuage Madel
Large Language Medel
musk
CAn orplone.
AAn oirpone
"An
*An arplane.
orpiene
(a)
(b)
(c)
(d)
Figure 3: Point Cloud Self-Supervised Learning Losses. In the pre-training stage, we explore common self-supervised learning losses for the encoder-free 3D LMM: (a) Masked Modeling Loss, (b) Reconstruction Loss, (c) Contrastive Loss, and (d) Knowledge Distillation Loss. The (e) represents our proposed Hybrid Semantic Loss, specifically designed for the encoder-free architecture
Reconstruction Loss. After the point feature tokens {P are encoded by the LLM, the tokens are transformed to the point patches { ) /== E Mxxx through linear layer. We utilize the 12 chamfer distance to align the predicted G pre with the ground truth G, reconstructing the original spatial information, as illustrated in Figure 3 (b). This approach encourages the LLM to learn the high-level semantics of the point cloud while preserving the critical structure and key features of the point cloud input. The optimization target Liecon can be written
Σ min ||a 6,112 + min lbb; ay 113 )
(2)
where a Gpres b G. The procedure for reconstructing feature F detailed in Appendix A.2.1
Contrastive Loss. We conduct contrastive learning Khosla et al. (2020) at the point cloud level, where we contrast two transformed versions of the point cloud in the Figure 3 (c). Given a sampled point cloud [ we apply two random geometric transformations Ti and Tz. including rotation and translation, to obtain PTi and Pr2. The two augmented point clouds are separately paired with the original text query and processed through the LLM to obtain their respective feature tokens Fr1 E RR MD. and FT2 E D.M.D. Within the bati-b the two feature tokens derived from the same point cloud serve positive pairs, while they considered negative pairs with other point clouds. Using NCESoftmaxL.ss we aim to maximize the similarity of positive pairs and minimize the similarity of negative pairs, encouraging the LLM to learn geometric equivariance of point clouds. The Leonrast is shown as below, where B stands for the training batch size.
B
exp(Frr FT2./T)
- log
B
Σド Σ exp( FT2,/T))
Knowledge Distillation Loss. We select the powerful Uni3D-L et al. (2023) as the teacher encoder, input the point cloud into the 3D encoder, and obtain the output feature I Piencher E M XD, The Mean Squared Error (MSE) between the LLM output tokens Fsuueent and Fiecher computed to align Fuuden as closely as possible to Fieachers thereby transferring the knowledge embedded in the 3D encoder to the L.LM. By obtaining additional supervision from the Uni3D, the L.MM better captures the complex structures in the point cloud data, as displayed in Figure 3 (d). The objective function is:
M
1 M
LXD
(I) [Faandent;
Preacher,
Experiments and Insights. As shown in Table we compare the effects of common self-supervised learning losses in the pre-training stage, where they are summed with the LLM cross-entropy loss Touvron et al. (023), each with a coefficient of 1. The observations are summarized as below:
The point cloud self-supervised losses generally benefit the encoder-free 3D LMM. Compared to previous experimental results, where the GPT scores for the classification and captioning tasks are 47.90 47.90% and 43.50% the self-supervised losses bring about the significant improvements. This

---

Published as a conference paper at ICLR 2026
Table 3: L.M-embedded Semantic Encoding, In pre-training, we explore the effects of different self-supervised learning losses targeting point tokens. v and & denote mask ratios of 60% and 30%, respectively, Subscripts patch and feat indicate loss targets. For Hybrid Semantic Loss, the subscripts patch and feat refer the masked modeling target, with reconstruction targeting the corresponding feat and patch.
Cls (Avg)
Cap
Method
GPT-4
GPT4
S-BERT
LM-77
53.00
44.85
47.47
Masked Modeling Loss gusch
47.00
43.64
45.36
Masked Modeling
I Loss patch
49.00
45.20
46.29
Masked Modeling Loss feat
48.50
43.90
45.30
Masked Modeling LosS
48.50
45.85
46,93
Reconstruction LOSSpatch
48.00
45,56
46.33
Roconstruction fost
47.50
44.05
46.18
Contrastive Loss
42.50
41.21
43.77
Knowledge Distillation Loss
48.00
43.87
46.09
Hybrid Semantic LosSpuaech
50.00
45.24
46.59
Hybrid Semantic feat
52.00
47.65
47.30
is because the self-supprvvsed learning loss forces transformations on the complex point clouds through certain task design. This encourages the LLM to not simply memorize specific point cloud data but learn the underlying geometric relationships and high-level semantic information.
Among the self-supervised learning losses, the Masked Modeling Loss demonstrates the strongest performance improvement. achieves GPT-4 scores of 48.5% and 45.85% for classification and captioning tasks, respectively. The application of the masked modeling to the point features facilitates the embedding of high-level semantics from point clouds into the LLM. However, higher mask ratio increases training difficulty, with 60% performing worse than 30%. In addition, explicitly reconstructing point patches helps capture complex structures and critical details in point clouds. Knowledge Distillation Loss falls short compared to the first two losses. Finally, Contrastive Loss, which fails to extract the detailed semantics, achieves the lowest performance.
Hybrid Semantic Loss. Based on the experimental results above, we propose the self- supersupereeeed learning loss specifically designed for the encoder-free 3D LMM Hybrid Semantic Loss, as show- cased in Figure (e). We apply a masking ratio to randomly mask point tokens from the token embedding. The masked tokens and the corresponding patches are referred as (Fmuck; 1.Mrr and
(mass; MMrr respectively. The remaining tokens are denoted as Fris, ) (1 and (Gvis, )MM((1-r) Considering the autoregressive nature of the LLM and the unordered attribute of point clouds, we directly concatenate learnable tokens {Ficam, J.Mr to the end of Fris- replacing the masked tokens. For the masked portion, we adopt masked modeling, and for the visible portion, we use the reconstruction strategy. After passing point tokens through the LLM,, we compute the MSE between Fleam and Fmask The visible features Fvis are transformed into Gpesd. and the L2 Chamfer distance is computed between Gpee and Gvis- These two are added to the original cross-entropy loss with coefficients all equal to 1. This approach not only embeds high-level semantics into the LLM but also ensures geometric consistency throughout the point cloud learning process. With a 30% mask ratio and per-layer positional encoding of point tokens, it achieves 52.00% and 47.65% on the classification and captioning tasks, respectively. The inverse modeling process described in Appendix AA.2.I
Our motivation arises from the observation that complex objectives, such as KD and contrastive learning, impose significant computational overhead yet often yield marginal gains compared to intrinsic data modeling losses like masked modeling. To address this, we propose the Hybrid Semantic Loss, which resolves the structural mismatch between 3D data and LLMs by exploiting two key properties: (I) the permutation invariance of point clouds, allowing learnable tokens to be appended after visible tokens without positional restoration; and (2) the encoder-fire free architecture, where 3D tokens integrated into causally-masker LLM instead of bidirectionalllymaan masked 3D encoder, fundamentally altering information flow between visible and masked tokens, enabling visible tokens to learn harder objectives while learnable tokens focus on lightweight reconstruction.
7

---

Published as a conference paper at ICLR 2026
Table Hierarchical Geometry Aggregation. In the instruction tuning stage, we conduct the experi- ments of Hierarchical Geometry Aggregation strat- egy. represents the number of aggregation and propagation operations. H refers the LLM layers between aggregation and propagation operations. Self-Attn. represents the incorporation of the gated self-attention in the aggregation.
Geometric Propagation
Transformer Layer
Geometric Aggregation
(AeanPooking
Gafed Self-Attention
Method
Cls (Avg)
Cap
Dynamic Grid
GPT4
GPT-4
S-BERT
Sompling
PointLLM-7B
53.00
44.85
47.47
1=1
52.50
48.70
48.07
1-2
51.00
46.67
48.12
Transformer Layer
1=3
53.00
48.93
48.06
1=4
45.00
45.48
46,90
Figure 4: Hierarchical Geometry Aggre- gation Strategy. In the instruction tuning stage, we apply aggregation and propaga- tion operations the point tokens cap- ture the local structural details.
H=2
54.25
49.56
48.52
H=4
52.50
48.61
47.81
H=8
52.25
48.95
47.90
+ Self-Attn.
55,55
51.03
48.79
3.3 HIERARCHICAL GEOMETRY AGGREGATION
3D encoders are designed with specific structures tailored for point clouds, such as local-to-global hierarchy Zhang et al. (2022) for exploring the geometric structure of the point cloud. However, in encoder free architectures, the LLM itself does not have an explicit local modeling module. The self-attention mechanism is intended for modeling global interactions. Therefore, building upon the proposed Hybrid Semantic Loss, we explore in the instruction tuning stage how to enable the LLM actively perceive 3D local details and complement the learned global semantics. To this end, propose the Hierarchical Geometry Aggregation strategy.
Implementation Details. As depicted in Figure from the LLM second layer, the input point tokens (Fupaa 3 based on their corresponding coordinates { (iipuu, J. are grouped by Dynamic Grid Sampling. The grid size follows cumulative scaling strategy across aggregation layers. At the i-th aggregation layer, the grid size is:
8; a Bs,
Bj = tanh(0,) + Berrs
(5)
where a = 0.02 m and 8; E [smin: Smax] = [0.02, 1] m. To ensure the cumulative scaling stays within bounds across I aggregation layers, we set:
Smax
(min
In Smaax +
Smin
Ber
21
21
where / is the total number of aggregation layers. Each 0, is randomly initialized from standard normal distribution. Points within the same grid cell form local neighbors, with the set of all neighbors denoted Gi having cardinality M,. The neighborhood features Fiip E R X are then collected, where & denotes the maximum number of points across all cells. To handle varying point numbers across grid cells, we employ a padding strategy: for cells with fewer than k points, we compute the mean-pooled feature of existing points and concatenate repeatedly until reaching points per cell. For iipuu we employ the gated self-attention mechanism for intra group interactions, grasping the local geometric structure. We multiply the self-attention output by a learnable parameter initialized from zero to adaptively adjust the required knowledge. We formulate it
F Fippot tanh(a) Self-Attn.(Fi)
F. Finput-
On top of this, we apply pooling to fuse the features F 'Input within each neighbor, yielding aggregated tokens {Fi aga, - formulated as
F agg
MeanPooling
FFD input
(8)
8

---

Published as a conference paper at ICLR 2026
Table 5: Comparison of different models on various 3D understanding tasks. A primary focus placed on GPT-4 evaluation, along with data-driven metrics (Sentence-BERT) The * indicates the Qwen2.5-7D LLM base and the ShapeLLM training data. The a denotes reproduced results. denotes the model is implemented based on the ShM baseline.
Cls (Ang)
Sentence-EERY
SimCSE
BL.EU-1
ROUGEL
METROR
GPT4
InstructBLPP DiDDeee
203
45.34
47.41
48.45
4.27
8:28
12.99
43.50
InstrccBBPP 13 ela
2005
44.97
45.90
88. 55
1.65
8.55
13.23
34.25
LLaYA-7 7i
2024
45.61
47.10
3.64
7.20
12.14
1LVA-113 ii
M024
38.28
46.37
45.00
4.02
8.15
12.58
51.75
47.90
Point CCee
2025
11.00
Umi3t ctal.
47:20
30-11
3025
44.45
43.68
16,91
45.35
Pount 1.
20133b
44.85
47.47
48.55
3.87
7.30
11.92
55.00
41.20
11
Xu
Shapet.LM-77-44 MMMM--00
al
46.92
48.20
49.23
54.50
47.40
SlupeI.1M-111 103---
211:4
MiniCPT-1D*
Tmg
7074
52.49
48.73
89 26
54.50
43.60
M-P-5X-77
20155
Poins11MPPSA. DOSSA-SSEmmmn
2025
30,52
48,60
89 64
3.75
7.54
12.56
55.00
46
ENEL-7B'
53.26
48.75
69.94
56.00
48.90
50.17
ENEI-138
54.78
49.37
$0.69
56.00
$4.80
We perform iterations of geometry aggregation, resulting in {P |,M To ensure that the LLM fully extracts the local information, we choose to perform further semantic modeling using H LLM layers after aggregation operations. This allows the model learn the interactions between local information while preventing the loss of fine- fine-grained geometric details. Subsequently, from the Lth layer, we perform I iterations of geometry propagation. Following the grid unpooling strategy, we the point-to-grid mappings propagate the aggregated features Fl FOR from each grid cell back to its corresponding set of points, generating { ( pro, J=1 Mi-1 After iterations, we obtain point tokens of length M,, which are then processed by the remaining LLM layers. After processing through H additional LLM layers, the geometry aggregation and propagation process is repeated.
Experiments and Insights. We conduct step-by-step experiments on the Hierarchical Geometry Aggregation strategy, sequentially evaluating the impacts the number of aggregation and propaga- tion operations (l), the number of LLM layers between aggregation and propagation (H), and the incorporation of the gated self-attention mechanism.
The best performance is achieved when set to As shown in Table performing three aggregation and propagation operations achieves 48. 48.93% and 53.00% performance captioning and classification tasks, respectively. Fewer aggregation layers limit the capture local geometric information, while too many layers oversimplify spatial relationships. Setting / = 3 achieves balanced modeling of local and global structures and realizes sampling ratio of approximately 1/8.
Compared to setting H or 8, highest performance is achieved when is set to 2. reaches 54.25% and 49.56% the classification and captioning tasks, respectively. The excessive number of LLM layers between aggregation and propagation can lead the oversmoothing of the aggregated local information, resulting in the loss of local structural details.
The gated self-attention mechanism effectively improves performance, reaching 55.55% and 51. 1.33% on classification and captioning tasks, respectively. The adaptive control of attention output ensures that global contextual information is utilized only when necessary, preventing it from disrupting local geometric structures. Additionally, allows the model to adjust to different tasks.
RESULTS AND VISUALIZATION
Results. In Table 5 the Objaverse benchmark Deitke et al. (2023). ENEL-7B achieves GPT score of 51.03% for 3D object captioning, setting a new SOTA. In traditional metrics, Sentence-BEi and SimCSE reach 48.79% and 49.52%, respectively, comparable object classification, ENEL-7B outperformes prior encoder- based 3D LMMss with a GPT score of 55.55%. Given the same training dataset PointI.LM, these results validate the effectiveness of
PoimLLLM-PiSA-133 For 3D

---

Published as a conference paper at ICLR 2026
PoIntLLM
(Ours)
(N)
Figure 5: Difference in Semantic Encoding, By visualizing the attention scores of the average text token to the point tokens on the Objaverse dataset, we compare the semantic encoding potential of encoder-ben based and encoder-fres architectures, where red indicates higher values. And (a) represents chairs, (b) represents airplanes, and (c) represents lamps.
WE'L ducis apos
desgror cocin oibore tnplessine qualicily spoert set iis poucing gar wiiie neher eduicing you cents
Dhr sokili inprssenniio
with mok
jassily ya the orcurti ochenl of
has you
Tee clikk
Figure 6: ENEL Response Examples. We demonstrate that ENEL is capable of providing accurate responses across different types of tasks, such as captioning and question answering. by effectively addressing a wide range of objects, including race cars, buildings, insects, and others.
our proposed LLM-embedded Semantic Encoding and Hierarchical Geometry Aggregation strategies for the encoder-free architecture. Additionally, on the 3D- ED-VAA task of the 3D MM- Vet dataset[Qi QQi et al. (2024); Li et al. (2025b). despite the lack of spatial and embodied interaction-relate data in the training set, ENEL achieves the GPT score of 43.8%, surpassing PointLL.M-7B by 2.6%. Replacing 7B Vicuna with 13B Vicuna, ENEL-13B achieves substantial performance gains across tasks. When replacing the Vicuna-7B with Qwen2.5-7B and using Shapel. training data, ENEL-7B achieves over 6% improvements across benchmarks. Details of the evaluation metric and classification performance on the ModelNet dataset are provided in Appendix A.2.2 and A.2.3 respectively.
Visualization. In the Figure 5 we visualize the attention scores between the average text token and the point tokens in the last layer of both PointLLM and ENEL. Three object categories, including the chair, the airplane, and the desk lamp, are selected from the Objaverse dataset Deitke et al. (2023). In the Figure 5 red indicates higher values. We observe that in encoder-base -aaed 3D LMMs, the semantic relevance between the text tokens and the processed point tokens is relatively low. In contrast, ENEL,, with its encoder-fiee free architecture, achieves high correlation between the features of the two different modalities, with the average text token focusing on key geometric structures of the objects, such as the backrest of the chair, the wings of the airplane, and the lampshade of the desk lamp.
Response Visualization. In the Figure we present a visualization of ENEL'S 's responses for both captioning and question answering (QA) formats. We observe that in the captioning task, ENEL can even accurately identify fine-graid fine-grained categories such as a cricket. Moreover, in the QA task, ENEL effectively handles visual challenges such as general object recognition (e.g., reasoning about numbers on dice) and spatial reasoning (e.g., accurately interpreting building structures).
5 CONCLUSION
In this study, we investigate the potential of the encoder-fre free architecture in 3D understanding. Through a systematic analysis, we demonstrate that transferring the functionality of the 3D encoder to the LLM itself can effectively compensate for the performance degradation caused by the removal of the 3D encoder. To achieve this, we introduce the LLM-embedded Semantic Encoding strategy and the Hierarchical Geometry Aggregation strategy in the pre-training and instruction tuning stages, These strategies enable the encoding of high-level point cloud semantics while capturing critical local information. Our experiments highlight the promising prospects of the encoder-fred architecture.
10

---

Published as a conference paper at ICLR 2026
6 ACKNOWLEDGEMENT
This work is supported by Shanghai AI Laboratory.
REFERENCES
Josh Achiam, Steven Adler, Sandhini Agarwal, Lama Ahmad, Ilge Akkaya, Florencia Leoni Aleman, Diogo Almeida, Janko Altenschmidt, Sam Altman, Shyamal Anadkat, et al. Gpt-4 technical report. arXiv preprint arXiv:2303.08774, 2023.
Jinze Bai, Shuai Bai, Yunfei Chu, Zeyu Cui, Kai Dang, Xiaodong Deng, Yang Fan, Wenbin Ge, Yu Han, Fei Huang, et al. Qwen technical report. arXiv preprint arXiv:2309.16600 2023.
Rohan Bavishi, Erich Elsen, Curtis Hawthorne, Maxwell Nye. Augustus Odena, Arushi Somani, and Sagnak Taşırlar. Introducing our multimodal models, 2023. URL htttp:///www http://wwwwwdppp.epe/p/pppt adept. ai/
blog/ (ath/fuyu-2200 8b
TeameleonTeam. Chameleon: Mixed-modal early-fusion foundation models. arXiv:2405.09818. 2024.
arXiv preprint
Guangyan Chen, Meiling Wang, Yi Yang, Kai Yu, Li Yuan, and Yufeng Yue. Pointgpt: Auto- regressively generative pre-training from point clouds. Advances in Neural Information Processing Systems, 36:29667-29679 2023.
Guiming Hardy Chen, Shunian Chen, Ruifei Zhang, Junying Chen, Xiangbo Wu, Zhiyi Zhang, Zhihong Chen, Jianquan Li, Xiang Wan, and Benyou Wang. Allava: Harnessing gpt4v-synthesized data for lite vision-language models. arXiv preprint arXiv:2402. 11684, 2024a.
Qizhi Chen, Delin Qu, Junli Liu, Yiwen Tang, Haoming Song. Dong Wang, Bin Zhao, and Xuelong Li. Freegaussian: Annotation-free control of articulated objects via 3d gaussian splats with flow derivatives. arXiv preprint arXiv:2410. 220000 2024b.
Yangyi Chen, Xingyao Wang, Hao Peng. and Heng Ji. A single transformer for scalable vision- language modeling. arXiv preprint arXiv:2407 .06488, 2024c.
Yilun Chen, Shuai Yang, Haifeng Huang, Tai Wang, Ruiyuan Lyu, Runsen Xu, Dahua Lin, and Jiangmiao Pang. Grounded 3d-llm with referent tokens. arXiv preprint arXiv:2405, 10370, 2024d.
Wei-Linn Chiang, Zhuohan Li, Zi Lin, Ying Sheng, Zhanghao Wu, Hao Zhang, Lianmin Zheng, Siyuan Zhuang, Yonghao Zhuang, Joseph E Gonzalez, et al. Vicuna: An open-source chatbot impressing gpt-4 with 90%* * chatgpt quality. See https://vicuna. Imsys. org (accessed 14 April 2023), 2(3):6, 2023.
Wenliang Dai, Junnan Li, Li, AMH Tiong, Zhao, W Wang, B Li, P Fung, and S Hoi. Instructblip: Towards general- purpose vision- languae models with instruction tuning. arxiv 2023. arXiv preprint arXiv:2305.000 065000 2, 2023.
Matt Deitke, Dustin Schwenk, Jordi Salvador, Luca Weihs, Oscar Michel, Eli VanderBilt, Ludwig Schmidt, Kiana Ehsani, Aniruddha Kembhavi, and Ali Farhadi. Objaverse: A universe of anno- tated 3d objects. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recogmition, pp. 13142-13153, 2023.
Haiwen Diao, Yufeng Cui, Xiaotong Li, Yueze Wang, Huchuan Lu, and Xinlong Wang. Unveiling encoder-free lanoua-la models. arXiv preprint arXiv:2406.1 1832 2024a.
Haiwen Diao, Yufeng Cui, Xiaotong Li, Yueze Wang, Huchuan Lu. and Xinlong Wang. Unveiling encoder-free vision- langugee models. arXiv preprint arXiv:2406. 1832, 2024b.
Haiwen Diao, Xiaotong Li, Yufeng Cui, Yueze Wang. Haoge Deng, Ting Pan, Wenxuan Wang, Huchuan Lu, and Xinlong Wang. Evev2: Improved baselines for encoder-free rision-language models. arXiv preprint a72.722502.06788. 2025.
11

---

Published as a conference paper at ICLR 2026
Patrick Esser, Robin Rombach, and Bjorn Ommer. Taming transformers for high-resolution image syn- thesis. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pp. 12873-12883, 2021.
Rao Fu, Jingyu Liu, Xilun Chen, Yixin Nie, and Wenhan Xiong. Scene-Ilm: Extending language model for 3d visual understanding and reasoning. arXiv preprint arXiv:2403. 1401, 2024.
Zilu Guo, Hongbin Lin, Zhihao Yuan, Chaoda Zheng, Pengshuo Qiu, Dongzhi Jiang, Renrui Zhang. Chun-Mei Feng, and Zhen Li. Pisa: A self- auffeeegeet data engine and training strategy for 3d understanding with large models. arXiv preprint 1002:55033..0599 2025a.
Ziyu Guo, Renrui Zhang, Xiangyang Zhu, Yiwen Tang, Xianzheng Ma, Jiaming Han, Kexin Chen, Peng Gao, Xianzhi Li, Hongsheng Li, et al. Point-bind & point-llm: Aligning point cloud with multi-modalliyy for 3d understanding, generation, and instruction following. arXiv preprint arXiv:2309.0611: 2023a.
Ziyu Guo, Xinyan Chen, Renrui Zhang, Ruichuan An, Yu Qi, Dongzhi Jiang, Xiangtai Li, Manyuan Zhang, Hongsheng Li, and Pheng-Ann Heng. Are video models ready as zero-shot reasoners? an empirical study with the mme-cof benchmark. arXiv preprint arXiv:2510.2680, 2025b.
Ziyu Guo, Renrui Zhang. Hongyu Li, Manyuan Zhang, Xinyan Chen, Sifan Wang, Yan Feng, Peng Pei, and Pheng-Ann Heng. Thinking-wiin wwhile-generating Interleaving textual reasoning throughout visual generation. arviv preprint 1..X::25111.66677 2025c.
Ziyu Guo, Renrui Zhang. Chengzhuo Tong, Zhizheng Zhao, Rui Huang, Haoquan Zhang, Manyuan Zhang, Jiaming Liu, Shanghang Zhang, Peng Gao, al. Can we generate images with cot? let's verify and reinforce image generation step by step. arXiv preprint arXiv:2501.1392: 2025d.
Zoey Guo, Yiwen Tang, Ray Zhang, Dong Wang, Zhigang Wang, Bin Zhao, and Xuelong Li. Viewrefer: Grasp the multi-view knowledge for 3d visual grounding. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pp. 15372-15383, 2023b.
Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recognition. In Proceedings of the IEEE conference on computer vision and pattern recognition, pp. 770-778, 2016.
Dan Hendrycks, Collin Burns, Steven Basart, Andy Zou, Mantas Mazeika, Dawn Song, and Jacob Steinhardt. Measuring massive multitask language understanding. arXiv preprint arXiv:2009.03300 2020.
Yining Hong, Haoyu Zhen, Peihao Chen, Shuhong Zheng, Yilun Du, Zhenfang Chen, and Chuang Gan. 3d-llm: Injecting the 3d world into large language models. Advances in Neural Information Processing Systems, 36:20482-20494, 2023.
Yining Hong, Haoyu Zhen, Peihao Chen, Shuhong Zheng, Yilun Du, Zhenfang Chen, and Chuang Gan. 3d-llm: Injecting the 3d world into large language models. Advances in Neural Information Processing Systems, 36, 2024.
Dongzhi Jiang, Ziyu Guo, Renrui Zhang, Zhuofan Zong. Hao Li. Le Zhuo, Shilin Yan. Pheng-Ann Heng, and Hongsheng Li. T2i-r1: Reinforcing image generation with collaborative semantic-level and token-level cot. arXiv preprint arXiv:2505.00m 00703, 2025.
Linglin Jing, Yuting Gao, Zhigang Wang. Wang Lan, Yiwen Tang, Wenhai Wang. Kaipeng Zhang, and Qingpei Guo. Evomoe: Expert evolution in mixture of experts for multimodal large language models. arXiv preprint ar..:2200. 23830, 2025.
Prannay Khosla, Piotr Teterwak, Chen Wang. Aaron Sarna, Yonglong Tian, Phillip Isola, Aaron Maschinot, Ce Liu, and Dilip Krishnan. Supervised contrastive learning. Advances in neural information processing systems, 33: 18:18666-1877. 2020.
Weixian Lei, Jiacong Wang, Haochen Wang. Xiangtai Li, Jun Hao Liew, Jiashi Feng, and Zilong Huang. The scalability of simplicity: Empirical analysis of lision-language learning with a single transformer. arXiv preprint arXiv:2504.1 10462, 2025.
12

---

Published as a conference paper at ICLR 2026
Feng Li, Renrui Zhang, Hao Zhang, Yuanhan Zhang, Bo Li, Wei Li, Zejun Ma, and Chunyuan Li. nuaaa-next-mteerlaave Tackling multi multi-image, video, and 3d in large multimodal models. arXiv preprint arXiv:2407.07885. 2024.
Handong Li, Yiyuan Zhang, Longteng Guo, Xiangyu Yue, and Jing Liu. Breaking the encoder barrier for seamless video-language understanding. arXiv preprint arXiv:2503. 18422, 2025a.
Xiaofan Li, Yanpeng Sun, Chenming Wu, Fan Duan, YuAn Wang, Weihao Bo, Yumeng Zhang, and Dingkang Liang. Video4edit: Viewing image editing as a degenerate temporal process. arXiv preprint arXiv:2511.18131 2025b..
Haotian Liu, Chunyuan Li, Qingyang Wu, and Yong Jae Lee. Visual instruction tuning. Advances in neural information processing systems, 36, 2024.
Gen Luo, Xue Yang, Wenhan Dou, Zhaokai Wang, Jifeng Dai, Yu Qiao, and Xizhou Zhu. Mono- internvl: Pushing the boundaries monolithic multimodal large language models with endogenous visual pre-training. arXiv preprint arXiv:2410.08200 2024.
Gen Luo, Xue Yang, Wenhan Dou, Zhaokai Wang, Jiawen Liu, Jifeng Dai. Yu Qiao, and Xizhou Zhu. Mono-internvl: Pushing the boundaries of monolithic multimodal large language models with endogenous visual trainingnn. Proceedings of the Computer Vision and Pattern Recognition Conference, pp. 11960-24977 2025.
Maxime Oquab, Timothée Darcet, Théo Moutakanni, Huy Vo, Mare Szafraniec, Vasil Khalidov, Pierre Fernandez, Daniel Haziza, Francisco Massa, Alaaeldin El-Nouby, et al. Dinov2: Learning robust visual features without supervision. arXiv preprint arXiv:2304.07192, 2023.
Yatian Pang, Wenxiao Wang, Francis EH Tay, Wei Liu, Yonghong Tian, and Li Yuan. Masked autoencoders for point cloud self-supervised learning. In European conference computer vision, pp. 604-621. Springer, 2022.
Zekun Qi, Runpei Dong, Guofan Fan, Zheng Ge, Xiangyu Zhang, Kaisheng Ma, and Li Yi. Contrast with reconstruct: Contrastive 3d representation learning guided by generative pretraining. In International Conference on Machine Learning, pp. 28223-28243, PMLR, 2023.
Zekun Qi, Runpei Dong, Shaochen Zhang, Haoran Geng, Chunrui Han, Zheng Ge, Li Yi, and Kaisheng Ma. Shapellm: Universal 3d object understanding for embodied interaction. arXiv preprint 11Xvv2440..17766. 2024.
Delin Qu, Haoming Song, Qizhi Chen, Yuanqi Yao, Xinyi Ye, Yan Ding, Zhigang Wang, Jia Yuan Gu, Bin Zhao, Dong Wang. et al. Spatialvla: Exploring spatial representations for visual- languae- action model. arXiv preprint arXiv:2501.15830 2025.
Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark. Gretchen Krueger, and Ilya Sutskever. Learning transferable visual models from natural language supervision. In ICML, volume 139, pp. 9748-8765 2021.
Yiwen Tang, Ray Zhang, Zoey Guo, Xianzheng Ma, Bin Zhao, Zhigang Wang, Dong Wang, and Xuelong Li. Point-peft: Parameter-efff efffccieet fine-tuning for 3d pre-trained models. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 38, pp. 5171 2024a.
Yiwen Tang, Ray Zhang, Jiaming Liu, Zoey Guo, Bin Zhao, Zhigang Wang, Peng Gao, Hongsheng Li. Dong Wang, and Xuelong Li. Any2point: Empowering any- any-modality large models for efficient 3d understanding. In European Conference on Computer Vision, pp. 456-473. Springer, 2024b.
Yiwen Tang, Zoey Guo, Kaixin Zhu, Ray Zhang, Qizhi Chen, Dongzhi Jiang, Junli Liu, Bohan Zeng, Haoming Song, Delin Qu, al. Are we ready for rl in text-to-3d generation? progressive investigation. arXiv preprint arXiv:2512, 109499 2025a.
Yiwen Tang, Qiuyu Zhao. Zenghui Sun, Jinsong Lan, Xiaoyong Zhu, Bo Zheng, and Kaifu Zhang. Revision: Reflective intent mining and online reasoning auxiliary for e-commerce visual search system optimization. arXiv preprint arXiv:2510.22739, 2025b.
13

---

Published as a conference paper at ICLR 2026
Yuan Tang, Xu Han, Xianzhi Li, Qiao Yu, Yixue Hao, Long Hu, and Min Chen. Minigpt-3d: Efficiently aligning 3d point clouds with large language models using 2d priors. In Proceedings of the 32nd ACM International Conference on Multimedia, pp. 6617-6626, 2024c.
Yuan Tang, Xu Han, Xianzhi Li, Qiao Yu, Jinfeng Xu, Yixue Hao, Long Hu, and Min Chen. More text, less point: Towards 3d data-efficient point-language understanding. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 39, pp. 7284-7292 2025c.
Chengzhuo Tong, Ziyu Guo, Renrui Zhang, Wenyu Shan, Xinyu Wei, Zhenghao Xing, Hongsheng Li, and Pheng-Ann Heng. Delving into rl for image generation with cot: A study on dpo vs. grpo. arXiv preprint Xi5: 7017, 2025.
Hugo Touvron, Thibaut Lavril, Gautier Izacard. Xavier Martinet, Marie-Anne Lachaux, Timothée Lacroix, Baptiste Rozière, Naman Goyal, Eric Hambro, Faisal Azhar, et al. Llama: Open and efficient foundation language models. arXiv preprint (.XXix:2302.197777 2023.
A Vaswani. Attention is all you need. Advances in Neural Information Processing Systems, 2017.
Jiaze Wang, Yi Wang, Ziyu Guo, Renrui Zhang, Donghao Zhou, Guangyong Chen, Anfeng Liu, and Pheng-Ann Heng. Mm-mixing: Multi-modal mixing alignment for 3d understanding. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 39, pp. 7744-7752, 2025.
Zehan Wang, Haifeng Huang. Yang Zhao, Ziang Zhang, and Zhou Zhao. Chat-3d: Data-efficiently tuning large language model for universal dialogue of 3d scenes. arXiv preprint wrXiv:2308.08766 2023.
Jinheng Xie, Weijia Mao, Zechen Bai, David Junhao Zhang, Weihao Wang, Kevin Qinghong Lin, Yuchao Gu, Zhijie Chen, Zhenheng Yang, and Mike Zheng Shou. Show-o: One single transformer to unify multimodal understanding and generation. arXiv preprint arXiv:2408. 12528, 2024.
Saining Xie, Jiatao Gu, Demi Guo, Charles R Qi, Leonidas Guibas, and Or Litany. Pointcontrast: Unsupervised pre-training for 3d point cloud understanding. In Computer Vision-ECCV 2020: 16th European Conference, Glasgow, UK. August 23-28, 2020, Proceedings, Part III 16, pp. 574- 591. Springer, 2020.
Runsen Xu, Xiaolong Wang, Tai Wang, Yilun Chen, Jiangmiao Pang, and Dahua Lin. Pointllm: Empowering large language models to understand point clouds. CoRR, abs/2308. 16911, 2023a.
Runsen Xu, Xiaolong Wang, Tai Wang. Yilun Chen, Jiangmiao Pang, and Dahua Lin. Pointllm: Empowering large language models to understand point clouds. arXiv preprint arXiv:2308. 16v11108.6691 2023b.
Runsen Xu, Xiaolong Wang, Tai Wang. Yilun Chen, Jiangmiao Pang, and Dahua Lin. Pointllm: Empowering large language models to understand point clouds. In European Conference on Computer Vision, pp. 131-147. Springer, 2025.
Yaotian Yang, Yiwen Tang, Yizhe Chen, Xiao Chen, Jiangjie Qiu, Hao Xiong, Haoyu Yin, Zhiyao Luo, Yifei Zhang, Sijia Tao, et Automat: Enabling automated crystal structure reconstruction from microscopy via agentic tool use. arXiv preprint Aix:v:250 12650, 2025.
Xumin Yu, Lulu Tang, Yongming Rao, Tiejun Huang, Jie Zhou, and Jiwen Lu. Point-bert: Pre- training 3d point cloud transformers with masked point modeling. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pp. 19313-19322. 2022.
Renrui Zhang, Ziyu Guo, Peng Gao, Rongyao Fang, Bin Zhao, Dong Wang. Yu Qiao, and Hong- sheng Li. Point-m2ae: multi-scale masked autoencoders for hierarchical point cloud pre-training. Advances in neural information processing systems, 35:27061 2022.
Renrui Zhang. Liuhui Wang, Yu Qiao, Peng Gao, and Hongsheng Li. Learning 3d representations from 2d pre-trained models via image-to-point masked autoencoders. In Proceedings of the IEEE/CVF Conference Computer Vision and Pattern Recognition, pp. 21769-17222 2178-2 2023a.
14

---

Published as a conference paper at ICLR 2026
Renrui Zhang, Liuhui Wang. Yali Wang, Peng Gao, Hongsheng Li, and Jianbo Shi. Starting from non non-paamettiic networks for 3d point cloud analysis. In Proceedings of the IEEE/CVF Conference Computer Vision and Pattem Recognition, pp. 5344-5353, 2023b.
Junsheng Zhou, Jinsheng Wang, Baorui Ma, Yu-Shen Liu, Tiejun Huang, and Xinlong Wang. Uni3d: Exploring unified 3d representation at scale. arXiv preprint arXiv:2310.0677. 2023.
15

---

Published as a conference paper at ICLR 2026
A APPENDIX
Mosked Modating Loss
Recotstruction Loss
Agdnnil
Large Language Model
Large Language Model
Large Language Model
"An airplane."
'An airplane."
"An airplane." "
(a)
(b)
(c)
Figure 7: Variants Point Cloud Supe-Supe Learning Losses. (a) The Variant of Masked
Modeling Loss, (b) The Variant of Reconstruction Loss, (c) The Variant of Hybrid Semantic Loss.
A.1 EXPERIMENTAL SETTINGS
Implementation Details. We use the LLaMA model Touvron et al. (2023) as our LLM backbone, with the 7B Vicuna-v1. 1.1 Chiang al. (2023) checkpoint as the default setting. In the token embedding layer, the point cloud is first processed by a linear layer to expand its dimension from 6 to 288. The input point cloud initially consists of 8192 points, followed by three iterations of farthest point sampling (FPS), reducing the size to 512, 256, and 128, respectively. After each FPS operation, k-Nearest Neighbors (k-NN) is applied with cluster size of 81. And geometric features are extracted using triangular encoding, followed by linear layers that progressively increase the dimension to 576, 1152. and 2304. Finally, the projection layer maps the features to the LLM dimension of 4096. In the pre-training stage, we unfreeze the first four LLM layers. Within the LLM- MLMeembedeed Semantic Encoding strategy, Hybrid Semantic Loss applies masked modeling to 30% of the tokens and reconstructs the patches for the remaining 70% visible tokens. During instruction tuning, geometric aggregation is applied at the end of the 1st, 2nd, and 3rd LLM layers reduce point tokens. MaxMean pooling is used to retain more information. After two LLM layers, geometric propagation is applied at the end of the 6th, 7th, and 8th layers to restore the number of point cloud to 128. After two LLM layers, geometric aggregation is applied the 1 11th-13th layers, followed by geometric propagation at the 16th-1 18th layers.
Training and Evaluation Details. During the two-stage training, each stage utilizes the same dataset and preprocessing method as PointLLM. All training are conducted on 4 x 80G A100 GPUs in BF16 precision, utilizing FlashAttention, the AdamW optimizer, and cosine learning rate schedule. During the pre-training stage, the model is trained for three epochs with a batch size of 128 and learning rate of 4e-4. In the instruction tuning stage, it is conducted for three epochs with batch size of 32 and a learning rate of 2e-5. The GPT-4 model Achiam et al. (2023) used for classification and captioning tasks evaluation refers to "gpt-4-0613" version consistent with PointLLM Xu et al.] (2023b). In contrast, the GPT-4 model employed for QA performance evaluation corresponds to "gpt-4-0125" version aligning with Shapel.LM Qi et al. (2024). Additionally, the GPT evaluation prompts for classification and captioning are identical to those used in PointLLM, while the prompts for QA follow those ShapeLLMM.
A.2
MORE EXPERIMENTS
A.2.1 VARIANTS OF POINT CLOUD SELF-SUPERVISED LEARNING LOSSES
In the Figure 7 exhibit the other variants of Masked Modeling Loss, Reconstruction Loss and Hybrid Semantic Loss.
As seen in Figure (a), in the Masked Modeling Loss, after the learnable tokens processed by the LLM, the tokens are transformed to the point patches [.pre)),f Mer e R M through a linear layer. We utilize the l chamfer distance align the predicted Gree with the point patches Grask
16

---

Published as a conference paper at ICLR 2026
Table 6: Ablation Experiments. We begin the ablation experiments by changing the single con- figuration of the module from ENEL. y and & denote mask ratios of 60% and 30%, respectively. For Hybrid Semantic Loss, the subscripts patch and feat refer the masked modeling target, with reconstruction targeting the corresponding feat and patch. I represents the number of aggregation and propagation operations. H refers the LLM layers between aggregation and propagation operations. O refers to the LLM layer between two individual aggregation or propagation operations.
Cap
Cls (Avg)
GPT-4
Sentence-BERT
SimCSE
BLEU-1
ROUGE-L
METEOR
GPT-4
ENEL-7B
51.03
48.79
49.52
3.91
7.20
12.68
55.55
Hybrid Semantic Loss
47.15
48.06
48.31
3.40
7.43
11.84
50.50
Hybrid Semantic Loss
49.13
3.99
7,20
52.30
Hybrid Semantic posck
48.79
48.30
49.00
3,65
6.90
11.88
52,10
Hybrid Semantic Loss
49.62
48.00
48.67
3.78
6.82
12.33
51.50
mechanism
49.61
48.41
48.97
3.79
7.12
12.48
53,60
1=2,012,,0=0
3.72
1=2.H=4.0=0
49.05
48.47
48.62
3.65
12.31
52.20
1=2,H22,022
48.96
47.95
48.88
3.80
7.05
12.55
52.00
1=2,H=4,0=2
49.68
48.70
48.85
3.84
7,56
12.76
53.10
Table 7:
Comparison of computational complexity between PoinLLM-77 and ENEL-7B. S1 and S2 refer to the pre-training and instruction tuning stages, respectively. Conv. Steps indicates the number of steps required for loss convergence.
Method
Time (H)
Memory (S1/S2)
FLOPs
Conv. Steps (S1/S2)
PointLLLM-BB
31.6
67G / 57G
2.0 x 10
10100 / 4300
ENEL-7B
22.2
56G 42G
1.59 1018
9790 3700
Improvement
29.7%
16.4% 16.3% 26.3%
20.5%
2.9% 14.0%
corresponding to the masked tokens, reconstructing the spatial information. The optimization is:
1 Mor
Σ
min ||a
b,l12 + min l|b, - -,,13
M *
where pre and Gmask-
As shown in Figure 7 (b). after the point feature tokens M are encoded by the LLM, the Mean Squared Error (MSE) is computed between the predicted Pr and the ground truth F. The optimization can be written as
M
1
L
(1 - F, F.112)
(10)
M
Finally, in the Figure 7 (c) Hybrid Semantic Loss, the masked tokens and the corresponding patches Mar and (Gmask, Me respectively. The remaining tokens are denoted
referred to (F Jiz t=1 M=(1-r) After passing point tokens through the LLM, we compute as{ Fii., 1 M+(1-)) and {Gvis, 1==1 the MSE between Fir and Fris- The learnable tokens Fleam transformed into Gpred, and the Le Chamfer distance is computed between Gpeed and Gmask These two are added to the original cross-entropy loss with coefficients all equal 1.
A.2.2 METRIC ANALYSIS
GPT-4 Evaluation is LLM-as-a-judge framework based custom prompts. Given model- generated description and human reference, GPT-4 identifies key attributes from the reference, measures how many accurately partially matched in the model output, and returns score from 0 to 100 with a brief explanation. It offers more comprehensive and human- evaluation.
Traditional metrics like BLEU measure n-gram precision. ROUGE-L uses longest common sub- sequence, and METEOR combines unigram precision and recall with lemmatization and synonym matching. However, these metrics struggle with semantic similarity and tend to favor shorter outputs.
17

---

Published as a conference paper at ICLR 2026
Table ModelNet40 classification results under instruction-typer and completion-typed prompts. The instruction-typed (I) prompt is "What is this?" and the completion-typed (C) prompt is "This is an object of."
Model
ModelNet (I)
ModelNet (C)
ModelNet-Avg
PointI.LMM7B
53.44
51.82
52.63
PointI.LMM13B
53.00
52.55
52.78
ShapeLLMM7B
53.08
ShapeLLM-13B
52.96
PointLLM-PiSA-77
54.58
52.60
53.59
PoinLLLM-PiSA-133B
55.03
53.81
54.42
ENEL-7B
54.82
53.69
54.26
ENEL-13B
55.59
54.38
55.00
ENEL-7B
61.25
60.47
60.86
Reasons for low traditional metries: 3D-LLM with high traditional metric scores generates captions averaging 20 words-much shorter than ENEL and other methods. However, this does not indicate better output quality and performs worse in human evaluations. Traditional metries often fail to assess the quality of detailed LLM outputs, as they favor shorter responses and struggle to capture semantic similarity. The GPT-4 score offers stronger semantic understanding, greater diversity, and better generalization.
Examples: Here is a typical example where GPT-4 gives high scores but traditional metrics give low scores. Given point cloud of an airplane, the model outputs:
"The 3D model portrays white cartoon airplane, styled in a simplistic and charm- ing fashion... This model can be inferred to be used in animated children's media as a playful element game or learning application design."
The ground truth:
"This 3D object is airplane with distinct wings and tail. has a long fuselage with glass windows at the front and sides. The round-shaped wings are located in the middle."
The model correctly identifies the object an airplane and captures key style features like simplicity. cartoon form, and whiteness. also reasonably infers use children's media, showing strong understanding. However, traditional metrics rely on n-gram overlaps. Phrases like "airplane body and wings" differ from the ground truth "fuselage with glass windows," leading to mismatches. The output is also longer and more descriptive, while the ground truth is concise and factual, and includes extra details like "white cartoon airplane," all contributing to low traditional scores.
A.2.3 NODEL.ET CLASSIFICATION TASK
As shown in Table 8 ENEL-7B achieves average accuracy of 54.26%, surpassing PointL.LM- 7B (52.63%), ShapeLLM-7B (53,00 (53.08% and PoinLLLM-PiSA-7B 53.59%). Similarly, ENEL-13B reaches 55,00%, outperforming both ShapeLLM-13B (52.96%) and Po (54.42%). These results demonstrate the effectiveness of the encoder- free design in 3D object understanding.
A.2.4 COMPLEXITY ANALYSIS
In Table compared to PointLLM-7B. ENEL-7B demonstrates significant improvements while using the same training dataset. It achieves 29.7% faster training time, reduces GPU memory usage by 16.4% and 26.3% Stage and Stage 2, respectively, lowers training FLOPs by 20.5%, and accelerates convergence speed by 2.9% (Stage 1) and 14.0% (Stage
18

---

Published as a conference paper at ICLR 2026
A.2.5 ENCODER-FREE ARCHITECTURE CLAIM.
Following the consensus in recent Large Multimodal Model (LMM) literature, we strictly define an architecture as "encoder-free based on two criteria: (1) the absence of a heavy, independently pretrained visual backbone, and (2) the utilization of end-to-end training from scratch. Unlike traditional 3D LMMs that rely on decoupled, pretrained encoders (e.g., Point-BERT Yu et al. (2022) for semantic extraction, our design integrates a lightweight, randomly initialized embedding layer trained jointly with the LLM.
Alignment with Community Standards. This design philosophy parallels established encoder- free paradigms in the 2D image and video domains. For instance, EVE Diao et al. (2024a) utilizes token embedding layer based on convolution and cross-attention (~16M parameters), while ELVA [i et al.) (2025a) employs a spatio-temporal attention layer (-9M parameters) for video framing. Similarly, Mono-Intern VL. [ et al. (2024) relies on a lightweight stack of convolutions (~10M parameters). As detailed in Table 2 our proposed point embedding layer comprises only 3M parameters. This is not only significantly more lightweight than its 2D counterparts but also orders of magnitude smaller than typical 3D encoders (e.g., 888 for PointBERT used in PointLLM), Our module functions strictly as tokenizer rather than a visual encoder.
Table 9: Comparison of Tokenizer Parameters across Domains.
Method
Domain
Tokenizer Structure
Tokenizer Params
Ratio (Tok./Total)
EVE/EVEV? Dian et al.
22024a
2025
Image
Conv + Cross-Attn
16 M
~0.23%
Mono-InternV Luo ct 2024
Image
Stucked Conv
10 M
~0.14%
ELVA Li et al. 2025a
Vidco
Spatio-temporal Attn
M
~0.13%
PointLLM Xu et al. 2023a
3D
PointBERT Encoder
8888 M
~ -1.244%
Ours
3D
Point Embedding
3 3
~0.04 %
Structural Formatting vs. Semantic Encoding, We explicitly distinguish the structural operations used in our embedding syyer-specificallly Farthest Point Sampling (FPS) and k-Nearest Neighbors (k-NN)-free semantic encoding. Due to the data irregularity of unstructured 3D point clouds, FPS and k--N serve as the mathematically necessary equivalents of the "patchify" or "stride" operations used in 2D Vision Transformers. They are required to group data points into processable tokens. Crucially, these operations are parameter-free. The subsequent learnable MLPs serve only to project these local geometric groupings into the feature dimension required by the LLM.
A.2.6 MORE ABLATION EXPERIMENTS
We begin the ablation experiments starting from the ENEL-7B, which is the reverse order compared to the experiments in the main text, as showcased in Table
The Effects of LLM-embedded Semantic Encoding Strategy. In the Table 6 on the basis of ENEL, removing the Hybrid Semantic Loss during the pre-training stage significantly degrades performance. The GPT-4 score for the captioning task drops from 51.03% to 47.15% and the GPT-4 score for the classification task decreases to 50.50%. This is because the proposed self-supervised learning function for point clouds effectively captures the detailed structures and high-level semantics.
Based on ENEL-7B, we find that setting the mask ratio in the Hybrid Semantic Loss to 30% con- sistently yields better results than 60%. Additionally, the configuration where the masked token part predicts features while the visible token part reconstructs patches outperforms the reverse set- ting-whers the masked token part predicts patches and the visible token part reconstructs features. This phenomenon can be explained as follows: mask ratio of 30% retains critical information while facilitating the model to effectively utilize the visible tokens to derive the masked parts. When the mask ratio set too high, the model fails to learn the global context knowledge adequately. Moreover, when the masked token part is tasked with predicting features, the model focuses on learning the high-level context semantics, while the patch reconstruction aids in accurately capturing low-level details. In contrast, when the masked token part predicts patches, the model becomes excessively dependent on local features during the process of semantic reconstruction.
The Effects of Hierarchical Geometry Aggregation Strategy. After removing the gating mechanism in the self self-attention of the aggregation operation, the performance drops to 49.61% % and 53.60% on the captioning and classification tasks, respectively. The gating mechanism helps the model
19

---

Published as a conference paper at ICLR 2026
Table 11: Generalization analysis on the ShapeLLM baseline. based on the ShapeLLM baseline.
denotes the model is implemented
Cap
Cls
QA
Model
GPT-4
S-BERT
GPT-4
GPT-4
PointLLM-7B
44.85
47.47
53.00
41.20
PointI.LMM-13B
48.15
47.91
54.00
46.60
ShapeLLM-7B
46.92
48.20
54.50
47.40
ShapeL.LM-13B
48.94
48.52
54.00
53.10
ENEL-7B
51.03
48.79
55.55
43.80
ENEL-7B
53.26
48.75
56.00
48,90
ENEL-I3B
53.24
48.92
56.00
48.50
ENEL-13B
54.78
49.37
56.00
54.80
to adaptively filter information, allowing it to focus more discriminative features. Without the dynamic adjustment to focus on different parts of the input, the generated text from the LLM lacks accuracy and coherence, leading decrease in performance.
As the number of aggregation and propagation operations decreases, overall performance declines, mainly due to insufficient layers failing adequately model complex spatial relationships point clouds. We observe that increasing the number of LLM layers between the final aggregation operation and the first propagation operation leads to improved performance. This suggests that fewer cascaded aggregation operations require deeper network architectures for high-level feature abstraction; other- wise, insufficient depth may lead to degraded hierarchical representations. Furthermore, the presence of LLM layers between each aggregation or propagation operation enhances performance by allowing the model to process and transform compressed information. Through self-attention mechanisms, these intermediate layers can recapture and restore details lost during the aggregation process.
General Language Capabilittes. We investigate whether 3D instruction tuning compromises the LLM inherent text gener- alization. As observed in 2D LMMs (c.g., LL)VA), training exclusively multimodal data often leads to catastrophic for- getting. As shown in Table TO ENEL-7B achieves 46.4% on the MMLU benchmark [eeddrycks et al. (2020), showing slight decline compared to the original Vicuna-7B (47.1%). %). To isolate the cause, we first removed the Hierarchical Geometry Aggre- gation (HGA) strategy. The marginal difference (46.5% vs.
Table 10: Ablation on general lan-
guage capabilities (MMLU).
Method
MMLU (5-shot)
Vicuna-7B
47.1
ENEL-7B
46.4
w/o HGA
46.5
Text Data
47.3
46.4%) indicates that the architectural design is not the primary factor in this degradation. However, by incorporating 12K pure text samples sampled from (2024a) during the SFT stage, the performance rebounds 47.3%, surpassing the original baseline. This confirms that mixing pure text data effectively mitigates catastrophic forgetting and preserves the general reasoning capabilities.
LinstructGGPT-Tuuuue 143K [Chnn et al.)
J43K[Chen
Generalizability across Basclines. To demonstrate that the effectiveness of our encoder-free architecture is not limited to a specific framework (i.e., PointLLM), we extend our evaluation to the ShapeLLM baseline. We replace the mecoder-based design in ShapeLLM with our proposed ENEL architecture while strictly maintaining the original Vicuna-based LLM backbone, training data, and hyper-parameter settings for fair comparison. As presented in Table I1 ENEL-7Bi achieves a GPT-4 score of 53.26% on the captioning task, significantly outperforming the original ShapeLLM-7B. Similar consistent improvements observed at the 13B scale across diverse tasks.
Table 12: Robustness to varying inference resolutions. We evaluate models on the Objaverse Captioning task using GPT-4 scores across different point cloud densities (2K to 16K).
Method
2K/128
4K/256
8K/512
12K/1024
16K/2048
Avg.
PMintLLM-7B
33.7
41.4
44,0
41.7
32,8
38.7
ENEL-7B
44.6
49.0
51.0
50.0
46.3
48.2
ENEL-Mix-7B
46.5
50.4
51.9
47.7
49.5
Robustness to Resolution Variations. A significant limitation of encoder-based 3D LMMs is
their sensitivity to input resolution discrepancies between training and inference. To evaluate the adaptability of our encoder-fre free architecture, introduce variant named ENEL-Mix. which is pre-trained by randomly sampling point clouds ranging from 2K to 16K points per batch. We evaluate performance consistency across varying inference resolutions (2K-16K) on the Objaverse
20

---

Published as a conference paper at ICLR 2026
Table Impact of mixed-resolution training on general tasks. Comparing ENEL-MMix against standard baselines on Captioning, Classification, and QA tasks.
Cap
Cls
QA
Model
GPT-4
S-BERT
GPT-4
GPT-4
PointLLLM-7B
44.85
47.47
53.00
41.20
ShapeLLMM7B
46.92
48.20
54.50
47.40
ENEL-7B
51.03
48.79
55.55
43.80
ENEL-Mix-7B
51.92
49.06
56.00
44.40
Captioning task. As detailed in Table 12 the standard ENEL-7B already demonstrates superior robustness compared PointLLM-7B. increasing the average GPT-4 score from 38.72 48.18 (49. and significantly reducing performance variance. This indicates that the encoder-free design inherently less brittle to density changes. Furthermore, ENEL-Mix-7H enhances stability, achieving the lowest relative performance drop (10.40%) between the best and worst resolutions. Beyond robustness, mixxuu-resolution training serves as an effective data augmentation strategy. As shown in Table 13 ENEL-Mix-7B yields consistent improvements across Captioning, Classification, and QA benchmarks, demonstrating that exposing the model to varying geometric densities facilitates a more generalized 3D understanding.
Hyperparameter Sensitivity Analysis. We evaluate the robustness to variations loss weighting and auxiliary hyperparameters, with results summarized in Table 14 We perform grid search for the auxiliary loss weights within the range {0.1, 1.0, 2.0) while fixing ALLM = 1. The model demonstrates high stability, with performance fluctuations restricted within 1.0 point across the [0.5, 2.0] interval. Extreme reductions (e.g., 0.1) lead to noticeable drops, validating the necessity of these semantic guidance components, Although Amask 0.5 yields marginal gains, we adopt unit weighting (A = 1.0) as the default for its simplicity and generalizability.
We further investigate the configuration of the alignment modules. For the contrastive objective, low temperature (r 0.07) proves critical for effective discriminative learning; increasing significantly degrades performance. Regarding Knowledge Distillation (KD), applying T-2 normalization both student and teacher features prior to calculating the MSE loss results in better convergence compared to using raw feature magnitudes.
Table 14: Hyperparameter Sensitivity Analysis.
Cap
Cls
Ablation Target
Setting
GPT-4 S-BERT
SimCSE
GPT-4
Part 1: Hyhrid Semantic Loss Coefficients
Amask 0.1
49.29
48.30
48.92
54.00
Amask = 0.5
51.64
48.92
49.70
56.00
Vary Amaak
Amask (Default)
51.03
48.79
49.52
55.55
Amaak 2.0
50.71
48.55
49.17
55.50
Arccon 0.1
49.87
48. 48.12
48.85
54.00
Arecon = 0.5
50.85
48.61
49.37
55.50
Vary Araan
Avecon
1.0 (Default)
51.03
48.79
49.52
55.55
Arecon 2.0
50.80
48.45
49.20
55.00
Contrastive
Hyperparameters
0.07
45.57
45.16
45.82
47.00
Contrastive (r)
0.2
45.23
44.92
45.29
46.00
0.5
44.81
44.68
45.20
46.50
w/o Normalization
47.60
47.56
48.02
$1.50
KD (MSE)
w/ Normalization
47.98
48.10
48.63
52.50
A.3 USE OF LLM
In this paper, the Large Language Model (LLM) is mainly used for grammar checking and spelling emrrr detection.
21

---

Published as a conference paper at ICLR 2026
A.4 MODEL OUTPUT
In Figure we showcase more model output, where our ENEL provides precise and diverse responses with multi-modal 3D instruction input.
Capriee 30 wold dees.
witer dystom muttud dightifurna nath wnler maitor will lriing memmalll
mudel
This
netteed nenioal aemi
deign
The 3omadli pormeut ail serber orgioee
cheening
atructured arginsaiie pur war The palitted
curmaaly pouttened slack seats
oes syrten. coe seriec infice inters tive,
anry after
paier und for the body and lygt
streegt eagineored
Ierepmend wrisk
te yone apol
mery iever drouen con sedethedd
Whech saie the Byy rell
At Aosmeel you
thet thres
whas pruce
chould bokk roel quide you SErRECT
stgact madel furhural
This masel portings can- carraeiic The vuliic' bady beert gerty shyt seigg gival car sperty, ogls vired plar soeiss that lend in et privery. indindastt binudo erys fuecheee
mondd at conicee bireet dengs the thit ofter
In sporte penikee ren-itegl aokk fieicee lol ansenvel
uned und murtaa video poica hable vertua enviravncets, browe color
enpearenc Purber meplemeett scher atoete songle anditencrin unioore aneen
appears
mathora watee commont
product.
Figure 8: ENEL Output Examples. We demonstrate that ENEL provides precise and diverse responses when addressing different problems.
22


Reviews:
[{"review_id": "GDQV8DMV4w", "reviewer": "Reviewer_t3wM", "summary": "Towards 3D understanding using LLM architecture, this work proposed an encoder-free method that works well on classification, captioning and QA tasks. First, during pre-training stage, multiple self-supervised losses are tried and compared. Second, at the instruction-tuning stage, geometry aggregation and propagation operations are baked into the LLM layers to better use the local details of 3D point clouds.", "questions": "1. What are other models that you could use as baseline except `PointLLM-7B`?\n2. What is the performance of the proposed method using following ablations? The reasoning behind this is to see how the vanilla version without the `claimed contributions` works. \n- a. without geometry aggregation, eg. $l=0$ \n- b. the variant with $H=0$", "limitations": "", "rating": 6, "confidence": 3, "soundness": 2, "presentation": 2, "contribution": 2, "strengths": "1. It is interesting to consider encode-free alternative for 3D LLM solutions. It has the potential to overcome the restricted resolution issue, and to reduce the semantic gaps between encoder and LLM. \n2. The proposed approach achieves good performance on 3D benchmarks and sometimes is state of the art.", "weaknesses": "1. The approach is quite incremental in terms of contributions. 1) For hybrid loss choices, combining masked and reconstruction loss seems less exciting to me. 2) The geometry aggregation is designed specifically for 3D point clouds. In some way, it moves the functionality of 3D point encoder inside the LLM. Although it might work, this leads to the loss of generality of using an general-purpose LLM. \n\n2.  Back to the claims of the paper. Two limitations motivate this work. 1) The variable resolution. However, there's no results to show that the method works for the case, and how does it work. Maybe benchmarks for this is missing. 2) The semantic gap between encoder and LLM. I would like to see some evidence of this first. Then, I would like to see some results of this work alleviating the issue.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "SVUTGiegBm", "reviewer": "Reviewer_FUhe", "summary": "This paper presents the systematic investigation into encoder-free architectures for 3D LMMs. The authors propose two methods: (1) LLM-embedded Semantic Encoding with a novel Hybrid Semantic Loss during pre-training to capture high-level 3D semantics, and (2) Hierarchical Geometry Aggregation during instruction tuning to enable better perception of local geometric structures. The resulting model, ENEL-7B, achieves competitive performance with state-of-the-art encoder-based models on classification, captioning, and VQA tasks, demonstrating the viability of encoder-free architectures for 3D understanding.", "questions": "1. Why not leverage the true advantage of encoder-free during training? The paper claims encoder-free architectures can \"adapt to varying point cloud resolutions\" (a key motivation in Abstract and Figure 1a). Can you train ENEL with mixed resolutions (e.g., randomly sample 2K-16K points per batch)? This would be infeasible for encoder-based models but natural for encoder-free models.\n\n2. Can ENEL scale to scene-level understanding? All experiments are conducted on object-level point clouds (Objaverse, ModelNet40). Does ENEL maintain its advantages when handling much larger scene-level point clouds (e.g., ScanNet, 3D-GRAND)? This is crucial to assess whether encoder-free architectures can serve as a general solution for 3D understanding beyond isolated objects.\n\nI appreciate the systematic investigation approach, but the evaluation does not sufficiently demonstrate the practical advantages of encoder-free design nor compare against modern baselines. If the authors can address these concerns, I would be willing to reconsider my score.", "limitations": "", "rating": 4, "confidence": 4, "soundness": 2, "presentation": 2, "contribution": 3, "strengths": "1. The paper conducts extensive ablation studies comparing various encoder-free strategies, including different self-supervised losses and architectural components. This thorough investigation provides valuable insights and practical guidance for the community when designing 3D LMMs.\n\n2. The experiments cover multiple benchmarks across different tasks, with both GPT-4 evaluation and traditional metrics reported.", "weaknesses": "1. All core investigations are built on PointLLM (Aug 2023), a 2-year-old model, while conveniently citing ShapeLLM (2024) only in Table 5. This is fundamentally problematic—the rapid evolution of LLMs makes findings from ancient baselines nearly irrelevant. The authors fail to justify why encoder-free strategies weren't explored on ShapeLLM, raising serious questions about whether these \"insights\" generalize at all or merely exploit weaknesses of an obsolete architecture.\n\n2. Figure 3 is confusing and poorly designed. The distinctions between loss functions are unclear, undermining a key contribution of the paper.\n\n3. When finally tested with modern components (Qwen2.5-7B + ShapeLLM data), ENEL-7B* achieves only 2% gain over ShapeLLM-13B on QA. This trivial improvement seriously undermines the paper's central claim that encoder-free is a promising direction—it suggests the approach offers minimal benefit as base models improve, questioning the entire motivation.\n\n4. Including BLEU/ROUGE/METEOR in Table 5 directly contradicts ShapeLLM's findings (and the authors' own admission in A.3.2) that these metrics are meaningless for 3D LMMs. \n\n5. The paper claims to be \"the first comprehensive investigation\" yet provides no related work discussion in the main paper (only in Appendix A.1). Reviewers are not obligated to read Appendix, and without proper context in the main text, it is impossible to assess the actual novelty and how this work differs from prior encoder-free efforts in 2D LMMs or recent 3D LMMs.\n\n6. Table 5 omits many recent 3D LMMs from 2024-2025 (e.g., MiniGPT-3D, GreenPLM, and others). Without related work in the main paper, it's unclear whether this is due to incomplete survey or intentional cherry-picking. This makes it impossible to fairly assess ENEL's standing relative to the current state-of-the-art.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "QuiJRZAO6V", "reviewer": "Reviewer_fG4v", "summary": "This paper introduces ENEL, an \"encoder-free\" 3D LMM. It features a pretraining scheme with a Hybrid Semantic Loss for learnable early LLM layers and a Hierarchical Geometry Aggregation strategy (gridding, attention, pooling/unpooling) for instruction tuning. The authors report their 7B model matches or exceeds encoder-based baselines on Objaverse captioning, classification, and VQA, supported by detailed ablations.", "questions": "1. “Encoder-free” claim is not substantiated. The proposed pipeline still performs hierarchical local aggregation (FPS/k-NN grouping, grid partitioning, intra-cell/gated self-attention, pooling/unpooling). Functionally, this is an encoder—even if its parameters are “moved” into early LLM blocks. Unless “encoder-free” is defined merely as “no separate backbone module,” the claim is misleading.\n\n2. Sweep the Hybrid loss coefficients (e.g., λmask, λrecon ∈ {0.1, 0.3, 1, 3}) and report stability/performance; try temperature/normalization changes for contrastive/KD, or curriculum schedules. \n\n3. Minimal tokenizer & hierarchy:\n(a) Replace FPS+k-NN with a single linear projection (no sampling) to bound the true need for geometric preprocessing.\n(b) Compare your aggregation to simple param-efficient adaptations closer to 2D “encoder-free” (e.g., LoRA on early LLM blocks; decoupled FFN) at equal parameter/FLOP budgets.\n\n4. You motivate with resolution mismatch (Fig. 1). Please add a systematic resolution sweep for all tasks with matched token budgets, and report failure modes. \n\n5. Provide a per-module FLOP/param/time breakdown for PointLLM vs ENEL: token embedding, aggregation/propagation (including grid building), and LLM compute.", "limitations": "", "rating": 4, "confidence": 5, "soundness": 2, "presentation": 3, "contribution": 2, "strengths": "This paper poses the ambitious question of whether 3D LMMs can be \"encoder-free.\" It clearly articulates two key challenges motivating this approach—resolution mismatch and semantic misalignment—using qualitative visualisations. To address these, the authors present a concrete system (ENEL) featuring two main contributions:\n\n1.  A thorough empirical exploration of self-supervised objectives (including masked modelling, reconstruction, KD, and contrastive), which culminates in a final \"Hybrid\" loss for pretraining.\n2.  The authors report competitive results against strong baselines like PointLLM and PointLLM-PiSA, while also noting potential compute and memory savings compared to PointLLM.", "weaknesses": "1.\t“Encoder-free” claim is overstated.\nThe “token embedding module” is explicitly a lightweight Point-PN–style hierarchy: FPS downsampling, k-NN grouping, and learnable layers repeated 2–4 times, followed by a projection. This is a parameter-efficient encoder, not the near-identity/VQ/linear tokenizer style commonly implied by “encoder-free” in 2D works. The paper itself calls it “a lightweight variant of Point-PN.” Please either narrow the claim or compare against true encoder-free tokenizers (e.g., single linear/conv projection without FPS/k-NN).  \n\n\t2.\tHierarchy moved into the LLM rather than removed.\nThe Hierarchical Geometry Aggregation repeatedly performs grid partitioning, intra-cell self-attention with a gate, mean-pooling, then later propagation (unpooling). Functionally, this recreates an encoder-like local-to-global hierarchy inside the decoder stack; it’s more of an architectural relocation of parameters than removal of hierarchical encoding. A head-to-head compute/param attribution (how many FLOPs/params sit in token-embedding + aggregation blocks vs a standard 3D encoder) is needed to justify “encoder-free” beyond naming. \n \n\t3.\tLoss weighting fixed to 1 without principled tuning.\nThe Hybrid Semantic Loss sums masked-token prediction and patch reconstruction with coefficients all set to 1, then adds the language CE loss—again with unit weight. This choice is arbitrary; one would expect a meaningful sensitivity analysis or a schedule (e.g., ramping reconstruction, temperature/ratio sweeps). Currently, the paper states “coefficients all equal to 1” with no further study.  \n\n\t4.\tFairness of comparisons / settings.\nSeveral results swap backbones or data (e.g., ENEL-7B* uses Qwen2.5-7B and ShapeLLM data), while SOTA comparisons use different sizes. Stronger controls are needed: same LLM, same data, same token budget, and matched FLOPs/epoch, to isolate the architectural effect.  \n\n\t5.\tWhat is the minimal “tokenizer” needed?\nTable 1 suggests deeper token-embedding helps (3 layers best), but the “-Encoder” baseline collapses. This invites a systematic study from pure linear projection → 1-layer MLP → Point-PN-lite (2–4 layers) at matched token counts, to identify the true necessity of FPS/k-NN (and their cost).  \n\n\t6.\tCompute story remains ambiguous.\nTable 8 claims wall-clock/VRAM/FLOP gains vs PointLLM, but the new costs introduced by repeated grid builds, gated self-attention within many cells, and propagation are not separately profiled; neither is the token-embedding block. A per-stage FLOP/param breakdown would clarify where savings come from.  \n\n\t7.\tEvaluation choices.\nHeavy reliance on GPT-4 scores for captioning is common, but adding human preference or task-specific structured metrics (beyond Sentence-BERT/SimCSE) would strengthen claims—especially since the method tunes the text generator itself. (No citation needed; suggestion.)", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (Poster)