Title: Spatially-aware Weights Tokenization for NeRF-Language Models
Venue: NeurIPS 2025
Accepted: yes
Keywords: MLLM

LLM

NeRF
Abstract: Neural Radiance Fields (NeRFs) are neural networks -- typically multilayer perceptrons (MLPs) -- that represent the geometry and appearance of objects, with applications in vision, graphics, and robotics. Recent works propose understanding NeRFs with natural language using Multimodal Large Language Models (MLLMs) that directly process the weights of a NeRF's MLP. However, these approaches rely on a global representation of the input object, making them unsuitable for spatial reasoning and fine-grained understanding. In contrast, we propose **weights2space**, a self-supervised framework featuring a novel meta-encoder that can compute a sequence of spatial tokens directly from the weights of a NeRF. Leveraging this representation, we build **Spatial LLaNA**, a novel MLLM for NeRFs, capable of understanding details and spatial relationships in objects represented as NeRFs. We evaluate Spatial LLaNA on NeRF captioning and NeRF Q&A tasks, using both existing benchmarks and our novel **Spatial ObjaNeRF** dataset consisting of $100$ manually-curated language annotations for NeRFs. This dataset features 3D models and descriptions that challenge the spatial reasoning capability of MLLMs. Spatial LLaNA outperforms existing approaches across all tasks.


Peer reviews:
{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe paper proposes an extension to Neural Radiance Fields (NeRFs), which are neural networks - typically multilayer perceptrons (MLPs) - that represent the geometry and appearance of visual objects. NeRFs represent a global representation of an input object and do not capture fine-grained details making them according to the authors unsuitable for spatial reasoning. The paper proposes weights2space, a self-supervised framework for encoding a sequence of spatial tokens directly from the weights of a NerRF. The paper also proposes a Spatial LLaNA , a multimodal large language model capable of understanding details and spatial relationships in which the objects are involved leveraging the NerRF representations of the objects. The paper also proposes a manually annotated dataset explicitly designed to evaluate spatial reasoning.\n\nSTRENGTHS AND WEAKNESSES:\nStrengths\n\n-\tThe related work section is extensive and well documented with the literature. \n-\tExtensive experimental results.\n\n\nWeaknesses\n\n-\tThe computation of a spatially aware representation of an object directly from the weights of its NeRF has resemblance to the nf2vec except for a pooling operation performed by nf2vec.\n-\tThe function of the seq2space module needs to be clarified.\n-\tA detailed ablation of the difference steps/modules of the spatially-aware NeRF weights tokenization is missing. \n-\tThe spatially-aware multimodal large language model should be evaluated on a broader range of downstream tasks and datasets beyond those requiring spatial reasoning, in order to ensure that its performance on other tasks is not degraded. As a foundation model, it is intended for general-purpose use.\n-\tThe writing of the paper  is not always very clear and precise. \n-\tThe claims on significant improvements compared to state-of-the-art results need to be supported by statical significance tests.\n\nQUESTIONS:\nSee weaknesses: \n- Content should be added with regard to the function of the seq2space module,  detailed ablation of the difference steps/modules of the spatially-aware NeRF weights tokenization,  and evaluation of  the spatially-aware multimodal large language model on a broader range of downstream tasks and datasets.\n\nLIMITATIONS:\nLimitations could be added regarding the chosen architectures and scalability.", "TITLE": "Official Review", "RECOMMENDATION": 3, "REVIEWER_CONFIDENCE": 3, "CLARITY": 2, "IMPACT": 3, "ORIGINALITY": 2, "DATE": "2025-06-17"}

comments: SUMMARY:
This paper proposes weights2space, a self-supervised method that extracts spatial tokens from NeRF weights via a meta-encoder, enabling spatially-aware understanding. Based on this, the authors develop Spatial LLaNA, an MLLM that performs well on NeRF captioning and Q&A. They also introduce the Spatial ObjaNeRF dataset. The method outperforms prior work in all tasks.

STRENGTHS AND WEAKNESSES:
Strengths:

1.	The idea of this paper is interesting—it uses a self-supervised approach to obtain spatially-aware visual tokens.
2.	The experiments show the proposed method clearly outperforms the previous SOTA LLaNA.
3.	A new dataset is introduced that focuses more specifically on object parts and their spatial relationships.


Weaknesses:

1.	The proposed method’s capability mainly stems from weights2space. However, the training of weights2space relies solely on 300K samples from ShapeNeRF-Text and ObjaNeRF-Text, which may limit its generalizability in real-world applications.
2.	It seems the method is limited to handling single-object NeRFs. It's unclear how it would scale to scenarios involving multiple objects in the same scene. For example, how would the model handle questions that require reasoning about spatial relationships between multiple objects? Addressing this limitation would be important for broader applicability.
3.	Lacking some ablation studies—for instance, the number of queries in weights2space.

QUESTIONS:
See Weaknesses

LIMITATIONS:
Yes
TITLE: Official Review
CLARITY: 3
IMPACT: 2
ORIGINALITY: 3
DATE: 2025-06-30

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe authors proposed a self-supervised framework featuring a novel meta-encoder (weights2space) that can compute a sequence of spatial tokens directly from NeRF weights. The idea is that it will output a sequence of tokens that are reorganized into a tri-plane structure so that it can extract more detailed and localized information. The authors then incorporate this idea into what they call Spatial LLaNA, a new version of Multimodal Large Language Models, and evaluate its performance in various datasets (including developing a benchmark).\n\nSTRENGTHS AND WEAKNESSES:\nQuality: The authors provide extensive experiment results, including a comparison to LLaNA, their direct competitor, as well as the other reference baselines that LLaNA compared against. They also compare against three tasks: brief captioning, detailed captioning, and single-round Q&A conversations.\nClarity: The authors include multiple figures to facilitate reading. However, the figures can be improved with better annotation and descriptive titles. The paper is overall easy to follow. But when authors introduce weights2space and S-LLaNA (together with NERF, MLLM) at the beginning, it can be a little confusing about how the ideas are related. \nWriting can also be improved in Section 4, especially. There are grammatical errors that make some sentences hard to understand.\nSignificance & Originality: This is my biggest question. While the experiment results demonstrate that S-LLaNA gains some performance improvement, the encoder that the authors proposed (weights2space) doesn't seem like a fundamental improvement. I understand that the authors discussed some of the aspects in Related work, but I'd like to ask the authors to clarify more about their contribution.\n\nQUESTIONS:\n(1) The significance of your work, especially compared to LLaNA. See Strengths And Weaknesses. \n(2)  Can you explain how to get the feature vector in your tri-plane representation? Specifically, you mentioned that \"for each projection, bilinear interpolation is performed on the four nearest neighbors’ features within the corresponding plane to extract three vectors of dimension.\"\n(3) Any intuition on why introducing the triplane representation can improve the model's understanding in part-whole hierarchies?\n\nLIMITATIONS:\nyes", "TITLE": "Official Review", "RECOMMENDATION": 4, "REVIEWER_CONFIDENCE": 3, "CLARITY": 2, "IMPACT": 2, "ORIGINALITY": 3, "DATE": "2025-07-02"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper works on Multimodal Large Language Models (MLLMs) equipped with Neural Radiance Fields (NeRFs). Previous work utilizes the weights of a NeRF’s MLP as a global representation of an object, which may not be suitable for capturing the details. This paper instead invents a self-supervised way to learn a meta encoder and feed the more representative tokens into the LLMs. A new manually annotated dataset is developed and the proposed method shows promising results in such a new dataset, compared to other NeRFs-based MLLMs approaches.\n\nSTRENGTHS AND WEAKNESSES:\nStrengths: Previous work uses the global representation of NeRFs as the input of MLLMs while this work uses more detailed tokens to enable fine-grained understanding in LLMs. I reckon this is a significant improvement for the NeRFs-based MLLMs. Compared to LLaNA, S-LLaNA consistently shows a large margin on different metrics of different datasets, which demonstrates the effectiveness of the propose method.  \n\nWeaknesses: It seems like all the experiments and ablations are conducted on the caption (e.g.  S-BERT, SimCSE, BLEU-1, ROUGE-L, and METEOR) while the multi-choice and numerical evaluation is missing. It would be helpful if the authors could provide such evaluation on the proposed dataset or existing datasets. Some efforts should be made to annotate some multi-choice and numerical Q&As on the proposed dataset, though the authors have claimed such a limitation.\n\nQUESTIONS:\nThe framework has two modules, self-supervised meta encoder and MLLMs. They are trained separately but how to determine the best checkpoint of the meta encoder for the second stage training, i.e. MLLMs? Does it use the checkpoint of the third epoch for each training stage?\n\nLIMITATIONS:\nIn the supplementary material, the authors state that the trained model may perform poorly in the out-of-distribution data. To improve the generalization of the model, the scale and diversity of data can be investigated.", "TITLE": "Official Review", "RECOMMENDATION": 4, "REVIEWER_CONFIDENCE": 4, "CLARITY": 3, "IMPACT": 2, "ORIGINALITY": 3, "DATE": "2025-07-03"}

comments: SUMMARY:
This paper introduces weights2space, a self-supervised meta-encoder that reorganises NeRF-MLP weights into 3 072 spatially-aware tokens arranged as a 32 × 32 tri-plane. Plugging these tokens into a frozen LLaMA-2 via a lightweight projector yields Spatial LLaNA, a NeRF-language model capable of fine-grained spatial reasoning. The authors introduce a new benchmark, Spatial ObjaNeRF. Across four datasets and three tasks (brief captioning, detailed captioning, single-round Q&A), S-LLaNA surpasses LLaNA and 2D/3D MLLM baselines.

STRENGTHS AND WEAKNESSES:
[+] Clear architecture, end-to-end self-supervised training.

[+] Strong empirical gains and insightful attention visualisations.

[+] New dataset that targets spatial understanding.

[-] Lack of Ablations: Please provide sensitivity to the number of queries N, token dimensionality C, and tri-plane resolution.

[-] Real-world robustness: Can the model handle photogrammetry or in-the-wild neural fields?

[-] Given that tri-plane representations empirically work well, why not train the tri-plane tokens directly instead of first fitting a standard NeRF MLP and then converting its weights into tri-plane tokens? What technical or practical advantages does the two-step pipeline offer over a native tri-plane approach?

QUESTIONS:
Please check the weaknesses.

LIMITATIONS:
No.
TITLE: Official Review
CLARITY: 3
IMPACT: 2
ORIGINALITY: 3
DATE: 2025-07-05