Spatially-aware Weights Tokenization for NeRF-Language Models
Authors: Andrea Amaduzzi
Pierluigi Zama Ramirez
Giuseppe Lisanti
Samuele Salti
Luigi Di Stefano
Venue: NeurIPS 2025 poster ()
Keywords: MLLM
LLM
NeRF
Primary area: deep_learning

Abstract: Neural Radiance Fields (NeRFs) are neural networks -- typically multilayer perceptrons (MLPs) -- that represent the geometry and appearance of objects, with applications in vision, graphics, and robotics. Recent works propose understanding NeRFs with natural language using Multimodal Large Language Models (MLLMs) that directly process the weights of a NeRF's MLP. However, these approaches rely on a global representation of the input object, making them unsuitable for spatial reasoning and fine-grained understanding. In contrast, we propose **weights2space**, a self-supervised framework featuring a novel meta-encoder that can compute a sequence of spatial tokens directly from the weights of a NeRF. Leveraging this representation, we build **Spatial LLaNA**, a novel MLLM for NeRFs, capable of understanding details and spatial relationships in objects represented as NeRFs. We evaluate Spatial LLaNA on NeRF captioning and NeRF Q&A tasks, using both existing benchmarks and our novel **Spatial ObjaNeRF** dataset consisting of $100$ manually-curated language annotations for NeRFs. This dataset features 3D models and descriptions that challenge the spatial reasoning capability of MLLMs. Spatial LLaNA outperforms existing approaches across all tasks.

Spatially-aware Weights Tokenization for NeRF-Language Models
Andrea Amaduzzi andrea. amaduzzi40uni it
Pierluigi Zama Ramirez pierluigi zama@unibo it
Giuseppe Lisanti giuseppe. lisanti@uniban it
Samuele Salti samuele. salti@unibo. it
Luigi Di Stefano
luigi www.ddisffnnnnnnnndooo it
CVLab, University of Bologna
https://andreemaduzzii wwittu...o/sppaeeeeeeoeeeeee..ee 11aana
Abstract
Neural Radiance Fields (NeRFs) are neural networks typically multilayer percep- trons (MLPs) that represent the geometry and appearance of objects, with applica- tions in vision, graphics, and robotics. Recent works propose understanding NeRFs with natural language using Multimodal Large Language Models (MLLMs) that directly process the weights of NeRF's MLP. However, these approaches rely on global representation of the input object, making them unsuitable for spatial rea- soning and fine-grained understanding. In contrast, we propose we beights 2sppac,, a self-supervised framework featuring novel meta-encoder that can compute a sequence of spatial tokens directly from the weights of NeRF, Leveraging this representation, build Spatial LLaNA, novel MLLM for NeRFs, capable of understanding details and spatial relationships in objects represented as NeRFs. We evaluate Spatial LLaNA NeRF captioning and NeRF Q&A tasks, using both existing benchmarks and our novel Spatial ObjaNeRF dataset consisting of 100 manually-curater language annotations for NeRFs. This dataset features 3D models and descriptions that challenge the spatial reasoning capability of MLLMs. Spatial LLaNA outperforms existing approaches across all tasks.
Introduction
Neural Radiance Fields (NeRFs) [45] can encode the 3D geometry and photorealistic appearance of an object compactly within the weights of neural network, typically implemented Multi-Layer Perceptron (MLP). The ability of NeRFs to capture intricate visual details has led to widespread adoption in vision, graphics, and robotics [21, 59].
With NeRFs emerging as an effective and popular modality to represent 3D objects, what one could effortlessly interact through words with an AI assistant capable of providing detailed information about a complex object stored a computer as a NcRF? For instance, one may ask the assistant describe specific parts of the object intuitive, language- based interaction with NeRF representations has recently been investigated by LLaNA [6, 7]. LLaNA employs meta-encoder [53] to extract global embedding of an object
explain intricate spatial relationships precisely. Such
39th Conference on Neural Information Processing Systems (NcurIPS 2025).

---

Exput NeRF
USER; Peovide vary detatled dascripetae of the sbyect, Foess an the pelative posicion of the of the object, Inclidee rany detalla as possible.
LLaNA Meta-Encodkr
S-LLaNA Moto-Encoder
vineage, reprosecte tradicionallly used tramupariatien. it La chasectaziced
reponsents vincage horse-drash carelage. veoden ceot and a cantiel anacing coppus uuint passusquca. On onu side, woodec spaza sheel, The hares, pasitioned at the teont, glves a sende of power and histoortss Nitherefelty to the modll..
ita four wheuld, sosden strocture and covered top-
side, there in weeden sbalf red coramio pot, on the other aide, sheld with two bosne on it, Inside the curziage there uaM thzee woodan benchee.
LLM
LLM
LLaNA
S-LLaVA
Figure 1: (left) LLaNA relies on a single global representation of the input NeRF. (right) S-LLaNA processes multiple spatially- tokens extracted by our new meta-encoder and, thus, can provide detailed descriptions that capture the relationships between objects. Spatial LLaNA contains more fine-grained information (e.g., it hints at the "wooden shelf with red ceramic pot" and enumerates the "four wheels" " and the "three wooden benches") ") while also reasoning spatially (e.g., it discriminates between objects "on one side" and objects "inside the carriage").
particular, the answer of
stored NeRF by directly processing the weigts of the MLP. Compared rendering views from the given NeRF so as to feed them to a Vision-Language Model, the strategy introduced by LLaNA remarkably faster and avoids decisions that impact object understanding, such as the number of views to render and their resolution. The global representation computed by the meta-encoder is then projected into the embedding space of Large Language Model (LLM) and prepended to language instruction to realize NeRF-Language reasoning. However, we reckon that reliance on single token represent a whole NeRF limits the ability of LLaNA to capture fine-grained details and understand precise spatial relationships among objects, as illustrated in Figure 1, left.
Multimodal Large Language Models (MLLMs) that operate explicit spatial representations, such as images [4, 8, 15, 32, 33, 39, 63, 70] and point clouds [49, 50, 62]. typically achieve fine-grained spatial reasoning by dividing inputs into localized regions- images into patches and point clouds into sets - in order to extract localized, spatially-aware tokens that capture geometric and visual context. However, the weights of a NeRF inherently lack such explicit spatial structure, as the spatial information is distributed across all of them, without any weight being directly associated with any specific object part. Consequently, traditional spatially-aware tokenization approaches not directly applicable to NeRF representations.
This observation raises intriguing research question: Is it possible elicit spatially-aware representations from the information distributed across the weights of a NeRF?
To answer this question, we propose novel self-supervised framework, weights2space, trained to reorganize the information encoded within the weights of a NeRF into patiillly-aaar tokens. Specifically, wei ghts22pace employs novel Transformer-based meta-encoder that ingests the MLP weights and outputs a sequence tokens that are reorganized into a tri-plane [13] structure during training, so to associate each token with specific, localized spatial region. The meta-encoder is trained alongside a decoder to render images from the tri ttiplane representations. This approach allows the meta-encoder to extract localized, spatially-grounded grounded representation of the input NeRF. By leveraging this novel representation, we introduce Spatial LLaNA (S-LLaNA), a novel Multimodal Large Language Model for NeRFs, capable of reasoning about fine-grained object details and spatial relationships, as shown in Figure 1, right.
To validate our approach, we thoroughly evaluate Spatial LLaNA on the existing NeRF Language datasets: ShapeNeRF- [53], HST [5], and ObjaNeRF-Text [7]. Moreover, we propose a novel benchmark, Sputial ObjuNeRF, consisting of 100 NeRF models from Objaverse [18] paired with detailed language descriptions focusing on object parts and their spatial relations. Experimental results show a substantial improvement in LanguLan reasoning with approach.

---

In summary, we make the following contributions:
weights2space: a self-supervised framework featuring novel Transformer-basee meta-encoder capable of extracting spatially-aware token representations directly from the weights of a NeRF.
Spatial LLaNA: a novel MLLM for NeRFs that leverages the localized representations from the weights2space meta-encoder to achieve detailed spatial reasoning.
Spatial ObjaNeRF: a new manually annotated dataset of 100 NeRFs explicitly designed to evaluate detailed spatial reasoning tasks involving NeRFs.
Spatial LLaNA achieves state-of-ther the-art performance on the new spatial reasoning tasks of our dataset well on existing benchmarks for NeRF captioning and NeRF Q&A, demonstrating the effectiveness of our approach.
2
Related work
Deep learning on neural networks. Processing the weights of neural network poses distinct challenges compared to standard input formats, primarily due to the high dimensionality of the weight space, inherent symmetries [23], and the influence of randomness on the convergence point of training [3, 20]. Early approaches to meta-networks use group theory to design architectures equivariant to permutation symmetries in network weights [46, 69], but were limited to specific architectures like MLPs or CÑNs. To generalize across architectures, Graph Meta-Networks (GMNs) were introduced [31, 37], reframing the problem as one of converting neural networks into graphs, which GMNs can process due to their inherent permutation equivariance. 2f2vec [53] is the first method to perform tasks on NeRFs by directly processing its weights. This model computes global embedding from the weights of the NeRF's MLP, leveraging an encoder-decoder architecture supervised with a rendering loss. LLaNA [6] uses the feature vector computed by this encoder to perform NeRI-Language tasks, such as NeRF captioning and NeRF Q&A. While nf2vec is designed to ingest MIPs, [11] proposes method to process a NeRF architecture consisting of tri-plane crtucture.performiin the tasks of classification and segmentation on NeRF data. All these methods compute a single feature vector from the weights of the input NeRF, encapsulating global information about the object. In contrast, the weights2space meta-encoder proposed in this work extracts spatially-aware tokens from the weights of the input NeRF, providing richer set of information.
Multimodal Large Language Models. Multimodal Large Language Models (MLLMs) expand the capabilities of existing LLMs [1, 29, 52, 56, 68] by combining language with other modalities like images, audio, or 3D data. Vision-language models [4, 8, 15, 32, 33, 39, 63, 70] encode images using pre-trained Vision Transformers (ViT) [5]] that convert image patches into sequences of visual tokens. These local representations capture fine-grained spatial information across the image. 3D MLLMs [25, 49, 50, 62] divide the input point cloud into local patches. Transformer-basee encoders like POiT [64] and Recon++ [49] compute discrete token sequences that preserve spatial relationships. MLLMs for audio [19, 26, 65] split acoustic inputs into temporal patches and encode them using audio-specific encoders [14], producing sequences of tokens that retain temporal structure. Similarly, MLLMS for video- languaae understanding [34, 41, 43, 44, 66) use visual encoders to compute local features on each video frame, preserving both spatial and temporal information. In contrast to these local representation approaches, the only existing MLLM for Neural Radiance Fields (NeRFs), LLaNA [6], computes single global embedding from the weights of NeRF's MLP. Although this approach enables basic language tasks on NeRFs, critically overlooks spatial information since it does not adopt the local token-based representation strategy successfully used in other modalities. In this work, we propose S-LLaNA, a novel MLLM framework for NeRF which relies new weights2space meta-encoder to compute set of spatial tokens from the weights of the input NeRF's s MLP, preserving crucial spatial information while also allowing to extract richer and more detailed information from the input NeRF.
Language and Neural Radiance Fields. Given a textual description of an object, generative models produce realistic objects represented as NeRFs [28, 48, 60, 67]. Similarly, other approaches focus on editing an input scene represented as a NeRF, through text control [17, 55, 57, 58). The interaction between NeRF and language is also leveraged for 3D semantic understanding. On this task, LeRF [30] predicts language features for each spatial location alongside density and color. While the primary focus of LeRF is object localization, 3D-OVS [40] distills the reasoning capability of CLIP [51]
3

---

NeRF
welchtss2space The enooder
NeRF
Leumabie quener
Previce detailed deacziption oc She buite.
Hel regghes
weights2sca
ses2space
Tokenizerr
Tokenizer
(*(+)
Projector v A
30
nolage
Comder
Ix Iu
LLM
Decoder
RGBo
Psy
imagge teay,
1-ghtt aushi....
PE
Figure 2: weights2space overview.
Figure 3: Spatial LLaNA overview.
and DINO [12] into NeRF, to perform 3D segmentation. However, these methods train semantic neural field, relying on the noisy and view-inconsistent semantics from CLIP. In contrast to this methods, OV-NeRF [36] addresses the task of open-vocabule vocabulary 3D segmentation by leveraging a regularization technique to improve the single-view semantics and ceff-enhancement strategy for cross-vies view semantics. The neural field adopted by these methods is parametrized by a neural network. The first method to perform captioning and Q&A tasks on NeRF is LLaNA [6] while [7] expands the capabilities of LLLNN to a larger dataset, ObjaNeRF-Textt by scaling its underlying L.M..
weights2space: spatially-aware NeRF weights tokenization
Our goal is to compute a spatially-aware representation of an object directly from the weights of its NeRF model. The only existing MLLM for NeRF, LLaNA [6], utilizes a global representation the input NeRF, in the form of single embedding. In contrast, state-of-the MLLMs typically operate on spatially-aware,, token-based representations computed by Transformer encoders [49, 51, 64]. These architectures allow the model to capture localized features crucial for detailed understanding and downstream language tasks.
Inspired by these approaches, we introduce a novel self-supervised framework, weights2space, that learns how to transform the weights of NeRF into sequence of spatially-aware embeddings, as shown in Figure 2 more implementation details are in the Appendix.
NeRF encoding. Similarly to LLaNA [6], we assume NeRFs be MLPs consisting of L hidden layers with neurons, an input layer, and an output layer. The input layer has a weight matrix W, E /x and biases b, E # where I depends on the frequency encoding [45] used to encode the input coordinates. Hidden layers have the same number of input and output neurons, H, thus their squared weight matrix has dimension W. E R # and biases b, R" The output layer, which predicts the RGB colour and density a at a given 3D coordinate, has a weight matrix of W. E # and biases bout E R44 The parameters of this MLP are packed in nf2vec [53] to provide the input to the meta-encoder. In particular, the weight matrices and biases of the input MLP are stacked along the row dimension to form matrix W E RSH where S + L (H 1) + H + 2. The weights and biases of the output layer, W. and b, are padded with zeros to obtain H columns.
Meta-encoder. It processes the parameters of the input MLP packed into matrix W through two components: a weights2seq module and a seq2space module.
The weights2seq module, consisting of series of linear layers 4 in our experiments batch normalizations [27] and ReLU activations [2], processes each row of W independently for efficient computation. The module yields a set of tokens V E R each representing row of parameters in the input MLP, with S 336 and G 1024 in our setup. We note that this per-row processing module is the same as nf 2vec [53], yet with one key modification: our architecture does not perform a final max-poolls max-pooling to aggregate all revel features into a single global representation. We conjecture that the pooled vector, which is the sole input utilized by LLaNA [6] to represent a NeRF,

---

contains compact and coarse information on the underlying object. Instead, by skipping this pooling step, we aim retaining more comprehensive and fine-graine ane-grained information.
In standard MILM pipelines for visual or 3D modalities, the encoder [51, 64] computes a sequence of spatially localized tokens, each representing specific image patch set of 3D points. However, the tokens computed by the weights2seq module have no direct correspondence to any specific spatial location or object part.
To elicit spatial information from the weights of the NeRF, we deploy the seq2space module, a Transformer decoder with 12 decoder layers, where the keys and values of the cross-attention layers are obtained from the sequence of tokens V. Input to seq2space is a sequence of learnable queries
X = (x1 X2, .... Xy) E RN where N is the number of queries and C' is the feature dimension. The Transformer decoder, hence, outputs a new sequence of tokens Y = (2,, .NN) E R.N In our experiments, we employ N 3072 and C = 516.
Tri-plane representation and decoder. To force each token of Y to contain localized information, we reshape the N tokens of Y into tri-plane feature map [13], defined as M (F'ay: 1 F F'y Each plane F E R x where H, W are the spatial dimensions, is obtained by arranging subset the tokens into a 2D grid. In particular, we reshape the 3072 tokens into three 32 x 32 spatial planes. In this way, each token of Y corresponds to specific location in one of the three orthogonal planes. This design introduces spatial structure and locality into the otherwise unstructured token sequence originally obtained by the weights2seq module.
Given a 3D point p E R3, we can obtain its 3D feature by projecting onto the three orthogonal planes to its corresponding 2D coordinates: Pxy. Pxz: Pyz Then, for each projection, bilinear interpolation is performed on the four nearest neighbors' features within the corresponding plane to extract three vectors of dimension C: fxy. fxx:fyx. fyx. These are summed element- wise to obtain a unified wisnt-wise feature vector f E RC, The detailed explanation of this step is reported in the Appendix.
This feature vector is then concatenated with a positional encoding [45] of p. PE(p). and fed into an MLP decoder (.) that predicts the radiance field value q = (p. 0), consisting of color and density components (R, G, B, o). 4 consists of 4 linear layers as the decoder used in [53].
Training. All modules of weights2space are optimized end-to-end using differentiable volumetric rendering [35]. At each training step, a set of camera poses is sampled. For each pose, multiple rays cast together forming training batch R and 3D points sampled along these rays. Each point is processed through the tri-plane module and the decoder as described above. The predicted radiance and density values are then aggregated along each ray using volumetric rendering to produce final RGB values. The framework is supervised using a smooth L1 loss between rendered and ground truth colors:
Lyender
(C() -C*)))
FER
where C'*(r (r) is the ground truth RGB color for ray r and (r) is the predicted color by the decoder. Since NeRF can be standalone modality with ground truth images unavailable, we adopt self- supervised approach. Specifically, we render images directly from the input NeRFs using volumetric rendering. These pseudo ground-truth images serve as reference images, making the entire training protocol fully self-supervised in the standalone NeRF scenario. The model is trained on 300K NeRFs from ShapeNeRF- and ObjaNeRF-Ter. Text for 500 epochs with learning rate of 0.00001, as done in LLaNA [7]. The training needs ~2 days on four 64GB A100 GPUs to reach convergence.
4 Spatial LLaNA: spatially-aware MLLM for NeRF
Architecture. S-LLaNA is the MLLM that performs NeRF-Language tasks by taking as input the NeRF tokens Y E NxC computed by the weights2space meta-encoder. Inspired by [6], we process these tokens with shared projector network V, composed of stack of 2 trainable linear layers, interleaved with GeLU activation functions [24]. This network maps each NeRF token into the internal embedding space of the language model, LLaMA-2 [56]. The input to the LLM consists of the projected NeRF tokens, encapsulated between two learnable special tokens <t st_start> and <t_ end?, and concatenated with sequence of & language tokens corresponding to the user's prompt.
5

---

The language model LLaMA processes the combined sequence to autoregressively generate relevant textual output.
Training. We adopt the training protocol introduced in [7], using the ShapeNeRF- and ObjaNeRF- Text datasets. In the first stage, S-I.LaNA is trained using brief textual descriptions, optimizing only the projection layers v and the trainable special tokens <t <t_start> and <t_ In the second stage, the full model, including the LLM, is fine-tuned using both brief and detailed descriptions, as well as Q&A annotations. Importantly the encoder responsible for computing the NeRF tokens Y is kept frozen during both training stages. The total training set consists of approximately 300K NeRFs with textual annotations. We supervise S-LLaNA by minimizing the negative log- oggllikelhhod of the text token at each position:
T Σ log P(w,, | Wp..., th Wh 1, V(Y))
LCE =
where P(We Lug.... 2 - (Y)) is the probability assigned by the LLM to the correct token w, at step t, conditioned on the previously predicted tokens and the projected NeRF token sequence (Y).
Implementation details. S-LLaNA shares the same LLM backbone as LLaNA [7], i.e. the 7B Vicuna and 13B Vicuna [16] of LLaMA-2 [56]. For experimental fairness, S-LLaNA follows the same training protocol as [7] and trained on 300K samples from ShapeNeRF- and ObjaNeRF-Text for 3 epochs for each training stage. The projection network & has 2 trainable linear layers which bring each token from dimension C=516 to the hidden dimension of the LLM, i.c., 4096 for LLaMA- 7B and 5192 for LLaMa-13B S-LLaNA is implemented in PyTorch and trained on NVIDIA A100 GPUs with 64GB of VRAM each. The 7B model requires 8 GPUs for training, while the 13B model requires 16 GPUs Training either version of the model takes approximately day.
Ablation experiments on the tri- tri-plane resolution and tokens dimensionality can be found in the Appendix.
5 Experimental setup
Baselines. The only direct competitor of S-LLaNA is LLaNA [7], as this is the only MLLM performing NeRF-Language tasks without rendering any 2D image or extracting an explicit 3D structure from input NeRFs. We evaluate also the same reference baselines as LLaNA [7]. i.c.. state-of-the-art MLLMs ingesting images or 3D explicit data like point clouds and meshes, with these models processing explicit data rendered or extracted from the input NeRF. As for 3D MLLMs, included baselines are GPT4Point [50], PointLLM [62]. 3D-LLM [25] and ShapeLLL [49]. As for 2D MLLMs, we consider VLLV111.6 [39] and B [33]. Since multiple views can be rendered from an input NeRF, these 2D baselines are evaluated using different viewpoints, i.e. front-view (FV), back-view (BV), random-view (RV), and in a multi-view (MV) scenario where we concatenate image tokens from N = images rendered from random viewpoints. For each method, we report standard metrics employed in MLLM literature [7, 50, 62]: Sentence-BEil BERT [54] and SimCSE [22] based on pre-trained encoders; BLEU-1 [47], ROUGE-L [38], and METEOR [9] based on n- n-gram statistics.
NeRF- Language datasets. As in [7], evaluate the baselines the test sets of the existing NeRF- Language datasets ShapeNeRRF Text and ObjaNeRF-Text. Two different test sets for ObjaNeRF- considered ensure fair comparison with PointLLM and GPT4Point models. Results on these datasets are divided into three tasks: brief captioning, detailed captioning, and single-round Q& conversations. In addition, within the brief captioning task, we evaluate the performance the methods on HST [5], subset of ShapeNeRF- Text objects with manually annotated short captions.
Spatial ObjaNeRF. To rigorously evaluate the ability of NeRF-Language models to move beyond simple object recognition and capture complex spatial relationships, we introduce Spatial ObjaNeRF, a manually curated benchmark. This dataset consists of 100 human-annotated 3D models from Tbj Crucially, to fully assess the spatial reasoning capability of Multi-modal Large Language Models (MI.LMs) within a realistic context, every data sample represents a complex scene fcaturing an arrangement of multiple interacting objects. For each scene, created detailed textual

---

Table 1: NeRF brief captioning ShapeNeRF-Texr and HST datasets. Best results are in bold, runner-up is underlined. (FV: front-view, BV: back-view, MV: multi-view)
StopeNeRK-Iedt
Modality
S-BERT
SinCNE
BLEU-1
ROLGE-I.
METEOR
S-BERT
SinoUSE:
ROUGE-L
Image (FV)
59.85
LLaVA-iiuu--7
Image (BV)
55.68
58.46
21.97
22.46
22.50
51.75
52:39
8.13
14.18
LLaYA vicun-
Image (MV)
59.77
61.42
23.15
22.
23.01
54.15
56.57
10.32
12.76
15:13
BIJP-2 Flan15--xd
Image (FV)
56.13
58.21
5.46
18.69
9.67
57.11
99.43
8.21
18.02
12:
BLIP-2 FlanT5 2 xcl
Image(DV)
52.48
54,05
5.67
15,20
9.50
54,11
$6.37
9.09
17.35
11.79
LLaVA-viaun-1B
laage (PV)
61.00
61.16
14.30
20.00
23.31
55.62
55.56
0.55
11.81
14.
L 11 13b
Imsge (BV)
54-35
$6.00
31 94
21, 67
22.00
50.00
50 70
9.39
12.76
14 46
13 =VA-vicum-11 13b
Imuge (MV)
50.64
61.01
22.
22,
23.08
$4.35
$5.56
9.78
14.13
14. 09
Faan55 15
Mesh MV
59.46
56.42
12.69
21.49
14.32
56.07
52.13
15.94
20.71
15:22
GP1--Piin-0p-22..
Pount
41.85
11.63
43.15
13.69
5959
ShupeLLLM770
38.34
20:71
M-1
cload
9.57
cloud
21.59
21.87
17.96
25.09
38.24
9:94
LLaNA-7b
NeRF
74.94
76.41
42.73
43.64
11.95
64.78
65.01
20.65
23.24
25.10
S-LaaW4-7b
NeRF
78.91
79.91
47.81
49.08
47.58
67.76
68.63
20.31
23.61
24.47
LLANA-136
NeRF
75.09
76.45
42.83
43.70
42.21
03.06
66.09
20.80
24.28
25.90
S-MaNV-133
NGRF
78.98
79.08
47.97
49.22
47.87
67.45
68. 28.36
20.01
23.85
25.08
Table 2: NeRF brief captioning on ObjaNeRF-Tess Best results are in bold, runner-up is underlined. (RV: random view, MV: multi-view)
ObjaNeKF- -est (Polnt LM
ObjoNcRF-Teet (GPTTPPoln
Model
S-BERT
SinCSE
BL.U--1
ROUGE-L
METEOR
S-BERT
SiaCUSF
BL.U-1
ROUGE-L
METEOR
39.29
7.24
Image (MV)
Image (XV)
7.69
1aaVA-viuua-
Image (RV)
38.57
39.05
4.62
7.63
42.08
04
6.53
15.67
LLaVA vicuna 13b
Image (MV)
41.01
41.10
4.87
3.03
12:35
44.15
43.19
6.75
10.85
16.
SD-LLM FlanTS-al
Mash MV
58.22
39.50
5.47
7.29
10.75
41.26
37.34
11.08
12.26
15.5%
GPT4PPint-Op-22 2.7b
Point cloua
34.42
31.89
8.76
13.62
13.55
PoiLLLM-27
Pount cloud
38.81
40.11
5.55
3.39
11.24
MaapelLMM-7o
Pocnt cloud
30.13
32.42
5.80
7.72
11.35
25.92
25.57
8,04
9.60
13:23
Point I M-1 3b
Peeat cloud
30.64
40.63
5.94
8.43
11.63
ShapeL.LM-133 LMM-LM--33 136
Point cloud
31.44
32.69
6.01
7.59
11,66
26.87
27.03
8.31
9.65
14.10
LLaNA 7b
NeRF
41.36
42.25
13.21
16.93
15.63
43.75
43.09
20.22
25.15
22.21
S-LLaNA-7b
NeKF
45.23
17.70
50.12
89.55
24.49
11,NA-136
23.08
42.40
44.36
43.75
70:51
25.62
23.36
17.61
50.08
$0.30
20.52
description that emphasizes spatial structure, highlighting the size, shape, and relative positioning of specific objects.
Examples of such annotations and their comparison with those present in ObjaNeRF-Text are provided in the Appendix.
6 Experiments
Results on ShapeNeRF-Text. HST, and ObjaNeRF-Text. We evaluate S-LLaNA the tasks of NeRF brief captioning, detailed captioning, and Q&A, using the datasets ShapeNeRF- HST, and ObjaNeRF-Text. Results presented in Tables to 4. Across all benchmarks and tasks, S-LLaNA consistently outperforms LLaNA and all other 2D and 3D MLLM baselines, often by substantial margins. The 13B variant of S-LLaNA achieves the highest performance in nearly every setting, while the 7B variant is typically the second-best model, outperforming all the other baselines. These results demonstrate the effectiveness of the NeRF spatially-aware tokenized representation learned by our wei weights2space meta-encoder in existing public NeRF-Language datasets. On the brief captioning task, reported in Tables and 2, S-LLaNA outperforms LLaNA by approximately Sentence-BERT and SimCSE metrics, consistently across all datasets and model sizes. This indicates that even in tasks requiring high-level object descriptions, the spatially-aware NeRF tokens produced by the weights2space meta-encoder provide more discriminative and semantically rich input than the global representation of LLaNA. Table 4 shows that, on the Q&A task, S-LLaNA outperforms LLaNA with performance improvements comparable to those observed in brief captioning. This suggests that the ability to reason over object structure and pert- features, supported by our spatial encoding, is also beneficial to this task. We observe larger gains the detailed captioning task, Table 3, where S-LJaNA surpasses LLaNA by up points on Sentence-BERT and SimCSE and over 10 points reference- based metrics. These results confirm that the rich, fine-graines geometric
points on
7

---

Table 3: NeRF detailed captioning on ShapeNeRF-Tees Best results are in bold, runner-up underlined. (FV: front-view, BV:
Table 4: NeRF single-round Q&A on ShapeNeRF- Text. Best results are in bold, runner-up is underlined. (FV: front-view, BV:
back-view, MV: mueti-view)
back-view, MV: multi-view)
Smag: (P9)
13.20
71:36
25.75
34.04
34.85
(501)
15.56
18:15
15.07
30.00
16:3
Llalkiine To
tmage (495
13.76
70.07
1167
15.4
11:30
23.16
18:00
11.00
7:33
1.g
15:00
17.00
11:11
kragect9)
73663
31:08
20a (89)
11.01
T1L35
23.23
33.42
52333
trugg
broge(041)
00:11
13.51
15.00
37.16
14:4
(49)
11.56
11.10
2154
33.20
33.45
Mad
10.00
34400
3.20
Madh- -MV
1940
07.35
43.0
15.33
59.00
12.38
19.76
10:55
Paan-diad
14.20
76-41
36.31
44.03
36.36
Praectoo)
64
18.55
10:53
19.44
10.83
Pozn-doad
14.05
76.15
44.00
48.13
Shopol
43.25
40.0%
12:47
19.42
10.30
Prseciiedd
47.20
46.99
20.03
17.45
13.22
TTae To
NARE
35.55
31:42
15.55
13.96
30.15
LLaNA-7
N:RF
$3.85
$1.61
45.95
33.21
40.90
44.6
61/8
36.01
11.09
81.36
01.81
58.38
$1.33
3331
73.3
NRR
$3.00
11.78
18.00
36.14
NARD
83660
56.17
51.66
$8.25
54.58
Table 5: Detailed spatial captioning on
Figure 4: Qualitative examples on Spatial ObjaNeRF
Spatial ObjaNeRF. Best results are in bold,
runner-up underlined. (RV: random view, MV: multi-view)
Caage (5S)
70:11
11.25
21:35
34:55
21.05
muportels.cd eci
Image 01V)
54.12
00.33
8.47
18:33
6.50
23:45
L
(mage-049)
13:22
T7.8
31:33
SDALL Phalss
Moa KMV
53.24
34,00
12.05
14:37
16.85
Pren-doad
12
13.08
25.03
24.15
19:35
stapel 18-77
23.48
61.01
17.08
36.34
Pean:lood
54.22
60.96
1.55
21.00
16.34
ILNNA-7
30
NLRF
13:01
71.30
20.33
28.85
21.30
KRF
18.35
$6.58
32.34
31.83
24.56
with bota teatte
information captured by the weights2space meta-encoder helps guide the language model toward more complete and structured descriptions. All the MLLMs operating on explicit data structures, i.e., Vision Language Models and 3D MLLMs, show significantly worse performance than LLaNA and S-LLaNA, highlighting the effectiveness of building NeRF- Language models operating directly on NERF weights. Additional qualitative examples are in the Appendix.
Results on Spatial ObjaNeRF. Unlike previous captioning tasks, Spatial ObjaNeRF targets ex- plicitly the understanding of relative positions, part-whole hierarchies, and size comparisons: skills that require structured and localized representation of geometry. This benchmark, described in Sec- tion 5, is proposed as a key tool to assess whether NeRF-Language models can achieve fine-grained 3D understanding. Quantitative results on Spatial ObjaNeRF are reported in Table 5. S-LLaNA- achieves the best performance across all metrics, followed by S-LLaNA-7b. The performance gap between S-LLaNA and LLaNA highlights the importance of explicitly modeling spatial features within the representation provided as input to the LLM. While LLaNA relies global embedding computed by processing the weights of the NeRF's MLP with linear layers, S-LLaNA benefits from spatially-aware tokens extracted by the weights2space meta-encoder. Qualitative examples in Figure 4 illustrate this advantage. In the first example, only S-LLaNA accurately recognizes the objects on the table and correctly describes their spatial arrangement. Indeed, it notes that the book is "next to the bowl". LLaNA mistakenly reports the presence of "two bowls" and "two books" and describes their relative position by stating that the books are "under the bowl of fruit". In the second example, S-LLaNA demonstrates precise part-level reasoning, describing the hat of the character as a "regal crown", while LLaNA incorrectly describes it as a "sombrero". These examples further demonstrate the superior ability of S-LLaNA to interpret complex scenes with fine-grained details, particularly when multiple objects interact spatially. More qualitative examples are the Appendix, together with experiments on the generalization capabilities of S-LLANA.
Ablation on LLM finetuning. As described in Section 5, both LLLNA and S-LLaNA are trained in two stages: first, the projection network & is trained independently; then, it is jointly fine-
8

---

Table 6: Experiments LLM finetuning. Best results are in bold, runner-up is underlined. (SN-T: ShapeNeRF-Texx ON-T: ObjaNeRF-Text PointLLM test set)
SN-T caplooiin
brief captionins
SN-T detailed caplioning
SN-T
Sputial
Modd
LLM Finetured
Trainable parons
S-BERT
SinCSE
|SBBERT
SinCSE
|.URRT
SlnC
S-BERT
SiiCSE
S-BERT
SInCSE
LLaNA-Th
6:75M
74.94
75:41
41:36
43.28
75.25
17:42
$1.05
81.61
7061
71.77
LLANA-bb
1-3M
67:75
63.35
33.36
33.73
71.25
71.25
71.74
62,45
5.LaNN T6
6,758
7881
79:91
45.25
45.80
80003
83.49
84,00
S.LLNN
140M
79.29
80.36
45.37
45,81
$1.03
$2.00
82.45
83.01
73.27
70,99
15018
79.09
76.45
41.08
42.40
25.11
17.1
75.01
77.30
136
178M
70.45
03.55
35.21
30,68
72.66
72.24
72,62
02.34
5-11aN--13
15/010
78.6
79:58
45.46
06.18
83.00
84.17
80.56
S-LLaN5-136
175M
79.05
80.17
44,57
45.41
90.63
$2.50
82.10
82.67
73.72
80.47
What color Is The base of the cup?
What ooter to the 1990 of the Flowers
place XV
plane VZ
plane XX
plane
place Y7
plane S7
plane
Figure 5: Visualization of the self-attention maps of the LLM (LLaMA 2)
tuned with the LLM. This protocol, common among state-of-the-art MLLMs [39, 62]. leads to high computational costs, particularly prohibitive when using large LLMs with tens of billions of parameters. This section investigates whether comparable performance can be achieved without fine-tuning the LLM, by optimizing only the projection network v on the full dataset. Results are shown in Table 6, where rows marked "LLM Finetuned" correspond to the two-stage protocol, while the remaining rows represent models trained solely with the first stage on the full dataset. For the brief captioning task on ObjaNeRF- larger than the GPT4Point test set. Interestingly, S-LLaNA maintains excellent performance even without finetuning LLaMA. In fact, the non-finetuned S-LLLNA-7bb and S-LLaNA- 13b closely match their finetuned counterparts with performance gaps consistently under point across all metrics. In some cases, the non- oon-fineteed variants even outperform the fine-tuned ones. For instance, on the brief captioning task of ShapeNeRF-Texx the Sentence-Bel BERT score improves from 78.91 to 79.29 for S-LLaNA-7B, and from 78.98 to 79.05 for S-LLaNA-133B In contrast, LLaNA struggles without LLM fine-tuning. Its non-finetuned versions exhibit substantial drops across all benchmarks, with gaps exceeding 10 points on challenging tasks like Spatial ObjaNeRF. This trend persists even for the larger 13B model. We hypothesize that the failure of LLaNA in the non-ffietunne regime stems from the global NeRF representation used input to the LLM. This representation cannot be directly mapped to a specific language token, making fine-tuning of the LLM - and thus adaptation of its crucial step. In contrast, to represent objects, S-LLaNA uses N = 3072 spatial
we evaluate the models on the PointLLM test set, which
input token space tokens that encode more local and lower-leve concepts. Thus, it is more likely that the projector can learn mapping from these tokens to the LLM's input space without requiring LLM fine-tuning This property of S-LLaNA allows for significantly faster and more cost-effective training, while still achieving comparable performance. In the Appendix, a discussion on the general language understanding of our model is proposed.
Visualization of attention maps in LLaMA. In Figure 5, we visualize the self-attention maps within the LLM of S-LLaNA-13b when prompted with questions designed force the model to focus on specific object parts of a NeRF from the ObjaNeRF test set. Specifically, we extract the self-attention maps computed the input sequence fed into LLaMa--3b,, which consists of N = 3002 NeRF tokens, concatenated with the text tokens of the question, for a total of B tokens. For visualization, we focus on the final decoder layer of LLaMA-13b and average the attention scores across all 40 attention heads, obtaining a matrix A E R B We visualize the part of a column in A corresponding to the attention between a word in the question and the 3072 spatial tokens. To better visualize these scores, we reshape them into three images corresponding to each tri- tri-plane of dimension H W, where H W 32. Several patterns emerge from these attention maps. First, in each plane of the tri-plane, the outline of the object is distinctly visible. This indicates that the supervision of the weights2space meta-encoder successfully injects spatial object information into the NeRF token representation. Moreover, attention is clearly localized: tokens with the highest attention scores to a given word correspond to spatial regions where the referenced element is located. For instance, in the right example, the word "flower" activates tokens in the upper region of the

---

object, and "vase" focuses attention on the lower portion. These visualizations highlight the ability of S-LLaNA ground linguistic concepts into spatially localized NeRF representations.
Conclusions
In this work, we demonstrated that extracting spatially-aware tokens directly from the weights of NeRF's MLP is possible, enabling fine-grained language interaction on NeRF data. To achieve this goal, we introduced weightssspace, a self-supervised framework featuring novel meta-encoder that computes a sequence of spatial tokens, and S-LLaNA, a novel Multimodal Large Language Model for NeRF. Our model significantly outperforms existing methods on NeRF-Language tasks, whether they operate directly NeRFs or use explicit representations. Notably, S-LLaNA achieves strong performance even without finetuning its underlying LLM, making extremely efficient to train. Moreover, we propose Spatial ObjaNeRRF. benchmark designed to evaluate fine- tne-grained spatial understanding in Neng-Language tasks. Finally, we show qualitatively that our approach leads to interpretable and meaningful attention patterns. This work contributes to the nascent field of Multimodal Large Language Models for Radiance Fields.
Limitations and Future work. Our spatially-aware representation may have the potential to address additional tasks, such as spatial Q&A on 3D scenes, 3D grounding, and 3D object detection. At present, the scarcity of large-scale NeRF-text datasets limits progress in these directions. Expanding this representation to such tasks remains a promising direction for future research.
Acknowledgements
We acknowledge ISCRA for awarding this project access to the LEONARDO supercomputer, owned by the EuroHPC Joint Undertaking, hosted by CINECA (Italy).
This work was partially funded by "FSE+ 2021-2027 ai sensi dell' art. 24, comma 3, lett. a). della Legge 240/2010 s.m.i. e del D.G.R. 693/2023 (RIF. PA: 2023-20090/RER CUP: J19/230007300002".
References
[1] Josh Achiam, Steven Adler, Sandhini Agarwal, Lama Ahmad, Ilge Akkaya, Florencia Leoni Aleman, Diogo Almeida, Janko Altenschmidt, Sam Altman, Shyamal Anadkat, Gpt-4 technical report. arXiv preprint arXiv:2303.08774 2023.
[2] Abien Fred Agarap. Deep learning using rectified linear units (relu). arXiv preprint wrXiv:1803.08375. 2018.
[3] Samuel K Ainsworth, Jonathan Hayase, and Siddhartha Srinivasa. Git re- re-basin: Morging models modulo permutation symmetries. arXiv preprint orXiv:2209.04833 2022.
[4] Jean-Raptiste Alayrac, Jeff Donahue, Pauline Luc, Antoine Miech, lain Barr, Yana Hasson, Karel Lene, Arthur Mensch, Katherine Millican, Malcolm Reynolds, et al. Flamingo: a visual language model for few-shot learning. Advances in neural information processing systems, 15:23716-23777 2022.
[5] Andrea Amaduzzi, Giuseppe Lisanti, Samucle Salti, and Luigi Di Stefano. Looking words and points with attention: a benchmark text-to-shape coherence. In 2023 IEEE/CVF International Conference on Computer Vision Workshops (ICCVW), pages 2860-2869. IEEE Computer Society, 2023.
[6] Andrea Amaduzzi, Pierluigi Zama Ramirez, Giuseppe Lisanti, Samuele Salti, and Luigi Di Stefano, Llana: Large language and nerf assistant. Advances in Neural Information Processing Systems, 37:1 1195. 2024.
Andrea Amaduzzi, Pierluigi Zama Ramirez, Giuseppe Lisanti, Samuele Salti, and Luigi Di Stefano. Scaling Ilana: Advancing nerf-language understanding through large-scale training. arXiv preprint arXiv:2504.1 13995, 2025.
[8] Shuai Bai, Keqin Chen, Xuejing Liu, Jialin Wang. Wenbin Ge, Sibo Song, Kai Dang, Peng Wang. Shijie Wang, Jun Tang, et al. Qwen2. 5-vl technical report. arXiv preprint 139222502..1323, 2025.
10

---

[9] Satanjeev Banerjee and Alon Lavie. Meteor: An automatic metric for mt evaluation with improved correlation with human judgments. In Proceedings of the ael workshop on intrinsic and extrinsic evaluation measures for machine translation and/or summarization, pages 65-72, 2005.
[10] Jonathan T Barron, Ben Mildenhall, Dor Verbin, Pratul P Srinivasan, and Peter Hedman. Mip-nerf 360:
Unbounded anti-aliased neural radiance fields. In Proceedings of the IEEF/CVF conference on computer vision and pattern recognition, pages 5470-5479, 2022.
[11] Adriano Cardace, Pierluigi Zama Ramirez, Francesco Ballerini, Allan Zhou, Samuele Salti, and Luigi Di Stefano. Neural processing of tri tllnp hybrid neural fields. arXiv preprint arXiv:2310.01 140, 2023.
[12] Mathilde Caron, Hugo Touvron, Ishan Misra, Hervé Jégou, Julien Mairal, Piotr Bojanowski, and Armand Joulin. Emerging properties in supe-supe vision transformers. In Proceedings of the IEEE/CVF international conference computer vision, pages 9650-9660, 2021.
[13] Eric R Chan, Connor Z Lin, Matthew A Chan, Koki Nagano, Boxiao Pan, Shalini De Mello, Orazio Gallo, Leonidas J Guibas, Jonathan Tremblay, Sameh Khamis, al. Efficient geometry-aware 3d generative adversarial networks. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 16123-16133, 2022.
[14] Ke Chen, Xingjian Du, Bilci Zhu, Zejun Ma, Taylor Berg- Kirkpatrick, and Shlomo Dubnov. Hts-at: hierarchical token-semantic audio transformer for sound classification and detection. In ICASSP 2022-2022 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), pages 646-650. IEEE 2022.
[15] Xiaokang Chen, Zhiyu Wu, Xingchao Liu, Zizheng Pan, Wen Liu, Zhenda Xie, Xingkai Yu, and Chong Ruan. Janus-pro: Unified multimodal understanding and generation with data and model scaling. arXiv preprint 150.:2550...8111, 2025.
[16] Wei-Lin Chiang, Zhuohan Li. Ziqing Lin, Ying Sheng, Zhanghao Wu, Hao Zhang, Lianmin Zheng, Siyuan Zhuang. Yonghao Zhuang. Joseph E Gonzalez, et al. Vicuna: An open open-source chatbot impressing gpt-4 with 90%* chatgpt quality. See hrtps://vicuna. Imsys. org (accessed 14 April 2023), 2(3):6, 2023.
[17] Peng Dai, Feitong Tan, Xin Yu, Yifan Peng, Yinda Zhang, and Xiaojuan Qi. Go-nerf: Generating objects in neural radiance fields for virtual reality content creation. IEEE Transactions on Visualization and Computer Graphics, 2025.
[18] Matt Deitke, Dustin Schwenk, Jordi Salvador, Luca Weihs, Oscar Michel, Fli VanderBilt, Ludwig Schmidt, Kiana Ehsani, Aniruddha Kembhavi. and Ali Farhadi. Objaverse: A universe of annotated 3d objects. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 13142-13153 2023.
[19] Soham Deshmukh, Benjamin Elizalde, Rita Singh, and Huamingg Wang. Pengi: An audio language model for audio tasks. Advances in Neural Information Processing Systems, 36: 1 18990-181 8108, 2023.
[20] Rahim Entezari, Hanie Sedghi, Olga Saukh, and Behnam Neyshabur. The role of permutation invariance in linear mode connectivity of neural networks. arXiv preprint 2r 10.00226,, 2021.
[21] Kyle Gao, Yina Gao, Hongjie He, Dening La, Linlin Xu, and Jonathan Li. Nerf: Neural radiance field in 3d vision, a comprehensive review, arXiv preprint arXiv:2210.0037 003779 2022.
[22] Tianyu Gao, Xingeheng Yao, and Danqi Chen. Simese: Simple contrastive learning of sentence embeddings. In 2021 Conference on Empirical Methods in Natural Language Processing, EMNLP 2021, pages 6894- 6910. Association for Computational Linguistics (ACL), 2021.
[23] Robert Hecht- On the algebraic structure of feedforward network weight spaces. In Advanced Neural Computers, pages 135. Elsevier, 1990.
[24] Dan Hendrycks and Kevin Gimpel. Gaussian error linear units (gelus). arXiv preprint arXiv:1606.08411
2016.
[25] Yining Hong, Haoyu Zhen, Peihao Chen, Shuhong Zheng, Yilun Du, Zhenfang Chen, and Chuang Gan. 3d-llm: Injecting the world into large language models. Advances in Neural Information Processing Systems, 36:20482-2099, 2023.
[26] Rongjie Huang. Mingze Li, Dongchao Yang. Jiatong Shi, Xuankai Chang, Zhenhui Ye, Yuning Wu, Zhiqing Hong, Jiawei Huang, Jinglin Liu, et al. Audiogpt: Understanding and generating speech, music, sound, and talking head. In Proceedings of the AAAI Conference on Artificial Intelligence, pages 23802-23804, 2024.
11

---

[27] Sergey loffe and Christian Szegedy. Batch normalization: Accelerating deep network training by reducing internal covariate shift. In International conference machine learning, pages 448-456. pmlr, 2015.
[28] Kyungmin Jo, Gyumin Shim, Sanghun Jung, Soyoung Yang, and Jaegul Choo. Cg-nerf: Conditional generative neural radiance fields for 3d-aware image synthesis. In Proceedings of the IEEF/CVF Winter Conference on Applications of Computer Vision, pages 724-733, 2023.
[29] Jacob Devlin Ming-Wai Wei Chang Kenton and Lee Kristina Toutanova. Bert: Pre-training of deep bidirectional transformers for language understanding. In Proceedings of naacl-HLT, page 2. Minneapolis, Minnesota, 2019.
[30] Justin Kerr, Chung Min Kim, Ken Goldberg, Angjoo Kanazawa, and Matthew Tancik. Lerf: Language embedded radiance fields. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 19729-19739, 2023.
[31] Miltiadis Kofinas, Boris Knyazev, Yan Zhang, Yunlu Chen, Gertjan J Burghouts, Efstratios Gavves, Cees GM Snock, and David W Zhang. Graph neural networks for learning equivariant representations of neural networks. arXiv preprint arXiv:2403. 12222443..14433 /2143, 2024.
[32] Feng Li, Renrui Zhang, Hao Zhang, Yuanhan Zhang, Bo Li, Wei Li, Zejun Ma, and Chunyuan Li. Llava-next-interlee Tackling multi--mage, video, and 3d in large multimodal models. arXiv preprint axXiv:407.007895, 2024.
[33] Junnan Li. Dongxu Li, Silvio Savarese, and Steven Hoi. BLIP-2: Bootstrapping language-image pre- training with frozen image encoders and large language models. arXiv preprint arXiv:2301. 12597, 2023.
[34] KunChang Li, Yinan He, Yi Wang. Yizhuo Li, Wenhai Wang, Ping Luo, Yali Wang. Limin Wang, and Yu Qiao. Videochat: Chat-centric video understanding. arXiv preprint arXiv:2305.063: 2023.
[35] Ruilong Li, Matthew Tancik, and Angjoo Kanazawa. Nerface: general nerf acceleration toolbox, arXiv preprint arXiv:2210004847 2022.
[36] Guibiao Liao, Kaichen Zhou, Zhenyu Bao, Kanglin Liu, and Qing Li. Ov-nerf: Ooca-voca neural radiance fields with vision and language foundation models for 3d semantic understanding. IEEE Transactions on Circuits and Systems for Video Technology, 2024.
[37] Derek Lim, Haggai Maron, Mare Law, Jonathan Lorraine, and James Lucas. Graph metanetworks for processing diverse neural architectures. arXiv preprint arXiv:2312.04501, 2023.
[38] Chin- Yew Lin. Rouge: A package for automatic evaluation of summarics. In Text summarization branches ouf, pages 74- 81, 2004.
[39] Haotian Liu, Chunyuan Li, Qingyang Wu, and Yong Jae Lee. Visual instruction tuning. In Advances in Neural Information Processing Systems, pages 11892-34916 Curran Associates, Inc., 2023.
[40] Kunhao Liu, Fangneng Zhan, Jiahui Zhang, Muyu Xu, Yingchen Yu, Abdulmotaleb El Saddik, Christian Theobalt, Eric Xing, and Shijian Lu. Weakly supervised open-vocabulary segmentation. Advances in Neural Information Processing Systems, 36:53433-53456, 2023.
[41] Ruyang Liu, Chen Li, Yixiao Ge, Thomas H Li, Ying Shan, and Ge Li. Bt-adapter: Video conversation is feasible without video instruction tuning. In Proceedings of the IEEF/CVF Conference Computer Vision and Pattern Recognition, pages 13658-13667, 2024.
[42] Ilya Loshchilov and Frank Hutter. arXiv:1711005001, 2017.
Decoupled weight decay regularization. arXiv preprint
[43] Fan Ma, Xiaojie Jin, Heng Wang, Yuchen Xian, Jiashi Feng, and Yi Yang. Vista-llama: Reducing hallucination in video language models via equal distance to visual tokens. In Proceedings of the IEFF/CVF Conference on Computer Vision and Pattern Recognition, pages 13151-13160, 2024.
[44] Muhammad Maaz, Hanoona Rasheed, Salman Khan, and Fahad Shahbaz Khan. Vidco-chatgpt Towards detailed video understanding via large vision and language models. arXiv preprint Xiv:2306.05424 2023.
[45] Ben Mildenhall, Pratul P Srinivasan, Matthew Tancik, Jonathan Barron, Ravi Ramamoorthi, and Ren Ng. Nerf: Representing scenes as neural radiance fields for view synthesis. Communications of the ACM, 65 (1):99 106, 2021.
12

---

[46] Aviv Navon, Aviv Shamsian, Idan Achituve. Ethan Fetaya, Gal Chechik, and Haggai Maron. Equivariant architectures for learning in deep weight spaces. In International Conference on Machine Learning, pages 25790-25816. PMLR, 2023.
[47] Kishore Papineni, Salim Roukos, Todd Ward, and Wei-Jing Zhu. Bleu: a method for automatic evaluation of machine translation. In Proceedings of the 40th annual meeting of the Association for Computational Linguistics, pages 311 311-318, 2002.
[48] Ben Poole. Ajay Jain, Jonathan T Barron, and Ben Mildenhall. Dreamfusion: Text-to-3d using 2d diffusion. arXiv preprint 2r0iv:2209 /4988, 2022.
[49] Zekun Qi. Rumpei Dong. Shaochen Zhang, Haoran Geng, Chunrui Han, Zheng Ge. Li Yi, and Kaisheng Ma. Shapellm: Universal object understanding for embodied interaction. European Conference on Computer Vision, pages 214-238. Springer, 2024.
[50] Zhangyang Qi. Ye Fang. Zeyi Sun, Xiaoyang Wu, Tong Wu, Jiaqi Wang, Dahua Lin, and Hengshuang Zhao. Gpt4point: A unified framework for point-language understanding and generation. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 26417-26443 2024.
[51] Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, al. Learning transferable visual models from natural language supervision. In International conference on machine learning, pages 8748-8763. PmI.R, 2021.
[52] Colin Raffel, Noam Shazeer, Adam Roberts, Katherine Lee, Sharan Narang, Michael Matena, Yanqi Zhou, Wei Li, and Peter J Liu. Exploring the limits of transfer lcarning with unified text-to-text transformer. Journal of machine learning research, 21(14)):1-67, 2020.
[53] Pierluigi Zama Ramirez, Luca De Luigi, Daniele Sirecchi, Adriano Cardace, Riccardo Spezialetti, Francesco Ballerini, Samuele Salti, and Luigi Di Stefano. Deep learning on object-centric 3d neural fields. IEEE Transactions on Pattern Analysis and Machine Intelligence, 2024.
[54] Nils Reimers and Iryna Gurevych. Sentence-bert: Sentence embeddings using siamese bert-networks. arXiv preprint arXiv:1908 10084, 2019.
[55] Hyeonseop Song, Seokhun Choi, Hoseok Do, Chul Lee, and Tachyeong Kim. Blending-nerf: Text-driven localized editing in neural radiance fields. In Proceedings of the IEEE/CVF international conference on computer vision, pages 14383-14393, 2023.
[56] Hugo Touvron, Thibaut Lavril, Gautier Izacard, Xavier Martinet, Marie-Anne Lachaux, Timothée Lacroix, Baptiste Rozière, Naman Goyal, Eric Hambro, Faisal Azhar, et al. Llama: Open and efficient foundation language models. arXiv preprint ar02v: 13971, 2023.
[57] Can Wang, Menglei Chai, Mingming He, Dongdong Chen, and Jing Liao. Clip-nerf: Text-and-image driven manipulation of neural radiance fields. In Proceedings of IEFF/CVF conference on computer vision and pattern recognition, pages 3835-3844, 2022.
[58] Can Wang, Ruixiang Jiang. Menglei Chai, Mingming He, Dongdong Chen, and Jing Liao. Nerf-art: Text-driven neural radiance fields stylization. IEEE Transactions on Visualization and Computer Graphics, 2023.
[59] Guangming Wang, Lei Pan, Songyou Peng, Shaohui Liu, Chenfeng Xu, Yanzi Miao, Wei Zhan, Masayoshi Tomizuka, Mare Pollefeys, and Hesheng Wang. Nerf in robotics: A survey. arXiv preprint arXiv:2405.0133 2024.
[60] Zhengyi Wang, Cheng Lu, Yikai Wang, Fan Bao, Chongxuan Li, Hang Su, and Jun Zhu, Prolifiedreamer: High-fidelity and diverse text-to-3d generation with variational score distillation. Advances in Neural Information Processing Systems, 36:8406 8441, 2023.
[61] Tong Wu, Jiarui Zhang, Xiao Fu, Yuxin Wang. Jiawei Ren, Liang Pan. Wayne Wu, Lei Yang, Jiaqi Wang. Chen Qian, al. Omniobject3d: Large-vocabulary 3d object dataset for realistic perception. reconstruction and generation. Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. pages 803-814, 2023.
[62] Runsen Xu, Xiaolong Wang. Tai Wang, Yilun Chen, Jiangmiao Pang. and Dahua Lin. Pointllm: Empower- ing large language models to understand point clouds. In European Conference on Computer Vision, pages 131-147. Springer, 2024.
13

---

[63] Jiabo Ye. Haiyang Xu, Haowei Liu. Anwen Hu, Ming Yan. Qi Qian. Ji Zhang, Fei Huang, and Jingren Zhou. mplug-ow13: Towards long image-sequence understanding in multi multi-modal large language models. arXiv preprint arXiv:2408. 04840, 2024.
[64] Xumin Yu, Lulu Tang, Yongming Rao, Tiejun Huang. Jie Zhou, and Jiwen Lu. Point-bert: Pre-training 3d point cloud transformers with masked point modeling. Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 19313-19322, 2022.
[65] Dong Zhang, Shimin Xin Zhang, Jun Zhan, Pengyu Wang. Yagian Zhou, and Xipeng Qiu. Speechgpt: Empowering large language models with intrinsic cross modal conversational abilities. arXiv preprint 10000:2305.11000, 2023.
[66] Hang Zhang, Xin Li, and Lidong Bing. Video-llama: An instruction tuned audio-visual language model for vidco understanding. arXiv preprint arXiv:2306.02833 2023.
[67] Jingbo Zhang, Xiaoyu Li. Ziyu Wan, Can Wang, and Jing Liao. Text2nerf: Text-driven 3d scene generation with neural radiance fields. IEEE Transactions on Visualization and Computer Graphics, 30(12):7749. 7762, 2024.
[68] Susan Zhang, Stephen Roller, Naman Goyal, Mikel Artetxe, Moya Chen, Shuohui Chen, Christopher Dewan, Mona Diab, Xian Li, Xi Victoria Lin, et al. Opt: Open pre-trate pre-trained transformer language models. arXiv preprint arXiv:2205. 01068, 2022.
[69] Allan Zhou, Kaien Yang. Kaylee Bums, Adriano Cardace, Yiding Jiang, Samuel Sokota, J Zico Kolter, and Chelsea Finn. Permutation equivariant neural functionals. Advances in neural information processing systems, 36:24966-24992, 2023.
[70] Deyao Zhu, Jun Chen, Xiaogian Shen, Xiang Mohamed Elhoseiny. Minigpt-4: Enhancing vision-language understanding with advanced large language models. arXiv preprint 2r0iv:2304. 10592, 2023.
14

---

NeurIPS Paper Checklist
1. Claims
Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?
Answer: [Yes]
Justification: the claims made in the article are also highlighted in the abstract and introduc- tion.
Guidelines:
The answer NA means that the abstract and introduction do not include the claims made the paper.
The abstract and/or introduction should clearly state the claims made, including the contributions made in the paper and important assumptions and limitations. A No or
NA answer to this question will not be perceived well by the reviewers.
The claims made should match theoretical and experimental results, and reflect how much the results can be expected to generalize to other settings.
is fine to include aspirational goals as motivation as long as is clear that these goals not attained by the paper.
2. Limitations
Question: Does the paper discuss the limitations the work performed by the authors?
Answer: [Yes]
Justification: the limitations of the proposed approach are detailed in section 7.
Guidelines:
The answer NA means that the paper has no limitation while the answer No means that the paper has limitations, but those are not discussed in the paper.
The authors are encouraged to create a separate "Limitations" section their paper.
The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g.. independence assumptions, noiseless settings, model well-specification, asymptotic approximations only holding locally). The authors should reflect how these assumptions might be violated in practice and what the implications would be.
The authors should reflect on the scope of the claims made, e.g., if the approach was only tested on few datasets or with few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.
The authors should reflect on the factors that influence the performance of the approach. For example, facial recognition algorithm may perform poorly when image resolution is low images are taken in low lighting. Or speech-to-text system might not be used reliably to provide closed captions for online lectures because it fails to handle technical jargon.
The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size.
If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and fairness.
While the authors might fear that complete honesty about limitations might used by reviewers as grounds for rejection, worse outcome might be that reviewers discover limitations that aren't 't acknowledged judgment and recognize that individual actions in favor of transparency play impor- tant role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed to not penalize honesty concerning limitations.
the paper. The authors should use their best
3. Theory assumptions and proofs
Question: For each theoretical result, does the paper provide the full set of assumptions and complete (and correct) proof?
Answer: [NA]
15

---

Justification:
the article does not introduce new theoretical results.
Guidelines:
The answer NA means that the paper does not include theoretical results.
All the theorems, formulas, and proofs in the paper should be numbered and cross- referenced.
All assumptions should be clearly stated referenced in the statement of any theorems.
The proof' can either appear the main paper or the supplemental material, but they appear in the supplemental material, the authors are encouraged to provide short proof sketch to provide intuition.
Inversely, any informal proof provided in the core of the paper should be complemented by formal proofs provided in appendix or supplemental material.
Theorems and Lemmas that the proof relies upon should properly referenced.
4. Experimental result reproducibility
Question: Does the paper fully disclose all the information needed to reproduce the main ex- perimental results of the paper to the extent that affects the main claims and/or conclusions of the paper (regardless of whether the code and data are provided or not)?
Answer: [Yes]
Justification: we provide all the information related to the training protocol used in our experiments and the implementation details Section 3, Section 4, and Section Moreover, we provide additional details in the appendix.
Guidelines:
The answer NA means that the paper does not include experiments.
If the paper includes experiments, a No answer to this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.
If the contribution is dataset and/or model, the authors should describe the steps taken make their results reproducible or verifiable.
Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is novel architecture, describing the architecture fully might suffice, or if the contribution a specific model and empirical evaluation, it may be necessary either make it possible for others to replicate the model with the same dataset, or provide access to the model. In general. releasing code and data is often one good way accomplish this, but reproducibility can also be provided via detailed instructions for how to replicate the results, access to a hosted model (e.g., in the case of large language model), releasing of a model checkpoint, or other means that appropriate to the research performed.
While NeurlPS does not require releasing code, the conference does require all submis- sions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example
(a) the contribution is primarily a new algorithm, the paper should make clear how reproduce that algorithm. (b) If the contribution is primarily new model architecture, the paper should describe the architecture clearly and fully.
(c) the contribution is a new model (e.g., a large language model), then there should either be way to access this model for reproducing the results or way to reproduce the model (e.g., with an open-source dataset instructions for how to construct the dataset).
(d) We recognize that reproducibility may be tricky in some cases, in which case authors are welcome deseribe the particular way they provide for reproducibility. In the case of closed-source models, it may be that access to the model is limited in some way (e.g., to registered users), but it should possible for other researchers to have some path to reproducing or verifying the results.
5. Open access to data and code
16

---

Question: Does the paper provide open access to the data and code, with sufficient instruc- tions faithfully reproduce the main experimental results, as described in supplemental material?
Answer: [Yes]
all our experiments have been conducted on publicly available data, i.e., ShapeNeRF-Text, HST and ObjaNeRF-Text. Our newly introduced dataset, Spatial ObjaNeRF, the source code and the weights for all models will be publicly released in case of acceptance.
Justification:
Guidelines:
The answer NA means that paper does not include experiments requiring code. Please see the NeurlPS code and data submission guidelines (https://nips) cc/
public/ cublic/guides// Saaiddds/CooeSuumisstt.t. ctt.uiiiiiiiooooicocioooooooo for more details.
While we encourage the release of code and data, we understand that this might not be possible, "No" is an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (e.g., for a new open-source benchmark).
The instructions should contain the exact command and environment needed to reproduce the results. See the NeurIPS code and data submission guidelines (https: //nips. /nips.ccppus wwtpccccppppccccccccccccccccccct http://gui.eeeottuuummiiiii
/CodeSubmii cetoommsssoonoo ssionPolicy) for more details.
The authors should provide instructions data access and preparation, including how to access the raw data, preprocessed data, intermediate data, and generated data, etc.
The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. only subset of experiments are reproducible, they should state which ones are omitted from the script and why.
At submission time, to preserve anonymity, the authors should release anonymized versions (if applicable).
Providing as much information as possible supplemental material (appended the paper) is recommended, but including URLs to data and code is permitted.
6. Experimental setting/details
Question: Does the paper specify all the training and test details (e.g., data splits, hyper- parameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?
Answer: [Yes]
Justification: all the training and test details are reported in Section 3, Section 4, and Section 5.
Guidelines:
The answer NA means that the paper does not include experiments.
The experimental setting should be presented the core the paper to level detail that is necessary to appreciate the results and make sense of them.
The full details can be provided either with the code, in appendix, as supplemental material.
7. Experiment statistical significance
Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?
Question:
Answer: [No]
Justification: we did not conduct multiple trials for each model training and evaluation due the large computational requirements needed fine-tune the LLMs employed in our approach.
Guidelines:
The answer NA means that the paper does not include experiments.
The authors should answer "Yes" if the results are accompanied by emoor bars, confi- dence intervals, or statistical significance tests, least for the experiments that support the main claims of the paper.
17

---

The factors of variability that the error bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall run with given experimental conditions).
The method for calculating the error bars should be explained (closed form formula, call to a library function, bootstrap, etc.)
The assumptions made should be given (e.g., Normally distributed errors).
It should be clear whether the error bar is the standard deviation the standard error of the mean.
is OK to report 1-sigma error bars, but one should state it. The authors should preferably report a 2-sigma error bar than state that they have 96% CI, if the hypothesis of Normality of errors is not verified.
For asymmetric distributions, the authors should be careful not to show in tables or figures symmetric error bars that would yield results that are out range (e.g. negative error rates).
If error bars reported in tables or plots, The authors should explain in the text how they were calculated and reference the corresponding figures or tables in the text.
8. Experiments compute resources
For each experiment, does the paper provide sufficient information on the com- puter resources (type of compute workers, memory, time of execution) needed to reproduce the experiments?
Question:
Answer: [Yes]
Justification: In Section 3 and Section provide all the details about the computational resources of our experiments.
Guidelines:
The answer NA means that the paper does include experiments.
The paper should indicate the type of compute workers CPU or GPU, internal cluster, cloud provider, including relevant memory and storage.
The paper should provide the amount of compute required for each of the individual experimental rums as well as estimate the total compute.
The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g., preliminary or failed experiments that didn't fidn't make it into the paper).
9. Code of ethics
Question: Does the research conducted in the paper conform, in every respect, with the NeurIPS Code of Ethics https: :tttps://neurr http///eerrrp..peeee wwt.eers....p...lppppppepe cc/public///pccccccccccccccccccc CopuliccEEthi Bechiic///hiiisteeeeeeeeeeee htt//ttthhisiiieeeeeeee csGuidelneed
Answer: [Yes]
Justification: reviewed the guidelines listed in the NeurIPS Code of Ethics and confirm that our approach does not violate them.
Guidelines:
The answer NA means that the authors have reviewed the NeurIPS Code of Ethics.
the authors answer No, they should explain the special circumstances that require deviation from the Code of Ethics.
The authors should make to preserve anonymity (e.g., if there is special consid- eration due to laws or regulations their jurisdiction).
10. Broader impacts
Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?
Answer: [NA]
Justification: we do not foresee any direct path to using our solution for negative applications it pertains describing digital twins of single objects.
Guidelines:
18

---

The answer NA means that there is no societal impact of the work performed.
If the authors answer NA or No. they should explain why their work has no societal impact or why the paper does not address societal impact.
Examples of negative societal impacts include potential malicious unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.
The conference expects that many papers will be foundational research and not tied to particular applications, let alone deployments. However, if there is direct path to any negative applications, the authors should point it out, For example, it is legitimate to point that improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people train models that generate Deepfakes faster.
The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology is being used intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology.
If there are negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition attacks, mechanisms for monitoring misuse, mechanisms to monitor how system learns from feedback over time, improving the efficiency and accessibility of ML).
11. Safeguards
Does the paper deseribe safeguards that have been put in place for responsible release of data or models that have a high risk for misuse (e.g., pretrained language models, image generators, scraped datasets)?
Question:
Answer: [Yes]
Justification: the large language models employed in our approach, i.c., LLaMA2-76 and LLaMA2-13b, are already equipped with safeguards mechanisms, and our finetuning does not affect such features. Indeed, the datasets we used for training are not scraped from the web. They contain only textual annotations on safe contexts curated by humans.
Guidelines:
The answer NA means that the paper poses no such risks.
Released models that have a high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example requiring that users adhere to usage guidelines or restrictions to access the model or implementing safety filters.
Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.
We recognize that providing effective safeguards is challenging, and many papers do not require this, but we encourage authors to take this into account and make best faith effort.
12. Licenses for existing assets
Question: Are the creators or original owners of assets (e.g., code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected?
Answer: [Yes]
Justification: In the Appendix, we provide details about the licenses for: (i) the large language models used in our approach, (ii) the employed open source codebases and (iii) the datasets used in all our experiments.
Guidelines:
The answer NA means that the paper does not use existing assets.
The authors should cite the original paper that produced the code package dataset.
19

---

The authors should state which version of the asset is used and, if possible, include URL.
The name of the license (e.g., CC-BY 4.0) should be included for each asset.
For scraped data from a particular source (e.g.. website), the copyright and terms of service of that source should be provided.
If assets are released, the license, copyright information, and terms of use in the package should be provided. For popular datasets, paperswi thcode com/ datasets has curated licenses for some datasets. Their licensing guide can help determine the license of dataset.
For existing datasets that re-packaged, both the original license and the license of the derived asset (if it has changed) should provided.
If this information is not available online, the authors are encouraged to reach out to the asset's creators.
13. New assets
Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?
Answer: [Yes]
Justification: our Spatial ObjaNeRF dataset will be made publicly available, in case of acceptance, together with the documentation required for reproducing the experiments. Moreover, in case of acceptance, we will also release the source code of our solution.
Guidelines:
The answer NA means that the paper does not release new assets.
Researchers should communicate the details the dataset/code/model as part of their submissions via structured templates. This includes details about training, license, limitations, etc.
The paper should discuss whether and how consent was obtained from people whose asset used.
At submission time, remember to anonymize your assets (if applicable). You can either create anonymized URL or include an anonymized zip file.
14. Crowdsourcing and research with human subjects
Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given participants and screenshots, if applicable, as well as details about compensation (if any)?
Answer: [NA]
Justification: the paper does not involve crowdsoureing nor research with human subjects.
Guidelines:
The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.
Including this information the supplemental material is fine, but if the main contribu- tion of the paper involves human subjects, then much detail as possible should be included the main paper.
According to the NeurIPS Code of Ethics, workers involved in data collection, curation, other labor should be paid at least the minimum wage in the country of the data collector.
15. Institutional review board (IRB) approvals or equivalent for research with human subjects
Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or equivalent approval/review based on the requirements of your country or institution) were obtained?
Answer: [NA]
Justification: the paper does not involve crowdsourcing nor research with human subjects.
20

---

Guidelines:
The answer NA means that the paper does not involve crowdsourcing nor research with
human subjects.
Depending on the country in which research is conducted, IRB approval (or equivalent)
may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.
We recognize that the procedures for this may vary significantly between institutions and locations, and expect authors to adhere to the NeurIPS Code of Ethics and the guidelines for their institution.
For initial submissions, do not include any information that would break anonymity (if applicable), such as the institution conducting the review.
16. Declaration of LLM usage
Question: Does the paper describe the usage of LLMs if it is important, original, or non-standard component of the core methods this research? Note that the LLM used only for writing, editing, formatting purposes and does not impact the core methodology. scientific rigorousness, or originality of the research, declaration is not required.
Answer: [NA]
Justification: the development of our method did not involve LLMs.
Guidelines:
The answer NA means that the core method development in this research does not involve LLMs as any important, original, non-standard components.
Please refer to our L.LM policy (https: 225 for what should or should not be described.
21

---

Appendix
A Bilinear Interpolation of tri-plane Features
In this Section, we describe the algorithm used obtain feature vectors from our tri-plane representa- tion, introduced in Section
Inputs:
Tri-plane feature maps: three orthogonal 2D feature planes (Fays Fuz, Fyz) of dimension xxxx" xC
A 3D point p = (z, y, z) R²
Axis-aligned bounding box for coordinate normalization: aabbmin aabbmax R3
Output: Interpolated feature vector at point f E CRC
A.1 Coordinate Normalization
The 3D point p is normalized to the unit cube [0, 1|3:
p aabbmin
coords01 aabbmax aabbmin
[0,1]
(3)
A.2
Projection 2D Triplane Coordinates
The normalized 3D point is projected onto the three feature planes:
Pay = (Uayy Vay) (coords01 .r, coord801-1/
(4)
Pzz (Uzzy Uxz) (coordso1 2t coordsa1.2).
(5)
Pys (Uyes Uys) = (coords01 -1 coordso1-2 -2).
(6)
A.3
Bilinear Interpolation on Each Plane
Given feature plane Fzy and the projected coordinate Pay:
Step Map image grid coordinates
Vimg May (W - 1),
Vimg Vny (H 1)
(7)
Step Compute integer and fractional parts
[imgg |,
|mmg
(8)
8,
Vimg i, 5, = Ving
j
(9)
Step Retrieve four neighboring features
foo Fry i f10 Fry [j, 1, for Fayli + 1,
(top-left)
(top-right)
(bottom-left)
fi FFyyl + 1111+1.1 1.
(bottom-right)
Step 4: Compute bilinear interpolation
= 61.) 6.) + 5, - -1)118))f10 + +(1 - 8.)6 + 8,86f11
(10)
Steps 1-4 are repeated for the other planes to obtain fax and fyz-
22

---

A.4
Feature Combination
The final feature vector is obtained by summing the interpolated features from the three planes:
f = fay + Jaz + fyz E R
(11)
B Additional implementation details
B.1 weighta2space: spatially-aware NeRF weights tokenization
NeRF encoding. Each NeRF is implemented as an MLP composed of an input layer, L. = 3 hidden layers with H 64 neurons each, and output layer. The input to the network is I-dimensional vector, with = 75, obtained by applying frequency encoding [45] to the input coordinates. The input layer has weight matrix W, E IR 17x664 and bias vector b, 1244, Each hidden layer uses weights W, E R64x64 and biases b, E RR4 The output layer, which predicts RGB values and density has weight matrix W. E 664x4 and bias vector b, Rt. The weights2seq module takes as input all weight matrices and bias vectors, stacked row-wise into a matrix W E RSX M where H = 64 and S I + L(H + 1) H + 336.
Tri-plane representation. The sequence of tokens (y1. .n E R with 3072 and C = 516 produced by seq2space, is reshaped into a tri-plane feature representation M (Fay, F Fe F ). Each plane FF a tensor of shape II x xC, where H = W 32 and = 516.
Decoder. The decoder & consists of 4 linear layers. A skip connection is applied after the second layer by adding a projection of the input. Each linear layer followed by a ReLU activation, except for the final output layer, which is followed by sigmoid for RGB and shifted exponential for o.
Training. At each training step, for every camera pose, 55000 rays are cast, forming training batch R.
B.2 Spatial LLaNA: spatially-aware MLLM for NeRF
Training. S-LLaNA-7B and S-LLaNA-13B are trained on NVIDIA A100 GPUs following two stages protocol as described in the main manuscript. The hyperparameters each training stage are the following:
Training on brief textual descriptions from ShapeNeRF- and ObjaNeRF- Text for epochs, using learning rate of 0.002. We leverage AdamW [42] optimizer, and cosine learning rate scheduler.
Stage 1:
Stage Training on brief and detailed textual descriptions, along with Q&A conversations from the same datasets, for 3 epochs, with a learning rate of 0.00002. We leverage AdamW [42] as optimizer, and a cosine learning rate scheduler.
Datasets. The full training set of Spatial LLaNA includes approximately 300K NeRFs with textual annotations. While the ObjaNeRF-Tex training set [7] contains annotations across all categories -brief, detailed, and Q& A its test sets (PointI.LM test set and GPT4Point test set) include only manually annotated brief descriptions. Therefore, in the main paper, the brief captioning task is evaluated both ShapeNeRF-Tere Text and ObjaNeRR- ShapeNeRR. Text. Spatial ObjaNeRF provides detailed spatial annotations for subset of 100 NeRFs from the GPT4Point test set of ObjaNeRF-Text The question paired each ground-truth annotation of this dataset is: "Provide a very detailed description of the object. Focus on the relative position of the parts of the object. Include as many detaily as possible." This question does belong the training set of S-LLaNA.
while the other tasks evaluated solely on
C Ablation experiment on tri-plane resolution and token dimensionality
We conducted several experiments to analyze the sensitivity of Spatial LLaNA to the number of queries N (and consequently tri-plane resolution) and token dimensionality C. We report results in Table 7, where the first row refers to the official version of S-I. presented the main paper. The second row shows an experiment analyze the impact of the number of queries, and thus of the
23

---

Table 7: Ablation on tri-plane and token configurations. Best results are in bold, runner-up is underlined. (SN-T: ShapeNeRF- ON-T: ObjaNeRF- Text PointLLM test set)
cageisning ON-T betef captioiing detailed captioning
Q&A
Spotial ObjaNeRr
Model
Tripplane Res
Token
S-BERT
SimCSE
S-BERT
SinCSE
S-BERT
SimCSE
S-BERT
SimUSE
S-BERT
SimCSE
5izaN1-135 136
32
516
78.95
79.95
45.44
46.15
80.95
82,90
53,50
84.17
75.25
50.58
SILAN 136
15
516
76.65
77.65
41.77
42.11
76.74
78,09
53.36
8388
75.48
77.26
54788-1136
32
252
76.77
76.33
42.5
42.86
76.47
75.00
63.37
$3.49
76.62
78.43
tri-plane resolution. weights2seq and S-LLaNA were trained using N=768 queries. As each plane is obtained by reshaping the queries into 3 planes of spatial dimension H W, with H W in our
experiments, the corresponding tri-plane resolution is v
768 16. Finally, the third row analyzes
3
the sensitivity of our model to the token dimensionality C. S-LLaNA was trained with N = 3072 tokens (i.e., tri-plane resolution 32) in the main paper, yet with smaller token dimensionality C' = 252.
Each column reports the S-BERT and SimCSE results various tasks and datasets: ShapeNeRF-Too (SN-T) brief and detailed captioning and QA, ObjaNeRF-Teor (ON-T) brief captioning, and Spatial ObjaNeRR.
We observe that reducing the tri-plane resolution or token dimensionality leads to marginal perfor- mance drop (around 2% across all metrics). This behaviour suggests that models with higher capacity (i.e., higher resolution and dimensionality) can encode richer and more fine-grained features from the input NeRF. However, the relatively small drop also indicates that S-LLaNA remains robust and effective even with more compact and computationally efficient representations.
D Ablation experiment on multi-token NeRF-Language Models
In this section, we report a set of experiments designed to isolate the impact of the spatial information encoded within multi-token NeRF representations for NeRF-Language Models. Originally, LLaNA [7] leverages a single-token representation computed from the ní2vec encoder [53]. To investigate on the impact of our proposed spatially NeRF encoding, max- max-pooling layer at the end this encoder, resulting in a multi-token representation. However, these tokens lack explicit spatial information, unlike those in Spatial LLaNA, where the spatial information encoded within the NeRF tokens thanks to the specific training methodology of the wei weights2space meta-encoder. The purpose of this experiment is to determine whether the superior performance of Spatial LLaNA compared to LLaNA stems primarily from the multi-token representation or from the spatial information that is explicitly encoded within the NeRF tokens. The results of this experiment are reported Table 8. Once again, S-I.LaNA emerges as the best performing model. For I. LLaNA, the transition multiple tokens yields minimal performance improvements and, in most tasks, actually degrades performance. The only exception is the detailed captioning task on ShapeNeRF-Texl, where multiple tokens produce limited gains of approximately points in Sentence BERT and 0.20 in SimCSE metrics. A marginal 1-point improvement in Sentence BERT is also observed for the Q&A task on ShapeNeRF- Text. Overall, these results suggest that employing multi-token representations for the multimodal input instead of single-token representations contributes negligibly to performance improvements in NeRF-Language tasks. In contrast, the spatially- -aware tokens utilized by S-LLaNA prove crucial for achieving superior performance across the considered tasks. These findings strongly suggest that the ability to extract spatially- representations from the weights of the input NeRF is significantly more important than increasing the number of tokens in the representation.
train LLaNA after removing the
Table 8: Ablation experiment on multi-token representation Best results are in bold, runner-up is underlined, (SN-T: ShapeNeRF-Text, ON-T: ObjaNeRF-Tex PointLLM test set)
SN-T brief captioning
ON-T bridf captioning
SN-T detailed captioning
SN-T Q&A
Spatial ObjaNeRP
Model
S-BERT
SimCSE
S-BERT
SimCSR
S-BERT
SimCSE
S-BERT
SimCSR
S.BERT
SImCSE
S-LLaNt-13b
78.98
79.98
45.44
46.18
$0.98
82.90
83.60
84.17
78.25
80.58
LLaNA-13b
26.45
42.08
42.40
25.01
LLaNA-133 multi-toket
74.05
74.81
36.31
38.62
77.02
77.83
82.01
81.51
74.59
76.55
24

---

Table 9: Zero-shot NeRF classification task ShapeNeRF-Text. Best results are in bold.
Model
Input modality
Accuracy (%)
LLaNA-7b
NeRF NeRF
67.56
S-I.LaNA-7b
68.67
LLaNA-13b
NeRF NeRF
69.27
S-LLaNA-13b
71.85
E
Experiments on the general language understanding of Spatial LLaNA
While our primary focus is spatial reasoning, we have included downstream tasks that go beyond spatial understanding to assess whether S-LLaNA retains general reasoning capabilities. First, the brief captioning task on ShapeNeRR-Texx, HST, and ObjaNeRR- involves generating concise textual descriptions focused on identifying the main subject a scene. This task primarily evaluates global scene understanding, rather than spatial reasoning. As shown in the main paper, in this setting. S-LLaNA consistently outperforms LLaNA, demonstrating its ability to preserve a holistic understanding of the input NeRF.
Furthermore, we evaluated S S-LLaNA on a zero-shot NeRF classification task on ShapeNeRF- following the protocol in [6], where the model must infer the object class based on its multimodal input. This task does not require fine-grained spatial reasoning. Table 9 provides results on this task.
These results confirm that S-LLaNA does not sacrifice holistic visual or linguistic understanding. Finally, the ablation study on LLM finetuning of Section 6 demonstrates that S-LLaNA performs robustly even without this step, achieving results comparable to the finetuned variant. Consequently, in this version of our framework, the pre-trained LLM remains untouched, and thus certainly preserves its original reasoning capabilities and world knowledge.
F Examples from Spatial ObjaNeRF
Figure 6 presents comparison between the textual annotations from the test set of ObjaNeRF. Text [7] and those from Spatial ObjaNeRF. Specifically, since Spatial ObjaNeRF includes 100 annotated NeRFs from the GPT4Point test split of ObjaNeRF-Texx, we provide examples from this subset. The annotations Spatial ObjaNeRF clearly offer rich descriptions with spatial relationships between the objects in the scene. In contrast, the original ObjaNeRF-Text test set provides brief descriptions that focus primarily on the main subject of the scene. For completeness, we include a JSON file alongside this document that contains all textual annotations from Spatial ObjaNeRF, each paired with the corresponding object_in ind of the 3D model from the Objaverse dataset [18].
Iaput
ObjsNeltF-ee Test
Spotisl ObjaNell
Iapat
ObjoNelt- Test
Spatial ObjaNelk
crove
corrlese covered
yelnericy til
mecepst repoeercation ned pappor vatenclii glassy AicA
baioe tazcle casae benites ASSACMO to woth cecaaco pot, do boues
wooder
cricced stocade Verse-ccov
slng i Goons, aich giees cioke
land, The aad inssicata
wnter coverne feyins pas kanglng Kosk cape. The tamess lieel
trans
has wnitee
glovss kanks The
appeace westing
1913 model tendy and att faoe consce the
bext boak
there
the buse snop Geneen pins txccico poans the ragss son pullic.. ang Aagis alie The saEl aitiking prstiicaas.
thess rad with citl inages the doner
coddy beer veariey net tla, avent cistoco
busd bocke.
supaics a etatiniiie composition, placed scok by sice.
Figure 6: Comparison between ObjaNeRF- (GPT4Point test set) and Spatial ObjaNeRF annota- tions. In green, the ane-igrined and spatial details.
25

---

G Real-world generalization of Spatial LLaNA
Although ObjaNeRF- already includes 3D models of real scenes, to further test the real-world generalization S-LLaNA, we perform qualitative experiments on OmniObject3D [61] and mip- NeRF360 [10] datasets. The former provides meshes of real-scanned objects, captured in a controlled environment with neutral lighting and background, while the latter includes complex real-world unbounded scenes. mip-NeRF360 is one of the most commonly used datasets for novel view synthesis tasks. Figure 7 shows qualitative examples on OmniObject3D. Such predictions highlight the generalization abilitt of S-LLaNA, which is able to provide meaningful and spatially-grounbl descriptions even for scenes that do not belong to the training distribution of the model. In Figure 8, can notice how S-LLaNA has the potential to generalize to unbounded in-the-wild NeRF scenes. Such data are extremely different from the training set of the model, from different points of view. First of all, our training data are purely object-centric with single large object in the center, while in this case the vase of flowers on the table is quite far from the camera view. Secondly, the 3D models of training set have neutral back background, while in-the-wild scenes have very complex backgrounds with rich details. Such features make the task of generalizing to in-the-wild scenes very challenging. Despite this, S-LLaNA able to recognize the main subjects of the scene ("a round wooden table with a vase flowers on top") ") and their spatial relations ("A vase is placed on top of the table"). "). Interestingly, S-LLaNA can also provide some information about the background of the scene ("The background is green, covered with plants. "), which has never seen in any training example.
Froviid very detailted desoription objest. Fozsa netacise position parts of the abject. Tnoliide as rany detadile as poesiste.
Input
S-LLANA
ground-truth
This rodel cartcen-style building, predininattly distinetive pink facade, large cocsout in buhind the building. The atraeaace appaasa to be whineteet representation building, possibly frox children's is anixated show ez cara, the vallow roef adda playeui teuch, eakiing the leitinng agpesr less resitatic and nore tragitarive.
color, with
mais building with doors the
right aides, standing in Exent, of the rain dees consnus tree bestnd it, The rasn part of the bouse is nainly Erows in color and the eavea sre yallow, peinted, teianquiaz and nst. eymerriost.
a
This Is model cartoon-style, green- coleaed tzuin, As the doous of the tsain, cartoon-etyle bead be virible. Ite vibrast
This toy train overall green, front castoon fane huad, the back in dack geeus. and with emsl: bimp, the resr part green with gold stripes, there is also a smill burp, there are sore trregelar patzerns on the eice, the frent and zeaz each have a buzp, total sheele, four zests, large, the overall structuxe the of oynnetey.
green color ackes it eyeneazehiag. fhis type of sbgect could be used in vartous applications wuch aninasion, video design, digital art to ste playtul and nodezn aestbetio.
Figure 7: Qualitative examples of S-LLaNA OmniObject3D
Iaput
S-LLaVA
tnne, prozido very ditailed description at the object. Focza parte many ponatble.
This ca 30 modal of as ceodecs soace droe a park or gurden. featurea round wooden
background is greet, covaced with plante. Tie acasa sugpasss a prscadul, pural lide assup,
caok: Muss is placed top the table?
A wase is pleced on top of the table,
20ER: Wbace the vase piaced in the
the zado placed os a cablo, shichh
posssloned
Figure 8: Qualitative examples of S-LLaNA on Mip-NeRF 360
26

---

sigmn,
lepar
ELANA
ILAVA
with restaralaa
aith
balle, there
proser seal
Pratie fos grese and the
konta Gails The skiin
the
stes beike
meating froms
for ond
wygoere Life copuatetin
This larps deteriieen
opasker the
cal cedkarplar
in hep the aie diferttto
syeake stel
modes paraug
kow tae, spees cosppecttion of goils.
suple The wase
tn peetects linee ves for
The spaite La mack buiper that.
with
donnd pounlated kockir,
cable bore
bas Alomssikle spu mask,
from
buy
Bhe
Thi desige
belf This colad
epcisslie aas vair veen
spidmas
beey
22 uuld spuds seeetee)
botth The
tieea) not
screed clsting
biikk appers (tct)
borkh Thes telagtale
lnd, tnlapion wall.
stik
la Plad
actiin coms.
vites gau.
rieal comm plouttuua
en the El00
tovecca Ca srourd and the
The
Junn atladda
bey then
stactiin will,
Jenn
will,
startisn mndd plare being sertinal, sllge
Figure 9: Qualitative evaluation on Spatial ObjaNeRF. In red, the hallucinations generated by the MLLMs. In green, the correct fine-grained and spatial details.
H Qualitative results on ShapeNeRF-Text, ObjaNeRF-Text and Spatial ObjaNeRF
Figure 9 shows qualitative results from the evaluation of the baselines Spatial ObjaNeRF. This dataset contains a large set of scenes involving multiple objects, as shown in the Figure (first, second, and fourth row). It can be noticed how Spatial LLaNA (S-LLaNA in short) is capable of generating detailed and accurate descriptions of the input scenes, including fine-grained spatial information, without exhibiting signs of hallucination. In contrast, the baseline methods either omit such details or produce incorrect descriptions. The ground-truth annotation column further confirms that Spatial ObjaNeRF provides rich and comprehensive textual descriptions of the input 3D models. Figures 10 to 12 report qualitatives examples on the language tasks evaluated on the test sets of ShapeNeRF-Text and ObjaNeRF-Text [7]. In particular, since the test set of ObjaNeRF-Text contains only brief descriptions, this is the only task evaluated on this dataset, as in the experiments reported in the main paper. Results show that also on these tasks Spatial L.NNA is able to recognize and describe very specific details about the input scene, while the competing basclines are not able to do so.
27

---

nyn sreplew
Baper
LLAVA
groass
strsston Siaded
propee Recertd Poans
convertiile
Eapus
SALLad
LIANA
PoiSLLM
LLaVA
grousd-rruth
Ter, sar Apuans
THEy the Appeaae
Figure 10: Qualitative evaluation of the single-round Q&A task on ShapeNeRF- Text. In red, the hallucinations generated by the MLLMs. In green, the correct fine- one-grained and spatial details.
Maar obfect Lizartiste
Eapst
SdiaNi
LLaNA
PORALEM
TLAVA
greanderuth
Stataccess siis black fremy comeey and wilver desaors
akade saltee boank, the distal porsilty
rellred sterd siis shald morere desice.
caliored the hiaa
makiny then fert
The andd cepornaie skauiaay
laggle
with goides see baok.
martin mak.
our ines The
eximt pussiby shay
grldes vall.
coft and cantly.
The deses
10 west wi wetter loaa
La
Mlack mnsert witm
Mari mor strs gellew
Figure 11: Qualitative evaluation of the brief captioning task on Shape and ObjaNeRF. Text. In red, the hallucinations generated by the MILMs. In green, the correct fine-grained and spatial details.
I Failure cases of Spatial LLaNA
Figure 13 presents some failure cases of Spatial LLaNA on Spatial ObjaNeRF. These failures can be broadly categorized into two main categories: (i) the model fails to recognize the main object in the scene (first and second row), or (ii) the model generates short descriptions lacking specific details (third row). Such failures may happen in cases where out-of-distribution objects are provided input to the model. For example, in the first row, the model is fooled by the unusual shape of the teapot. leading to the weird description "a square device for making tea", while the "pink teapot" had already been correctly included in the answer. In the second row, since most of the snowmen of ObjaNeRF-Tess T-xxt are simple white snowballs with outstretched arms made of sticks, this particular snowman is incorrectly described as "cartoon character". Another interesting case appears in the last row, where Spatial LLaNA correctly detects a "blue toy truck" but omits any distinctive features in the generated description. This may be attributed to the complexity or uniqueness of the object, which challenges the ability of Spatial LLaNA to generate a detailed description.
J Additional visualization examples of the attention maps in LLaMA
In this section, we report additional visualization examples of the self-attention maps LLaMA, the underlying LLM of Spatial LLaNA. While in the main manuscript we present the attention patterns from single question word to the 3072 spatial NeRF tokens, here we show the opposite. More precisely, for given NeRF token at a specific object location, we visualize the normalized attention scores to each text token in the question (Figure 14). These maps are derived from the final decoder layer of LLAMA-13BB, with attention scores averaged across all 40 heads. These attention maps confirm the ability of the 2eights2space meta-encoder to generate spatially-aware tokens from the weights of the input NcRF, As shown in the first row of the Figure, tokens located at the base of
28

---

Binsspion sIjas1
laper
sporles will
dowk wite silme maser
sase hears lop sidme
mike of sealeamndn dig
silmn
en
vsuail yrain pansers criong
whine Sakal the froe
atenalatiin, reairelly deeee snai
here the basn
werble men semeers
vies of synour aue sissagis
the comemnttii shape meed pny fan
lood aquated spuent
mee puuully your fur, whic
the spnkeer hos steple and
shaps and musile was
in alloer in mice and
wask
disstration polley
vopy unes unn
cledk
mille's
18 dows mis repomale
seaplial repreee lake
wae uame madglt and caiin
las skille witi bar
boly pill
Tantegt
alla uad diar
saplons and
in
the stege
Coling the but
vall bus vy
Lobil ressuneds slle
with toa lesttig per retructed
tay wit passod Lim.
ceokles leskerm yetlio and
momenial worienlee
Figure 12: Qualitative evaluation of the detailed captioning task on ShapeNeRF- In red, the hallucinations generated by the MLLMs. In green, the correct fine- -neggriiedd and spatial details.
USER: Peovida verv detailed description of the cbject. Facual an the relstive position of the parta the object. includs details possible.
Input
S-LLANA
ground-truth
Legpot is Eiee-ddinasionn ctjeee. with nstte finish. has equaze shape dour laga, handle on one side three spoute, one on each of the other sides, Each spout di dieeennn geonatric shape, One is rounded, one straight with straight
3D mode- at anall pink beapot placed
grey table, poseibly accorpanted square rakinng
seation, ciemlar cross section. the lid is separate piece with a knob on the top. The coler of the Leapot is cons tast alads of pink, and there vialile
pattezno.
snowas nodel is conposed of zain parts: the head and the body, Each part made by large anovball. the head is whi ta with blse hat and green aunglasee. the body in with blse and green
30 nodel of cartoon character wearing sunglazzes, scart, and avaates,
with
body ia vnite.
The object In boy Lruck with predeminantlyy alse colour.
rade ratal
and the platforn is auszouated by a scodan frans. On the platforn, there is stack of cardboard
The wheels black with aged aуpажranоe vi th rust and chipped paint, dezign of the truck includes de details auch the grille, headlights and taillights. On the scont geslla or the vebicie is yellow the insoription '2", the right the roate emblen yullow. the tip the bood are the borna of bull.
nodel blue boy tzuck with wooden
blue rin. body
bed.
Figure 13: Examples of failure cases of Spatial LLaNA on Spatial ObjaNeRF. In red, the hallucina- tions generated by the model. In green, the correct fine-grained and spatial details.
the cup and on the vase of the flower exhibit the strongest attention to the words "base" and "vase". respectively, while showing much lower scores for the remaining words. Similarly, in the second row, tokens covering the top of the objects correlate most strongly with the words "cup" and "flower".
These visualizations further highlight the ability of S- to connect the words of the question with the NeRF tokens representing specific parts of the object.
29

---

What: te haze the
color flower?
Figure 14: Visualization of the normalized self-attention scores of single NeRF tokens towards the text tokens of the question. yellow, surrounded by red circle, the reference NeRF token.
K Details about datasets, models and source code licenses
This section provides details about the datasets, models, and source code licenses used the work, ensuring proper credit to the creators original owners, and adherence to license terms.
Datasets: the datasets employed in our work and the relative licenses are listed below:
ShapeNeRF-Text: licensed under MIT License.
ObjaNeRF-Text: licensed under MIT License.
ShapeNet: licensed under GNU Affero General Public License v3.0.
Objaverse: licensed under Open Data Commons Attribution License v1.0 (ODC-By v1.0).
GPT2Shape HST: licensed under Creative Commons Iiabuttion-onnCommriioo
ShareAlike 4.0 International License.
Models: the models used in our experiments and their relative licenses are detailed below:
LLaNA: licensed under MIT License.
nf2vec:
licensed under MIT License.
PointLLM: licensed under Creative Commons Atrrharios-onnnComeerraaaaaaaae 4.0 International License.
licensed under Creative Commons Atributioe-NnnCCommrcciillaaaaaa 4.0 International License.
GPT4Point:
ShapeLLM: licensed under Apache License 2.0.
3D-LLM: licensed under MIT License.
BLIP-2: licensed under BSD 3-Clause License.
LLAVA: licensed under Apache License 2.0.
LLAMA-2: licensed under META LLAMA COMMUNITY LICENSE AGREEMENT!.
'https://at wwt.mmaa....mmmmmmmmmmmm /eeee//ccce///mmm////nee/////
30


Reviews:
[{"review_id": "WD2H4edBHv", "reviewer": "Reviewer_GQGs", "summary": "This paper introduces weights2space, a self-supervised meta-encoder that reorganises NeRF-MLP weights into 3 072 spatially-aware tokens arranged as a 32 × 32 tri-plane. Plugging these tokens into a frozen LLaMA-2 via a lightweight projector yields Spatial LLaNA, a NeRF-language model capable of fine-grained spatial reasoning. The authors introduce a new benchmark, Spatial ObjaNeRF. Across four datasets and three tasks (brief captioning, detailed captioning, single-round Q&A), S-LLaNA surpasses LLaNA and 2D/3D MLLM baselines.", "questions": "Please check the weaknesses.", "limitations": "No.", "rating": 4, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 3, "significance": 2, "originality": 3, "strengths_and_weaknesses": "[+] Clear architecture, end-to-end self-supervised training.\n\n[+] Strong empirical gains and insightful attention visualisations.\n\n[+] New dataset that targets spatial understanding.\n\n[-] Lack of Ablations: Please provide sensitivity to the number of queries N, token dimensionality C, and tri-plane resolution.\n\n[-] Real-world robustness: Can the model handle photogrammetry or in-the-wild neural fields?\n\n[-] Given that tri-plane representations empirically work well, why not train the tri-plane tokens directly instead of first fitting a standard NeRF MLP and then converting its weights into tri-plane tokens? What technical or practical advantages does the two-step pipeline offer over a native tri-plane approach?", "was_revised": true, "final_justification": "The authors addressed my concerns. I keep my score", "rebuttal": "## 1. Lack of Ablations: Please provide sensitivity to the number of queries N, token dimensionality C, and tri-plane resolution.\n\nWe conducted several ablations to analyze sensitivity to the number of queries N (and consequently tri-plane resolution) and token dimensionality C. We report results in the table below. In particular, each table row reports:\n* First row: the official S-LLaNA version of the main paper.\n* Second row: an experiment to analyze the impact of the number of queries, and thus of the tri-plane resolution. Here, weights2space and S-LLaNA were trained using N=768 queries. As each tri-plane is obtained by reshaping the queries into 3 planes of spatial dimension $H \\times W$, with $H = W$ in our experiments, the corresponding tri-plane resolution is $\\sqrt{\\frac{768}{3}} = 16$.\n* Third row: an experiment to analyze the sensitivity to token dimensionality $C$. S-LLaNA was trained with N=3072 tokens (i.e., tri-plane resolution 32) as in the main paper, yet with a smaller token dimensionality $C=252$.\n\n|Model|Tri-plane resolution|Token dimensionality|SN-T brief||ON-T brief||SN-T detailed||SN-T QA||Spatial ObjaNeRF||\n|:---:|:---:|:---:|:---:|:---:|:---:|:---:|:---:|:---:|:---:|:---:|:---:|:---:|\n||||S-BERT|SimCSE|S-BERT|SimCSE|S-BERT|SimCSE|S-BERT|SimCSE|S-BERT|SimCSE|\n|S-LLaNA-13b|32|516|78.98|79.98|45.44|46.18|80.98|82.90|83.60|84.17|78.25|80.58|\n|S-LLaNA-13b|16|516|76.65|77.68|41.77|42.11|76.74|78.69|83.36|83.83|75.48|77.26|\n|S-LLaNA-13b|32|252|76.77|76.33|42.25|42.36|76.47|78.07|83.27|83.49|76.62|78.43|\n\nEach column reports the S-BERT and SimCSE results on various tasks and datasets: ShapeNeRF-Text (SN-T) brief and detailed captioning and QA, ObjaNeRF-Text (ON-T) brief captioning, and Spatial ObjaNeRF.  \n\nWe observe that reducing the tri-plane resolution or token dimensionality leads to a marginal performance drop (∼2% across all metrics). This behaviour suggests that models with higher capacity (i.e., higher resolution and dimensionality) can encode richer and more fine-grained features from the input NeRF. However, the relatively small drop also indicates that S-LLaNA remains robust and effective even with more compact and computationally efficient representations.\n\nMoreover, we point out that in Sec. 6 (supplementary), we report an additional ablation study of the modules of the weights2space meta-encoder proposed in this paper. This experiment clearly shows that the weights2seq module alone performs poorly compared to the full architecture.\n\n## 2. Real-world robustness: Can the model handle photogrammetry or in-the-wild neural fields?\n\nOur experiments already demonstrate that S-LLaNA can handle real-world data. Indeed, the Objaverse dataset [17] used in the experiments of Table 2 and Table 5 (main paper) includes several real-world objects and scenes captured through diverse 3D acquisition methods, photogrammetry among them. Below, we show two representative examples from our test set featuring a real scene (Example 1) and a real-world object (Example 2).\n\n---\n\n__Example 1 (Real Scene)__\n\n__Objaverse ID__\nba341c4ce89647ea9f6996ec58e3eacf \n\n__Spatial detailed description, ground truth:__\n_This is a 3D scene with two tractors, a loader, and a trailer parked on a plot of land with plenty of wood waiting to be loaded. The tractors are positioned side by side, with the loader attached to one of them. The trailer is parked in front of the tractors, and the wood is stacked neatly beside the tractors. The ground appears to be a mix of dirt and patches of grass, and there are no visible signs of recent activity, suggesting the scene might be a still moment in time._\n\n__Spatial detailed description, prediction:__\n_The 3D model represents a typical farming scene. It features two tractors, one of which is a large, robust machine with a visible driver's cabin, indicating it's likely used for heavy-duty tasks. The other tractor is smaller, possibly a compact model, which might be used for lighter tasks or for navigating narrower spaces. Both tractors are parked on a plot of land, suggesting they are either in a state of rest or not currently in use. There is also a loader, a versatile piece of equipment used for lifting and moving materials, which implies that the land might be a construction or building site._\n\n---\n\n__Example 2 (Real Object)__\n\n__Objaverse ID:__ c4a0c2e2fb624bc0af9928b0ae6407ff\n\n__Spatial detailed description, ground truth:__\n_The blue teddy bear is depicted with a soft, plush texture, and its color is a consistent shade of blue throughout. The bear has a round head with two small, round ears, a snout, and a pair of eyes. It has a body with a round belly, and its arms are short and stubby. The bear's legs are also short and rounded. The teddy bear is in a sitting position. Around its neck, there is a white ribbon._\n\n__Spatial detailed description, prediction:__\n_The 3D object model is a toy, specifically a teddy bear, painted in a vibrant shade of blue. The teddy bear is in a seated position, showcasing a white plush neck, enhancing its cute appeal. The teddy bear, being a universal symbol of comfort and care, is usually used as a plaything for children or as a decorative item in a nursery room or any child-oriented space._\n\n--- \n\nThe examples above show that S-LLaNA produces accurate descriptions of real-world entities. For instance, in Example 1, it can properly describe the scene: _it features two tractors_ ... _both tractors are parked on a plot of land_ ... _there is also a loader_ ...\nMoreover, in the second example, S-LLaNA can recognize main features of the object: _a teddy bear, painted in a vibrant shade of blue_ ... _in a seated position_ ... _a white plush neck_.\n\nFurthermore, to validate the applicability of S-LLaNA to in-the-wild NeRFs, we also test it on another dataset containing real objects, OmniObject3D. To conduct this experiment, we fit NeRFs for a few objects employing the images officially released with the dataset. These NeRFs are provided as input to our framework. Below, we show some qualitative results:\n\n---\n\n__OmniObject3D ID:__\n_house_004_\n\n__Ground Truth:__\n_The house has a main building and an annex with doors on the left and right sides, a man standing in front of the main door and a coconut tree behind it, the main part of the house is mainly brown in color and the eaves are yellow, pointed, triangular and not symmetrical._\n\n__Prediction:__\n_This is a 3D model of a cartoon-style building, predominantly brown in color, with a distinctive pink facade. A large coconut tree is behind the building. The structure appears to be a whimsical representation of a building, possibly from a children's animated show or game. The yellow roof adds a playful touch, making the building appear less realistic and more imaginative._\n\n---\n\n__OmniObject3D ID:__ _toy_train_044_\n\n__Ground Truth:__\n_This toy train is overall green, the front is a cartoon face head, the back is dark green and with a small bump, the rear part is green with gold stripes, there is also a small bump, there are some irregular patterns on the side, the front and rear each have a bump, a total of six wheels, four small, two large, the overall structure of the axis of symmetry._\n\n__Prediction:__\n_This is a 3D model of a cartoon-style, green-colored train. At the front of the train, a cartoon-style head is visible. Its vibrant green color makes it eye-catching. This type of object could be used in various applications such as animation, video game design, or digital art due to its playful and modern aesthetic._\n\n---\n\nFrom the examples above, we can appreciate how S-LLaNA generalizes effectively to real-world NeRFs belonging to a dataset __never seen during training__. For instance, in the first example, it correctly describes relative positioning and object properties: _A large coconut tree is behind the building_ ... _The yellow roof_. In the second example, S-LLaNA can capture object properties and fine-grained details: _green-colored train_ ... _at the front of the train, a cartoon-style head_.\n\n---\n### Note to visualize example models\n\nRegarding ObjaNeRF-Text, the 3D models can be accessed by downloading the Objaverse dataset from the official website and locating the models using the Objaverse IDs provided above.\n\nRegarding OmniObject3D, the dataset can be downloaded using the _openxlab_ library specifying the data folder _house_, or _toy_train_ for the first and second example, respectively. To visualize the objects, refer to the folders _house_004_ and _toy_train_044_.\n\n## 3. Given that tri-plane representations empirically work well, why not train the tri-plane tokens directly instead of first fitting a standard NeRF MLP and then converting its weights into tri-plane tokens? What technical or practical advantages does the two-step pipeline offer over a native tri-plane approach?\n\nOur decision to adopt the NeRF MLP representation was driven by both practical and methodological considerations.\n\nFirst, our work follows the same setup as LLaNA, which focuses on MLP-based NeRFs. The MLP representation is the original parameterization used in the first NeRF paper, and consequently, many NeRFs available online adopt it. Moreover, this is the only representation used in all existing NeRF-Language datasets. Therefore, using this setup allows for direct comparison with LLaNA and compatibility with existing NeRF-Language benchmarks.\n\nSecond, as there are currently no publicly available NeRF-language datasets built on tri-plane-based representations, transitioning from MLP to tri-plane NeRFs would require rendering multi-view images for each object and subsequently fitting the corresponding tri-plane NeRF. This process introduces a substantial computational burden and complexity.\n\nNevertheless, we agree that designing a NeRF-Language model that takes as input also the tri-plane representation is a promising research direction towards a universal NeRF assistant."}, {"review_id": "rKw4lFrVjk", "reviewer": "Reviewer_QcvA", "summary": "This paper works on Multimodal Large Language Models (MLLMs) equipped with Neural Radiance Fields (NeRFs). Previous work utilizes the weights of a NeRF’s MLP as a global representation of an object, which may not be suitable for capturing the details. This paper instead invents a self-supervised way to learn a meta encoder and feed the more representative tokens into the LLMs. A new manually annotated dataset is developed and the proposed method shows promising results in such a new dataset, compared to other NeRFs-based MLLMs approaches.", "questions": "The framework has two modules, self-supervised meta encoder and MLLMs. They are trained separately but how to determine the best checkpoint of the meta encoder for the second stage training, i.e. MLLMs? Does it use the checkpoint of the third epoch for each training stage?", "limitations": "In the supplementary material, the authors state that the trained model may perform poorly in the out-of-distribution data. To improve the generalization of the model, the scale and diversity of data can be investigated.", "rating": 4, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 3, "significance": 2, "originality": 3, "strengths_and_weaknesses": "Strengths: Previous work uses the global representation of NeRFs as the input of MLLMs while this work uses more detailed tokens to enable fine-grained understanding in LLMs. I reckon this is a significant improvement for the NeRFs-based MLLMs. Compared to LLaNA, S-LLaNA consistently shows a large margin on different metrics of different datasets, which demonstrates the effectiveness of the propose method.  \n\nWeaknesses: It seems like all the experiments and ablations are conducted on the caption (e.g.  S-BERT, SimCSE, BLEU-1, ROUGE-L, and METEOR) while the multi-choice and numerical evaluation is missing. It would be helpful if the authors could provide such evaluation on the proposed dataset or existing datasets. Some efforts should be made to annotate some multi-choice and numerical Q&As on the proposed dataset, though the authors have claimed such a limitation.", "was_revised": false, "final_justification": "", "rebuttal": "## Weakness. It seems like all the experiments and ablations are conducted on the caption (e.g. S-BERT, SimCSE, BLEU-1, ROUGE-L, and METEOR) while the multi-choice and numerical evaluation is missing. It would be helpful if the authors could provide such evaluation on the proposed dataset or existing datasets. Some efforts should be made to annotate some multi-choice and numerical Q&As on the proposed dataset, though the authors have claimed such a limitation.\n\nTo better assess NeRF-language understanding and text generation capabilities, we built a new test set of spatial multiple-choice questions focused on the 100 objects from the Spatial ObjaNeRF dataset. These questions are crafted to probe fine-grained spatial reasoning within the 3D scenes. Each object is associated with one or more questions, resulting in a total of 190 multiple-choice tasks, each with three answer options.\n\nBelow are two representative examples from the dataset:\n\n**Object id**: ac58cb3a28cb4a0d92ce1016f74fa5ff\n\n**Question**: How many spouts does the teapot have?\n- A: One\n- B: Two\n- C: Four\n\n**Ground-truth answer**: C: Four\n\n-------------------------------------------------------\n\n**Object id**: 103989411047470ab9f86341fd016539\n\n**Question**: Where is the bowl of the fruit placed with respect to the book?\n- A: The bowl of fruit is next to the book\n- B: The bowl of fruit is on top of the book\n- C: The bowl of fruit is under the book\n\n**Ground-truth answer**: A: The bowl of fruit is next to the book\n\n-------------------------------------------------------\n\n\nThe results of our evaluation on this dataset are summarized in the table below:\n\n| Model               |     Modality     | Accuracy (%) | LLM Accuracy (%) |\n|---------------------|:----------------:|-------------:|------------------:|\n| GPT4Point-Opt-2.7b  | Point cloud      |        15.26 |            47.89  |\n| LLaVA-vicuna-7b     | Image (RV)       |       63.16  |            63.16  |\n| PointLLM-7b         | Point cloud      |        24.74 |            49.47  |\n| ShapeLLM-7b         | Point cloud      |        23.16 |            44.21  |\n| LLaNA-7b            | NeRF             |         6.32 |            47.89  |\n| S-LLaNA-7b          | NeRF             |        22.63 |            54.74  |\n||\n| LLaVA-vicuna-13b    | Image (RV)       |        64.74 |            64.74  |\n| PointLLM-13b        | Point cloud      |        24.74 |            54.21  |\n| ShapeLLM-13b        | Point cloud      |        23.16 |            44.74   |\n| LLaNA-13b           | NeRF             |         7.37 |            46.32  |\n| S-LLaNA-13b         | NeRF             |        23.68 |            55.79  |\n\nThe \"Accuracy\" column reflects cases in which the model's output exactly matched the ground truth answer with the correct format, e.g., \"C: Four\". As the results show, LLaNA, S-LLaNA, and the other 3D MLLMs generally struggle on this task, while LLaVA achieves significantly higher accuracy.\n\nWe hypothesize that this is largely due to LLaVA having been trained on multiple-choice QA tasks, whereas the 3D MLLMs and NeRF-language models evaluated in this work were not. Consequently, these models often produce free-form textual responses rather than selecting an explicit multiple-choice answer in the expected format (i.e., \"letter: answer\"). While the correct answer is frequently mentioned in their generated text, it is not structured in a way that permits direct comparison with the ground truth. For example, we report the answers of S-LLaNA to the two questions shown above:\n\n**Object id**: ac58cb3a28cb4a0d92ce1016f74fa5ff\n\n**S-LLaNA**: The teapot has four spouts.\n\n-------------------------------------------------------\n\n**Object id**: 103989411047470ab9f86341fd016539\n\n**S-LLaNA**: The fruit bowl is placed on the table next to the book.\n\nTo address this limitation and ensure a fair comparison, we implement an auxiliary evaluation protocol. Specifically, we leverage a strong instruction-following LLM (LLaMA-3.3-70B-Instruct) to interpret the free-form outputs of each model and map them to the closest valid multiple-choice option, when possible. The resulting \"LLM Accuracy\" column reflects this refined evaluation. \n\nThe scores of LLaVA remain consistent across both evaluation strategies, proving the effectiveness of the LLM-based evaluation protocol and the proper answer formatting of this model. Among all baselines, LLaVA remains the top performer, likely due to its explicit exposure to multiple-choice QA during training.\n\nImportantly, among the models that were not trained on multiple-choice QA tasks, S-LLaNA emerges as the best performer by a notable margin, outperforming both LLaNA and other 3D MLLMs. This suggests the potential to perform excellently on such diverse NeRF-language tasks.\n\nThis multi-choice QA dataset will be publicly released, and the evaluation results will be included in the final version of the paper.\n\n## Questions. The framework has two modules, self-supervised meta encoder and MLLMs. They are trained separately but how to determine the best checkpoint of the meta encoder for the second stage training, i.e. MLLMs? Does it use the checkpoint of the third epoch for each training stage?\nTo determine the best checkpoint of the self-supervised meta encoder for the second-stage MLLM training, we select the one that achieves the highest reconstruction PSNR on the validation set. In representation learning scenarios such as ours, where the objective is to learn rich and spatially grounded embeddings of the input scene, PSNR provides a strong and interpretable proxy for feature quality. This approach has also been adopted in prior works such as [6], [7], and [51] for downstream language and semantic tasks, respectively, demonstrating strong empirical performance. By selecting the checkpoint with the highest PSNR, we ensure that the MLLM is trained on top of a robust and well-aligned scene representation.\n\n## Limitations. In the supplementary material, the authors state that the trained model may perform poorly in the out-of-distribution data. To improve the generalization of the model, the scale and diversity of data can be investigated.\n\nDuring the rebuttal phase, we conducted additional experiments suggested by the reviewers to probe the generalization capabilities of S-LLaNA further. In particular, we evaluated S-LLaNA also on another dataset containing real objects, OmniObject3D. To conduct this experiment, we fit NeRFs for a few objects employing the images officially released with the dataset. These NeRFs are provided as input to our framework. Below, we show some qualitative results:\n\n---\n\n__OmniObject3D ID:__\n_house_004_\n\n__Ground Truth:__\n_The house has a main building and an annex with doors on the left and right sides, a man standing in front of the main door and a coconut tree behind it, the main part of the house is mainly brown in color and the eaves are yellow, pointed, triangular and not symmetrical._\n\n__Prediction:__\n_This is a 3D model of a cartoon-style building, predominantly brown in color, with a distinctive pink facade. A large coconut tree is behind the building. The structure appears to be a whimsical representation of a building, possibly from a children's animated show or game. The yellow roof adds a playful touch, making the building appear less realistic and more imaginative._\n\n---\n\n__OmniObject3D ID:__ _toy_train_044_\n\n__Ground Truth:__\n_This toy train is overall green, the front is a cartoon face head, the back is dark green and with a small bump, the rear part is green with gold stripes, there is also a small bump, there are some irregular patterns on the side, the front and rear each have a bump, a total of six wheels, four small, two large, the overall structure of the axis of symmetry._\n\n__Prediction:__\n_This is a 3D model of a cartoon-style, green-colored train. At the front of the train, a cartoon-style head is visible. Its vibrant green color makes it eye-catching. This type of object could be used in various applications such as animation, video game design, or digital art due to its playful and modern aesthetic._\n\n---\n\nFrom the examples above, we can appreciate how S-LLaNA generalizes effectively to real-world NeRFs belonging to a dataset __never seen during training__, demonstrating that S-LLaNA can effectively handle some degree of domain shift. For instance, in the first example, it correctly describes relative positioning and object properties: _A large coconut tree is behind the building_ ... _The yellow roof_. In the second example, S-LLaNA can capture object properties and fine-grained details: _green-colored train_ ... _at the front of the train, a cartoon-style head_.\n\nNevertheless, we acknowledge that increasing the diversity and scale of the training data represents a promising direction to further improve the generalization capabilities of the model on out-of-distribution scenarios, and is therefore worth investigating in future work."}, {"review_id": "UbycKvYX31", "reviewer": "Reviewer_1Y9M", "summary": "The authors proposed a self-supervised framework featuring a novel meta-encoder (weights2space) that can compute a sequence of spatial tokens directly from NeRF weights. The idea is that it will output a sequence of tokens that are reorganized into a tri-plane structure so that it can extract more detailed and localized information. The authors then incorporate this idea into what they call Spatial LLaNA, a new version of Multimodal Large Language Models, and evaluate its performance in various datasets (including developing a benchmark).", "questions": "(1) The significance of your work, especially compared to LLaNA. See Strengths And Weaknesses. \n(2)  Can you explain how to get the feature vector in your tri-plane representation? Specifically, you mentioned that \"for each projection, bilinear interpolation is performed on the four nearest neighbors’ features within the corresponding plane to extract three vectors of dimension.\"\n(3) Any intuition on why introducing the triplane representation can improve the model's understanding in part-whole hierarchies?", "limitations": "yes", "rating": 4, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 2, "significance": 2, "originality": 3, "strengths_and_weaknesses": "Quality: The authors provide extensive experiment results, including a comparison to LLaNA, their direct competitor, as well as the other reference baselines that LLaNA compared against. They also compare against three tasks: brief captioning, detailed captioning, and single-round Q&A conversations.\nClarity: The authors include multiple figures to facilitate reading. However, the figures can be improved with better annotation and descriptive titles. The paper is overall easy to follow. But when authors introduce weights2space and S-LLaNA (together with NERF, MLLM) at the beginning, it can be a little confusing about how the ideas are related. \nWriting can also be improved in Section 4, especially. There are grammatical errors that make some sentences hard to understand.\nSignificance & Originality: This is my biggest question. While the experiment results demonstrate that S-LLaNA gains some performance improvement, the encoder that the authors proposed (weights2space) doesn't seem like a fundamental improvement. I understand that the authors discussed some of the aspects in Related work, but I'd like to ask the authors to clarify more about their contribution.", "was_revised": true, "final_justification": "I increased my score because the authors clarified my question about their contribution.", "rebuttal": "## Weakness 1. Clarity: The authors include multiple figures to facilitate reading. However, the figures can be improved with better annotation and descriptive titles. The paper is overall easy to follow. But when authors introduce weights2space and S-LLaNA (together with NERF, MLLM) at the beginning, it can be a little confusing about how the ideas are related. Writing can also be improved in Section 4, especially. There are grammatical errors that make some sentences hard to understand.\n\nWe thank the reviewer for the suggestion. We will include more descriptive captions for Figures 2 and 3 to enhance clarity. The captions will be:\n\nFigure 2: Overview of weights2space. Given the weights of a NeRF MLP, our meta-encoder computes a sequence of spatially-aware tokens, $Y$. These tokens are reshaped into tri-plane feature maps, $F_{xy}$, $F_{xz}$, and $F_{yz}$, which are used to obtain a feature $f$ for each 3D point $p_i$. This feature is then processed by a decoder that computes the radiance field values for $p_i$. For a given viewpoint, radiance field values are sampled along camera rays, and pixel RGB values are obtained through volumetric rendering. The meta-encoder and decoder are jointly trained in a self-supervised manner to reconstruct the image rendered by the original NeRF from the same viewpoint.\n\nFigure 3: Overview of Spatial-LLaNA. The sequence of spatially-aware tokens $Y$ is processed by a Linear Projection Network to map such vectors into the input embedding space of the LLM we employ, LLAMA. The projected tokens are concatenated with the text tokens of the input question and fed into the LLM for the generation of the output text. Both the Projector and the LLM are optimized through language supervision.\n\nWe will also revise Section 4 in the final version of this paper to improve its fluency.\n\n## Weakness 2. Significance & Originality: This is my biggest question. While the experiment results demonstrate that S-LLaNA gains some performance improvement, the encoder that the authors proposed (weights2space) doesn't seem like a fundamental improvement. I understand that the authors discussed some of the aspects in Related work, but I'd like to ask the authors to clarify more about their contribution.\n## &\n## Question (1) - The significance of your work, especially compared to LLaNA. See Strengths And Weaknesses. \n\nThank you for your comment. We would like to clarify the novelty and significance of our contribution compared to previous work, i.e., LLaNA.\n\nLLaNA introduced the idea of encoding a NeRF scene into a single global latent vector, which enabled a language assistant to reason about neural radiance fields. However, this global representation discards all spatial structure, limiting the ability of the assistant to capture spatial and local information. In contrast, S-LLaNA proposes a fundamentally different and more expressive representation computed thanks to our original meta-encoder architecture, dubbed weights2space. Unlike the meta-encoder design described in LLaNA, weights2space does not collapse the NeRF weights into a single embedding. Instead, it produces a sequence of spatially-aware tokens enabling the assistant to reason on spatial relationships among objects, capture fine details, and understand part-whole hierarchies. \n\nFrom a slightly more technical standpoint, we wish to point out that S-LLaNA meta-encoder key originality pertains to the seq2space module, a transformer decoder that cross-attends to the encoded weights of the input NeRF, along with the downstream tri-plane representation. These two components enable the model to extract spatially grounded features from the information spread across the weights of the input NeRF. In particular, because the output tokens of seq2space are reshaped into a tri-plane to enable rendering views from the input NeRF, the features learned by the S-LLaNA meta-encoder are strongly correlated with the object's parts and spatial layout. This novel strategy for eliciting spatially-aware features from the information distributed across the weights of the input NeRF marks a clear shift from the approach described in LLaNA, which compresses the weights into a global embedding and lacks any mechanism to preserve spatial structure.\n\nTherefore, by enabling rich and detailed spatial reasoning on 3D objects, the design of S-LLaNA represents a significant step forward compared to LLaNA in advancing the capabilities of language assistants that can accept NeRFs as inputs.\n\n## Question (2) - Can you explain how to get the feature vector in your tri-plane representation? Specifically, you mentioned that \"for each projection, bilinear interpolation is performed on the four nearest neighbors’ features within the corresponding plane to extract three vectors of dimension.\"\n\nWe provide below the algorithm employed to obtain feature vectors in our tri-plane representation. We will add this information to the supplementary material.\n\n**Inputs:**\n\n* Tri planes feature maps – three orthogonal 2D feature planes $(F_{xy}, F_{xz}, F_{yz})$ of dimension $\\mathbb{R}^{H \\times W \\times C} $.\n\n* A 3D point, $p = (x, y, z) \\in \\mathbb{R}^3$.\n\n* Axis-aligned bounding box for coordinate normalization $aabb_{min},aabb_{max} \\in \\mathbb{R}^3$.\n\n**Output:**\n* Interpolated feature vector at point $p$, $f \\in \\mathbb{R}^C$.\n\n---\n\n__1. Normalize coordinates__\n\nNormalize the 3D point $p$ to $[0, 1]$:\n\n$$\ncoords_{01} = \\frac{p - aabb_{min}}{aabb_{max} - \\text{aabb}_{min}} \\quad \\in [0, 1]\n$$\n\n---\n\n#### 2. Project 3D point to 2D coordinates on triplanes\n\n$$\np_{xy} = (u_{xy}, v_{xy}) = (coords_{01}.x, coords_{01}.y)\n$$\n\n$$\np_{xz} = (u_{xz}, v_{xz}) = (coords_{01}.x, coords_{01}.z)\n$$\n\n$$\np_{yz} = (u_{yz}, v_{yz}) = (coords_{01}.y, coords_{01}.z)\n$$\n\n---\n\n#### 3. Perform bilinear interpolation on each plane\n\nGiven the $xy$ feature plane, $F_{xy}$, and the projected coordinate of $p$, $p_{xy}$:\n\n1. Map from $[0, 1]$ to triplane grid coordinates:\n\n$$\nu_{img} = u_{xy}(W - 1) \\qquad v_{img} = v_{xy}(H - 1)\n$$\n\n2. Compute integer and fractional parts:\n\n$$\ni = \\lfloor u_{img} \\rfloor, \\quad\nj = \\lfloor v_{img} \\rfloor\n$$\n\n$$\n\\delta_u = u_{img} - i, \\quad\n\\delta_v = v_{img} - j\n$$\n\n3. Retrieve the four neighboring features:\n\n- $f_{00} = F_{xy}[j, i, :]$ (top-left)  \n- $f_{10} = F_{xy}[j, i+1, :]$ (top-right)  \n- $f_{01} = F_{xy}[j+1, i, :]$ (bottom-left)  \n- $f_{11} = F_{xy}[j+1, i+1, :]$ (bottom-right)\n\n4. Perform bilinear interpolation:\n\n$$\nf = (1 - \\delta_u)(1 - \\delta_v)f_{00} + \\delta_u(1 - \\delta_v)f_{10} + (1 - \\delta_u)\\delta_v f_{01} + \\delta_u \\delta_v f_{11}\n$$\n\nRepeat 1. to 4. for other planes and projected coordinates to obtain:\n* $f_{xz}$ from $F_{xz}$ and $p_{xz}$\n* $f_{yz}$ from $F_{yz}$ and $p_{yz}$\n\n---\n\n#### 4. Combine features\n\nSum the three interpolated features to obtain the final feature vector for point $p$:\n\n$$\nf = f_{xy} + f_{xz} + f_{yz} \\quad \\in \\mathbb{R}^C\n$$\n\n## Question (3) - Any intuition on why introducing the triplane representation can improve the model's understanding in part-whole hierarchies?\n\nThe explicit tri-plane spatial structure allows the model to associate specific tokens with specific regions of the scene. \nConsequently, as tokens capture local part-level information, while the overall spatial configuration provides cues about how these parts compose the global scene, this representation can be effectively processed by the downstream LLM for understanding part-whole hierarchies.\n\nFor instance, if we have two spatial tokens representing a _lamp_ and a _chair_ that are located one above the other in the tri-plane representation, then a language model processing these tokens can infer that the _lamp_ is positioned above the _chair_. The same reasoning applies in the case of individual objects, where spatial tokens correspond to different parts of the object. Based on their relative positions in the tri-plane, the model can correctly infer that the _head_ is above the _torso_, and the _torso_ is above the _legs_. Furthermore, by understanding how these parts are composed, the model can also infer the identity of the whole object. For instance, if the head, torso, and legs are vertically aligned, the object is likely a standing person.\n\nThese considerations are supported by the visualization in Figure 5 (main paper) and Figure 7 (supplementary), where the attention maps are clearly localized. Specifically, the tokens receiving the highest attention scores correspond to spatial regions that align with the referenced elements in the scene. This suggests that the LLM is capable of identifying both what and where the relevant parts of the scene are in response to a given question, thanks to the triplane representation, thereby enabling it to generate more effective and spatially grounded answers."}, {"review_id": "3tSszUWn4h", "reviewer": "Reviewer_DMsk", "summary": "This paper proposes weights2space, a self-supervised method that extracts spatial tokens from NeRF weights via a meta-encoder, enabling spatially-aware understanding. Based on this, the authors develop Spatial LLaNA, an MLLM that performs well on NeRF captioning and Q&A. They also introduce the Spatial ObjaNeRF dataset. The method outperforms prior work in all tasks.", "questions": "See Weaknesses", "limitations": "Yes", "rating": 4, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 3, "significance": 2, "originality": 3, "strengths_and_weaknesses": "Strengths:\n\n1.\tThe idea of this paper is interesting—it uses a self-supervised approach to obtain spatially-aware visual tokens.\n2.\tThe experiments show the proposed method clearly outperforms the previous SOTA LLaNA.\n3.\tA new dataset is introduced that focuses more specifically on object parts and their spatial relationships.\n\n\nWeaknesses:\n\n1.\tThe proposed method’s capability mainly stems from weights2space. However, the training of weights2space relies solely on 300K samples from ShapeNeRF-Text and ObjaNeRF-Text, which may limit its generalizability in real-world applications.\n2.\tIt seems the method is limited to handling single-object NeRFs. It's unclear how it would scale to scenarios involving multiple objects in the same scene. For example, how would the model handle questions that require reasoning about spatial relationships between multiple objects? Addressing this limitation would be important for broader applicability.\n3.\tLacking some ablation studies—for instance, the number of queries in weights2space.", "was_revised": true, "final_justification": "Thanks for your thoughtful response, which addresses most of my concerns. Although the authors responded to the question about generalization, I believe that using a real scene from Objaverse-NeRF-Text does not provide strong evidence, since the model has already seen data from this domain during training. Hence, I keep my score.", "rebuttal": "## 1. The proposed method’s capability mainly stems from weights2space. However, the training of weights2space relies solely on 300K samples from ShapeNeRF-Text and ObjaNeRF-Text, which may limit its generalizability in real-world applications.\n\nWhile the training set comprises 300K samples, we highlight that ObjaNeRF-Text includes a wide variety of synthetic and real objects and scenes, the latter featuring multiple objects. This diversity makes the test scenarios on ObjaNeRF-Text (including Spatial ObjaNeRF) a good first-level evaluation benchmark for generalization. As shown in Table 2 and Table 5 of the paper, S-LLaNA performance on the test set of ObjaNeRF-Text is superior to that of other MLLMs, highlighting the effectiveness of our approach on unseen NeRFs.\n\nMoreover, to qualitatively assess the capability of S-LLaNA in handling __real-world data__, we also report here a result on a real scene from ObjaNeRF-Text:\n\n--- \n__Objaverse ID__\nba341c4ce89647ea9f6996ec58e3eacf \n\n__Spatial detailed description, ground truth:__\n_This is a 3D scene with two tractors, a loader, and a trailer parked on a plot of land with plenty of wood waiting to be loaded. The tractors are positioned side by side, with the loader attached to one of them. The trailer is parked in front of the tractors, and the wood is stacked neatly beside the tractors. The ground appears to be a mix of dirt and patches of grass, and there are no visible signs of recent activity, suggesting the scene might be a still moment in time._\n\n__Spatial detailed description, prediction:__\n_The 3D model represents a typical farming scene. It features two tractors, one of which is a large, robust machine with a visible driver's cabin, indicating it's likely used for heavy-duty tasks. The other tractor is smaller, possibly a compact model, which might be used for lighter tasks or for navigating narrower spaces. Both tractors are parked on a plot of land, suggesting they are either in a state of rest or not currently in use. There is also a loader, a versatile piece of equipment used for lifting and moving materials, which implies that the land might be a construction or building site._\n\n--- \n\nThe example above shows that S-LLaNA can produce accurate descriptions of real-world scenes. Notably, it can properly count objects and understand their relative positions: _it features two tractors_ ... _both tractors are parked on a plot of land_ ... _there is also a loader_.\n\nFurthermore, to validate the applicability of S-LLaNA to in-the-wild NeRFs, we also test it on another dataset containing real objects, OmniObject3D. To conduct this experiment, we fit NeRFs for a few objects employing the images officially released with the dataset. These NeRFs are provided as input to our framework. Below, we show some qualitative results:\n\n---\n\n__OmniObject3D ID:__\n_house_004_\n\n__Spatial detailed description, ground truth:__\n_The house has a main building and an annex with doors on the left and right sides, a man standing in front of the main door and a coconut tree behind it, the main part of the house is mainly brown in color and the eaves are yellow, pointed, triangular, and not symmetrical._\n\n__Spatial detailed description, prediction:__\n_This is a 3D model of a cartoon-style building, predominantly brown in color, with a distinctive pink facade. A large coconut tree is behind the building. The structure appears to be a whimsical representation of a building, possibly from a children's animated show or game. The yellow roof adds a playful touch, making the building appear less realistic and more imaginative._\n\n---\n\n__OmniObject3D ID:__ _toy_train_044_\n\n__Ground Truth:__\n_This toy train is overall green, the front is a cartoon face head, the back is dark green and with a small bump, the rear part is green with gold stripes, there is also a small bump, there are some irregular patterns on the side, the front and rear each have a bump, a total of six wheels, four small, two large, the overall structure of the axis of symmetry._\n\n__Prediction:__\n_This is a 3D model of a cartoon-style, green-colored train. At the front of the train, a cartoon-style head is visible. Its vibrant green color makes it eye-catching. This type of object could be used in various applications such as animation, video game design, or digital art due to its playful and modern aesthetic._\n\n---\n\nFrom the examples above, we can appreciate how S-LLaNA generalizes effectively to real-world NeRFs belonging to a dataset __never seen during training__. For instance, it correctly describes relative positioning (_A large coconut tree is behind the building_) and captures fine-grained details (_at the front of the train, a cartoon-style head_).\n\nTo better visualize these results, please refer to section __\"Note to visualize example models\"__ at the end of __response 2 of reviewer GQGs__.\n\n## 2. It seems the method is limited to handling single-object NeRFs. It's unclear how it would scale to scenarios involving multiple objects in the same scene...\n\nWe would like to clarify that ObjaNeRF-Text and Spatial ObjaNeRF already contain a substantial number of scenes featuring multiple objects. Moreover, Spatial ObjaNeRF provides spatially grounded descriptions for many of these scenes, including references to inter-object relationships, making it a suitable benchmark for evaluating the ability of the model to reason about spatial relations. Qualitative examples of such multi-object scenes can be found in Figure 2 of the supplementary material (i.e., first row: table, fruit bowl, and book; fourth row: telephone booth, wall, pavement, street lamp, and octopus)\n\nMoreover, we include additional results below on multi-object scenes from our datasets:\n\n__Objaverse ID__\n0fa42f5b83084f0eb32533b760c8d146 \n\n__Spatial detailed description, ground truth:__\n_This is the 3D model of scene with a man, two sheep and a sheepdog. The man is standing between the sheep and the sheepdog is standing in front of them. The man is wearing green trousers, a yellow jacket and a blue cap._\n\n__Spatial detailed description, prediction:__\n_The 3D model depicts a scene from a farm. A person, clad in a blue hat, light yellow coat, and brown pants with a green lining, is standing next to two white sheep and in front of a sheepdog. The scene suggests a tranquil, rural life setup. The attire of the person suggests a casual, possibly outdoor setting._\n\n---\n\n__Objaverse ID__\n998cccab49ed496586994d9f4774daa0\n\n__Spatial detailed description, ground truth:__\n_The 3D model represents a stack of three antique brown suitcases. The suitcases are positioned one on top of the other, with the middle one being the largest. Each suitcase has a rectangular shape with rounded corners, and they are secured with metal rivets. The suitcases have a textured surface that appears to be worn, suggesting age and use. The handles of the suitcases are visible, and they are attached to the body of the suitcases. In particular, the suitcases at the top and at the bottom of the stack have a single handle in the middle of their front side. The suitcase in the middle also has a couple of leather straps and the handle is between them._\n\n__Spatial detailed description, prediction:__\n_The 3D model represents a stack of three antique brown suitcases. Each suitcase is of the same design, exhibiting a vintage aesthetic. The suitcase in the middle is larger than the others. The suitcases are vertically aligned, with the topmost one tilted slightly forward._\n\n---\n\nThese examples show that S-LLaNA can understand scenes featuring multiple objects and reason about spatial relationships between them. For instance, S-LLaNA can describe various spatial properties such as the relative position of objects and their number in the first example (e.g., _a person_ ... _is standing next to two white sheep and in front of a sheepdog_) and in the second example (e.g., _a stack of three antique brown suitcases_ ... _The suitcases are vertically aligned, with the topmost one tilted slightly forward_). Notably, we appreciate how S-LLaNA can also capture relative object dimensions (e.g., _The suitcase in the middle is larger than the others_).\n\n## 3. Lacking some ablation studies — for instance, the number of queries in weights2space.\n\nWe conduct ablations to analyze sensitivity to the number of queries N (and consequently tri-plane resolution) and token dimensionality C, reporting results in the table below:\n\n|Model|Tri-plane resolution|Token dimensionality|SN-T brief||ON-T brief||SN-T detailed||SN-T QA||Spatial ObjaNeRF||\n|:---:|:---:|:---:|:---:|:---:|:---:|:---:|:---:|:---:|:---:|:---:|:---:|:---:|\n||||S-BERT|SimCSE|S-BERT|SimCSE|S-BERT|SimCSE|S-BERT|SimCSE|S-BERT|SimCSE|\n|S-LLaNA-13b|32|516|78.98|79.98|45.44|46.18|80.98|82.90|83.60|84.17|78.25|80.58|\n|S-LLaNA-13b|16|516|76.65|77.68|41.77|42.11|76.74|78.69|83.36|83.83|75.48|77.26|\n|S-LLaNA-13b|32|252|76.77|76.33|42.25|42.36|76.47|78.07|83.27|83.49|76.62|78.43|\n\nEach column reports results on various tasks and datasets: ShapeNeRF-Text (SN-T) brief and detailed captioning and QA, ObjaNeRF-Text (ON-T) brief captioning, and Spatial ObjaNeRF. \n\nWe observe that reducing the tri-plane resolution or token dimensionality leads to a marginal performance drop (∼2% across all metrics). This behaviour suggests that models with higher capacity (i.e., higher resolution and dimensionality) can encode richer and more fine-grained features from the input NeRF. However, the relatively small drop also indicates that S-LLaNA remains robust and effective even with more compact and computationally efficient representations.\n\nMoreover, we point out that in Sec. 6 (supplementary), we report an additional ablation study of the modules of the weights2space meta-encoder proposed in this paper. This experiment clearly shows that the weights2seq module alone performs poorly compared to the full architecture."}, {"review_id": "YTLpBHiumn", "reviewer": "Reviewer_Vie1", "summary": "The paper proposes an extension to Neural Radiance Fields (NeRFs), which are neural networks - typically multilayer perceptrons (MLPs) - that represent the geometry and appearance of visual objects. NeRFs represent a global representation of an input object and do not capture fine-grained details making them according to the authors unsuitable for spatial reasoning. The paper proposes weights2space, a self-supervised framework for encoding a sequence of spatial tokens directly from the weights of a NerRF. The paper also proposes a Spatial LLaNA , a multimodal large language model capable of understanding details and spatial relationships in which the objects are involved leveraging the NerRF representations of the objects. The paper also proposes a manually annotated dataset explicitly designed to evaluate spatial reasoning.", "questions": "See weaknesses: \n- Content should be added with regard to the function of the seq2space module,  detailed ablation of the difference steps/modules of the spatially-aware NeRF weights tokenization,  and evaluation of  the spatially-aware multimodal large language model on a broader range of downstream tasks and datasets.", "limitations": "Limitations could be added regarding the chosen architectures and scalability.", "rating": 3, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 2, "significance": 3, "originality": 2, "strengths_and_weaknesses": "Strengths\n\n-\tThe related work section is extensive and well documented with the literature. \n-\tExtensive experimental results.\n\n\nWeaknesses\n\n-\tThe computation of a spatially aware representation of an object directly from the weights of its NeRF has resemblance to the nf2vec except for a pooling operation performed by nf2vec.\n-\tThe function of the seq2space module needs to be clarified.\n-\tA detailed ablation of the difference steps/modules of the spatially-aware NeRF weights tokenization is missing. \n-\tThe spatially-aware multimodal large language model should be evaluated on a broader range of downstream tasks and datasets beyond those requiring spatial reasoning, in order to ensure that its performance on other tasks is not degraded. As a foundation model, it is intended for general-purpose use.\n-\tThe writing of the paper  is not always very clear and precise. \n-\tThe claims on significant improvements compared to state-of-the-art results need to be supported by statical significance tests.", "was_revised": true, "final_justification": "Most of the issues that I raised were answered in a satisfactory way. I have added my final rating and have raised my score. I value the good results of the paper but I worry about the imprecise writing style, that should be improved in a final version.", "rebuttal": "## Summary. The paper proposes an extension to Neural Radiance Fields (NeRFs), which are neural networks - typically multilayer perceptrons (MLPs) - that represent the geometry and appearance of visual objects. NeRFs represent a global representation of an input object and do not capture fine-grained details making them according to the authors unsuitable for spatial reasoning. The paper proposes weights2space, a self-supervised framework for encoding a sequence of spatial tokens directly from the weights of a NerRF. The paper also proposes a Spatial LLaNA , a multimodal large language model capable of understanding details and spatial relationships in which the objects are involved leveraging the NerRF representations of the objects. The paper also proposes a manually annotated dataset explicitly designed to evaluate spatial reasoning.\nFirst of all, we would like to clarify that our paper is not an extension to Neural Radiance Fields, i.e., we do not propose a novel NeRF architecture that may better capture fine-grained details. Instead, we propose S-LLaNA: an AI assistant to perform language tasks on 3D data represented as standard NeRFs. A key ability of language tasks on 3D data deals with understanding the spatial relations between objects and their parts. Purposely, S-LLaNA includes a novel meta-encoder design, weights2space, that computes a sequence of spatially-aware tokens from the weights of the input NeRF. \n\n## Weakness 1. The computation of a spatially aware representation of an object directly from the weights of its NeRF has resemblance to the nf2vec except for a pooling operation performed by nf2vec.\n\nWe clarify the difference between our novel meta-encoder, weights2space, and nf2vec using Fig. 2 of the paper. The weights2seq module is the nf2vec model without its final pooling layer. However, the key difference between weights2space and nf2vec does not pertain the absence/presence of the pooling layer in weights2seq, but, instead, in weights2space including as additional components the seq2space module and the downstream tri-plane representation (see Fig.2). These two components endow weights2seq with its peculiar ability to elicit spatial information from the weights of the input NeRF and obtain a sequence of spatially-aware tokens to be fed to the LLM. Hence, unlike nf2vec, our proposed weights2space meta-encoder is explicitly designed to extract spatially grounded features from NeRF weights and produce a sequence of tokens that are strongly and explicitly correlated with object parts in the input scene.\n\n## Weakness 2. The function of the seq2space module needs to be clarified.\nThe function of the seq2space module is to work in conjunction with the tri-plane representation to convert the sequence coming from weights2seq, where spatial information is distributed across the tokens, into a sequence to be fed to the LLM, where each token corresponds to a specific spatial location. More precisely, seq2space is a transformer decoder that, by cross-attending to the output from weights2seq, produces a sequence of tokens that, at training time, is reshaped into the features associated with the locations of the tri-plane to enable rendering images from the input NeRF. The intuition behind this design is that, by cross-attending to all the tokens from weights2space, each learnable query of seq2space, which corresponds to a specific position in the tri-plane, can extract distributed spatial cues and rearrange them into spatially-localized features. Besides, the self-attention layers in seq2space realize interactions among the spatially-aware tokens that enable the capture of the relationships between the objects and object parts present in the input NeRF.\n\n##  Weakness 3. A detailed ablation of the difference steps/modules of the spatially-aware NeRF weights tokenization is missing.\nIn Section 6 of the Supplementary Material, we provide an ablation of the modules of weights2space. In particular, *LLaNA-13b-multitoken* uses the original nf2vec encoder without its pooling layer. As previously highlighted, this setup corresponds to directly interfacing weights2seq to the LLM, without leveraging the key original components in our design, namely seq2space and the tri-plane representation. The results show clearly that weights2seq alone performs poorly compared to the meta-encoder design proposed in this paper. \n\nMoreover, we conduct ablations to analyze sensitivity to the number of queries N (and consequently tri-plane resolution) and token dimensionality C. The results of these experiments are reported in our response to reviewer GQGs.\n\n##  Weakness 4. The spatially-aware multimodal large language model should be evaluated on a broader range of downstream tasks and datasets beyond those requiring spatial reasoning, in order to ensure that its performance on other tasks is not degraded. As a foundation model, it is intended for general-purpose use.\n\nWhile our primary focus is spatial reasoning, we have explicitly included downstream tasks that go beyond spatial understanding to assess whether S-LLaNA retains general reasoning capabilities.\n\nFirst, the brief captioning task on ShapeNeRF-Text, HST, and ObjaNeRF-Text involves generating concise textual descriptions focused on identifying the main subject of a scene. This task primarily evaluates global scene understanding, rather than spatial reasoning. In this setting, S-LLaNA consistently outperforms LLaNA, demonstrating its ability to preserve a holistic understanding of the input NeRF.\n\nFurthermore, we evaluated S-LLaNA on a zero-shot NeRF classification task following the protocol in [6], where the model must infer the object class based on its multimodal input. This task does not require fine-grained spatial reasoning. As shown below, S-LLaNA outperforms LLaNA: \n\n|Model|Modality|Accuracy (%)|\n|:---:|:---:|:---:|\n|LLaNA-7b|NeRF|67.56|\n|S-LLaNA-7b|NeRF|68.67|\n||\n|LLaNA-13b|NeRF|69.27|\n|S-LLaNA-13b|NeRF|71.85|\n\nThese results confirm that S-LLaNA does not sacrifice holistic visual or linguistic understanding.\n\nRegarding the general language understanding and reasoning of the underlying LLM (LLaMA-2), both LLaNA and S-LLaNA incorporate LLM finetuning in their training protocol to adapt the model to our scenario.\nWhile in theory this finetuning stage could degrade performance on other general language tasks, we note that S-LLaNA correctly answers in the single-round QA task on ShapeNeRF-Text, which includes questions that tap into commonsense reasoning and real-world knowledge (as explained in [7]). Below are two representative examples showing how S-LLaNA provides answers consistent with human-level commonsense:\n\n---\n__Example 1 (Phone)__\n\n__ShapeNet ID__\n57056c8f465dd1aec03bc4569d70377c\n\n__Question:__\n_What is the purpose of the flash on the right side of the phone?_\n\n__Ground-truth:__\n_The flash is used to enhance camera performance in low-light conditions._\n\n__Prediction:__\nThe flash is used to illuminate the subject in low-light conditions, providing a better quality image.\n\n--- \n__Example 2 (car)__\n\n__ShapeNet ID__\n431ca41fdf0897c628ccbb4eb8965b05\n\n__Question:__\nCan the blue car be used for daily driving?\n\n__Ground-truth:__\nYes, the blue car is a modern hatchback, making it suitable for daily driving.\n\n__Prediction:__\nYes, the blue car's compact design and modern features make it suitable for daily driving.\n\n---\n\nFinally, our ablation study in Section 6 (main paper) demonstrates that S-LLaNA performs robustly even without this step, achieving results comparable to the finetuned variant. Consequently, in this version of our framework, the pre-trained LLM remains untouched, and thus certainly preserves its original reasoning capabilities and world knowledge.\n\n##  Weakness 5. The writing of the paper is not always very clear and precise.\nAs pointed out by reviewer 1Y9M in response to weakness 1, we will revise Section 4 and update captions of Figures 2 and 3 to enhance readability.\n\n##  Weakness 6. The claims on significant improvements compared to state-of-the-art results need to be supported by statistical significance tests.\n\nWe acknowledge that statistical significance testing would offer a more rigorous foundation for comparing performance. \n\nBased on our internal experiments, we observed that our framework achieves consistent performance across multiple independent training runs. We are currently running additional training of our framework to quantify the consistency of our performance, yet, due to the short rebuttal period, we will be able to report statistics of our results in the camera-ready version of the paper. \n\nNevertheless, we would like to highlight that the methods we benchmark against do not provide results from multiple runs, likely due to the substantial computational cost of training and evaluating multimodal large language models (MLLMs). Consequently, meaningful significance testing would still be infeasible due to the absence of comparable data from other approaches."}]

Author rebuttal:

Meta-review: The authors propose a method of tokenising Nerfs so that they can effectively become a new input modality for MLLMs. In contrast to prior work (LLana), the authors' Nerf tokenisation is location-aware. Unlike Llana which just directly encodes the Nerf weights as tokens, the proposed tokens are location aware by projecting to a tri-plane representation of xy, yz and xz coordinates.
To verify the location awareness, the authors also propose a new dataset, Spatial ObjaNeRF, for spatial reasoning tasks. The authors show strong performance on Nerf-captioning and QA tasks.

Reviewers appreciated the strong empirical results, motivation of the approach and evaluation dataset proposed by the authors. The authors posted convincing rebuttals to the reviewers, addressing most of their concerns during the review period, by adding additional ablation studies, adding multiple choice questions to the dataset and several clarification questions by the reviewers.

The sole negative-leaning reviewer (Vie1) mentioned the quality of the writing as the main limitation of the paper. However, after some deliberation, the AC and SAC found the writing quality to be sufficient and believe that the remaining issues can be fixed by addressing the reviewers' comments for the camera-ready version. Therefore, on the balance, the final decision is to accept the paper. Please incorporate all updates from the rebuttal into the camera-ready.
Decision: Accept (poster)