LLaNA: Large Language and NeRF Assistant
Authors: Andrea Amaduzzi
Pierluigi Zama Ramirez
Giuseppe Lisanti
Samuele Salti
Luigi di Stefano
Venue: NeurIPS 2024 poster (2024)
Keywords: LLM
NeRF
VQA
Primary area: machine_vision
TL;DR: We propose LLaNA, che first Multimodal Large Language Model (MLLM) able to perform NeRF-language tasks, such as NeRF captioning and NeRF QA.

Abstract: Multimodal Large Language Models (MLLMs) have demonstrated an excellent understanding of images and 3D data. However, both modalities have shortcomings in holistically capturing the appearance and geometry of objects. Meanwhile, Neural Radiance Fields (NeRFs), which encode information within the weights of a simple Multi-Layer Perceptron (MLP), have emerged as an increasingly widespread modality that simultaneously encodes the geometry and photorealistic appearance of objects. This paper investigates the feasibility and effectiveness of ingesting NeRF into MLLM. We create LLaNA, the first general-purpose NeRF-language
assistant capable of performing new tasks such as NeRF captioning and Q&A. Notably, our method directly processes the weights of the NeRF’s MLP to extract information about the represented objects without the need to render images or materialize 3D data structures. Moreover, we build a dataset of NeRFs with text annotations for various NeRF-language tasks with no human intervention.
Based on this dataset, we develop a benchmark to evaluate the NeRF understanding capability of our method. Results show that processing NeRF weights performs favourably against extracting 2D or 3D representations from NeRFs.

LLaNA: Large Language and NeRF Assistant
Andrea Amaduzzi andrea. wamaduzzi4@uniiii it
Pierluigi Zama Ramirez it
Giuseppe Lisanti giuseppe. www.ntiiuniio..........i......im
Samuele Salti samuele salti@unibo.is
Luigi Di Stefano
iiigi.distefanoeee it
CVLAB, University of Bologna
https://andrreamduuzii github. io/llana/
Abstract
Multimodal Large Language Models (MLLMs) have demonstrated an excellent understanding of images and 3D data. However, both modalities have shortcomings in holistically capturing the appearance and geometry of objects. Mcanwhile, Neural Radiance Fields (NeRFs), which encode information within the weights of a simple Multi-Layer Perceptron (MLP), have emerged as an increasingly widespread modality that simultaneously encodes the geometry and photorealistic appearance of objects. This paper investigates the feasibility and effectiveness of ingesting NeRF into MLLM. We create LLaNA, the first general purposs NeRF- language assistant capable of performing new tasks such as NeRF captioning and Q&A. Notably, our method directly processes the weights of the NeRF's MLP to extract information about the represented objects without the need to render images or materialize 3D data structures. Moreover, we build a dataset of NeRFs with text annotations for various NeRF-language tasks with human intervention. Based on this dataset, we develop benchmark to evaluate the NeRF understanding capability of our method. Results show that processing NeRF weights performs favourably against extracting 2D or 3D representations from NeRFs.
Introduction
Large Language Models (LLMs) 70 I have revolutionized the field of Natural Language Processing, demonstrating incredible text comprehension and generation capabilities. These results have fostered the development of Multimodal LLMs (MLLMs) [4 modalities such as images, videos, and audio, generate text describing and reasoning about the content of such modalities. Recently, MLLMs have also been extended to 3D data 78),,), primarily represented through colored point clouds, yielding remarkable results even this scenario.
82 44 [1 10], which can ingest various
Beyond images and 3D data, another paradigm is emerging to represent objects and scenes: Neural Radiance Fields (NeRFs) [50]. NeRFs coordinate-haan hasedd neural networks, typically Multi-Laayr Perceptrons (MLPs), designed to capture both the geometry and the photorcalistic appearance of an object by learning continuous radiance field each 3D spatial location. After training. a NeRF model can be queried to render realistic images or to reconstruct the 3D surface of the encoded object. Therefore, capturing an object as NeRF provides an interesting alternative create a digital twin
38th Conference on Neural Information Processing Systems (NcurIPS 2024).

---

NeRF captioning
NeRF Question Answering
Male
Helt
what the color schee clew inattr des dfow
cipal
repltton tie retred offereteree perdee
The color scheme the meror desig la de% powitry aocor
ford singfttie bank, Ih buturs, ciraleii dengg
word apourien ous cust bo coler purtering mvral briggs nor
unet?
NORF Classification
dent ta and aqgll vida for par
Rhw No cimn
Weite Biikiig
The roof the kes seder fature surnect andingg
Figure I: LLaNA. The first Multimodal Large Language Model that understands and reasons on an input NeRF. Our framework directly processes the NeRF weights and performs tasks such as captioning, Q&A, and zero-shot classification of NeRFs.
with respect to standard representations such as multi-view images or point clouds. For instance, thanks to its continuous formulation, from single NeRF, one can generate an infinite number of photorcalistic images at any resolution while storing only the weights of an MLP instead of the entire image set. See Appendix A.4 for more on the memory advantages of using NeRFs. Due to their advantages, NeRFs are effectively becoming a new modality stored and communicated independently, with datasets of NeRFs being made publicly available (25 61 and companies providing digital twins of objects represented as NeRFs (e.g..
ai/).
The increasing adoption of NeRFs and their appealing characteristics prompted us to the following research question: possible to build an MILM able to ingest directly NeRFs? Inspired by recent studies on meta-networks that can process neural fields [8] 42), we answer this question in the positive by showing that it is possible to process the weights of a given NeRF with a meta-network encoder that projects the NeRF weights into the embedding space of pre-trained LLM such as LLaMA 2 [70]. By doing so, we create the first MLLM for NeRFs, dubbed Large Language and NeRF Assistant (LLaNA), which can solve NeRF-language tasks such as NeRF captioning, Q&A and zero-shot NeRF classification (see Fig. D.
We also introduce new NeRF- languaae dataset, that we will make publicly available, to train LLaNA and test the capabilities of our assistant. To collect this dataset, we designed an automated annotation framework that leverages MLLMs to produce text annotations for NeRFs trained on Shapenet g Using this dataset alongside an additional split containing manually curated textual descriptions we establish benchmark for NeRF textual assistants.
Since a straightforward way to create an assistant for NeRFs would be to render images or extract 3D point clouds out of it and provide them as input to existing MLLMs specifically designed to handle such modalities, we thoroughly compare LLaNA against these baselines on the proposed benchmark. We show how the resolution of the extracted 3D geometry or images, and for images also the vantage point used for rendering, negatively impact the quality of the MLLM's output. Important details might be lost by rendering from the wrong angle, or the extracted geometry might not be detailed enough. Vice versa, by operating directly on the MLP weights, we extract all the information they hold about the object without any other design decision. Our approach turns out to be the most effective way to create NeRF assistant as consistently outperforms MI.LMs processing images or 3D geometries extracted by querying NeRFs Our contributions can be summarized as follows:
LLaNA, the first MLLM capable of performing tasks such as captioning and Q&A on NeRFs.
We show that is possible to build such an assistant by directly processing the NeRFs weights with meta-encoder, which is faster and captures more information than rendering images extracting 3D data.
We automatically create a NeRF NeRF-language benchmark based on ShapeNet, and we thoroughly evaluate I.LNNA on it, showing that performs better than applying popular MI.LMs on discrete representations obtained from NeRFs.
2

---

2 Related work
Multimodal Large Language Models (MLLMs). Significant advancements have been made by Large Language Models (LLMs) in language understanding, reasoning, and generalization capabilities [62
60]. These models have been extended into Multimodal Large Language Models Ms)Ms), which broaden their reasoning abilities by including other modalities
1 54
70
75
MILMS generally align target features with textual ones and then integrate them into LLMs for various text inference tasks. Some MLLMs
like images 14
82
19 audio (26, and videos 47 10
are trained entirely from scratch [27 [6], others utilize pretrained LLMs [37 4
44
38 I 3D
MLLMs focus understanding the 3D world typically represented as colored point clouds [58]
24
86 20 78 or multi-view images [23]. Some of these models are trained using 2D images [4
86 23
while others directly align textual phrases with points 20 78 58
Neural radiance fields. NeRF 150 have been applied in several visual tasks such as novel view
synthesis [48], generative media 557 and robotics [79]. The base formulation employs MLPs to convert spatial coordinates into colors and densities. Recent advancements substitute enhance for faster training and inference.
MLPs with explicit data structures 9 68
16
52
Neural radiance fields and language. The interaction between NeRF and language has been recently investigated for several practical applications. Many works address the problem of generating
geometrically consistent views of objects or scenes described by textual prompts (66 49
31 65
36 57. Other approaches focus on editing the scene represented by NeRF from text, e.g., by
changing the appearance and shape of objects [73 28
67 74 69] 21 80, 87]. or by inserting/removing
objects in the scene 3 []] Some techniques investigate new types of radiance fields that predict language features for each spatial location alongside density and color [32 (4]. By distilling knowledge from vision-language models into these models, the neural fields can be queried by textual prompts. LERF [32] extends the original radiance field formulation, considering functions which model density, color and language features at each spatial coordinate. Such language fields are parametrized by neural network. Unlike all previous methods, Ballerini et al. [5] is the first to utilize the weights of a NeRF's MLP as an input modality. They aim to learn mapping between the NeRF and CLIP [59 embedding spaces to perform tasks such as NeRF retrieval from textual or image queries. Differently, our goal is develop an MLLM capable of reasoning about NeRFs.
Deep learning on neural networks. Several studies have explored using meta-networks, i.e. neural networks that analyze other neural networks. Initially, researchers concentrated on predicting network characteristics, such as accuracy and hyperparameters, by processing their weights [7] 64 30
Several recent works focus on processing networks implicitly representing data (Implicit Neural Representations or Neural Fields). These methods perform tasks such as classifying or segmenting the data by processing solely the weights of the input neural networks. Among these works, Functa [IS] trains a shared network on the entire dataset and then learns a compact embedding for each sample for downstream tasks. Later works concentrate data samples, e.g., a specific object. By leveraging a novel encoder architecture for MLP weights, inr2vec [12] extracts compact embeddings from INRs of 3D shapes, which are employed inputs for downstream tasks. nf2vec [61 extends 2nrevec to ingest the NeRF's network weights to classify, segment, retrieve similar NeRFs. Cardace et al. ( develop a strategy to process neural fields represented by a hybrid tri-plane structure. Other approaches
processing networks representing individual
[53] 84
83 85
develop equivariant
architectures to handle MLPs by exploiting weight space symmetries 22 an inductive bias. Also, Graph Neural Networks have been investigated to compute network representation [35] 42). Since we aim process NeRFs directly from the network weights, we employ 2fvv as our meta-encoder due to its efficient and scalable architecture.
Methodology
This section describes the proposed Large Language and NeRF Assistant (LLaNA)). We provide an overview of NeRFs and the meta-encoder that maps NeRF weights into global embedding. Then, present the overall LLaNA framework and discuss training protocol.
3

---

NeRF
0000
Dokomber
LSBR sarrr
Pool
Projector
1/ke red sports ene. mit uteel Notter
austo
Tokenizer
W
bm
b:
Figure 2: Framework overview. Example of NeRF captioning.
Neural Radiance Fields (NeRF) Neural Radiance Field (NeRF) (0 is a framework that employs coordinate-basee neural networks, typically MultiLayer Perceptrons (MLP) and is trained on collection of images of an object or scene taken from various vantage points. The main application of NeRFs is the task of novel views synthesis, i.c. photorealistic rendering of images from viewpoints unseen at training time. In its base formulation, the MLP is function of continuous 3D coordinates p= (x. y, 2) IR3, that yields four-dimensional outputs, RG E [0, 1 This output encodes the RGB color and the volume density of each 3D location in the scene. The volume density a can be interpreted as the differential probability of a ray terminating at point p. After training, a NeRF can render images from any desired viewpoint at arbitrary resolution by querying it for the values of RGB and a at several points along the ray corresponding to each pixel and applying the volumetric rendering equation 550
In this work, we realize NeRFs as MLPs composed of hidden layers, an input layer, and an output layer. An example of MLP with input, output, and I hidden layer is shown in Fig, 2 (left). A layer is parameterized by weight matrix plus bias vector. More in detail, the hidden layers in our architecture have the same number of input and output neurons, H, thus having squared weight matrices W, E TR.H H for I I....: and H di-meensional biases by E R As input p goes through a 24-freqe encoding [50], the first layer has W in E R 1444x and b,n E R The final one has W out E R and bost E R2, Refer to Appendix A for more details on NeRFs.
Meta-encoder In this work, we explore how a NeRF assistant can be realized by processing the NeRF weights directly. We expect the NeRF weights to contain comprehensive information about the represented object, such as its geometry and appearance. Thus, an encoder processing them might extract all the necessary information for downstream language tasks such as captioning and Q&A.
Inspired by the recent development of meta-network capable of processing neural fields 42 81). we employ as our meta-encoder architecture nf2vec (8] takes as input the weights of NeRF and yields global embedding that distills the content of the input NeRF. In particular, the weight matrices and biases of the input NeRF are stacked along the row dimension where the number of rows S depends on the number of hidden layers J., the number of units per hidden layer H, and the dimension of the input, which is a 144-dimensional array obtained by frequency encoding of the 3D coordinates. Before stacking, we pad the output layer weights W out and biases bout with zeros to obtain H columns (see Fig. center).
form matrix M Sx
The meta-encoder is parametrized as an MLP with batch normalization layers 29 and ReLU non- linearities. To scale gracefully with the input MLP dimensions, the encoder processes each row of M independently. extracting total of S tokens, each of length G. from an input NeRF. They are then max- moo-pp obtain global representation g E HC of the NeRF, with G 1024 in our experiments. The encoder is pre-trained using the self-training protocol of nf2vec [8]] i.e., jointly with a decoder architecture that, given as input the NeRF global embedding, reconstructs the same images as the input NeRF from arbitrary viewpoints. More details in Appendix
Large language and NeRF assistant Inspired by recent approaches that created effective Mul- timodal Large Language Models, we build LLaNA by leveraging on pre-trained LLM with transformer backbone [72], in our experiments I.L.MA (70), and injecting the NeRF modality into its embedding input space, proposed for images and 3D data [4 78 (see Fig. right). Thanks to

---

the self-attention mechanism, the transformer can understand the contextual relationships between text and NeRF tokens, enabling to generate responses based on both text and NeRF inputs.
We employ a trainable linear projection layer, Φ, to project the embedding of the input NeRF computed by the meta-encoder into the LLaMA 2 embedding space. The projection layer has weights W Wpros E _ where 1' is the word embedding dimension of the employed LLaMA model. This embedding is encapsulated between two special tokens, whose embeddings are learned end-to-end while training, namely <n_start> and <n_end>.
Then, given an input sequence of mixed NeRF and word tokens, (<n_ hon_star>>,co <n (0))<n___dd,,,_ W1, W2, .... Wx), where is the number of word tokens, the large language model returns sequence of predicted word tokens (1x++1, Wk+2 Weos).
Training protocol Our framework is trained on the ShapeNeRF-Tex dataset, described in detail in Sec. This dataset is organized into set of prompts from the user and expected ground-trute answers that are used to optimize the original auto-regressive objective of the LLM. For the meta-encoder, we employ the nf2vec encoder pre-trained on ShapeNet released by the authors [8I], and we keep it frozen during training. We follow the two-stage training protocol delineated in Liu et al. (44).
Stagel: projector training. In the first stage, we train the projector network ø to align the NeRF and the word embedding spaces while keeping the LLM weights fixed. We train on an instruction dataset of brief descriptions to learn the projection layer efficiently. We also train the embeddings of the special tokens used to encapsulate the NeRF one. We optimize the projector weights and the embeddings for 3 epochs with a learning rate of 0.002 and batch size of 64.
Stage2: instruction tuning. During the second stage, we train on complex instructions to help the model understand and reason about NeRF data. In this phase, we optimize both the projector and the LLM for epochs on the detailed descriptions, single-round and multi-round Q&A conversations available in our dataset. For this phase, we employ a learning rate of 0.0002 and a batch size of 16.
Our model is implemented in Py and trained on 4 NVIDIA A100 with 64GB of VRAM each. Completing both stages requires 1 day of training.
4 Benchmark
4.1 ShapeNeRF-Tex dataset
To train and validate our NeRF assistant, we automatically created a dataset of conversations about NeRFs, the ShapeNeRF-Text dataset.
features paired NeRFs and language annotations for ShapeNet objects (8 in particular for all the 40K NeRFs available in the nf2vec dataset [61]. We followed the struc- ture defined in PointLLM (78) to create the textual annotations. More in detail, for each object, we generated a brief descrip- tion, detailed description, 3 single-round Q&As, and one multi-round Q&A. The brief descriptions are concise captions of the object, taking into account its global structure and ap- pearance. The detailed descriptions are longer sentences that describe all the details of the object. The single-round Q&As consist of a question about the object and the correspond- ing ground-truth answer. Finally, the multi-round Q&As are longer conversations formed by 3 questions and the relative answers. The automatic data annotation pipeline is inspired by Cap3D [46] and is shown in Fig. 3 First, multiple views of each ShapeNet object have been rendered from different perspectives. Then, each view has been provided as input to LLaVA (LLaVA2-13b) 44 to get a detailed description of the object from that point of view. Afterward, starting from the captions generated by LLaVA, ILaMA (LLaMA3-8B-chal was used to generate the final ground-truth text data (brief and
LLaV
Liama
Figure 3: Automatic annotation pipeline. Given a 3D model, N views are rendered and processed by VLM (LLaVA) to generate view- specific captions. These are aggre- gated by an LLM (LLaMA) for final descriptions and Q&A.
5

---

detailed descriptions, single and multi-round Q&As). Both the frozen LLMs employed to create our benchmark (LLaVA2-13b, LLaMA3-8b-chat) are equipped with safeguards.
When building the ground-truth data, to ensure diversity in the language annotations, each brief and detailed description has been associated with a question randomly sampled from 30 instructions for each kind of description. Such instructions, together with the carefully engineered request prompts for LLaVA VA and LLaMA, are reported in Appendix C.I
ShapeNeRT-Text provides 30939, 3846 and 3859 objects for the train, validation and test sets, respectively. Overall, the dataset features 13 object classes, and the train, validation and test splits are obtained by randomly sampling objects within each class, i.e., holding out a fixed percentage of objects per class (80%, 10%, and 10% for the sets, respectively). Appendix provides more dataset statistics. As quantitatively proven in Appendix C.3 and Appendix(D.] many of the questions belonging to the Q&A set require holistic 3D understanding of the object, to be answered correctly.
4.2 Language tasks and metrics
We evaluate NeRF assistants on three different language tasks, given an input NeRF: brief captioning, detailed captioning, and single-round Q&A. We evaluate all tasks on the objects from the ShapeNeRF- Text test set. For brief captioning, we additionally evaluate the methods on the GPT2Shape Human Shape Text (HST) dataset (2) a subset of ShapeNet for which human-curated brief descriptions publicly available. To generate the dialogues for HST, we randomly pair each of its captions with one of the 30 instructions requesting a brief description, used in ShapeNeRE-Text and reported in Appendix C.T We employ standard language similarity metrics to evaluate these methods. We compute the cosine similarity between the global embeddings of the generated and ground-truth sentences provided by the pre-trained encoders Sentence- BERT [63] and SimCSE [18]. These metrics based on learned networks are the most effective at measuring the quality of the generated output. We also include standard handerafted metrics based on n-gram statisties, like BLEU-1 (55), ROUGE-L 43, and METEOR 6
5 Experiment results
5.1 Foundation models as baselines
As our method is the first to investigate language tasks on NeRF, there are no baselines in the literature. However, given a NeRF, straightforward way to create an assistant for it could be to render an image and use an MLLM capable of ingesting images. Alternatively, we could extract the 3D shape from the NeRF and use one of the recent 3D MLLMs. Hence, in a first set of experiments, we use MLLMs as off-the-sheit foundation models, trained on hundreds of thousands of shapes or millions of images, without performing any fine-tuning on the training set of ShapeNeRF- Text, and consider such pipelines as natural baselines. Specifically, we use LLaVA (v1.6) 44 and BL.I--2 (39 for images, as well as PointIIMM (78) and GPT4Point 58 for colored point clouds. Since NeRFs can render arbitrary viewpoints after training, we also include the evaluation of L.L.V 44 in a
multi-view scenario. More in detail, we render images from N viewpoints randomly sampled between the set of camera poses used to train each NeRF; then, we concatenate tokens from these N images and fed them into LLaVA alongside text instructions. We set N=3 because the model cannot process a higher number of images correctly. In addition, we test 3D-LLM 24] to compare its performance to LLaNA. We employ the official code and pre- traitrd models released by the respective authors for such evaluations We note that the only official GPT4Point weights available at submission time were those obtained from fine-tuning OPT-2.7B performance of all methods under the more realistic scenario where NeRFs are treated as the only input data to the assistant. Hence, images and point clouds can only be extracted from NeRFs. Details on the extraction procedure are provided in Appendix A.3 As for 3D-LLLM, we extract colored 3D) meshes from the NeRFs of ShapeNeRF-Text and process such data with the official 3D-LLM code to render images from multiple views and compute both the 2D and 3D features required by the model
Cap3D [46). In Tabs. to we present the
http://hootaannntuL/LLaaLLa liu/LLaVA BLIP-2: http::///gittub..oom/eeeeeeeeee LAVIS/ tttes/ree////eee/ haaaln//projetssbblppp PoinLLM: https: :/g1 thub com/ http///onnoootla/Pooonttt
'LLaVA:
GPT4Point: http://hhhhhu.cmcomoooooctt///gp UMass Poundattiooo--oooe//---
3D-LLM: https://github. com/

---

Table 1: NeRF brief captioning on ShapeNeRF- Text. Frozen baselines.
Table 2: NeRF brief captioning on the HST dataset. Frozen baselines.
Best results are in bold, runner-up underlined. (FV: fien,-view, BV: back-view, MV: multi View
Best results are bold, runner-up underlined.
(FV: front-view, BV: back-view, MV: multi-view)
Modality
Isege (P4)
61.00
52.10
14.30
21.0
23:31
Irage (T0)
50.52
55.55
0.56
11:31
18.57
ixage
Iraa (MX)
12.76
(MSV)
56.01
53:01
12.84
12:17
23/05
5125
55.55
15:90
Isage (PC)
50.85
$2.85
2161
23,4
23:35
brag: (F5)
$4.31
56.28
1008
14.71
14.33
Image
BLBR Ean Baad
Isea: (P4)
55,13
$3.21
5.45
13.00
3.57
SUR A
(94)
$2.11
20.43
5.31
1912
12.14
Irage (IV)
$3.43
11005
5.0
14:20
4.40
BUP- FartSood
51.11
9.00
11.34
Piezz cloal
40.59
08:30
16:26
12.00
14.56
42.40
14.50
13:64
Pasti
41.85
40.22
11.76
16.54
11.53
GPPiNici-Qge2 78
Prenictua)
43.15
42.32
12.02
18.23
15.99
Medt-MV MV
10
16.03
13.00
14.33
30EM
Med
96.07
1532
MaNN--hh
NeR7
(6.03
78.54
3064
2633
30.26
LENG
N.FF
Table 3: NeRF detailed captioning on ShapeNeRF-Texx Text. Frozen baselines. Best results are in bold, runner-up underlined. (FV: front-view, BV: back -iew, MV: multi-
Model
Modality
S-BERT
SInCSE
BLEU-1
ROUGE-L
METEOR
LLaVA-vicuma 13b
Image (FV)
59.08
58.87
23.63
23.55
22.55
LLaVA-vicuna-1
Image (BV)
50.33
13:77
13.18
LLaVA-vicuma 13b
Image (MV)
60.21
59.51
15.07
32
14.64
LLaVA-vicuma-7h
Image (FV)
57.55
$7.68
14,99
22.82
14:36
LLaVA-vicune-7b
Image (BV)
54.46
BLIP-2 FlanT5-xxl
Image (FV)
41.27
40.69
0.18
7.83
2.60)
RIJI-2 FlanTS-xx]
Image (BV)
38.49
37.89
0.19
7.72
2.58
Poin
cloud
59.02
58.30
10.55
GPT-4Ponn-00e22.76
Point cloud
42.44
38.33
3.72
9.21
5.13
3D-LLM
Mesh MV
60.00
53.91
1,58
14.40
5.28
LLaNA-7b
NeRF
77.43
79.81
41.32
36.18
32.39
at inference time. Moreover, in Appendix E. report the results dealing with the images used to train the NeRF or the original 3D point cloud from ShapeNet, which confirms the methods' ranking. When rendering an image, a non-obvious design decision for the pipeline is from which vantage point to render it. ShapeNet artificially simplifies this task since all objects have been canonically aligned common reference frame, but this may not be the case in a general setting. To show the vantage point's effect on the assistant's 's results, we report results processing a frontal or back view.
5.2 NeRF captioning
We test the assistants' ability to describe the NeRF content in the captioning experiments. We prompt them with the NeRF, or the image/cloud extracted from it, followed by the question which has been paired with its ground-truth description, as detailed in Section 4.2 e.g. "What's the content of this NeRF DeRF//mage/cloud"" We then collect the answers generated by the models and compare them with the ground-truit -ruth description according to the selected metrics.
Brief description. We report results for the brief description tasks on ShapeNeRF-Text and the HST dataset Tab. and Tab. 2 respectively. Comparing LLaNA with the baselines described in Sec.15.] we appreciate how LLaNA achieves the best performance in most metrics, often by large margins against runner-ups. For instance, for the Sentence- BERT similarity on the ShapeNeRF- Text dataset, LLaNA achieves 68.63, 7.63 points more than LLa 133, even if LLaNA uses smaller LLM. Results on the HST dataset, which provides ground-truth descriptions validated by humans, are generally lower for all methods. Yet, LLaNA provides again the best performance according to most metrics. The difference in the quality of the brief description provided by LLaNA compared the baselines is showcased by the qualitative result reported in the first row of Fig. where the description provided by LLaNA is the most accurate.
Tab.
A clear trend in both tables and qualitative results is that image-based models tend to perform better than models processing point clouds. This is likely due the larger amount of data used during training of the modality encoder, i.e. millions of images versus hundreds of thousands of shapes, which enhances their generalization ability, as well as the capability of images to capture more details than point clouds at the input resolutions required by image-based MILMM versus 3D ML.Mss. Nonetheless, our method, which operates on NeRFs, benefits from a holistic view of the object and provides the most accurate descriptions. Remarkably, in LLaNA, all the necessary information for this
7

---

Table 4: NeRF single-round Q&A on ShapeNeRF-Text. Frozen baselines.
Best results are in bold, runner-up is underlined. (FV: front-view, BV: biek, MV: multi-view)
Model
Modality
S-BERT
SImCSF
BLEU--
ROUGE-I
METEOR
HaVA--vccum-118 135
Image (FV)
71.61
70.98
20.19
32.53
LLaVA-vicuna-IB 13b
Image (BV)
68.25
69.06
20.03
29.84
32.27
LLaVA-vicuma-1 133
Image (MV)
71.84
71.16
20.04
30,20
33.46
LLaVA-vicuma-70
Image (FV)
71.79
71.96
25.79
34,04
34.86
LLaVA-vicumal 7b
Image (DV)
70.88
70.93
25.17
33.30
34,22
BLIP-2 FlanTS-xxl
Image (PV)
45.20
47.92
11.50
20.16
13.49
BLIP-2 FlanT5-xxl
Image (BV)
45.06
47.66
11.50
19.98
13.44
LM-7
Point cloud
74.70
74.40
36.81
44,41
39,76
GPT-Peinn-OPt-221 Optl-...b
Point cloud
27.62
31.41
6.26
3D-LLM
Mosh + MV
69.62
67.55
32.19
40.95
35.83
ILaNA-7b
NoRF
81.03
81.56
46.16
53.17
50.15
LLaNA
PointLLM
GPT4Point
LLavA
BLIP-2
Brief Description
Small
SONO
poatir and The seeeeeee
Detailed Description
pooi 1000
jui cocer aertarding
5an1 leper ard
Roun-Rooddddd Q&A
Single-Re Single-Round
Figure 4: Qualitative results of NeRF captioning and Q&A. Results on ShapeNeRF-Text. From top to bottom: brief and detailed descriptions, single-round Q&A
language task can be extracted from single global embedding obtained by directly processing the NeRF weights. It is also worth pointing out that, while LLaNA directly processes weights and thus is independent by design from spatial resolution, the baselines face a computational overhead growing with the desired resolution due to the necessity of extracting spatial data from NeRF (Appendix [A.3). Results show that 3D-I.LM performs better than the point-based models and comparably to image- based models. Comparing the results of image-based MLLMs when processing front versus back views, we can see that the vantage point has non-negligible effect on the performance of such baselines, with SentenceBERT and SimCSE metrics diminishing by about 4 points in all baselines. In a dataset without canonical poses for objects, this would be relevant limitation that processing NeRF weights seamlessly sidesteps. Finally, we observe that the multi-view setup of LLa provides similar performance to the single-view counterpart. In Appendix G additional qualitative examples are provided.
Detailed description. We evaluate the performance for the detailed description tasks on the pro- posed ShapeNeRF-Text. reporting the results in Tab. 3 For this task, the point- model PointLLM 78 performs similarly to the image-based based one, LLaVA [44). However, we appreci- ate that LLaNA achieves the best performance in all metrics by large margins. For instance, for the Sentence- -ERT metric, LLaNA achieves 77.43, notably 18.35 points more than LLa VA-vicun----- 13b single-view and 17.22 for the LLaVA-vicuna-11. -3b multi-view setup. These large improvements indicate that, while individual or aggregated images may be sufficient for brief descriptions, they may lack all the details needed to provide a comprchensive description. Moreover, the dependency
8

---

Table 5: Zero-Shot NeRF Frozen baselines. Best results are in bold, runner-up is underlined. (FV: front-view, BV: back-view, MV: multi- view)
Classification.
(WE IN Bhys rodlic NiriMo BAM
Madel
Medally
Accurary (S)
tite glum C/raabl
ELAN -ican 23b
Inage (PV)
SADV Sip
LLAN vicuse 36
Ieage (B9)
63.50
36
Irage (NP))
73.45
Cor Fib oolee uned on cotoody pione
Image
ML/WWioo Tb
(BV)
$7.00
Flar75 uxl
Trage (PV)
63,67
Fla775 uxl
Irage (BV)
61.47
Point cloul
Figure 5: NeRF multi-round Q&A example.
Print cloud
41,95
30LLMM
Meah MVV
LlaN4-77
NoRt"
Table 6:
NeRF brief captioning on ShapeNeRF- Text. Trained baselines. Best results are in bold, runner-up underlined. (FV: fiew
Table 7: NeRF brief captioning on the HST dataset. Trained baselines.
Best results are in bold, runner-up is underlined. (FV: fiie)
Medaliy
S-BERT
ROUGE-L
BLBU-1
MOUGE-L
NETROR
ILahiiinnss Hb
IPV)
43.56
43.22
15.56
13.74
15.27
lrage (PV)
33.99
42.66
16:38
13.32
12.19
Past choal
35.48
$8.28
21.77
25.84
24.54
Potti cloee
48.55
44.68
8.51
12.33
12.54
GPP apr
17.0
1000
21.3
35.0
31.4
Op
Piker ciee
9190
11
117
1335
11.0
Nek
6863
70.34
20.4
18.3
31.56
LLaNA.
Sea
86.2
55.06
6.6
11.11
1432
of the output quality on the selected vantage points remains strong. Contrarily, the NeRF weights contain detailed and complete information about the object, which is fundamental for more granular description tasks, with the additional advantage of not requiring tuning such hyperparameters. The ability of NeRF to capture holistic information about the object is also shown in the second row of Fig. where only the direct processing of NeRF weights lets LLaNA understand that the object is TV. Lo and LLaVA provide detailed but wrong descriptions, likely because of the need to extract the intermediate discrete representation as a point cloud or an image, losing information. Indeed, in both cases, it is hard even for human observer to provide the right description from the intermediate modalities shown in the figure. More qualitative examples of this task are shown in Appendix G
5.3 NeRF single-round Q&A
In the single-round Q&A experiment, we test the ability of the assistants to provide precise answers to specific questions about the object instead of open-ended general descriptions. We prompt the models with the NeRF, or the image/cloud extracted from it, followed by one of the questions in the single-round Q& A annotations associated with the NeRF, We then collect the answer generated by the model and compare it against the ground-truth answer with the selected metrics. Results are reported in Tab. 4 Interestingly, PointLLM (78) performs better than LLaVA (44) in this task, likely because it has been specifically trained to answer detailed questions about objects represented as point clouds. Nevertheless, similarly to the detailed description results, LLaNA the oopperforming method across all metrics, again by large margins. This result suggests that the meta-encoder and the projector can extract tnn-ggainedd information from the NeRF, even if they are processing directly NeRF weights. Remarkably, the amount of information they can extract lets LLaNA answer more precisely than when images or point clouds are extracted from the NeRF, Indeed, as shown in the third row of Fig. 4 which reports qualitative example, the only assistant able to answer correctly to a precise question about the appearance of the tyres of the car is L.aNN. In Appendix G additional qualitative examples of this task are provided. Finally, another qualitative result confirming the ability of LLaNA to provide high-quality answers to specific questions, in this case in a multi-round A&A experiment, is reported in Fig. 5
5.4 Zero-shot NeRF classification
Finally, we compare assistants on the task of zero-shot classification. We query the models with the sentence "What is the class of the NeRF/mmage/cloud? Choose among these: <Shapenet classess>

---

Table 8: NeRF detailed captioning on ShapeNeRF-Text Trained baselines. Best results are bold, runner-up underlined. (FV: front-view)
Table 9: NeRF single-round Q&A on ThapeNeRF-TTet Trained baselines. Best results are in bold, runner-up is underlined. (FV: front-view)
(P
44.99
42.31
10.06
22.56
22.10
Tenge (PV)
35.29
62.35
26.61
29.35
30.40
Poer
83.31
19.30
14:30
23.22
11:37
Parza cloal
33:28
11.18
40.00
62.03
23.33
Oye.PPa000002...0.0 2.76
Prex dieed
41.35
00.32
14.0
19:15
13.20
Partchaal
25.59
875
1338
14.77
12 74.46777
NARF
77.43
11.33
36.18
1.aNA-7
NaRT
46.16
8LU5
where bilcpenne_classese are the 10 ShapeNet classes available in our dataset. We consider the answer correct only if the ground truth class appears in the response. We report results in Tab. 5 the ShapeNeRF-Texx dataset. Using multiple views boosts the zero-shot classification performance of LLaVA, which turns out be the best model for this task, followed by LLaNA.
5.5 Training baselines on ShapeNeRF-Tex.
Tabs. 6 to report results language tasks of several baselines trained ShapeNeRR-Text, while Tab. 13 the appendix, shows zero-shot NeRF classification performance of such models. We employed those baselines on ShapeNeRF-Text, for which we were able to run the official training code. Accordingly, we followed their protocol, which, for of them, keeps the modality-speciffe encoder frozen and trains adaptor and the LLM in two steps. We notice that the trained baselines exhibit different behaviors to their frozen counterparts, with LLaVA performing significantly worse and PointLLM showing clear improvements. As for GPT4Point, we observe greater variability across metrics; however, overall, it shows no significant benefit from training on ShapeNeRF- yields the best performance compared to all baselines, either frozen trained on ShapeNeRF-Texx Finally, Appendix F shows the generalization performance on Objaverse of LLaNA and the trained baselines.
LLaNA
Limitations and future directions
Despite the promising results of our framework, it is the first study in this direction and several limitations are yet to be addressed. First, the pre-trained nf2vec encoder, having been trained exclusively on synthetic data from ShapeNet, may not generalize well real-world objects. To address this, future work should create NeRF-Text dataset including a more diverse set of objects, like the ones provided by Objaverse 13] and OmniObject3D (76). Another limitation is that nf2vec currently processes only MLPs, restricting our model to MLP-only NeRFs. However, with the rapid advancements in meta-networks, may become very soon possible extend LLaNA more complex NeRF architectures, such as InstantNGP [52]. For instance, the approach by Lim et al. [2] suggests the feasibility of processing various input architectures. although is currently limited to small networks. Finally, our framework has been tested solely on object-centrr NeRFs. Expanding its application NeRFs representing entire scenes would compelling direction for future research.
Concluding remarks
This paper addressed the novel task creating language assistant for NeRF, We have tackled this problem by leveraging recent advances in MILMs and meta-networks processing neural fields. We have shown that is feasible and effective to directly process the weights NeRF to project it into the input embedding space of an LLM. We have built and made publicly available dataset of textual annotations of NeRFss and have shown that our approach compares favourably with respect to several MLLMs used baselines for the novel tasks of brief and detailed captioning, question answering, and zero-shot classification of NsRFs.
Acknowledgements
We acknowledge the CINECA award under the ISCRA initiative, for the availability of high- performance computing resources and support.
10

---

References
[1] Josh Achiam, Steven Adler, Sandhini Agarwal, Lama Ahmad, Igge Akkaya, Florencia Lconi Aleman, Diogo Almeida, Janko Altenschmidt, Sam Altman, Shyamal Anadkat, al. Gpt-4 technical report. arXiv preprint or xrxv:23333007774 2023.
[2] Andrea Amaduzzi, Giuseppe Lisanti, Samuele Salti, and Luigi Di Stefano. Looking words and points with attention; a benchmark for text-to-shape coherence, In 2023 IEFE/CVP International Conference on Computer Vision Workshops (ICCVW), pages 2860-2869. IEEE Computer Society, 2023.
[3] Haotian Bai, Yuanhuiyi Lyu, Lutao Jiang. Sijia Li, Haonan Lu, Xiaodong Lin, and Lin Wang. Com- ponerf: Text-guided multi-object compositional nerf with editable 3d scene layout. arXiv preprint arXiv:2303. 13843, 2023.
[4] Jinze Bai, Shuai Bai, Shusheng Yang, Shijic Wang, Sinan Tan, Peng Wang, Junyang Lin, Chang Zhou, and Jingren Zhou. Qwen-vl: A frontier large vision-language model with versatile abilitics. arXiv preprint ar0iv:2308. /2966, 2023.
[5] Francesco Ballerini, Pierluigi Zama Ramirez, Roberto Mirabella, Samuele Salti, and Luigi Di Stefamo. Connecting nerfs, images, and text. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) Workshops, 2024.
[6] Satanjeev Banerjee and Alon Lavie. Meteor: An automatic metric for mt evaluation with improved conelation with human judgments. In Proceedings of the acl workshop on intrinsic and extrinsic evaluation measures for machine translation and/or summarization,, pages 65-72, 2005.
[7] Adriano Cardace, Picrluigi Zama Ramirez, Francesco Ballerini, Allan Zhou, Samuele Salti, and Luigi di Stefano. Neural processing of tri-plane hybrid neural fields. In The Twelfth International Conference on Learning Representations, 2024.
[8] Angel X Chang. Thomas Funkhouser, Leonidas Guibas, Pat Hanrahan, Qixing Huang, Zimo Li, Silvio Savarese, Manolis Savva, Shuran Song. Hao Su, et al. Shapenet: An information-rich 3d model repository. arXiv preprint
2015.
[9] Anpei Chen. Zexiang Xu, Andreas Geiger. Jingyi Yu, and Hao Su. Tensorf: Tensorial radiance fields. In European Conference on Computer Vision (ECCV), 2022.
[10] Guo Chen, fin-Dong Zheng, Jiahao Wang, Jilan Xu, Yifei Huang, Junting Pan, Yi Wang, Yali Wang. Yu Qiao, Tong Lu, et al. Videollm: Modeling video sequence with large language models. arXiv preprint arXiv:2305. 13292, 2023.
[11] Wenliang Dai, Junnan Li, Dongxu Li. Anthony Meng Huat Tiong, Junqi Zhao, Weisheng Wang, Boyang Li, Pascale N Fung, and Steven Hoi. Instructblip: Towards general-purpuse vision-language models with instruction tuning. Advances in Neural Information P Processing Systems, 36, 2024.
[12] Luca De Luigi, Adriano Cardace, Riccardo Spezialetti, Pierluigi Zama Ramirez, Samuele Salti, and Luigi Di Stefano. Deep learning on implicit neural representations of shapes. In International Conference on Learning Representations (ICLR), 2023.
[13] Matt Deitke, Dustin Schwenk, Jordi Salvador, Luca Weihs, Oscar Michel, Eli VanderBilt, Ludwig Schmidt, Kiana Ehsani, Aniruddha Kembhavi, and Ali Farhadi. Objaverse: A universe of annotated 3d objects. In Proceedings of the IEEF/CVF Conference on Computer Vision and Pattern Recognition, pages 13142- 13153, 2023.
[14] Danny Driess, Fei Xia, Mehdi SM Sajjadi, Corey Lynch, Aakanksha Chowdhery, Brian Ichter, Ayzaan Wahid, Jonathan Tompson, Quan Vuong. Tianhe Yu, et al. Palm-e: embodied multimodal language model. In International Conference on Machine Learning, pages 8469 8488. PMLR, 2023.
[15] Emilien Dupont, Hyunjik Kim, SM Ali Eslami, Danilo Jimenez Rezende, and Dan Rosenbaum. From data to functa: Your data point is function and you can treat it like onc. International Conference on Machine Learning, pages 5694-5725. PMLR, 2022.
[16] Sara Fridovich-Keil, Alex Yu, Matthew Tancik, Qinhong Chen, Benjamin Recht, and Angjoo Kanazawa. Plenoxels: Radiance fields without neural networks. 2022 JEEE/CVF Conference on Computer Vision and Puttern Recognition (CVPR). 2022.
[17] Peng Gao, Jiaming Han, Renrui Zhang, Ziyi Lin, Shijie Geng, Aojun Zhou, Wei Zhang, Pan Lu, Conghui He, Xiangyu Yue, et al. Llama-adapte v2: parameter-efficiee visual instruction model, arXiv preprint 150000, 2023,
11

---

[18] Tianyu Gao. Xingcheng Yao, and Danqi Chen. Simese: Simple contrastive learning of sentence embeddings. In 2021 Conference on Empirical Methods in Natural Language Processing, EMNLP 2021, pages 6894- 6910. Association for Computational Linguisties (ACL), 2021.
[19] Rohit Girdhar, Alaaeldin El-Nouby, Zhuang Liu, Mannat Singh, Kalyan Vasudev Alwala, Armand Joulin, and Ishan Misra. Imagehind: One emhedding space to bind them all. In Proceedings of the IFEF/CVF Conference on Computer Vision and Pattern Recognition, pages 151 80-15119,, 2023.
[20] Ziyu Guo, Renrui Zhang, Xiangyang Zhu, Yiwen Tang, Xianzheng Ma, Jiaming Han, Kexin Chen, Peng Gao, Xianzhi Li, Hongsheng Li, et al. Point-bind & point-llm: Aligning point cloud with multi- multi-modality for 3d understanding, generation, and instruction following. arXiv preprint arXiv:2309.cm 200.02330000055,, 2023.
[21] Ayaan Haque, Matthew Tancik, Alexei A. Efros, Aleksander Holynski, and Angjoo Kanazawa. Instruct- nerf2nerf: Editing 3d scenes with instructions. In Proceedings of the IEEE/CVF International Conference on Computer Vixion (ICCV), pages 19740-19750, 2023.
[22] Robert Hecht-Nielsee On the algebraic structure of feedforward network weight spaces. In Advanced Neural Computers, pages 129-135. Elsevier, 1990.
[23] Yining Hong, Chunru Lin, Yilun Du, Zhenfang Chen, Joshua B. Tenenbaum, and Chuang Gan, concept learning and reasoning from multi-view images. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pages 9202-9212, 2023.
[24] Yining Hong, Haoyu Zhen, Peihao Chen, Shuhong Zheng, Yilun Du, Zhenfang Chen, and Chuang Gan. 3d-LIIM: Injecting the 3d world into large language models. In Thirry-sevnnth Conference on Neural Information Processing Systems, 2023.
[25] Benran Hu, Junkai Huang. Yichen Liu, Yu- Wiig Tai, and Chi-Keung Tang. Nerf-rpn: A general framework for object detection in nerfs. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 23528-23538, 2023.
[26] Rongjic Huang, Mingze Li, Dongchao Yang, Jiatong Shi, Xuankai Chang, Zhenhui Ye, Yuning Wu, Zhiqing Hong. Jiawei Huang. Jinglin Liu, et al. Audiogpt: Understanding and generating speech, music, sound, and talking head. In P Proceedings of the AAAI Conference on Artificial Intelligence, pages 23802-23804. 2024.
[27] Shaohan Huang, Li Dong, Wenhui Wang, Yaru Hao, Saksham Singhal, Shuming Ma. Tengchao Lv, Lei Cui, Owais Khan Mohammed, Barun Patra, et al. Language is not all you need: Aligning perception with language models. Advances in Neural Information Processing Systems, 36, 2024.
[28] Sungwon Hwang, Junha Hyung, Daejin Kim, Min-Jung Kim, and Jaegul Choo. Faceclipnerf: Text- driven face manipulation using deformable neural radiance fields. In Proceedings of the IEEE/CVF International Conference on Computer Vision (ICCV), pages 3469-3479, 3479, 2023.
[29] Sergey Toffe and Christian Svegedy. Batch normalization: Accelerating deep network training by reducing internal covariate shift. In International conference on machine learning, pages 448-456. pmlr, 2015.
[30] Florian Jaeckle and M Pawan Kumar. Generating adversarial examples with graph neural networks. In Uncertainty in Artificial Intelligence, pages 1556-15566 1564, PMLR, 2021.
[31] Kyungmin Jo, Gyumin Shim, Sanghun Jung, Soyoung Yang, and Jacgul Choo. Cg-nerf: Conditional generative neural radiance fields for 3d-aware image synthesis. In Proceedings of the IEFF/CVF Winter Conference on Applications of Computer Vision (WACV), pages 724-733, 2023.
[32] Justin* Kenr, Chung Min Kim, Ken Goldberg, Angjoo Kanazawa, and Matthew Tancik. Lerf: Language embedded radiance fields. International Conference on Computer Vision (ICCV), 2023.
[33] Boris Knyazev, Michal Drozdzal, Graham W. Taylor, and Adriana Romero. Parameter prediction for unseen deep architectures. In Advances in Neural Information Processing Systems. 2021.
[34] Sosuke Kobayashi. Eiichi Matsumoto, and Vincent Sitzmann. Decomposing nerf for editing via feature field distillation. In Advances in Neural Information Processing Systems, pages 23311-23330 Curran Associates, Inc., 2022.
[35] Miltiadis Kofinas, Boris Knyazev, Yan Zhang, Yunlu Chen, Gertjan J Burghouts, Efstratios Gavves, Cees GM Snoek, and David W Zhang, Graph neural networks for learning equivariant representations of neural networks. In The Twelfth International Conference on Learning Representations, 2024.
[36] Han-Hung Lee and Angel X. Chang. Understanding pure clip guidance for voxel grid nerf models, 2022.
12

---

[37] Bo Li. Yuanhan Zhang, Liangyu Chen, Jinghao Wang, Fanyi Pu, Jingkang Yang. Chunyuan Li. and Ziwei Liu. Mimic-it: Multi-modal in-context instruction tuning. arXiv preprint arXiv:2306.05422 054425,, 2023.
[38] Junnan Li, Dongxu Li, Silvio Savarese, and Steven Hoi. Blip-2: 2: Bootstrapping language image pre- vre-training with frozen image encoders and large language models. In International conference on machine learning, pages 19730-19742. PMIR, 2023.
[39] Junnan Li. Dongxu Li, Silvio Savarese, and Steven Hoi. BLIP-2: Bootstrapping language-image pre- training with frozen image encoders and large language models, arXiv preprint arXiv:2301 12597, 2023.
[40] Jianhui Li, Shilong Liu, Zidong Liu, Yikai Wang, Kaiwen Zheng, Jinghui Xu, Jianmin Li, and Jun 7hu. Instructpix2neRRF Instructed 3d portrait editing from a single image. In The Twelfth International Conference on Learning Representations, 2024.
[41] Ruilong Li, Hang Gao, Matthew Tancik, and Angjoo Kanazawa, Nerface: Efficient sampling accelerates nerfs, arXiv preprint arXiv:2305. 04996, 2023.
[42] Derek Lim, Haggai Maron, Marc T. Law, Jonathan Lorraine, and James Lucas. Graph metanetworks for pro- cessing diverse neural architectures. In The Twelfth International Conference on Learning Representations, 2024.
[43] Chin Yew Lin. Rouge: A package for automatic evaluation of summaries. In Text summarization branches oud, pages 74- 81, 2004.
[44] Haotian Liu, Chunyuan Li, Qingyang Wu, and Yong Jac Lec, Visual instruction tuning. Advances in neural information processing systems, 36, 2024.
[45] Jingyue Lu and M. Pawan Kumar. Neural network branching for neural network verification. In Interna- tional Conference on Learning Representations, 2020.
[46] Tiange Luo, Chris Rockwell, Honglak Lee, and Justin Johnson. Scalable 3d captioning with pretrained models. arXiv preprint arXiv:2506.07279, 2023.
[47] Muhammad Maaz, Hanoona Rasheed, Salman Khan, and Fahad Shahbaz Khan. Video- Aido--cattpp:: Towards detailed video understanding via large vision and language models. arXiv preprint arXiv:2306.05422 05424. 2023.
[48] Ricardo Martin-Brualla, Noha Radwan, Mehdi SM Sajjadi, Jonathan T Barron, Alexey Dosovitskiy, and Daniel Duckworth. Nerf in the wild: Neural radiance fields for unconstrained photo collections. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 7210-7219 2021.
[49] Gal Metzer, Elad Richardson, Or Patashnik, Raja Giryes, and Daniel Cohen-Or. Latent-nerf for shape- guided generation of 3d shapes and textures. In Proceedings of the IEEE/CVF Conference Computer Vision and Patiern Recognition (CVPR), pages 12663-12673, 2023.
[50] Ben Mildenhall, Pratul P Srinivasan, Matthew Tancik, Jonathan T Barron, Ravi Ramamoorthi, and Ren Ng. Nerf: Representing scenes as neural radiance fields for view synthesis. In European conference on computer vision, pages 405-421. Springer, 2020.
[51] Ashkan Mirzaei, Tristan mamnenttdo-Ammstroon Marcus A. Brubaker, Jonathan Kelly, Alex Levinshtein. Konstantinos G. Derpanis, and Igor Gilitschenski. Reference-guidee controllable inpainting of neural radiance fields. In Proceedings of the IEEE/CVF International Conference Computer Vision (ICCV), pages 17815 17825, 2023.
[52] Thomas Müller, Alex Evans, Christoph Schied, and Alexander Keller. Instant neural graphics primitives with multiresolution hash encoding. ACM Trans. Graph., 11(4):102:1-00::55 2022.
[53] Aviv Navon, Aviy Shamsian, Idan Achituve, Ethan Fetaya, Gal Chechik, and Haggai Maron. Equivariant architectures for learning deep weight spaces. International Conference on Machine Learning, 2023.
[54] Long Ouyang, Jeffrey Wu, Xu Jiang, Diogo Almeida, Carroll Wainwright, Pamela Mishkin, Chong Zhang, Sandhini Agarwal, Katarina Slama, Alex Ray, et al. Training language models to follow instructions with human feedback. Advances in neural information processing systems, 35:27730-27744 2022.
[55] Kishore Papineni, Salim Roukos, Todd Ward, and Wei-Jing Zhu. Bleu: a method for automatic evaluation of machine translation. In Proceedings of the 40th annual meeting of the Association for Computational Linguistics, pages 311-318, 2002.
13

---

[56] Zhiliang Peng, Wenhui Wang. Li Dong, Yaru Hao, Shaohan Huang. Shuming Ma. and Furu Wei. Kosmos-2: Grounding multimodal large language models to the world. arXiv preprini arXiv:2306. 14824, 2023.
[57] Ben Poole. Ajay Jain, Jonathan T Barron, and Ben Mildenhall. Dreamfusion: Text-to-3d using 2d diffusion. In The Eleventh International Conference on Learning Representations, 2022.
[58] Zhangyang Qi. Ye Fang, Zeyi Sun, Xiaoyang Wu, Tong Wu, Jiaqi Wang, Dahua Lin, and Hengshuang Zhao. Gpt4point: A unified framework for point-language understanding and generation. In CVPR, 2024.
[59] Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, al. Learning transferable visual models from natural language supervision. In International Conference on Machine Learning, pages 8748-8763. PMLR, 2021.
[60] Colin Raffel, Noam Shazeer. Adam Roberts, Katherine Lee, Sharan Narang, Michael Matena, Yanqi Zhou, Wei Li, and Peter Liu. Exploring the limits of transfer learning with unified text-to-text transformer. Journal of machine learning research. 21(140):1 1-67)))-:72000 2020.
[61] Pierluigi Zama Ramirez, Luca De Luigi, Daniele Sirocchi, Adriano Cardace, Riccardo Spezialetti. Francesco Ballerini, Samuele Salti, and Luigi Di Stefano. Deep learning on 3d neural fields. arXiv preprint Xiv: 13277, 2023.
[62] Partha Pratim Ray. Chatgpt: A comprehensive review on background, applications, key challenges, bias, ethics, limitations and future scope. Internet of Things and Cyber-Physical Systems, 3:121-154, 2023.
[63] Nils Reimers and Iryna Gurevych. Sentence-bert: Sentence embeddings using siamese bert-networks. arXiv preprint arXiv:1908. 10084, 2019.
[64] Konstantin Schürholt, Dimche Kostadinov, and Damian Borth. sulf-supe representation learning on neural network weights for model characteristic prediction. In Advances in Neural Information Processing Systems, 2021.
[65] Bipasha Sen, Gaurav Singh, Aditya Agarwal, Rohith Agaram, Madhava Krishna, and Srinath Sridhar. Hyp-nerf: Learning improved nerf priors using hypemetwork. In Advances in Neural Information Processing Systems, pages 51050-51064. Curran Associates, Inc., 2023.
[66] Hoigi Sco, Haycon Kim, Gwanghyun Kim, and Se Young Chun. Ditto-nerf: Diffusion- based iterative text to aunnidirectional 3d model, 2023.
[67] Hyeonseop Song, Scokhun Choi, Hoseok Do Chul Lee, and Tachyeong Kim. Blending-nerf: Text-driven localized editing in neural radiance fields. In Proceedings of the IEEE/CVF International Conference on Computer Vision (ICCV), pages 14383 14393, 2023.
[68] Cheng Sun, Min Sun, and Hwann- Tzong Chen. Direct voxel grid optimization: Super-fast convergence for radiance fields reconstruction. In Proceedings of the IEEE/CVF Conference Computer Vision and Pattern Recognition, pages 5459 5469, 2022.
[69] Chunyi Sun, Yanbin Liu, Junlin Han, and Stephen Gould, Nerfeditor: Differentiable style decomposition for 3d scene editing. In Proceedingy of the IEEE/CVF Winter Conference on Applications of Computer Vision (WACV), pages 7306-7315, 2024.
[70] Hugo Touvron, Thibaut Lavril, Gautier Izacard, Xavier Martinet, Marie-Anne Lachaux, Timothée Lacroix. Baptiste Rozière, Naman Goyal. Eric Hambro, Faisal Azhar, et al. Llama: Open and efficient foundation language models. arXiv preprint arXiv: 02.3021.9777 13971, 2023.
[71] Thomas Unterthiner, Daniel Keysers, Sylvain Gelly, Olivier Bousquet, and Ilya O. Tolstikhin. Predicting neural network accuracy from weights. arXiv, Abs/20022.11443 2020.
[72] Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Lukasz Kaiser, and Illia Polosukhin. Attention is all you need. In Advances in Neural Information Processing Systems. Curran Associates, Inc., 2017.
[73] Can Wang, Menglei Chai, Mingming He, Dongdong Chen, and Jing I. Liao. Clip-nerf: Text-and-image driven manipulation of neural radiance fields. In Proceedings of the IEEE/CVE Conference on Computer Vision and Pattern Recognition, pages 3835-3844, 2022.
[74] Can Wang, Ruixiang Jiang, Menglei Chai, Mingming He, Dongdong Chen, and Jing Liao. Nerf-art: Text-driven neural radiance fields stylization. IEEE Transactions on Visualization and Computer Graphics. pages 1-15, 2023,
14

---

[75] Jason Wei, Maarten Bosma, Vincent Zhao. Kelvin Guu, Adams Wei Yu. Brian Lester, Nan Du, Andrew M Dai, and Quoe V Le. Finetuned language models are zero-shot leamerr, In International Conference on Learning Reprexemtations 2021.
[76] Tong Wu, Jiarui Zhang, Xiao Fu, Yuxin Wang. Liang Pan Jiawei Ren, Wayne Wu, Lei Yang. Jiaqi Wang. Chen Qian, Dahua Lin, and Ziwei Liu. Omniobject3d: Large-vocabular 3d object dataset for realistic perception, reconstruction and generation. In IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), 2023.
[77] Qiangeng Xu, Weiyue Wang, Duygu Ceylan, Radomir Mech, and Ulrich Neumann. Dism: Deep implicit surface network for high-quality single-view 3d reconstruction. In Advances in Neural Information Processing Systems. Curran Associates, Inc., 2019.
[78] Runsen Xu, Xiaolong Wang, Tai Wang, Yilun Chen, Jiangmiao Pang, and Dahua Lin, Pointllm Empower- ing large language models to understand point clouds. arXiv preprint arXiv:2308 16911, 2023.
[79] Lin Yen-Chen, Pete Florence. Jonathan T Barron, Tsung- Yi Lin, Alberto Rodriguez, and Phillip Isola. Nerf-supervision: Learning dense object descriptors from neural radiance fields. In 2022 international conference robotics and automation (ICRA), pages 6496-6503. IEEE, 2022.
[80] Yingchen Yu, Rongliang Wu, Yifang Men, Shijian Lu, Miaomiao Cui, Xuansong Xie, and Chunyan Miao. Morphnerf: Text- Text-guided 3d aware editing via morphing generative neural radiance fields. IEEE Transactions on Multimedia, pages 1-13, 2024.
[81] Pierluigi Zama Ramirez, Luca De Luigi Danicle Sirocchi, Adriano Cardace, Riccardo Spezialetti, Francesco Ballerini, Samuele Salti, and Luigi Di Stefano. Deep learning on 3D neural fields. arXiv preprint 2ix::331..13277, 2023.
[82] Renrui Zhang, Jiaming Han, Chris Liu, Peng Gao, Aojun Zhou, Xiangfei Hu, Shilin Yan, Pan Lu, Hongsheng Li, and Yu Qiao. Llama-adapper: Efficient fine-tuning of language models with zero-init attention. arXiv preprint arXiv:2303. 16/99, 2023.
[83] Allan Zhou, Kaien Yang, Kaylee Burns, Adriano Cardace, Yiding Jiang, Samuel Sokota, J Zico Kolter, and Chelsea Finn. Permutation equivariant neural functionals. Advances in neural information processing systems, 37, 2023.
[84] Allan Zhou, Kaien Yang, Yiding Jiang, Kaylee Burns, Winnie Xu, Samuel Sokota, Zico Kolter, and Chelsea Finn. Neural functional transformers. Advances in neural information processing systems, 37, 2023.
[85] Allan Zhou, Chelsea Finn, and James Harrison. Universal neural functionals. arXiv preprint urXiv:240.005232. 2024.
[86] Ziyu Zhu, Xiaojian Ma, Yixin Chen, Zhidong Deng, Siyuan Huang, and Qing Li. 3d-vista: Pre-trained transformer for 3d vision and text alignment. ICCV, 2023.
[87] Jingyu Zhuang, Chen Wang, Liang Lin, Lingji Liu, and Guanbin Li. Dreameditor: Text-driven 3d scene editing with neural fields, SIGGRAPH Asia 2023 Conference Papers, pages 1-10, 2023.
15

---

A Details on NeRFs
We report here some details regarding the NeRFs of the ShapeNeRR-Tex dataset, which were trained by Zama Ramirez al. 81 The NeRF code is implemented leveraging the NerfAcc library 41
A.1 Architecture
An instance of the employed NeRFs consists of multi-layer perceptron (MLP) that contains three hidden layers, each with 64 neurons. The ReLU activation function is applied between all layers except for the last one, which calculates the density and RGB values directly without any activation function. A frequency encoding (50) is applied to the input 3D coordinates, in order to improve the NeRF reconstruction quality. NeRFs do not take as input the view direction. The MLP processes an input coordinate p IR3 to produce d-dimensional vector containing RGBa.
A.2 Training
Training NeRF consists of minimizing the error between the rendered images from the NeRF and the ground truth images. Our NeRFs were trained using an L1 loss between the predicted and ground truth RGB pixel intensities, weighting background pixels less than foreground pixels (0.8 foregrounds vs. 0.2 background). Image rendering involves querying the neural network by feeding it 3D coordinates to obtain RGB color values and density estimates. By integrating these outputs along camera rays using volumetric rendering techniques [50), colors and opacities are accumulated to produce the final rendered image. Each NeRF is trained until it reaches good reconstruction quality, approximately for 2000 steps.
A.3 Generating images and point clouds from NeRFs
To compare with 2D and 3D MLLMs on the new tasks of NeRF captioning and NeRF Q&A, we need to render images or reconstruct point clouds from the NeRF. To render images, we employ same volumetric rendering procedure used during the NeRF's training. In order to extract a point cloud, the marching cubes algorithm first applied to the volumetric density field derived from the NeRF. This process generates mesh by identifying isosurfaces within the density field. The mesh is then converted into point cloud by considering only the mesh vertices, uniformly distributed the 3D space. We sample RGB values from NeRF for each point coordinate to approximate point cloud colors. An example of data extracted from NeRF is depicted in Fig. 6
Generating images and point clouds requires the user to make some decisions, the effects of which on the assistant's performance are not easy to anticipate. When dealing with images, is difficult to select the rendering viewpoint. It might happen that the object is not clearly visible from the chosen viewpoint or that important elements are missing. Another decision is the resolution of the generated image, which, if too coarse, may prevent the identification of fine-grained details. The same concerns regarding the resolution also apply to point clouds. Yet, the modality encoder may not handle large resolutions or may greatly increase the processing time. Another important point is the additional computational time required to extract data from NeRF For instance, extracting point clouds from NeRF with only 8192 points requires approximately 620ms. Moreover, the time for sampling the MLP and running a marching cube algorithm scales cubically with the desired spatial resolution. On the other hand, the time required to process the MLP weights is independent of the spatial resolution.
GT Front
Rendered Front
GT Back
Rendered Back
GT Points
Extracted Points
Figure 6: Example data extracted from NeRF, From left to right: GT front view, rendered front view, GT back view, rendered back view, GT point cloud, extracted point cloud.
16

---

A.4 NeRF memory occupation compared to images or point clouds
An important benefit of using NeRF to represent objects is that memory consumption is decoupled from spatial resolution. In Fig- 7 we analyze the number of parameters needed for point clouds and images compared to neural fields by altering the spatial resolution of the data. We account for all variables required by explicit representation in their parameter count. For instance, each point in point cloud has six parameters corresponding to its coordinates (æ, y, 2) and color (R, G. B), while each pixel has only three channels (R, G. B). The orange line represents the parameters of the NeRF MLP, while the blue lines indicate the parameters for 3D points (Fig. Hefft and image pixels (Fig. right).
We observe that the space occupied by the NeRF MLP is comparable to that used by point clouds in experiments (i.e., 8192 points, the data size used in GPT4Point [58] and PointLLM (78). However, NeRF becomes advantageous for representing data as soon as the point cloud size is greater than 8621 points. This is crucial, considering real datasets may contain point clouds meshes with significantly more points or faces; for example, Objaverse 13 features meshes with over 107 polygons.
The advantages are even more pronounced for images, where a single NeRF MLP corresponds to 36 images at resolution of 22 x 22. Storing the 36 pictures from ShapeNetRender at 256 256 resolution, used to train our NeRF on a single object, requires substantially more memory.
Point cloud vs NeRF
Pixels vs NeRF
60000
Point cloud
5419008
Images
MLP
Number of Poraiaterss
Number of
51728
49152
224 224 x (ShapelietRenne
81.9
862]
22
Number Points
Number Pixels
Figure 7: Memory usage of NeRF compared to images point clouds. Left: NeRF point clouds. Right: NeRF vs pixels.
B Details on the Meta-Encoderr
We employ nf2vec 81 as the meta-encoder of LLaNA. Thus, in the following paragraphs, we describe the details of the nf2vec architecture and training protocol.
B.1 Architecture
The nf2vec encoder consists of 4 linear layers with 512, 512, 512, and 1024 output neurons each. The final embedding extracted for a given MLP is a 1024-dimensional array. Each linear layer followed by batch normalization and a ReLU activation function. The weight matrices and biases of the input NeRF are stacked along the row dimension to form matrix M E RSX where = 144 + L (H + 1) + + I. * H + + H + 146 and L. the number of units per hidden layer, while H is the dimension of the input.
17

---

NeRF
Child code unloursity
eng
Brief description
eng go leduiig cocoonnd bedy
conttal anso vode bur lee pntcectleen ville, eaeige acoear the bi halium and shilratto com lis seratymanics
Detailed description
contoal
preatt
White desige the vndiy
ligh torteeggee Mary marks
pass dls deagged accual local Fadii
The siigle argine provides poouiiirn unille shipe all
act biga ar sel, cokre anco wousi
Single-round QA
Multi-round QA
Figure 8: Example of data sample from ShapeNeRF-Text dataset.
B.2 Training
We employ the official code and weights of 2feved pre-trained on an augmented version of ShapeNetRender [77]. The encoder was trained in an end-to-end manner together with an implicit decoder. The decoder takes in input 3D coordinates after frequency encoding and the global 1024- dimensional output of the encoder. consists of linear layers with 1024 neurons each, followed by ReLU activations except for the last layer. It yields 4-dimensional output RG Bo, similar to the NeRF given input to the encoder. The framework supervision comes from the pixel-wise rendering Ln error computed between the ground truth RGB image and the predicted image, which is obtained through volumetric rendering after encoding and decoding the NeRF.
C Details on ShapeNeeF--Tss dataset
The proposed ShapeNeRF- dataset consists of 40K paired NeRFs and language annotations for ShapeNet objects [8) In particular, for every 3D model, multiple annotations have been provided: brief description, a detailed description, 3 single-round Q& and one multi-round Q&A. Figure g
shows example of such annotations. These annotations have been obtained by exploiting VA
and LLaMA 3 as described in section f of the main paper.
C.1 Instruction prompts and ground-truth questions
In this section, we provide the instruction prompts used to generate the ground -ound answers of ShapeNeRF-Ters and the list of questions used to build the ground-truth questions for the brief and detailed descriptions.
Instruction prompts for LLaVA and LLaMA to generate the dataset For constructing ShapeNNrffText, first, descriptive captions for multiple views of each object have been obtained using the following input request to LLaVA:
https://cvlab-unii github. 10 /n/nffvvc/
18

---

"USER: <inage>imgeeeai You will be provided the image of an object, seen from the <view point>. Describe the object in detail. Include as much information as possible, but do not infer anything that is not in the image. Avoid describing the background. Generate an answer with a maximum length of 30 words. words.wAASSISAANT:
The placcholder <view point> was replaced with "back", "side", or "front" according the viewpoint of the image provided as input. To expedite computation and leverage the high symmetry of ShapeNet objects, 7 views have been employed for this process.
After obtaining the captions for each view, LLaMA was queried to aggregate these single-view captions into comprehensive descriptions and Q&A rounds. The input provided to LLaMA was:
You will be shown 7 different descriptions of an object, obtained from different points of view. Please provide two descriptions, which aggregates all these ones. The first description must concise, the second one will more descriptive. Both these description must refer to the same subject. Avoid repetitions. Important: The output descriptions must be followed by the string "Final concise description." and "Final more detailed description.". Notice: There are errors in some descriptions, due to occlusion and improper angle. You need combine all the descriptions and eliminate possible wrong details (please fix the errors directly, do not tell me). Input descriptions: [list of the single-view captions generated by VA]
The detailed description was then used to generate multiple Q&A rounds, through the following request:
Given this description of an object, generate 6 short Q&A dialogues regarding diverse aspects of the object described, ensuring logical relevance between the questions and answers. Include always question about how this object can be used. Question begins with 'Q', Answer begins with 'A', IMPORTANT: Do not mention size, background. Do not mention "how many". Do not add text after the last answer.".
From the 6 generated Q&A pairs, were randomly sampled to build the sequence multi-round Q&A, while the remaining pairs were used as single-round Q&A.
Ground-truth questions for the brief and detailed descriptions Tab. 10 and Tab. provide the list of questions used to build the ground-truth data of ShapeNeRF-Texx as explained in Sec. 4.1
Table 10: List of questions prompt the model to produce brief descriptions. An instruction from the list is randomly selected and coupled with a ShapeNeRF- brief caption to form a ground-truth data sample.
Summarize the 3D object briefly.
What kind of object is depicted by this NeRF?
Provide a short explanation of this object.
What does this NeRF represent?
Can you give a brief overview of this object?
Characterize the object this NeRF is illustrating.
Share brief interpretation of this NeRF
Provide an outline of this 3D shape's characteristics.
What object is this NeRF rendering?
Deliver a quick description of the object represented here.
How would you describe the 3D form shown in this NeRF?
What the nature of the object this NeRF is representing?
Present compact account of this 3D object's key features.
What can you infer about the object from this NeRF?
Offer a clear and concise description of this object.
How would you summarize this 3D data?
Give brief explanation of the object that this NeRF represents.
What kind of structure does this NeRF depict?
Could you delineate the object indicated by this NeRF?
Express in brief, what this NeRF is representing.
19

---

Give quick overview of the object represented by this NeRF.
Convey a summary of the 3D structure represented in this NeRF.
What kind of object is illustrated by this NeRF?
Describe the object that this NeRF forms.
How would you interpret this NeRF?
Can you briefly outline the shape represented by this NeRF?
Give a concise interpretation of the data presented here.
Explain the object this NeRF depicts succinctly.
Offer a summary of the 3D object illustrated by this NeRF.
Table 11: List of questions to prompt the model to produce detailed descriptions. An instruction from the list is randomly selected and paired with a ShapeNeRF- detailed caption to form ground-truth data sample.
Can you tell me more about this?
What does this represent?
Can you describe this more detail?
I'm interested in this. Can you explain?
Could you provide more information about this?
What exactly am looking at here?
What is this?
Could you describe the detailed structure of this?
This looks interesting. Can you expand on it?
Can you explain more about this form?
What can you tell me about the shape of this object?
Could you delve deeper into this?
want to know more about this. Can you help?
Can you walk me through the details of this object?
Can you provide comprehensive account of this object?
Offer a detailed interpretation of this NeRF
Please elucidate
the characteristies of this form.
Could you provide an in-depth description of this structure?
What does this NeRF represent in its entirety?
Elaborate on the details of this NeRF, please.
Kindly furnish me with more information about this object.
Please expand on the intricate structure of this form.
Provide meticulous explanation of what this NeRF represents.
Provide a detailed explanation of what this NeRF represents.
request detailed breakdown of this structure.
Give a thorough rundown of this NeRF.
Can you offer a complete analysis of this object?
would like comprehensive explanation of this form.
Please detail the specific features of this NeRF
Could you elaborate extensively on what this represents?
C.2 ShapeNeRF-Ters statistics
The average lengths words of the instractions/respoones are 8.51/22.76 for brief descriptions, 7.82/77.90 for detailed descriptions, 8.81/14.25 for single-round QAs and 8.80/14.14 (per round) for multi-round QAs. Fig. ( and Fig. [0]reprrt instruction/response length histograms and the word clouds obtained after removing generic words like "model", "object" and "NeRF", emphasizing frequent words in the ground-trun instructions and responses.
20

---

Figure 9: ShapeNeRF-Teex statistics for ground-truth brief and detailed descriptions.
Brief Descriptions
Word clouds
3D
Perruting ueur m mend Patarepresente
shart ood Sourtien
WORH four Straight
O
repre appressnt oo Lanaaaatt
INTR
Summariz ze
CORT ROCOIUE
design
modern de
Braethy
curved
WILLS chair
Dacknest
BUTCK
brief
our
Careu
COD
egs
Str Straight leg
Provide
Offer Ao
simple
d describn summar
brigt
key
white rrectann
Teatur ing
Langulaa
legs made
100 Gastd
Instructions
Responses
Brief Descriptions
Lengths (Words)
8000
Total: 38644
Total: 38644
7000
Average length: 8.51
2500
Average length: 22.76
6000
2000
Number
Number
1500
3000
2000
1000
Length
Length
Instructions
Responses
Detailed Descriptions Word clouds
shape
tell
rectanacler to
truturee
eprese
design
gn
a
Sreegge
struct
TO
cecturr
f
shaps gg
curved
backrest
Please detoil
orrmnn
S
Simpl
Flat
Cop
Pericatee
or
Offer
Tov
dde
excty
Please
expand
expand
modern
IHe
Sassee leek core ece broenass
chair
tal persiby
featurre
deive
deeper
Texplain
loots
interetton
O
descFibe
tableblace
backgr ound
ou Lggs
Instructions
Responses
Detailed Descriptions
Lengths (Words)
Total: 38644
Total: 38544
Average length: 7.80
3000
Average length: 77.90
8000
2500
Number
Number
6000
2000
1500
4000
1030
2000
500
10
11
20
60
80
100
120
144
160
180
Length
Length
Instructions
Responses
21

---

Figure 10: ShapeNeRF-Text statistics for ground-truth single-round and multi-round Q&A A.
Single-round Q&A Word clouds
ourpose
design
te
Rodorn destRn
PritRty color mat teriiall jused
Sppiinal ny fmat waater
Yes
uuse
S S
See
Sae ne
face spon
Car
black s0fa Bhaskress wat
Jos Faii
Eable
tapl e aetaute mackrest Halp
style
dirplancerr
cna
cniat
Ba Bood
airplañe
TAS Comtn color
Light colored dere
bench 3
make Th
natecial possibly tis pr prinary coloc made
Instructions
Responses
Single-roume Q&A Lengths (Words)
36000
Total: 115932
8000
Total: 115932
30000
Average length:
7000
Average length: 14.25
25000
6000
Number
Number
8000
20000
4000
15000
3000
10000
2000
5000
1000
15.0
20.0
Length
Length
Instructions
Responses
Multi-round Q&A
Word clouds
polepprrose
Cab
perproos
snooth
made
black
primmry toler pritary naterial
Satee
white
SUP face
OF
sleek
Sarrplann
LOON
make
design 1
Backrest
ma
material
used Tin
des100
style
bench
designedy
aod
color
simple
Jee di
pessiblyy
sofa
make
suitable
Instructions
Responses
Multi-round Q&A Lengths (Words)
Total: 111254
8000
Total: 111254
30000
Average length: 8.80
7000
Average length: 14, 14
6000
Number
Number
5000
20000
4000
19000
3000
2000
5000
1000
5.0
7.5
10.0
12.5 Length
15.0
17.5
20.0
22.5
10
20
30
40
50
Length
Instructions
Responses
22

---

C.3 ShapeNeRF-Text quality analysis
We have carried out several experiments to assess the quality of the questions in ShapeNeRF-Texx More specifically, the purpose of this analysis was to understand how many questions referred to detail that is visible only from specific viewpoint of the object. First, we evaluated our dataset questions with language-only model, LLaMA3. For each question Q. we asked LLaMA3: Is random viewpoint of the object enough to answer this question? <Q> so, reply "YES"; : if specific viewpoint is needed, answer "NO".
By doing so, we obtained 5163 "YES" and 5847 "NO", highlighting that most questions refer to some details which are visible only from point of view.
Second, we ran a Vision-Language model, LLaVA-1.6-13b, on each question of the single-round Q&A dataset, on the front and back views of objects. Then, selected only the LLaVA responses where the answer for the front or back view achieves a SimCSE score higher than 80%, i.c., likely correct answers, which selects approximately 45% of the answers. Among these correct responses, we calculated the percentage of those where the front and back answers are extremely different (i.e., difference SimCSE scores 10). Remarkably, 26% of such answers are correct from one point of view but wrong from the other: these questions would have required multi-view information to be answered correctly. We report two qualitative examples in Fig. In the first row, the Mercedes-Bent logo cannot be recognized from the back view. In the second row, from the rear viewpoint, the monitor seems turned off, and thus not possible to identify correctly the helicopter displayed on the screen. Similarly, Fig. o4 the Appendix shows other examples of this kind of cases.
Back View
Front View
What is the logo the grille?
What grille?
The image you've provided too small and bluryy to discern any specific details, such logo grille. If you can provide a clearer image
The logo on the grille of the Mercedes-Benz emblem
context, might be able to assist you better
Ground
logo grille Mercedes-Benz logo
Back View
Front View
What displayed the TV?
What is displayed on the TV?
image simple model of two white cubes against background. is nothing displayed TV screen this image.
The image shows a Television displaying to be helicopter. The helicopter likely part video or a graphic being shown the screen.
appears
Ground Truth
The IV displaying captivating image of helicopter.
Figure 11: Front vs back results with LLaVA. The dataset contains many view-dependen questions.
D Additional baselines results and details
D.1 Language-only baseline
To assess potential spurious patterns in the question-answet annwerr relationships, we evaluate the performance of LLaMA 2, the LLM on which LLaNA relies, fine-tuned on ShapeNeRF-Ters evaluation protocol, the LLM is provided with questions belonging to the dataset and must return the correct answers without having access to the NeRF data. Therefore, the predicted answers may be generated only based on the textual patterns present in the training set. Results are shown in Tab. T2 significant performance gap exists between LLaMA 2 and L.ANN, highlighting that our dataset consists of questions that can only be answered with access to information about 3D objects.
this training and
23

---

Table 12: Language-only baseline.
Model
Modality
Sentence-BERr
SimCSE
BLEU-1
ROUGE-I.
METEOR
LLaMA2
Text-only
42.62
40.70
25.14
24.53
25.53
LLaNA-7b
NeRF
68.63
70.54
20.64
28.33
31.76
ILaMAZ
Text-only
49.73
47.68
15.15
23.27
14.78
LLaNA-7b
NeRF
77.43
79.81
41.32
36.18
32.39
LLaMA2
Text-only
68.37
68.46
44.07
51.15
48.00
LLaNA-7b
NeRF
81.03
81,56
46.16
53.17
50.15
D.2 Zero-Shot NeRF classification trained baselines
We report in Tab. 13 the results obtained zero-shot NeRF classification task by the baselines trained on ShapeNeRF-Texi. Results follow the same trend as the other language tasks, reported in the main paper.
Table 13: Zero-shot NeRF classification ShapeNeRF-Text. Trained baselines.
Best results are in bold, runner-up is underlined. (FV: front-view)
Model
Modality
Accuracy (%)
LLaVA-vicuma-133 13b
Image (FV)
36.49
MM-7t
Point cloud
49.69
Point cloud
26.30
LLaNA-7b
NeRF
67.14
E
Ground-truth images and point clouds
This section presents the results of an experiment in which the baseline 2D and 3D MLLMs have been provided with ground-truth input images and point clouds extracted from the original 3D meshes in the dataset rather than from the NeRFs. This scenario estimates an upper bound for the performance of such approaches when used as NeRF assistants, by simulating perfect extraction of images or point clouds from the NeRFs. In other words, simulates the ideal scenario in which the encoding of information inside NeRF is lossless, a non-realisite situation in which the baselines can achieve their best performance. Tab. [4, Tab. [5| and Tab. show the results of this experiments on the tasks of brief description, detailed description, and single-round Q&A, respectively. For brevity, the best-performing 2D model, i.c., LLaVA 44 (on front views) and the best- pest-performing 3D model, i.c., PointLLM 778), have been tested in this scenario. The results demonstrate that, in this idealized and most favorable scenario for the baselines, LLaNA outperforms them.
Model
Modality
Sentence-BERT
SimCSE
BLEU-1
ROUGE-L
METEOR
LIa AlaVA-vicuna-1.33 133b
Image (FV)
68.61
67.99
17.48
23.08
27.03
PointLLM-7b
Point cloud
51.99
51.70
17.19
18.63
15.03
LLaNA-7b
NeRF
68.63
70.54
20.64
28.33
31.76
Table 14: NeRF brief captioning on ShapeNeRF- dataset. Frozen baseline results obtained on data extracted from ShapeNet mesh data. Best results in bold. Runner-up underlined. (FV: front-view)
F Generalization experiments
conducted an experiment to probe the generalization capabilities LLaNA against the trained baselines. We evaluate the models on the subset of 200 Objaverse [13] objects with human-annotated captions used as a test set by PointLLMM (78.. This evaluation protocol sets forth challenging out-of-domain and open-set experiment (164 out of 200 Objaverse objects belong to categories not

---

Model
Modality
Sentence-BERT
SimCSE
BLEU-1
ROUGE-L
METEOR
LLaVA-vicuna-Il
Image (FV)
68.32
67.35
27.46
26.62
24.40
PointLLLM-7b
Point cloud
61.87
61.77
10.65
19.90
10.93
LLaNA-7b
NeRF
77.43
79.81
41.32
36.18
32.39
Table 15: NeRF detailed captioning on ShapeNeRF- dataset. Frozen baseline results obtained on data extracted from ShapeNet mesh data. Best results in bold. Runner-up underlined.
Model
Modality
Sentence-BERT
SimCSE
BLEU-1
ROUGE-L
METEOR
LLa VLaVVu-ieuna-
Image (FV)
78.40
75.68
22.65
33.04
35.70
PointLLM-7b
Point cloud
74.98
74.90
36.93
44.60
39.87
LLaNA-7b
NeRF
81.03
81.56
46.16
53.17
50.15
Table 16: NeRF single-round Q&A on ShapeNeRF-Texx dataset. Frozen baseline results obtained on data extracted from ShapeNet mesh data. Best results in bold. Runner-up underlined.
present in ShapeNeRF-Tewe Text). To test LLaNA, we fit NeRFs for all the objects the test set. Then, extracted colored point clouds and rendered front views from NeRFs, in order to test the baselines. In Tab. 17 we can observe that the scores of all models are significantly lower compared to Tab. 6 which hints at all models struggling when evaluated on objects very different from those belonging to the training domain. LLaNA achieves the second-best generalization performance after PointLLM. Yet, it is worth highlighting that the frozen modality-specific encoder PointLLM (and GPT4Point) is PointBERT, which was pre-trained on Objaverse. In contrast, LLaNA meta-encoder, nf2vec, has been trained only on ShapeNet, meaning it has never encountered objects outside the ShapeNet categories.
Table 17: Generalization results on Objaverse.
NeRF captioning
Model
Modality
Sentence-BERT
SimCSE
BLEU-1
ROUGE-L
METEOR
LLaVA-vicuna- 13h
Image (FV)
27.07
26.82
4.41
6.81
9.77
..i-
Point cloud
33.88
33.04
5.37
8.14
12.28
GPT4Point-Opt221
Point cloud
25.94
29.04
4.25
7.99
10.42
LLaNA-7b
NeRF
30.07
28.17
4.93
7.47
11.15
G Additional qualitative examples
This section provides additional qualitative comparisons between the proposed method, i.c. LLaNA which directly processes NeRF, and the baselines that take as input images 44] 39 or 3D representa- tions 78 58 In particular, Figs. 12 to 14show additional brief descriptions, detailed descriptions, and single-round Q&A provided as output by the different methods. Many examples, such as the white speaker in the third row of Fig. [3] are not described properly by MLLMS operating on point clouds. Indeed, due to the input point cloud containing only 8192 points, these methods cannot perceive the object details, such as the curved surface of the speaker, therefore they predict that the object is a "cubic white object" or an "ice cube". In other examples, such the white screen sample in the last row of Fig. 14 the LLM operating on images cannot give the right answer to the question the button location as it is not visible from the given viewpoint. Contrarily, by operating directly on the holistic representation provided by NeRFs, LLaNA provides the right answer in these situations.
H Information about datasets, models, and source code licenses
This section provides details about the datasets, models, and source code licenses used in the paper, ensuring proper credit to the creators or original owners, and adherence to license terms.
25

---

LLaNA
PointLLM
GPT4Paint
LLaVA
BUP-22
Figure 12: Additional qualitative examples for the brief description task.
LLaNA
PointLIM
GPT4Paint
LLaVA
BLIP-2
Figure 13: Additional qualitative examples for the detailed description task.
26

---

Frat THE EPINT PUSED
charrs bosc surtie
The dlall
Figure 14: Additional qualitative examples for the single-round Q&A task.
Datasets: the datasets employed in our work and the relative licenses are listed below:
ShapeNet: licensed under GNU Affero General Public License v3.0.
GPT2Shape HST: licensed under Creative Commons
ShareAlike 4.0 International License.
Models: the models used in all our experiments and their relative licenses are detailed in the following:
nf2vec: licensed under MIT License.
PointLLM: licensed under Creative Commons AttibutionNNannCommecaalllll Alike 4.0 International License.
GPT4Point:
licensed under Creative Commons Attrbbtion-NonCCommeriaSSS Alike
4.0 International License.
LLAMA-2: licensed under META LLAMA 2 COMMUNITY LICENSE AGREEMENT?
LLAMA-3: licensed under META LLAMA COMMUNITY LICENSE AGREEMENT!
LLAVV: licensed under Apache License 2.0.
Proper care has been taken to ensure that all licenses and terms of use are explicitly mentioned and respected throughout this paper.
https://ai www.. homma/aammmm/me/esseee./////
27

---

NeurIPS Paper Checklist
1. Claims
Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?
Answer: [Yes]
Justification: the claims made in the article are also highlighted in the abstract and introduc- tion.
Guidelines:
The answer NA means that the abstract and introduction do not include the claims made the paper.
The abstract and/or introduction should clearly state the claims made, including the contributions made in the paper and important assumptions and limitations. A No or
NA answer to this question will not be perceived well by the reviewers.
The claims made should match theoretical and experimental results, and reflect how much the results can be expected to generalize to other settings.
is fine to include aspirational goals as motivation as long as is clear that these goals are not attained by the paper.
2. Limitations
Question: Does the paper discuss the limitations the work performed by the authors?
Answer: [Yes]
Justification: the limitations of the proposed approach are detailed in Sec.
Guidelines:
The answer NA means that the paper has no limitation while the answer No means that the paper has limitations, but those are not discussed in the paper.
The authors are encouraged to create a separate "Limitations" section their paper.
The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g.. independence assumptions, noiseless settings, model well-specification, asymptotic approximations only holding locally). The authors should reflect how these assumptions might be violated in practice and what the implications would be.
The authors should reflect on the scope of the claims made, e.g., if the approach was only tested on few datasets or with few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.
The authors should reflect on the factors that influence the performance of the approach. For example, facial recognition algorithm may perform poorly when image resolution is low images are taken in low lighting. Or speech-to-text system might not be used reliably to provide closed captions for online lectures because it fails to handle technical jargon.
The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size.
If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and fairness.
While the authors might fear that complete honesty about limitations might used by reviewers as grounds for rejection, worse outcome might be that reviewers discover limitations that aren't 't acknowledged judgment and recognize that individual actions in favor of transparency play impor- tant role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed to not penalize honesty concerning limitations.
the paper. The authors should use their best
3. Theory Assumptions and Proofs
Question: For each theoretical result, does the paper provide the full set of assumptions and complete (and correct) proof?
Answer: [NA].
28

---

Justification:
the article does not introduce new theoretical results.
Guidelines:
The answer NA means that the paper does not include theoretical results.
All the theorems, formulas, and proofs in the paper should be numbered and cross- referenced.
All assumptions should be clearly stated or referenced
statement of any theorems.
The proofs can either appear in the main paper or the supplemental material, but they appear in the supplemental material, the authors are encouraged provide short proof sketch to provide intuition.
Inversely, any informal proof provided in the core of the paper should be complemented by formal proofs provided in appendix or supplemental material.
Theorems and Lemmas that the proof relies upon should properly referenced.
4. Experimental Result Reproducibility
Does the paper fully disclose all the information needed to reproduce the main perimental results of the paper to the extent that it affects the main claims and/or conclusions of the paper (regardless whether the code and data are provided or not)?
Question:
Answer: [Yes]
Justification: we provide all the information related to the training protocol used in our
experiments and the implementation details in Sec. B In addition, Sec. Dexeessiiii details the experimental setup used for all the evaluated tasks. Moreover, we provide additional details the appendix.
Guidelines:
The answer NA means that the paper does not include experiments. If the paper includes experiments, a No answer this question will not perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.
the contribution is a dataset and/or model, the authors should describe the steps taken to make their results reproducible or verifiable.
Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is a novel architecture, describing the architecture fully might suffice, or if the contribution a specific model and empirical evaluation, may be necessary to either make it possible for others to replicate the model with the same dataset, or provide access the model. In general. releasing code and data often one good way to accomplish this, but reproducibility can also be provided via detailed instructions for how to replicate the results, access to a hosted model (e.g., in the case of large language model), releasing of a model checkpoint, or other means that are appropriate to the research performed.
While NeurIPS does not require releasing code, the conference does require submis- sions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example
(a) If the contribution is primarily a new algorithm, the paper should make it clear how reproduce that algorithm.
(b) If the contribution is primarily
model architecture, the paper should describe the architecture clearly and fully.
(c) the contribution is a new model (e.g., a large language model), then there should either be way access this model for reproducing the results way to reproduce the model (e.g., with an open-source dataset or instructions for how to construct the dataset).
(d) We recognize that reprodueibility may be tricky in some cases, in which authors are welcome to describe the particular way they provide for reproducibility. In the case of closed-source models, it may be that access to the model is limited in some way (e.g., to registered users), but it should be possible for other researchers have some path reproducing verifying the results.
5. Open access to data and code
29

---

Does the paper provide open access to the data and code, with sufficient instruc- tions faithfully reproduce the main experimental results, as described in supplemental material?
Question:
Answer: [Yes]
Justification: all our experiments have been conducted on publicly available data, i.e., ShapeNet, HST and Objaverse datasets. Our newly introduced benchmark, the source code and the weights for all our models will be publicly released case of acceptance.
Guidelines:
The answer NA means that paper does not include experiments requiring code.
Please see the NeurIPS code and data submission guidelines (https://nips... cc/
for more details.
While we encourage the release of code and data, we understand that this might not be possible, so "No" an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (e.g., for new open-source benchmark).
The instructions should contain the exact command and environment needed to run to reproduce the results. See the NeurIPS code and data submission guidelines (https
/
www.c//wweeeeeeeee Saateees//ooSSmmiiiiaa SwwpeddeuumisssiooPeo. wwiiiisssiiiiiyiyioo for more details.
The authors should provide instructions on data access and preparation, including how access the raw data, preprocessed data, intermediate data, and generated data, etc.
The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. If only subset of experiments reproducible, they should state which ones are omitted from the script and why.
At submission time, to preserve anonymity, the authors should release anonymized versions (if applicable).
Providing as much information as possible in supplemental material (appended the paper) is recommended, but including URLs to data and code is permitted.
6. Experimental Setting/eetails
Question: Does the paper specify all the training and test details (e.g., data splits, hyper- parameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?
Answer: [Yes]
and in Justification: all implementation details and data splits detailed in Sec. Sec. 4 the appendix.
Guidelines:
The answer NA means that the paper does not include experiments.
The experimental setting should be presented in the core the paper to level of detail that is necessary to appreciate the results and make sense of them.
The full details can be provided either with the code, in appendix, as supplemental material.
7. Experiment Statistical Significance
Question: Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?
Answer: [No]
Justification: did not conduct multiple trials for each model training and evaluation due to the large computational requirements needed to fine-tune the LLMs employed in our approach.
Guidelines:
The answer NA means that the paper does not include experiments.
The authors should answer "Yes" if the results are accompanied by emoo bars, confi- dence intervals, or statistical significance tests, least for the experiments that support the main claims of the paper.
30

---

The factors of variability that the error bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall run with given experimental conditions).
The method for calculating the error bars should be explained (closed form formula, call to a library function, bootstrap, etc.)
The assumptions made should be given (e.g., Normally distributed errors).
It should be clear whether the error bar is the standard deviation the standard error of the mean.
is OK to report 1-sigma error bars, but one should state it. The authors should preferably report a 2-sigma error bar than state that they have 96% CI, if the hypothesis of Normality of errors is not verified.
For asymmetric distributions, the authors should be careful not to show in tables or figures symmetric error bars that would yield results that are out range (e.g. negative error rates).
If error bars reported in tables or plots, The authors should explain in the text how they were calculated and reference the corresponding figures or tables in the text.
8. Experiments Compute Resources
For each experiment, does the paper provide sufficient information on the com- puter resources (type of compute workers, memory, time of execution) needed to reproduce the experiments?
Question:
Answer: [Yes]
Justification: we provide details about the training time and computational resources required by our approach in Sec.
Guidelines:
The answer NA means that the paper does include experiments.
The paper should indicate the type of compute workers CPU or GPU, internal cluster, cloud provider, including relevant memory and storage.
The paper should provide the amount of compute required for each of the individual experimental rums as well as estimate the total compute.
The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g., preliminary or failed experiments that didn't make it into the paper).
9. Code Of Ethics
Question: Does the research conducted in the paper conform, in every respect, with the http:///wrrpp..cccpplppllee/eeaa
NeurIPS Code of Ethics
Answer: [Yes]
Justification: reviewed the guidelines listed in the NeurIPS Code of Ethics and confirm that our approach does not violate them.
Guidelines:
The answer NA means that the authors have reviewed the NeurIPS Code of Ethics.
the authors answer No, they should explain the special circumstances that require deviation from the Code of Ethics.
The authors should make to preserve anonymity (e.g., if there is special consid- eration due to laws or regulations their jurisdiction).
10. Broader Impacts
Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?
Answer: [NA]
Justification: we do not foresee any direct path to using our solution for negative applications it pertains describing digital twins of single objects.
Guidelines:
31

---

The answer NA means that there is no societal impact of the work performed.
If the authors answer NA or No, they should explain why their work has no societal impact or why the paper does not address societal impact.
Examples of negative societal impacts include potential malicious or unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairesss considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.
The conference expects that many papers will be foundational research and tied to particular applications, let alone deployments. However, if there is direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, is not needed to point out that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster.
The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology is being used as intended but gives incorrect results, and harms following from (intentional unintentional) misuse of the technology.
If there are negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition to attacks, mechanisms for monitoring misuse, mechanisms monitor how system learns from feedback over time, improving the efficiency and accessibility of ML).
11. Safeguards
Does the paper describe safeguards that have been put in place for responsible release of data or models that have a high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?
Question:
Answer: [Yes]
Justification: the LLaNA model will be released under the following terms of use, reported on the Github page of the released code: "By using this service, users are required agree to the following terms: The service is research preview intended for non-commercial use only. It only provides limited safety measures and may generate offensive content. It must not be used for any illegal, harmful, violent, racist, sexual purposes. The service may collect user dialogue data for future research."
Guidelines:
The answer NA means that the paper poses such risks.
Released models that have high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere usage guidelines or restrictions to access the model or implementing safety filters.
Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.
We recognize that providing effective safeguards challenging, and many papers do not require this, but we encourage authors to take this into account and make best faith effort.
12. Licenses for existing assets
Question: Are the creators or original owners of assets (e.g., code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected?
Answer: [Yes]
Justification: in Appendix H of the supplemental material, we provide details about the licenses for: (i) the large language models used in our approach, (ii) the employed source codes and (iii) the datasets used in all our experiments.
Guidelines:
The answer NA means that the paper does not use existing assets.
32

---

The authors should cite the original paper that produced the code package or dataset.
The authors should state which version of the asset is used and, if possible, include URL.
The name of the license ge.g.. CC-BY 4.0) should be included for each asset. For scraped data from a particular source (e.g., website), the copyright and terms of service of that source should be provided.
assets are released, the license, copyright information, and terms use in the package should be provided. For popular datasets, pappessiithheeeocc thoode com datasets has curated licenses for some datasets. Their licensing guide can help determine the license of dataset.
For existing datasets that are re-pane re-packaged, both the original license and the license of the derived asset (if it has changed) should be provided.
If this information is not available online, the authors are encouraged to reach out to the asset's s creators.
13. New Assets
Question: Are assets introduced in the paper well documented and is the documentation provided alongside the assets?
Answer: [Yes]
Justification: the dataset defined for our benchmark will be made publicly available, in case of acceptance, together with the documentation required for reproducing the experiments. Moreover, in case of acceptance, we will also release the source code of our model.
Guidelines:
The answer NA means that the paper does not release new assets.
Researchers should communicate the details of the dataset/code/model as part of their submissions via structured templates. This includes details about training, license, limitations, etc.
The paper should discuss whether and how consent was obtained from people whose asset is used.
At submission time, remember to anonymize your assets (if applicable). You can either create an anonymized URL or include an anonymized zip file.
Crowdsourcing and Research with Human Subjects
Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well details about compensation (if any)?
Answer: [NA]
Justification: the paper does not involve crowdsourcing nor research with human subjects.
Guidelines:
The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.
Including this information in the supplemental material is fine, but if the main contribu- tion of the paper involves human subjects, then as much detail as possible should be included the main paper.
According to the NeurIPS Code of Ethics, workers involved data collection, curation, or other labor should be paid at least the minimum wage in the country of the data collector.
15. Institutional Review Board (IRB) Approvals or Equivalent for Research with Human Subjects
Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or an equivalent approval/review based on the requirements of your country or institution) were obtained?
Answer: [NA]
33

---

Justification:
the paper does not involve crowdsourcing nor research with human subjects.
Guidelines:
The answer NA means that the paper does not involve crowdsoureing nor research with
human subjects.
Depending the country in which research is conducted, IRB approval (or equivalent) may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.
recognize that the procedures for this may vary significantly between institutions and locations, and we expect authors to adhere to the NeurlPS Code of Ethics and the guidelines for their institution.
For initial submissions, do not include any information that would break anonymity (if applicable), such the institution conducting the review.
34


Reviews:
[{"review_id": "RERADwoc3w", "reviewer": "Reviewer_W4u1", "summary": "This work proposes LLaNA, a multimodal large language model (MLLM) that is capable of aligning language with 3D scene fields embedded with (learned) NeRF models. By projecting the NeRF fields into an LLM’s embedding space, LLaNA utilizes a meta encoder to transform fields seamlessly into the token space of an LLM (Llama model in this work) and hence can be adapted to downstream higher-level reasoning tasks (centered around 3D understanding). This work also features a newly created dataset called NeRF-language, which has the emphasis on leveraging NeRF features in answering various visual questions.", "questions": "- Are there any questions that exactly depend on the variant of the views where the NeRF fields would benefit?\n- And if so, what is the detailed percentage analysis of these types of questions, and what do they look like in general?", "limitations": "- The authors addressed some limitations of the work in the manuscript.", "rating": 5, "confidence": 4, "soundness": 2, "presentation": 3, "contribution": 2, "strengths": "- The proposed LLaNA method that directly projects a learned radiance field weights into LLM’s space is an interesting idea.\n- The training on multi-turn 3D-aware question answering style makes good use of both fields and LLM’s capabilities.\n- A new challenge is proposed with the emphasis on multi-turn QAs on 3D-awareness.", "weaknesses": "- A more detailed statistical analysis of the dataset is required, such as detailed sizes, diversities of the questions, human performance or judgment on the dataset, type-token ratios, and some frequent word analysis.\n- The questions introduced in NeRF-language do not seem to require a very thorough understanding of 3D scenes.\n- There is an increasing amount of work aimed at marrying the benefits of radiance fields with the strong ability of LLMs, just to name a few [1,2,3]. How does this work compared to them?\n- Additionally, the 3D-LLM [1] also presents question answering capabilities from 3D input scenes, at least a comparison of the LLaNA method with it is required.\n- Minor but not of least importance: additional standard VQA baselines can provide more in-depth analysis on the dataset, as well as language-only baseline models to gauge the potential spurious patterns of the curated question-answer relationships.\n\n[1] Hong, Yining, et al. \"3d-llm: Injecting the 3d world into large language models.\" NeurIPS 2023.\n\n[2] Kerr, Justin, et al. \"Lerf: Language embedded radiance fields.\" ICCV 2023.\n\n[3] Qin, Minghan, et al. \"LangSplat: 3D Language Gaussian Splatting.\" CVPR 2024.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "__W1__\n\nThe proposed ShapeNeRF-Text is built upon the 3D object dataset ShapeNet. ShapeNeRF-Text has 13 classes of ordinary objects, such as cars, airplanes, and chairs. The shapes are divided into 30939, 3846, and 3859 for the train, val, and test set. A NeRF is trained for each of these objects. Regarding the text, we have a brief and detailed description, three single-rounds and a three multi-round QA for each object.\n\nThe average lengths in words of the instructions/responses are 8.81/14.25 for single-round QAs, 8.80/14.14 (per round) for multi-round QAs, 8.51/22.76 for brief descriptions, and 7.82/77.90 for detailed descriptions. We report instruction/response length histograms in __Figure 1__ - bottom (rebuttal PDF; see global response).\n\n__Figure 1__ - top (rebuttal PDF) shows word clouds obtained after removing generic words like “model,” “object,” and “NeRF”, emphasizing frequent words in the detailed description instruction and response texts.\n\nFinally, we would like to point the reviewer to Appendix C, where we included several details on ShapeNeRF-Text.\n\n__W2 - Q1 - Q2__\n\nAkin PointLLM and GPT4Point, our paper focuses on an object-centric setup rather than scenes. The proposed ShapeNeRF-Text dataset contains many questions that require a holistic understanding of 3D objects. We prove that with the following two experiments.\n\nFirst, we evaluate our dataset questions with an LLM (LLaMA3). For each question $Q$, we ask LLaMA3:\n\n\n_Is a random viewpoint of the object enough to answer this question?_\n\n$Q$ \n\n_If so, reply \"YES\"; if a specific viewpoint is needed, answer \"NO\"_\n\nBy doing so, we obtained 5163 “YES” and 5847 “NO”, highlighting that most questions require multi-view information to be answered correctly.\n\nSecond, we run a Vision-Language model, LLaVA13b, on each question of the single round QA dataset on the front and back views of objects. Then, we select only the LLaVA responses, where the answer for the front or back view achieves a SimCSE score higher than 80%, i.e., likely correct answers, which selects approximately 45% of the answers. Among these correct responses, we calculate the percentage of those where the front and back answers are extremely different (i.e., a difference in SimCSE scores > 10). Remarkably, 26 % of answers are correct from one point of view but wrong from the other: these questions would have required multi-view information to be answered correctly. We report two qualitative examples in __Figure 2__. In the first row, the Mercedez-Benz logo cannot be recognized from the back view. In the second row, the monitor seems turned off, and thus it is not possible to identify correctly the helicopter displayed on the screen. Similarly, __Figure 11__ of the Appendix shows other examples of this kind of QA.\n\n__W3 - W4__\n\nLeRF [2] ([32] in our paper) and LangSplat [3] are innovative representations of 3D objects and scenes. They extend the radiance field formulation, considering functions that model density, color, and language features at each spatial coordinate. These _language_ fields are parameterized by either a neural network (LeRF) or a set of 3D Gaussians (LangSplat). We believe that rather than competing with LLaNA -- a multimodal LLM that considers NeRFs as an input modality -- [2] and [3] propose field-based representations semantically richer than standard NeRFs.\n\n3D-LLM [1] ([24] in our paper) is a multimodal LLM that processes a colored mesh or a set of posed images. Thus, like PointLLM and GPT4Point, it can be applied to data extracted from NeRFs. In our experiments, we did not include it among the baselines because 3D-LLM addresses scenes and scene-specific tasks, such as 3D grounding and navigation. On the contrary, LLaNA focuses on object-centric scenarios, like PointLLM and GPT4Point.\n\nYet, as requested by the reviewer, we have evaluated 3D-LLM on our ShapeNeRF-Text dataset to compare its performance to LLaNA. Purposely, we have extracted colored 3D meshes from the NeRFs belonging to the test set of ShapeNeRF-Text and processed these data by the official 3D-LLM code to render the images and compute both the 2D and 3D features required by the model at inference time. The results are reported in __Table 4__ (Rebuttal PDF). We notice that 3D-LLM provides performance somehow comparable to PointLLM (see __Tables 1, 3, 4, and 5__ of the paper). Yet, as highlighted by the last row of __Table 4__ (Rebuttal PDF), LLaNA performs much better on all tasks.\n\nMoreover, we highlight that 3D-LLM has several disadvantages compared to LLaNA.\n1) 3D-LLM requires several components to extract 3D language features which are the input of their LLM. Indeed, it requires a pre-trained vision-language feature extractor (CLIP), a general-purpose semantic segmentation network (SAM), and a way to handle noisy 3D data (gradSLAM) to project language features in the 3D space. Conversely, LLaNA extracts NeRF features by a single forward pass of the meta-encoder, which takes just a few ms on a GPU.\n2) As it operates on meshes and images, 3D-LLM does not scale well with the resolution of the input signal. Conversely, the number of NeRF weights is decoupled from the spatial resolution of the underlying signal, making LLaNA resolution-agnostic. \n3) In our scenario where NeRF is the input modality, we must extract a 3D mesh and render N views to apply 3D-LLM. However, as already stated in the paper (see L53-57), this process is not trivial and time-consuming. Moreover, important details might be lost when the extracted geometry from NeRF is too noisy or low resolution.\n\n__W5__ \n\nTo assess potential spurious patterns in the question-answer relationships, we evaluated the performance of LLaMA 2 fine-tuned on ShapeNeRF-Text (__Table 5__ rebuttal PDF). There is a significant performance gap between LLaMA2 and LLaNA, highlighting that our dataset consists of questions that can only be answered with access to information about objects.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "18yRyxks1E", "reviewer": "Reviewer_ZJvT", "summary": "The authors propose a method to ingest NeRFs and project them into a language model's latent space for question answering and chat applications on NeRFs directly.", "questions": "See weaknesses for specific questions.", "limitations": "Yes the authors address limitations in a specific section.", "rating": 6, "confidence": 4, "soundness": 2, "presentation": 3, "contribution": 3, "strengths": "S1. Very clear abstract, which nicely frames the paper\n\nS2. Strong related work section\n\nS3. The proposed dataset may be useful to others working on related investigations\n\nS4. Many different (standard) metrics considered\n\nS5. As NeRFs become a more standard 3D representation, dealing with them in the context of chat applications is increasingly relevant and forward thinking", "weaknesses": "W1. Consider adding basic statistics about the proposed dataset in the intro (e.g., the dataset size in number of samples).\n\nW2. Consider adding information about the kind of \\emph{generalization} that can be expected from the method. Can the method generalize to novel classes? To different instances of fixed classes with held out attributes, colors etc.?\n\nW3. ShapeNet seems like a good starting point, but datasets like Objaverse and Objaverse-XL may be interesting to probe generalization. How does the proposed method, trained on ObjectNet generalize to more complex NeRFs?\n\nW4. I dont quite follow the addition on L141. Consider giving a more high-level description of this and deferring details to the Appx.\n\nW5. L181-182 about retaining safety of the model reads an unsupported claim. Either point to an experiment to this effect, run this experiment, or remove this claim.\n\nW6. Figure 3 does not stand alone. Just looking at it I am not able to understand what I should take away. Consider adding more information to the caption so it is clear how to interpret the figure and what the takeaway is.\n\nW7. I am not 100% clear on how the training dataset is split in L216. Are there some held out classes? Or is there some data that is held out per class? \n\nW8. The proposed method uses fine-tuning while the baseline methods do not apply fine-tuning. Hence, I do not consider these fair comparisons calling into question the validity of the baselines and the gains of the prosed method for NeRFs over traditional 3D and 2D input representations. Adding fine-tuned models as baselines seems important to contextualize the results.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "__W1__\n\nShapeNeRF-Text is built upon ShapeNet. 3D shapes are divided into 30939, 3846, and 3859 for the train, val, and test set, respectively. A NeRF is trained for each of these objects. As for text, the dataset features a brief and detailed description and 3 single-rounds and one three-rounds QA for each object. We computed word clouds and the instruction/response length statistics, partially shown in __Figure 1__ (rebuttal PDF; see global response). We will add basic details on the dataset size in the intro and all the details in the Appendix.\n\n__W2-W3__\n\nTo address __W8__ (see below), we have trained all the baselines considered in our paper on ShapeNeRF-Text (__Table 1__, Rebuttal PDF).  To train the baselines, we have used the official training code, which, for all of them, keeps the modality-specific encoder frozen and trains in two steps both an adaptor and the LLM. Thus, to probe generalization, we have evaluated LLaNA and the trained baselines on the subset of 200 Objaverse objects with human-annotated captions used as a test set in the PointLLM paper [77]. This sets forth a pretty challenging out-of-domain and open-set experiment (164 out of the 200 Objaverse objects belong to categories not present in ShapeNet).  To conduct our experiments, we created NeRFs for all 200 objects. Then, we extracted the point clouds and rendered front views from NeRFs to test the baselines. The results are in __Table 4__ (Rebuttal PDF). We can observe that the scores of all models are significantly lower compared to __Table 1__ (Rebuttal PDF), which hints at all models struggling when evaluated on objects very diverse to those belonging to their training domain. LLaNA achieves the second-best generalization performance after PointLLM. Yet, it is worth highlighting that the frozen modality-specific encoder of PointLLM (and GPT4Point) is PointBERT, which has been trained on Objaverse. In contrast, LLaNA meta-encoder, i.e, nf2vec, has been trained only on ShapeNet and thus has never seen any object outside the ShapeNet categories.\n\n__W4__\n\nThe addition shows how to calculate the number of rows of the input matrix $M$ obtained by stacking the NeRF’s weights. The number of rows of $M$ depends on the number of hidden layers, $L$, the number of units per hidden layer, $H$, and the dimension of the input, which is a $144$-dimensional array obtained by frequency encoding of 3D coordinates. We will move the detailed calculation in the Appendix.\n\n__W5__\n\nWe apologize for any confusion. We meant that, as our ShapeNeRF training dataset was generated using LLaMA's automated text creation, which includes built-in safeguards, and as ShapeNet is a manually curated dataset with only common items such as chairs and cars, our model trained on these data should also be safe. However, we acknowledge that we have not conducted any experiments to validate this claim and will remove that statement.\n\n__W6__\n\nThe image represents the automatic pipeline used to create the ShapeNeRF dataset. Given a 3D model corresponding to a NeRF, we render $N$ views by a computer graphic engine. Each view is then processed by a VLM, i.e. LLaVA, obtaining $N$ view-specific captions. These captions are finally aggregated by an LLM, i.e., LLaMA 3, to obtain brief and detailed descriptions and single-round and multi-round Q&As. We will add this information to the caption. \n\n__W7__\n\nThe proposed ShapeNeRF-Text dataset features 13 classes. Following [61], the train, val and test splits are obtained by randomly sampling objects within each class, i.e., holding out a fixed percentage of objects per class.  So, there are no held-out classes. \n\n__W8__\n\nOur choice of using frozen models as baselines has been motivated by LLaNA being the only assistant that works on NeRFs, i.e. whose input modality is a radiance field parametrized as a neural network. Indeed, the papers most closely related to ours -- those proposing object-centric assistants that ingest point clouds (PointLLM, GPT4Point) -- use frozen VLMs as baselines operating on a different modality (i.e. images). \n\nNevertheless, we agree with the reviewer on the importance of adding experiments with baselines trained on ShapeNeRF-Text. Thus, we trained all the baselines on ShapeNeRF-Text following their official protocol and using the official training code (see also answer to __W3__).  We report the results of the new experiments alongside those already included in the paper (Frozen baseline models) in __Table 1 and 2__ (Rebuttal PDF).  We notice that the trained baselines exhibit different behaviors than their frozen counterparts, with LLaVA performing significantly worse and PointLLM showing clear improvements.  As for GPT4Point, we observe more variability across metrics, though, overall, we are led to reckon that it does not benefit from training on ShapeNerf-Text.  The last row in __Tables 1 and 2__ shows how LLaNA performs best compared to all baselines, frozen or trained on ShapeNerf-Text.  \n\nEventually, we highlight that the new experiments fortify the key finding of our paper: directly processing the weights of a NeRF is the most effective way to reason about the underlying radiance field. Moreover, converting a NeRF into a different modality, e.g. a point cloud or an image or set of images, is inefficient and cumbersome (see A.3, main paper). For instance, extracting a point cloud from a NeRF mandates voxelizing the 3D space to evaluate the density function, the required computation scaling cubically with the desired resolution. Images are also not easy to handle, as NeRFs do not come with an object-centered 3D reference frame, so one would need to guess from which viewpoint(s) to render meaningful images. Obtaining the rendering(s) can be pretty slow, especially if high-resolution images are needed to capture important object details. Yet, rephrasing our key finding, it is not worth undertaking any costly NeRF-to-X conversion, as processing a NeRF as a NeRF is more effective.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "CNsi1gdiLu", "reviewer": "Reviewer_aHSF", "summary": "In this submission the authors propose a new pipeline that enables Large Language models to interact with trained object-centric NeRF models. They achieve this by utilising a pretrained meta-network that ingests the weights of a NeRF MLP and outputs a low-dimensional feature vector. This low-dimensional feature vector is then transformed to the language models input space via a projection layer similar to LLaVA. The method is trained on a dataset of 40K NeRF models which the authors extend with 240K paired text descriptions. Experiments on this dataset show favourable performance compared to prior works that solely work with images or point clouds.", "questions": "To focus the discussion about the weaknesses raised above here are some questions for the authors: \n\n1. Please further justify not finetuning the baselines on the proposed dataset. How is this evaluation done in prior work and what justifications do they give for why they did or did not do this? Do these arguments also apply to this submission?\n2. Is it possible to train a limited amount of NeRF models on Objaverse or ModelNet40 and then use the proposed pipeline without finetuning on them and compare to PointLLM and GPT4Point? This would be a very interesting experiment to showcase the generalisation ability of the proposed method. \n3. Can you evaluate the 2D baselines with more views ? (related to Weakness 2)", "limitations": "The authors addressed the technical limitations of their work quite well but limitations with regards to prior work are not discussed, as they do not exist in the current evaluation scheme.\nSocietal Impact is briefly discussed in the paper checklist.", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 4, "contribution": 3, "strengths": "1. The proposed problem and method of interacting with NeRFs via a LLM is novel and very interesting, enabling potential new applications in robotics and AR. \n2. The experiments performed in the submission show promising results, where the proposed method is achieving impressive results over the chosen baselines. \n3. The dataset/benchmark and code will be released upon acceptance, which will make follow-up works and comparisons much easier. \n4. The paper is very well written and concepts are easy to understand.", "weaknesses": "In order of severity: \n1. The comparisons that were performed in the experiments seem unfair towards the baselines. The baselines are not trained or fine-tuned on the dataset proposed in the submission and it is assumed that they would generalise on this new domain since they were trained on millions of images or hundreds of thousands of 3D shapes. Since the proposed dataset consists only of ShapeNet objects there will be a domain gap for the baselines to overcome, since they were trained on the Objaverse (Deitke, Matt, et al. \"Objaverse: A universe of annotated 3d objects.\" CVPR 2023)  or ModelNet40 (Wu, Zhirong, et al. \"3d shapenets: A deep representation for volumetric shapes.\" CVPR 2015) Datasets in the case of the GPT4Point[58] and PointLLM[77]. The same logic applies to the BLIP2 and LLaVA baselines, which will not have seen many synthetic object centric images during their training. This weakens the overall argument to consider NeRFs as an input modality to LLMs because the presented comparisons are flawed.\n2. While it is a valid argument that view selection is an issue for 2D-LLMs, as stated in line 123, NeRFs can render arbitrary viewpoints after training. It would therefore be possible to render multiple images from varying viewpoints and use all of them as input to modern Multimodal LLMs jointly, e.g. by concatenating their text-tokens after the projection layer. This would provide a more balanced comparison than the single-view baselines that are chosen in the submission, since a lot more information can be passed onto the LLM with multiple views. Another avenue to make a fair comparison to 2D-LLMs like LLaVA would be to encode 2D images in an MLP (as shown for example in: Sitzmann, Vincent, et al. \"Implicit neural representations with periodic activation functions.\" Advances in neural information processing systems 33 (2020): 7462-7473.) and then use these weights as input to the proposed method. Both of these points will probably be infeasible to address in a rebuttal but would be interesting experiments to support the use of implicit representations as input to LLMs. \n3. It is a bit unclear from the description in line 215 if the test set contains only object classes not seen during training or if they were seen before. This should be clarified. \n4. In table 5 it is not discussed what a ‘hard’ view is and what constitutes its ‘hardness/complicatedness’. This makes the table confusing and not self-explanatory. \n5. Stating in line 90 that Ballerini et al. [5] are the first to utilise NeRFs as an input modality is not correct, what about NeSF (Vora, Suhani, et al. \"Nesf: Neural semantic fields for generalizable semantic segmentation of 3d scenes.\" arXiv preprint arXiv:2111.13260 (2021)) or Nerf-RPN (Hu, Benran, et al. \"Nerf-rpn: A general framework for object detection in nerfs.\" CVPR 2023)? It probably refers to being the first to directly utilise NeRF MLP weights as an input modality when considering language tasks. This sentence should be re-written.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "__W1-Q1__\n\nOur choice of using frozen models as baselines has been motivated by LLaNA being the only assistant that works on NeRFs, i.e. whose input modality is a radiance field parametrized as a neural network. Indeed, the papers most closely related to ours -- those proposing object-centric assistants that ingest point clouds (PointLLM, GPT4Point) -- employ frozen VLMs as baselines operating on a different modality (i.e., images). \n\nNevertheless, we agree with the reviewer on the importance of complementing our evaluation with experiments to assess the performance of the baselines trained on ShapeNerf-Text. Thus, we trained all the baselines with their official code on ShapeNerf-Text following their official protocol, which, for all of them, keeps the modality-specific encoder frozen and trains an adaptor and the LLM in two steps.\n\nWe report the results of the new experiments alongside those already included in the paper (Frozen baseline models) in __Table 1 and 2__ (rebuttal PDF; see global comment).  We notice that the trained baselines exhibit different behaviors wrt their frozen counterparts, with LLaVA performing significantly worse and PointLLM showing clear improvements.  As for GPT4Point, we observe more variability across metrics, though, overall, we are led to reckon that it does not benefit from training on ShapeNerf-Text.  The last row in both __Table 1 and 2__ points out how LLaNA yields the best performance compared to all baselines, either frozen or trained on ShapeNerf-Text. \n\nEventually, we highlight that the new experiments fortify the key finding of our paper: directly processing the weights of a NeRF is the most effective way to reason about the underlying radiance field. Moreover, converting a NeRF into a different modality, e.g. a point cloud or images, is inefficient and cumbersome (see A.3, main paper). For instance, extracting a point cloud from a NeRF mandates voxelizing the 3D space to evaluate the density function, the required computation scaling cubically with the desired resolution. Images are also not easy to handle, as NeRFs do not come with an object-centered 3D reference frame, so one would need to guess from which viewpoint(s) to render meaningful images. Obtaining the rendering(s) can be pretty slow, especially if high-resolution images are needed to capture important object details. Yet, rephrasing our key finding, it is not worth undertaking any costly NeRF-to-X conversion, as processing a NeRF as a NeRF is more effective.\n\n__Q2__\n\nTo assess the generalization ability of LLaNA, we trained NeRFs for the subset of 200 Objaverse objects with human-annotated captions used as a test set in the PointLLM paper [77].  This sets forth a challenging out-of-domain and open-set experiment (164 out of the 200 Objaverse objects belong to categories not present in ShapeNet).  We could also perform a comparative evaluation as we trained the baselines on ShapeNerf-Text (see __W1-Q1__). Purposely, we extracted point clouds and rendered front views from the 200 Objaverse NeRFs. Results are reported in __Table 4__ (Rebuttal PDF). We can observe that the scores of all models are significantly lower compared to __Table 1__ (Rebuttal PDF), which hints at all models struggling when evaluated on objects very diverse to those belonging to the training domain. LLaNA achieves the second-best generalization performance after PointLLM. Yet, it is worth highlighting that the frozen modality-specific encoder of PointLLM (and GPT4Point) is PointBERT, which has been trained on Objaverse, while the meta-encoder used by LLaNA, i.e nf2vec, has been trained only on ShapeNet and thus has never seen any object outside such categories. We also wondered whether the comparison to PointLLM and GPT4Point suggested by the reviewer concerned frozen versions of these models.  So, we conducted this experiment and found out that both frozen models provide better scores than LLaNA on the considered 200 Objaverse objects. For example, PointLLM and GPT4Point yield a Sentence-BERT equal to 38.09 and 33.61, respectively, while the score for LLaNA is 30.07.  We consider these results reasonable because the frozen models have been trained on Objaverse, so their performance deals with in-domain data. Yet, the gap between GPT4Point and LLaNA, for which this experiment implies reasoning on out-of-domain data, is relatively limited.\n\n__W2 – Q3__\n\nFollowing the reviewer's suggestion, we realized a multi-view baseline by rendering images from N viewpoints randomly chosen between the set of camera poses used to train the given NeRF. Then, we concatenated tokens from the N images and fed them into LLaVA alongside text instructions.  We set N=3 because the model cannot process correctly a higher number of images. Results are in __Table 4__ (Rebuttal PDF). As for all reasoning tasks, we observe a slight improvement of LLaVA in the multi-view set-up. Yet, LLaNA keeps outperforming LLaVA by large margins. Conversely, using multiple images boosts the zero-shot classification performance of LLaVa, which turns out to be the best model for this task. \n \n\n__W3__\n\nThe dataset does not contain classes not seen during training. Indeed, it features 13 classes, and the train, val, and test splits are obtained by randomly sampling objects within each class, i.e., holding out a fixed percentage of objects per class (80%, 10%, and 10% for train, val, and test, respectively).  \n\n__W4__\n\nWe apologize for the typo in __Table 5__; we should have used “Back View,” which has the same meaning as in __Tables 1 to 4__.\n\n\n__W5__\n\nRegarding NeSF, it proposes a novel semantic field representation of 3D scenes rather than seeing neural fields as an input modality. On the other hand, we agree with the reviewer that NeRF-RPN also considers NeRFs as an input modality. Therefore, we agree that the sentence at L20 needs rewriting, and we will modify it.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:
We thank the reviewers for their valuable feedback, which allowed us to carry out a more extensive investigation, improving the quality of our work. 

We reported the results of these extensive experiments in the attached PDF, referred to as the “__rebuttal PDF__”, which we believe provides a clearer picture of the merits of our proposal. 

We also highlight that, in case of acceptance, we will include all new tables, figures, and considerations, including discussion on the related papers pointed out by the reviewers, in the revised manuscript, either in the main paper or in the Appendix.

Meta-review: This submission received very positive reviews from three reviewers. The authors did a good job in the rebuttal and addressed all of the concerns from all the reviewers. The AC agreed with the reviewers' comments and recommendations and decided to accept this submission. The authors need to include all the new materials/comments in the final camera-ready submission.
Decision: Accept (poster)