3D Concept Grounding on Neural Fields
Authors: Yining Hong
Yilun Du
Chunru Lin
Joshua B. Tenenbaum
Chuang Gan
Venue: NeurIPS 2022 Accept (2022)
Keywords: 3D Visual Reasonng
Neural Fields
Concept Grounding
Neural Implicit representations

Abstract: In this paper, we address the challenging problem of 3D concept grounding (i.e., segmenting and learning visual concepts) by looking at RGBD images and reasoning about paired questions and answers. Existing visual reasoning approaches typically utilize supervised methods to extract 2D segmentation masks on which concepts are grounded. In contrast, humans are capable of grounding concepts on the underlying 3D representation of images. However, traditionally inferred 3D representations (e.g., point clouds, voxelgrids and meshes) cannot capture continuous 3D features flexibly, thus making it challenging to ground concepts to 3D regions based on the language description of the object being referred to. To address both issues, we propose to leverage the continuous, differentiable nature of neural fields to segment and learn concepts. Specifically, each 3D coordinate in a scene is represented as a high dimensional descriptor. Concept grounding can then be performed by computing the similarity between the descriptor vector of a 3D coordinate and the vector embedding of a language concept, which enables segmentations and concept learning to be jointly learned on neural fields in a differentiable fashion.  As a result, both 3D semantic and instance segmentations can emerge directly from question answering supervision using a set of defined neural operators on top of neural fields (e.g., filtering  and counting). Experimental results show that our proposed framework outperforms unsupervised / language-mediated segmentation models on semantic and instance segmentation tasks, as well as outperforms existing models on the challenging 3D aware visual reasoning tasks. Furthermore, our framework can generalize well to unseen shape categories and real scans. 

3D Concept Grounding on Neural Fields
Yining Hong
Yilun Du Massachusetts Institute of Technology
University California, Los Angeles
Chunru Lin Shanghai Jiao Tong University
Joshua B. Tenenbaum
MIT BCS, CBMM, CSAIL
Chuang Gan
UMass Amherst and MIT-IBM Watson Al Lab
Abstract
In this paper, we address the challenging problem of 3D concept grounding (i.e. seg- menting and learning visual concepts) by looking at RGBD images and reasoning about paired questions and answers. Existing visual reasoning approaches typically utilize supervised methods to extract 2D segmentation masks which concepts are grounded. In contrast, humans are capable of grounding concepts on the underlying 3D representation of images. However, traditionally inferred 3D representations (e.g.. point clouds, voxelgrids and meshes) cannot capture continuous 3D features flexibly, thus making challenging to ground concepts to 3D regions based the language description of the object being referred to. To address both issues, we propose to leverage the continuous, differentiable nature of neural fields segment and learn concepts. Specifically, each 3D coordinate in a scene represented high dimensional descriptor. Concept grounding can then be performed by computing the similarity between the descriptor vector of 3D coordinate and the vector embedding of a language concept, which enables segmentations and concept learning to be jointly learned on neural fields in a differentiable fashion. As a result, both 3D semantic and instance segmentations can emerge directly from question answering supervision using a set of defined neural operators top of neural fields (e.g., filtering and counting). Experimental results on our collected MARTNET-RFASONE dataset show that our proposed framework outperforms unsupervised language-mede mediated segmentation models on semantic and instance segmentation tasks, as well as outperforms existing models on the challenging 3D aware visual reasoning tasks. Furthermore, our framework can generalize well to unseen shape categories and real scans".
Introduction
Visual reasoning, the ability to utilize compositional operators to perform complex visual question answering tasks (e.g., counting, comparing and logical reasoning). has become a challenging problem these years since they go beyond pattern recognition and bias exploitation [19, 22, 2]. Consider an image of a table pictured Figure 1. We wish to construct a method that is able to accurately ground the concepts and reason about the image such the number of legs the pictured table has. Existing works typically address this problem by utilizing a supervised segmentation model for legs, and then applying count operator extracted segmentation masks [28]. However, illustrated in Figure 1, in many visual reasoning problems, the correct answer depends on a very small portion of
PProecct page: http://3ddcgg http:://ddcc..cco.
catpaaccc.cc........oo..mee csail. mit.edu
36th Conference on Neural Information Processing Systems (NcurIPS 2022).

---

Shape Question
Segmentation
Answer
Shape & Question
Segmentation
Answer
Answer: 31
Answer:
Question:
x
Question:
X
How many legs does this table have?
How many wheels
Segment Legs
Segment Wheels
does this cart have?
Figure I: Issues with 2D Concept Grounding and Visual Reasoning, Existing methods typically answer questions by relying on 2D segmentation masks obtained from supervised models. However, heavy occlusion leads to incorrect segmentations and answers.
given image, such highly occluded back leg, which many existing 2D segmentation systems may neglect. Humans are able to accurately ground the concepts from images and answer such questions by reasoning the underlying 3D representation of the image [43]. In this underlying 3D representation, the individual legs of a table are roughly similar in size and shape, without underlying issues of occlusion of different legs, making reasoning significantly easier and more flexible. In addition, such an intermediate 3D representation further abstracts confounding factors towards reasoning, such the underlying viewing direction from which we see the input shape. To resemble more human-lin like reasoning capability, in this paper, we propose novel concept grounding framework by utilizing an intermediate 3D representation.
natural question arises what makes a good intermediate 3D representation for concept grounding? Such an intermediate representation should be discovered in a ceakly-supervieed manner and efficient to infer, as well as maintain correspondences between 3D coordinates and feature maps in a fully differentiable and flexible way, so that segmentation and concept learning can directly emerge from this inferred 3D representation with supervision of question answering. While traditional 3D representations (e.g.. point clouds, voxelgrids and meshes) are efficient to infer, they can not provide continuous 3D feature correspondences flexibly, thus making it challenging to ground concepts to 3D coordinates based on the language descriptions of objects being referred to. To address both issues, in this work, we propose to leverage the continuous, differentiable nature of neural fields as the intermediate 3D representation, which could be easily used for segmentation and concept learning through question answering.
To parameterize neural field for reasoning, utilize recently proposed Neural Descriptor Fields (NDFs) [41] which assign each 3D point in a scene a high dimensional descriptor. Such descriptors learned in weakly-supervised manner, and implicitly capture the hierarchical nature of scene, by leveraging implicit feature hierarchy learned by a neural network. Portions of scene relevant to particular concept could be then differentiably extracted through vector similarity computation between each descriptor and corresponding vector embedding of the concept, enabling concept segmentations on NDFs be differentiably learned. In contrast, existing reasoning approaches utilize supervised segmentation models to pre-detined concepts, which prevents reasoning on concepts unseen by the segmentation model [28], and further prevents models from adapting segmentations based on language descriptions.
On top of NDFs, define set of neural operators for visual reasoning. First, construct filter operator, and predict the existence of a particular concept in an image. We also have query operator which queries about attribute of the image. In addition, we define neural counting operator, which quantifies the number of instances of a particular concept.
To evaluate the performances of 3D Concept Grounding (3D-CG) framework, we conduct experiments on our collected PARTNET-REASONE images of 3D shapes and 20K quesion-answed pairs. We find that by integrating our neural operators with NDF, our framework able to effectively and robustly answer a set of visual reasoning questions. Simultaneously, observe the emergence of 3D segmentations of concepts, at the semantic and instance level, directly from the underlying supervision of downstream visual question answering.
dataset, which contains approximately 3K
Our contributions can be summed up follows: 1) we propose 3D-CG, which utilizes the differ- entiable nature of neural descriptor fields (NDF) to ground concepts and perform segmentations; 2) define set of neural operators, including neural counting operator top of the NDF; 3) with

---

3D-CG, semantic and instance segmentations can emerge from question answering supervision; 4) our 3D-CG outperforms baseline models in both segmentation and reasoning tasks; 5) it can also generalize well to unseen shape categories and real scans.
2 Related works
Visual Reasoning. There have been different tasks focusing on learning visual concepts from natural language, such as visually -oouuddd question answering [11, 12] and text-image retrieval [45]. Visual question answering that evaluates machine's 's reasoning abilities stands out as a challenging task it requires human-like understanding of the visual scene. Numerous visual reasoning models have been proposed in recent years. Specifically, MAC [16] combined multi-modal attention for compositional reasoning. LCGN [15] built contextualized representations for objects to support relational reasoning. These methods model the reasoming process implicitly with neural networks. Neural-symbolic methods [53, 28, 3] explicitly perform symbolic reasoning on the objects representations and language representations. Specifically, they use perception models to extract 2D masks for 3D shapes first step, and then execute operators and ground concepts on these pre-segmented masks, but are limited set of pre-defined concepts. In this paper, we present an approach towards grounding 3D concepts in fully differentiable manner, with which 3D segmentations can emerge from question answering supervision.
Neural Fields. Our approach utilizes neural fields also known as neural implicit representations, to parameterize an underlying 3D geometry of a shape for reasoning. Implicit fields have been shown to accurately represent shape topology and 3D geometry [1, 20, 32]. Recent works improve traditional shape fields by using continuous neural networks [33, 5, 36, 38, 39, 51, 13]. These works are typically used for reconstruction [33, 36, 29] or geometry processing [51. 37. 8]. We use a neural descriptor field (NDF) similar to [29] to infer the 3D representations from 2D images, and leverage the features learnt from the NDF for visual reasoning and language grounding. Neural fields have also been used to represent dynamic scenes [34, 10], appearance [42, 30, 35, 52, 40], physics [23], robotics [18], acoustics [27] and more general multi-moda signals [9]. Conditional neural field such as [25, 48] allows us to encode mnstance-speciff information into the field, which is also utilized by our framework to encode the RGBD data of 3D shapes. There are also some works that integrate semantics or language in neural fields [17, 46]. However, they mainly focus on using language for manipulation, editing or generation.
this work, we utilize descriptors defined on neural fields [41] to differentiably reason in 3D given natural language questions. We refer readers to recent surveys [49, 44] for more related works on neural fields.
Language-driven Segmentation. Recent works have been focused on leveraging language for segmentation. Specifically, BiLD [14] distills the knowledge from a pre-trained image classification model into two-stage detector. MDETR [21] is an end-to-end modulated detector that detects objects in an image conditioned on raw text query, like caption or a question. LSeg [24] uses text encoder to compute embeddings of descriptive input labels together with a transformer-basee image encoder that computes dense per-pixel embeddings of the input image. Group ViT [50] learns to group image regions into progressively larger arbitrary bhhare segments with a text encode. These methods typically use an encoder to encode the text and do not have the ability to modify the segmentations based on the question answering loss. LORL [47] uses the part-centric concepts derived from language to facilitate the learning of part-centric representations. However, they can only improve the segmentation results but cannot generate segmentations from scratch with language supervision.
3 3D Concept Grouding
In Figure 2, we show an overall framework of our 3D Concept Grounding (3D-CG), which seamlessly bridges the gap between 3D perception and reasoning by using Neural Descriptor Field (NDF). Specifically, we first use NDF assign a high dimensional descriptor vector for each 3D coordinate, and run semantic parsing module to translate the question into neural operators to be executed on the neural field. The concepts, also the parameters of the operators, possess concept embeddings as well. Upon executing the operators and grounding concepts in the neural field, we perform dot product attention between the descriptor vector of each coordinate and the concept embedding vector to calculate the score (or mask probability) of a certain concept being grounded on a coordinate. We also propose a count operator which assigns point coordinates into different slots for counting the number of instances of part category. Both the NDF and the neural operator have a fully differentiable
3

---

Mask Probabilities:
Mask Probublities:
0.00
0.04
Maxpool in Each Slot
0.78
maxa
0.09
My 0.97
0.12
Mgg 0.89
mi
0.69
0.02
m, 0.78
0.96
mess 0.00
Mags 0.83
Myz 0.09
0.92
0.90
m,
0.91
" 0.18
0.01
0,78
Myg 0.83
may 0.91
m 0.91
Encoder
Occupuncy Descriptor Voctors
Min
m46 0.78
Min
NDF
F2 pe 0.12
p 9d6 >= 0.02
Sum
X Point Coordinates
Attention
Attention
M11 m2 3.57
Concept Embeddings
Slot Embeddings
Answer: 3,57
How many: legs does the chuir have?
Seq2Seq
Parsing
Count (Filter(Leg))
Operutors
(Filter (Log))
Count
GT. 4
Compute Loss
Figure 2: Our 3D Concept Grounding (3D-CG) framework. Given input image and a question, 3D-CG first utilizes a Neural Descriptor Field (NDF) to extract a descriptor vector for each 3D point coordinate (here we take coordinates with 5 different colors as examples), and a semantic parsing network to parse the question into operators. The mask probabilities of the coordinates are initialized with occupancy values. Concepts are also mapped to embedding vectors. Attention is calculated between the descriptors and concept embeddings to yield new mask probabilities and filter the set coordinates being referred as the concept. We also define count operator which segments a part category into different slots. The sum of the maxpooled mask probabilities of all slots can be used for counting, and the loss can be back propagated to optimize NDF as well as the embeddings. Semantic and instance segmentations emerge from question-answeree supervision with 3D-CG.
design. Therefore, the entire framework can be end-to-end trained, and the gradient from the question answering loss can be utilized to optimize both the NDF perception module and the visual reasoning module and jointly learn all the embeddings and network parameters.
3.1
Model Details
3.1.1
Neural Descriptor Fields
In this paper, we utilize a neural field to map a 3D coordinate x using a descriptor function f(x) and extract the feature descriptor of that 3D coordinate:
fxx) : R + R", x f(x) =v
(1)
where the descriptor representation which encodes information about shape properties (e.g.. geometry, appearance, etc).
In our setting, we condition the neural field on the partial point cloud P derived from RGB-D images. We use a PointNet-basee encoder & to encode P. The descriptor function then becomes ((xEE(P) v. This continuous and differentiable representation maps each 3D coordinate x to descriptor vector v. Since the concepts to be grounded in most reasoning tasks focus on geometry and appearance, leverage two pre-training tasks to parameterize f and learn the features concerning these properties.
Shape Reconstruction. Inspired by recent works on using occupancy networks [29] to reconstruct shapes and learn shape representations, we also use an MLP decoder D7 which maps each descriptor vector v to an occupancy value: D1 v + o E [0, 1], where the occupancy value indicates whether the 3D coordinate the surface of shape.
Color Reconstruction. D2 v e R-, where the color value indicates the color of the 3D coordinate on the shape.
We use another MLP D2 to decode an RGB color of each coordinate:
After learning decoders D, through these pre-training tasks, to construct the Neural Descriptor Field (NDF) f(x), we concatenate all the intermediate activations of D; when decoding a 3D point [41]. The resultant descriptor vector v can then be used for concept grounding. Since v consists of intermediate activations of D,, they implicitly capture the hierarchical nature of scene, with earlier activations corresponding to lower level features and later activations corresponding to higher level features.

---

3.1.2 Concept Quantization
Visual reasoning requires determining the attributes (e.g,, color, category) of shape shape part, where each attribute contains a set of visual concepts (e.g, blue, leg). As shown in Figure 2, visual concepts such as legs, are represented as vectors in the embedding space of part categories. These concept vectors are also learned by our framework. To ground concepts in the neural fields, we calculate the dot products ( between the concept vectors and the descriptor vectors from the neural field. For example, to calculate the score of 3D coordinate belonging to the category 1eg, we take its descriptor vector v and compute p = (v, e,)) where yااy the concept embedding of leg.
3.1.3 Semantic Parsing
To transform natural language questions into of primitive operators that be executed on the neural field, a semantic parsing module similar to that in [53] is incorporated into our framework. This module utilizes LSTM-based Seq2Seq to translate questions into set of fundamental operators for reasoning. These operators takes concepts and attributes as their parameters (e.g., Filter (1eg), Query (color)))
3.1.4 Neural operators on Neural Descriptor Fields
The operators extracted from natural language questions can then be executed on the neural field. Due the differentiable nature of the neural field, the whole execution process also fully-diiferenniaabl. We represent the intermediate results in probabilistic manner: for the i-th sampled coordinate x,, is represented by descriptor vector v; and has an attention mask m, E [o. 1]. m, denotes the probability that a coordinate belongs to a certain set, and is initialized using the occupancy value output by Dr: m, O;,
There are three kinds of operators output by the semantic parsing module, we will illustrate how each operator executes the descriptor vectors yield the outputs.
Filter Operator. The filter operator "selects outs" a set of coordinates belonging to the concept by outputting new mask probabilities:
Filter(c) m = min m; (v,,2"))
min(m;;
Query Operator. The query operator asks about an attribute on selected coordinates and outputs concept è of the maximum probability:
Query(a): è = arg max min(m;; (ve, (
Count Operator. The count operator intends to segment the coordinates in the same part category into instances and count the number of instances. Inspired by previous unsupervised segmentation methods which output different parts in various slots [26, 4], we also allocate a set of slots for different instances of category. The j-th slot 8j has its embedding vector vz- The score 8ij of the i-th coordinate belonging the j-th slot is also computed by dot product 8ij = (v, v*>). We further take softmax to normalize the probabilities across slots. Since we want to count the instances of part that was previously selected out, also take the minimum of previous mask probabilities and the mask probabilities of each slot. The result of counting (i.e., the number of part instances n) is obtained by summing up the maximum probability in each slot.
8ij
Count(c) m.j min(mm
(4)
Σ
s/s
n Σ max mij
(5)
3.1.5 Segmentation
Note that based the above operators, not only visual reasoning can be performed, but also do semantic segmentation and instance segmentation.
Semantic Segmentation. We can achieve semantic segmentations by executing Query (category) on each coordinate and output the category é with the maximum probability for each point.
5

---

Instance Segmentation. We can perform instance segmentations by first doing semantic segmentation, and then for each output è we further execute count(è).
3.2 Training Paradigm Optimization Objective. by minimizing the loss
During training, we jointly optimize the NDF and the concept embeddings
L -= a. LNDF + B Lreasoning
Specifically, the NDF loss consists of two parts: the binary cross entropy classification loss between the ground-truth occupancy and the predicted occupancy, and the MSE loss between the ground-truth rgb value and the predicted rgb value.
We also define three kinds of losses for three types of questions: 1) For questions that ask about whether part exist, we use MSE loss ||a max where is the ground-truth answer and max,m;) takes the maximum mask probability among all sampled 3D coordinates; 2) For questions that query about attribute, we take the cross entropy classification loss between the predicted concept category è and the ground -ruth concept category; 3) For counting questions, we first use an MSE loss |/a - n|l between the answer and the number output by summing up the maxpool values of all the slots, and further add a loss to ensure that the mask probabilities of the top K values in the top a slots (i.e., the slots with the maximum maxpool values) should be close 1, where K is hyper-parameter. During training, we first train the NDF module only for N₁ epochs and then jointly optimize the NDF module and the concept embeddings. The Seq2seq model for semantie parsing is trained independently with supervision.
Curriculum Learning. Motivated by previous curriculum strategies for visual reasoning [28], we employ curriculum learning strategy for training. We split the questions according to the length of operators they are parsed into. Therefore, we start with questions with only one neural operator (e.g.. "is there yellow part of the chair" can be parsed into Filter Filter(Color)). (Color)).
4 Experiments
4.1
Experimental Setup
4.1.1 Dataset
Instead of object-based visual reasoning [19, 53, 28] where objects spatially disentangled, which makes segmentations quite trivial we seek to explore part-based visual reasoning where segmenta- tions and reasoning are both harder. To this end, we collect new dataset, -ertieetRReasninng which focuses on visual question answering on the PartNet [31] dataset. Specifically, we render ap- proximately 3K RGB-D images from shapes of 4 categories: Chair, Table, Bag and Cart, with question-answer pairs on average for each shape. We have three question types: exist_ _errtt_rtttt__ queries about whether a certain part exists by having the filter operator as the last operator; query-part -prrt uses query operator to query about an attribute (e.g., part category or color) of filtered part; count porrt_part utilizes the count operator to count the number of instances of filtered part. We are interested in whether the reasoning tasks can benefit the segmentations of the fine treegga parts, as well as whether the latent descriptor vectors by NDF can result better visual reasoning.
4.1.2 Evaluation Tasks
Reasoning. We report the visual question answering accuracy on the PartNet-Reasoning dataset w.r.t the three types of questions on all four categories.
Segmentation. We further evaluate both the performances of semantic segmentation and instance segmentation on our dataset. As stated in 3.1.5, semantic segmentation can be performed by querying the part category with the maximum mark probability of each coordinate, and filtering the coordinates according to category labels, and instance segmentation can be achieved by using the count operator on each part category. We report the mean per-label Intersection Over Union (IOU).
4.1.3 Baselines
We compare our approach to a set of different baselines, with details provided in the supp. material.

---

PointNet+LSTM
MAC
NDF+LSTM
CVX+L
BAE+L
3D-CG
exist_pe part
52.3
65.2
55.7
71.9
72.4
85.4
Chair
query_ _arrt
41.6
53.9
54.2
72.3
70.5
68.7
count part
63.4
78.1
71.6
48.8
68.0
92.2
exist_part
55.1
66.4
61.3
68.5
71.0
80.3
Table
query_part
43.6
51.2
54.4
69.7
73.6
75.1
count_ part
35.5
55.3
50.1
30.7
45.7
90.9
exist_part
65.4
85.4
69.2
87.3
73.8
89.2
Bag
query_part
53.1
74.8
64.0
88.1
70.6
85.2
count_part
51.2
70.9
72.3
53.4
31.4
68.3
exist_ part
49.7
75.3
61.7
79.1
91.0
90.1
Cart
query _prr
50.1
64.0
57.0
72.3
81.5
86.3
count_part
41.6
82.1
67.3
46.6
47.3
74.8
Table I: Visual question answering accuracies. exist_part queries about whether a certain part exists by having the filter operator as the last operator; query-part uses query operator to query about an attribute of a filtered part: count_ ___aattttttttt utilizes the operator count the number of instances of filtered part. Point+LSTM. MAC NDF+LSTM are methods based on neural networks. CVX+L and BAE+L are neural- symbolic methods which pre-segment the masks and ground concepts on masks. 3D-CG outperforms baseline models by a large margin.
Unsupervised Segmentation Approaches. We compare 3D-CG with two additional 3D unsuper- vised segmentation models, and further consider how we may integrate such approaches with concept grounding. First, consider CVXNet [7], which decomposes solid object into collection of convex polytope using neural implicit fields. In this baseline, a number of primitives are selected and convex part is generated in each primitive. Next,
consider BA shape into set of pre-assigned branches, each specifying a portion of a whole shape. In practice, we tuned with the original codes of BAENet on our dataset and found that nearly all coordinates would be assigned to single branch. This is probably because the original model is trained on voxel models, while ours is on partial point cloud and contains more complex shapes, thus posing more challenges. Therefore, for BAENet we use an additional loss which enforces that at least some of the branches should have multiple point coordinates assigned to positive oceupancy value. The primitives in CVXNet and the branches in BAENet are similar to slots in our paper.
which decomposes
Reasoning Baselines. We compare our approach to a set of different reasoning baselines. First consider PointNet+LSTM and NDF+LSTM, which module, concatenated with the features of the questions encoded by a LSTM model. The final answer distribution is predicted with an MLP. Next, we compare with MAC, commonly-used attention-based model for visual reasoning. We add a depth channel to the input to the model. Finally we compare with CVX+L and BAE+L are neural-symbole mediated segmentation framework from [47]. Specifically, they first initiate the segmentations in the slots, and each slot has slot feature vector. The operators from the questions are executed symbolically on the slot features. Question answering loss can be also propagated back to finetune the slot segmentations and features.
the features by PointNet our NDF
models that use the similar language-
Segmentation Baselines. For the segmentation tasks, compare approach with unsupervised segmentation approaches CVX and BAE described above. We further construct language-mediated variants CVX-L and BAE-L. For semantic segmentation by CVX and BAE, the manual grouping methods as in [7]. For semantic segmentation by language-mediated models, we use the filter operator to filter the slots belonging the part categories.
4.2
Experimental Results
4.2.1 Reasoning Concept Grounding
Table shows the VQA accuracy among all models on our dataset. We can see that overall, our 3D-CG outperforms baseline modeles by a large margin. Language-medinn mediated neural-symbolic methods (CVX+L & BAE+L) are better than neural methods question types, but in general they are far worse than our method. This is because the slot-based representations pre-select the parts and make easier for the concepts attend to specific regions. However, wrong pre-segmentations are also hard to be corrected during the training of reasoning.
exist_part and query_parr _part

---

Shape & Question
Operator, Reference & Output Answer
Shape & Question
Operator, Reference & Output Answer
Stepl: Filter (Leg)
Step1: Filter (Green)
Step2: Query (Color)
Dark Blue
Question:
Stop2: Query (Category) Seat
Question:
Step3: Other & Filter (Dark Blue)
What the category of the green part of the chair?
Is there other part of same color the legs?
Answer:
Step1: Filter (Leg)
Answer: Yes
Filter (Yellow)
Question:
Step2: Count
Question: Is there a yellow part in the cart?
How many logs does this table have?
Answer: Yes
Answer:
Figure 3: Qualitative Illustration. Examples of the reasoning process of proposed 3D-CG. Given input image a shape and question, we first parse the question into steps of operators. We visualize the set of points being referred to by the operator via highlighting the regions where mask probabilitics > 0.5, As shown,our 3D-CG can make reference to the right set of coordinates, thus correctly answering the questions.
The accuracies for the count count_part type further demonstrates this point, where language-mediatee baselines have extremely low accuracies, and segmentations from Figure 4a also show that even with supervision from question answering loss, BAE and CVX cannot segment the instances of a part category (e.g., legs and wheels). Neural methods such as MAC and NDF+LSTM perform well in the count_part type in some categories, probably due some shorteuts of the PartNet dataset (e.g.. most bags have two shoulder straps). In comparison to both neural and neural-symbolic methods, our method performs well in all three question types. This is because the regions attended by concepts are dynamically improved with the question answering loss, while the advantage of explicit disentanglement of concept learning and reasoning process maintained.
Figure 3 shows some qualitative examples of the reasoning process of our method. Specifically, the questions parsed into set of neural operators. For each step, the neural operator takes the output of the last step as its input. For the filter operator, we visualize the attended regions with mask probabilities greater than 0.5. From the figure, we can that our method attend the correct region to answer the questions, as well as segment the instances correctly for counting problems. This attention-based reasoning pipeline is also closer than previous neural-symbolic methods the way that humans perform reasoning. For example, when asked the question "What is the category of the green part of the chair?", humans would directly pay attention to the green region regardless of the segmentations of the rest of the chair. However, previous neural symbolic methods [53, 28, 47] segment the chair into different parts first and then select the green part, which is very unnatural.
4.2.2 Segmentation
Table 2 shows the semantic segmentation and instance segmentation results, and Figure visualizes some qualitative examples. We can see that overall, our 3D-CG can better segment parts than unsupervised or language-mediated methods. Other methods experience some common problems such as failing to segment the instances within a part category (e.g.. one of the legs is always merged with the seat), or experience unclear boundaries and segment one part into multiple parts (e.g., segmenting the chair back or the tabletop into two parts). In general, language-mediated methods are better than pure unsupervised methods, which indicates that the supervision question answering does benefit the segmentation modules. However, lot of the wrong segmentations cannot be corrected by reasoning, resulting in bad results for counting questions for these models. In contrast, our 3D-CG can learn segmentations well using question answering loss, mainly because the segmentations emerge from question-answee
supervision without any restrictions.
For fair comparison with baselines, we first control the reconstruction part and use the ground-truth voxels for the segmentation evaluations only. We also provide additional experimental results when the segmentations performed on the reconstructed voxels, which is shown in Table 3.

---

Generalization to New Categories
Shapes
Semantic Segmentation
tu widheee
Count (Wheel)
CVX+L
BAE
Filter (Leg)
BAE+L
3D-CG
Generalization to Real Scans
Instance Segmentation
CVX+L
Instance Segmentation
BAE
BAE+L
Filter (Back)
3D-CG
(b) Generalization.
(a) Visualization segmentation results.
Figure 4: Visualization of segmentation well generalization. Here ground-truth voxel values for better visualization. CVX and BAE unsupervised segmentaton methods. CVX+L and BAE+L language-mediated methods which utilize question answering loss to finetune segmentation module. 3D-CG has hetter performances in both semantic and instance segmentations, while other methods suffer from merging parts or segmenting part into multiple parts. It can also generalize well to unseen categories and real scans.
4.2.3 Generalization
In Figure 4b, we show some qualitative examples of how our 3D-CG trained on seen categories can be generalized directly to unseen categories and real scans. We show results semantic and instance segmentations, well visualize the parts that are referred to in the questions.

---

Chair
Table
Bag
Cart
Mean
Chair
Table
Bag
Cart
Mcan
CVX
62.3
74.2
66.0
44.2
61.7
CVX
44.1
40.6
31.2
22.5
34.6
CVX+L
64.6
74.5
70.2
51.3
65.2
CVX+L
42.6
51.2
41.3
20.4
38.9
BAE
49.5
71.0
64.1
49.7
58.6
BAE
53.3
32.2
39.8
34.0
39.9
BAE+L
56.3
69.8
46.9
61.3
BAE+L
50.1
47.0
34.1
36.6
42.0
3D-CG
76.6
79.3
67.3
54.2
69.4
3D-CG
68.5
71.2
47.2
40.5
56.9
(a) Semantic Segmentation IOU
(b) Instance Segmentation IOU
Table 2: Segmentation Results on Ground- Tround-Truth Occupancy Values. 3D-CG outperforms all unsupervised / landuage-mediated baseline modcls.
Chair
Table
Bag
Cart
Mean
Chair
Table
Bag
Cart
Mean
CVX
38.1
49.5
40.3
26.0
38.5
CVX
29.4
26.4
18.2
13.4
21.9
CVX+L
34.0
39.0
38.3
29.2
35.1
CVX+L
22.9
28.2
22.6
13.8
21.9
BAE
36.3
50.1
50.0
38.5
43.8
BAE
38.7
22.7
28.1
24.8
28.6
BAE+L
41.1
51.6
51.7
34.5
44.8
BAE+L
37.3
34.5
25.1
26.7
30,9
3D-CG
69.3
65.7
56.9
48.2
60.0
3D-CG
57.1
58.7
42.0
35.4
48.3
(a) Semantic Segmentation IOU
(b) Instance Segmentation IOU
Table 3: Segmentation Results on Preidcted Occupancy Values.
For generalizing to new categories, we use the model trained on carts to ground and count the instances of the concept "wheel" cars. We can see that the instance segmentation results on the wheels are not perfect because one wheel is in wrong position. However, most of the wheels are detected and the model manages to output the right count. We also use the model trained on chairs to filter the legs of bed. All the legs are successfully selected out by our 3D-CG.
We also use real 3D scans from the RedWood dataset [6] estimate 3D-CG's ability to generalize real scenes. We use a single-view scan to reconstruct partial point cloud and remove the background such the floor. We input the partial point cloud into our 3D-CG and output the segmentations on the ground-truth voxels. For generalizing to bicycles, we use the 3D-CG model trained on carts. We can see that 3D-CG can find all instances the bicycle and detect both wheels. Furthermore, 3D-CG trained on chairs can also be generalized to chairs in real scans.
Discussion
Conclusion. In this paper, propose 3D-CG, which leverages the continuous and differentiable nature of neural descriptor fields to segment and learn concepts in the 3D space. We define a set of neural operators top of the neural field, with which not only can semantic and instance segmentations emerge from question-answering supervision, but visual reasoning can also be well performed.
Limitations and Future Work. A limitation of underlying framework with 3D-CG is that while we show transfer results on real scenes, our approach is only trained with synthetic scenes and questions. While we believe our proposed operations is general-purpose, of future work would be scaling our framework to directly train on complex real-world scenes, and ground more concepts from natural language these real scenes. A promising direction of future work would to explore the combination large pre-trained visual- Lsugul-an models and volumetric rendering multi-view images.
interesting direction
Acknowledgments and Disclosure of Funding
This work was supported by MIT-IBM Watson AI Lab and its member company Nexplore, Amazon Research Award, ONR MURI, DARPA Machine Common Sense program, ONR 1-00014-18-1- 2847), and Mitsubishi Electric.
10

---

References
[1] J. Bloomenthal and B. Wyvill. Introduction to implicit surfaces. 1997. 3
[2] L. Bottou. From machine learning to machine reasoning. Machine Learning, 94:133-149, 2013.
[3] Z. Chen, J. Mao, J. Wu, K.-Y. K. Wong, J. B. Tenenbaum, and C. Gan. Grounding physical concepts of objects and events through dynamic visual reasoning. ICLR, 2021.
[4] Z. Chen, K. Yin, M. Fisher, S. Chaudhuri, and H. Zhang. Bae-net: Branched autoencoder for shape co-segmentation. 2019 IEEE/CVF International Conference on Computer Vision (ICCV), pages 8489-8498, 2019. 5,
[5] Z. Chen and H. Zhang. Learning implicit fields for generative shape modeling. In Proc. CVPR, pages 5939-5948, 2019.
[6] S. Choi, Q.-Y. Zhou, S. D. Miller, and V. Koltun. A large dataset of object scans. ArXiv, Abs//602..2281, 2016. 10
[7] B. Deng, K. Genova, S. Yazdani, S. Bouaziz, G. E. Hinton, and A. Tagliasacchi. Cvxnet: Learnable convex decomposition. 2020 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pages 31-41, 2020.
[8] Y. Deng, J. Yang, and X. Tong. Deformed implicit field: Modeling 3d shapes with learned dense correspondence. 2021 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pages 10281-10299 2021.
[9] Y. Du, M. K. Collins, B. J. Tenenbaum, and V. Sitzmann. Learning signal-agnostic manifolds of neural fields. In Advances Neural Information Processing Systems, 2021.
[10] Y. Du, Y. Zhang, H.-X. Yu, J. B. Tenenbaum, and J. Wu. Neural radiance flow for 4d view synthesis and video processing. In Proceedings of the IEEE/CVF International Conference on Computer Vision, 2021. 3
[11] C. Gan, Y. Li, H. Li, C. Sun, and B. Gong. Vgs: Linking segmentations to questions and answers for supervised attention in vqa and guestion-focuus semantic segmentation. In ICCV, pages 1811-1820 2017. 3
[12] S. Ganju, O. Russakovsky, and A. K. Gupta. What's in question: Using visual questions as a form of supervision. 2017 IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pages
2017.
[13] K. Genova, Cole, D. Vlasic, A. Samna, W. T. Freeman, and T. A. Funkhouser. Learning shape templates with structured implicit functions. 2019 JEFF/CVF International Conference on Computer Vision (ICCV), pages 7153-7163, 2019.
[14] X. Gu, T.-Y. Lin, W. Kuo, and Y. Cui. Zero-shot detection via vision and language knowledge distillation. ArXiv, abs/2104. 13921, 2021.
[15] R. Hu, A. Rohrbach, T. Darrell, and K. Saenko. Language-conditiones graph networks for relational reasoning. 2019 IEEF/VV International Conference on Computer Vision (ICCV), pages 10293-10302, 2019.
[16] D. A. Hudson and Manning. Compositional attention networks for machine reasoning. ICLR, 2018. 3
[17] A. Jain, B. Mildenhall, J. T. Barron, Abbeel, and B. Poole. Zero-shot text-guided object generation with dream fields. arXiv, December 2021. 3
[18] Z. Jiang, Zhu, M. Svetlik, K. Fang, and Y. Zhu. Synergies between affordance and geometry: 6-dof grasp detection via implicit representations. ArXiv, abs/2104.01542, 2021. 3

---

[19] Johnson, B. Hariharan, L. van der Maaten, Fei-Fei, C. L. Zitnick, and R. B. Girshick. Clevr: diagnostic dataset for compositional language and elementary visual reasoning. 2017 IEEE Conference on Computer Vision and Patlern Recognition (CVPR), pages 1988-1997, 2017. 1, 6
[20] M. W. Jones, J. A. Bærentzen, and M. Srámek. 3d distance fields: survey of techniques and applications. IEEE Transactions on Visualization and Computer Graphics, 12:581-599, 2006. 3
[21] A. Kamath, M. Singh, Y. LeCun, I. Misra, G. Synnaeve, and N. Carion. Mdetr modulated detection for end-to-end multi-modal understanding. 2021 IEEE/CVF International Conference on Computer Vision (ICCV), pages 1760-1770, 2021. 3
[22] C. Kervadec, Jaunet, G. Antipov, M. Baccouche, R. Vuillemot, and C. Wolf. How transferable are reasoning patterns in vqa? 2021 /EEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pages 4205 4214, 2021. 1
[23] S. Kollmannsberger, D. D' Angeela,, M. Jokeit, and L. A. Herrmann. Physics-informet neural networks. Deep Learning in Computational Mechanics, 2021. 3
[24] B. Li, K. Q. Weinberger, S. Belongie, V. Koltun, and R. Ranftl. Language-driven semantic segmentation, 2022. 3
[25] S. Liu, X. Zhang, Z. Zhang, R. Zhang, J. Zhu, and B. C. Russell. Editing conditional radiance fields. 2021 IEEE/CVF International Conference on Computer Vision (ICCV), pages 5753-5763,
2021. 3
[26] F. Locatello, D. Weissenborn, T. Unterthiner, A. Mahendran, G. Heigold, J. Uszkoreit, A. Doso- vitskiy, and T. Kipf. Object-centric learning with slot attention. ArXiv, abs/2006. 15055, 2020. 5
[27] A. Luo, Y. Du, M. J. Tarr, B. Tenenbaum, A. Torralba, and C. Gan. Learning neural acoustic fields. arXiv preprint wxXiv2204.006222 2022. 3
[28] J. Mao, C. Gan, P. Kohli, J. B. Tenenbaum, and J. Wu. The ncuro-symbolic concept learner: Interpreting scenes words and sentences from natural supervision. ArXiv, abs/1904 12584, 2019. 1, 2, 3, 6, 8
[29] L. M. Mescheder, M. Oechsle, M. Niemeyer, S. Nowozin, and A. Geiger. Occupancy networks: Learning 3d reconstruction in function space. 2019 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pages 4455-4465, 2019. 3,
[30] B. Mildenhall, P. P. Srinivasan, M. Tancik, J. T. Barron, R. Ramamoorthi, and R. Ng. Nerf: Representing scenes as neural radiance fields for view synthesis. In Proc. ECCV, 2020. 3
[31] K. Mo, S. Zhu, A. X. Chang, L. Yi, S. Tripathi, L. J. Guibas, and H. Su. Partnet: A large-scale benchmark for fine-grained and hierarchical part-level 3d object understanding. 2019 IEFF/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pages 909-918, 2019, 6
[32] K. Museth, D. E. Breen, R. T. Whitaker, and A. H. Barr. Level set surface editing operators. Proceedings of the 29th annual conference on Computer graphies and interactive techniques, 2002. 3
[33] M. Niemeyer, L. Mescheder, M. Oechsle, and A. Geiger. Occupancy flow: 4d reconstruction by learning particle dynamics. In Proc. ICCV, 2019. 3
[34] M. Niemeyer, L.. Mescheder, M. Oechsle, and A. Geiger. Occupancy flow: 4d reconstruction by learning particle dynamics. In Proceedings of the IEEE International Conference on Computer Vision, pages 5379-5389, 2019. 3
[35] M. Niemeyer, L. Mescheder, M. Oechsle, and A. Geiger. Differentiable volumetric rendering: Learning implicit 3d representations without 3d supervision. In Proc. CVPR, 2020.
[36] J. Park, P. Florence, J. Straub, R. Newcombe, and S. Lovegrove. Deepsdf: Learning continuous signed distance functions for shape representation. In Proc. CVPR, 2019. 3
12

---

[37] K. Park, U. Sinha, J. T. Barron, S. Bouaziz, D. B. Goldman, S. M. Seitz, and R. M. Brualla. Deformable neural radiance fields. ArXiv, abs/2011. 2020.
[38] Peng, M. Niemeyer, L. Mescheder, M. Pollefeys, and A. Geiger. Convolutional occupancy networks. In Proc. ECCV, 2020. 3
[39] D. Rebain, K. Li, V. Sizzmann, S. Yazdani, K. M. Yi, and A. Tagliasacchi. Deep medial fields. arXiv preprint arXiv:2 2021. 3
[40] S. Saito, Z. Huang, R. Natsume, S. Morishima, A. Kanazawa, and H. Li. Pifu: Pixel-aligned implicit function for high-resolution clothed human digitization. In Proc. ICCV, pages 2304- 2314, 2019. 3
[41] A. Simeonov, Y. Du, A. Tagliasacchi, J. B. Tenenbaum, A. Rodriguez, P. Agrawal, and V. Sitz- mann. Neural descriptor fields: Se (3)-equivariant object representations for manipulation. arXiv preprint 1 2021. 2, 3, 4
[42] V. Sitzmann, M. Zollhöfer, and G. Wetzstein. Scene representation networks: Continuous dd-structure-awaan aware neural scene representations. In Proc. NeurIPS 2019, 2019. 3
[43] E. S. Spelke, K. Breinlinger, K. Jacobson, and A. Phillips. Gestalt Relations and Object Perception: A Developmental Study. Perception,
1993. 2
[44] A. Tewari, J. Thies, B. Mildenhall, Srinivasan, Tretschk, W. Yifan, C. Lassner, V. Sitzmann, R. Martin-Brualla, S. Lombardi, et al. Advances in neural rendering, In Computer Graphics Forum, volume 41, pages 703-735. Wiley Online Library, 2022.
[45] Vendrov, Kiros, S. Fidler, and R. Urtasun. Order-embeddings of images and language. CoRR, aabs//51..03610 2016.
[46] C. Wang, M. Chai, M. He, D. Chen, and J. Liao. Clip-nerf: foxt-and-image driven manipulation of neural radiance fields. ArXiv, abs/2 12.051199, 2021. 3
[47] R. Wang, J. Mao, S. J. Gershman, and J. Wu. Language-mediated, object-centric representation learning. In FINDINGS, 2021. 3, 7,
[48] S. Wang, L. Li, Y. Ding, C. Fan, and X. Yu. Audio2head: Audio-driven one-shot talking-head generation with natural head motion. In IJCAI, 2021. 3
[49] Y. Xie, T. Takikawa, S. Saito, O. Litany, S. Yan, N. Khan, F. Tombari, J. Tompkin, V. Sitzmann, and S. Sridhar. Neural fields in visual computing and beyond. Computer Graphics Forum, 41, 2022.
[50] J. Xu, S. D. Mello, S. Liu, W. Byeon, T. Breuel, J. Kautz, and X. Wang. Groupvit: Semantic segmentation emerges from text supervision. ArXiv, abs/2202. 1094, 2022.
[51] G. Yang, S. Belongie, B. Hariharan, and V. Koltun. Geometry processing with neural fields. In A. Beygelzimer, Y. Dauphin, P. Liang, and J. W. Vaughan, editors, Advances in Neural Information Processing Systems, 2021.
[52] L. Yariv, Y. Kasten, D. Moran, M. Galun, M. Atzmon, B. Ronen, and Y. Lipman. Multiview neural surface reconstruction by disentangling geometry and appearance. Proc. NeurIPS, 2020.
3
[53] K. Yi, J. Wu, C. Gan, A. Torralba, P. Kohli, and J. B. Tenenbaum. Neural-symbolic vqa: Disentangling reasoning from vision and language understanding. In NewrIPS, 2018. 3, 5, 6,
13

---

Checklist
For all authors...
(a) Do the main claims made in the abstract and introduction accurately reflect the paper's
contributions and scope? [Yes]
(b) Did you describe the limitations of your work? [Yes]
(c) Did you discuss any potential negative societal impacts of your work? [Yes]
(d) Have you read the ethics review guidelines and ensured that your paper conforms to them? [Yes]
2. If you are including theoretical results...
(a) Did you state the full set of assumptions all theoretical results? [N/A]
(b) Did you include complete proofs of all theoretical results? [N/A]
3. If you ran experiments...
(a) Did you include the code, data, and instructions needed to reproduce the main experi- mental results (either in the supplemental material or as URL)? [No]
(b) Did you specify all the training details (e.g., data splits, hyperparameters, how they were chosen)? [Yes]
(c) Did you report error bars (e.g., with respect to the random seed after running experi- ments multiple times)? [No]
(d) Did you include the total amount compute and the type of resources used (e.g., type of GPUs, internal cluster, or cloud provider)? [No]
4. If you are using existing assets (e.g., code, data, models) or curating/releasing new assets...
(a) If your work uses existing assets, did you cite the creators? [Yes]
(b) Did you mention the license of the assets? [Yes]
(c) Did you include any new assets either in the supplemental material or as URL? [No]
(d) Did you discuss whether and how consent obtained from people whose data you're using/curating? [Yes]
(e) Did you discuss whether the data you are using/curating contains personally identifiable information or offensive content? [Yes]
5. If you used crowdsoureing conducted research with human subjects...
(a) Did you include the full text of instructions given to participants and screenshots, if applicable? [N/A]
(b) Did you deseribe any potential participant risks, with links to Institutional Review Board (IRB) approvals, if applicable? [N/A]
(c) Did you include the estimated hourly wage paid participants and the total amount spent on participant compensation? [N/A]
14


Reviews:
[{"review_id": "dzL8_qSIax2", "reviewer": "Reviewer_1ERZ", "summary": "This paper presents a method for grounding natural language concepts in 3D neural fields of shape and appearance. The main contribution is to use neural descriptor fields, a continuous field and ground it in language concepts. The input to the method is a set of shapes/appearance and QA about that shape. No other supervision is used. Specifically, the paper looks at binary questions, counting questions, and segmentation questions. Furthermore, the paper presents a slot attention module and neural operators for counting/segmentation.", "questions": "In the rebuttal, it would be nice to see a plan to include more details that are currently missing in the paper (see above).\n\nWill the PartNet-Reasoning dataset be publicly released?", "limitations": "Yes, there is a discussion of limitations and societal impact. This discussion is adequate.", "rating": 8, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 4, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "## Strengths\n\n- I quite like the idea of grounding concepts to neural fields rather than to 3D shape/appearance. This makes it continuous and offers previously unseen differentiability properties.\n- The paper grounds concepts to descriptor fields without any explicit supervision. This is an exciting direction that could open up new research directions.\n- The proposed neural operators, particularly for counting is novel to my knowledge.\n- The results shown are promising and show improvement compared to baselines. I especially appreciated the inclusion of real image results.\n\n## Weaknesses\n\n- Firstly, it's not very clear what class of data the method operates on. Is it scenes or object classes? Only in the experiment section is this made clear (PartNet-Reasoning). This is an important detail that the paper ought to mention earlier.\n\n- The paper seems to introduce a depth-map conditioned neural field (line 123), but details and differences to conventional neural shape fields are missing.\n\n- Generally, the paper is lacking many details, especially in the shape and color representation parts, and the language parsing part. For instance, what kind of parsing does the seq2seq model do? Why were these three operators chosen? In the loss description, how exactly is L_{NDF} computed? Is there a reconstruction module that is missing in Figure 2.\n\n- In terms of related work, the paper could have done a better job summarizing the latest advances. I am afraid that the related work is limited to a small set of works. I would recommend the authors take a look at a couple of recent works and papers cited within.\n\n*Geometry Processing with Neural Fields, Yang et al. 2021\n\n*Neural Fields in Visual Computing and Beyond, Xie et al. 2022\n\n*Advances in Neural Rendering, Tewari et al. 2022\n\n- In the datasets, how are the ~8 questions+answers per shape selected? Are these all related to the categories of tasks that the paper intends to solve, or are they more general? Also, the dataset has *both* questions and answers, not just questions. This should be mentioned in line 214.\n\nOverall, this is really nice work that I would like to see at NeurIPS. The paper opens up new research directions that move the needle both in terms of 3D shape representations, learning with fewer examples, and concept grounding to 3D shapes. However, the related work section needs to be more comprehensive.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ""}, "extra_text": {}}, {"review_id": "dbNsexIP1qY", "reviewer": "Reviewer_gaS1", "summary": "The paper proposes a weakly-supervised method that learns to ground 3D concepts with RGBD inputs. The key idea is to introduce Neural Descriptor Field (NDF) to neural operators (filter, query, and counting). 3D concepts are shown to emerge from training on VQA dataset. They extend their method to part semantic segmentation and part instance segmentation and show some genarlziation ability to unseen classes and real scans.\n", "questions": "see weakness 2 ", "limitations": "The authors have addressed the limitations and potential negative societal impact of their work", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 4, "contribution": 3, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Strength\n+ The paper aims to lift concept reasoning to 3D space. I think the biggest strength is the weakly-supervised method. The method leverages differentiable neural operators such that 3D grounding can be trained with question answer pairs. It also naturally allows for better interpretability. \n+ With a bunch of modules, the paper is very well written and easy to follow. Toy examples are nicely introduced to help readers understand. \n+ Part semantic segmentation and instance segmentation are also shown as special cases of the proposed method.\n+ The results look convincing to me. Since the problem is quite novel, the authors do a good job to compare with baselines. \n\nWeakness\n1. The proposed method focuses on object-centric reasoning and the method may be hard to scale up to more complicated scenes because the NDF considers the whole 3D space as a continuous, per-point query space. However, Table1 may suggest that some primitive-based methods may have advantages if their emerging part aligns with the language guide. -- It is more of my personal suggestion than weakness. \n2. The generalization claim is quite weak as only some qualitative results are shown. For example, I wonder about the quantitative performance on novel categories for both baselines and their method. \n\n", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ""}, "extra_text": {}}, {"review_id": "YqVBDJ4aL8", "reviewer": "Reviewer_owKC", "summary": "This paper proposes a method for grounding concepts onto 3D representations inferred from images. The idea builds implicit 2d-to-3d methods like occupancy networks and neural descriptor fields, which encode an image to an embedding space, where, paired with a 3D query vector, the model returns a vector describing quantities of interest. This setup is paired with operators similar to \"neuro-symbolic concept learner\", which gives the model a way to answer questions about the input image. The model is first supervised for occupancy and color prediction, and then supervised to answer synthetic questions. The model is trained and evaluated on toy-ish data consisting of single objects on white backgrounds, where it outperforms some baselines.\n", "questions": "I asked a few questions in the Weaknesses part of my review. ", "limitations": "I think the written limitations greatly understate the true limitations. \n\nIt says...\n> while we show transfer results on real scenes\n\nI don't think this was convincingly shown.\n\n> an interesting direction of future work would be scale our framework to directly train on complex real-world scenes and ground more concepts from natural language on these real scenes is worth delving into in the future\n\nThis is so far beyond the paper's scope that it doesn't feel serious. \n\n\n\n\n****\nPost-rebuttal update: During the discussion phase the authors gracefully engaged with my complaints and toned down the contribution claims, so I am increasing my rating. I think people are generally interested in efforts to attach task-relevant features to neural fields, and this paper presents a reasonable step in that direction. ", "rating": 6, "confidence": 4, "soundness": 2, "presentation": 2, "contribution": 3, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "I like the idea of using an intermediate 3D representation for the grounding task. I think this is a good idea. But I think that the difficult part here is in inferring the intermediate 3D representation for the grounding -- which is not this paper's focus. Instead, the paper is focused on toy scenes where 2D-3D lifting is relatively easy, and then combines neurosymbolic-style operators to perform (and supervise) some grounding tasks.\n\nI am not very familiar with the baselines here, so I cannot judge the improvement over them.\n\nA major factor in my negative impression for this paper is the overstatement of results. When a paper claims to have done something, when in fact it only achieved a version so simplified as to be trivial compared to the real issue, it feels frustrating to me as a reader. \n\nThe paper claims that segmentation emerges: \n> \"we observe the emergence of natural 3D segmentations of concepts, at the semantic and instance level, directly from the underlying supervision of downstream reasoning.\"\n\nI don't understand why this claim is here at all. As explained later, semantic and instance segmentation are supervised through the querying task (line 192). \n\n\nThe paper claims that much of its training is self-supervised:\n> \"an intermediate representation should be discovered in a self-supervised manner\"\n> \"descriptors are learned in a self-supervised manner, and implicitly capture the hierarchical nature of a scene\"\n> \"After learning decoders Di through these self-supervised pre-training tasks...\"\n\nThis is in reference to 3D shape and color reconstruction, from image input. This requires a 3D model or detailed multi-view reconstruction of the object created beforehand, before the method begins! (Not to mention it requires object-centric data carefully collected by a human.) \n\n\nThe paper claims that it generalizes to unseen categories and real scenes:\n> \"It can also generalize well to unseen categories and real scans.\"\n> \"our 3D-CG trained on seen categories can be generalized directly to unseen categories and real scans\"\n> \"estimate 3D-CG’s ability to generalize to real scenes\" \n\nIn fact, the experiment here completely abstracts away the model's ability to infer a 3D representation from a 2D image, and instead queries the model's segmentations at ground-truth voxels. As for the categories, the results on the synthetic car and bed in Fig4b are quite unusual -- ignoring the fact that the blue wheel is in the wrong place, how did the model know to estimate the occluded side of the car and put 4 wheels, when only 2 are visible? Was 3D ground truth provided here as well? \n", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ""}, "extra_text": {}}, {"review_id": "Pqc6lu1Kwm", "reviewer": "Reviewer_Xamz", "summary": "This paper studies the problem of the approach and the effectiveness of the 3D concept grounding using neural implicit field representation. The authors focus on several types of questions and develop a pipeline including multiple stages. They also create a dataset containing the necessary types of questions to assist the experiment. The results show that by using 3D concept grounding information, the trained models outperform previous works and generalize well to unseen shapes.", "questions": "Please see my comments in the \"weaknesses\" part.", "limitations": "The limitation is addressed in the paper.", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 4, "contribution": 4, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Pros:\n+ The paper is well written.\n+ This paper studies a novel and interesting problem.\n+ A dataset contains 3K images and 3 types of questions are created to facilitate the experiments.\n+ A novel framework based on the neural descriptor field with multiple stages and attention blocks is proposed.\n+ The results that demonstrate superior performance and generalization are promising.\n\nCons:\n- The experimenting dataset is relatively small.\n- The framework seems to be designed for a specific purpose (i.e. answering limited types of questions) rather than a general-purpose one. I don't see this framework can be easily extendable.\n- A concern would be that it is well known that 3D labeling/datasets are hard to obtain, not to mention combined with question-answering data.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ""}, "extra_text": {}}]

Author rebuttal:

Decision: Accept