Venue: NeurIPS 2022 Conference

Reviews:

final_score: rating: 6: Weak Accept: Technically solid, moderate-to-high impact paper, with no major concerns with respect to evaluation, resources, reproducibility, ethical considerations.
review_content: summary: This paper proposes a method for grounding concepts onto 3D representations inferred from images. The idea builds implicit 2d-to-3d methods like occupancy networks and neural descriptor fields, which encode an image to an embedding space, where, paired with a 3D query vector, the model returns a vector describing quantities of interest. This setup is paired with operators similar to "neuro-symbolic concept learner", which gives the model a way to answer questions about the input image. The model is first supervised for occupancy and color prediction, and then supervised to answer synthetic questions. The model is trained and evaluated on toy-ish data consisting of single objects on white backgrounds, where it outperforms some baselines.

strengths_and_weaknesses: I like the idea of using an intermediate 3D representation for the grounding task. I think this is a good idea. But I think that the difficult part here is in inferring the intermediate 3D representation for the grounding -- which is not this paper's focus. Instead, the paper is focused on toy scenes where 2D-3D lifting is relatively easy, and then combines neurosymbolic-style operators to perform (and supervise) some grounding tasks.

I am not very familiar with the baselines here, so I cannot judge the improvement over them.

A major factor in my negative impression for this paper is the overstatement of results. When a paper claims to have done something, when in fact it only achieved a version so simplified as to be trivial compared to the real issue, it feels frustrating to me as a reader. 

The paper claims that segmentation emerges: 
> "we observe the emergence of natural 3D segmentations of concepts, at the semantic and instance level, directly from the underlying supervision of downstream reasoning."

I don't understand why this claim is here at all. As explained later, semantic and instance segmentation are supervised through the querying task (line 192). 


The paper claims that much of its training is self-supervised:
> "an intermediate representation should be discovered in a self-supervised manner"
> "descriptors are learned in a self-supervised manner, and implicitly capture the hierarchical nature of a scene"
> "After learning decoders Di through these self-supervised pre-training tasks..."

This is in reference to 3D shape and color reconstruction, from image input. This requires a 3D model or detailed multi-view reconstruction of the object created beforehand, before the method begins! (Not to mention it requires object-centric data carefully collected by a human.) 


The paper claims that it generalizes to unseen categories and real scenes:
> "It can also generalize well to unseen categories and real scans."
> "our 3D-CG trained on seen categories can be generalized directly to unseen categories and real scans"
> "estimate 3D-CG’s ability to generalize to real scenes" 

In fact, the experiment here completely abstracts away the model's ability to infer a 3D representation from a 2D image, and instead queries the model's segmentations at ground-truth voxels. As for the categories, the results on the synthetic car and bed in Fig4b are quite unusual -- ignoring the fact that the blue wheel is in the wrong place, how did the model know to estimate the occluded side of the car and put 4 wheels, when only 2 are visible? Was 3D ground truth provided here as well? 

questions: I asked a few questions in the Weaknesses part of my review. 
limitations: I think the written limitations greatly understate the true limitations. 

It says...
> while we show transfer results on real scenes

I don't think this was convincingly shown.

> an interesting direction of future work would be scale our framework to directly train on complex real-world scenes and ground more concepts from natural language on these real scenes is worth delving into in the future

This is so far beyond the paper's scope that it doesn't feel serious. 




****
Post-rebuttal update: During the discussion phase the authors gracefully engaged with my complaints and toned down the contribution claims, so I am increasing my rating. I think people are generally interested in efforts to attach task-relevant features to neural fields, and this paper presents a reasonable step in that direction.
review_title: null

final_score: rating: 8: Strong Accept: Technically strong paper, with novel ideas, excellent impact on at least one area, or high-to-excellent impact on multiple areas, with excellent evaluation, resources, and reproducibility, and no unaddressed ethical considerations.
review_content: summary: This paper presents a method for grounding natural language concepts in 3D neural fields of shape and appearance. The main contribution is to use neural descriptor fields, a continuous field and ground it in language concepts. The input to the method is a set of shapes/appearance and QA about that shape. No other supervision is used. Specifically, the paper looks at binary questions, counting questions, and segmentation questions. Furthermore, the paper presents a slot attention module and neural operators for counting/segmentation.
strengths_and_weaknesses: ## Strengths

- I quite like the idea of grounding concepts to neural fields rather than to 3D shape/appearance. This makes it continuous and offers previously unseen differentiability properties.
- The paper grounds concepts to descriptor fields without any explicit supervision. This is an exciting direction that could open up new research directions.
- The proposed neural operators, particularly for counting is novel to my knowledge.
- The results shown are promising and show improvement compared to baselines. I especially appreciated the inclusion of real image results.

## Weaknesses

- Firstly, it's not very clear what class of data the method operates on. Is it scenes or object classes? Only in the experiment section is this made clear (PartNet-Reasoning). This is an important detail that the paper ought to mention earlier.

- The paper seems to introduce a depth-map conditioned neural field (line 123), but details and differences to conventional neural shape fields are missing.

- Generally, the paper is lacking many details, especially in the shape and color representation parts, and the language parsing part. For instance, what kind of parsing does the seq2seq model do? Why were these three operators chosen? In the loss description, how exactly is L_{NDF} computed? Is there a reconstruction module that is missing in Figure 2.

- In terms of related work, the paper could have done a better job summarizing the latest advances. I am afraid that the related work is limited to a small set of works. I would recommend the authors take a look at a couple of recent works and papers cited within.

*Geometry Processing with Neural Fields, Yang et al. 2021

*Neural Fields in Visual Computing and Beyond, Xie et al. 2022

*Advances in Neural Rendering, Tewari et al. 2022

- In the datasets, how are the ~8 questions+answers per shape selected? Are these all related to the categories of tasks that the paper intends to solve, or are they more general? Also, the dataset has *both* questions and answers, not just questions. This should be mentioned in line 214.

Overall, this is really nice work that I would like to see at NeurIPS. The paper opens up new research directions that move the needle both in terms of 3D shape representations, learning with fewer examples, and concept grounding to 3D shapes. However, the related work section needs to be more comprehensive.
questions: In the rebuttal, it would be nice to see a plan to include more details that are currently missing in the paper (see above).

Will the PartNet-Reasoning dataset be publicly released?
limitations: Yes, there is a discussion of limitations and societal impact. This discussion is adequate.
review_title: null

final_score: rating: 6: Weak Accept: Technically solid, moderate-to-high impact paper, with no major concerns with respect to evaluation, resources, reproducibility, ethical considerations.
review_content: summary: This paper studies the problem of the approach and the effectiveness of the 3D concept grounding using neural implicit field representation. The authors focus on several types of questions and develop a pipeline including multiple stages. They also create a dataset containing the necessary types of questions to assist the experiment. The results show that by using 3D concept grounding information, the trained models outperform previous works and generalize well to unseen shapes.
strengths_and_weaknesses: Pros:
+ The paper is well written.
+ This paper studies a novel and interesting problem.
+ A dataset contains 3K images and 3 types of questions are created to facilitate the experiments.
+ A novel framework based on the neural descriptor field with multiple stages and attention blocks is proposed.
+ The results that demonstrate superior performance and generalization are promising.

Cons:
- The experimenting dataset is relatively small.
- The framework seems to be designed for a specific purpose (i.e. answering limited types of questions) rather than a general-purpose one. I don't see this framework can be easily extendable.
- A concern would be that it is well known that 3D labeling/datasets are hard to obtain, not to mention combined with question-answering data.
questions: Please see my comments in the "weaknesses" part.
limitations: The limitation is addressed in the paper.
review_title: null

final_score: rating: 6: Weak Accept: Technically solid, moderate-to-high impact paper, with no major concerns with respect to evaluation, resources, reproducibility, ethical considerations.
review_content: summary: The paper proposes a weakly-supervised method that learns to ground 3D concepts with RGBD inputs. The key idea is to introduce Neural Descriptor Field (NDF) to neural operators (filter, query, and counting). 3D concepts are shown to emerge from training on VQA dataset. They extend their method to part semantic segmentation and part instance segmentation and show some genarlziation ability to unseen classes and real scans.

strengths_and_weaknesses: Strength
+ The paper aims to lift concept reasoning to 3D space. I think the biggest strength is the weakly-supervised method. The method leverages differentiable neural operators such that 3D grounding can be trained with question answer pairs. It also naturally allows for better interpretability. 
+ With a bunch of modules, the paper is very well written and easy to follow. Toy examples are nicely introduced to help readers understand. 
+ Part semantic segmentation and instance segmentation are also shown as special cases of the proposed method.
+ The results look convincing to me. Since the problem is quite novel, the authors do a good job to compare with baselines. 

Weakness
1. The proposed method focuses on object-centric reasoning and the method may be hard to scale up to more complicated scenes because the NDF considers the whole 3D space as a continuous, per-point query space. However, Table1 may suggest that some primitive-based methods may have advantages if their emerging part aligns with the language guide. -- It is more of my personal suggestion than weakness. 
2. The generalization claim is quite weak as only some qualitative results are shown. For example, I wonder about the quantitative performance on novel categories for both baselines and their method. 


questions: see weakness 2 
limitations: The authors have addressed the limitations and potential negative societal impact of their work
review_title: null

Meta-review: metareview: The paper received all positive reviews (3x weak accept ratings, 1x strong accept rating). The meta-reviewer agrees with the reviewers' assessment of the paper.
confidence: Certain
Decision: Accept