Venue: NeurIPS 2022 Conference

Reviews:

final_score: rating: 5: Borderline accept: Technically solid paper where reasons to accept outweigh reasons to reject, e.g., limited evaluation. Please use sparingly.
review_content: summary: This paper proposes the SIG module to project RGB-D point clouds of object proposals to multiple views of RGB images, which acts as another stream of visual clues to help point cloud-based 3D visual grounding. In contrast to SAT, this method does not rely on actual 2D images to generate the 2D clues. The proposed method, also called LAR, fuses the aforementioned 2D visual tokens and 3D visual tokens by a visual transformer, and then the enhanced tokens in each stream are respectively fused with language tokens for the identification of referred objects, while these two streams of visual-linguistic tokens can be fused again by a fusion transformer for the final referring identification. The experiments on Nr3D, Sr3D, and Scanrefer datasets, show that the proposed SIG can consistently improve the performance in comparison to the SOTA. 
strengths_and_weaknesses: --- Strengths ---

1. The proposed SIG module can directly exploit the 2D visual clues from RGB-D point clouds, without acquiring real 2D images associated with point clouds, thus is more flexible than SAT. 
2. The experiments contain a list of comparisons, validating that the proposed SIG can consistently improve the performance than the way that SAT is used for exploiting 2D visual data.

--- Weaknesses ---

1. Other than the SIG module, the overall novelty of this paper is limited, in comparison to SAT. However, the proposed LAR model has to explicitly employ the SIG module both at the training and testing stages, it thus requires more computational budgets than the previous 2D-3D combined methods, such as SAT. The computational cost of the LAR model should be discussed.
2. The SIG module may require accurate object proposals for the rendering of 2D images, which is okay for Nr3D and Sr3D, but for actual scenarios such as ScanRefer, how to maintain the reliability of SIG-generated 2D data is not guaranteed. How will the performance degrade if the proposals are predicted?
3. Even with a bunch of sophisticated fusion techniques and extra computational cost introduced by the 2D visual stream, the proposed method just achieve marginal gains than SAT, which harms the significance of the claimed contributions.
4. The presentation of this paper needs significant improvement. For example, 
    - Tab. 2: What do referit3D and non-SAT mean in this table?
    - Tab. 3: is SAT$^\dagger$ means the re-implemented SAT?
    - How the camera and image augmentations are applied in the experiments other than Tab. 4. 
    - A number of typos and grammar errors throughout the whole paper.
questions: It is encouraging if the authors can answer the questions listed in the paper's weaknesses.
limitations: Yes, the authors have adequately addressed the limitations and potential negative societal impact of their work.
review_title: null

final_score: rating: 5: Borderline accept: Technically solid paper where reasons to accept outweigh reasons to reject, e.g., limited evaluation. Please use sparingly.
review_content: summary: This paper presents LAR, a novel method for the task of 3D language-visual grounding. The key idea of LAR is to leverage synthetic 2D images as an additional modality of data to the downstream object reference task. LAR outperforms all baselines on the selected 3D grounding benchmarks that do not use additional training data.

strengths_and_weaknesses: === Strengths ===
1. The paper is well written and the presentation is clear.
2. Ablation studies on the design choices are mostly thorough.
3. Strong empirical performance over previous methods.

=== Weaknesses ===
1. The proposed SIG module uses heavy heuristics and hand-engineering. I am wondering if this can be learned – given 3D scene point clouds and a single object proposal bounding box, predict (multiple) camera extrinsics. It seems this can be learned end-to-end as the projection function seems also differentiable.
2. While LAR uses BERT embeddings, I am wondering if switching to a image-language embedding will improve performance given the task requires modeling language and visuals together in the first place.
3. I am not sure if learning simple 1D conv kernel weights (seems more like learned weighted average…) is a good practice to fuse multi-view images, since it ignores the context.
4. Typo in L181-182.
5. The current model seems to ignore interaction among objects and predicts reference for them individually. I am wondering whether modeling objects together might make a difference in the performance.

questions: Same as 1, 2, 5 raised above in the weakness section

limitations: Same as 3 mentioned above in weaknesses. Additionally I would have liked to see if the setup is set closer with object detection: instead of assuming access to object proposal, predict objects directly given language description, since principally it seems language should also help bounding box proposal and localization. However, this is out of the scope of this paper and has not factored into my scoring.
review_title: null

final_score: rating: 7: Accept: Technically solid paper, with high impact on at least one sub-area, or moderate-to-high impact on more than one areas, with good-to-excellent evaluation, resources, reproducibility, and no unaddressed ethical considerations.
review_content: summary: The paper proposes a method to enhance the visual part of a 3D visual grounding system by integrating 2D semantic information. The system receives a 3D point cloud, a text query, and point subsets for the candidate objects as input. The paper proposes to render each candidate point set into multiple virtual cameras and use a transformer to fuse 2D and 3D information. The visual information is merged further with the text to classify the objects for finding the desired candidate.

strengths_and_weaknesses: Strengths:
1. In my opinion, the overall technical idea of the paper, to use 2D information together with 3D information for visual grounding, is novel and exciting. In contrast to 3D point cloud encoders, 2D feature encoders can be pretrained on large datasets and aggregate more information. Generating synthetic images through point cloud rendering also generates multiple views per object, increasing the chance of generating a dominant view of the object. 
2. The method achieves strong results in terms of classification accuracy, outperforming methods that use ground truth 2D images through rendering synthetic multi-view images. 
3. I appreciated the qualitative examples in the supplementary material of both success and failure cases. Maybe showing qualitative results of SAT and adding a figure to the main paper would increase the readers understanding of the significance of the results.
4. The paper contains code and a supplementary video.

Weaknesses:
1. Section 3.1, lines 145 - 168. In my opinion, this section is overly wordy, explaining a well-known concept (pinhole projection), which takes space away from other parts of the paper and could use additional context. The paper explains how to project 3D points to a pinhole camera plane. As this is a very well-known topic in the literature, it suffices to mention which conventions are used for the extrinsic and intrinsic camera matrices and how the camera axes are defined. This should at most take a short paragraph and can even be shortened by using equations cleverly. 

3. In contrast, section 3.2 does not contain enough information, in my opinion. Equation 1 contains the definition of how image information is fused with bounding box information, but this is merely stated as fact, without giving intuition about why this might be beneficial. Similarly, in lines 213-223, the paper presents the loss terms without a proper introduction. Most losses are not defined or only are merely referring to previous papers. In my opinion, this makes the paper hard to understand and not self-contained. Shortening section 3.1, the paper should contain enough space to add the missing details. 

4. Section 3.3 Software Details: Here, the paper mentions for the first time that image augmentations were used, without specifying what kind of augmentations and why this is needed. The type of augmentations is mentioned later in Line 299, but there might be a better place.

5. Section 4: The paper does not introduce the used evaluation metrics. These should be contained within the paper so that the reader can evaluate what the quantitative results represent.

6. The paper does not contain a qualitative comparison of the detection performance with state-of-the-art. I am guessing that the quantitative score measures classification accuracy, but it is not apparent how large of a difference the improvement the method provides quantitatively.

Minor issues, spelling, and grammar:
* Figures 4, 5, and 6 are of different sizes and, therefore, are not aligned. Please align them such that the captions align.
* Table 6, Overall and hard performance on Nr3D. The paper marks its approach as best, although SAT performs slightly better. Please change this. I do not think it is an issue that the presented approach does not outperform the reported numbers from SAT, especially regarding the reproducibility issues. One of the paper's strong points is that the SAT approach has been retrained in this case. 

I think the authors should address these weaknesses to polish the paper and make it more self-contained. If these issues are addressed, I am willing to increase my score.

questions: I have a couple of questions regarding the paper:
1. In the limitation section, the paper mentions using a detector instead of getting the ground truth point set proposals. How would this work with noisy detections? If points are cut out or added, this will drastically alter the information gained from the 2D encoder especially compared to a method like SAT, which uses ground truth 2D images. Also, if the goal is to use a robot, it would most likely be equipped with a camera and get a stream of 2D images. What was the rationale for not using this information and only using the synthetic 2D images instead? Line 110 mentions: "SAT [51 ] leveraged 2D semantics to perform the 3D VG task. However, requiring extra 2D inputs limits potential application scenarios." What limitations does the paper refer to here? As the motivation seems to be mainly for VR and robotics applications, where cameras are used either on the VR device or on the robot itself, are images not widely available?

2. The paper experiments show that it outperforms methods that have access to ground truth 2D images. How can this be explained? Are the single view images in the dataset taken from arbitrary angles? Is this paper just using a more advantageous camera angle? Or is this mainly due to the multi-view setting?

3. The paper focuses on indoor scenes by how it selects the viewing directions of the virtual cameras. How would this work in more open-world settings where the camera does not always point outwards? Would this approach still work?
limitations: The paper contains a concise section on the technical limitations, namely using ground truth candidate point sets (see my mention regarding this under questions). Negative societal impacts are not discussed at all.
review_title: null

final_score: rating: 5: Borderline accept: Technically solid paper where reasons to accept outweigh reasons to reject, e.g., limited evaluation. Please use sparingly.
review_content: summary: This paper studies 3D Visual Grounding. Its goal is to identify from a 3D point cloud the 3D object referred in a sentence. The idea in this paper is to utilize RGB image as another input modality to help the grounding task. Different from the previous method SAT [51], the RGB info in the proposed method is obtained by rendering the RGB point cloud from multiple viewpoints, while SAT [51] obtains it from camera input from one single view. The paper further proposes a novel architecture to fuse the rendered RGB input and the point cloud, leading to better performance than SAT [51].
strengths_and_weaknesses: Strengths:
+ The paper designs a clever rendering pipeline (SIG) to render the input point cloud into multiple images, which are fed into the network together with the point cloud. This strategy provides more RGB signal than pure camera inputs, and together with the novel network that fuses multi-modal inputs, the proposed method outperform SAT (which requires an RGB camera input). This is an effective strategy, and as the number of provided view increases, the performance increases (Table2).


Weaknesses:
- I think the strategy designed to render the point cloud seems debatable, and it is possible that there is another object blocking the path from the camera center to the proposal object, resulting in occlusion. In this case the rendered image would be incorrect. 
- It seems that the proposed method requires a trained image classifier? If that is the case wouldn't that make the comparison with SAT [51] unfair because more information is given to train this framework? 
questions: Overall I feel that a lot of details are missing from the paper. I think adding those details are necessary to make the paper self-contained and easier to be understood. For example:
1. Where do the 3D object proposals come from during training and inference?  How do you know what is the upper and lower faces of the bounding box as stated in line 136?
2. Where does the predicted label z'_i in line 197 come from? From a trained image classifier? How is it trained? How many classes is it producing?
3. What is the "language classification head" in line 203? What is its purpose?
4. How exactly are the losses in line 213 to 220 defined?


I am also curious to know what the performance would be if the proposed method uses the same RGB input as SAT, i.e. one single image from camera. That would make the proposed SIG more justified.
limitations: The authors have adequately addressed the limitations and potential negative societal impact of their work
review_title: null

Meta-review: metareview: After the rebuttal and discussion the paper received one weak accept, and three borderline ratings (2 ba, 1br). The concerns of the only reviewer leaning towards rejection are well addressed, as such the AC sees no reason to reject this paper.
confidence: Less certain
Decision: Accept