Venue: NeurIPS 2021 Conference

Reviews:

final_score: rating: 7: Good paper, accept
review_content: summary: The paper proposes a scene graph prediction method for 3D point clouds. The paper uses the graph network to learn class label based embeddings. It does not use appearance information from the point clouds to avoid perceptual inconsistencies. The class label dependent prototypical embeddings are learned using a graph auto-encoder with directed edges and a message passing mechanism tuned for such edges. The scene graph prediction works by extracting embeddings from subject and object points and object pair relationship in form of an edge embedding. The embeddings (geometric, meta embeddings) are fused with an MLP. The network is trained with focal loss. The proposed method is evaluated on 3DSSS dataset and achieves state of the art results on various metrics. 
main_review: - The main novelty of the method lies in utilizing meta embeddings from class labels and fusing them with geometric embeddings to obtain the scene graphs. The approach, in my opinion, builds upon geometry based methods on point clouds [A], where the appearance information is not necessary to learn robust descriptors. However, overall adding class label based embeddings and the fusion mechanism seems novel with reference to the existing literature. 

- The paper is well written and motivated. The contributions are highlighted clearly. However, main premise of the paper is to avoid appearance noise. In order to validate this hypothesis, authors may include such features and demonstrate the change in performance.

- I was able to reproduce the resutls (with 2% variance from the reported results on various metrics) with the code provided by the authors on an NVIDIA RTX 8000, 256 GB RAM.

- How do you chose the five best meta embeddings for fusion. Is it based on the attention maps ? (Line 167). What happens if the number of embeddings is more/less ?

- What is the motivation behind using Focal Loss ? The main advantage of using focal loss is during class imbalanc. Is it due to sparse mapping between the object-subject pairs here ?

- How does the technique perform with noisy data/points say sampled at a lower resolution ?

- There are a few typos
(i) Fig 1 "Labal" -> Label
(ii) Line 169, "sapces" -> spaces

[A]: Guo, Yulan, et al. "Deep learning for 3d point clouds: A survey." IEEE transactions on pattern analysis and machine intelligence (2020).
limitations_and_societal_impact: The authors have described the limitations adequately in Section 5. However, mentioning the constraints on the subject-object pairs and type of relationshio would help readers to extend the work in other domains.
review_title: null

final_score: rating: 6: Marginally above the acceptance threshold
review_content: summary: The paper provides a novel two-stage pipeline for 3D scene graph prediction tasks on point cloud data.
Novel relation and object embedding methods are proposed.
Experiment results show that the proposed pipeline gives higher performance compared with related methods. 
main_review: Originality:

The two-state pipeline is of great novelty.
The embedding method proposed is effective.

Clarity:

The overall idea is expressed clearly.
But details on network structures are missing or fuzzy.

In meta-embedding learning auto-encoder, it is unclear to me how the network structure is obtained.
Judging by Fig 1 and Fig 2, the input contains only one-hot embeddings.
But GNN requires a network structure to operate on.
I conjecture that the graphs come from all scene graph in the dataset.
Then in that case, the input should further include graph structure instead of only labels.

Judging by Fig 3, there is a recurrent structure in the scene graph prediction model.
It is not clear to me what will happen in the first iteration before any meta-embedding can be obtained.
Without meta-embedding, updates described by equation 8 cannot be executed.
Other questions are how many iterations are required and will the classification accuracy improve over iterations.

Quality:

The work is complete and the model is sound.
Evaluation is comprehensive and convincing.
The limitation is discussed in detail.

Significance:

3D scene understanding is an important research direction with great potential.
The performance of the model proposed in this paper surpasses a large body of previous works.
The embedding method is inspiring and could potentially be used in other similar tasks.
limitations_and_societal_impact: Yes
review_title: null

final_score: rating: 7: Good paper, accept
review_content: summary: This paper proposes a knowledge-inspired method for 3D scene graph prediction on point cloud. The key is to decompose the problem into two sub-tasks. First, a graph auto-encoder called meta-embedding is introduced to learn class-dependent prototypical representations only from the class labels to avoid visual confusions. Then a scene graph prediction model selects related meta-embedding from the first task as prior knowledge to predict credible relation triplets of 3D scene graph. Benefited from the learned commonsense knowledge, the scene graph prediction model achieves good performance on 3DSSG dataset.
main_review: Strong points: 
1. The idea to learn knowledge solely based on class labels and their regular graphic structures is novel, which avoids the visual confusion introduced by the perceptual errors in previous works. It may bring some insights to other knowledge-inspired tasks. 
2. The experiments and visualizations are thoroughly provided, while the weaknesses are also discussed. 
3. The paper is well-written and easy to follow.

Weak points:
1. Line 244-249 argues that the limitation of the model in object identification can be possibly solved by using a stronger point set encoder than PointNet. However, Table 1 shows that the method w/o ME can also get a good result (using PointNet), where the problem is that the performance gain in SGCLs task is not enough to fully prove the effectiveness of ME. This is expected to be discussed in the rebuttal.
2. In Line 268, the performance on frequent categories is damaged by the model. However, the reason to cause this problem is not fully discussed.
3. Line 167 states that the five most confident embeddings are selected, will the number of the embeddings influence the result? Since the selection of prior knowledge is also the key in to utilize the knowledge, this detail is expected to be discussed.
4. Figure 6 of Schemata [1] clearly confirms that object representations will get into more separable clusters after injecting their prior knowledge. Although Line 255 argues similar effects, the same visualization comparison is expected to be provided.
limitations_and_societal_impact: The societal impacts are fully discussed. 
For the limitations of the method, the reviewer hope to know the robustness of the proposed method to the perturbation such as rotation, translation and scaling. The robustness to the perturbation on 3D point clouds is very important in real-world applications. Since the proposed method learn the knowledge solely from labels, it may be robust to the real-world perturbations, which is expected to be discussed in the rebuttal.
review_title: null

final_score: rating: 5: Marginally below the acceptance threshold
review_content: summary: In this paper, the authors study the 3D scene graph prediction task. The major contribution is the proposed “object/predicate meta-embedding,” which learns class-specific embedding to assist scene graph prediction. Ablation study verifies the effectiveness of the proposed embedding learning method.
main_review: Strengths: 
1. The additional object and predicate embedding improves the scene graph prediction performance.

2. The analyses and feature visualization show that the meta-embedding does learn class-specific representation as desired.

Weaknesses: 
1. In the claimed contribution (Line 53), it is unclear what does the first point (“we introduce a novel ... method that directly functions on 3D scene point clouds”) refers to. The framework seems to have the same input-output as the previous 3D scene graph prediction studies [29]. 

2. The “Knowledge-inspired Scene Graph Prediction Model” section and Figure 3 are confusing. The two-round approach (the loop from object prediction to initial feature) is barely introduced in the text, and thus makes the inference pipeline difficult to understand.

    A question is that whether more than two rounds with more accurate object/predicate prediction further improves the performance.

3. In addition to the reported metrics in image scene graph prediction, it might be helpful to also benchmark the studies following previous 3D scene graph studies’ metrics [29].

4. a. The term “commonsense” could be misleading. The learned “object/predicate meta-embedding” is more like external class-specific memories, instead of “commonsense” or “knowledge.”

5. It would be helpful to conduct additional ablation studies and analyses on the meta-embedding design. For example, what if keeping the meta-embedding but learn it E2E instead of using the auto-encoder-like structure.

6. Despite the performance improvement, the extra embedding also increases the training and inference cost, as it requires two inference rounds and storing additional memory-related parameters.

7. The proposed method seems generalizable to image scene graph prediction. It would be helpful to also benchmark the study on image scene graph prediction, or show the unique design for the 3D task.

limitations_and_societal_impact: The limitations and societal impacts are discussed in the conclusion section.
review_title: null

Meta-review: This paper proposes a method for 3D scene graph prediction from point clouds in which  a graph auto-encoder model learns prototypical representations for object categories using scene graph annotations, that are used  as prior knowledge during scene graph inference from a point cloud input.  Reviewers acknowledge the novelty of the autoencoding model proposed for learning categorical priors. 
Reviewers point out that the paper is not clear regarding the input of the autoencoding graph model. They further point out that vague words like ``common sense” are not well defined and are used arbitrarily in the paper. Authors are strongly encouraged to clarify the paper writing (and corresponding figures), following reviewers’ comments.

Decision: Accept (Poster)