Knowledge-inspired 3D Scene Graph Prediction in Point Cloud
Authors: Shoulong Zhang
Shuai Li
Aimin Hao
Hong Qin
Venue: NeurIPS 2021 Poster (2021)
Keywords: 3D Scene Graph
Knowledge Embedding
Deep Learning

Abstract: Prior knowledge integration helps identify semantic entities and their relationships in a graphical representation, however, its meaningful abstraction and intervention remain elusive. This paper advocates a knowledge-inspired 3D scene graph prediction method solely based on point clouds. At the mathematical modeling level, we formulate the task as two sub-problems: knowledge learning and scene graph prediction with learned prior knowledge. Unlike conventional methods that learn knowledge embedding and regular patterns from encoded visual information, we propose to suppress the misunderstandings caused by appearance similarities and other perceptual confusion. At the network design level, we devise a graph auto-encoder to automatically extract class-dependent representations and topological patterns from the one-hot class labels and their intrinsic graphical structures, so that the prior knowledge can avoid perceptual errors and noises. We further devise a scene graph prediction model to predict credible relationship triplets by incorporating the related prototype knowledge with perceptual information. Comprehensive experiments confirm that, our method can successfully learn representative knowledge embedding, and the obtained prior knowledge can effectively enhance the accuracy of relationship predictions. Our thorough evaluations indicate the new method can achieve the state-of-the-art performance compared with other scene graph prediction methods.

Knowledge-inspired 3D Scene Graph Prediction in Point Cloud
Shoulong Zhang Beihang University
Shuai Li Beihang University
shoulong ong zhang9buaa, edu. cn
Peng Cheng Laboratory lishuai catiaai@bbaa..mm edu. cn
Aimin Hao
Hong Qin Stony Brook University (SUNY)
Beihang University
Peng Cheng Laboratory ham@buaa edu. en
qinêcs stt.ecbrosttoboooooooooo edu
Abstract
Prior knowledge integration helps identify semantic entities and their relationships in a graphical representation, however, its meaningful abstraction and intervention remain elusive. This paper advocates a knowledge-inspired 3D scene graph pre- diction method solely based point clouds. At the mathematical modeling level, we formulate the task as two sub-problems: knowledge learning and scene graph prediction with learned prior knowledge. Unlike conventional methods that learn knowledge embedding and regular patterns from encoded visual information, we propose suppress the misunderstandings caused by appearance similarities and other perceptual confusion. At the network design level, we devise a graph auto- encoder automatically extract dependent representations and topological patterns from the one-hot class labels and their intrinsic graphical structures, so that the prior knowledge can avoid perceptual errors and noises. We further devise a scene graph prediction model to predict credible relationship triplets by incorporat- ing the related prototype knowledge with perceptual information. Comprehensive experiments confirm that, our method can successfully learn representative knowl- edge embedding, and the obtained prior knowledge can effectively enhance the accuracy of relationship predictions. Our thorough evaluations indicate the new method can achieve the state-of-the-art performance compared with other scene graph prediction methods.
Introduction and Motivation
Scene graph has been studied as a viable means towards better interpretation of scene context, which encodes the semantic elements and their complex relationships of scene [6 32 33
31
As
an enriched scene structure representation, scene graph parsing from 3D data has recently interested researchers 3D vision for the tasks of scene understanding 2]. VR/AR scene interactions 25 and robot navigation [7]. However, the scanned 3D data has intrinsic nature of incompleteness [1 24).
other visual challenges. These perception errors result in extra difficulties in object and relationship recognition purely based on visual inputs.
geometrical similarity across categories,
Fortunately, in cognitive psychology, besides visual perception, human beings also associate objects and concepts with certain abstract symbolic class-dependent representations and regular combinations
"Corresponinng authors.
35th Conference on Neural Information Processing Systems (NcurIPS 2021).

---

Meta-embedding Auto-encoder
010000
010000
000100 Class Label Inputs
000100
Class Label Outputs
Knowledge Upéate
Meta- embedding
Knowledge Injection
Knowledge Selection
Scene Graph Prediction Model
Meta-embeddiing Latent Features
Point Cloud Inputs
Scene Graph Outputs
Figure 1: Overview of the knowledge-inspired 3D scene graph prediction method. We decompose the problem into two sub-tasks. First, we learn the meta-embedding only from the class labels with their graphical structures avoiding perceptual errors. The figure on the right shows that our meta-embedding (stars) can effectively represent the latent class features (points) in the latent space. Then, our scene graph prediction model selects related meta-embedding as prior knowledge to classify object entities and their relationships with point cloud perceptual features.
of scene context, known as knowledge. However, utilizing prior knowledge still remains chal- lenging in scene graph prediction. Prior knowledge-incorporated methods have several limitations. Static knowledge (e.g., heuristic co-occurrence statistics (37). mined triplet facts (8), or knowledge graph (35)) could be incomplete and inaccurate depending on the quality of knowledge sources and the domain knowledge covered. Thus, several methods tend to use learning- knowledge capture the graphical structure and regular combinations in the exact domain. However, the knowledge learned from visual information also inherits perceptual confusion into the knowledge
embedding and network parameters 222 36
To ameliorate, this paper advocates a novel 3D scene graph prediction method with Iearned prior knowledge. The knowledge can be encoded in various forms (e.g., vectors, graphs, and natural language). In our work, we choose class-dependent prototypical memories as the carrier of lndgledge to tackle the key challenges mentioned above. We re- formulate the target problem into two sub-tasks: knowledge learning and knowledge-nnspired scene graph prediction. Unlike previous works, we learn the knowledge only depending on semantic categories and regular structural patterns without any visual confusion. The knowledge learning model is graph auto-encoder that takes the one-hot class labels as inputs with scene graph structure annotations. Since we do not involve any geometric and visual features, the latent embedding indeed encodes only category- category-related regular structure patterns. To correctly record this prior knowledge, we enforce group of prototypes [23), namely meta-embedding to approach the latent areas of the corresponding category in the metric space. As we illustrated in Figure [ the meta-embedding is capable of representing each semantic class as prototypical knowledge. Once trained, the scene graph prediction model unites the geometric features and the corresponding knowledge meta-embedding to predict each element in a triplet with the <subject, predicute, object> structure. Extensive experiments confirm that the effectiveness of knowledge extraction and the injection of the meta-embedding can improve the scene graph prediction accuracy.
The primary contributions of this paper could be summarized as follows: (1) We introduce novel knowledge-inspiree scene graph prediction method that directly functions on 3D scene point clouds. To achieve our goal, we re-formulate the problem into two sub-tasks: knowledge learning and scene graph prediction with knowledge intervention; (2) We propose graph auto-encoder, called meta-embeddingg learn group of -lassddepnndent prototypical representations. The learned class prototypes can avoid perceptual errors and effectively encode prior structural knowledge in a more distinguishable latent space; and (3) We design a scene graph prediction model benefiting from the prior learned knowledge in order to predict credible relation triplets. With our learned knowledge, our scene graph prediction model can achieve state-of-the-art performance on the 3D semantic scene graph (3DSSG) dataset (29] compared to currently available methods.

---

2 Related Work
Knowledge- -aaed Scene Graph Classification, Prior knowledge has been proven as effective source of information to enhance object and relation recognition. Early methods use statistical co-occurrence extra knowledge for scene graph inference. Zellers al. observed that the predicate distribution is strongly dependent on the head and tail categories. It introduces a pre-compute bias into the final prediction [37]. The biased results naturally improve the most frequent combinations. To improve the classification performance on the rarely sampled ones, knowledge-embee embedded routing network (KERN) [4] implicitly fuses co-occurrence probabilities as weights in the message pass step to neutralize the long-tail phenomenon. Nevertheless, this integrated knowledge can hardly be used and studied in isolation since it is hard-coded as intrinsic parameters. Besides, some methods bring external knowledge bases into scene graph inference. Gu et al. extracted knowledge triplets from the ConceptNet knowledge bases [8). Graph bridging network (GB-Net) adopts auxiliary edges as bridges that enable and facilitate message passing between knowledge graphs and scene graph [35]. This hand-crafted external information is highly dependent on the organization of knowledge and the quality of sources. Furthermore, several methods try to incorporate learning- based commonsense into scene graph inference. Zareian et al. proposed a local-global graph transformer to complete masked scene graph by embedding the regular patterns into the network parameters [36]. More recently, Sharifzadeh et al. encoded inductive biases into the class-level prototypical representations learned from all previous perceptual outputs [22] Inspired by [22]. our method also adopts prototypical representations as prior knowledge. Nevertheless, our method focuses on class-dependent and graph structure information, which aims to combat the confusion resulting from perceptual errors.
3D Scene Graph Prediction. 3D scene graph has only recently been studied in the 3D vision community. There are mainly two approaches for 3D scene graph construction. The first group of works benefits from image-based scene graph techniques. Armeni et al. firstly introduced a 3D scene graph construction framework with 3D meshes and registered RGB panoramic images [2]. It adopts multi-view consistency to enforce the accuracy of object identification by 2D-3D projection. Furthermore, succeeding methods [13 20 execute image- based scene graph prediction within
dynamic simultaneously localization and mapping (SLAM) [2] 17 30]. Kim et al. proposed a framework that merges the local graphs into global one by fusing the same nodes in different images [33. Spatial Perception eNgine (SPIN) distinguishes human and object nodes by dynamic masking 120]. The second group of approaches directly process 3D data. Recently, Wald et al. proposed an end-to-end network, scene graph prediction network (SPGN) [29]. which takes scene scans as input. Along with the network, they also published the 3DSSG dataset of reconstructed scene scans. Similar to this work, we investigate a novel model to extend the scene graph prediction methods using 3D scene data solely based on point clouds.
3 New Method
Problem Formulation. We define scene graph 9 as a directed graph with sets of nodes and edges, G = [V,E). The node set V includes all object instances of scene, and the edge set & consists of all possible connections of nodes as predicates in a relation triplet <subject, predicate, object>. The edges are directed by connecting neighbor nodes from subject to object. Unlike previous knowledge- embedded approaches [37 36. we formulate the seowedde-insppree scene graph prediction task as multi- -tsk problem: we intend to learn knowledge and scene graphs from 3D data simultancously. Mathematically, given a 3D scene data D with scene graph annotations, the training process is aimed to predict the probability of P(G, K D). where K is the prior knowledge. Furthermore, we can decompose the joint probability into two individual components:
P(g, D) = P(G)) K).
(1)
This decomposition breaks the problem into two sub-tasks. The first sub-task, the predication of P(K)D), extracts knowledge from existing 3D scene configurations. In this paper, the learned knowledge is in the form of prototypical embedding of each object and predicate category, named metad-emed
= ) The set of meta-ent eeetaeembediing is expected to represent each semantic class and also encode the corresponding class's 's regular patterns in scene graphs. Towards this goal, we design a graph auto-encoder network, which only takes the one-hot labels as input without any perceptual information. The graphical encoder also can extract the topological interactions of each class. We close the distance of latent outputs and the related meta-embedding in metric space so that the meta mebbddnm records the prior knowledge of the training data.
3

---

Object Knowledge Update
Object
Latent Embedding
Object Label
Object Prediction
Object Meta-embedding
MLP
MLP
Predicate Label
Predicate Prediction
Predicate Meta-embedding
MLP
GNN
MLP
Predicate Latent Embedding
Predicate Knowledge Update
Figure 2: Overview of the meta-embedding learning auto-encoder.
The second sub-task, D, incorporates the meta-embedding and perceptual features to predict the possible scene graph elements. It is also the expected mathematical model in the evaluation stage with unseen 3D scene data.
Meta-embedding and Learning Model. This model is designed to learn class-dependent prototypi- cal knowledge embedding with a graph auto-encoder. The overview of the meta-embedding learning model is illustrated in Figure 2 Since we consider the class-related patterns instead of perceptual information, the auto-encoder only takes the one-hot class labels as input. The first two multi-layer perceptron (MLP) blocks lift one-hot vectors into latent space with the same dimension size. Then, we assume that each object pair can have a relation (including none relation) and fully connect them graph where predicates encode the graphical patterns each node and edge in scene graphs by message passing [6]. At the network design level, adapt the message passing mechanism the directed graph structure. To calculate the latent representations (h for all nodes E V and (har) for all directed edges (u, v) E & in every layer the message passing of nodes and edges can be formulated as:
represented edges. We design graph neural network to further
h h GRU (br.,, mmd (v)),
e.1+1 GRU (had m m vu,v),
(3)
where m² and m are the parameterized neural networks that calculate messages in layer I. We adopt the gated recurrent unit (GRU) [5] to update the hidden state for reserving initial information. worth noting that a node could be subject or object in relation triplet. We consider both conditions in message computation. For node and one of its neighbors the message as subject m²d(w) and as object m (w) are computed by:
m s (v) I.N (bar) (h?")),
(4)
m o (v) = (o, (h?")))
(5)
where &... 6p. and & are three non-linear transformations for subject, predicate, and object respec- tively. To stabilize the numerical calculation recurrent mechanism, we adopt the layer normalization (LN) 3 after feature fusion. The final message of node is the average of m and m
m md (v) = R
Σ (m))(1)) + Σ (m²+(())
(6)
R2
MER:
HER:
where |- denotes cardinality and RS and R, are the set of connections of node v as subject and as object respectively. The calculation of edge (u, message is:
m m,
(o. (b%.') (h.
We use the sum pooling operation the stacked hidden states as the output model of the graph encoder. Since we know the class labels of the latent output embedding of the graph encoder, we update the corresponding meta-embeddding vectors by reducing their distances (e.g., Euclidean distance) in metric space. Given the latent embedding of nodes and those of edges, we select the corresponding class vectors from the object meta-embedding and the predicate meta-embedding. Then reduce the distance between those embeddings loss function. As the values of

---

Object Meta-embedding
S Addd
Object Point Cloud
MS
MLP
Object Prediction
PointNet
Norm
Knowledge
Knowloddee
Injection
Point Cloud
Object Pair
MS PointNet
3 MLPP
2 RPP
Bard &
GNN
MLP
Predicate Prediction
Predicate Meta-embedding
Figure 3: Overview of the scene graph prediction model with the learned meta-embeddding.
eeetadembdding are initialized network parameters, they can eventually be updated with back- propagation [21]. The meta-embedding is updated only when seeing the related category, not depending on the number of training samples. This property makes the meta-embeddings qualified to be prototypical representations for even less frequent categories. The last two MLPs predict one-hot labels to enforce the latent embedding class-related and complete the auto-encoder structure.
nooweedge-nsppieed Scene Graph Prediction Model. This model is aimed to fuse prior meta- embedding into the scene graph prediction task. Figure 3 shows the pipeline of the knowledge- inspired scene graph prediction model. Taking scene point cloud with instance masks as input, we expect to encode geometrical features from both objects and object pairs. Based on the pioneering point set encoder, PointNet [18], we enhance the original structure to capture multi-scale geometric information like several point completion methods [10 34
For the object nodes, we directly use our modified multi-scale PointNet (MS PointNet) on the normalized point clouds. We first encode subject and object point sets separately for the object-pair edges and concatenate the latent features with the center coordinates of two point sets. To better encode the contradictory relationships (e.g.. left and right, lower than and higher than), we adopt subtraction as an mnti-symmetric operation on these two features. After transformation by a two-layer MLP, we obtain the initial edge embedding from the object pairs.
We adopt the same graph neural network design in the meta- meta-embedding learning stage to capture the global context with perceptual information. In the first iteration, the graph neural network only takes the geometric features of the PointNet as input. The last two MLPs output the node and edge classification possibilities and offer the clues to select meta-embedding vectors of related categories. We select the five most confident meta-embedding for each node and edge and fuse them with the perception features. Based on our experiments, the large number of selected embedding is inevitable to introduce wrong knowledge, while small meta-embedding set might contain proper knowledge. In practice, we choose five meta-embedding as a compromise. In the second iteration, since the perceptual embedding and the knowledge are in different latent spaces, we transform them by two-layer fred-forward networks if and g respectively before directly fusing them. Denoting by (emetaa)i= the selected meta-embeddeee and x the features encoded by multi-scale PointNet, fuse the knowledge and perceptual information similar as in [22]
LN(((x) Σ g(eenea)))
(8)
X = LN + (Z))).
(9)
where db is a two-layer MI.P and & the Enowledge-embedded input of the graph neural network in the second iteration. In practice, we only implement two iterations considering the time consumption since the classification performance can not be significantly improved over more itcrations.
Loss Function Design. We use the focal loss for object and predicate classification 14 29 in both embedddn learning and scene graph prediction tasks due the data distribution imbalance. The focal loss we used is formulated as in [29]
Lforat ael 7) log(p),
(10)
where p the logits of a prediction of object or predicate, the normalized inverse frequency for object classification and an edge/no-edge weight for the predicates, and the y is hyper-parameter
5

---

We follow the parameter settings in 129]. To enforce the meta-embeddding close to the class latent vectors in metric space, we adopt the Euclidean distance d between the output of the graph encoder and the corresponding meta-embedding as loss:
Laist
Σ d( dh,, Sineta) DEV
Σ
Ed(he )meta); EEE
(11)
where meto and mete are the corresponding mela-embedding of the categories of node v and edge In the meta- eeteaddibddiigg learning task, we formulate the loss function as:
meta
- + Froat + -disst-
(12)
Empirically, use the summation of the three components with the same weight. In the scene graph prediction task, we only use the focal loss for object and predicate classification.
Lag =
XC focat
foeat
(13)
where À is hyper-parameter to re-weight the components in 29
4 Experiments and Evaluations
Dataset and Task Description. We train the meta-learning auto-encoder and the scene graph prediction model on the 3DSSG dataset [29] p a 3D scene graph dataset based on 3RScan [28]. The dataset annotates support, proximity, and comparative relationships among daily indoor objects. With the same sub-scene split in 29). there are 3582 scenes in the training set and 548 for evaluation. On average, there are 9 objects per scene of 160 categories with 27 kinds of relationships, including none relation. We evaluate our model and compare it to others in two standard tasks proposed [32]. (1) The scene graph classification (SGCIs) scenario is aimed to predict both object and predicate semantic classes. (2) In the predicate classification (PredCIs) task, the model only needs to predict the predicate categories with the ground truth labels of object entities.
Implementation Details. Our model is implemented in PyTorch. We trained our model on an Nvidia RTX 2080Ti GPU in personal computer platform for 40 epochs with the ADAM optimizer. The initial learning rate is set to 0,0001, and the decay rate is 0.7 for every ten epochs. We trained the meta- embedding learning network and the scene graph prediction network separately. We followed the focal loss parameter settings in [29. In the scene graph prediction model, we used pre-trained multi- scale PointNets to encode the point cloud. The multi-scald PointNet is also trained on the 3DSSG dataset. We replaced the RCNN [9 image encoder with the same pre-trained PointNet used in our model to compare with image-based methods 44 22 29
We selected the meta-embedding vectors according to the five most confident class predictions in the knowledge fusion stage. In the PredCIs task, assigned the corresponding object meta-embedding as the input of the graph encoder since the ground truth of objects is known. We trained each model three times to calculate the standard deviation. All models are trained on the 3DSSG dataset with the same random seeds and the same split for a fair comparison.
compute the recall at the top-n (R (n)n triplets predicted by each model. The triplet is considered correct when the subject, predicate, and object are all valid. As defined in (15), the recall is the fraction of the correct top-n triplets against the ground truth. Additionally, since the long-tail phenomenon in the dataset annotations, we also adopt mean recalls @)) [4] 26] to evaluate the performance on the unevenly sampled relations. The mean recall is computed as the average of the recalls on each predicate class.
Metrics. Conventionally,
We introduce several baselines to compare with our proposed method. Co-Occurrence: we propose a relation prediction baseline with only co-occurrence statis- ties of the training set. This baseline outputs the most frequent predicate given the subject and object categories predicted by the multi-scale PointNet. This simple model draws a scratch line for
Comparison with Related Methods.
We used the 3DSSG dataset 29 under the permission of its creators and authors. We contacted the author by email (johanna. wald @ tum.de). We implemented the KERN model based on its released code (https: //gi::///tttbb Sean///wwihhao//BB KEERN with MIT license. The SGPN and the Sechemata models have no public code for now. We reproduced them based on their papers.

---

pillow
left right
pillow
left
taying on
hand dryer
door
sink-GT
slanding ONL
len neta
Jamp-GT
sofa
Cacaacai on
right
close by
by
bench
stardling on
trash
manddin
right
wall
Waanding
lofk
door
wall-GT
floor
floor
pillow
pillow
clese
armchalr
safa
Welt
Fight
coffe table
Jeva
Diiqger thar
DONE
Bipre
pillow
pillow
Sying
thas
plant
Handing do
wandins
plant
floor
GT
Klanding
standing
standing on
floor
wall
plant
ottoman
-G1
Landing
blinds-GT
picture
table
Positive Node
Negative Node
True Positive Edge
False Positive Edge
False Negative
Figure 4: Examples of predicted scene graphs from our proposed method. The green nodes are correctly classified by our model. The orange nodes are false negatives. The green edge results are true positives at the R @50 setting. The blue edges are correct in commonsense but not annotated in the ground truth. The orange edges the annotated relationships missed by our model.
knowledge- based methods on the evaluated dataset. Since this model is deterministic. the standard deviation is zero for both SGCIs and PredCls tasks, KERN: the knowledge-embedder routing network
a robust statistical knowledge-base baaee method that incorporates the co-occurrence probability into the scene graph prediction [4. SGPN: the scene graph prediction network is proposed with the 3DSSG dataset, which the first scene graph prediction model evaluated on this dataset [29]. Schemata: this baseline is recently proposed scene graph prediction model with learning sarning-based knowledge. It
encodes the class-dependent prototypes from the previous decoded perceptual results 22 which is similar approach to ours.
Table 1: Quantitative results of the evaluated methods in SGCIs and PredCIs tasks in recall, with and without graph constraint (GC), We additionally report our method with and without the meta- embedding (ME) intervention. The standard deviation is reported after the mean recall value.
SGCIs
PredCIs
Model
R @ 20
R@550 50
R@ 100
@0
@@50
R@ 100
Co-Occurrence
0,148 1:0000
0.197
0.199
0.347 1:0000
0.474
0.479
KERN 4
0.2031.0000
0.224
0.227:0008
0.468 1.0001
0.557=0..00
0.565
w/
SGPN 29
0.270
0.288-0.mm
0.290=20001
0.51944.m4
0.580=
0.585
GC
Schemata
0.27410003
0.292_000054
0.
00.4871..004
0.58220.007
0.591 006
Ours (w/o ME)
0.282
0.299_0.00
0.301 0:0
0.529
0.592=00.004
0.598
Ours
0.285 +-0001
0.300 00,001
0.301 +0.001
0.593
0.650
0.653+4.000
Co-Occurrence
0. 0.141 1:0000
0,202=0.000
0.0
0.351 1:0000000
0.556=0.000
0.706
KERN 4
0.208 000
0.247=0.000
0.276:1000
0.483
0.648
0.772.1001
w/o SGPN
0.282-10001
0.326=0.001
0.353
00.45510000
0.701=2.000
0.824:10.00
GC
Schemata
0.288
0.335 0.335=0.015 -0.01s
0.363
0.496-+0.00z
0.67 1:0.00
0.800-+0.009
Ours (w/o ME)
0.293++0001
1.,388-0.md
0.367
0.549
0.7 0.00000
0.824
Ours
0.298
0..43
0.370
0.622+ta.0s
0.784
0..83420000
Table 1 shows that our model achieves the best relation recall than other approaches and the state-
of-the-art performance in both SGCIs and PredCIs tasks. It confirms the effectiveness of our graph network design and the intervention of the meta-embeddingg, Additionally, show the results

---

Table 2: Quantitative results of the evaluated methods in SGCIs and PredCIs tasks in mean recall. We additionally report our method ablated on the meta-embedding (ME) intervention. The standard deviation is reported after the mean recall value.
SGCIs
PredCIs
Model
mR @:20
50
mR @:20
mR @ @50
mR 100
Do-Oecurreecee
0,088
0.127 1:0000
0.129 0.000
0,338 0.0000
0.474:10.000
0,479-00000
KERN 4
0.015:10001
0.115 1.0.12
00009
1..007
000007
0.256
0.265 0,009
SGPN 22
0.197
0.226 1:00066
0.23 0.000
0.321 0.3211::00 0.0004
0.384-100.00
0,389
Schemata 22
0.23810012
0.270
0.272200000
0.3522
0.426
0.433_0..0s
Ours (w/o MF)
0.219
0.242+
0,245
0.353
0.4 0.4
0.415
Ours
0.244
0.286=1000
0.288=000077
0,566:10011
0.635
0.638_0.00
evaluated in mean recall in Table Our model can still achieve the best performance. Our model improves the mR 100 of 4.3% and 22.3% in SGCIs and PredCIs tasks with the meta-embedding compared to our ablated version without learned knowledge. This comparison proves that our meta-em can improve the predictions marginally sampled predicate categories. We also illustrate several examples of predicted scene graphs in Figure 4 We provide more examples in the supplementary material.
Compared with the Schemata method, our model performs better in relationship prediction tasks. The reason behind the difference could be that our metedembe encoded from a more distinguishable latent space than the schemata, as shown in Figure The visualized results confirm that the knowledge learned from the class information is more representative than that from visual information with inevitable perceptual confusion.
However, there is massive gap in recall values while comparing the SGCIs and the PredCIs tasks. The performance of relationship prediction drops gravely without the object annotations. It presents the limitation of our model in object identification. This failure caused by the severe geometric incompleteness of point clouds and the long-tail distribution of the annotations. A potential solution using stronger point encoder than the PointNet extract robust geometric features from incomplete scans.
Schemata Embesting
* Mara embedding Later:: Festures
Labert Feotures
(b)
Figure 5: The t-SNE visualization (27 of the object latent space of the Schemata 22 method (b). shows clearly that our prototypical knowledge embedding is learned from a more distinguishable latent space than the Schemata, which extracts the knowledge from perceptual latent embedding.
(a) and our
Analysis on the Meta-embedding. We illustrate the t-SNE visualization of the output object embedding of the graph encoder in the meta-learning stage and the object meta-embedding in (b) of Figure [ There are two points worth noticing. Firstly, the learned meta-embedding can locate at a similar area of its related class in metric space, which validates its potential be prototypes of semantic categories. Secondly, the sub-spaces represented by the meta- eetta-ebeedding are distinguishable that help to enlarge the distance between categories after injecting knowledge we visualize the latent spaces in the predicate classification
into perception inference. In Figure 6 task with and without the meta-embedding. illustrates that the predicate representations are more separable after knowledge intervention.

---

(1)
(b)
Figure 6: The t-SNE visualization the latent spaces in the predicates classification task. The figures show the geometrical features from PointNet in (a), the contextualized features by the graph neural network in (b), and the latent space after knowledge intervention in (c).
Table 3: Quantitative results of our proposed method in SGCIs and PredCls tasks on different splits of the 3DSSG dataset. We report our method ablated on the meta-embedding (ME) intervention. The percentage of the used training set is noted in parentheses. The standard deviation is reported after mean recall value.
SGCIs
PredCIs
Model
mR @-20
mR@50
mR @ 100
@R@20
mR 500
@ 100
Ours w/o ME (1005)
0.2194:004
0.24220004
0.245
0.353
0.410
0.4152m010
Ours w/ MR (0%)
0.365
0.430=mm.4
Ours w/ ME (258)
0.185
0.219
0.0005
0.505
00:
0.559
Ours w/ ME (50%)
0.227++0.000
0.249 +-0.001
-.002
0.557 +
0.600
0.603
Ours w/ ME
0.244++0011
0.286-20.008
0.288
0.566+a001
0.635 +0.001
0.638=
Additionally, understand the importance of the meta-embedding intervention, train the scene graph classification model with more minor splits. The results are reported in Table For the SGCIs task, model can achieve competitive results with the backbone model with only 50% of the training set with the meta-embedding. For the PredCIs task, our model can even give reasonable predicate classification results without any perception information with the object labels and corresponding meta-embeddingg This attempt could also he extended for scene generation applications. For example, our model can output plausible scene organization with only graph inference based on the learned meta-embedding given object numbers and categories. Another interesting observation is that our design preserves the possibility of training the meta-embedding from other data sources, which could be larger than the training set used in the scene graph prediction model. It is also the reason that we maintain the independence of the two modules instead of training the whole model an end-to-end manner.
Improvereett of Preddicttes Ri8
Exalustien Set Predicaee Distrbuttin
836
032
5.30
0.08
sente sp memeerte as covel
build in- stardingg ding in leaning agains carrin
smaller do
hidver bishind
lower tiran
altac attached aed to
Close by
standing oodiingg on
bigzer than
right
lying on
connected to front
supported by-
doove Iy
keft
front
behind attcched to
lying on supported by
lower an
higher than
on Banning on
buw thnn Ihann
same gymmetery 25
liing in
bolonging to
standing in leaning sgainst
tower part of hamping
hangint
bellonging
standing
same
smaller
build
(a)
Figure 7: The relative improvement of predicate recalls on R@5 with the intervention of the meta- embedding(a) and the predicates statistical distributions (b). The meta-embedding intervention enhances the predictions especially on rarely sampled predicate categories.

---

Furthermore, we illustrate the improvements of predicate predictions on @@5 with the meta- embedding intervention and the distribution of each predicate category in the evaluation set in Figure 7 We can see that the meta-embedding intervention can notably improve the recall of the less frequently sampled predicates. However, our model could damage the performance of predicting categories with considerable frequency. Since the fused meta-embedding could introduce extra infor- mation of the most related categories, could be noises for the most frequent classes but guidance for rare-sampled ones. Fortunately, the recalls of the frequent predicates are qualified enough because of their large proportion in the training set. We provide more quantitative results of the predicate predictions in the supplementary material.
5 Conclusion, Discussion, and Future Work
We proposed a novel knowledge-inspired 3D scene graph prediction method in this paper. We decomposed our final destination into two sub-tasks. For the first task of knowledge embedding learning, we designed a graph auto-encoder to extract dlass-dependent prototypical knowledge solely based semantic category information without adverse effect of perceptual confusion. Secondly, we incorporated the learned knowledge embedding in the scene graph prediction task for an accurate object and predicate classification performance. Comprehensive experiments verified that our method could extract and inject the prior knowledge to achieve more accurate relationship prediction than the existing approaches pertaining to this subject.
However, our method still has several limitations yet to overcome. First, our multi-scale point set encoder could fail to predict confident initial class guesses when experiencing severe incomplete- ness. The inaccurate object identification impairs the relation predictions. Second, our knowledge intervention could not help the predictions for the most frequent semantic categories since the chosen knowledge might be noises for the inference of those classes. Moreover, our prior learned knowledge highly dependent on the variation and the completeness of the training set. Considering the personal information security, the data used in our paper contains only uncolored scene scatter scans without any personally identifiable information nor any sensitive content. However, for the potential negative societal impacts, in real-world robot navigation tasks, inaccurate prediction of object entities and their relationships could lead agents to the wrong target and raise human safety concerns. To avoid this potential problem, we suggest proposing a security protocol in case of dysfunction of our scene understanding algorithm in real-world robotic applications.
Our near-term research efforts are geared towards immediate improvement with better solutions. Other aspects of 3D scene understanding and applications could be researched in the long term. For example, we could formulate the dynamic scene graph generation task to capture spatio-temporal cues from other data modalities (c.g., vidcos). Additionally, it would be of interest researchers to explore the registration of virtual agents in dynamic scene graphs and enhance interactions with the real world in augmented reality (AR) applications.
Acknowledgements
This research is supported in part by National Key R&D Program of China NN0.2018YFBI700600 National Natural Science Foundation of China (No.61672077 and 61532002), Beijing Natural Science Lanndation-Haddian Primitive Innovation Join Fund (L182016), and National Science Foundation of USA: 18SS6666666 and IIS-1715985.
References
[1] P. Achlioptas, O. Diamanti, I. Mitliagkas, and L. J. Guibas. Learning representations and generative models for 3d point clouds. In JCML, volume 80 of PMLR, pages 40-49, 2018.
[2] I. Armeni, Z. He, A. R. Zamir, J. Gwak, J. Malik, M. Fischer, and S. Savarese. 3d scene graph:
A structure for unified semantics, 3d space, and camera. In ICCV, pages 5663-5672, 2019.
[3] L. J. Ba, J. R. Kiros, and G. E. Hinton. Layer normalization. CoRR, abs//100.00445 2016.
[4] Chen, W. Yu, R. Chen, and I.. Lin. Knowledge-embedder routing network for scene graph generation. In CVPR, pages 6163-6171, 2019.
10

---

[5] K. Cho, B. van Merrienboer, Ç. Gülgehre, D. Bahdanau, F. Bougares, H. Schwenk, and Bengio. Learning phrase representations using RNN encoder-decoder for statistical machine translation. In EMNLP, pages 1724-1734, 2014.
[6] Gilmer, S. S. Schoenholz. Riley, O. Vinyals, and G. Dahl. Neural message passing for quantum chemistry. In ICML, volume 70 of PMLR, pages 1263-1277 1272, 2017.
[7] C. Gómez, M. Fehr, A. Millane, A. C. Hernández, Hybrid topological and 3d dense mapping through autonomous exploration for large indoor environments. In ICRA, pages 9673-9679, 2020.
Nieto, R. Barber, and R. Siegwart.
[8] J. Gu, H. Zhao, Z. Lin, S. Li, J. Cai, and M. Ling. Scene graph generation with external knowledge and image reconstruction. In CVPR, pages 1969-1978, 2019.
[9] R. Herzig, M. Raboh, G. Chechik, J. Berant, and A. Globerson. Mapping images to scene graphs with permutation-invarian structured prediction. In NeurIPS, pages 7211-7221, 2018.
[10] Z. Huang, Y. Yu, J. Xu, Ni, and X. Le. Pf-net: Point fractal network for 3d point cloud completion. In CVPR, pages 7659-7667 2020.
[11] Johnson, A. Gupta, and Fei-Fei. Image generation from scene graphs. In CVPR, pages 1219-1228, 2018.
[12] M. Keller, Lefloch, M. Lambers, Izadi, T. Weyrich, and A. Kolb, Real-time 3d reconstruc- tion in dynamic scenes using point-based fusion. In 3DV, pages 1 18, 2013.
[13] U. Kim, Park, T. Song, and Kim. 3-d scene graph: A sparse and semantic representation of physical environments for intelligent agents. JEEE Transactions on Cybernetics, 50(12):4921- 4933, 2020.
[14] T. Lin, P. Goyal, R. B. Girshick, K. He, and P. Dollár. Focal loss for dense object detection. In ICCV, pages 2999-3007, 2017.
[15] C. Lu, R. Krishna, M. S. Bernstein, and F. Li. Visual relationship detection with language priors. In ECCV, volume 9905 of LNCS, pages 852-869, 2016.
[16] K. Murphy, A. Torralba, and W. T. Freeman. Using the forest to see the trees: A graphical model relating features, objects, and scenes. In NIPS, pages 1499-1506, 2003.
[17] M. Nießner, M. Zollhöfer, Izadi, and M. Stamminger. Real-time 3d reconstruction at scale using voxel hashing. ACM Transactions Graphics, 32(6): 169:1-169:11 2013.
[18] C. R. Qi, H. Su, K. Mo, and J. Guibas. Pointnet: Deep learning on point sets for 3d classification and segmentation. In CVPR, pages 77-85, 2017.
[19] S. Ren, K. He, K. He, R. B. Girshick, and J. Sun. Faster R-CNN: Towards real-time object detection with region proposal networks. IEEE Transactions Pattern Analysis and Machine Intelligence, 39(6): 137-114441 149, 2017.
[20] A. Rosinol, A. Gupta, M. Abate, J. Shi, and L. Carlone. 3d dynamic scene graphs: Actionable spatial perception with places, objects, and humans. CoRR, abs/2002.06289,, 2020.
[21] D. E. Rumelhart, G. F. Hinton, and R. Williams. Learning representations by back prok-prppgggiigggg errors. Nature, 323(6088):333-56, 1986.
[22] S. Sharifzadeh, S. M. Baharlou, and V. Tresp. Classification by attention: Scene graph classifi- cation with prior knowledge. CoRR, abs/201 1.10084, 2020.
[23] Snell, K. Swersky, and R. S. Zemel. Prototypical networks for few-shot learning. In NIPS, pages 4077-4087,2011
[24] S. Song, F. Yu, Zeng, X. Chang, M. Savva, and T. A. Funkhouser. Semantic scene completion from single depth image. In CVPR, pages 190-198, 2017.

---

[25] T. Tahara, T. Seno, G. Narita, and T. Ishikawa. Retargetable AR: Context-aware augmented reality indoor scenes based on 3d scene graph. In ISMAR, pages 249-255, 2020.
[26] K. Tang, H. Zhang, Wu, W. Luo, and W. Liu. Learning to compose dynamic tree structures for visual contexts. In CVPR, pages 6619-6628, 2019.
[27] L. Van der Maaten and G. Hinton. Visualizing data using t-sne. JMLR, 9(11), 2008.
[28] Wald, A. Avetisyan, N. Navab, F. Tombari, and M. Nießner. RIO: 3d object instance re-localization in changing indoor environments. In ICCV, pages 7657-7666, 2019.
[29] J. Wald, H. Dhamo, N. Navab, and F. Tombari. Learning 3d semantic scene graphs from 3d indoor reconstructions. CVPR, pages 19660-969,, 2020.
[30] Whelan, S. Leutenegger, R. Salas-Moreno, B. Glocker, and A. J. Davison. Elasticfusion: Dense SLAM without pose graph. In Robotics: Science and Systems XI, 2015.
[31] S. Woo, D. Kim, D. Cho, and I. S. Kweon. Linknet: Relational embedding for scene graph. In NeurlPS, pages 558-568, 2018.
[32] D. Xu, Y. Zhu, C. B. Choy, and L. Fei-Fei. Scene graph generation by iterative message passing. In CVPR. pages 3097-3106, 2017.
[33] Yang, Lu, S. Lee, D. Batra, and Parikh, Graph R-CNN for scene graph generation. In ECCV, volume 11205 of LNCS, pages 690-706, 2018.
[34] L. Yu, X. Li, C. Fu, D. Cohen-Or, and P. Heng. Pu-net: Point cloud upsampling network. In CVPR, pages 2790-2799, 2018.
[35] A. Zareian, S. Karaman, and S. Chang. Bridging knowledge graphs to generate scene graphs. In ECCV, volume 12368 of LNCS, pages 606-623, 2020.
[36] A. Zareian, Z. Wang. H. You, and S. Chang. Learning visual commonsense for robust scene graph generation. In ECCV, volume 12368 of LNCS, pages 642-657, 2020.
[37] R. Zellers, M. Yatskar, S. Thomson, and Y. Choi. Neural motifs: Scene graph parsing with global context. In CVPR, pages 5831-5840, 2018.
Checklist
For all authors...
(a) Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope? [Yes] The central claims in the abstract and introduction are detailly discussed in each part of Section 3 which are the primary contributions of our work.
(b) Did you describe the limitations of your work? [Yes] The limitations detailly discussed in Section 4 after the analysis of quantitative results. The summary of the limitations is reported in Section 5
(c) Did you discuss any potential negative societal impacts of your work? [Yes] The potential negative societal impacts is discussed in Section
(d) Have you read the ethics review guidelines and ensured that your paper conforms to them? [Yes] We have already read the ethics review guidelines and ensured that our paper conforms them.
2. If you are including theoretical results...
(a) Did you state the full set of assumptions all theoretical results? [Yes] All assumptions of the theoretical results are listed and discussed in the problem formulation part of Section 3
(b) Did you include complete proofs of all theoretical results? [Yes] The theoretical results mentioned in this paper are obtained by basic probability. They are conventional and easy to understand.
12

---

3. If you ran experiments...
(a) Did you include the code, data, and instructions needed to reproduce the main ex- perimental results (either in the supplemental material or URL)? [Yes] The code and instructions are included in the supplemental material. Our pipeline are detailly reported in Section
3 The only dataset used in this work, 3DSSG dataset, is public and
well cited in our paper.
(b) Did you specify all the training details (e.g., data splits. hyperparameters, how they were chosen)? [Yes] We report the training details and hyperparameters at the begining of Section 4
(c) Did you report error bars (e.g., with respect the random seed after running experi- ments multiple times)? [Yes] We report the error bars in the form of text in the tables
of Section 4
(d) Did you include the total amount of compute and the type of resources used (e.g., type of GPUs, internal cluster, cloud provider)? [Yes] We report our compute hardware in the implementation detail part of Section 4
4. If you using existing assets (e.g., code, data, models) or curating/releasing new assets...
(a) If your work uses existing assets, did you cite the creators? [Yes] We cite all used code, data and models in this paper.
(b) Did you mention the license of the assets? [Yes] We report the license of the used code repository the implementation detials of Section 4 in the footnote.
(c) Did you include any new assets either in the supplemental material or as a URL? [No] We only show more experimental examples of the dataset we used in the paper. There no new assets to cite in the supplemental material or as URL.
(d) Did you discuss whether and how consent was obtained from people whose data you're using/curating? [Yes] We report the way have the permission of the dataset in Section 4 in the footnote.
(e) Did you discuss whether the data you are using/curating contains personally identifiable information or offensive content? [Yes] We report that the used data does not involve any personally identifiable information nor offensive content in Section 5
5. If you used crowdsourcing or conducted research with human subjects...
(a) Did you include the full text of instructions given to participants and screenshots, applicable? [N/A] Our work do not involve any crowdsoureing conducted research with human subjects.
(b) Did you describe any potential participant risks, with links to Institutional Review Board (IRB) approvals, if applicable? [N/A] Our work do not involve any crowdsourcing or conducted research with human subjects.
(c) Did you include the estimated hourly wage paid to participants and the total amount spent on participant compensation? [N/A] Our work do not involve any crowdsourcing or conducted research with human subjects.
13


Reviews:
[{"review_id": "w5Bnv3d4Gn", "reviewer": "Reviewer_8qYe", "summary": "The paper provides a novel two-stage pipeline for 3D scene graph prediction tasks on point cloud data.\nNovel relation and object embedding methods are proposed.\nExperiment results show that the proposed pipeline gives higher performance compared with related methods. ", "questions": "", "limitations": "", "rating": 6, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "Originality:\n\nThe two-state pipeline is of great novelty.\nThe embedding method proposed is effective.\n\nClarity:\n\nThe overall idea is expressed clearly.\nBut details on network structures are missing or fuzzy.\n\nIn meta-embedding learning auto-encoder, it is unclear to me how the network structure is obtained.\nJudging by Fig 1 and Fig 2, the input contains only one-hot embeddings.\nBut GNN requires a network structure to operate on.\nI conjecture that the graphs come from all scene graph in the dataset.\nThen in that case, the input should further include graph structure instead of only labels.\n\nJudging by Fig 3, there is a recurrent structure in the scene graph prediction model.\nIt is not clear to me what will happen in the first iteration before any meta-embedding can be obtained.\nWithout meta-embedding, updates described by equation 8 cannot be executed.\nOther questions are how many iterations are required and will the classification accuracy improve over iterations.\n\nQuality:\n\nThe work is complete and the model is sound.\nEvaluation is comprehensive and convincing.\nThe limitation is discussed in detail.\n\nSignificance:\n\n3D scene understanding is an important research direction with great potential.\nThe performance of the model proposed in this paper surpasses a large body of previous works.\nThe embedding method is inspiring and could potentially be used in other similar tasks.", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "Yes", "review_text": "", "extra_scores": {"needs_ethics_review": "No", "ethics_review_area": "", "time_spent_reviewing": "5"}, "extra_text": {}}, {"review_id": "t8TevxATiIe", "reviewer": "Reviewer_rms6", "summary": "This paper proposes a knowledge-inspired method for 3D scene graph prediction on point cloud. The key is to decompose the problem into two sub-tasks. First, a graph auto-encoder called meta-embedding is introduced to learn class-dependent prototypical representations only from the class labels to avoid visual confusions. Then a scene graph prediction model selects related meta-embedding from the first task as prior knowledge to predict credible relation triplets of 3D scene graph. Benefited from the learned commonsense knowledge, the scene graph prediction model achieves good performance on 3DSSG dataset.", "questions": "", "limitations": "", "rating": 7, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "Strong points: \n1. The idea to learn knowledge solely based on class labels and their regular graphic structures is novel, which avoids the visual confusion introduced by the perceptual errors in previous works. It may bring some insights to other knowledge-inspired tasks. \n2. The experiments and visualizations are thoroughly provided, while the weaknesses are also discussed. \n3. The paper is well-written and easy to follow.\n\nWeak points:\n1. Line 244-249 argues that the limitation of the model in object identification can be possibly solved by using a stronger point set encoder than PointNet. However, Table 1 shows that the method w/o ME can also get a good result (using PointNet), where the problem is that the performance gain in SGCLs task is not enough to fully prove the effectiveness of ME. This is expected to be discussed in the rebuttal.\n2. In Line 268, the performance on frequent categories is damaged by the model. However, the reason to cause this problem is not fully discussed.\n3. Line 167 states that the five most confident embeddings are selected, will the number of the embeddings influence the result? Since the selection of prior knowledge is also the key in to utilize the knowledge, this detail is expected to be discussed.\n4. Figure 6 of Schemata [1] clearly confirms that object representations will get into more separable clusters after injecting their prior knowledge. Although Line 255 argues similar effects, the same visualization comparison is expected to be provided.", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "The societal impacts are fully discussed. \nFor the limitations of the method, the reviewer hope to know the robustness of the proposed method to the perturbation such as rotation, translation and scaling. The robustness to the perturbation on 3D point clouds is very important in real-world applications. Since the proposed method learn the knowledge solely from labels, it may be robust to the real-world perturbations, which is expected to be discussed in the rebuttal.", "review_text": "", "extra_scores": {"needs_ethics_review": "No", "ethics_review_area": "", "time_spent_reviewing": "6"}, "extra_text": {}}, {"review_id": "Sii0wMZ-mTC", "reviewer": "Reviewer_r35L", "summary": "The paper proposes a scene graph prediction method for 3D point clouds. The paper uses the graph network to learn class label based embeddings. It does not use appearance information from the point clouds to avoid perceptual inconsistencies. The class label dependent prototypical embeddings are learned using a graph auto-encoder with directed edges and a message passing mechanism tuned for such edges. The scene graph prediction works by extracting embeddings from subject and object points and object pair relationship in form of an edge embedding. The embeddings (geometric, meta embeddings) are fused with an MLP. The network is trained with focal loss. The proposed method is evaluated on 3DSSS dataset and achieves state of the art results on various metrics. ", "questions": "", "limitations": "", "rating": 7, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "- The main novelty of the method lies in utilizing meta embeddings from class labels and fusing them with geometric embeddings to obtain the scene graphs. The approach, in my opinion, builds upon geometry based methods on point clouds [A], where the appearance information is not necessary to learn robust descriptors. However, overall adding class label based embeddings and the fusion mechanism seems novel with reference to the existing literature. \n\n- The paper is well written and motivated. The contributions are highlighted clearly. However, main premise of the paper is to avoid appearance noise. In order to validate this hypothesis, authors may include such features and demonstrate the change in performance.\n\n- I was able to reproduce the resutls (with 2% variance from the reported results on various metrics) with the code provided by the authors on an NVIDIA RTX 8000, 256 GB RAM.\n\n- How do you chose the five best meta embeddings for fusion. Is it based on the attention maps ? (Line 167). What happens if the number of embeddings is more/less ?\n\n- What is the motivation behind using Focal Loss ? The main advantage of using focal loss is during class imbalanc. Is it due to sparse mapping between the object-subject pairs here ?\n\n- How does the technique perform with noisy data/points say sampled at a lower resolution ?\n\n- There are a few typos\n(i) Fig 1 \"Labal\" -> Label\n(ii) Line 169, \"sapces\" -> spaces\n\n[A]: Guo, Yulan, et al. \"Deep learning for 3d point clouds: A survey.\" IEEE transactions on pattern analysis and machine intelligence (2020).", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "The authors have described the limitations adequately in Section 5. However, mentioning the constraints on the subject-object pairs and type of relationshio would help readers to extend the work in other domains. ", "review_text": "", "extra_scores": {"needs_ethics_review": "No", "ethics_review_area": "", "time_spent_reviewing": "12"}, "extra_text": {}}, {"review_id": "R5Rkh6D35HC", "reviewer": "Reviewer_YwYy", "summary": "In this paper, the authors study the 3D scene graph prediction task. The major contribution is the proposed “object/predicate meta-embedding,” which learns class-specific embedding to assist scene graph prediction. Ablation study verifies the effectiveness of the proposed embedding learning method.", "questions": "", "limitations": "", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "Strengths: \n1. The additional object and predicate embedding improves the scene graph prediction performance.\n\n2. The analyses and feature visualization show that the meta-embedding does learn class-specific representation as desired.\n\nWeaknesses: \n1. In the claimed contribution (Line 53), it is unclear what does the first point (“we introduce a novel ... method that directly functions on 3D scene point clouds”) refers to. The framework seems to have the same input-output as the previous 3D scene graph prediction studies [29]. \n\n2. The “Knowledge-inspired Scene Graph Prediction Model” section and Figure 3 are confusing. The two-round approach (the loop from object prediction to initial feature) is barely introduced in the text, and thus makes the inference pipeline difficult to understand.\n\n    A question is that whether more than two rounds with more accurate object/predicate prediction further improves the performance.\n\n3. In addition to the reported metrics in image scene graph prediction, it might be helpful to also benchmark the studies following previous 3D scene graph studies’ metrics [29].\n\n4. a. The term “commonsense” could be misleading. The learned “object/predicate meta-embedding” is more like external class-specific memories, instead of “commonsense” or “knowledge.”\n\n5. It would be helpful to conduct additional ablation studies and analyses on the meta-embedding design. For example, what if keeping the meta-embedding but learn it E2E instead of using the auto-encoder-like structure.\n\n6. Despite the performance improvement, the extra embedding also increases the training and inference cost, as it requires two inference rounds and storing additional memory-related parameters.\n\n7. The proposed method seems generalizable to image scene graph prediction. It would be helpful to also benchmark the study on image scene graph prediction, or show the unique design for the 3D task.\n", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "The limitations and societal impacts are discussed in the conclusion section.", "review_text": "", "extra_scores": {"needs_ethics_review": "No", "ethics_review_area": "", "time_spent_reviewing": "3"}, "extra_text": {}}]

Author rebuttal:

Meta-review: This paper proposes a method for 3D scene graph prediction from point clouds in which  a graph auto-encoder model learns prototypical representations for object categories using scene graph annotations, that are used  as prior knowledge during scene graph inference from a point cloud input.  Reviewers acknowledge the novelty of the autoencoding model proposed for learning categorical priors. 
Reviewers point out that the paper is not clear regarding the input of the autoencoding graph model. They further point out that vague words like ``common sense” are not well defined and are used arbitrarily in the paper. Authors are strongly encouraged to clarify the paper writing (and corresponding figures), following reviewers’ comments.

Decision: Accept (Poster)