Knowledge-inspired 3D Scene Graph Prediction in Point Cloud
Year: 2022

Abstract:
Prior knowledge integration helps identify semantic entities and their relationships in a graphical representation, however, its meaningful abstraction and intervention remain elusive. This paper advocates a knowledge-inspired 3D scene graph prediction method solely based on point clouds. At the mathematical modeling level, we formulate the task as two sub-problems: knowledge learning and scene graph prediction with learned prior knowledge. Unlike conventional methods that learn knowledge embedding and regular patterns from encoded visual information, we propose to suppress the misunderstandings caused by appearance similarities and other perceptual confusion. At the network design level, we devise a graph autoencoder to automatically extract class-dependent representations and topological patterns from the one-hot class labels and their intrinsic graphical structures, so that the prior knowledge can avoid perceptual errors and noises. We further devise a scene graph prediction model to predict credible relationship triplets by incorporating the related prototype knowledge with perceptual information. Comprehensive experiments confirm that, our method can successfully learn representative knowledge embedding, and the obtained prior knowledge can effectively enhance the accuracy of relationship predictions. Our thorough evaluations indicate the new method can achieve the state-of-the-art performance compared with other scene graph prediction methods.

heading: 1 Introduction and Motivation
text: Scene graph has been studied as a viable means towards better interpretation of scene context, which encodes the semantic elements and their complex relationships of a scene [16, 32, 33, 11, 31, 9]. As an enriched scene structure representation, scene graph parsing from 3D data has recently interested researchers in 3D vision for the tasks of scene understanding [2], VR/AR scene interactions [25], and robot navigation [7]. However, the scanned 3D data has a intrinsic nature of incompleteness [1, 24], geometrical similarity across categories, and other visual challenges. These perception errors result in extra difficulties in object and relationship recognition purely based on visual inputs.
Fortunately, in cognitive psychology, besides visual perception, human beings also associate objects and concepts with certain abstract symbolic class-dependent representations and regular combinations
∗Corresponding authors.
35th Conference on Neural Information Processing Systems (NeurIPS 2021).
of scene context, known as knowledge. However, utilizing prior knowledge still remains challenging in scene graph prediction. Prior knowledge-incorporated methods have several limitations. Static knowledge (e.g., heuristic co-occurrence statistics [37], mined triplet facts [8], or knowledge graph [35]) could be incomplete and inaccurate depending on the quality of knowledge sources and the domain knowledge covered. Thus, several methods tend to use learning-based knowledge to capture the graphical structure and regular combinations in the exact domain. However, the knowledge learned from visual information also inherits perceptual confusion into the knowledge embedding and network parameters [22, 36].
To ameliorate, this paper advocates a novel 3D scene graph prediction method with learned prior knowledge. The knowledge can be encoded in various forms (e.g., vectors, graphs, and natural language). In our work, we choose class-dependent prototypical memories as the carrier of knowledge to tackle the key challenges mentioned above. We re-formulate the target problem into two sub-tasks: knowledge learning and knowledge-inspired scene graph prediction. Unlike previous works, we learn the knowledge only depending on semantic categories and regular structural patterns without any visual confusion. The knowledge learning model is a graph auto-encoder that takes the one-hot class labels as inputs with scene graph structure annotations. Since we do not involve any geometric and visual features, the latent embedding indeed encodes only category-related regular structure patterns. To correctly record this prior knowledge, we enforce a group of prototypes [23], namely meta-embedding, to approach the latent areas of the corresponding category in the metric space. As we illustrated in Figure 1, the meta-embedding is capable of representing each semantic class as prototypical knowledge. Once trained, the scene graph prediction model unites the geometric features and the corresponding knowledge meta-embedding to predict each element in a triplet with the <subject, predicate, object> structure. Extensive experiments confirm that the effectiveness of knowledge extraction and the injection of the meta-embedding can improve the scene graph prediction accuracy.
The primary contributions of this paper could be summarized as follows: (1) We introduce a novel knowledge-inspired scene graph prediction method that directly functions on 3D scene point clouds. To achieve our goal, we re-formulate the problem into two sub-tasks: knowledge learning and scene graph prediction with knowledge intervention; (2) We propose a graph auto-encoder, called meta-embedding, to learn a group of class-dependent prototypical representations. The learned class prototypes can avoid perceptual errors and effectively encode prior structural knowledge in a more distinguishable latent space; and (3) We design a scene graph prediction model benefiting from the prior learned knowledge in order to predict credible relation triplets. With our learned knowledge, our scene graph prediction model can achieve state-of-the-art performance on the 3D semantic scene graph (3DSSG) dataset [29] compared to currently available methods.

heading: 2 Related Work
text: Knowledge-based Scene Graph Classification. Prior knowledge has been proven as an effective source of information to enhance object and relation recognition. Early methods use statistical co-occurrence as extra knowledge for scene graph inference. Zellers et al. observed that the predicate distribution is strongly dependent on the head and tail categories. It introduces a pre-computed bias into the final prediction [37]. The biased results naturally improve the most frequent combinations. To improve the classification performance on the rarely sampled ones, knowledge-embedded routing network (KERN) [4] implicitly fuses co-occurrence probabilities as weights in the message pass step to neutralize the long-tail phenomenon. Nevertheless, this integrated knowledge can hardly be used and studied in isolation since it is hard-coded as intrinsic parameters. Besides, some methods bring external knowledge bases into scene graph inference. Gu et al. extracted knowledge triplets from the ConceptNet knowledge bases [8]. Graph bridging network (GB-Net) adopts auxiliary edges as bridges that enable and facilitate message passing between knowledge graphs and scene graph [35]. This hand-crafted external information is highly dependent on the organization of knowledge and the quality of sources. Furthermore, several methods try to incorporate learning-based commonsense into scene graph inference. Zareian et al. proposed a local-global graph transformer to complete a masked scene graph by embedding the regular patterns into the network parameters [36]. More recently, Sharifzadeh et al. encoded inductive biases into the class-level prototypical representations learned from all previous perceptual outputs [22]. Inspired by [22], our method also adopts prototypical representations as prior knowledge. Nevertheless, our method focuses on class-dependent and graph structure information, which aims to combat the confusion resulting from perceptual errors.
3D Scene Graph Prediction. 3D scene graph has only recently been studied in the 3D vision community. There are mainly two approaches for 3D scene graph construction. The first group of works benefits from image-based scene graph techniques. Armeni et al. firstly introduced a 3D scene graph construction framework with 3D meshes and registered RGB panoramic images [2]. It adopts multi-view consistency to enforce the accuracy of object identification by 2D-3D projection. Furthermore, succeeding methods [13, 20] execute image-based scene graph prediction within dynamic simultaneously localization and mapping (SLAM) [12, 17, 30]. Kim et al. proposed a framework that merges the local graphs into a global one by fusing the same nodes in different images [13]. Spatial PerceptIon eNgine (SPIN) distinguishes human and object nodes by dynamic masking [20]. The second group of approaches directly process 3D data. Recently, Wald et al. proposed an end-to-end network, scene graph prediction network (SPGN) [29], which takes scene scans as input. Along with the network, they also published the 3DSSG dataset of reconstructed scene scans. Similar to this work, we investigate a novel model to extend the scene graph prediction methods using 3D scene data solely based on point clouds.

heading: 3 New Method
text: Problem Formulation. We define a scene graph G as a directed graph with sets of nodes and edges, G = {V, E}. The node set V includes all object instances of a scene, and the edge set E consists of all possible connections of nodes as predicates in a relation triplet <subject, predicate, object>. The edges are directed by connecting neighbor nodes from subject to object. Unlike previous knowledgeembedded approaches [37, 4, 36], we formulate the knowledge-inspired scene graph prediction task as a multi-task problem: we intend to learn knowledge and scene graphs from 3D data simultaneously. Mathematically, given a 3D scene data D with scene graph annotations, the training process is aimed to predict the probability of P (G,K|D), where K is the prior knowledge. Furthermore, we can decompose the joint probability into two individual components:
P (G,K|D) = P (K|D)P (G|D,K). (1) This decomposition breaks the problem into two sub-tasks. The first sub-task, the predication of P (K|D), extracts knowledge from existing 3D scene configurations. In this paper, the learned knowledge is in the form of prototypical embedding of each object and predicate category, named meta-embedding K = {emeta}. The set of meta-embedding is expected to represent each semantic class and also encode the corresponding class’s regular patterns in scene graphs. Towards this goal, we design a graph auto-encoder network, which only takes the one-hot labels as input without any perceptual information. The graphical encoder also can extract the topological interactions of each class. We close the distance of latent outputs and the related meta-embedding in metric space so that the meta-embedding records the prior knowledge of the training data.
The second sub-task, P (G|D,K), incorporates the meta-embedding and perceptual features to predict the possible scene graph elements. It is also the expected mathematical model in the evaluation stage with unseen 3D scene data.
Meta-embedding and Learning Model. This model is designed to learn class-dependent prototypical knowledge embedding with a graph auto-encoder. The overview of the meta-embedding learning model is illustrated in Figure 2. Since we consider the class-related patterns instead of perceptual information, the auto-encoder only takes the one-hot class labels as input. The first two multi-layer perceptron (MLP) blocks lift one-hot vectors into latent space with the same dimension size. Then, we assume that each object pair can have a relation (including none relation) and fully connect them as a graph where predicates are represented as edges. We design a graph neural network to further encode the graphical patterns of each node and edge in scene graphs by message passing [6]. At the network design level, we adapt the message passing mechanism to the directed graph structure. To calculate the latent representations {hn,lv } for all nodes v ∈ V and {h e,l uv} for all directed edges (u, v) ∈ E in every layer l, the message passing of nodes and edges can be formulated as:
hn,l+1v = GRU n,l(hn,lv ,m n,l(v)), (2)
he,l+1uv = GRU e,l(he,luv,m e,l(u, v)), (3)
where mn,l and me,l are the parameterized neural networks that calculate messages in layer l. We adopt the gated recurrent unit (GRU) [5] to update the hidden state for reserving initial information. It is worth noting that a node could be a subject or object in a relation triplet. We consider both conditions in message computation. For node v and one of its neighbors u, the message as subject ms,l(v) and as object mo,l(v) are computed by:
ms,l(v) = LN(φp(he,lvu) + φo(h e,l u )), (4)
mo,l(v) = LN(φp(he,luv) + φs(h e,l u )), (5)
where φs, φp, and φo are three non-linear transformations for subject, predicate, and object respectively. To stabilize the numerical calculation in recurrent mechanism, we adopt the layer normalization (LN ) [3] after feature fusion. The final message of node v is the average of ms,lv and m o,l v :
mn,l(v) = 1 |Rsv|+ |Rov| ( ∑ u∈Rsv (ms,l(u)) + ∑ u∈Rov (mo,l(u))), (6)
where | · | denotes cardinality andRsv andRov are the set of connections of node v as subject and as object respectively. The calculation of edge (u, v) message is:
me,l(u, v) = LN(φs(hn,lu ) + φo(h n,l v )). (7)
We use the sum pooling operation to the stacked hidden states as the output model of the graph encoder. Since we know the class labels of the latent output embedding of the graph encoder, we update the corresponding meta-embedding vectors by reducing their distances (e.g., Euclidean distance) in metric space. Given the latent embedding of nodes and those of edges, we select the corresponding class vectors from the object meta-embedding and the predicate meta-embedding. Then we reduce the distance between those embeddings as a loss function. As the values of
meta-embedding are initialized as network parameters, they can eventually be updated with backpropagation [21]. The meta-embedding is updated only when seeing the related category, not depending on the number of training samples. This property makes the meta-embeddings qualified to be prototypical representations for even less frequent categories. The last two MLPs predict one-hot labels to enforce the latent embedding is class-related and complete the auto-encoder structure.
Knowledge-inspired Scene Graph Prediction Model. This model is aimed to fuse prior metaembedding into the scene graph prediction task. Figure 3 shows the pipeline of the knowledgeinspired scene graph prediction model. Taking a scene point cloud with instance masks as input, we expect to encode geometrical features from both objects and object pairs. Based on the pioneering point set encoder, PointNet [18], we enhance the original structure to capture multi-scale geometric information like several point completion methods [10, 34]. For the object nodes, we directly use our modified multi-scale PointNet (MS PointNet) on the normalized point clouds. We first encode subject and object point sets separately for the object-pair edges and concatenate the latent features with the center coordinates of two point sets. To better encode the contradictory relationships (e.g., left and right, lower than and higher than), we adopt subtraction as an anti-symmetric operation on these two features. After a transformation by a two-layer MLP, we obtain the initial edge embedding from the object pairs.
We adopt the same graph neural network design in the meta-embedding learning stage to capture the global context with perceptual information. In the first iteration, the graph neural network only takes the geometric features of the PointNet as input. The last two MLPs output the node and edge classification possibilities and offer the clues to select meta-embedding vectors of related categories. We select the five most confident meta-embedding for each node and edge and fuse them with the perception features. Based on our experiments, the large number of selected embedding is inevitable to introduce wrong knowledge, while a small meta-embedding set might contain no proper knowledge. In practice, we choose five meta-embedding as a compromise. In the second iteration, since the perceptual embedding and the knowledge are in different latent spaces, we transform them by two-layer feed-forward networks f and g respectively before directly fusing them. Denoting by {eimeta}i=1...k the selected meta-embedding and x the features encoded by multi-scale PointNet, we fuse the knowledge and perceptual information similar as in [22]:
z = LN(f(x) + k∑ i=1 g(eimeta)), (8)
x̂ = LN(z + ψ(z)). (9) where ψ is a two-layer MLP and x̂ is the knowledge-embedded input of the graph neural network in the second iteration. In practice, we only implement two iterations considering the time consumption since the classification performance can not be significantly improved over more iterations.
Loss Function Design. We use the focal loss for object and predicate classification [14, 29] in both meta-embedding learning and scene graph prediction tasks due to the data distribution imbalance. The focal loss we used is formulated as in [29]:
Lfocal = α(1− p)γ log(p), (10) where p is the logits of a prediction of object or predicate, α is the normalized inverse frequency for object classification and an edge/no-edge weight for the predicates, and the γ is a hyper-parameter.
We follow the parameter settings in [29]. To enforce the meta-embedding close to the class latent vectors in metric space, we adopt the Euclidean distance d between the output of the graph encoder and the corresponding meta-embedding as a loss:
Ldist = ∑ v∈V d(hv, evmeta) + ∑ e∈E d(he, eemeta), (11)
where evmeta and eemeta are the corresponding meta-embedding of the categories of node v and edge e. In the meta-embedding learning task, we formulate the loss function as:
Lmeta = Lobjfocal + L pred focal + Ldist. (12)
Empirically, we use the summation of the three components with the same weight. In the scene graph prediction task, we only use the focal loss for object and predicate classification.
Lsg = λLobjfocal + L pred focal, (13)
where λ is a hyper-parameter to re-weight the components as in [29].

heading: 4 Experiments and Evaluations
text: Dataset and Task Description. We train the meta-learning auto-encoder and the scene graph prediction model on the 3DSSG dataset [29] 1, a 3D scene graph dataset based on 3RScan [28]. The dataset annotates support, proximity, and comparative relationships among daily indoor objects. With the same sub-scene split in [29], there are 3582 scenes in the training set and 548 for evaluation. On average, there are 9 objects per scene of 160 categories with 27 kinds of relationships, including none relation. We evaluate our model and compare it to others in two standard tasks proposed in [32]. (1) The scene graph classification (SGCls) scenario is aimed to predict both object and predicate semantic classes. (2) In the predicate classification (PredCls) task, the model only needs to predict the predicate categories with the ground truth labels of object entities.
Implementation Details. Our model is implemented in PyTorch. We trained our model on an Nvidia RTX 2080Ti GPU in a personal computer platform for 40 epochs with the ADAM optimizer. The initial learning rate is set to 0.0001, and the decay rate is 0.7 for every ten epochs. We trained the metaembedding learning network and the scene graph prediction network separately. We followed the focal loss parameter settings in [29]. In the scene graph prediction model, we used pre-trained multiscale PointNets to encode the point cloud. The multi-scale PointNet is also trained on the 3DSSG dataset. We replaced the faster-RCNN [19] image encoder with the same pre-trained PointNet used in our model to compare with image-based methods [4, 22, 29] 2. We selected the meta-embedding vectors according to the five most confident class predictions in the knowledge fusion stage. In the PredCls task, we assigned the corresponding object meta-embedding as the input of the graph encoder since the ground truth of objects is known. We trained each model three times to calculate the standard deviation. All models are trained on the 3DSSG dataset with the same random seeds and the same split for a fair comparison.
Metrics. Conventionally, we compute the recall at the top-n (R@n) triplets predicted by each model. The triplet is considered correct when the subject, predicate, and object are all valid. As defined in [15], the recall is the fraction of the correct top-n triplets against the ground truth. Additionally, since the long-tail phenomenon in the dataset annotations, we also adopt mean recalls (mR@n) [4, 26] to evaluate the performance on the unevenly sampled relations. The mean recall is computed as the average of the recalls on each predicate class.
Comparison with Related Methods. We introduce several baselines to compare with our proposed method. Co-Occurrence: we propose a relation prediction baseline with only co-occurrence statistics of the training set. This baseline outputs the most frequent predicate given the subject and object categories predicted by the multi-scale PointNet. This simple model draws a scratch line for
1We used the 3DSSG dataset [29] under the permission of its creators and authors. We contacted the author by email (johanna.wald@tum.de).
2We implemented the KERN model based on its released code (https://github.com/yuweihao/KERN) with MIT license. The SGPN and the Sechemata models have no public code for now. We reproduced them based on their papers.
knowledge-based methods on the evaluated dataset. Since this model is deterministic, the standard deviation is zero for both SGCls and PredCls tasks. KERN: the knowledge-embedded routing network is a robust statistical knowledge-based method that incorporates the co-occurrence probability into the scene graph prediction [4]. SGPN: the scene graph prediction network is proposed with the 3DSSG dataset, which is the first scene graph prediction model evaluated on this dataset [29]. Schemata: this baseline is a recently proposed scene graph prediction model with learning-based knowledge. It encodes the class-dependent prototypes from the previous decoded perceptual results [22], which is a similar approach to ours.
Table 1 shows that our model achieves the best relation recall than other approaches and the stateof-the-art performance in both SGCls and PredCls tasks. It confirms the effectiveness of our graph network design and the intervention of the meta-embedding. Additionally, we show the results
evaluated in mean recall in Table 2. Our model can still achieve the best performance. Our model improves the mR@100 of 4.3% and 22.3% in SGCls and PredCls tasks with the meta-embedding compared to our ablated version without learned knowledge. This comparison proves that our meta-embedding can improve the predictions in marginally sampled predicate categories. We also illustrate several examples of predicted scene graphs in Figure 4. We provide more examples in the supplementary material.
Compared with the Schemata method, our model performs better in relationship prediction tasks. The reason behind the difference could be that our meta-embedding is encoded from a more distinguishable latent space than the schemata, as shown in Figure 5. The visualized results confirm that the knowledge learned from the class information is more representative than that from visual information with inevitable perceptual confusion.
However, there is a massive gap in recall values while comparing the SGCls and the PredCls tasks. The performance of relationship prediction drops gravely without the object annotations. It presents the limitation of our model in object identification. This failure is caused by the severe geometric incompleteness of point clouds and the long-tail distribution of the annotations. A potential solution is using a stronger point set encoder than the PointNet to extract robust geometric features from incomplete scans.
Analysis on the Meta-embedding. We illustrate the t-SNE visualization of the output object embedding of the graph encoder in the meta-learning stage and the object meta-embedding in (b) of Figure 5. There are two points worth noticing. Firstly, the learned meta-embedding can locate at a similar area of its related class in metric space, which validates its potential to be prototypes of semantic categories. Secondly, the sub-spaces represented by the meta-embedding are distinguishable that help to enlarge the distance between categories after injecting knowledge into perception inference. In Figure 6, we visualize the latent spaces in the predicate classification task with and without the meta-embedding. It illustrates that the predicate representations are more separable after knowledge intervention.
Additionally, to understand the importance of the meta-embedding intervention, we train the scene graph classification model with more minor splits. The results are reported in Table 3. For the SGCls task, our model can achieve competitive results with the backbone model with only 50% of the training set with the meta-embedding. For the PredCls task, our model can even give reasonable predicate classification results without any perception information with the object labels and corresponding meta-embedding. This attempt could also be extended for scene generation applications. For example, our model can output a plausible scene organization with only graph inference based on the learned meta-embedding given object numbers and categories. Another interesting observation is that our design preserves the possibility of training the meta-embedding from other data sources, which could be larger than the training set used in the scene graph prediction model. It is also the reason that we maintain the independence of the two modules instead of training the whole model in an end-to-end manner.
Furthermore, we illustrate the improvements of predicate predictions on R@5 with the metaembedding intervention and the distribution of each predicate category in the evaluation set in Figure 7. We can see that the meta-embedding intervention can notably improve the recall of the less frequently sampled predicates. However, our model could damage the performance of predicting categories with considerable frequency. Since the fused meta-embedding could introduce extra information of the most related categories, it could be noises for the most frequent classes but guidance for rare-sampled ones. Fortunately, the recalls of the frequent predicates are qualified enough because of their large proportion in the training set. We provide more quantitative results of the predicate predictions in the supplementary material.

heading: 5 Conclusion, Discussion, and Future Work
text: We proposed a novel knowledge-inspired 3D scene graph prediction method in this paper. We decomposed our final destination into two sub-tasks. For the first task of knowledge embedding learning, we designed a graph auto-encoder to extract class-dependent prototypical knowledge solely based on semantic category information without adverse effect of perceptual confusion. Secondly, we incorporated the learned knowledge embedding in the scene graph prediction task for an accurate object and predicate classification performance. Comprehensive experiments verified that our method could extract and inject the prior knowledge to achieve more accurate relationship prediction than the existing approaches pertaining to this subject.
However, our method still has several limitations yet to overcome. First, our multi-scale point set encoder could fail to predict confident initial class guesses when experiencing severe incompleteness. The inaccurate object identification impairs the relation predictions. Second, our knowledge intervention could not help the predictions for the most frequent semantic categories since the chosen knowledge might be noises for the inference of those classes. Moreover, our prior learned knowledge is highly dependent on the variation and the completeness of the training set. Considering the personal information security, the data used in our paper contains only uncolored scene scatter scans without any personally identifiable information nor any sensitive content. However, for the potential negative societal impacts, in real-world robot navigation tasks, inaccurate prediction of object entities and their relationships could lead agents to the wrong target and raise human safety concerns. To avoid this potential problem, we suggest proposing a security protocol in case of dysfunction of our scene understanding algorithm in real-world robotic applications.
Our near-term research efforts are geared towards immediate improvement with better solutions. Other aspects of 3D scene understanding and applications could be researched in the long term. For example, we could formulate the dynamic scene graph generation task to capture spatio-temporal cues from other data modalities (e.g., videos). Additionally, it would be of interest to researchers to explore the registration of virtual agents in dynamic scene graphs and enhance interactions with the real world in augmented reality (AR) applications.

heading: Acknowledgements
text: This research is supported in part by National Key R&D Program of China (No.2018YFB1700603), National Natural Science Foundation of China (No.61672077 and 61532002), Beijing Natural Science Foundation-Haidian Primitive Innovation Join Fund (L182016), and National Science Foundation of USA: IIS-1812606 and IIS-1715985.

Review summaries:
In this paper, the authors study the 3D scene graph prediction task. The major contribution is the proposed “object/predicate meta-embedding,” which learns class-specific embedding to assist scene graph prediction. Ablation study verifies the effectiveness of the proposed embedding learning method.

The paper proposes a scene graph prediction method for 3D point clouds. The paper uses the graph network to learn class label based embeddings. It does not use appearance information from the point clouds to avoid perceptual inconsistencies. The class label dependent prototypical embeddings are learned using a graph auto-encoder with directed edges and a message passing mechanism tuned for such edges. The scene graph prediction works by extracting embeddings from subject and object points and object pair relationship in form of an edge embedding. The embeddings (geometric, meta embeddings) are fused with an MLP. The network is trained with focal loss. The proposed method is evaluated on 3DSSS dataset and achieves state of the art results on various metrics. 

The paper provides a novel two-stage pipeline for 3D scene graph prediction tasks on point cloud data. Novel relation and object embedding methods are proposed. Experiment results show that the proposed pipeline gives higher performance compared with related methods. 

This paper proposes a knowledge-inspired method for 3D scene graph prediction on point cloud. The key is to decompose the problem into two sub-tasks. First, a graph auto-encoder called meta-embedding is introduced to learn class-dependent prototypical representations only from the class labels to avoid visual confusions. Then a scene graph prediction model selects related meta-embedding from the first task as prior knowledge to predict credible relation triplets of 3D scene graph. Benefited from the learned commonsense knowledge, the scene graph prediction model achieves good performance on 3DSSG dataset.