Knowledge-inspired 3D Scene Graph Prediction in Point Cloud

1 Introduction and Motivation . Scene graph has been studied as a viable means towards better interpretation of scene context , which encodes the semantic elements and their complex relationships of a scene [ 16 , 32 , 33 , 11 , 31 , 9 ] . As an enriched scene structure representation , scene graph parsing from 3D data has recently interested researchers in 3D vision for the tasks of scene understanding [ 2 ] , VR/AR scene interactions [ 25 ] , and robot navigation [ 7 ] . However , the scanned 3D data has a intrinsic nature of incompleteness [ 1 , 24 ] , geometrical similarity across categories , and other visual challenges . These perception errors result in extra difficulties in object and relationship recognition purely based on visual inputs . Fortunately , in cognitive psychology , besides visual perception , human beings also associate objects and concepts with certain abstract symbolic class-dependent representations and regular combinations ∗Corresponding authors . 35th Conference on Neural Information Processing Systems ( NeurIPS 2021 ) . of scene context , known as knowledge . However , utilizing prior knowledge still remains challenging in scene graph prediction . Prior knowledge-incorporated methods have several limitations . Static knowledge ( e.g. , heuristic co-occurrence statistics [ 37 ] , mined triplet facts [ 8 ] , or knowledge graph [ 35 ] ) could be incomplete and inaccurate depending on the quality of knowledge sources and the domain knowledge covered . Thus , several methods tend to use learning-based knowledge to capture the graphical structure and regular combinations in the exact domain . However , the knowledge learned from visual information also inherits perceptual confusion into the knowledge embedding and network parameters [ 22 , 36 ] . To ameliorate , this paper advocates a novel 3D scene graph prediction method with learned prior knowledge . The knowledge can be encoded in various forms ( e.g. , vectors , graphs , and natural language ) . In our work , we choose class-dependent prototypical memories as the carrier of knowledge to tackle the key challenges mentioned above . We re-formulate the target problem into two sub-tasks : knowledge learning and knowledge-inspired scene graph prediction . Unlike previous works , we learn the knowledge only depending on semantic categories and regular structural patterns without any visual confusion . The knowledge learning model is a graph auto-encoder that takes the one-hot class labels as inputs with scene graph structure annotations . Since we do not involve any geometric and visual features , the latent embedding indeed encodes only category-related regular structure patterns . To correctly record this prior knowledge , we enforce a group of prototypes [ 23 ] , namely meta-embedding , to approach the latent areas of the corresponding category in the metric space . As we illustrated in Figure 1 , the meta-embedding is capable of representing each semantic class as prototypical knowledge . Once trained , the scene graph prediction model unites the geometric features and the corresponding knowledge meta-embedding to predict each element in a triplet with the < subject , predicate , object > structure . Extensive experiments confirm that the effectiveness of knowledge extraction and the injection of the meta-embedding can improve the scene graph prediction accuracy . The primary contributions of this paper could be summarized as follows : ( 1 ) We introduce a novel knowledge-inspired scene graph prediction method that directly functions on 3D scene point clouds . To achieve our goal , we re-formulate the problem into two sub-tasks : knowledge learning and scene graph prediction with knowledge intervention ; ( 2 ) We propose a graph auto-encoder , called meta-embedding , to learn a group of class-dependent prototypical representations . The learned class prototypes can avoid perceptual errors and effectively encode prior structural knowledge in a more distinguishable latent space ; and ( 3 ) We design a scene graph prediction model benefiting from the prior learned knowledge in order to predict credible relation triplets . With our learned knowledge , our scene graph prediction model can achieve state-of-the-art performance on the 3D semantic scene graph ( 3DSSG ) dataset [ 29 ] compared to currently available methods . 2 Related Work . Knowledge-based Scene Graph Classification . Prior knowledge has been proven as an effective source of information to enhance object and relation recognition . Early methods use statistical co-occurrence as extra knowledge for scene graph inference . Zellers et al . observed that the predicate distribution is strongly dependent on the head and tail categories . It introduces a pre-computed bias into the final prediction [ 37 ] . The biased results naturally improve the most frequent combinations . To improve the classification performance on the rarely sampled ones , knowledge-embedded routing network ( KERN ) [ 4 ] implicitly fuses co-occurrence probabilities as weights in the message pass step to neutralize the long-tail phenomenon . Nevertheless , this integrated knowledge can hardly be used and studied in isolation since it is hard-coded as intrinsic parameters . Besides , some methods bring external knowledge bases into scene graph inference . Gu et al . extracted knowledge triplets from the ConceptNet knowledge bases [ 8 ] . Graph bridging network ( GB-Net ) adopts auxiliary edges as bridges that enable and facilitate message passing between knowledge graphs and scene graph [ 35 ] . This hand-crafted external information is highly dependent on the organization of knowledge and the quality of sources . Furthermore , several methods try to incorporate learning-based commonsense into scene graph inference . Zareian et al . proposed a local-global graph transformer to complete a masked scene graph by embedding the regular patterns into the network parameters [ 36 ] . More recently , Sharifzadeh et al . encoded inductive biases into the class-level prototypical representations learned from all previous perceptual outputs [ 22 ] . Inspired by [ 22 ] , our method also adopts prototypical representations as prior knowledge . Nevertheless , our method focuses on class-dependent and graph structure information , which aims to combat the confusion resulting from perceptual errors . 3D Scene Graph Prediction . 3D scene graph has only recently been studied in the 3D vision community . There are mainly two approaches for 3D scene graph construction . The first group of works benefits from image-based scene graph techniques . Armeni et al . firstly introduced a 3D scene graph construction framework with 3D meshes and registered RGB panoramic images [ 2 ] . It adopts multi-view consistency to enforce the accuracy of object identification by 2D-3D projection . Furthermore , succeeding methods [ 13 , 20 ] execute image-based scene graph prediction within dynamic simultaneously localization and mapping ( SLAM ) [ 12 , 17 , 30 ] . Kim et al . proposed a framework that merges the local graphs into a global one by fusing the same nodes in different images [ 13 ] . Spatial PerceptIon eNgine ( SPIN ) distinguishes human and object nodes by dynamic masking [ 20 ] . The second group of approaches directly process 3D data . Recently , Wald et al . proposed an end-to-end network , scene graph prediction network ( SPGN ) [ 29 ] , which takes scene scans as input . Along with the network , they also published the 3DSSG dataset of reconstructed scene scans . Similar to this work , we investigate a novel model to extend the scene graph prediction methods using 3D scene data solely based on point clouds . 3 New Method . Problem Formulation . We define a scene graph G as a directed graph with sets of nodes and edges , G = { V , E } . The node set V includes all object instances of a scene , and the edge set E consists of all possible connections of nodes as predicates in a relation triplet < subject , predicate , object > . The edges are directed by connecting neighbor nodes from subject to object . Unlike previous knowledgeembedded approaches [ 37 , 4 , 36 ] , we formulate the knowledge-inspired scene graph prediction task as a multi-task problem : we intend to learn knowledge and scene graphs from 3D data simultaneously . Mathematically , given a 3D scene data D with scene graph annotations , the training process is aimed to predict the probability of P ( G , K|D ) , where K is the prior knowledge . Furthermore , we can decompose the joint probability into two individual components : P ( G , K|D ) = P ( K|D ) P ( G|D , K ) . ( 1 ) This decomposition breaks the problem into two sub-tasks . The first sub-task , the predication of P ( K|D ) , extracts knowledge from existing 3D scene configurations . In this paper , the learned knowledge is in the form of prototypical embedding of each object and predicate category , named meta-embedding K = { emeta } . The set of meta-embedding is expected to represent each semantic class and also encode the corresponding class ’ s regular patterns in scene graphs . Towards this goal , we design a graph auto-encoder network , which only takes the one-hot labels as input without any perceptual information . The graphical encoder also can extract the topological interactions of each class . We close the distance of latent outputs and the related meta-embedding in metric space so that the meta-embedding records the prior knowledge of the training data . The second sub-task , P ( G|D , K ) , incorporates the meta-embedding and perceptual features to predict the possible scene graph elements . It is also the expected mathematical model in the evaluation stage with unseen 3D scene data . Meta-embedding and Learning Model . This model is designed to learn class-dependent prototypical knowledge embedding with a graph auto-encoder . The overview of the meta-embedding learning model is illustrated in Figure 2 . Since we consider the class-related patterns instead of perceptual information , the auto-encoder only takes the one-hot class labels as input . The first two multi-layer perceptron ( MLP ) blocks lift one-hot vectors into latent space with the same dimension size . Then , we assume that each object pair can have a relation ( including none relation ) and fully connect them as a graph where predicates are represented as edges . We design a graph neural network to further encode the graphical patterns of each node and edge in scene graphs by message passing [ 6 ] . At the network design level , we adapt the message passing mechanism to the directed graph structure . To calculate the latent representations { hn , lv } for all nodes v ∈ V and { h e , l uv } for all directed edges ( u , v ) ∈ E in every layer l , the message passing of nodes and edges can be formulated as : hn , l+1v = GRU n , l ( hn , lv , m n , l ( v ) ) , ( 2 ) he , l+1uv = GRU e , l ( he , luv , m e , l ( u , v ) ) , ( 3 ) where mn , l and me , l are the parameterized neural networks that calculate messages in layer l. We adopt the gated recurrent unit ( GRU ) [ 5 ] to update the hidden state for reserving initial information . It is worth noting that a node could be a subject or object in a relation triplet . We consider both conditions in message computation . For node v and one of its neighbors u , the message as subject ms , l ( v ) and as object mo , l ( v ) are computed by : ms , l ( v ) = LN ( φp ( he , lvu ) + φo ( h e , l u ) ) , ( 4 ) mo , l ( v ) = LN ( φp ( he , luv ) + φs ( h e , l u ) ) , ( 5 ) where φs , φp , and φo are three non-linear transformations for subject , predicate , and object respectively . To stabilize the numerical calculation in recurrent mechanism , we adopt the layer normalization ( LN ) [ 3 ] after feature fusion . The final message of node v is the average of ms , lv and m o , l v : mn , l ( v ) = 1 |Rsv|+ |Rov| ( ∑ u∈Rsv ( ms , l ( u ) ) + ∑ u∈Rov ( mo , l ( u ) ) ) , ( 6 ) where | · | denotes cardinality andRsv andRov are the set of connections of node v as subject and as object respectively . The calculation of edge ( u , v ) message is : me , l ( u , v ) = LN ( φs ( hn , lu ) + φo ( h n , l v ) ) . ( 7 ) We use the sum pooling operation to the stacked hidden states as the output model of the graph encoder . Since we know the class labels of the latent output embedding of the graph encoder , we update the corresponding meta-embedding vectors by reducing their distances ( e.g. , Euclidean distance ) in metric space . Given the latent embedding of nodes and those of edges , we select the corresponding class vectors from the object meta-embedding and the predicate meta-embedding . Then we reduce the distance between those embeddings as a loss function . As the values of meta-embedding are initialized as network parameters , they can eventually be updated with backpropagation [ 21 ] . The meta-embedding is updated only when seeing the related category , not depending on the number of training samples . This property makes the meta-embeddings qualified to be prototypical representations for even less frequent categories . The last two MLPs predict one-hot labels to enforce the latent embedding is class-related and complete the auto-encoder structure . Knowledge-inspired Scene Graph Prediction Model . This model is aimed to fuse prior metaembedding into the scene graph prediction task . Figure 3 shows the pipeline of the knowledgeinspired scene graph prediction model . Taking a scene point cloud with instance masks as input , we expect to encode geometrical features from both objects and object pairs . Based on the pioneering point set encoder , PointNet [ 18 ] , we enhance the original structure to capture multi-scale geometric information like several point completion methods [ 10 , 34 ] . For the object nodes , we directly use our modified multi-scale PointNet ( MS PointNet ) on the normalized point clouds . We first encode subject and object point sets separately for the object-pair edges and concatenate the latent features with the center coordinates of two point sets . To better encode the contradictory relationships ( e.g. , left and right , lower than and higher than ) , we adopt subtraction as an anti-symmetric operation on these two features . After a transformation by a two-layer MLP , we obtain the initial edge embedding from the object pairs . We adopt the same graph neural network design in the meta-embedding learning stage to capture the global context with perceptual information . In the first iteration , the graph neural network only takes the geometric features of the PointNet as input . The last two MLPs output the node and edge classification possibilities and offer the clues to select meta-embedding vectors of related categories . We select the five most confident meta-embedding for each node and edge and fuse them with the perception features . Based on our experiments , the large number of selected embedding is inevitable to introduce wrong knowledge , while a small meta-embedding set might contain no proper knowledge . In practice , we choose five meta-embedding as a compromise . In the second iteration , since the perceptual embedding and the knowledge are in different latent spaces , we transform them by two-layer feed-forward networks f and g respectively before directly fusing them . Denoting by { eimeta } i=1 ... k the selected meta-embedding and x the features encoded by multi-scale PointNet , we fuse the knowledge and perceptual information similar as in [ 22 ] : z = LN ( f ( x ) + k∑ i=1 g ( eimeta ) ) , ( 8 ) x̂ = LN ( z + ψ ( z ) ) . ( 9 ) where ψ is a two-layer MLP and x̂ is the knowledge-embedded input of the graph neural network in the second iteration . In practice , we only implement two iterations considering the time consumption since the classification performance can not be significantly improved over more iterations . Loss Function Design . We use the focal loss for object and predicate classification [ 14 , 29 ] in both meta-embedding learning and scene graph prediction tasks due to the data distribution imbalance . The focal loss we used is formulated as in [ 29 ] : Lfocal = α ( 1− p ) γ log ( p ) , ( 10 ) where p is the logits of a prediction of object or predicate , α is the normalized inverse frequency for object classification and an edge/no-edge weight for the predicates , and the γ is a hyper-parameter . We follow the parameter settings in [ 29 ] . To enforce the meta-embedding close to the class latent vectors in metric space , we adopt the Euclidean distance d between the output of the graph encoder and the corresponding meta-embedding as a loss : Ldist = ∑ v∈V d ( hv , evmeta ) + ∑ e∈E d ( he , eemeta ) , ( 11 ) where evmeta and eemeta are the corresponding meta-embedding of the categories of node v and edge e. In the meta-embedding learning task , we formulate the loss function as : Lmeta = Lobjfocal + L pred focal + Ldist . ( 12 ) Empirically , we use the summation of the three components with the same weight . In the scene graph prediction task , we only use the focal loss for object and predicate classification . Lsg = λLobjfocal + L pred focal , ( 13 ) where λ is a hyper-parameter to re-weight the components as in [ 29 ] .

Summary: This paper proposes a knowledge-inspired method for 3D scene graph prediction on point cloud. The key is to decompose the problem into two sub-tasks. First, a graph auto-encoder called meta-embedding is introduced to learn class-dependent prototypical representations only from the class labels to avoid visual confusions. Then a scene graph prediction model selects related meta-embedding from the first task as prior knowledge to predict credible relation triplets of 3D scene graph. Benefited from the learned commonsense knowledge, the scene graph prediction model achieves good performance on 3DSSG dataset.