Knowledge-inspired 3D Scene Graph Prediction in Point Cloud

1 Introduction and Motivation . Scene graph has been studied as a viable means towards better interpretation of scene context , which encodes the semantic elements and their complex relationships of a scene [ 16 , 32 , 33 , 11 , 31 , 9 ] . As an enriched scene structure representation , scene graph parsing from 3D data has recently interested researchers in 3D vision for the tasks of scene understanding [ 2 ] , VR/AR scene interactions [ 25 ] , and robot navigation [ 7 ] . However , the scanned 3D data has a intrinsic nature of incompleteness [ 1 , 24 ] , geometrical similarity across categories , and other visual challenges . These perception errors result in extra difficulties in object and relationship recognition purely based on visual inputs . Fortunately , in cognitive psychology , besides visual perception , human beings also associate objects and concepts with certain abstract symbolic class-dependent representations and regular combinations ∗Corresponding authors . 35th Conference on Neural Information Processing Systems ( NeurIPS 2021 ) . of scene context , known as knowledge . However , utilizing prior knowledge still remains challenging in scene graph prediction . Prior knowledge-incorporated methods have several limitations . Static knowledge ( e.g. , heuristic co-occurrence statistics [ 37 ] , mined triplet facts [ 8 ] , or knowledge graph [ 35 ] ) could be incomplete and inaccurate depending on the quality of knowledge sources and the domain knowledge covered . Thus , several methods tend to use learning-based knowledge to capture the graphical structure and regular combinations in the exact domain . However , the knowledge learned from visual information also inherits perceptual confusion into the knowledge embedding and network parameters [ 22 , 36 ] . To ameliorate , this paper advocates a novel 3D scene graph prediction method with learned prior knowledge . The knowledge can be encoded in various forms ( e.g. , vectors , graphs , and natural language ) . In our work , we choose class-dependent prototypical memories as the carrier of knowledge to tackle the key challenges mentioned above . We re-formulate the target problem into two sub-tasks : knowledge learning and knowledge-inspired scene graph prediction . Unlike previous works , we learn the knowledge only depending on semantic categories and regular structural patterns without any visual confusion . The knowledge learning model is a graph auto-encoder that takes the one-hot class labels as inputs with scene graph structure annotations . Since we do not involve any geometric and visual features , the latent embedding indeed encodes only category-related regular structure patterns . To correctly record this prior knowledge , we enforce a group of prototypes [ 23 ] , namely meta-embedding , to approach the latent areas of the corresponding category in the metric space . As we illustrated in Figure 1 , the meta-embedding is capable of representing each semantic class as prototypical knowledge . Once trained , the scene graph prediction model unites the geometric features and the corresponding knowledge meta-embedding to predict each element in a triplet with the < subject , predicate , object > structure . Extensive experiments confirm that the effectiveness of knowledge extraction and the injection of the meta-embedding can improve the scene graph prediction accuracy . The primary contributions of this paper could be summarized as follows : ( 1 ) We introduce a novel knowledge-inspired scene graph prediction method that directly functions on 3D scene point clouds . To achieve our goal , we re-formulate the problem into two sub-tasks : knowledge learning and scene graph prediction with knowledge intervention ; ( 2 ) We propose a graph auto-encoder , called meta-embedding , to learn a group of class-dependent prototypical representations . The learned class prototypes can avoid perceptual errors and effectively encode prior structural knowledge in a more distinguishable latent space ; and ( 3 ) We design a scene graph prediction model benefiting from the prior learned knowledge in order to predict credible relation triplets . With our learned knowledge , our scene graph prediction model can achieve state-of-the-art performance on the 3D semantic scene graph ( 3DSSG ) dataset [ 29 ] compared to currently available methods . 2 Related Work . Knowledge-based Scene Graph Classification . Prior knowledge has been proven as an effective source of information to enhance object and relation recognition . Early methods use statistical co-occurrence as extra knowledge for scene graph inference . Zellers et al . observed that the predicate distribution is strongly dependent on the head and tail categories . It introduces a pre-computed bias into the final prediction [ 37 ] . The biased results naturally improve the most frequent combinations . To improve the classification performance on the rarely sampled ones , knowledge-embedded routing network ( KERN ) [ 4 ] implicitly fuses co-occurrence probabilities as weights in the message pass step to neutralize the long-tail phenomenon . Nevertheless , this integrated knowledge can hardly be used and studied in isolation since it is hard-coded as intrinsic parameters . Besides , some methods bring external knowledge bases into scene graph inference . Gu et al . extracted knowledge triplets from the ConceptNet knowledge bases [ 8 ] . Graph bridging network ( GB-Net ) adopts auxiliary edges as bridges that enable and facilitate message passing between knowledge graphs and scene graph [ 35 ] . This hand-crafted external information is highly dependent on the organization of knowledge and the quality of sources . Furthermore , several methods try to incorporate learning-based commonsense into scene graph inference . Zareian et al . proposed a local-global graph transformer to complete a masked scene graph by embedding the regular patterns into the network parameters [ 36 ] . More recently , Sharifzadeh et al . encoded inductive biases into the class-level prototypical representations learned from all previous perceptual outputs [ 22 ] . Inspired by [ 22 ] , our method also adopts prototypical representations as prior knowledge . Nevertheless , our method focuses on class-dependent and graph structure information , which aims to combat the confusion resulting from perceptual errors . 3D Scene Graph Prediction . 3D scene graph has only recently been studied in the 3D vision community . There are mainly two approaches for 3D scene graph construction . The first group of works benefits from image-based scene graph techniques . Armeni et al . firstly introduced a 3D scene graph construction framework with 3D meshes and registered RGB panoramic images [ 2 ] . It adopts multi-view consistency to enforce the accuracy of object identification by 2D-3D projection . Furthermore , succeeding methods [ 13 , 20 ] execute image-based scene graph prediction within dynamic simultaneously localization and mapping ( SLAM ) [ 12 , 17 , 30 ] . Kim et al . proposed a framework that merges the local graphs into a global one by fusing the same nodes in different images [ 13 ] . Spatial PerceptIon eNgine ( SPIN ) distinguishes human and object nodes by dynamic masking [ 20 ] . The second group of approaches directly process 3D data . Recently , Wald et al . proposed an end-to-end network , scene graph prediction network ( SPGN ) [ 29 ] , which takes scene scans as input . Along with the network , they also published the 3DSSG dataset of reconstructed scene scans . Similar to this work , we investigate a novel model to extend the scene graph prediction methods using 3D scene data solely based on point clouds . 3 New Method . Problem Formulation . We define a scene graph G as a directed graph with sets of nodes and edges , G = { V , E } . The node set V includes all object instances of a scene , and the edge set E consists of all possible connections of nodes as predicates in a relation triplet < subject , predicate , object > . The edges are directed by connecting neighbor nodes from subject to object . Unlike previous knowledgeembedded approaches [ 37 , 4 , 36 ] , we formulate the knowledge-inspired scene graph prediction task as a multi-task problem : we intend to learn knowledge and scene graphs from 3D data simultaneously . Mathematically , given a 3D scene data D with scene graph annotations , the training process is aimed to predict the probability of P ( G , K|D ) , where K is the prior knowledge . Furthermore , we can decompose the joint probability into two individual components : P ( G , K|D ) = P ( K|D ) P ( G|D , K ) . ( 1 ) This decomposition breaks the problem into two sub-tasks . The first sub-task , the predication of P ( K|D ) , extracts knowledge from existing 3D scene configurations . In this paper , the learned knowledge is in the form of prototypical embedding of each object and predicate category , named meta-embedding K = { emeta } . The set of meta-embedding is expected to represent each semantic class and also encode the corresponding class ’ s regular patterns in scene graphs . Towards this goal , we design a graph auto-encoder network , which only takes the one-hot labels as input without any perceptual information . The graphical encoder also can extract the topological interactions of each class . We close the distance of latent outputs and the related meta-embedding in metric space so that the meta-embedding records the prior knowledge of the training data . The second sub-task , P ( G|D , K ) , incorporates the meta-embedding and perceptual features to predict the possible scene graph elements . It is also the expected mathematical model in the evaluation stage with unseen 3D scene data . Meta-embedding and Learning Model . This model is designed to learn class-dependent prototypical knowledge embedding with a graph auto-encoder . The overview of the meta-embedding learning model is illustrated in Figure 2 . Since we consider the class-related patterns instead of perceptual information , the auto-encoder only takes the one-hot class labels as input . The first two multi-layer perceptron ( MLP ) blocks lift one-hot vectors into latent space with the same dimension size . Then , we assume that each object pair can have a relation ( including none relation ) and fully connect them as a graph where predicates are represented as edges . We design a graph neural network to further encode the graphical patterns of each node and edge in scene graphs by message passing [ 6 ] . At the network design level , we adapt the message passing mechanism to the directed graph structure . To calculate the latent representations { hn , lv } for all nodes v ∈ V and { h e , l uv } for all directed edges ( u , v ) ∈ E in every layer l , the message passing of nodes and edges can be formulated as : hn , l+1v = GRU n , l ( hn , lv , m n , l ( v ) ) , ( 2 ) he , l+1uv = GRU e , l ( he , luv , m e , l ( u , v ) ) , ( 3 ) where mn , l and me , l are the parameterized neural networks that calculate messages in layer l. We adopt the gated recurrent unit ( GRU ) [ 5 ] to update the hidden state for reserving initial information . It is worth noting that a node could be a subject or object in a relation triplet . We consider both conditions in message computation . For node v and one of its neighbors u , the message as subject ms , l ( v ) and as object mo , l ( v ) are computed by : ms , l ( v ) = LN ( φp ( he , lvu ) + φo ( h e , l u ) ) , ( 4 ) mo , l ( v ) = LN ( φp ( he , luv ) + φs ( h e , l u ) ) , ( 5 ) where φs , φp , and φo are three non-linear transformations for subject , predicate , and object respectively . To stabilize the numerical calculation in recurrent mechanism , we adopt the layer normalization ( LN ) [ 3 ] after feature fusion . The final message of node v is the average of ms , lv and m o , l v : mn , l ( v ) = 1 |Rsv|+ |Rov| ( ∑ u∈Rsv ( ms , l ( u ) ) + ∑ u∈Rov ( mo , l ( u ) ) ) , ( 6 ) where | · | denotes cardinality andRsv andRov are the set of connections of node v as subject and as object respectively . The calculation of edge ( u , v ) message is : me , l ( u , v ) = LN ( φs ( hn , lu ) + φo ( h n , l v ) ) . ( 7 ) We use the sum pooling operation to the stacked hidden states as the output model of the graph encoder . Since we know the class labels of the latent output embedding of the graph encoder , we update the corresponding meta-embedding vectors by reducing their distances ( e.g. , Euclidean distance ) in metric space . Given the latent embedding of nodes and those of edges , we select the corresponding class vectors from the object meta-embedding and the predicate meta-embedding . Then we reduce the distance between those embeddings as a loss function . As the values of meta-embedding are initialized as network parameters , they can eventually be updated with backpropagation [ 21 ] . The meta-embedding is updated only when seeing the related category , not depending on the number of training samples . This property makes the meta-embeddings qualified to be prototypical representations for even less frequent categories . The last two MLPs predict one-hot labels to enforce the latent embedding is class-related and complete the auto-encoder structure . Knowledge-inspired Scene Graph Prediction Model . This model is aimed to fuse prior metaembedding into the scene graph prediction task . Figure 3 shows the pipeline of the knowledgeinspired scene graph prediction model . Taking a scene point cloud with instance masks as input , we expect to encode geometrical features from both objects and object pairs . Based on the pioneering point set encoder , PointNet [ 18 ] , we enhance the original structure to capture multi-scale geometric information like several point completion methods [ 10 , 34 ] . For the object nodes , we directly use our modified multi-scale PointNet ( MS PointNet ) on the normalized point clouds . We first encode subject and object point sets separately for the object-pair edges and concatenate the latent features with the center coordinates of two point sets . To better encode the contradictory relationships ( e.g. , left and right , lower than and higher than ) , we adopt subtraction as an anti-symmetric operation on these two features . After a transformation by a two-layer MLP , we obtain the initial edge embedding from the object pairs . We adopt the same graph neural network design in the meta-embedding learning stage to capture the global context with perceptual information . In the first iteration , the graph neural network only takes the geometric features of the PointNet as input . The last two MLPs output the node and edge classification possibilities and offer the clues to select meta-embedding vectors of related categories . We select the five most confident meta-embedding for each node and edge and fuse them with the perception features . Based on our experiments , the large number of selected embedding is inevitable to introduce wrong knowledge , while a small meta-embedding set might contain no proper knowledge . In practice , we choose five meta-embedding as a compromise . In the second iteration , since the perceptual embedding and the knowledge are in different latent spaces , we transform them by two-layer feed-forward networks f and g respectively before directly fusing them . Denoting by { eimeta } i=1 ... k the selected meta-embedding and x the features encoded by multi-scale PointNet , we fuse the knowledge and perceptual information similar as in [ 22 ] : z = LN ( f ( x ) + k∑ i=1 g ( eimeta ) ) , ( 8 ) x̂ = LN ( z + ψ ( z ) ) . ( 9 ) where ψ is a two-layer MLP and x̂ is the knowledge-embedded input of the graph neural network in the second iteration . In practice , we only implement two iterations considering the time consumption since the classification performance can not be significantly improved over more iterations . Loss Function Design . We use the focal loss for object and predicate classification [ 14 , 29 ] in both meta-embedding learning and scene graph prediction tasks due to the data distribution imbalance . The focal loss we used is formulated as in [ 29 ] : Lfocal = α ( 1− p ) γ log ( p ) , ( 10 ) where p is the logits of a prediction of object or predicate , α is the normalized inverse frequency for object classification and an edge/no-edge weight for the predicates , and the γ is a hyper-parameter . We follow the parameter settings in [ 29 ] . To enforce the meta-embedding close to the class latent vectors in metric space , we adopt the Euclidean distance d between the output of the graph encoder and the corresponding meta-embedding as a loss : Ldist = ∑ v∈V d ( hv , evmeta ) + ∑ e∈E d ( he , eemeta ) , ( 11 ) where evmeta and eemeta are the corresponding meta-embedding of the categories of node v and edge e. In the meta-embedding learning task , we formulate the loss function as : Lmeta = Lobjfocal + L pred focal + Ldist . ( 12 ) Empirically , we use the summation of the three components with the same weight . In the scene graph prediction task , we only use the focal loss for object and predicate classification . Lsg = λLobjfocal + L pred focal , ( 13 ) where λ is a hyper-parameter to re-weight the components as in [ 29 ] .

Summary: In this paper, the authors study the 3D scene graph prediction task. The major contribution is the proposed “object/predicate meta-embedding,” which learns class-specific embedding to assist scene graph prediction. Ablation study verifies the effectiveness of the proposed embedding learning method.