Joint Modeling of Visual Objects and Relations for Scene Graph Generation

1 Introduction . Modern object recognition [ 32 , 10 , 35 ] and detection [ 28 , 27 , 57 ] systems excel at the perception of visual objects , which has significantly boosted many industrial applications such as intelligent surveillance [ 18 , 49 ] and autonomous driving [ 23 , 38 ] . To have a deeper understanding of a visual scene , detecting and recognizing the objects in the scene is however insufficient . Instead , a comprehensive cognition of visual objects and their relationships is more desirable . Scene Graph Generation ( SGG ) [ 13 ] is a natural way to achieve this goal , in which a graph incorporating all objects and their relations within a scene image is derived to represent its semantic structure . Most previous works for SGG [ 48 , 55 , 53 , 36 , 4 , 37 ] usually first independently predict different objects in a scene and then predict their relations independently . In practice , though such methods are very efficient , they ignore the dependency between different objects and between the relations of different object pairs . For example , a car could frequently co-occur with a street , and the relation eating could always appear along with the relation sitting on . Modeling such dependency could be very important for accurate scene graph prediction , especially for rare objects and relations . There are indeed some recent works [ 6 , 5 ] along this direction . For example , Dai et al . [ 6 ] explored the triplet-level label dependency among a head object , a tail object and their relation . These methods have shown very promising results , while they only explored the limited dependency within a triplet . How to capture the full dependency between different objects and between their relations within a whole scene graph remains very challenging and unexplored . 35th Conference on Neural Information Processing Systems ( NeurIPS 2021 ) . To attain such a goal , in this paper , we propose a principled approach called Joint Modeling for Scene Graph Generation ( JM-SGG ) to predict the whole scene graph by jointly capturing all the label dependency within it , i.e . the dependency between different objects and their relations and also the interdependency between them . Specifically , we model the joint distribution of all objects and relations in a scene graph with the conditional random field ( CRF ) framework [ 17 ] . To flexibly model the joint distribution , the key is to define effective potential functions on both nodes ( i.e . objects ) and edges ( i.e . relations between objects ) . We define the potential functions on objects according to the object representations extracted by existing neural network based object detector . It is however nontrivial to design effective potential functions on edges , since these potential functions have to capture the relation between two objects in an edge and meanwhile allow relational reasoning among different edges , which models the dependency among the relations on various edges . Inspired by the existing work of knowledge graph embedding [ 20 ] , which represents entities and relations in the same embedding space and performs relational reasoning in that space , we define our potential functions according to the knowledge graph embedding method and hence allow efficient relational reasoning between different object pairs in a scene graph . Such a fully expressive model also brings challenges to both learning and inference due to the complicated structures between different random variables in the CRF , i.e . objects and their relations . We therefore further propose an efficient and effective inference algorithm based on mean-field variational inference , which is able to assist the gradient estimation for learning and derive the most likely scene graph for test . Traditional mean-field methods usually suffer from the problem of slow convergence . Instead of starting from a randomly initialized variational distribution as in traditional mean-field methods , we propose to initialize the variational distribution , i.e . the marginal distribution of each object and each relation , with a factorized tweak of JM-SGG model , and then perform a few iterations of message passing induced by the fixed-point optimality condition of mean field to refine the variational distribution , which allows our approach to enjoy both good precision and efficiency . To summarize , in this paper , we make the following contributions : • We propose Joint Modeling for Scene Graph Generation ( JM-SGG ) which is a fully expressive model that can capture all the label dependency in a whole scene graph . • We propose a principled mean-field variational inference algorithm to enable the efficient learning and inference of JM-SGG model . • We verify the superior performance of our method on both relationship retrieval and zero-shot relationship retrieval tasks under various settings and metrics . Also , we illustrate the efficiency and efficacy of the proposed inference algorithm by thorough analytical experiments . 2 Related Work . Scene Graph Generation ( SGG ) . This task aims to extract structured representations from scene images [ 13 ] , including the category of objects and their relationships . Previous works performed SGG by propagating the information from different local regions [ 48 , 53 , 50 , 36 ] , introducing external knowledge [ 9 , 52 ] , employing well-designed loss functions [ 56 , 14 , 34 ] and performing unbiased scene graph prediction [ 4 , 19 , 37 ] . Most of these methods predict each object and relation label independently based on an informative representation , which fails to capture the rich label dependency within a scene graph and is thus less expressive . Several former works [ 6 , 5 ] attempted to model such label dependency within a single relational triplet but not on the whole scene graph . Improvements over existing methods . The proposed JM-SGG model is , to our best knowledge , the first approach that jointly models all the label dependency within a scene graph , including the one within object or relation labels and the one between these two kinds of labels . To attain this goal , a unified CRF is constructed for graphical modeling , and a mean-field variational inference algorithm is designed for efficient learning and inference , which show technical contributions . Conditional Random Fields ( CRFs ) . CRFs are a class of probabilistic graphical modeling methods which perform structured prediction upon the observed data . CRF-based approaches have been broadly studied on various computer vision problems , including segmentation [ 17 , 43 , 51 , 26 ] , superresolution [ 39 , 46 ] , image denoising [ 29 , 42 ] and scene graph generation [ 6 , 5 ] . These former works utilizing CRF for SGG [ 6 , 5 ] aimed to model the conditional distribution of a single triplet upon visual representations . By comparison , our approach models the conditional distribution of a whole scene graph upon the observed scene image , which is more expressive . 3 Problem Definition and Preliminary . 3.1 Problem Definition . This work focuses on extracting a scene graph , i.e . a structured representation of visual scene [ 13 ] , from an image . Formally , we define a scene graph as G = ( yO , R ) . yO denotes the category labels of all objects O in the image , and it holds that yo ∈ C for each object o ∈ O , where C stands for the set of all object categories , including the “ background ” category . R = { ( oh , r , ot ) } is the set of relational triplets/edges with r ∈ T as the relation type from head object oh to tail object ot ( oh , ot ∈ O ) , where T represents all relation types , including the type of “ no relation ” . In this work , we aim at jointly modeling visual objects and visual relations as defined below : Joint Scene Graph Modeling . Given an image I , we aim to jointly predict object categories yO and the relationships R among all objects , which models the joint distribution of scene graphs , i.e . p ( G|I ) = p ( yO , R|I ) , with comprehensively considering the dependency within yO and R and also the interdependency between them . 3.2 Conditional Random Fields . Conditional Random Field ( CRF ) is a discriminative undirected graphical model . Given a set of observed variables x , it models the joint distribution of labels y based on a Markov network G that specifies the dependency among all variables : p ( y|x ) = 1 Z ( x ) ∏ C φC ( xC , yC ) , Z ( x ) = ∑ y ∏ C φC ( xC , yC ) . ( 1 ) where φC denotes the nonnegative potential function defined over the variables in clique C ( a clique is a fully-connected local subgraph ) , and Z ( x ) is a normalization constant called partition function . 4 Model In this section , we introduce Joint Modeling for Scene Graph Generation ( JM-SGG ) . Current methods solve the problem by independently predicting each object and relation label upon an informative representation , and thus the prediction of different labels can not fully benefit each other . JM-SGG tackles the limitation by jointly modeling all the objects and relationships in a visual scene with a unified conditional random field , which enables the prediction of various object and relation labels to sufficiently interact with each other . Nevertheless , learning and inferring this complex CRF is nontrivial , and we thus propose to use maximum likelihood estimation combined with mean-field variational inference , yielding an efficient algorithm for learning and inference . Next , we elucidate the details of our approach . 4.1 Representation . In the JM-SGG model , we organize the observed scene image I and all object and relation labels in the latent scene graph ( i.e . yO and R ) as the nodes in a unified conditional random field . Since the interactions of these nodes are either for a single object or for the relationship between an object pair , we decompose the graphical structure of whole network into two sets of components . ( 1 ) Object components : For an object o ∈ O , we consider the dependency of its category label on its visual representation and thus connect yo with I , as shown in Fig . 1 ( a ) . ( 2 ) Relation components : for a relational triplet ( oh , r , ot ) ∈ R , we consider the dependency of relation type r on the visual cues in image I , and we also model the interdependency among the object and relation labels in this triplet ( i.e . yoh , yot and r ) , which forms a relation component as Fig . 1 ( b ) shows . By combining all object and relation components , the CRF can capture the comprehensive label dependency within a scene graph . We now define the joint distribution of scene graphs upon the observed scene image as below : pΘ ( G|I ) = 1 ZΘ ( I ) fΘ ( G , I ) , ( 2 ) fΘ ( G , I ) = ∏ o∈O φ ( yo , I ) ∏ ( oh , r , ot ) ∈R ψ ( r , yoh , yot , I ) , ( 3 ) where Θ summarizes the parameters of whole model , fΘ is an unnormalized likelihood function , ZΘ denotes the partition function , and φ and ψ are the potential functions defined on object and relation components , respectively . Next , we define these potential functions based on the extracted visual representations and the correlation among different labels . Visual representation extraction . Given a scene image I , we first utilize a standard object detector ( e.g . Faster R-CNN [ 28 ] in our implementation ) to obtain a set of bounding boxes which potentially contain the objects in the image , and object representations zO = { zo|o ∈ O } ( zo ∈ RD ) are then derived by RoIAlign [ 11 ] . We regard the union bounding box over a pair of objects as their context region and again use RoIAlign to get all context representations zR = { zht| ( oh , r , ot ) ∈ R } ( zht ∈ RD ) . Here , D denotes the latent dimension of objects and contexts . By denoting the whole object detector as gθ , this feature extraction process can be represented as : ( zO , zR ) = gθ ( I ) . Potential function definition . The potential function φ ( yo , I ) for object component models the dependency of object category yo on object representation zo by measuring their affinity . To conduct such a measure , we represent each object category with a prototype [ 33 ] ( i.e . a learnable embedding vector ) in the continuous space , which forms a prototype set C = { Ci ∈ RD|i ∈ C } for all object categories ( D denotes the dimension of object space ) . On such basis , we define φ ( yo , I ) by computing the distance between object representation zo and the prototype of object category yo : φ ( yo , I ) = exp ( −d ( Cyo , zo ) ) , ( 4 ) where d is a distance measure ( e.g . Euclidean distance in our practice ) . The potential function ψ ( r , yoh , yot , I ) for relation component models the dependency of relation type r on the relevant visual representations in image I , and it also models the interdependency among the object and relation labels of a triplet ( i.e . yoh , yot and r ) . Therefore , we can factorize ψ ( r , yoh , yot , I ) into a term ψvisual ( r , I ) for modeling visual influence and another term ψtriplet ( r , yoh , yot ) for modeling the label consistency within a triplet : ψ ( r , yoh , yot , I ) = ψvisual ( r , I ) ψtriplet ( r , yoh , yot ) . ( 5 ) Similarly , for measuring r in the continuous space , a prototype set T = { Tj ∈ RK |j ∈ T } is constructed for all relation types ( K denotes the dimension of relation space ) . We consider two kinds of visual representations that affect the prediction of relation type r , i.e . the context representation zht and the head and tail object representations zoh and zot . The influence of context representation can be easily measured by projecting context representation zht to the relation space and computing its distance to the prototype of relation type r. However , measuring the influence of head and tail object representations and evaluating the label consistency within a triplet are nontrivial , which require to model the ternary correlation among head object , tail object and their relationship . Inspired by the idea of TransR [ 20 ] , an effective knowledge graph embedding technique , we model such ternary correlation by treating each relation as a translation vector from head object embedding to tail object embedding in the same embedding space . Specifically , we first apply the translation vector Tr specified by relation r to head object embedding , and then compute the distance between the translated embedding and tail object embedding . Based on these thoughts , we define ψvisual ( r , I ) and ψtriplet ( r , yoh , yot ) as follows : ψvisual ( r , I ) = exp ( − ( d ( Tr , Mczht ) + d ( Mozoh + Tr , Mozot ) ) ) , ( 6 ) ψtriplet ( r , yoh , yot ) = exp ( −d ( MoCyoh + Tr , MoCyot ) ) , ( 7 ) where Mc ∈ RK×D denotes the projection matrix mapping from context space to relation space , and Mo ∈ RK×D is the projection matrix mapping from object space to relation space . Next , we state how to learn the parameters in JM-SGG model .

Summary: This paper proposes a joint prediction of the entire scene graph that fully captures the dependency among different objects and relations using a unified conditional random field.  The proposed model can be summarized as starting from joint modeling of the object and relation component label and visual features. Joint modeling uses CRF to model comprehensive dependency of the object using unnormalized likelihood and partition function. In addition, the potential function used for object and relation components computed affinity through a distance-based learnable prototype. Finally, the knowledge graph embedding techniques for projection of different embedding to the same embedding space (e.g. context space to relation space, object space to relation space) have been used.  All the learnable parameters are trained with maximum log-likelihood function and the partition function uses a mean-field variational inference for efficient sampling. Mean-field inference initialized with independent relation labels for triplet component factor update.  Authors have conducted extensive experiments on the Visual Genome dataset and a brief ablation study that provide a good insight into the method.