Joint Modeling of Visual Objects and Relations for Scene Graph Generation

1 Introduction . Modern object recognition [ 32 , 10 , 35 ] and detection [ 28 , 27 , 57 ] systems excel at the perception of visual objects , which has significantly boosted many industrial applications such as intelligent surveillance [ 18 , 49 ] and autonomous driving [ 23 , 38 ] . To have a deeper understanding of a visual scene , detecting and recognizing the objects in the scene is however insufficient . Instead , a comprehensive cognition of visual objects and their relationships is more desirable . Scene Graph Generation ( SGG ) [ 13 ] is a natural way to achieve this goal , in which a graph incorporating all objects and their relations within a scene image is derived to represent its semantic structure . Most previous works for SGG [ 48 , 55 , 53 , 36 , 4 , 37 ] usually first independently predict different objects in a scene and then predict their relations independently . In practice , though such methods are very efficient , they ignore the dependency between different objects and between the relations of different object pairs . For example , a car could frequently co-occur with a street , and the relation eating could always appear along with the relation sitting on . Modeling such dependency could be very important for accurate scene graph prediction , especially for rare objects and relations . There are indeed some recent works [ 6 , 5 ] along this direction . For example , Dai et al . [ 6 ] explored the triplet-level label dependency among a head object , a tail object and their relation . These methods have shown very promising results , while they only explored the limited dependency within a triplet . How to capture the full dependency between different objects and between their relations within a whole scene graph remains very challenging and unexplored . 35th Conference on Neural Information Processing Systems ( NeurIPS 2021 ) . To attain such a goal , in this paper , we propose a principled approach called Joint Modeling for Scene Graph Generation ( JM-SGG ) to predict the whole scene graph by jointly capturing all the label dependency within it , i.e . the dependency between different objects and their relations and also the interdependency between them . Specifically , we model the joint distribution of all objects and relations in a scene graph with the conditional random field ( CRF ) framework [ 17 ] . To flexibly model the joint distribution , the key is to define effective potential functions on both nodes ( i.e . objects ) and edges ( i.e . relations between objects ) . We define the potential functions on objects according to the object representations extracted by existing neural network based object detector . It is however nontrivial to design effective potential functions on edges , since these potential functions have to capture the relation between two objects in an edge and meanwhile allow relational reasoning among different edges , which models the dependency among the relations on various edges . Inspired by the existing work of knowledge graph embedding [ 20 ] , which represents entities and relations in the same embedding space and performs relational reasoning in that space , we define our potential functions according to the knowledge graph embedding method and hence allow efficient relational reasoning between different object pairs in a scene graph . Such a fully expressive model also brings challenges to both learning and inference due to the complicated structures between different random variables in the CRF , i.e . objects and their relations . We therefore further propose an efficient and effective inference algorithm based on mean-field variational inference , which is able to assist the gradient estimation for learning and derive the most likely scene graph for test . Traditional mean-field methods usually suffer from the problem of slow convergence . Instead of starting from a randomly initialized variational distribution as in traditional mean-field methods , we propose to initialize the variational distribution , i.e . the marginal distribution of each object and each relation , with a factorized tweak of JM-SGG model , and then perform a few iterations of message passing induced by the fixed-point optimality condition of mean field to refine the variational distribution , which allows our approach to enjoy both good precision and efficiency . To summarize , in this paper , we make the following contributions : • We propose Joint Modeling for Scene Graph Generation ( JM-SGG ) which is a fully expressive model that can capture all the label dependency in a whole scene graph . • We propose a principled mean-field variational inference algorithm to enable the efficient learning and inference of JM-SGG model . • We verify the superior performance of our method on both relationship retrieval and zero-shot relationship retrieval tasks under various settings and metrics . Also , we illustrate the efficiency and efficacy of the proposed inference algorithm by thorough analytical experiments . 2 Related Work . Scene Graph Generation ( SGG ) . This task aims to extract structured representations from scene images [ 13 ] , including the category of objects and their relationships . Previous works performed SGG by propagating the information from different local regions [ 48 , 53 , 50 , 36 ] , introducing external knowledge [ 9 , 52 ] , employing well-designed loss functions [ 56 , 14 , 34 ] and performing unbiased scene graph prediction [ 4 , 19 , 37 ] . Most of these methods predict each object and relation label independently based on an informative representation , which fails to capture the rich label dependency within a scene graph and is thus less expressive . Several former works [ 6 , 5 ] attempted to model such label dependency within a single relational triplet but not on the whole scene graph . Improvements over existing methods . The proposed JM-SGG model is , to our best knowledge , the first approach that jointly models all the label dependency within a scene graph , including the one within object or relation labels and the one between these two kinds of labels . To attain this goal , a unified CRF is constructed for graphical modeling , and a mean-field variational inference algorithm is designed for efficient learning and inference , which show technical contributions . Conditional Random Fields ( CRFs ) . CRFs are a class of probabilistic graphical modeling methods which perform structured prediction upon the observed data . CRF-based approaches have been broadly studied on various computer vision problems , including segmentation [ 17 , 43 , 51 , 26 ] , superresolution [ 39 , 46 ] , image denoising [ 29 , 42 ] and scene graph generation [ 6 , 5 ] . These former works utilizing CRF for SGG [ 6 , 5 ] aimed to model the conditional distribution of a single triplet upon visual representations . By comparison , our approach models the conditional distribution of a whole scene graph upon the observed scene image , which is more expressive . 3 Problem Definition and Preliminary . 3.1 Problem Definition . This work focuses on extracting a scene graph , i.e . a structured representation of visual scene [ 13 ] , from an image . Formally , we define a scene graph as G = ( yO , R ) . yO denotes the category labels of all objects O in the image , and it holds that yo ∈ C for each object o ∈ O , where C stands for the set of all object categories , including the “ background ” category . R = { ( oh , r , ot ) } is the set of relational triplets/edges with r ∈ T as the relation type from head object oh to tail object ot ( oh , ot ∈ O ) , where T represents all relation types , including the type of “ no relation ” . In this work , we aim at jointly modeling visual objects and visual relations as defined below : Joint Scene Graph Modeling . Given an image I , we aim to jointly predict object categories yO and the relationships R among all objects , which models the joint distribution of scene graphs , i.e . p ( G|I ) = p ( yO , R|I ) , with comprehensively considering the dependency within yO and R and also the interdependency between them . 3.2 Conditional Random Fields . Conditional Random Field ( CRF ) is a discriminative undirected graphical model . Given a set of observed variables x , it models the joint distribution of labels y based on a Markov network G that specifies the dependency among all variables : p ( y|x ) = 1 Z ( x ) ∏ C φC ( xC , yC ) , Z ( x ) = ∑ y ∏ C φC ( xC , yC ) . ( 1 ) where φC denotes the nonnegative potential function defined over the variables in clique C ( a clique is a fully-connected local subgraph ) , and Z ( x ) is a normalization constant called partition function . 4 Model In this section , we introduce Joint Modeling for Scene Graph Generation ( JM-SGG ) . Current methods solve the problem by independently predicting each object and relation label upon an informative representation , and thus the prediction of different labels can not fully benefit each other . JM-SGG tackles the limitation by jointly modeling all the objects and relationships in a visual scene with a unified conditional random field , which enables the prediction of various object and relation labels to sufficiently interact with each other . Nevertheless , learning and inferring this complex CRF is nontrivial , and we thus propose to use maximum likelihood estimation combined with mean-field variational inference , yielding an efficient algorithm for learning and inference . Next , we elucidate the details of our approach . 4.1 Representation . In the JM-SGG model , we organize the observed scene image I and all object and relation labels in the latent scene graph ( i.e . yO and R ) as the nodes in a unified conditional random field . Since the interactions of these nodes are either for a single object or for the relationship between an object pair , we decompose the graphical structure of whole network into two sets of components . ( 1 ) Object components : For an object o ∈ O , we consider the dependency of its category label on its visual representation and thus connect yo with I , as shown in Fig . 1 ( a ) . ( 2 ) Relation components : for a relational triplet ( oh , r , ot ) ∈ R , we consider the dependency of relation type r on the visual cues in image I , and we also model the interdependency among the object and relation labels in this triplet ( i.e . yoh , yot and r ) , which forms a relation component as Fig . 1 ( b ) shows . By combining all object and relation components , the CRF can capture the comprehensive label dependency within a scene graph . We now define the joint distribution of scene graphs upon the observed scene image as below : pΘ ( G|I ) = 1 ZΘ ( I ) fΘ ( G , I ) , ( 2 ) fΘ ( G , I ) = ∏ o∈O φ ( yo , I ) ∏ ( oh , r , ot ) ∈R ψ ( r , yoh , yot , I ) , ( 3 ) where Θ summarizes the parameters of whole model , fΘ is an unnormalized likelihood function , ZΘ denotes the partition function , and φ and ψ are the potential functions defined on object and relation components , respectively . Next , we define these potential functions based on the extracted visual representations and the correlation among different labels . Visual representation extraction . Given a scene image I , we first utilize a standard object detector ( e.g . Faster R-CNN [ 28 ] in our implementation ) to obtain a set of bounding boxes which potentially contain the objects in the image , and object representations zO = { zo|o ∈ O } ( zo ∈ RD ) are then derived by RoIAlign [ 11 ] . We regard the union bounding box over a pair of objects as their context region and again use RoIAlign to get all context representations zR = { zht| ( oh , r , ot ) ∈ R } ( zht ∈ RD ) . Here , D denotes the latent dimension of objects and contexts . By denoting the whole object detector as gθ , this feature extraction process can be represented as : ( zO , zR ) = gθ ( I ) . Potential function definition . The potential function φ ( yo , I ) for object component models the dependency of object category yo on object representation zo by measuring their affinity . To conduct such a measure , we represent each object category with a prototype [ 33 ] ( i.e . a learnable embedding vector ) in the continuous space , which forms a prototype set C = { Ci ∈ RD|i ∈ C } for all object categories ( D denotes the dimension of object space ) . On such basis , we define φ ( yo , I ) by computing the distance between object representation zo and the prototype of object category yo : φ ( yo , I ) = exp ( −d ( Cyo , zo ) ) , ( 4 ) where d is a distance measure ( e.g . Euclidean distance in our practice ) . The potential function ψ ( r , yoh , yot , I ) for relation component models the dependency of relation type r on the relevant visual representations in image I , and it also models the interdependency among the object and relation labels of a triplet ( i.e . yoh , yot and r ) . Therefore , we can factorize ψ ( r , yoh , yot , I ) into a term ψvisual ( r , I ) for modeling visual influence and another term ψtriplet ( r , yoh , yot ) for modeling the label consistency within a triplet : ψ ( r , yoh , yot , I ) = ψvisual ( r , I ) ψtriplet ( r , yoh , yot ) . ( 5 ) Similarly , for measuring r in the continuous space , a prototype set T = { Tj ∈ RK |j ∈ T } is constructed for all relation types ( K denotes the dimension of relation space ) . We consider two kinds of visual representations that affect the prediction of relation type r , i.e . the context representation zht and the head and tail object representations zoh and zot . The influence of context representation can be easily measured by projecting context representation zht to the relation space and computing its distance to the prototype of relation type r. However , measuring the influence of head and tail object representations and evaluating the label consistency within a triplet are nontrivial , which require to model the ternary correlation among head object , tail object and their relationship . Inspired by the idea of TransR [ 20 ] , an effective knowledge graph embedding technique , we model such ternary correlation by treating each relation as a translation vector from head object embedding to tail object embedding in the same embedding space . Specifically , we first apply the translation vector Tr specified by relation r to head object embedding , and then compute the distance between the translated embedding and tail object embedding . Based on these thoughts , we define ψvisual ( r , I ) and ψtriplet ( r , yoh , yot ) as follows : ψvisual ( r , I ) = exp ( − ( d ( Tr , Mczht ) + d ( Mozoh + Tr , Mozot ) ) ) , ( 6 ) ψtriplet ( r , yoh , yot ) = exp ( −d ( MoCyoh + Tr , MoCyot ) ) , ( 7 ) where Mc ∈ RK×D denotes the projection matrix mapping from context space to relation space , and Mo ∈ RK×D is the projection matrix mapping from object space to relation space . Next , we state how to learn the parameters in JM-SGG model .

Summary: This paper presents a conditional random field (CRF) based joint modeling of objects and relations for the task of scene graph generation. Unlike previous relation discovery methods, this paper employs embedding-based relation feature representation, and thus enables an unified modeling of the unitary and clique potential functions. Using MCMC sampler and mean-field variational inference, the proposed JM-SGG can effectively update the relation triplets utilizing the contextual label dependency. The experiments show that JM-SGG has good performances on the relationship retrieval task, even under the zero-shot setting.