Learning to Compose Visual Relations

1 Introduction . The ability to reason about the component objects and their relations in a scene is key for a wide variety of robotics and AI tasks , such as multistep manipulation planning [ 11 ] , concept learning [ 25 ] , navigation [ 43 ] , and dynamics prediction [ 3 ] . While a large body of work has explored inferring and understanding the underlying objects in a scene , robustly understanding the component relations in a scene remains a challenging task . In this work , we explore how to robustly understand relational scene description ( Figure 1 ) . Naively , one approach towards understanding relational scene descriptions is to utilize existing multi-modal language and vision models . Such an approach has recently achieved great success in DALL-E [ 36 ] and CLIP [ 35 ] , both of which show compelling results on encoding object properties with language . However , when these approaches are instead utilized to encode relations between objects , their performance rapidly deteriorates , as shown in [ 36 ] and which we further illustrate in Figure 7 . We posit that the lack of compositionality in the language encoder prevents it from capturing all the underlying relations in an image . To remedy this issue , we propose instead to factorize the scene description with respect to each individual relation . Separate models are utilized to encode each individual relation , which are then subsequently composed together to represent a relational scene description . The most straightforward ⇤indicates equal contribution 35th Conference on Neural Information Processing Systems ( NeurIPS 2021 ) . approach is to specify distinct regions of an image in which each relation can be located , as well as a composed relation description corresponding to the combination of all these regions . Such an approach has significant drawbacks . In practice , the location of one pair of objects in a relation description may be heavily influenced by the location of objects specified by another relation description . Specifying a priori the exact location of a relation will thus severely hamper the number of possible scenes that can be realized with a given set of relations . Is it possible to factorize relational descriptions of a scene and generate images that incorporate each given relation description simultaneously ? In this work , we propose to represent and factorize individual relations as unnormalized densities using Energy-Based Models . A relational scene description is represented as the product of the individual probability distributions across relations , with each individual relation specifying a separate probability distribution over images . Such a composition enables interactions between multiple relations to be modeled . We show that this resultant framework enables us to reliably capture and generate images with multiple composed relational descriptions . It further enables us to edit images to have a desired set of relations . Finally , by measuring the relative densities assigned to different relational descriptions , we are able to infer the objects and their relations in a scene for downstream tasks , such as image-to-text retrieval and classification . There are three main contributions of our work . First , we present a framework to factorize and compose separate object relations . We show that the proposed framework is able to generate and edit images with multiple composed relations and significantly outperforms baseline approaches . Secondly , we show that our approach is able to infer the underlying relational scene descriptions and is robust enough in understanding semantically equivalent relational scene descriptions . Finally , we show that our approach can generalize to a previously unseen relation description , even if the underlying objects and descriptions are from a separate dataset not seen during training . We believe that such generalization is crucial for a general artificial intelligence system to adapt to the infinite number of variations of the world around it . 2 Related Work . Language Guided Scene Generation . A large body of work has explored scene generation utilizing text descriptions [ 13 , 17 , 20 , 30 , 34 , 36 , 38–40 , 46–48 ] . In contrast to our work , prior work [ 17 , 38 , 47 , 48 ] have focused on generating images given only a limited number of relation descriptions . Recently , [ 36 ] shows compelling results on utilizing text to generate images , but also explicitly state that relational generation was a weakness of the model . In this work , we seek to tackle how we may generate images given an underlying relational description . Visual Relation Understanding . To understand visual relations in a scene , many works applied neural networks to graphs [ 1–3 , 5 , 14 , 15 , 18 , 20 , 28 , 29 , 32 , 37 ] . Raposo et al . [ 37 ] proposed Relation Network ( RN ) to explicitly compute relations from static input scenes while we implicitly encode relational information to generate and edit images based on given relations and objects . Johnson et al . [ 20 ] proposed to condition image generation on relations by utilizing scene graphs which were further explored in [ 1 , 14 , 15 , 18 , 29 , 31 , 44 ] . However , these approaches require excessive supervisions , such as bounding boxes , segmentation masks , etc. , to infer and generate the underlying relations in an image . Such a setting restricts the possible combinations of individual relations in a scene . In our work , we present a method that enables us to generate images given only a relational scene description . Energy Based Models . Our work is related to existing work on energy-based models [ 6 , 8 , 10 , 12 , 22 , 33 , 42 , 45 ] . Most similar to our work is that of [ 7 ] , which proposes a framework of utilizing EBMs to compose several object descriptions together . In contrast , in this work , we study the problem of how we may compose relational descriptions together , an important and challenging task for existing text description understanding systems . 3 Method . Given a training dataset C = { xi , Ri } Ni=1 with N distinct images xi 2 RD and associated relational descriptions Ri , we aim at learning the underlying probability distribution p✓ ( x|R ) — the probability distributions of an image given a corresponding relational description . To represent p✓ ( x|R ) , we split a relational description R into K separate relations { r1 · · · , rK } and model each component relation separately using a probability distribution p✓ ( x|rk ) which is represented as an Energy-Based Model . Our overall scene probability distribution is then modeled by a composition of individual probability distributions for each relation description p✓ ( x|R ) / Q k p✓ ( x|rk ) . In this section , we give an overview of our approach towards factorizing and representing a relational scene description . We first provide a background overview of Energy-Based Models ( EBMs ) in Section 3.1 . We then describe how we may parameterize individual relational probability distributions with EBMs in Section 3.2 . We further describe how we compose relational probability distributions to model a relational scene description in Section 3.3 . Finally , we illustrate downstream applications of our relational scene understanding model in Section 3.4 . 3.1 Energy-Based Models . We model each relational probability distribution utilizing an Energy-Based Model ( EBM ) [ 6 , 26 ] . EBMs are a class of unnormalized probability models , which parameterize a probability distribution p✓ ( x ) utilizing a learned energy function E✓ ( x ) : p✓ ( x ) / e E✓ ( x ) . ( 1 ) EBMs are typically trained utilizing contrastive divergence [ 16 ] , where energies of training datapoints are decreased and energies of sampled data points from p✓ ( x ) are increased . We adopt the training code and models from [ 8 ] to train our EBMs . To generate samples from an EBM p✓ ( x ) , we utilize MCMC sampling on the underlying distribution , and Langevin dynamics , which refines a data sample iteratively from a random noise : x̃m = x̃m 1 2 rxE✓ ( x̃m 1 ) + ! m , ! m ⇠ N ( 0 , ) ( 2 ) where m refers to the iteration and is the step size , utilizing the gradient of the energy function with respect to the input sample rxE✓ , and ! m is sampled from a Gaussian noise . EBMs enable us to naturally compose separate probability distributions together [ 7 ] . In particular , given a set of independent marginal distributions { pi✓ ( x ) } , the joint probability distribution is represented as : Y i pi✓ ( x ) / e P i E i ✓ ( x ) , ( 3 ) where we utilize Langevin dynamics to sample from the resultant joint probability distribution . 3.2 Learning Relational Energy Functions . Given a scene relation ri , described using a set of words { w1i , . . . wni } , we seek to learn a conditional EBM to model the underlying probability distribution p✓ ( x|ri ) : p✓ ( x ; ri ) / e E i ✓ ( x|Enc ( ri ) ) , ( 4 ) where p✓ ( x|ri ) represents the probability distribution over images given relation ri and Enc ( ri ) denotes a text encoder for relation ri . The most straightforward manner to encode relational scene descriptions is to encode the entire sentence using an existing text encoder , such as CLIP [ 35 ] . However , we find that such an approach can not capture scene relations as shown in Supplement Section A . An issue with such an approach is that the sentence encoder loses or masks the information captured by the relation tokens in ri . To enforce that the underlying relation tokens in ri is effectively encoded , we instead propose to decompose the relation ri into a relation triplet ( r0i , o1i , o2i ) , where r0i is the relation token , e.g . “ below ” , “ to the right of ” , o1i is the description of the first object , and o2i is the description of the second object appeared in ri . Each separate entry in the relation triplet is then separately embedded . Such an encoding scheme encourages the models to encode underlying objects and relations in a scene , enabling us to effectively model the relational distribution . We explored two separate approaches to embed the underlying object descriptions into our relational energy function . CLIP Embedding . One approach we consider is to directly utilize CLIP to obtain the embedding of an object description . Such an approach may potentially enable us to generalize relations in a zero-shot manner to new objects by utilizing CLIP ’ s underlying embedding of the object , but may also hurt learning if the underlying object embedding does not distinctly separate two object descriptions . Random Initialization . Alternatively , we may encode an object description using a learned embedding layer that is trained from scratch . In this approach , we extract a scene embedding by concatenating the learned embeddings of color , shape , material , and size of any two objects , o1i , o2i , and their relation ri . 3.3 Representing Relational Scene Descriptions . Given an underlying scene description R , we represent the underlying probability distribution p ( x|R ) by factorizing it as a product of probabilities over the the underlying relations rk inside R. Given the separate relational energy functions learned in Section 3.2 , this probability p ( x|R ) is proportional to p ( x|R ) = e E k ✓ ( x|R ) / Y k p ( x|rk ) = e P k Ek✓ ( x|Enc ( rk ) ) , ( 5 ) which is a new EBM with underlying energy function E✓ ( x|R ) = P k Ek✓ ( x|Enc ( rk ) ) . The overview of the proposed method is shown in Figure 2 , where E1 , · · · , EK correspond to the K individual relational energy functions Ek✓ ( x|Enc ( rk ) ) .

Summary: The paper introduces an energy-based model to compose separate relations in a factorized manner.  A relational scene description is decomposed into individual relations, and modeled as the product of individual probability distributions across relations. The proposed method is shown to be able to generate images with good quality from the relational descriptions and understand the semantic differences between relational descriptions. Zero-shot evaluation also suggest that the proposed approach can generalize to unseen relational descriptions.