Learning to Compose Visual Relations

1 Introduction . The ability to reason about the component objects and their relations in a scene is key for a wide variety of robotics and AI tasks , such as multistep manipulation planning [ 11 ] , concept learning [ 25 ] , navigation [ 43 ] , and dynamics prediction [ 3 ] . While a large body of work has explored inferring and understanding the underlying objects in a scene , robustly understanding the component relations in a scene remains a challenging task . In this work , we explore how to robustly understand relational scene description ( Figure 1 ) . Naively , one approach towards understanding relational scene descriptions is to utilize existing multi-modal language and vision models . Such an approach has recently achieved great success in DALL-E [ 36 ] and CLIP [ 35 ] , both of which show compelling results on encoding object properties with language . However , when these approaches are instead utilized to encode relations between objects , their performance rapidly deteriorates , as shown in [ 36 ] and which we further illustrate in Figure 7 . We posit that the lack of compositionality in the language encoder prevents it from capturing all the underlying relations in an image . To remedy this issue , we propose instead to factorize the scene description with respect to each individual relation . Separate models are utilized to encode each individual relation , which are then subsequently composed together to represent a relational scene description . The most straightforward ⇤indicates equal contribution 35th Conference on Neural Information Processing Systems ( NeurIPS 2021 ) . approach is to specify distinct regions of an image in which each relation can be located , as well as a composed relation description corresponding to the combination of all these regions . Such an approach has significant drawbacks . In practice , the location of one pair of objects in a relation description may be heavily influenced by the location of objects specified by another relation description . Specifying a priori the exact location of a relation will thus severely hamper the number of possible scenes that can be realized with a given set of relations . Is it possible to factorize relational descriptions of a scene and generate images that incorporate each given relation description simultaneously ? In this work , we propose to represent and factorize individual relations as unnormalized densities using Energy-Based Models . A relational scene description is represented as the product of the individual probability distributions across relations , with each individual relation specifying a separate probability distribution over images . Such a composition enables interactions between multiple relations to be modeled . We show that this resultant framework enables us to reliably capture and generate images with multiple composed relational descriptions . It further enables us to edit images to have a desired set of relations . Finally , by measuring the relative densities assigned to different relational descriptions , we are able to infer the objects and their relations in a scene for downstream tasks , such as image-to-text retrieval and classification . There are three main contributions of our work . First , we present a framework to factorize and compose separate object relations . We show that the proposed framework is able to generate and edit images with multiple composed relations and significantly outperforms baseline approaches . Secondly , we show that our approach is able to infer the underlying relational scene descriptions and is robust enough in understanding semantically equivalent relational scene descriptions . Finally , we show that our approach can generalize to a previously unseen relation description , even if the underlying objects and descriptions are from a separate dataset not seen during training . We believe that such generalization is crucial for a general artificial intelligence system to adapt to the infinite number of variations of the world around it . 2 Related Work . Language Guided Scene Generation . A large body of work has explored scene generation utilizing text descriptions [ 13 , 17 , 20 , 30 , 34 , 36 , 38–40 , 46–48 ] . In contrast to our work , prior work [ 17 , 38 , 47 , 48 ] have focused on generating images given only a limited number of relation descriptions . Recently , [ 36 ] shows compelling results on utilizing text to generate images , but also explicitly state that relational generation was a weakness of the model . In this work , we seek to tackle how we may generate images given an underlying relational description . Visual Relation Understanding . To understand visual relations in a scene , many works applied neural networks to graphs [ 1–3 , 5 , 14 , 15 , 18 , 20 , 28 , 29 , 32 , 37 ] . Raposo et al . [ 37 ] proposed Relation Network ( RN ) to explicitly compute relations from static input scenes while we implicitly encode relational information to generate and edit images based on given relations and objects . Johnson et al . [ 20 ] proposed to condition image generation on relations by utilizing scene graphs which were further explored in [ 1 , 14 , 15 , 18 , 29 , 31 , 44 ] . However , these approaches require excessive supervisions , such as bounding boxes , segmentation masks , etc. , to infer and generate the underlying relations in an image . Such a setting restricts the possible combinations of individual relations in a scene . In our work , we present a method that enables us to generate images given only a relational scene description . Energy Based Models . Our work is related to existing work on energy-based models [ 6 , 8 , 10 , 12 , 22 , 33 , 42 , 45 ] . Most similar to our work is that of [ 7 ] , which proposes a framework of utilizing EBMs to compose several object descriptions together . In contrast , in this work , we study the problem of how we may compose relational descriptions together , an important and challenging task for existing text description understanding systems . 3 Method . Given a training dataset C = { xi , Ri } Ni=1 with N distinct images xi 2 RD and associated relational descriptions Ri , we aim at learning the underlying probability distribution p✓ ( x|R ) — the probability distributions of an image given a corresponding relational description . To represent p✓ ( x|R ) , we split a relational description R into K separate relations { r1 · · · , rK } and model each component relation separately using a probability distribution p✓ ( x|rk ) which is represented as an Energy-Based Model . Our overall scene probability distribution is then modeled by a composition of individual probability distributions for each relation description p✓ ( x|R ) / Q k p✓ ( x|rk ) . In this section , we give an overview of our approach towards factorizing and representing a relational scene description . We first provide a background overview of Energy-Based Models ( EBMs ) in Section 3.1 . We then describe how we may parameterize individual relational probability distributions with EBMs in Section 3.2 . We further describe how we compose relational probability distributions to model a relational scene description in Section 3.3 . Finally , we illustrate downstream applications of our relational scene understanding model in Section 3.4 . 3.1 Energy-Based Models . We model each relational probability distribution utilizing an Energy-Based Model ( EBM ) [ 6 , 26 ] . EBMs are a class of unnormalized probability models , which parameterize a probability distribution p✓ ( x ) utilizing a learned energy function E✓ ( x ) : p✓ ( x ) / e E✓ ( x ) . ( 1 ) EBMs are typically trained utilizing contrastive divergence [ 16 ] , where energies of training datapoints are decreased and energies of sampled data points from p✓ ( x ) are increased . We adopt the training code and models from [ 8 ] to train our EBMs . To generate samples from an EBM p✓ ( x ) , we utilize MCMC sampling on the underlying distribution , and Langevin dynamics , which refines a data sample iteratively from a random noise : x̃m = x̃m 1 2 rxE✓ ( x̃m 1 ) + ! m , ! m ⇠ N ( 0 , ) ( 2 ) where m refers to the iteration and is the step size , utilizing the gradient of the energy function with respect to the input sample rxE✓ , and ! m is sampled from a Gaussian noise . EBMs enable us to naturally compose separate probability distributions together [ 7 ] . In particular , given a set of independent marginal distributions { pi✓ ( x ) } , the joint probability distribution is represented as : Y i pi✓ ( x ) / e P i E i ✓ ( x ) , ( 3 ) where we utilize Langevin dynamics to sample from the resultant joint probability distribution . 3.2 Learning Relational Energy Functions . Given a scene relation ri , described using a set of words { w1i , . . . wni } , we seek to learn a conditional EBM to model the underlying probability distribution p✓ ( x|ri ) : p✓ ( x ; ri ) / e E i ✓ ( x|Enc ( ri ) ) , ( 4 ) where p✓ ( x|ri ) represents the probability distribution over images given relation ri and Enc ( ri ) denotes a text encoder for relation ri . The most straightforward manner to encode relational scene descriptions is to encode the entire sentence using an existing text encoder , such as CLIP [ 35 ] . However , we find that such an approach can not capture scene relations as shown in Supplement Section A . An issue with such an approach is that the sentence encoder loses or masks the information captured by the relation tokens in ri . To enforce that the underlying relation tokens in ri is effectively encoded , we instead propose to decompose the relation ri into a relation triplet ( r0i , o1i , o2i ) , where r0i is the relation token , e.g . “ below ” , “ to the right of ” , o1i is the description of the first object , and o2i is the description of the second object appeared in ri . Each separate entry in the relation triplet is then separately embedded . Such an encoding scheme encourages the models to encode underlying objects and relations in a scene , enabling us to effectively model the relational distribution . We explored two separate approaches to embed the underlying object descriptions into our relational energy function . CLIP Embedding . One approach we consider is to directly utilize CLIP to obtain the embedding of an object description . Such an approach may potentially enable us to generalize relations in a zero-shot manner to new objects by utilizing CLIP ’ s underlying embedding of the object , but may also hurt learning if the underlying object embedding does not distinctly separate two object descriptions . Random Initialization . Alternatively , we may encode an object description using a learned embedding layer that is trained from scratch . In this approach , we extract a scene embedding by concatenating the learned embeddings of color , shape , material , and size of any two objects , o1i , o2i , and their relation ri . 3.3 Representing Relational Scene Descriptions . Given an underlying scene description R , we represent the underlying probability distribution p ( x|R ) by factorizing it as a product of probabilities over the the underlying relations rk inside R. Given the separate relational energy functions learned in Section 3.2 , this probability p ( x|R ) is proportional to p ( x|R ) = e E k ✓ ( x|R ) / Y k p ( x|rk ) = e P k Ek✓ ( x|Enc ( rk ) ) , ( 5 ) which is a new EBM with underlying energy function E✓ ( x|R ) = P k Ek✓ ( x|Enc ( rk ) ) . The overview of the proposed method is shown in Figure 2 , where E1 , · · · , EK correspond to the K individual relational energy functions Ek✓ ( x|Enc ( rk ) ) .

Summary: This paper demonstrates the potential usage of our models across tasks of compositional multi-object relational image generation, editing, and even generalization on unseen datasets. The proposed model represents each relation as an unnormalized density (an energy-based model), enabling it to compose separate relations in a factorized manner. The experimental results significantly outperform baseline approaches and even can generalize to a previously unseen relation description.