Language-Mediated, Object-Centric Representation Learning

1 INTRODUCTION . Cognitive studies show that human infants develop object individuation skill from diverse sources of information : spatial-temporal information , object property information , and language ( Xu , 1999 ; 2007 ; Westermann & Mareschal , 2014 ) . Specifically , young infants develop object-based attention that disentangles the motion and location of objects from their visual appearance features . Later on , they can leverage the knowledge acquired through word learning to solve the problem of object individuation : words provide clues about object identity and type . The general picture from cognitive science is that object perception and language co-develop in support of one another ( Bloom , 2002 ) . Our long-term goal is to endow machines with similar abilities . In this paper , we focus on how language may support object segmentation . Many recent works have studied the problem of unsupervised object representation learning , though without language . As an example , factorized , object-centric scene representations have been used in various kinds of prediction ( Goel et al. , 2018 ) , reasoning ( Yi et al. , 2018 ; Mao et al. , 2019 ) , and planning tasks ( Veerapaneni et al. , 2020 ) , but they have not considered the role of language and how it may help object representation learning . As a concrete example , consider the input images shown in Fig . 1 and the paired questions . From language , we can learn to associate concepts , such as black , pan , and legs , with the referred object ’ s visual appearance . Further , language provides cues about how an input scene should be segmented into individual objects : a wrong parsing of the input scene will lead to an incorrect answer to the question . We can learn from such failure that the handle belongs to the frying pan ( Fig . 1a ) and the chair has four legs ( Fig . 1b ) . Motivated by these observations , we propose a computational learning paradigm , Language-mediated , Object-centric Representation Learning ( LORL ) , associating learned object-centric representations to their visual appearance ( masks ) in images , and to concepts—words for object properties such as color , shape , and material—as provided in language . Here the language input can be either descriptive sentences or question-answer pairs . LORL requires no annotations on object masks , categories , or properties during the learning process . In LORL , four modules are jointly trained . The first is an image encoder , learning to encode an image into factorized , object-centric representations . The second is an image decoder , learning to reconstruct masks for individual objects from the learned representations by reconstructing the input . These two modules share the same formulation as recent unsupervised object segmentation research , and may be realized as methods such as MONet ( Burgess et al. , 2019 ) and Slot Attention ( Locatello et al. , 2020 ) . The third module in LORL is a pre-trained semantic parser that translates the input sentence into a semantic , executable program , where each concept ( i.e. , words for object properties such as ‘ red ’ ) is associated with a vector space embedding . Finally , the last module , a neural-symbolic program executor , takes the object-centric representation from Module 1 , intermediate representations from Module 2 , and concept embeddings and the semantic program from Module 3 as input , and outputs an answer if the language input is a question , or TRUE/FALSE if it ’ s a descriptive sentence . The correctness of the executor ’ s output and the quality of reconstructed images ( as output of Module 2 ) are the two supervisory signals we use to jointly train Modules 1 , 2 , and 4 . We integrate the proposed LORL with state-of-the-art unsupervised segmentation methods , MONet ( Burgess et al. , 2019 ) and Slot Attention ( Locatello et al. , 2020 ) . The evaluation is based on two datasets : ShopVRB ( Nazarczuk & Mikolajczyk , 2020 ) contains images of daily objects and question-answer pairs ; PartNet ( Mo et al. , 2019 ) contains images of furniture with hierarchical structure , supplemented by descriptive sentences we collected ourselves . We show that LORL consistently improves existing methods on unsupervised object segmentation , much more likely to group different parts of a single object into a single mask . We further analyze the object-centric representations learned by LORL . In LORL , conceptually similar objects ( e.g . objects of similar shapes ) appear to be clustered in the embedding space . Moreover , experiments demonstrate that the learned concepts can be used in new tasks , such as visual grounding of referring expressions , without any additional fine-tuning . 2 RELATED WORK . Unsupervised object representation learning . Given an input image , unsupervised object representation learning methods segment objects in the scene and build an object-centric representation for them . A mainstream approach has focused on using compositional generative scene models that decompose the scene as a mixture of component images ( Greff et al. , 2016 ; Eslami et al. , 2016 ; Greff et al. , 2017 ; Burgess et al. , 2019 ; Engelcke et al. , 2020 ; Greff et al. , 2019 ; Locatello et al. , 2020 ) . In general , these models use an encoder-decoder architecture : the image encoder encodes the input image into a set of latent object representations , which are fed into the image decoder to reconstruct the image . Specifically , Greff et al . ( 2019 ) ; Burgess et al . ( 2019 ) ; Engelcke et al . ( 2020 ) use recurrent encoders that iteratively localize and encode objects in the scene . Another line of research ( Eslami et al. , 2016 ; Crawford & Pineau , 2019 ; Kosiorek et al. , 2018 ; Stelzner et al. , 2019 ; Lin et al. , 2020 ) leverages object locality to attend to different local patches of the image . These models often use a pixel-level reconstruction loss . In contrast , we propose to explore how language , in addition to visual observations , may contribute to object-centric representation learning . There have also been work that uses other types of supervisions , such as dynamic prediction ( Kipf et al. , 2020 ; Bear et al. , 2020 ) and multi-view consistency ( Prabhudesai et al. , 2020 ) . In this paper , we focus on unsupervised learning of object-centric representations from static images and language . Visual concept learning . Learning visual concepts from language and other forms of supervisions provides useful representations for various downstream tasks , such as cross-modal retrieval ( Wu et al. , 2019 ) , visual-question answering ( Yi et al. , 2018 ; Mao et al. , 2019 ) , and scene manipulation ( Prabhudesai et al. , 2020 ) . Previous work has been focusing on various types of representations ( Ren et al. , 2016 ; Wu et al. , 2017 ) , training algorithms ( Faghri et al. , 2018 ) and supervisions ( Johnson et al. , 2016 ; Yang et al. , 2018 ) . In this paper , we focus on learning visual concepts that can be grounded on object-centric representations . Recent works on object-centric grounding of visual concepts ( Wu et al. , 2017 ; Mao et al. , 2019 ; Prabhudesai et al. , 2020 ) have shown great success in achieving high performance in downstream tasks and strong generalization from a small amount of data . However , these methods assume pre-trained object detectors to generate object proposals in the scene . In contrast , our LORL learns to individuate objects and associates concepts with the learned object-centric representations without any annotations on object segmentation masks or properties . 3 PRELIMINARIES . Before delving into our language-mediated object-centric representation learning paradigm , we first discuss a general formulation that unifies multiple concurrent unsupervised object representation learning methods and a neuro-symbolic framework for learning visual concepts from language . 3.1 UNSUPERVISED OBJECT-CENTRIC REPRESENTATION LEARNING . Given an image I , a typical unsupervised object representation learning model will decompose the scene into a series of slot profiles { ( z1 , x1 , m1 ) , . . . , ( zK , xK , mK ) } , where each slot profile is expected to represent an object ( or nothing , as the number of slots may be greater than the actual number of objects in the scene ) . Here zi is the object feature , xi is the object image , and mi is the object mask specifying its location in the scene . In our paper , we focus on two recent models , MONet ( Burgess et al. , 2019 ) and Slot Attention ( Locatello et al. , 2020 ) . MONet uses a recurrent spatial attention network ( Ronneberger et al. , 2015 ) to segment out objects in the scene , and adopts a variational autoencoder ( Kingma & Welling , 2014 ) to encode objects as well as reconstructing object images for self-supervision . At a very high level , its objective function is calculated as L = ∥∥∥∥∥ ( K∑ k=1 mkxk ) − I ∥∥∥∥∥ 2 2 + β · K∑ k=1 KL ( zk ) , ( 1 ) where the first term is a pixel-wise L2 reconstruction loss and the second term computes the KL divergence between the distribution of zk ’ s and a prior Gaussian distribution . Slot Attention uses a transformer-like attention network ( Vaswani et al. , 2017 ) to extract object features , and decode them with convolutional neural networks to component images and object masks . The model is trained by the same reconstruction loss in the form of L2-norm : L = ∥∥∥∥∥ ( K∑ k=1 mkxk ) − I ∥∥∥∥∥ 2 2 . ( 2 ) 3.2 NEURO-SYMBOLIC CONCEPT LEARNING . The neuro-symbolic concept learner ( NS-CL ; Mao et al. , 2019 ) learns visual concepts by looking at images and reading paired questions and answers . NS-CL takes a set of segmented objects in a given image as its input , extracts their visual features with a ResNet ( He et al. , 2015 ) , translates the input question into an executable program by a semantic parser , and executes the program based on the object-centric representation to answer the question . The key idea of NS-CL is to explicitly represent individual concepts in natural language ( colors , shapes , spatial relationships , etc . ) as vector space embeddings , and associate them with the object embeddings . NS-CL answers the input question by executing the program based on the object-centric representation . For example , in order to query the name of the white object in Fig . 2 , NS-CL first filters out the object by computing the cosine similarity between the concept white and individual object representations , which produces a “ mask ” vector where each entry denotes the probability that an object has been selected . The output “ mask ” on the objects will be fed into the next module and the execution will continue . The last query operation will produce the answer to the question . The vector embeddings of individual objects and the concepts will be jointly trained based on language supervision . 4 LANGUAGE-MEDIATED , OBJECT-CENTRIC REPRESENTATION LEARNING . Marrying the ideas of unsupervised object-centric representation learning and neuro-symbolic concept learning , we are able to learn an object-centric representation using both visual and language supervision . Fig . 2 shows an overview of Language-mediated , Object-centric Representation Learning ( LORL ) . In LORL , four modules are optimized jointly : an image encoder , an image decoder , a semantic parser , and a neuro-symbolic program executor . Image encoder . Given an input image , we first use the image encoder ( Fig . 2a ) to individuate objects in the scene and extract an object-centric scene representation . It takes the input image as its input , individuates objects in the scene , and produces a collection of latent slot embeddings { zi } . Image decoder . The decoder ( Fig . 2b ) takes the object-centric representation produced by the image encoder and produces a 3-tuple for each individual slot ( xk , mk , sk ) , where xk reconstructs the RGB image of the slot , mk reconstructs the mask , and sk ∈ [ 0 , 1 ] is a scalar indicating the objectness of the slot . That is , whether k-th slot corresponds to a single object in the scene . Here , we have extended the general pipeline we described in Section 3.1 with an objectness indicator . It serves dual purposes . First , it weights each reconstructed component image while generating the reconstructed image . Mathematically , the reconstructed image I ′ is computed as : I ′ = ∑K k=1 sk · ( mkxk ) . Second , it mediates the output of all filter operations in the program executor . In this paper , we will experiment with two image encoder-decoder options : MONet ( Burgess et al. , 2019 ) and Slot Attention ( Locatello et al. , 2020 ) . They are both compatible with the learning paradigm described above . For both models , we use a single linear layer to predict the objectness score for each slot on top of the second-last layer of their image decoders . Semantic parser . A pre-trained semantic parser ( Fig . 2c ) will translate the input question into an executable program composed of primitive operations , such as filter , which filters out objects with certain concepts and query , which queries the attribute of the input object . We use roughly the same domain-specific language ( DSL ) for representing programs as CLEVR ( Johnson et al. , 2017a , see also Appendix A for details ) . All concepts that appear in the program , such as white , are associated with distinct , learnable concept embedding vectors . Neuro-symbolic program executor . The program executor ( Fig . 2d ) takes the object-centric representation from the image encoder { zk } , the objectness score { sk } from the image decoder , the concept embeddings and the program generated by the semantic parser as its input . It executes the program based on the visual and concept representations to answer the question . The original program executor in NS-CL ( Section 3.2 ) assumes a pre-trained object detector for generating object proposals . In LORL , we associate each object representation with an objectness score sk . Recall that a filter operation in NS-CL produces a mask vector indicating whether an object has been selected . Here , we mediate the output of an filter ( c ) operation as min ( sk , filter ( c ) ) . Intuitively , a slot will be selected only if , first , it has concept c and , second , it corresponds to a single object in the scene . Training paradigm . During training , we jointly optimize the image encoder , the image decoder , and the concept embeddings . They are trained by minimizing the loss L : L = α · Lperception + β · Lreasoning . For MONet-based image encoder-decoder , we use Equation 1 as the perception loss Lperception , while for Slot Attention-based encoder-decoder , we use Equation 2 . The neuro-symbolic program executor produces a distribution over candidate answers to the input question . We use the cross-entropy loss between the predicted answer and the ground truth answer as Lreasoning . We use a three-stage training paradigm in LORL . First , we train the model with only visual inputs with Lperception for N1 epochs . Next , we fix the image encoder and the image decoder , and optimize the concept embeddings with the loss term Lreasoning for N2 epochs . During this second stage , the image encoder and the decoder can already produce descent object segmentation results . Finally , we jointly optimize all three modules for N3 epochs . We provide detailed information about the hyperparameters for different models in Appendix B .

Summary: The paper proposes a framework for object-centric representation learning with additional language supervision such as e.g. questions and answers, denoted as Language-mediated, Object-centric Representation Learning (LORL). The authors combine two ideas from prior work, the unsupervised object-centric representation learning and the neural-symbolic concept learning, in one architecture. The model obtains object representations by learning to reconstruct the input image (as in MONet and Slot Attention). The learned representations are used as input to the neural-symbolic program executor, which learns to answer questions about objects. The entire model is trained in three stages: first the reconstruction objective, then the QA objective, and, finally, jointly. Experiments on two datasets demonstrate that the obtained object segmentations have better quality that those of the original unsupervised models. The learned representations are also shown to be effective in several other down-stream tasks.