Language-Mediated, Object-Centric Representation Learning

1 INTRODUCTION . Cognitive studies show that human infants develop object individuation skill from diverse sources of information : spatial-temporal information , object property information , and language ( Xu , 1999 ; 2007 ; Westermann & Mareschal , 2014 ) . Specifically , young infants develop object-based attention that disentangles the motion and location of objects from their visual appearance features . Later on , they can leverage the knowledge acquired through word learning to solve the problem of object individuation : words provide clues about object identity and type . The general picture from cognitive science is that object perception and language co-develop in support of one another ( Bloom , 2002 ) . Our long-term goal is to endow machines with similar abilities . In this paper , we focus on how language may support object segmentation . Many recent works have studied the problem of unsupervised object representation learning , though without language . As an example , factorized , object-centric scene representations have been used in various kinds of prediction ( Goel et al. , 2018 ) , reasoning ( Yi et al. , 2018 ; Mao et al. , 2019 ) , and planning tasks ( Veerapaneni et al. , 2020 ) , but they have not considered the role of language and how it may help object representation learning . As a concrete example , consider the input images shown in Fig . 1 and the paired questions . From language , we can learn to associate concepts , such as black , pan , and legs , with the referred object ’ s visual appearance . Further , language provides cues about how an input scene should be segmented into individual objects : a wrong parsing of the input scene will lead to an incorrect answer to the question . We can learn from such failure that the handle belongs to the frying pan ( Fig . 1a ) and the chair has four legs ( Fig . 1b ) . Motivated by these observations , we propose a computational learning paradigm , Language-mediated , Object-centric Representation Learning ( LORL ) , associating learned object-centric representations to their visual appearance ( masks ) in images , and to concepts—words for object properties such as color , shape , and material—as provided in language . Here the language input can be either descriptive sentences or question-answer pairs . LORL requires no annotations on object masks , categories , or properties during the learning process . In LORL , four modules are jointly trained . The first is an image encoder , learning to encode an image into factorized , object-centric representations . The second is an image decoder , learning to reconstruct masks for individual objects from the learned representations by reconstructing the input . These two modules share the same formulation as recent unsupervised object segmentation research , and may be realized as methods such as MONet ( Burgess et al. , 2019 ) and Slot Attention ( Locatello et al. , 2020 ) . The third module in LORL is a pre-trained semantic parser that translates the input sentence into a semantic , executable program , where each concept ( i.e. , words for object properties such as ‘ red ’ ) is associated with a vector space embedding . Finally , the last module , a neural-symbolic program executor , takes the object-centric representation from Module 1 , intermediate representations from Module 2 , and concept embeddings and the semantic program from Module 3 as input , and outputs an answer if the language input is a question , or TRUE/FALSE if it ’ s a descriptive sentence . The correctness of the executor ’ s output and the quality of reconstructed images ( as output of Module 2 ) are the two supervisory signals we use to jointly train Modules 1 , 2 , and 4 . We integrate the proposed LORL with state-of-the-art unsupervised segmentation methods , MONet ( Burgess et al. , 2019 ) and Slot Attention ( Locatello et al. , 2020 ) . The evaluation is based on two datasets : ShopVRB ( Nazarczuk & Mikolajczyk , 2020 ) contains images of daily objects and question-answer pairs ; PartNet ( Mo et al. , 2019 ) contains images of furniture with hierarchical structure , supplemented by descriptive sentences we collected ourselves . We show that LORL consistently improves existing methods on unsupervised object segmentation , much more likely to group different parts of a single object into a single mask . We further analyze the object-centric representations learned by LORL . In LORL , conceptually similar objects ( e.g . objects of similar shapes ) appear to be clustered in the embedding space . Moreover , experiments demonstrate that the learned concepts can be used in new tasks , such as visual grounding of referring expressions , without any additional fine-tuning . 2 RELATED WORK . Unsupervised object representation learning . Given an input image , unsupervised object representation learning methods segment objects in the scene and build an object-centric representation for them . A mainstream approach has focused on using compositional generative scene models that decompose the scene as a mixture of component images ( Greff et al. , 2016 ; Eslami et al. , 2016 ; Greff et al. , 2017 ; Burgess et al. , 2019 ; Engelcke et al. , 2020 ; Greff et al. , 2019 ; Locatello et al. , 2020 ) . In general , these models use an encoder-decoder architecture : the image encoder encodes the input image into a set of latent object representations , which are fed into the image decoder to reconstruct the image . Specifically , Greff et al . ( 2019 ) ; Burgess et al . ( 2019 ) ; Engelcke et al . ( 2020 ) use recurrent encoders that iteratively localize and encode objects in the scene . Another line of research ( Eslami et al. , 2016 ; Crawford & Pineau , 2019 ; Kosiorek et al. , 2018 ; Stelzner et al. , 2019 ; Lin et al. , 2020 ) leverages object locality to attend to different local patches of the image . These models often use a pixel-level reconstruction loss . In contrast , we propose to explore how language , in addition to visual observations , may contribute to object-centric representation learning . There have also been work that uses other types of supervisions , such as dynamic prediction ( Kipf et al. , 2020 ; Bear et al. , 2020 ) and multi-view consistency ( Prabhudesai et al. , 2020 ) . In this paper , we focus on unsupervised learning of object-centric representations from static images and language . Visual concept learning . Learning visual concepts from language and other forms of supervisions provides useful representations for various downstream tasks , such as cross-modal retrieval ( Wu et al. , 2019 ) , visual-question answering ( Yi et al. , 2018 ; Mao et al. , 2019 ) , and scene manipulation ( Prabhudesai et al. , 2020 ) . Previous work has been focusing on various types of representations ( Ren et al. , 2016 ; Wu et al. , 2017 ) , training algorithms ( Faghri et al. , 2018 ) and supervisions ( Johnson et al. , 2016 ; Yang et al. , 2018 ) . In this paper , we focus on learning visual concepts that can be grounded on object-centric representations . Recent works on object-centric grounding of visual concepts ( Wu et al. , 2017 ; Mao et al. , 2019 ; Prabhudesai et al. , 2020 ) have shown great success in achieving high performance in downstream tasks and strong generalization from a small amount of data . However , these methods assume pre-trained object detectors to generate object proposals in the scene . In contrast , our LORL learns to individuate objects and associates concepts with the learned object-centric representations without any annotations on object segmentation masks or properties . 3 PRELIMINARIES . Before delving into our language-mediated object-centric representation learning paradigm , we first discuss a general formulation that unifies multiple concurrent unsupervised object representation learning methods and a neuro-symbolic framework for learning visual concepts from language . 3.1 UNSUPERVISED OBJECT-CENTRIC REPRESENTATION LEARNING . Given an image I , a typical unsupervised object representation learning model will decompose the scene into a series of slot profiles { ( z1 , x1 , m1 ) , . . . , ( zK , xK , mK ) } , where each slot profile is expected to represent an object ( or nothing , as the number of slots may be greater than the actual number of objects in the scene ) . Here zi is the object feature , xi is the object image , and mi is the object mask specifying its location in the scene . In our paper , we focus on two recent models , MONet ( Burgess et al. , 2019 ) and Slot Attention ( Locatello et al. , 2020 ) . MONet uses a recurrent spatial attention network ( Ronneberger et al. , 2015 ) to segment out objects in the scene , and adopts a variational autoencoder ( Kingma & Welling , 2014 ) to encode objects as well as reconstructing object images for self-supervision . At a very high level , its objective function is calculated as L = ∥∥∥∥∥ ( K∑ k=1 mkxk ) − I ∥∥∥∥∥ 2 2 + β · K∑ k=1 KL ( zk ) , ( 1 ) where the first term is a pixel-wise L2 reconstruction loss and the second term computes the KL divergence between the distribution of zk ’ s and a prior Gaussian distribution . Slot Attention uses a transformer-like attention network ( Vaswani et al. , 2017 ) to extract object features , and decode them with convolutional neural networks to component images and object masks . The model is trained by the same reconstruction loss in the form of L2-norm : L = ∥∥∥∥∥ ( K∑ k=1 mkxk ) − I ∥∥∥∥∥ 2 2 . ( 2 ) 3.2 NEURO-SYMBOLIC CONCEPT LEARNING . The neuro-symbolic concept learner ( NS-CL ; Mao et al. , 2019 ) learns visual concepts by looking at images and reading paired questions and answers . NS-CL takes a set of segmented objects in a given image as its input , extracts their visual features with a ResNet ( He et al. , 2015 ) , translates the input question into an executable program by a semantic parser , and executes the program based on the object-centric representation to answer the question . The key idea of NS-CL is to explicitly represent individual concepts in natural language ( colors , shapes , spatial relationships , etc . ) as vector space embeddings , and associate them with the object embeddings . NS-CL answers the input question by executing the program based on the object-centric representation . For example , in order to query the name of the white object in Fig . 2 , NS-CL first filters out the object by computing the cosine similarity between the concept white and individual object representations , which produces a “ mask ” vector where each entry denotes the probability that an object has been selected . The output “ mask ” on the objects will be fed into the next module and the execution will continue . The last query operation will produce the answer to the question . The vector embeddings of individual objects and the concepts will be jointly trained based on language supervision . 4 LANGUAGE-MEDIATED , OBJECT-CENTRIC REPRESENTATION LEARNING . Marrying the ideas of unsupervised object-centric representation learning and neuro-symbolic concept learning , we are able to learn an object-centric representation using both visual and language supervision . Fig . 2 shows an overview of Language-mediated , Object-centric Representation Learning ( LORL ) . In LORL , four modules are optimized jointly : an image encoder , an image decoder , a semantic parser , and a neuro-symbolic program executor . Image encoder . Given an input image , we first use the image encoder ( Fig . 2a ) to individuate objects in the scene and extract an object-centric scene representation . It takes the input image as its input , individuates objects in the scene , and produces a collection of latent slot embeddings { zi } . Image decoder . The decoder ( Fig . 2b ) takes the object-centric representation produced by the image encoder and produces a 3-tuple for each individual slot ( xk , mk , sk ) , where xk reconstructs the RGB image of the slot , mk reconstructs the mask , and sk ∈ [ 0 , 1 ] is a scalar indicating the objectness of the slot . That is , whether k-th slot corresponds to a single object in the scene . Here , we have extended the general pipeline we described in Section 3.1 with an objectness indicator . It serves dual purposes . First , it weights each reconstructed component image while generating the reconstructed image . Mathematically , the reconstructed image I ′ is computed as : I ′ = ∑K k=1 sk · ( mkxk ) . Second , it mediates the output of all filter operations in the program executor . In this paper , we will experiment with two image encoder-decoder options : MONet ( Burgess et al. , 2019 ) and Slot Attention ( Locatello et al. , 2020 ) . They are both compatible with the learning paradigm described above . For both models , we use a single linear layer to predict the objectness score for each slot on top of the second-last layer of their image decoders . Semantic parser . A pre-trained semantic parser ( Fig . 2c ) will translate the input question into an executable program composed of primitive operations , such as filter , which filters out objects with certain concepts and query , which queries the attribute of the input object . We use roughly the same domain-specific language ( DSL ) for representing programs as CLEVR ( Johnson et al. , 2017a , see also Appendix A for details ) . All concepts that appear in the program , such as white , are associated with distinct , learnable concept embedding vectors . Neuro-symbolic program executor . The program executor ( Fig . 2d ) takes the object-centric representation from the image encoder { zk } , the objectness score { sk } from the image decoder , the concept embeddings and the program generated by the semantic parser as its input . It executes the program based on the visual and concept representations to answer the question . The original program executor in NS-CL ( Section 3.2 ) assumes a pre-trained object detector for generating object proposals . In LORL , we associate each object representation with an objectness score sk . Recall that a filter operation in NS-CL produces a mask vector indicating whether an object has been selected . Here , we mediate the output of an filter ( c ) operation as min ( sk , filter ( c ) ) . Intuitively , a slot will be selected only if , first , it has concept c and , second , it corresponds to a single object in the scene . Training paradigm . During training , we jointly optimize the image encoder , the image decoder , and the concept embeddings . They are trained by minimizing the loss L : L = α · Lperception + β · Lreasoning . For MONet-based image encoder-decoder , we use Equation 1 as the perception loss Lperception , while for Slot Attention-based encoder-decoder , we use Equation 2 . The neuro-symbolic program executor produces a distribution over candidate answers to the input question . We use the cross-entropy loss between the predicted answer and the ground truth answer as Lreasoning . We use a three-stage training paradigm in LORL . First , we train the model with only visual inputs with Lperception for N1 epochs . Next , we fix the image encoder and the image decoder , and optimize the concept embeddings with the loss term Lreasoning for N2 epochs . During this second stage , the image encoder and the decoder can already produce descent object segmentation results . Finally , we jointly optimize all three modules for N3 epochs . We provide detailed information about the hyperparameters for different models in Appendix B .

Summary: This paper proposes to combine the neuro-symbolic concept learner for visual reasoning from language (NS-CL; Mao et al., 2019) with recent unsupervised approaches to learning object-centric representations such as MONet (Burgess et al., 2019) and Slot-Attention (Locatello et al., 2020). While NS-CL normally relies on pre-trained object-detectors (in a supervised fashion) to extract visual representations, the proposed combination (dubbed LORL) use MONet or Slot Attention for this. By additionally back-propagating error signals from language-driven visual reasoning tasks obtained via NS-CL into MONet/Slot-Attention, it is shown how LORL is better able at learning object-centric representations and perform instance segmentation.