Learning to encode spatial relations from natural language

1 INTRODUCTION . Through natural language humans are able to evoke representations in each others ’ minds . When one person describes their view of a scene their interlocutors are able to form a mental model of the situation and imagine how the objects described would look from different viewpoints . At the simplest level , if someone in front of you describes an object as situated to their left , you understand that it is to your right . Current models for embedding the meaning of natural language are not able to achieve such viewpoint integration . In fact , as shown by Gershman & Tenenbaum ( 2015 ) , distributed representations of natural language extracted from monolingual corpora fail to understand semantic equivalences such as that ‘ A is in front of B ’ describes the same situation as ‘ B is behind A. ’ . In this paper we seek to understand how to build representations that can capture these invariances , an important first step toward a human-level ability to perform abstract spatial reasoning . We hypothesize that to capture spatial invariance in language , representation learning must be grounded in vision . To this end , we introduce a dataset and task designed for the specific purpose of learning and evaluating representations of spatial language . We couple 3D scenes with language descriptions of several different viewpoints of a scene , facilitating the learning of spatial relations , spatial paraphrases and rotational equivalences . Unlike visual question answering tasks such as CLEVR ( Johnson et al. , 2017 ) , this task requires systems to read descriptions of a scene , each paired with a camera position , and create an internal model of the described world . They must then demonstrate the efficacy of their model by generating visual representation of the world conditioned on a viewpoint . Training a multi-modal generative model on this dataset , we can learn scene representations in an unsupervised manner . Our approach is to train the model to translate ( multiple ) natural language descriptions into images , taking into account the viewpoints both of the input ( text ) and the output ( image ) . Solving this task requires understanding the language of spatial relations . Finding that the model performs well on the image generation task , we analyse the constituent parts of the model , solve the challenge posed by Gershman & Tenenbaum ( 2015 ) , and provide a roadmap to incorporating spatial understanding in language models more generally . The main contributions of this paper are : • We introduce two new datasets which contain language descriptions ( synthetic and natural ) and visual renderings of 3D scenes from multiple viewpoints . • We use those datasets to learn language representations of a scene grounded in the visual domain . We show that the model is able to form a representation of the 3D scene from language input . • We analyse these representations and demonstrate that they encode spatial relations and their corresponding invariances in a way consistent with human understanding , and that these language-based invariances arise at the level of the language encoder , with additional viewpoint-invariance as an emergent property of the overall system . 2 RELATED WORK . We consider two lines of related work : previous efforts towards understanding spatial relations across NLP , Computer Vision and Robotics , and more broadly work in visual question answering where explicit visual reasoning is required . Spatial natural language is notoriously ambiguous and difficult to process computationally . Even seemingly simple prepositions like behind are impossible to describe categorically and require a graded treatment ( e.g . how far may a person move from behind a tree before we no longer describe them as such ) . Likewise , understanding the difference between allocentric and egocentric reference frames can easily confound ( Vogel & Jurafsky , 2010 ; Kranjec et al. , 2014 ) . The lexicalisation of spatial concepts can vary widely across languages and cultures ( Haun et al. , 2011 ) , with added complexity in how humans represent geometric properties when describing spatial experiences ( Landau & Jackendoff , 1993 ) and in the layering of locatives ( Kracht , 2002 ) . While there has been considerable research into the relationship between categorical spatial relation processing , perception and language understanding in humans , there are few definite conclusions on how to encode this relationship computationally ( Kosslyn , 1987 ; Johnson , 1990 ; Kosslyn et al. , 1998 ; Haun et al. , 2011 ) . In the field of Natural Language Processing , work on spatial relations has focused on the extraction of spatial descriptions from text and their mapping into formal symbolic languages ( Kordjamshidi et al. , 2012a ; b ) , with numerous such annotation schemes and methods proposed ( Shen et al. , 2009 ; Bateman et al. , 2010 ; Rouhizadeh et al. , 2011 ) . In other related work , Vogel & Jurafsky ( 2010 ) grounded spatial language in an environment through reinforcement learning . The action- and statespaces of the environment however were simplified to the extent that spatial relations were represented by tuples such as west-of ( A , B ) , turning this again into a semantic parsing problem more than anything else . Meanwhile research on visualising spatial descriptions has predominantly employed heavily hand engineered representations that do not offer the generic cross task advantages of distributed representations ( Chang et al. , 2014 ; Hassani & Lee , 2016 ) . In robotics , spatial relations feature in research on human-robot interaction such as ( Tellex et al. , 2011 ) . Here , similar to Vogel & Jurafsky ( 2010 ) , spatial description clauses are translated into grounding graphs , with objects represented by bounding boxes and relations by binary , geometric features such as supports ( x , y ) . There have been a number of datasets that bear some similarity to the dataset produced as part of this work . The SHAPES ( Andreas et al. , 2016 ) and CLEVR datasets ( Johnson et al. , 2017 ) contain images and ( synthetic ) language questions that test visual reasoning . CLEVR in particular uses a form of 3D rendering that produces images similar to the ones used here . While on the surface , these tasks may look similar , it is important to note that this paper is not about visual question answering , but rather about the understanding and modelling of spatial relations . This is achieved by making available multiple viewpoints of the same scene and presenting scenes in viewpoint-specific formats . The key results in this work concern the text representations learnt alongside the visual ones rather than the mapping from one to the other . 3 DATASET OF VISUALLY GROUNDED SCENE DESCRIPTIONS . We construct virtual scenes with multiple views each presented in multiple modalities : image , and synthetic or natural language descriptions ( see Figure 1 ) . Each scene consists of two or three objects placed on a square walled room , and for each of the 10 camera viewpoints we render a 3D view of the scene as seen from that viewpoint as well as a synthetically generated description of the scene . For natural language we selected a random subset of these scenes and had humans describe a given scene and viewpoint based on the rendered image . Details on data generation are found in Appendix B . We targeted a fairly low visual complexity of the rendered scenes since this factor is orthogonal to the spatial relationships we want to learn . The generated scenes still allow for large linguistic variety ( see Table 1 ) . We publish the datasets described in this paper at https : //anonymous . Synthetic language data : We generated a dataset of 10 million 3D scenes . Each scene contains two or three coloured 3D objects and light grey walls and floor . The language descriptions are generated programmatically , taking into account the underlying scene graph and camera coordinates so as to describe the spatial arrangement of the objects as seen from each viewpoint . Natural language data : We generated further scenes and used Amazon Mechanical Turk to collect natural language descriptions . We asked annotators to describe the room in an image as if they were describing the image to a friend who needs to draw the image without seeing it . We asked for a short or a few sentence description that describes object shapes , colours , relative positions , and relative sizes . We provided the list of object names together with only two examples of descriptions , to encourage diversity , while focusing the descriptions on the spatial relations of the objects . The annotators annotated 6,604 scenes with 10 descriptions each , one for each view . 4 SCENE ENCODING TASK & MODEL . We use this dataset in a scene encoding experiment , where the task is to draw an image given a number of written descriptions of a scene . Considering our overarching goal of learning to encode spatial language from text , this task has multiple desirable properties . Having multiple descriptions per scene encourages learning paraphrases , and particularly those of spatial language . By combining input descriptions with viewpoints and requiring the model to generate outputs from different perspectives , this is reinforced , and in addition learning of viewpoint invariant representations is encouraged . This is further encouraged by setting up the task in such a way that the encoder does not have access to the output viewpoint , which puts pressure on the model to encode the descriptions into a viewpoint invariant representation . For the experiments presented in this paper , we adapted the model by Eslami et al . ( 2018 ) , the main difference to their model being the use of language encoders , rather than visual ones . While a number of models could plausibly be used for the work presented here , their setup particularly lent itself to integrating information from multiple viewpoints . This is useful as we want to learn a viewpoint invariant representation for a single scene . We provide full details of the model implementations in Appendix A , and refer to the model as the Spatial Language Interpretation Model ( SLIM ) , when in need of an acronym . On a high level , the model consists of two parts . First an encoder combines all input descriptions and their viewpoints to learn a single vector representation of a scene . Second , this representation is fed into a conditional generative model ( DRAW , Gregor et al . ( 2015 ) ) to produce an output image . See Figure 2 for a schematic description of the model . 4.1 A NOTE ON EVALUATION . While pixel loss compared with gold images is an obvious metric to optimise against when generating images , human judgements are more suitable for evaluating this task . More precisely , pixel loss is too strict a metric for evaluating the generative model due to the differing degrees of specificity between language and visual data points . Consider the images and descriptions in Figure 1 . Adjectives such as big or relative positions such as to the far right are sufficient to give us a high-level idea of a scene , but lack the precision required to generate an exact copy of a given image . The same description can be satisfied by an infinite number of visual renderings—implying that pixel loss with a gold image is not a precise measure of whether a given image is semantically consistent with the matching scene description . Human judgements on the other hand allow us to evaluate this task on the desired property , preferring spatial consistency over pixel-accuracy in the generated images . We showed annotators model samples with corresponding language descriptions , and asked them to judge whether the descriptions matched , partially matched , or did not match the image . We additionally asked for binary choices on whether ( 1 ) all object shapes and ( 2 ) all object colours from the text are in the image ; whether ( 3 ) all shape and colour combinations are correct ; and whether ( 4 ) all objects are correctly positioned . Annotators saw randomly mixed samples from all the models and benchmarks . For the UPPER BOUND evaluation we showed gold images and matching descriptions . Irrespective of the training setup below , we use synthetic descriptions of the output in the human annotation , as these guarantee us a consistent degree of specificity versus natural language descriptions that can widely vary in that respect . Note that this only concerns the description shown to the human annotators . In the evaluation , we feed synthetic language into the model for the experiments in Section 4.2 , and natural language descriptions in the case of Section 4.3 .

Summary: The authors present a large synthetic dataset for 3D scenes with templated descriptions.  They then use the model of Eslami 2018 to this new domain.  The previous work appears to already introduce all the necessary mechanisms for 3D generalization from multiple viewpoints, though this work embeds language instead of a scene in the process.  Minor note: A bit more discussion on this distinction would be appreciated.  Also, it appears that the previous work includes many of the rendered scenes also present here, so the primary focus of this paper is on the use of a language encoder (not necessarily a trivial extension).