Learning to Refer to 3D Objects with Natural Language

Human world knowledge is both structured and flexible . When people see an object , they represent it not as a pixel array but as a meaningful arrangement of semantic parts . Moreover , when people refer to an object , they provide descriptions that are not merely true but also relevant in the current context . Here , we combine these two observations in order to learn fine-grained correspondences between language and contextually relevant geometric properties of 3D objects . To do this , we employed an interactive communication task with human participants to construct a large dataset containing natural utterances referring to 3D objects from ShapeNet in a wide variety of contexts . Using this dataset , we developed neural listener and speaker models with strong capacity for generalization . By performing targeted lesions of visual and linguistic input , we discovered that the neural listener depends heavily on part-related words and associates these words correctly with the corresponding geometric properties of objects , suggesting that it has learned task-relevant structure linking the two input modalities . We further show that a neural speaker that is ‘ listener-aware ’ — that plans its utterances according to how an imagined listener would interpret its words in context — produces more discriminative referring expressions than an ‘ listener-unaware ’ speaker , as measured by human performance in identifying the correct object . 1 INTRODUCTION . Human world knowledge is both structured and flexible . For example , when people see a chair , they represent it not as a pixel array but as a semantically meaningful combination of parts , such as arms , legs , seat , and back . How to obtain and flexibly deploy such structured knowledge remains an outstanding problem in machine learning ( Lake et al. , 2017 ) . One promising approach is to harness the rich conceptual and relational structure latent in language ( Andreas et al. , 2017 ) . Natural languages have been optimized across human history to solve the problem of efficiently communicating those aspects of the world most relevant to current goals ( Kirby et al. , 2015 ; Gibson et al. , 2017 ) . Consequently , language reflects the structured nature of our world knowledge : we not only conceive of a chair in terms of its semantic parts , but can combine multiple words to refer to its ‘ curved back ’ or ‘ cushioned seat ’ , and provide more informative descriptions if the context requires it , e.g. , refer to a different distinguishing part if all the chairs have a cushioned seat . Our goal is to leverage these insights to develop systems that can make fine-grained distinctions between complex object geometries across a wide variety of contexts . Our approach is to leverage natural language produced by people in an interactive communication task to develop neural network models of the speaker and listener roles in this task . We find that the resulting representations learned by these models exhibit structure that is crucial for robust communication : first , they capture taskrelevant correspondences between individual parts of objects and individual tokens of language , and second , they have strong capacity to generalize to novel contexts , objects , utterances , and other related object classes . We make the following contributions : close context far context 2 DATASET AND TASK . Our dataset consists of triplets of 3D objects coupled with referential utterances that aim to distinguish one object ( the “ target ” ) from the remaining two ( the “ distractors ” ) . To obtain such utterances , we paired participants from Amazon Mechanical Turk to play an online , reference game ( Hawkins , 2015 ) . On each round of the game , the two players were shown a triplet of objects . The designated target object was privately highlighted for one player ( the “ speaker ” ) who was asked to send a message through a chat box such that their partner ( the “ listener ” ) could successfully select it from the context ( see Appendix Fig . 16 ) . To ensure speakers used geometric information rather than color , texture , orientation , or position on the screen , we scrambled the positions of the objects for each participant and used textureless , colorless renders of 3D objects taken from the same viewpoint . Additionally , to ensure communicative interaction was natural , no constraints were placed on the chat box : referring expressions from the speaker were occasionally followed by clarification questions from the listener or other discourse . A crucial decision in building our dataset concerned the construction of useful contexts that would reliably elicit fine-grained contrastive language . Perceptually identical objects can not be distinguished with language at all , while wildly different objects ( a chair and a car ) can be easily distinguished with a single word ( “ it ’ s a chair ” ) . To solve this problem , we considered three objectives . First , the set of objects must be familiar so we can tap existing visual and linguistic representations . Second , the objects should be complex and variable to provide wide coverage of interesting geometries . Third , different contexts must contain diverse combinations of objects to ensure variation in the relevant distinctions required . To satisfy the first two objectives , we utilize the collection of about 7,000 chairs from ShapeNet ( Chang et al. , 2015 ) . This class is geometrically complex , densely sampled , highly diverse , and abundant in the real world . To satisfy the third objective in a scalable and unsupervised manner , we estimated object similarity between different chairs using the Point Cloud-AutoEncoder ( PC-AE ) from Achlioptas et al . ( 2018 ) . This representation allowed us to leverage the fact that point-clouds extracted from a 3D surface provide an intrinsic 3D representation of an object , oblique to color or texture . To deal with the inhomogeneity of data in repositories like ShapeNet we used a sampling 1The dataset and our code will be publicly available upon acceptance . strategy to construct our triplets . First , we computed the 2-nearest-neighbor graph of all ShapeNet chairs based on their PC-AE embedding distances . On this graph , we selected chairs of highest indegree as seeds and for each seed chair we generated two kinds of triplets . Close contexts sampled nearby chairs while Far contexts sampled highly dissimilar chairs ( see Fig . 1 ) . Additional details of triplet construction are provided in the Appendix ( Section 9.1 ) . In total , we collected a corpus containing 78,789 referring expressions for 4,054 triplets , containing 4,511 unique chairs . In doing so we recruited 2,124 unique participants . Human performance on the reference game was high in general , but listeners made significantly more errors in the close triplets ( 94.2 % vs. 97.2 % , z = 13.54 , p < 0.001 ) . Also , significantly longer utterances were used on average to describe targets in close triplets ( approximately 8.4 words vs. 6.1 , t = −35 , p < 0.001 ) . A wide spectrum of descriptions was elicited , ranging from the more holistic/categorical common for far triplets ( e.g. , “ the rocking chair ” ) to more complex , geometric language common for close triplets ( e.g. , “ thinner legs but without armrests ” ) . 78 % of utterances used at least one part word : “ back ” , “ legs ” , “ seat , ” “ arms ” , or closely related synonyms ( e.g. , “ armrests ” ) . 3 NEURAL LISTENERS . Constructing neural listeners that reason about geometric relationships is a key contribution of our work . It lays the foundation for creating speakers that utter discriminative utterances and enables the creation of an object retrieval system that operates with linguistic queries . Given its importance , below we conduct a detailed comparison between three distinct architectures , highlight the effect of different regularization techniques , and investigate the merits of two different representations of 3D objects for the listening task , namely , images and point clouds . In what follows , we denote the three objects of a communication context as O = { o1 , o2 , o3 } , the corresponding word-tokenized utterance , which has at most K tokens , as U = u1 , u2 , . . . and as t ∈ O the referential target . Our proposed listener is inspired by Monroe et al . ( 2017 ) . It takes as input a ( latent code ) vector for each of the three objects in O and a ( latent code ) vector for each token of U , and outputs an object–utterance compatibility score L ( oi , U ) ∈ [ 0 , 1 ] for each of the three objects . At its core lies a multi-modal LSTM ( Hochreiter & Schmidhuber , 1997 ) that takes as input ( “ is grounded ” with ) the vector of a single object , processes the word-sequence U , and is read out by a final MLP to yield a single number ( the compatibility score ) . This is repeated for each oi , sharing all network parameters across the objects . We then apply a soft-max to the three compatibility scores to yield a distribution over the three objects , and compute a cross-entropy loss between this distribution and the ground-truth indicator vector of the target . Object encoders We experimented with three object representations to capture the underlying geometries : ( a ) the bottleneck representation of a pretrained Point Cloud-AutoEncoder ( PC-AE ) , ( b ) the embedding provided by a convolutional network operating on single-view images of nontextured 3D objects , or ( c ) a combination of ( a ) and ( b ) . Specifically , for ( a ) we use the PC-AE architecture of Achlioptas et al . ( 2018 ) trained with single-class point clouds extracted from the surfaces of 3D CAD models , while for ( b ) we use the activations of the penultimate layer of a VGG16 ( Simonyan & Zisserman , 2014 ) neural network , pre-trained on ImageNet ( Deng et al. , 2009 ) , and fine-tuned on an 8-way classification task with images of objects from ShapeNet . For each representation we project the corresponding code vector to the input space of the LSTM using a fully connected ( FC ) layer with L2-norm weight regularization . The addition of these projectionlike layers improves the training and convergence of our system . While there are many ways to simultaneously incorporate the two modalities in the LSTM , we found that the best performance resulted when we ground the LSTM with the image code , concatenate the LSTM ’ s final output ( after processing U ) with the point cloud code , and finally feed this result into a shallow MLP to produce the final compatibility ( see Figure 2 for an overview of this architecture ) . We note that grounding the LSTM with point clouds and using images towards the end of the pipeline , resulted in a significant performance drop ( ∼ 4.8 % on average ) . Also , adding dropout at the input layer of the LSTM and L2 weight regularization and dropout at and before the FC projecting layers was crucial ( giving improvements of more that 10 % ) . The token codes of each sentence where initialized with the GloVe embedding ( Pennington et al. , 2014 ) and fine-tuned for the listening task . Incorporating context information Critically , our proposed listener architecture first scores each object separately then applies softmax normalization to yield a score distribution over the three objects . In order to evaluate the importance of this design choice , we consider two alternative architectures that incorporate context earlier , at encoding . The first alternative ( Separate-Augment ) , is identical to the proposed architecture , except for it uses a convolutional layer to augment each object ’ s grounding vector with information about the other two objects in context before yielding its score . Specifically , if vi is the image code vector of the i-th object ( oi ∈ O ) , to produce the grounding vector for oi , the convolutional layer receives f ( vj , vk ) ||g ( vj , vk ) ||vi , where f , g are order invariant functions , such as the average or max-pooling and || denotes feature-wise concatenation . The second alternative architecture ( At-Once ) first feeds the image vectors for all three objects sequentially to the LSTM and then proceeds to process the tokens of U once , to yield the entire score distribution . Similarly to the proposed architecture , point clouds are incorporated in both alternatives via a separate MLP after the LSTM . Attention mechanism over words We hypothesize that a listener forced to prioritize a few words in each utterance would learn to prioritize words that express properties that distinguish the target from the distractors ( and , thus , perform better ) . To test this hypothesis , we augment the listener models with a bilinear attention mechanism over words . Specifically , to estimate the “ importance ” of each text-token ui we compare the output of the LSTM for ui ( denoted as ri ) with the hidden state after the entire utterance has been processed ( denoted as h ) . The ideas is that the hidden state acts as a summary of the grounded sentence ( Shen & Lee , 2016 ) , that can be used to assess the relative importance of each word as ai = ∆ rTi ×Watt × h , where Watt is a trainable diagonal matrix . With the attention mechanism in place , the final output of the LSTM is defined as ∑|U | i=1 ri âi , where âi = exp ( ai ) ∑|U| j exp ( aj ) and is the point-wise product . The optimal parameters of each listener ( and speaker ) , the hyper-parameter search strategy , and the exact details of training are provided in the Appendix 9.2 and 9.3 .

Summary: The paper investigates how chairs are being described "in the context of other similar or not-so-similar chairs", by humans and neural networks. Humans perceive an object's structure, and use it to describe the differences to other objects "in context". The authors collected a corresponding "chairs in context" corpus, and build models that can describe the "target" chair, that can be used to retrieve the described object, and that can create more discriminative language if given information about the listener.