Imagine That! Leveraging Emergent Affordances for Tool Synthesis in Reaching Tasks

1 INTRODUCTION . Deep generative models are gaining in popularity for unsupervised representation learning . In particular , recent models like MONet ( Burgess et al. , 2019 ) have been proposed to decompose scenes into objectcentric latent representations ( cf . Greff et al. , 2019 ; Engelcke et al. , 2019 ) . The notion of such an objectcentric latent representation , trained from examples in an unsupervised way , holds a tantalising prospect : as generative models naturally capture factors of variation , could they also be used to expose these factors such that they can be modified in a task-driven way ? We posit that a task-driven traversal of a structured latent space leads to affordances emerging naturally as directions in this space . This is in stark contrast to more common approaches to affordance learning where it is commonly achieved via direct supervision or implicitly via imitation ( e.g . Tikhanoff et al. , 2013 ; Myers et al. , 2015 ; Liu et al. , 2018 ; Grabner et al. , 2011 ; Do et al. , 2018 ) . The setting we choose for our investigation is that of tool synthesis for reaching tasks as commonly investigated in the cognitive sciences . The ability to make and use tools has been a cornerstone of humanity ’ s cognitive , technological and cultural development . It has long been hailed as a defining characteristic of intelligence ( Ambrose , 2001 ) . Consequently , the ability of artificial agents to select and use appropriate tools has received significant attention in , for example , the robotics community ( e.g . Sinapov & Stoytchev , 2007 ; Tikhanoff et al. , 2013 ; Toussaint et al. , 2018 ; Xie et al. , 2019 ) . A very limited number of works such as Wicaksono ( 2017 ) also consider tool making and typically deploy strong inductive biases such as simulations in which the agent adjusts numerical parameters given a set of design constraints . However , while much stands to be gained by endowing robots with an aptitude for tool use , more recent evidence in the cognitive sciences suggests that habitual tool use can not in and off itself be taken as a sign of intelligence ( e.g . Ambrose , 2001 ; Emery & Clayton , 2009 ) . Some animals , which are recognised as prodigious users of tools , do not show any sign of understanding either cause or effect or , indeed , the difference between appropriate and inappropriate tools ( e.g . Ambrose , 2001 ) . Instead , relatively recent results suggest that tool selection and manufacture – especially once demonstrated – is a significantly easier task than tool innovation : the step , prior to manufacture , of imagining the type of tool suitable for a particular task ( Beck et al. , 2011 ) . We posit that it is tool innovation which ultimately unlocks the full potential of tool manufacture and , thus , creative tool use . In order to demonstrate that a task-aware , object-centric latent space encodes useful affordance information we require a mechanism to train such a model as well as to purposefully explore the space . To this end we propose an architecture in which a task-based performance predictor ( a classifier ) operates on the latent space of an object-centric generative model ( see Figure 1 ) . During training the classifier is used to provide an auxiliary objective , aiding in shaping the latent space . Importantly , however , during test time the performance predictor is used to guide exploration of the latent space via activation maximisation ( Erhan et al. , 2009 ; Zeiler & Fergus , 2014 ; Simonyan et al. , 2014 ) , thus explicitly exploiting the structure of the space . While our desire to affect factors of influence is similar in spirit to the notion of disentanglement , it contrasts significantly with approaches such as β-VAE ( Higgins et al. , 2017 ) , where the factors if influence are effectively encouraged to be axis aligned . Our approach instead relies on a high-level auxiliary loss to discover the direction in latent space to explore . Our experimental design is inspired by a long history in the cognitive sciences and behavioural ecology of investigating reaching tasks which require tool use ( e.g . Kohler , 1925 ; van Leeuwen et al. , 1994 ; Chappell & Kacelnik , 2002 ; 2004 ; Emery & Clayton , 2009 ; Beck et al. , 2011 ) . The results demonstrate that artificial agents are indeed able not only to select – but to imagine – an appropriate tool for a variety of reaching tasks . The dataset consisting of 52,000 synthetic example reaching tasks will be made available to the community . To the best of our knowledge , ours is the first work to demonstrate an artificial agent ’ s ability to imagine , or synthesise , images of tools appropriate for a given task via optimisation in a structured latent embedding . Similarly , while activation maximisation has been used to visualise modified input images before ( e.g . Mordvintsev et al. , 2015 ) , we believe ours to be the first to effect deliberate manipulation of factors of influence by chaining the outcome of a task predictor to the latent space , and then decoding the latent representation back into an image . Beyond the application of tool synthesis , in demonstrating that object affordances can be viewed as directions in a structured latent space as well as by providing a novel architecture adept at deliberately manipulating interpretable factors of influence we believe our work to provide novel perspectives on affordance learning and disentanglement . 2 RELATED WORK . The idea of an affordance , which describes a potential action to be performed on an object ( e.g . a doorknob affords being turned ) , goes back to Gibson ( 1977 ) . Together , robotics and computer vision boast a rich literature on learning task-affordances ( e.g . Tikhanoff et al. , 2013 ; Mar et al. , 2015 ; Stoytchev , 2005 ) , which includes work on learning multiple attention-like affordance masks ( provided labelled data ) for common objects such as kitchen utensils ( Do et al. , 2018 ; Myers et al. , 2015 ) . To learn affordances without supervision , another research thread has employed simulations in virtual environments . Grabner et al . ( 2011 ) is an example in which a simulated humanoid was re-positioned , via trial-and-error , to find all of the surfaces in a 3D scene in which it is possible to sit . Other studies have successfully learned affordances through imitation ( e.g . Liu et al. , 2018 ) . Closely related to the the affordance literature is a mature body of research dealing with almost all aspects of robot tool use . Tasks addressed include reaching ( Jamone et al. , 2015 ) , grasping ( Takahashi et al. , 2017 ) , pushing ( Stoytchev , 2005 ) and hammering ( Fang et al. , 2018 ) . This research commonly depends on a more traditional pipeline approach , consisting of tool-recognition ( Tikhanoff et al. , 2013 ; Fang et al. , 2018 ) , toolselection ( Xie et al. , 2019 ; Saito et al. , 2018 ) , planning ( Toussaint et al. , 2018 ) and execution ( Tikhanoff et al. , 2013 ; Fang et al. , 2018 ; Stoytchev , 2005 ; Jamone et al. , 2015 ) . Recently , the advent of deep generative models has led to an active research area using world models ( Ha & Schmidhuber , 2018 ; Lesort et al. , 2018 ; Nair et al. , 2018 ) , in which an artificial agent can train itself on tasks using a kind of imagination . Our work is directly inspired by this , though we approach imagination modelling in a different way . In particular , we propose an architecture in which a generative model for unsupervised scene decomposition ( Burgess et al. , 2019 ) is paired with the use of activation maximisation ( e.g . Erhan et al. , 2009 ; Zeiler & Fergus , 2014 ; Simonyan et al. , 2014 ) of an auxiliary , task-driven loss-signal back into the generative model ’ s latent representations to drive the imagination process . 3 DATASET . To investigate tool imagination , we designed a set of simulated reaching tasks with clear and controllable factors of influence . Each task image is comprised of a green target button , a red line delineating the workspace area , and , optionally , a set of blue obstacles . We also vary the goal location and the sizes and positions of the obstacles . For each task image , we provide a second image depicting a set of ‘ tools ’ – straight sticks , L-shaped hooks , and J-shaped claws – with varying dimensions , shapes , colours , and poses . Given a pair of images ( i.e . the task image and the toolkit image ) , the goal is to select a tool for a given task scene that can reach the target ( the green dot ) whilst avoiding obstacles ( blue areas ) and remaining on the exterior of the workspace ( i.e . behind the red line ) . Depending on the task image , the applicability of a tool is determined by different subsets of its attributes . For example , if the target button is unobstructed , then any tool of sufficient length will satisfy the constraints ( regardless of its width or shape ) . However , when the target is hidden behind a corner , or only accessible through a narrow gap , an appropriate tool also needs to feature a long-enough hook , or a thin-enough handle , respectively . By design , the colour of a tool does not influence its applicability and is introduced as a nuisance factor . As depicted in Figure 2 , we have designed eight scenario types to study these factors of influence in isolation and in combination . Importantly , the scenario types are designed to enable an investigation into our approach ’ s performance on in-sample tasks , interpola- tion tasks and an extrapolation task . In-sample scenarios differ from interpolation scenarios in that the tasks are different but can be solved with tools ( sticks or hooks ) that were seen during training . The extrapolation scenario consists of a novel scenario but almost always also requires the use of a new tool : a J-shaped claw . For each task , toolkit images are constructed by scattering tools from a catalogue with randomised colours and poses in a separate visible workspace . A geometric applicability check , described in Appendix B , verifies whether or not any of the tools can reach the target , while satisfying the task constraints . We consider a task–toolkit pair feasible if the toolkit image includes at least one tool that can achieve the task under the specified constraints . More formally , our dataset can be written as a set of N triplets : { ( InG , InT , ρn ) } Nn=1 , where each example features a task image IG , a toolkit image IT , and a binary label ρ indicating the feasibility of reaching the target with one of the given tools ( see Figure 3 ) . We also save instance segmentation masks Mnk for the toolkits , as well as the applicability label ρnk for each tool , indexed by k. In all our experiments , we restrict the training input to the sparse high-level triplets and use the additional ground-truth labels for evaluation purposes only . For additional details on the dataset , see Appendix B .

Summary: This paper proposes an architecture for synthesizing tools to be used in a reaching task. Specifically, during training the agent jointly learns to segment an image of a set of three tools (via the MONet architecture) and to classify whether one the tools will solve the given scene. At test time, one of the three tools is selected based on which seems most feasible, and then gradient descent is used to modify the latent representation of the tool in order to synthesize a new tool to (hopefully) solve the scene. The paper demonstrates that this approach can achieve ok performance on familiar scenes with familiar tools, but that it fails to generalize when exposed to unfamiliar scenes or unfamiliar tools. The paper reports a combination of the quantitative results showing that optimizing the latent space can lead to successful synthesis in some cases, and qualitative results showing that the synthesized tools change along interpretable dimensions such as length, width, etc. The combination of these results suggest that the model has learned something about which tool dimensions are important for being able to solve the types of reaching tasks given in the paper.