Imagine That! Leveraging Emergent Affordances for 3D Tool Synthesis

1 INTRODUCTION . The advent of deep generative models ( e.g . Burgess et al. , 2019 ; Greff et al. , 2019 ; Engelcke et al. , 2019 ) with their aptitude for unsupervised representation learning casts a new light on learning affordances ( Gibson , 1977 ) . This kind of representation learning raises a tantalising question : Given that generative models naturally capture factors of variation , could they also be used to expose these factors such that they can be modified in a task-driven way ? We posit that a task-driven traversal of a structured latent space leads to affordances emerging naturally along trajectories in this space . This is in stark contrast to more common approaches to affordance learning where it is achieved via direct supervision or implicitly via imitation ( e.g . Tikhanoff et al. , 2013 ; Myers et al. , 2015 ; Liu et al. , 2018 ; Grabner et al. , 2011 ; Do et al. , 2018 ) . The setting we choose for our investigation is that of tool synthesis for reaching tasks as commonly investigated in the cognitive sciences ( Ambrose , 2001 ; Emery & Clayton , 2009 ) . In order to demonstrate that a task-aware latent space encodes useful affordance information we require a mechanism to train such a model as well as to purposefully explore the space . To this end we propose an architecture in which a task-based performance predictor ( a classifier ) operates on the latent space of a generative model ( see fig . 1 ) . During training the classifier is used to provide an auxiliary objective , aiding in shaping the latent space . Importantly , however , during test time the performance predictor is used to guide exploration of the latent space via activation maximisation ( Erhan et al. , 2009 ; Zeiler & Fergus , 2014 ; Simonyan et al. , 2014 ) , thus explicitly exploiting the structure of the space . While our desire to affect factors of influence is similar in spirit to the notion of disentanglement , it contrasts significantly with models such as β-VAE ( Higgins et al. , 2017 ) , where the factors of influence are effectively encouraged to be axis-aligned . Our approach instead relies on a high-level auxiliary loss to discover the direction in latent space to explore . Our experiments demonstrate that artificial agents are able to imagine an appropriate tool for a variety of reaching tasks by manipulating the tool ’ s task-relevant affordances . To the best of our knowledge , this makes us the first to demonstrate an artificial agent ’ s ability to imagine , or synthesise , 3D meshes of tools appropriate for a given task via optimisation in a structured latent embedding . Similarly , while activation maximisation has been used to visualise modified input images before ( e.g . Mordvintsev et al. , 2015 ) , we believe this work to be the first to effect deliberate manipulation of factors of influence by chaining the outcome of a task predictor to the latent space , and then decoding the latent representation back into a 3D mesh . Beyond the application of tool synthesis , we believe our work to provide novel perspectives on affordance learning and disentanglement in demonstrating that object affordances can be viewed as trajectories in a structured latent space as well as by providing a novel architecture adept at deliberately manipulating interpretable factors of influence . 2 RELATED WORK . The concept of an affordance , which describes a potential action to be performed on an object ( e.g . a doorknob affords being turned ) , goes back to Gibson ( 1977 ) . Because of their importance in cognitive vision , affordances are extensively studied in computer vision and robotics . Commonly , affordances are learned in a supervised fashion where models discriminate between discrete affordance classes or predict masks for image regions which afford certain types of human interaction ( e.g . Stoytchev , 2005 ; Kjellström et al. , 2010 ; Tikhanoff et al. , 2013 ; Mar et al. , 2015 ; Myers et al. , 2015 ; Do et al. , 2018 ) . Interestingly , most works in this domain learn from object shapes which have been given an affordance label a priori . However , the affordance of a shape is only properly defined in the context of a task . Hence , we employ a task-driven traversal of a latent space to optimise the shape of a tool by exploiting factors of variation which are conducive to task success . Recent advances in 3D shape generation employ variational models ( Girdhar et al. , 2016 ; Wu et al. , 2016 ) to capture complex manifolds of 3D objects . Besides their expressive capabilities , the latent spaces of such models also enable smooth interpolation between shapes . Remarkable results have been demonstrated including ‘ shape algebra ’ ( Wu et al. , 2016 ) and the preservation of object part semantics ( Kohli et al. , 2020 ) and fine-grained shape styles ( Yifan et al. , 2019 ) during interpolation . This shows the potential of disentangling meaningful factors of variation in the latent representation of 3D shapes . Inspired by this , we investigate whether these factors can be exposed in a taskdriven way . In particular , we propose an architecture in which a generative model for 3D object reconstruction ( Liu et al. , 2019 ) is paired with activation maximisation ( e.g . Erhan et al. , 2009 ; Zeiler & Fergus , 2014 ; Simonyan et al. , 2014 ) of a task-driven performance predictor . Guided by its loss signal , activation maximisation traverses the generative model ’ s latent representations and drives an imagination process yielding a shape suitable for the task at hand . A key application of affordance-driven shape imagination is tool use . Robotics boasts a mature body of literature studying how robots can utilise tools to improve their performance across a wide range of tasks like reaching ( Jamone et al. , 2015 ) , grasping ( Takahashi et al. , 2017 ) , pushing ( Stoytchev , 2005 ) and hammering ( Fang et al. , 2018 ) . The pipeline executing tool-based tasks typically starts with models for tool recognition and selection ( e.g . Tikhanoff et al. , 2013 ; Zhu et al. , 2015 ; Fang et al. , 2018 ; Saito et al. , 2018 ; Xie et al. , 2019 ) before tool properties and affordances are leveraged to compute higher-order plans ( Toussaint et al. , 2018 ) . Our proposed model lends itself to robotics applications like these , as the learned latent space encodes a rich object-centric representation of tools that are biased for specific tasks . 3 METHOD . Our overarching goal is to perform task-specific tool synthesis for 3D reaching tasks . We frame the challenge of tool imagination as an optimisation problem in a structured latent space obtained using a generative model . The optimisation is driven by a high-level , task-specific performance predictor , which assesses whether a target specified by a goal image IG is reachable given a particular tool and in the presence of obstacles . To map from tool images into manipulable 3D tools , we first train an off-the-shelf 3D single-view reconstruction model taking as input tool images IiT , I j T and corresponding tool silhouettes IiS , I j S as rendered from two different vantage points i and j . After training , the encoder can infer the tool representation that contains the 3D structure information given a single-view RGB image and its silhouette as input . This representation is implicitly used to optimise the tool configuration to make it suitable for the task at hand . An overview of our model is given in fig . 2 . More formally , we consider N data instances : { ( InG , In , iT , In , jT , In , iS , In , jS , ρn ) } Nn=1 , where each example features a task image IG , tool images IT in two randomly selected views i and j , and their corresponding silhouettes IS , as well as a binary label ρ indicating the feasibility of reaching the target with the given tool . Examples of task images and model inputs are shown in fig . 3 . In all our experiments , we restrict the training input to such sparse high-level instances . For additional details on the dataset , we refer the reader to the supplementary material . 3.1 REPRESENTING TASKS AND TOOLS . Given that our tools are presented in tool images IT , it is necessary for the first processing step to perform a 3D reconstruction of IT , from pixels into meshes . To achieve this single view 3D reconstruction of images into their meshes , we employ the same architecture as proposed by ( Kato et al. , 2018 ; Wang et al. , 2018 ) . The 3D reconstruction model consists of two parts : an encoder network and a mesh decoder . Given the tool image and its silhouette in view i , i.e IiT and I i S , we denote the latent variable encoding the tool computed by the encoder , ψ , as ψ ( IiT , I i S ) = zT . ( 1 ) The mesh decoder takes zT as input and synthesises the mesh by deforming a template . A differentiable renderer ( Liu et al. , 2019 ) predicts the tool ’ s silhoutte ÎjS in another view j , which is compared to the ground-truth silhouette IjS to compute the silhouette loss Ls . This silhouette loss Ls together with an auxiliary geometry loss Lg formulates the total 3D reconstruction loss : Lrecon = Ls + µLg , ( 2 ) where µ is the weight of the geometry loss . We refer the reader to Liu et al . ( 2019 ) regarding the exact hyper-parameter and training setup of the 3D reconstruction model . Task images IG are similarly represented in an abstract latent space . For this we employ a task encoder , φ , which consists of a stack of convolutional layers.1 φ takes the task image IG as input and maps it into the task embedding zG . 3.2 TOOL IMAGINATION . Task-driven learning The tool representation zT contains task-relevant information such as tool length , width , and shape . In order to perform tool imagination , the sub-manifold of the latent space that corresponds to the task-relevant features needs to be accessed and traversed . This is achieved by adding a three-layer MLP as a classifier σ . The classifier σ takes as input a concatenation hcat of the task embedding zG and the tool representation zT , and predicts the softmax over the binary task 1Architecture details are provided in the supplementary material . success . The classifier learns to identify the task-relevant sub-manifold of the latent space by using the sparse success signal ρ and optimising the binary-cross entropy loss , such that Ltask ( σ ( hcat ) , ρ ) = − ( ρ log ( σ ( hcat ) ) + ( 1− ρ ) log ( 1− σ ( hcat ) ) ) , ( 3 ) where ρ ∈ { 0 , 1 } is a binary signal indicating whether or not it is feasible to solve the task with the given tool . The whole system is trained end-to-end with a loss given by L � IG , I i T , I i S , I j T , I j S , ρ � = Lrecon + Ltask . ( 4 ) Note that the gradient from the task classifier σ propagates through both the task encoder φ and the toolkit encoder ψ , and therefore helps to shape the latent representations of the toolkit with respect to the requirements for task success . Tool imagination Once trained , our model can synthesise new tools by traversing the latent manifold of individual tools following the trajectories that maximise classification success given a tool image and its silhouette ( fig . 2 ) . To do this , we first pick a tool candidate and concatenate its representation zT with the task embedding zG . This warm-starts the imagination process . The concatenated embedding hcat is then fed into the performance predictor σ to compute the gradient with respect to the tool embedding zT . We then use activation maximisation ( Erhan et al. , 2009 ; Zeiler & Fergus , 2014 ; Simonyan et al. , 2014 ) to optimise zT with regard to Ltask of the success estimation σ ( hcat ) and a feasibility target ρs = 1 , such that zT = zT + η ∂Ltask ( σ ( zT ) , ρs ) ∂zT , ( 5 ) where η denotes the learning rate for the update . Finally , we apply this gradient update for S steps or until the success estimation σ ( zT ) reaches a threshold γ , and use ψ� ( zT ) to generate the imagined 3D tool mesh represented by zT .

Summary: This paper presents a method with two goals: (1) estimate if a given tool can solve a given task, and (2) generate a tool that can solve a given task. One encoder maps a tool image and silhouette into a latent code, and a decoder maps this code into a mesh; another encoder maps a task image into a latent code; this code is concatenated with the tool code, then mapped to a task success probability. The networks are trained together, so that task success probability has some impact on the tool encoder's latent space. Results show that the model is largely successful in both tasks, but no challenging baselines are here.