Imagine That! Leveraging Emergent Affordances for Tool Synthesis in Reaching Tasks
Authors: Yizhe Wu
Sudhanshu Kasewa
Oliver Groth
Sasha Salter
Li Sun
Oiwi Parker Jones
Ingmar Posner
Venue:  (2020)
Keywords: Affordance Learning
Imagination
Generative Models
Activation Maximisation

Abstract: In this paper we investigate an artificial agent's ability to perform task-focused tool synthesis via imagination. Our motivation is to explore the richness of information captured by the latent space of an object-centric generative model - and how to exploit it. In particular, our approach employs activation maximisation of a task-based performance predictor to optimise the latent variable of a structured latent-space model in order to generate tool geometries appropriate for the task at hand. We evaluate our model using a novel dataset of synthetic reaching tasks inspired by the cognitive sciences and behavioural ecology. In doing so we examine the model's ability to imagine tools for increasingly complex scenario types, beyond those seen during training. Our experiments demonstrate that the synthesis process modifies emergent, task-relevant object affordances in a targeted and deliberate way: the agents often specifically modify aspects of the tools which relate to meaningful (yet implicitly learned) concepts such as a tool's length, width and configuration. Our results therefore suggest, that task relevant object affordances are implicitly encoded as directions in a structured latent space shaped by experience. 

Under review conference paper at ICLR 2020
IMAGINE THAT! LEVERAGING EMERGENT AFFORDANCES FOR TOOL SYNTHESIS IN REACHING TASKS
Anonymous authors Paper under double-blind review
ABSTRACT
In this paper we investigate an artificial agent's ability to perform task-focused tool synthe- sis via imagination. Our motivation is explore the richness of information captured by the latent space of an object-centric generative model and how to exploit it. In particular, our approach employs activation maximisation of task-based performance predictor to optimise the latent variable of structured latent-space model in order to generate tool ge- ometries appropriate for the task at hand. We evaluate our model using a novel dataset of synthetic reaching tasks inspired by the cognitive sciences and behavioural ecology. In do- ing so we examine the model's 's ability to imagine tools for increasingly complex scenario types, beyond those seen during training. Our experiments demonstrate that the synthesis process modifies emergent, task-relevant object affordances in a targeted and deliberate way: the agents often specifically modify aspects of the tools which relate to meaningful (yet implicitly learned) concepts such as a tool's length, width and configuration. Our results therefore suggest, that task relevant object affordances are implicitly encoded as directions structured latent space shaped by experience.
INTRODUCTION
Deep generative models are gaining in popularity for unsupervised representation learning. In particular, recent models like MONet (Burgess et al., 2019) have been proposed to decompose scenes into object- centric latent representations (ct. Greff et al. 2019 Engelcke et al. 2019). The notion of such an object- centric latent representation, trained from examples in an unsupervised way, holds tantalising prospect: generative models naturally capture factors of variation, could they also be used to expose these factors such that they can be modified in a task-driven way? We posit that task-driven traversal of a structured latent space leads to affordances emerging naturally directions in this space. This is in stark contrast to more common approaches to affordance learning where it is commonly achieved via direct supervision or implicitly via imitation (e.g. Tikhanoff et al. 2013 Myers al. 2015 Liu al. 2018 Grabner et al. 2011 Do et al. 2018).
The setting we choose for our investigation is that of tool synthesis for reaching tasks as commonly inves- tigated in the cognitive sciences. The ability to make and use tools has been a cornerstone of humanity's s cognitive, technological and cultural development. It has long been hailed as a defining characteristic of intelligence (Ambrose)
2001). Consequently, the ability of artificial agents select and use appropriate tools has received significant attention in, for example, the robotics community (c.g. [Sinapov & Stoytchev 2007 Tikhanoff et al. 2013) Toussaint et al. 2018 Xie et al., 2019). A very limited number of works such as Wicaksono (2017) 7) also consider tool making and typically deploy strong inductive biases such as simu- lations in which the agent adjusts numerical parameters given set of design constraints. However, while much stands to be gained by endowing robots with an aptitude for tool use, more recent evidence in the cog- nitive sciences suggests that habitual tool use cannot in and off itself be taken as a sign of intelligence (c.g. Ambrose 2001 Emery& & Clayton| 2009). Some animals, which are recognised as prodigious users of tools,

---

Under review conference paper at ICLR 2020
do not show any sign of understanding either cause or effect or, indeed, the difference between appropriate and inappropriate tools (e.g. [mmrrsee| (001). Instead, relatively recent results suggest that tool selection and manufacture - especially once demonstrated is a significantly easier task than tool innovation: the step, prior to manufacture, of imagining the type of tool suitable for a particular task (Beck et al. 2011). We posit that it is tool innovation which ultimately unlocks the full potential of tool manufacture and, thus, creative tool use.
In order to demonstrate that a task-aware, object-centric latent space encodes useful affordance informa- tion we require a mechanism to train such a model as well purposefully explore the space. To this end we propose an architecture in which a task-based performance predictor (a classifier) operates on the latent space of an object-centric generative model (see Figure D During training the classifier is used provide an auxiliary objective, aiding in shaping the latent space. Importantly, however, during test time the performance predictor is used to guide exploration of the latent space via activation maximisation (Erhan jet al. 2009. Zeiler & Fergus| 2014, SSioonyan et al.) 2014), thus explicitly exploiting the structure of the space. While our desire to affect factors of influence is similar in spirit to the notion of disentanglement, it contrasts significantly with approaches such as 3-VAE (Higgins et al.) 2017). where the factors if influence are effectively encouraged to be axis aligned. Our approach instead relies on a high-level auxiliary loss to discover the direction in latent space explore.
Our experimental design is inspired by a long history in the cognitive sciences and behavioural ecology of investigating reaching tasks which require tool use (e.g. Kohler [925; van Leeuwen et al.) 1994 Chappell & Kacelnik] 2002] 2004 EEmer& & Clayton| 2009 Beck al. [201T). The results demonstrate that artificial agents are indeed able not only to select- but to imagine an appropriate tool for variety of reaching tasks. The dataset consisting of 52,000 synthetic example reaching tasks will be made available the community.
To the best of our knowledge, ours is the first work to demonstrate an artificial agent's 's ability to imagine, synthesise, images of tools appropriate for given task via optimisation in structured latent embedding. Similarly. while activation maximisation has been used to visualise modified input images before (c.g. Mord- vintsev et al. 2015), we believe ours to be the first to effect deliberate manipulation of factors of influence by chaining the outcome of task predictor to the latent space, and then decoding the latent representation back into an image. Beyond the application of tool synthesis, in demonstrating that object affordances can be viewed directions liberately manipulating interpretable factors of influence we believe our work to provide novel perspectives on affordance learning and disentanglement.
structured latent space as well as by providing a novel architecture adept de-
2 RELATED WORK
The idea of an affordance, which describes potential action to be performed affords being turned), goes back to Gibson (1977). Together, robotics and computer vision boast a rich lit- erature learning task-affordances (e.g. Tikhanoff et al. 2013 Mar et al. 2015 Stoytchev 2005). which includes work on learning multiple attention-like like affordance masks (provided labelled data) for common objects such as kitchen utensils (Do et al. 2018 Myers et al. 2015). To learn affordances without supervi- sion, another research thread has employed simulations in virtual environments. Grabner et al. (201T) an example in which simulated humanoid was re- rospoiiiioee,, via trial-and-eree to find all of the surfaces in a 3D scene in which it is possible sit. Other studies have successfully learned affordances through imitation (e.g. Liu et al. 2018).
object (e.g. doorknob
Closely related to the the affordance literature is mature body of research dealing with almost all aspects of robot tool use. Tasks addressed include reaching (Jamone et al. 2015),, grasping (Takahashi et al. (017). pushing /Stoytchev/20033
and hammering (Fang et al.) 2018). This research commonly depends on more traditional pipeline approach, consisting of tool-recognition (Tikhanoff al. 2013 Fang et al.) 2018). tool-
selection (Xie et al., [2019] Saito el al., 2018), planning (Toussaint et al. 2018) and execution (Tikhanoff
et al. 2013 Fang al. 2018 Stoytchev
2005 Jamone et 2l.1
2

---

Under review conference paper at ICLR 2020
toolkit
latent space model
tool synthesis
task
X
X
X
feasibility
success
X
performance predictor
optimisation steps
Figure I: Innovating tools in a simulated reaching task. The model is trained on data triplets comprising the toolkit, the task environment, and indicator of success (left). The toolkit is image of stick-like tools (here in yellow, light green, and pink). The goal of the task to determine if any tool can reach the dark green target whilst touching the red boundary and avoiding any blue obstacles. When no object in the toolkit satisfics these geometric constraints, our model (TasMON) is able to take tool from the toolkit and imagine suitable modifications until it does satisfy these constraints (right). Here we see TasMON imagine the yellow stick transforming into a hook with thin enough handle to both pass through the gap between the blue obstacles and reach the dark green target, all whilst still crossing over the red boundary.
Recently, the advent of deep generative models has led to an active research area using world models (Ha & Schmidhuber] 2018 Lesort et al. 2018 Nair et al. 2018), 3 which an artificial agent can train itself on tasks using kind of imagination. Our work is directly inspired by this, though we approach imagination modelling a different way. In particular. we propose an architecture in which generative model for unsupervised scene decomposition (Burgess et 2019) is paired with the use of activation maximisation (e.g. EErhan et al.) (2009] ( ZZiler & Fergus (imonnan et al. 2014) of an auxiliary, task-driven loss-signal back into the generative model's 's latent representations to drive the imagination process.
3 DATASET
To investigate tool imagination, we designed of simulated reaching tasks with clear and controllable fac- tors of influence. Each task image comprised of green target button, a red line delineating the workspace area, and, optionally, a set of blue obstacles. We also vary the goal location and the sizes and positions of the obstacles. For each task image, we provide a second image depicting a set of 'tools' straight sticks, L-shaped hooks, and J-shaped claws with varying dimensions, shapes, colours, and poses. Given a pair of images (i.e. the task image and the toolkit image), the goal to select a tool for a given task scene that can reach the target (the green dot) whilst avoiding obstacles (blue areas) and remaining on the exterior of the workspace (i.e. behind the red line). Depending the task image, the applicability of tool is deter- mined by different subsets of its attributes. For example, if the target button is unobstructed, then any tool of sufficient length will satisfy the constraints (regardless of its width shape). However, when the target is hidden behind a corner, or only accessible through a narrow gap, an appropriate tool also needs feature a Long-enough! hook, or thin-enough handle, respectively. By design, the colour of tool does not influence its applicability and introduced a nuisance factor. As depicted in Figure - we have designed eight scenario types to study these factors of influence in isolation and in combination. Importantly, the scenario types designed to enable investigation into approach's 's performance in-sample tasks, interpola-
3

---

Under review conference paper at ICLR 2020
In-sample Scenarios
Interpolation
Extrapolation
X
X
X
X
X X
Figure 2: Illustrative examples of tasks in our dataset. Each task has a green target, a red line indicating the movement limits of the robot, and potentially some blue obstacles. We train the first four in-sample scenario types: A - D. During evaluation, we also include interpolation scenario types: E. - G, which can be solved by the tools seen in training data, as well be solved by claw tools never seen in the training data. We choose these scenarios to set constraints on the dimensions of an appropriate tool; for example, in scenario C (third column), tool needs to be thin enough to reach the goal. Other attributes which affect applicability are tool length and type. In the top row, we overlay tools in yellow, which can press the buttons without colliding with the obstacles or leaving the movement space, indicated by the green tick. The bottom row, marked with red crosses, shows tasks being attempted by orange tools, which cannot succeed without violating the constraints. Tools are rendered here for visualisation and not present in the task images.
extrapolation scenario type: H, which
only
tion tasks and an extrapolation task. In-sample scenarios differ from interpolation scenarios in that the tasks are different but can be solved with tools (sticks or hooks) that were seen during training. The extrapolation scenario consists of a novel scenario but almost always also requires the use of a new tool: a J-shaped claw.
For each task, toolkit images are constructed by scattering tools from a catalogue with randomised colours and poses in a separate visible workspace. A geometric applicability check, described in Appendix [B] verifies whether or not any of the tools reach the target, while satisfying the task constraints. We consider a task-toolkit pair feasible the toolkit image includes at least one tool that can achieve the task under the specified constraints.
More formally, our dataset can be written as set of N triplets: p?, where each example features a task image Ic. toolkit image Ir, and a binary label p indicating the feasibility of reaching the target with one of the given tools (see Figure 3). We also save instance segmentation masks MG for the toolkits, as well as the applicability label p% for each tool, indexed by In all our experiments, we restrict the training input to the sparse high-level triplets and use the additional ground- truth labels for evaluation purposes only. For additional details on the dataset, see Appendix
4 METHOD
We frame the challenge of tool imagination as optimisation problem in structured latent space obtained using generative models. The optimisation is driven by a high-level, task-specific performance predictor, which assesses whether a goal position is reachable given particular tool. To reflect real-world complexity, the system must choose a tool from a of potentially unsuitable tools (Figure T prime the optimisation. In effect, we ask how an unsuitable tool might be improved to succeed on a task. This motivates an object- centric, generative model with latent space suited to task success classification detailed in Figure
4

---

Under review conference paper at ICLR 2020
input
latent space encoding
tool-task atlention
success estimation
lG
Leask
Zect
1.3
lT
2
Zám
argmax (Zeae1)) Zt1.3
tool innovation rendering from latent space
V/
optimisation steps
Figure 3: TasMON model architecture. A convolutional encoder á represents the task image IG as a latent vector Zg. In parallel, the encoder y of MONet scene decomposition model represents the toolkit image Ir as a set latent vectors 27,1..3 where each latent vector represents one of the tools. The concatenated tool-task representations 1.. are used to compute soft-attention 01..3 over tools. A weighted sum 2 the tool representations concatenated to the task representation again forming heat which used by a classifier a to estimate the success of any of the tools solving the task. The task loss Ltask is computed over a's output. We use the MONet decoder y render latent tool representations into images. Given the gradient signal produced when optimising task success for the classifier a, a chosen latent tool representation Zim gets updated to render an increasingly suitable tool for the task, depicted the bottom right in a callback to Figure [ A latent vector 24 for the toolkit background, and a second loss LONNe used to train the MONet model, are omitted from the figure for clarity of presentation.
4.1 OBJECT- TOOL REPRESENTATION
necessary for all subsequent pro- cessing to perform a decomposition of Ir, from pixels into discrete objects. To achieve this decomposition of images into their component objects, we employ an unsupervised architecture called MONet (Burgess al. 2019). MONet consists of two parts: an attention network and a component VAE (Kingma & Welling [2014 Rezende al. 2014). The attention network recurrently proposes K regions represent individual parts an image. These components hopefully correspond to individual objects (plus the scene background). Each component is represented as VAE latent code and used to explain certain part of the input image by reconstructing it. MONet therefore serves two purposes at once: its encoder & represents an image as a set of object-centric latent vectors, and its decoder v/ can be used to render any such latent vector into an individual object. We train MONet using the image-reconstruction loss LONNet described in Burgess al, 2019). We denote the set of latent variables encoding the toolkit computed by the MONet encoder, 4/, as
Given that our tools are presented toolkit images Ir (see Figure 3).
(Tr)) = {
(1..K).
After encoding the toolkit image, we need represent the task image IG an abstract latent space. The task encoder, d, consists a stack of convolutional layers followed by two dense layers ø takes a concatenation of the task image Ig and a normalised meshgrid of the image's (x. y) coordinates into the task embedding za. Without loss of generality the (z, y) meshgrid can be generated for any input and provides with sense of relative scale via coordinate querying (see Appendix
input and maps this
fo for further details).
Architecture details provided in Appendix
5

---

Under review conference paper at ICLR 2020
In order to interpret the task success signal, we need select the component that dominates the feasibility of the toolkit. This is done via a soft-attention approach which passes each tool representation Z con- catenated with the task embedding zg as & through three-layer MLP to compute attention logit (ef. Figure 3). A softmax activation is then applied to the logits to normalise the attention score such that
(Beat.k)
(2)
The attention over &eat.k is indicative of the influence of the k-th tool on global task success. We use the approximated soft attention to compute toolkit context vector 2 containing information about all the tools and their respective contributions to the task solution, where
akk ZT.
(3)
4.2 TOOL IMAGINATION
Task-driven learning The context vector 2 contains both task-relevant information (e.g. tool length, width, and shape), as well as task-irrelevant information such as tool location and colour. order to perform tool imagination, the sub-manifold of the latent space that corresponds to the task-relevant features has to be found. This achieved by adding three-layer MLP as classifier σ. The classifier a takes a concate- nation heat of the task embedding and the context vector as input and predicts the softmax over the binary task success. The classifier learns identify the task-relevant sub-manifold of the latent space by using the sparse success signal p and optimising the binary-cross entropy loss, such that
Lrask (a (heat) p) = log (o + (1 p) log (1 o (bboat))),
(4)
where p {0, 1} is a binary signal indicating whether or the toolkit contains feasible tool to solve the task. The whole system is trained end-to-end with a loss given by
C ( (UG.IT,p) LMONEt + ACtask-
(5)
The hyper- asper-parameter X denotes a weight factor balancing the task and MONet losses. Note that the gradient from the task classifier propagates through both the task encoder ó and the toolkit encoder vù and therefore helps to shape the latent representations of the toolkit with respect the requirements for task success.
Tool imagination Once trained, TasMON can synthesise new tools by traversing the latent manifold of individual tools in directions that maximise classification success given toolkit image (Figure 3 To do this, we first concatenate each individual toolkit component 2ţ with the task embedding zg and select the tool embedding vector Z that corresponds to the highest tool utility (Zeot,,k) (i.c. the tool most likely succeed). This warm-starts the imagination process. We denote the chosen tool latent Zim and its concatenation with the task embedding vector as Zeat.im- We then use activation maximisation (Erhan et al., 2009 Zeiler & Fergus 2014 (Simonyan al. 2014) to optimise the tool encoding Zim with regard to the Ltask of the success estimation o (Zeat,im) and a feasibility target p. = 1, such that
Zim Zim + Dinak (a (Zeat,im) PP))
(6)
Zz.m
where 7) denotes the learning rate for the update. Finally, we apply this gradient update for S steps until the success estimation a im) reaches a threshold y, and use (z(m))) to generate the imagined tool represented by Zim-
6

---

Under review conference paper at ICLR 2020
5 EXPERIMENTS
Our aim in this section is to demonstrate TasMON's ability to imagine suitable tool given task image and an image putative set of tools. Coneretely, we examine the model's performance when imagination requires interpolating between properties of known tool types well as when it requires innovating new tool types that have not previously been seen during training. To provide insight into the key aspects of TasMON's operation, our main evaluation in Table I
focuses on imagination success, while we present performance on key sub-tasks such tool utility, decomposition, and selection supporting work in Appendix
Tool Utility (TU) We compute tool utility as the average precision aggregated over classifier scores for all tools when compared to their ground truth feasibilities. Details are provided in Appendix A
Tool Imagination (TI) We evaluate whether our model can generate tools achieve tasks. For each instance, the target signal is set to success such that the latent vector of the tool is then modified via back- propagation using learning rate of 0.01 for 5, 000 steps until o (Zeattmmm) reaches 0.997. The imagined tool is then rendered via the latent space decoder v and tested using the same geometric applicability test as described in Appendix B
We report TI as the percentage of imagined tools that successfully pass this test.
5.1 QUANTITATIVE RESULTS
In order to investigate the efficacy of TasMON under different operating conditions, we present models trained with different training regimes. TasMON trained in curriculum learning manner. we first pre-train the MONet weights on toolkit images and then train the performance predictor with the MONet backbone jointly i.e. the gradient from the predictor is allowed back-propagate into the MONet. This curriculum allows MONet to learn object-centric decompositions before the performance predictor overfits the training data. To investigate the impact of jointly training the latent representation with the performance predictor we report results on an ablation model FroMON in which we keep the pre- pre-trained MONet weights frozen during predictor training. Both TasMON and FroMON are trained on in-sumple scenario types only. We select trained checkpoints that have the best performance prediction (TU) on a validation split. In addi- tion to comparing the imagination outcomes for TasMON and FroMON, we also include a simple Random
Table Like-for-like comparison of imagination processes with FroMON and TasMON, when artificially warmstarting from the same (infcasibile or feasible) tool in both cases. TI-Infeasible are the imagination results warmstarted with infeasible tools and TI-Feasible are the imagination results initialised with feasible tools.
TI-Infeasible [%]
Ti-Feasible [%]
RW
FroMON
TasMON
RW
FroMON
TasMON
Scenario
A
250
65.2%
87.6%
92.0 %
250
97.2%
98.8%
99.2%
B
250
25.2%
38.0%
40.0 %
250
87.6%
90.4%
84.8%
C
250
14.4%
26.4%
33.2%
250
37.6%
56.4%
58.4%
D
250
8.4%
19.6%
18.0%
250
28.8%
30.0%
37.6%
E
250
8.4%
20.4%
16.40%
250
62.8 %
61.2%
58.8%
F
250
6.0%
10.8%
13.2%
250
32.4%
36.8 %
30.8%
G
250
2.0%
5.2%
4.4%
250
25.2%
27.6% %
24.4%
H
250
0.0%
0.0%
0.0%
250
0.4%
0.4%
0.0%
Overall
2000
16.2%
26.0%
27.2%
2000
46.5%
50.2%
49.3%
7

---

Under review conference paper at ICLR 2020
(a) Increasing tool length
(b) Decreasing tool width
(c) Morphing hetween configurations
Figure 4: Qualitative results from our imagination experiments. The figure shows the evolution of tools during the imagination process (overlayed manually over the corresponding task image). Of these, each illustrates how the imagination procedure increasing the tool length, (B) decreasing the tool width, and (C) changing the configuration between a stick and hook. The overlapping area is marked in bright yellow. The arrows indicates tool synthesis with increasing optimisation steps.
succeed at constructing tools that solve task, by (A)
Walk (RW) baseline, where in place of taking steps in the direction of the steepest gradient, we move in random direction in the latent space for 5000 steps. Specifically, the latent vector of the selected tool is updated by a sample drawn from an isotropic Gaussian with mean and the absolute value of the ground- truth gradient derived by back mack-propagating from the predictor the variance. For 250 infeasible instances per scenario type, we warmstart each imagination attempt with the same tool across RW, TasMON and FroMON to enable a like-for-like comparison. We also test to see how the models modify feasible tools; for 250 instances per scenario type, imagination is warmstarted with a feasible tool observe how it is modi- fied. The reader is referred to Table and Appendix A for more quantitative results examining the model's performance in related sub-tasks like toolkit decomposition and tool selection.
TasMON successfully imagines applicable tools in 46.0% of in-sample test cases, and in 11.3% of tool inter- polation test cases (Table [). A significant drop in TI performance is incurred for the interpolation scenarios (E, F, G) in which the model encounters distinctly novel scenarios. The model fails in the extrapolation sce- nario where it needs to imagine previously unseen tool type. TasMON outperforms FroMON in tool utility prediction (Table (2 and tool imagination most tasks, suggesting that, although the predictor is powerful enough to guide the imagination through an already existing underling structure of toolkii-representations. task-aware latent space
still provide benefits. As expected, the random walk fails in most scenarios except A and B. This is because there are no obstacles in scenario A, so any blob large enough to reach the target will suffice. We conjecture that similar situation occurs in scenario B when the obstacle is small, which imposes only weak constraint on the required tool. Interestingly, the imagination could also destroy some of the already feasible tools showed in the TI-Feasible column in Table | Upon examination, we find that this is due the combined effect of underconfident behaviour from the performance predic- tor, reconstruction errors, and rendering choices. We refer readers to Appendix E for further details about rendering.
8

---

Under review conference paper at ICLR 2020
5.2 QUALITATIVE RESULTS
Qualitative examples of the tool imagination process are provided in Figure 4 as well as in Figure 6 in the supplementary material. A striking feature of the optimisation process is the fact the performance predictor often drives the tools to evolve in a deliberate and interpretable way by modifying aspects such as length, width and configuration. This suggests that these properties are encoded directions in the structured latent space learnt by TasMON and deliberately traversed via a high-level task objective in the form of the performance predictor. Also, since the tools are modified in smooth manner, we hypothesise that tools are embedded in a continuous manifold of changing length, width and configuration. Moreover, instead of always turning stick to hook, the model can adjust the tool shape between these two configurations according to the tasks as depicted in block (c) of Figure which indicates that the model learns a non-biased interpretation of the tasks.
6 CONCLUSION
In this paper we investigate an agent's ability to synthesise tools for simulated reaching tasks via task- driven imagination. Our approach, TasMON, uses a novel architecture in which a high-level performance predictor drives an optimisation process in a structured latent space. TasMON successfully generates tools for scenario types beyond its own training regime. Intriguingly, TasMON leads to interpretable modifications of the tools considered. The system tends to modify aspects such tool length, width and configuration. Our results thus suggest that these object affordances are encoded as directions in the latent space learnt by TasMON and sought out during the optimisation process. We posit that this may help our understanding of object affordances and offers up novel avenues towards disentangling interpretable factors of variation. Nevertheless, the TasMON struggles to generalise to significantly different scenarios requiring, for example, novel tool configurations. We conjecture that this is due to the performance predictor not generalising well to these novel scenarios as the TU drops drastically in these tasks (see Table Remedying this degradation performance prediction and thereby enabling actual tool innovation remains interesting challenge for future work. To facilitate further advances in this area, we intend to release the reaching dataset and trained TasMON model to the community.
REFERENCES
Stanley H. Ambrose. Paleolithic technology and human evolution. Science, 291 (5569)) 2001.
Sarah R. Beck, Ian A. Apperly, Jackie Chappell, Carlie Guthrie, and Nicola Cutting. Making tools isn't child's play. Cognition, 119(2):301-300 2011.
Christopher P. Burgess, Loic Matthey, Nicholas Watters, Rishabh Kabra, Irina Higgins, Matt Botvinick, and Alexander Lerchner. MONet: Unsupervised scene decomposition and representation. arXiv preprint arXiv: / 1390, 2019.
Jackie Chappell and Alex Kacelnik. Tool selectivity in a non-primate, the New Caledonian crow (Corvus monedaloides). Animal Cognition, 5(2):71-78, 2002.
Jackie Chappell and Alex Kacelnik, Selection of tool diameter by New Caledonian crows Corvus monedu- loides. Animal Cognition, 7(2):121-12) 2004.
Thanh-Tome Do. Anh Nguyen, and Ian D. Reid. Affordancenet: An end-to-end deep learning approach for object affordance detection. In 2018 IEEE International Conference on Robotics and Automation, ICRA 2018, Brisbane, Australia, May 2/-25, 2018, pp. 1-5, 2018.
Nathan J. Emery and Nicola S. Clayton. Tool use and physical cognition in birds and mammals. Current Opinion in Neurobiology, 19(:227 33,, 2009.
9

---

Under review conference paper at ICLR 2020
Martin Engelcke, Adam R. Kosiorek, Oiwi Parker Jones, and Ingmar Posner. GENESIS: Generative scene inference and sampling with object-centric latent representations. arXiv preprint arXiv:1907.13055 2019.
Dumitru Erhan, Yoshua Bengio, Aaron Courville, and Pascal Vincent. Visualizing higher-layer features of deep network. Technical report, University of Montreal, 2009.
Kuan Fang. Yuke Zhu, Animesh Garg, Andrey Kurenkov, Viraj Mehta, Li Fei-Fei, and Silvio Savarese. Learning -rientede grasping for tool manipulation from simulated self-supervision. arXiv preprint ar0iv:1806.0 09266, 2018. 100.609680.006
James Gibson. The theory of affordances, volume 1977.
Helmut Grabner, Juergen Gall, and Luc Van Gool. What makes a chair a chair? In The 24th IEEE Conference on Computer Vision and Puttern Recognition, CVPR 2011, I, Colorado Springs, CO, USA, 20-25 June 201 pp. 1529-1536, 2011.
Klaus Greff, Raphaël Lopez Kaufman, Rishabh Kabra, Nick Watters, Christopher Burgess, Daniel Zoran, Loic Matthey, Matthew Botvinick, and Alexander Lerchner. Multi-object representation learning with iterative variational inference. In International Conference Machine Learning (ICMI.), 2019.
David Ha and Jürgen Schmidhuber. World models. arXiv preprint arXiv: 1803. 10122, 2018.
Irina Higgins, Loic Matthey, Arka Pal, Christopher Burgess, Xavier Glorot, Matthew Botvinick, Shakir Mohamed, and Alexander Lerchner. 3-vae: Learning basic visual concepts with a constraied variational framework. In ICLR, 2017.
Lawrence Hubert and Phipps Arabie. Comparing partitions. Journal of Classification, 2(1):193-2 19--118 1985.
Lorenzo Jamone, Giovanni Saponaro, Alexandre Antunes, Rodrigo Ventura, Alexandre Bernardino, and José Santos-Victos Learning object affordances for tool use and problem solving in cognitive robots. In Proceedings of the 2nd Italian Workshop on Artificial Intelligence and Robotics, pp. 68 82, 2015.
Diederik Kingma and Max Welling. Auto-Encoding Variational Bayes. International Conference on Learning Representations, 2014.
Wolfgang, Kohler. The mentality of apes. Vintage Books, 1925.
Harold W. Kuhn. The Hungarian method for the assignment problem. Naval Research Logistics Quarterly, 2:83-97, 1955.
Timothée Lesort, Natalia Díaz-Rodríguez, Jean-Franois Goudou, and David Filliat. State representation learning for control: An overview. Neural Networks, 2018.
YuXuan Liu, Abhishek Gupta, Pieter Abbeel, and Sergey Levine. Imitation from observation: Learning to imitate behaviors from raw video via context translation. In 2018 IEEE International Conference on Robotics and Automation (ICRA), pp. 1118-1125. IEEE, 2018.
Andrew L. Maas, Awni Y. Hannun, and Andrew Y. Ng. Rectifier nonlinearities improve neural network acoustic models. In ICML Workshop on Deep Learning for Audio, Speech, and Language Processing (WDLASL 2013), pp. 1-6, 2013.
Tanis Mar, Vadim Tikhanoff, Giorgio Metta, and Lorenzo Natale. Self-supervised learning of grasp depen- dent tool affordances on the icub humanoid robot. In 2015 IEEE International Conference on Robotics and Automation (ICRA), pp. 3200-32202 3206. IEEE, 2015.
10

---

Under review conference paper at ICLR 2020
Alexander Mordvintsev, Christopher Olah, and Mike Tyka. Inceptionism: Going deeper into neural networks, 2015. URL og incept on sm- penprrgg--epee---tt---eeeeerrrr hht.m.arraaaaam.ll.l........
James Munkres. Algorithms for the assignment and transportation problems. Journal of the society for
industrial and applied mathematics, 5(1):32-38, 1957.
Austin Myers, Ching L. Teo, Cornelia Fermüller, and Yiannis Aloimonos. Affordance detection of tool parts from geometric features. In 2015 IEEE International Conference on Robotics and Automation (ICRA), pp. 1374-1381. IEEE, 2015.
Ashvin V. Nair, Vitchyr Pong, Murtaza Dalal, Shikhar Bahl, Steven Lin, and Sergey Levine. Visual re- inforcement learning with imagined goals. In Advances in Neural Information Processing Systems, pp. 9191-9200, 2018.
William M. Rand. Objective criteria for the evaluation of clustering methods. Journal of the American Statistical association, 66(336):84688555) 1971.
Danilo Jimenez Rezende, Shakir Mohamed, and Daan Wierstra. Stochastic Backpropagation and Approxi- mate Inference in Deep Generative Models. International Conference on Machine Learning, 2014.
Namiko Saito, Kitae Kim, Shingo Murata, Tetsuya Ogata, and Shigeki Sugano. Tool-use model consider- ing tool selection by a robot using deep learning. In 2018 IEEE-RAS 18th International Conference on Humanoid Robots (Humanoids), pp. 200 276. IEEE, 2018.
Karen Simonyan, Andrea Vedaldi, and Andrew Zisserman. Deep inside convolutional networks: Visualising image classification models and saliency maps. In Workshop at International Conference on Learning Representations, 2014.
Jivko Sinapov and Alexander Stoytchev. Learning and generalization of behavior- tool affordances. In IEEE International Conference Development and Learning. pp. 19-24, 2007.
Alexander Stoytchev. Behavvor-grounded representation of tool affordances. In IEEE International Confer- ence on Roboties and Automation (ICRA). pp. 3071 0071-3076, 2005.
Kuniyuki Takahashi, Kitae Kim, Tetsuya Ogata, and Shigeki Sugano. Tool-body assimilation model consid- ering grasping motion through deep learning. Robotics and Autonomous Systems, 91: 115-127, 2017.
Vadim Tikhanoff, Ugo Pattacini, Lorenzo Natale, and Giorgio Metta. Exploring affordances and tool use on the iCub. In JEEE-RAS International Conference on Humaaood Robots (Humanoids)) pp. 130-137, 2013.
Mare Toussaint, Kelsey Allen, Kevin Smith, and Joshua Tenenbaum. Differentiable physics and stable modes for tool-use and manipulation planning. In Roboties: Science and Systems XIV. Robotics: Science and Systems Foundation, 2018
Lieselotte van Leeuwen, Ad Smitsman, and Cees van Leeuwen. Affordances, perceptual complexity, and the development of tool use. Journal of Experimental Psychology: Human Perception and Performance, 20(1):174, 1994.
Handy Wicaksono. Towards relational approach for tool creation by robots. In International Joint Confer- ence on Artificial Intelligence, Melbourne, Australia, 2017.
Annie Xie, Frederik Ebert, Sergey Levine, and Chelsea Finn. Improvisation through physical understanding: Using novel objects as tools with visual foresight. arXiv preprint arXiv:1904.055.0 1904.05538, 2019.
Matthew D Zeiler and Rob Fergus. Visualizing and understanding convolutional networks. In Proceedings of the IEEE European Conference on Computer Vision, pp. 318-833 2014.
11

---

Under review conference paper at ICLR 2020
A ADDITIONAL QUANTITATIVE RESULTS
In this appendix, we present performance on key sub-tasks useful for understanding model behaviour.
Tool Utility (TU) The performance predictor is responsible for estimating the probability of the toolkit containing an applicable tool for the given task based on the encoding of the task image concatenated with toolkit context vector. However, toolkit image need not depict more than a single tool, and so a can provide measure of tool utility of individual, possibly imagined, tool. We intersection-over-unii (IOU) as similarity measure to match tool decompositions to ground truth tool masks according to the Kuhn-Munkres algorithm (Kuhn 1955 Munkres 1957). All matches below the conservative threshold of IOU 0.8 are discarded as they correspond to spurious or poor quality decompositions. Matched ground-truth masks are assigned their corresponding tool decomposition' tool utility probability from the classifier. We report the aggregated tool utility as average precision. tool utility probability of is assigned any unmatched ground truth masks. Since we report average precision, this rubric can only deflate our scores by increasing false negatives at the cost of true positives, leading conservative estimates of performance.
Toolkit Decomposition (TD) We the Adjusted Rand Index (ARI) (Rand 1971 Hubert & Arabie 1985) to evaluate the quality of the instance segmentation, as in Greff et al. (019). An ARI score treats the pixel- co-nstaace-aasiinmeen as clustering problem and computes the similas larity between the predicted clustering and ground truth. ARI scores range from (complete misalignment) to (identical clustering).
Tool Selection (TS) The TasMON model performs tool selection as described in Section 4.2 In feasible scenarios, select the tool decomposition with the highest tool utility, and compare it with the ground truth mask of the applicable tool. We consider tool selection to successful if the predicted mask has an IOU overlap with the ground-truth mask of at least 0.8, which we take to be conservative threshold.
Results table, comparing TasMON with frozen MONet+ Task ablation, i.c. FroMON, across all tasks. Other abbreviations: TU tool utility, aggregated as the average- veraagepreccisoon score over all ground truth tools; TD toolkit decomposition, measured by the Adjusted Rand Index (Rand) [971] Hubert & Arabie 1985): TS tool-selection success in feasible scenarios, % of times pixel-wise IOU of most-likely tool and the ground-truth feasible tool was 0.8 or more; TI = unconditional tool-imagination success for each model, % of all test scenarios; The best model for task and metric are shown in bold, except for TD where ARI scores were similar for different models. All metric scores are shown as percentages for consistency. Please refer to Figure for illustrative tasks by
Table 2:
Like-for-like tool cool-mmagination results are shown in Table I scenario.
TU [%]
TD [%]
TS [%]
TI [[]] TasMON
FroMON
TasMON
FroMON
TasMON
FroMON
TasMON
FroMON
Scenario
95.4%
97.6%
97.7%
97.7%
95.2%
96.2%
83.9%
87.4%
B
88.9%
91.3%
97.8%
97.8%
82.8% %
79.8%
48.6%
48.9%
83.2%
85.8%
97.7%
97.9%
88.0%
88.2%
38.2%
40.8%
D
77.2%
79.4%
97.8%
97.8%
78.6%
75.8%
20.2%
20.2%
76.3%
77.3%
97.7%
97.8%
72.0 %
71.4%
29.1%
24.9%
37.9%
36.5%
97.7%
97.9%
62.4%
65.2%
15.8%
16.0%
48.6%
48.4%
97.8%
97.8%
76.6% %
73.6%
14.2%
12.4%
37.8%
38.4%
96.9%
97.1%
69.6%
70.0%
0.1%
0.0%
Overall
68.2%
69.3%
97.6%
97.7%
78.2% %
77.5%
31.3%
31.3%
12

---

Under review conference paper at ICLR 2020
Table 2 presents model performance across all four metrics. Note that the colliimaiination (TI) results here differ from Table Aubleueeeasse in this table, we do not warmstart imagination with the same tool for each model, instead letting each model select tool with highest TU. As result, models might select different tools bootstrap imagination, leading to a comparison that is not exactly like-for-like.
We see that both models demonstrate strong ability gauge tool utility (Table 2 scenarios. However, prediction performance decreases as scenario types grow more distinct from the training data. Table Table(ssuggest suggest a strong correlation between successful tool imagination (TI) and effective and robust tool utility prediction (TU). This unsurprising as the performance predictor constitutes the main driver for the optimisation. Finally, we note that task decomposition (TD) is commensurate across all models and scenario types. This is an intuitive result both models trained with an explicit decomposition objective, which is independent of scenario type.
TU column) in in-sample
B DATASET CONSTRUCTION
Figure 5: More illustrative examples of tasks (cf. Figure 2 We procedurally generated 196,390 unique instances across eight scenario types for our reaching experiments.
Our synthetic dataset consists of pairs of tasks and toolkits rendered from top-down views as 64 x 64 pixel RGB images. We create the dataset in multiple sampling steps ensuring balance between scenarios and tool configurations. For each scenario, we generate 5,000 tasks. Each paired with 50 tools from a tool catalogue containing million unique tool geometries, evenly balanced between sticks, hooks and claw tools. Then we perform the tool applicability check each of the task-tool pairs and reject tasks which had fewer than five more than 40 applicable tools. The applicability of tool to task is determined by sampling 200 interior points of the tool polygon, overlaying the sampled point with the target and rotating the tool polygon and vertically mirrored copy of it by 360 degrees. If any such pose of the tool satisfies all constraints (i.e. touching the space behind the red line while not colliding with any obstacle), we consider the tool applicable for the given task. In this way, we construct up to five feasible and five infeasible toolkits for each task, by sampling one to three tools per toolkit and rendering them with randomised poses and colours. We constrain the feasible toolkits to contain exactly one applicable tool. All toolkits are constrained to have only sticks and hooks tools, with the exception of the tasks, which require claws in order to be feasible.
The dataset contains total of 52,000 scenarios. Table 3 shows breakdown of each by scenario type and split. Each split has equal number of feasible and infeasible instances, for each scenario type.
13

---

Under review conference paper at ICLR 2020
Table 3: Number of instances by scenario type.
Type
Training
Validation
Test
10,000
1,000
1,000
10,000
1,000
1,000
10,000
1,000
1,000
10,000
1,000
1,000
1,000
1,000
1,000
H
1,000
Total
40,000
4,000
8,000
C ARCHITECTURE AND TRAINING DETAILS
C.I MODEL ARCHITECTURE
As described the main text, our model (TasMON) comprises two main parts (as depicted in Figure 3 On the one hand, we used a task-based classifier (Tas), while on the other we used MONet scene decomposition model (MON). We briefly describe each component turn.
C.I.1 MONET
As scene decomposition model, faithfully re-implement MONet (Burgess al. 2019) with the minor modification of using leaky ReLUs (Maas et al. 2013) as activation functions in place of ReLUs.
C.1.2 TASK-BASED CLASSIFIER
The task-based classifier consists of three sub-parts: a task encoder, an attention module, and a classifier.
The task encoder stacks three convolutional blocks followed by single convolutional layer and two-layer MLP. First, the three consecutive convolutional blocks make use 16, 32, 64 output channels respectively. Each convolutional block contains two consecutive sub-convolutional blocks, each with a kernel size of and padding of 1 (and with stride of and stride of 1, respectively). Second, the additional convolutional layer has stride 2, padding 1, and 128 output channels. All convolutional layers use a leaky ReLU nonlinearity. Finally, the MLP takes flattend output of the convolutional layers of dimension 2048 (4 4 x 128) and passes to a hidden layer with 128 neurons using leaky ReLU as the activation function. This is then passed through another layer, giving 16 dimensional output for the task-based classifier.
The attention module a three layer MLP with input dimension 32, and successive hidden layers 128 and 16 neurons respectively. Each of these layers use leaky Rel.U activation functions. To predict the logit of the attention probability, the attention module has an output dimension of 1.
The structure of the classifier part of the task-classifier model mirrors the attention module. In place the output layer, it consists of three layer MLP with the same dimensions. Given that the classifier outputs logits for binary classifier, the output dimension is 2.
C.2 TRAINING DETAILS
All experiments are performed PyTorch, using the ADAM optimiser with learning rate of 0.0001 and a batch size of 32. We vary the weight factor À between 0.1 to 4, and we select TasMON to be the best performing model, À MONet's hyperparameters tuned to our dataset are given in Table
14

---

Under review conference paper at ICLR 2020
Table 4: Hyperaparmeters used for pretrained MONet.
MONet Hyperparameters
Attention Blocks
5
Attention Channels
8
Beta
0.1
Gamma
0.3
Pixel variance background
0.18
Pixel variance foreground
0.2
VAE Channels
32
VAE Latents
16
D COORDINATE QUERYING
The CNN is local and its parameters shared across spatial locations; however, the location of object an image is global information. In order to infer the global location of an object, the CNN must work together with the permutation-varian dense-layers. Assume that in CNN-Dense network structure the output shape of the feature tensor just before the flatten operation anchor point feature, i.e. the one located at(1, 1), whose receptive field does not touch the boundaries of the task image. Based only on that feature vector the network will not be capable to predict the global coordinates of the object observed within that receptive filed. The global coordinates of an object can only be inferred by analysing the information of subset of all the anchor point features that have a joint receptive field larger than or equal to the whole image. This will also utilise some capacity of the dense layers. But given the x-y meshgrid attached to the task image, even the kemnel in the input layer with receptive field of (assuming a kernel size of 3 x 3) can know where it is, which allows the network to query the global coordinate information of an object locally. In conclusion, the x-y meshgrid is free additional information that can help the task module to capture the relative scale of the scene.
x and we only take the middle
E RENDERING METHOD
Although the MONet aims to decompose scene into object-centric representations, the imagined compo- nent cannot be rendered independently from the rest parts the image. This because the reconstruction outputs of the decoder are the unnormalised logits and directly using softmax to normalise the mask of the imagined component together with the rest parts will constrain the growth of the tool. Imagine that the imag- ination process that tries to enlarge a tool is assigning higher logits to the area where it thinks the tool should of the background component
grow towards, while the logits of that process is not applied to it. During normalisation the imagined component must beat the background to pos- sess the pixels it wants to grow towards and thus the growth is constrained implicitly. We tackle this problem by rather comparing the logits of the imagined tool with the rest components but comparing with itself. We use the minimum logit of the area where the attention mask has (normalised) probability higher than 0.2 threshold before the imagination starts. Any imagined pixel that has a logit higher than that threshold is treated as belonging to the imagined tool. It's common that when people develop a scene decomposition model they normalise the component masks altogether in the end since "sum up one" is an important in- ductive bias by design. Nevertheless, our work indicates that this might constrain the down-streaming tasks such as optimising the ceject-centric representation independently for planning purpose.
still high since the imagination
15

---

Under review conference paper at ICLR 2020
F ADDITIONAL IMAGINATION EXAMPLES
7
Figure 6
16

---

Under review conference paper at ICLR 2020
Figure 6: More examples of imagined tools for all tasks. The first two columns are the task and toolkit inputs, and the next four columns imagination reconstructions for the selected tool overlaid on the task image. The overlapping area is marked bright yellow.
17


Reviews:
[{"review_id": "rygLWwJ6Yr", "reviewer": "AnonReviewer1", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "This paper proposes an algorithm that learns to synthesize tools for the task of reaching. The main idea is to first use unsupervised segmentation algorithm to decompose an image of a set of tools into individual objects; then, with a trained feasibility model, search through the latent space of encoded tools to 'imagine' new ones for reaching. \n\nThis paper has clear strengths and weaknesses. It's studying an important problem from a cognitive perspective; it also proposes a novel model for the task, building upon SOTA models. However, the current problem formulation and experiment setup are not well justified, and the experiments are quite limited. I lean toward rejection.\n\nMost importantly, while this paper argues for the importance of an object-centric representation, it conducts most of its search in the pixel space (both as input to the model, and as the output of the imagination). This leads to some unnatural and unphysical results: in the teaser figure, it's true that the final, imagined tool reaches the target; however, the tool itself shouldn't be able to pass the gap/hole on the wall, due to its angular shape. Objects, in essence, have shapes and physical occupancy. Without modeling physics, it's unclear how useful the object-centric representation is.\n\nImagination is done by searching over the latent space, which limits the model's generalization power to novel tools or new configurations. This is revealed in the results on case H, where the model doesn't work at all.\n\nThe experimental results, as just mentioned, are not very impressive, especially given the simplified setup. There are no results on other tasks except reaching. In addition, comparisons with published methods are missing. For example, what if I just train a Pix2Pix from the inputs to successful reaches? That sounds like a reasonable baseline and should be compared with.\n\nDue to all these limitations, I lean toward rejection.", "extra_scores": {"experience_assessment": "I have published in this field for several years.", "review_assessment:_thoroughness_in_paper_reading": "I read the paper thoroughly.", "review_assessment:_checking_correctness_of_experiments": "I assessed the sensibility of the experiments.", "review_assessment:_checking_correctness_of_derivations_and_theory": "I assessed the sensibility of the derivations and theory."}, "extra_text": {}}, {"review_id": "rJe7toL2Fr", "reviewer": "AnonReviewer3", "summary": "", "questions": "", "limitations": "", "rating": 1, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "This paper proposes an architecture for synthesizing tools to be used in a reaching task. Specifically, during training the agent jointly learns to segment an image of a set of three tools (via the MONet architecture) and to classify whether one the tools will solve the given scene. At test time, one of the three tools is selected based on which seems most feasible, and then gradient descent is used to modify the latent representation of the tool in order to synthesize a new tool to (hopefully) solve the scene. The paper demonstrates that this approach can achieve ok performance on familiar scenes with familiar tools, but that it fails to generalize when exposed to unfamiliar scenes or unfamiliar tools. The paper reports a combination of the quantitative results showing that optimizing the latent space can lead to successful synthesis in some cases, and qualitative results showing that the synthesized tools change along interpretable dimensions such as length, width, etc. The combination of these results suggest that the model has learned something about which tool dimensions are important for being able to solve the types of reaching tasks given in the paper.\n\nWhile I think this paper tackles a very interesting, important, and challenging problem, I unfortunately feel it is not ready for publication at ICLR and thus recommend rejection. Specifically, (1) neither the particular task, results, or model are not very compelling, (2) there are no comparisons to meaningful alternatives, and (3) overall I am not quite sure what conclusions I should draw from the paper. However, given the coolness of the problem of tool synthesis, I definitely encourage the authors to continue working on this line of work!\n\n1. The task, results, and model are not very compelling. Any of these three things alone would not necessarily be a problem, but given that all three are true the paper comes across as a bit underwhelming.\n \n- First, while the task can be construed as a tool synthesis task, it doesn’t come across to me as very ecologically valid. In fact, the task seems to be more like a navigation task than a tool synthesis task: what’s required is simply to draw an unbroken line from one part of the scene to another, rather than actually generate a tool that has to be manipulated in an interesting way. Navigation has been studied extensively, while synthesis of tools that can be manipulated has not, which makes this task both not very novel and disappointing in comparison to what more ecologically-valid tool synthesis would look like. For example, consider a variation of the task where you would have to start the tool at the red region and move it to the green region. Many of the tools used here would become invalid since you wouldn’t actually be able to fit them through the gaps (e.g. Figure 2E).\n \n- Second, given that the “synthesis” task is more like a navigation task, the results are somewhat disappointing. When provided with a feasible solution, the model actually gets *worse* even in some of the in-sample scenes that it has seen during training (e.g. scene types C and D) which suggests that it hasn’t actually learned a good generative model of tools. Generalization performance is pretty bad across the board and is only slightly better than random, which undermines the claim in the abstract that “Our experiments demonstrate that the synthesis process modifies emergent, task-relevant object affordances in a targeted and deliberate way”. While it’s clear there is successful synthesis in some cases, I am not sure that the results support the claim that the synthesis is “targeted” or “deliberate” given how poor the overall performance is.\n \n- Third, the model/architecture is a relatively straightforward combination of existing components and is highly specialized to the particular task. As mentioned above, this wouldn’t necessarily be a problem if the task were more interesting (i.e. not just a navigation task) and if the results were better. I do think it is cool to see this use of MONet but I’m skeptical that the particular method of optimizing in the latent space is doing anything meaningful. While there is prior work that has optimized the latent space to achieve certain tasks (as is cited in the paper), there is also a large body of work on adversarial examples which demonstrate that optimizing in the latent space is also fraught with difficulty. I also suspect this is the reason why the results are not particularly good.\n \n2. While I do appreciate the comparisons that are in the paper (to a “Random” version of TasMON that moves in a random direction in the latent space, and to “FroMON” agent which is not allowed to backpropagate gradients from the classification loss into MONet), these comparisons are not particularly meaningful. The difference between FroMON performance and TasMON tool imagination performance (I didn’t test tool utility) across tasks is not statistically significant (z(520, 544)=-0.8588, p=.38978), so I don’t think it is valid to claim that “a task-aware latent space can still provide benefits.” The Random baseline is a pretty weak baseline and it would be more interesting to compare to an alternative plausible architecture (for example, which doesn’t use a structured latent space, or which doesn’t have a perceptual frontend and operates directly on a symbolic representation of the tools/scene).\n \n3. Overall, I am not quite sure what I am supposed to get out of the paper. Is it that “task relevant object affordances are implicitly encoded as directions in a structured latent space shaped by experience”? If so, then the results do not support this claim and so I am not sure what to take away. Is it that the latent space encodes information about what makes a tool feasible? If so, then this is a bit of a weak argument---of *course* it must encode this information if it is able to do the classification task at all. Is it that tool synthesis is a challenging problem? If so, then the lack of strong or canonical baselines makes it hard to evaluate whether this is true (and the navigation-only synthesis task also undermines this a bit).\n\nSome additional suggestions:\n \nIt would be good to include a discussion of other recent work on tool use such as Allen et al. (2019) and Baker et al. (2019), as well as on other related synthesis tasks such as Ha (2018) or Ganin et al. (2018).\n \nThe introduction states that “tool selection and manufacture – especially once demonstrated – is a significantly easier task than tool innovation”. While this may be true, it is a bit misleading in the context of the paper as the agent is doing something more like tool selection and modification rather than tool innovation (and actually the in-sample scenes are more like “manufacture”, which the agent doesn’t always even do well on).\n \nIt would be helpful to more clearly explain scene types. Here is some suggested phrasings: in-sample = familiar scenes with familiar tools, interpolation = novel scenes with familiar tools, extrapolation = novel scenes with novel tools.\n \nI was originally confused how psi’ knew where to actually place the tool and at what orientation, and whether the background part of the rendering process shown in Figure 1. I realized after reading the supplemental that this is not done by the agent itself but by separate code that tries to find the orientation and position of the tool. This should be explained more clearly in the main text.\n \nIn Table 1 it would be helpful to indicate which scene types are which (in-sample, interpolation, extrapolation).\n \nAllen, K. R., Smith, K. A., & Tenenbaum, J. B. (2019). The Tools Challenge: Rapid Trial-and-Error Learning in Physical Problem Solving. arXiv preprint arXiv:1907.09620.\n \nBaker, B., Kanitscheider, I., Markov, T., Wu, Y., Powell, G., McGrew, B., & Mordatch, I. (2019). Emergent tool use from multi-agent autocurricula. arXiv preprint arXiv:1909.07528.\n \nGanin, Y., Kulkarni, T., Babuschkin, I., Eslami, S. M., & Vinyals, O. (2018). Synthesizing programs for images using reinforced adversarial learning. arXiv preprint arXiv:1804.01118.\n \nHa, D. (2018). Reinforcement learning for improving agent design. arXiv preprint arXiv:1810.03779.", "extra_scores": {"experience_assessment": "I have published one or two papers in this area.", "review_assessment:_checking_correctness_of_derivations_and_theory": "N/A", "review_assessment:_checking_correctness_of_experiments": "I assessed the sensibility of the experiments.", "review_assessment:_thoroughness_in_paper_reading": "I read the paper at least twice and used my best judgement in assessing the paper."}, "extra_text": {}}, {"review_id": "SJxPoldPtB", "reviewer": "AnonReviewer2", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "The authors constructed an interesting dataset named reaching task, where the model need to predict if the given toolkit is able to solve the corresponding task or not. They showed that combining variational auto-encoding with an auxiliary loss (in this case, the predictor of solving the tasks, could help shaping a latent space where affordance of the tool is encoded as directions in such latent space.) Using the activation maximisation technique (which was phrased as the imagination process), they are able to modify the input tools into the ones that is suitable to solve the corresponding task. I found the idea of using an auxiliary loss when training a VAE may cause the latent space coding direction change novel and interesting. However, I do not find the authors has a strong case of proven it is the case in this manuscript.\n\n1. The performance difference between FroMON and TasMON is not clear.\n  The most critical control model in this paper is the FroMON (frozen MoNet). In this control model, the gradient from the success predictor is not flowing back into the VAE encoder. So, based on the author's assumption, it should not be benefit of having the tool affordance directions in the latent space. However, in the main results in Table 1. We found the performance between FroMON and TasMON is not quite clear. This is particularly true for the Scenario E, F, G (the interpolation tasks), which is more about generalization and is more important.\n\n2. Are the affordance 'directions' in the latent space?\n  The authors used activation maximisation approach to travel in the latent space. My understanding of the approach is it follow the gradient to maximise the predictor's success prediction in an iterative approach. So, at each optimization step, the z_im can move in different direction. This seems to not fit as a sense of 'direction', as I would assume it is moving along a particular line (not necessarily axis aligned.). Maybe this does explain whey FroMON and TasMon perform equally well. As long as the possible shapes is encoded in a smooth way in the latent space, the activation maximisation could find a path toward the target object. Unfortunately, is that a 'direction'? Would it be possible to train an optimization algorithm that is only allow to move in a linear direction, and see how well that work?\n\n3. Why MoNet and multiple tools in the toolkit. A simplified version could drive the point as well.\nUsing MoNet to decompose tools from a toolkit is nice. However, is it really necessary to drive the main point (an auxillary loss of success prediction can shape the latent space of a VAE model) in this paper. In a simplified version, where there is only one tool in the toolkit, one may not need MoNet (maybe still need it for object-background separation?) May the authors comment why multiple tools in the toolkit is important?\n\nMinor:\n1. typo: page 1, (2nd to the last line). '...that habitual tool use cannot in and off itself ..' --> of\n2. A simple video showing how the tool shape change sequentially during the activation maximisation process would be interesting. ", "extra_scores": {"experience_assessment": "I have read many papers in this area.", "review_assessment:_checking_correctness_of_derivations_and_theory": "I assessed the sensibility of the derivations and theory.", "review_assessment:_checking_correctness_of_experiments": "I carefully checked the experiments.", "review_assessment:_thoroughness_in_paper_reading": "I read the paper at least twice and used my best judgement in assessing the paper."}, "extra_text": {}}]

Author rebuttal:

Meta-review: This paper investigates the task of learning to synthesize tools for specific tasks (in this case, a simulated reaching task). The paper was reviewed by 3 experts and received Reject, Weak Reject, and Weak Reject opinions. The reviews are very encouraging of the topic and general approach taken by the paper -- e.g. R3 commenting on the "coolness" of the problem and R1 calling it an "important problem from a cognitive perspective" -- but also identify a number of concerns about baselines, novelty of proposed techniques, underwhelming performance on the task, whether experiments support the conclusions, and some missing or unclear technical details. Overall, the feeling of the reviewers is that they're "not sure what I am supposed to get out of the paper" (R3). The authors posted responses that addressed some of these issues, in particular clarifying their terminology and contribution, and clearing up some of the technical details. However, in post-rebuttal discussions, the reviewers still have concerns with the claims of the papers. In light of these reviews, we are not able to recommend acceptance at this time, but I agree with reviewers that this is a "cool" task and that authors should revise and submit to another venue.
Decision: Reject