Imagine That! Leveraging Emergent Affordances for 3D Tool Synthesis
Authors: Yizhe Wu
Sudhanshu Kasewa
Oliver Groth
Sasha Salter
Kevin Li Sun
Oiwi Parker Jones
Ingmar Posner
Venue:  (2021)
Keywords: Affordance Learning
Imagination
Generative Models
Activation Maximisation

Abstract: In this paper we explore the richness of information captured by the latent space of a vision-based generative model.  The model combines unsupervised generative learning with a task-based performance predictor to learn and to exploit task-relevant object affordances given visual observations from a reaching task, involving a scenario and a stick-like tool.  While the learned embedding of the generative model captures factors of variation in 3D tool geometry (e.g. length, width, and shape), the performance predictor identifies sub-manifolds of the embedding that correlate with task success. Within a variety of scenarios, we demonstrate that traversing the latent space via backpropagation from the performance predictor allows us to imagine tools appropriate for the task at hand.  Our results indicate that affordances – like the utility for reaching – are encoded along smooth trajectories in latent space. Accessing these emergent affordances by considering only high-level performance criteria (such as task success) enables an agent to manipulate tool geometries in a targeted and deliberate way.

Under review as a conference paper at ICIR 2021
IMAGINE THAT!
LEVERAGING EMERGENT AFFORDANCES FOR 3D TOOL SYNTHESIS
Anonymous authors Paper under double-blind review
ABSTRACT
In this paper we explore the richness of information captured by the latent space of basonn generative model. The model combines unsupervised genera- tive learning with lask-based performance predictor learn and exploit task- relevant object affordances given visual observations from reaching task, involv- ing a scenario and stick-like tool. While the learned embedding of the generative model captures factors of variation in 3D tool geometry (e.g. length, width, and shape), the performance predictor identifies sub-manite manifolds of the embedding that correlate with task success. Within variety of scenarios, we demonstrate that traversing the latent space via backpropagation from the performance predictor allows us to imagine tools appropriate for the task at hand. Our results indicate that affordances like the utility for reaching- tories latent space. Accessing these emergent affordances by considering only high-level performance criteria (such as task success) enables an agent to manip- ulate tool geometries in a targeted and deliberate way.
encoded along smooth trajec-
INTRODUCTION
The advent of deep generative models (e-g. Burgess et al., 2019; Greff et al., 2019; Engelcke et al., 2019) with their aptitude for unsupervised representation learning casts new light on learning affordances (Gibson, 1977). This kind of representation learning raises tantalising question: Given that generative models naturally capture factors of variation, could they also be used to expose these factors such that they can modified in a task-driven way? We posit that a task-driven traversal of a structured latent space leads to affordances emerging naturally along trajectories in this space. This is in stark contrast to more common approaches to affordance learning where it is achieved via direct supervision or implicitly via imitation (e.g. Tikhanoff et al., 2013; Myers et al., 2015; Liu al., 2018: Grabner et al., 2011: I; et al.. 2018). The setting we choose for our investigation that of tool synthesis for reaching tasks as commonly investigated in the cognitive sciences (Ambrose, 2001; Emery & Clayton, 2009).
In order to demonstrate that a task-aware latent space encodes useful affordance information we require mechanism to train such model well purposefully explore the space. To this end we propose an architecture in which a task-based performance predictor (a classifier) operates on the latent space of a generative model (see fig. 1). During training the classifier is used to provide an auxiliary objective, aiding in shaping the latent space. Importantly, however, during test time the performance predictor is used to guide exploration of the latent space via activation maximisation (Erhan al.. 2009; Zeiler& & Fergus, 2014; Simonyan et al., 2014), thus explicitly exploiting the structure of the space. While our desire to affect factors of inffuence is similar in spirit to the notion of disentanglement, it contrasts significantly with models such as 3- VAE (Higgins et al., 2017), where the factors of influence effectively encouraged to be axis-aligned. Our approach instead relies on a high-level auxiliary loss discover the direction in latent space explore.
Our experiments demonstrate that artificial agents are able to imagine an appropriate tool for a vari- ety of reaching tasks by manipulating the tool's task-relevant affordances. To the best of our knowl- edge, this makes us the first to demonstrate an artificial agent's 'ss ability to imagine, or synthesise, 3D meshes of tools appropriate for given task via optimisation in structured latent embedding.

---

Under review as conference paper at ICTR 2021
latent space model
tool synthesis
feasibility
X
performance predictor
optimisation steps
Figure I: Tool synthesis for reaching task. Our model is trained on data-triplets (task observation, tool observation, success indicator). Within a scenario, the goal is to determine if given tool can reach the goal (green) while avoiding barriers (blue) and remaining behind the boundary (red). If tool cannot satisfy these constraints, one may augment order to solve the task. Our interest is in what these augmentations, imagined during tool synthesis, imply about the learned object representations.
approach (via the performance predictor) imagines how
Similarly, while activation maximisation has been used to visualise modified input images before (e.g. Mordvintsev et al., 2015), we helieve this work to of factors of influence by chaining the outcome of a task predictor to the latent space, and then decoding the latent representation back into 3D mesh. Beyond the application of tool synthesis, believe work to provide novel perspectives demonstrating object affordances can viewed trajectories a structured latent space as well as by providing a novel architecture adept at deliberately manipulating interpretable factors influence.
first effect deliberate manipulation
affordance learning and disentanglement in
RELATED WORK
The concept of an affordance, which describes potential action performed on object (e.g. doorknob affords being turned), goes back to Gibson (1977). Because of their importance in cogni- tive vision, affordances are extensively studied in computer vision and robotics. Commonly, affor- dances are learned in supervised fashion where models discriminate between discrete affordance classes predict masks for image regions which afford certain types of human interaction (e.g. Staytchev, 2005; Kjellström al., 2010; Tikhanoff et al., 2013; Mar et al., 2015; Myers el al., 2015; Do et al., 2018). Interestingly, most works in this domain learn from object shapes which have heen given an affordance label priori. However, the affordance of a shape is only properly defined in the context of task. Hence, we employ task-driven traversal of latent space optimise the shape of tool by exploiting factors variation which are conducive task success.
Recent advances in shape generation employ variational models (Girdhar et al., 2016: et al., 2016) to capture complex manifolds of 3D objects. Besides their expressive capabilities. the latent spaces of such models also enable smooth interpolation between shapes. Remarkable results have been demonstrated including 'shape algebra' (Wu et al., 2016) and the preservation of object part semantics (Kohli et al.. 2020) and fine- fine-grained shape styles (Yifan et al., 2019) during interpolation. This shows the potential of disentangling meaningful factors variation the latent representation of 3D shapes. Inspired by this, we investigate whether these factors can be exposed in a task- driven way. In particular, reconstruction (Liu al., 2019) paired with activation maximisation (e.g. Erhan et al., 2009; Zeiler & Fergus, 2014; Simonyan et al., 2014) of a task-driven performance predictor. Guided by its loss signal, activation maximisation traverses the generative model's latent representations and drives imagination process yielding shape suitable for the task at hand.
propose architecture in which generative model for 3D object
2

---

Under review as a conference paper at ICTR 2021
performance predictor
hp
Zg
Z
task
heat
tool synthesis
task encoder
feasibility
7
optimisation steps
V/
4/
T
Zy
tool encoder
The model architecture. A convolutional encoder & represents the task image LG as latent vector zg. In parallel, the 3D tool encoder takes input image 1i. and its silhouette I/ produces latent representation xr. The concatenated task-tool representation heat used by a classifier a to estimate the success of the tool at solving the task (i.e. reaching the goal). Given the gradient signal from this performance predictor for success, the latent tool representation zr gets updated to render an increasingly suitable tool (via the 3D tool decoder ş²"), We pretrained the encoding decoding models (v, e/') together prior work (Kato al., 2018; Wang et al., 2018).
Figure 2:
key application of affordance driven shape imagination tool use. Roboties boasts mature body literature studying how robots can utilise tools to improve their performance across a wide range of tasks like reaching (Jamone et al.. 2015), grasping (Takahashi et al., 2017), pushing (Stoytchev, 2005) and hammering (Fang al., 2018). The pipeline executing tool- tasks typically starts with models for tool recognition and selection (e.g. Tikhanoff et al., 2018; Saito et al., 2018; Xie et al., 2019) before tool properties and affordances are leveraged to compute higher-order plans (Toussaint et al., 2018). Our proposed model lends itself to rohotics applications like these, the learned latent space encodes a rich object-centric representation of tools that are biased for specific tasks.
2013: Zhu al., 2015; Fang
METHOD
Our overarching goal is to perform Lask-specific tool synthesis for 3D reaching tasks. We frame the challenge of tool imagination generative model. The optimisation is driven high-level, task-speeific performance predictor, which assesses whether larget specified by goal image Ig reachable given a particular tool and in the presence of obstacles. To map from tool images into manipulable 3D tools, we first train an off-the-shelf 3D single-view reconstruction model laking as input tool images 17. 12 and corresponding silhouettes I3. rendered from two different vantage points and j. After training, the encoder can infer the tool representation that contains the 3D structure information given single-vies view RGB image and its silhouette as input. This representation is implicitly used to optimise the tool configuration make suitable for the task hand. An overview of our model is given in fig.
optimisation problem in structured latent space obtained using
TT More formally, we consider N data instances: {(T pot In T example features task image Ic. tool images In in two randomly selected views i and and their
(") where cach
3

---

Under review as a conference paper at ICTR 2021
Scenarios
Inputs
A
B
C
D
E
Figure 3: (Left) Task examples from our dataset. Top and bottom rows correspond unsuccessful and successful tool examples respectively, Columns A represent five different task scenario types each imposing different tool constraints including width, length, orientation and shape. Note that the robot fixed at its base the table and constrained remain outside the red boundary. Hence, it can only reach the green target with a tool while avoiding collisions with the blue obstacles. (Right) Model inputs (task observation, tool observation] during training and test time.
corresponding silhouettes Is. as well as a binary label p indicating the fcasibility reaching the target with the given tool. Examples of task images and model inputs are shown in fig. 3. In all our experiments, we restrict the training input such sparse high-level instances. For additional details on the dataset, we refer the reader the supplementary material.
3.1 REPRESINTING TASKS AND TOOLS
Given that our tools are presented in tool images 47. it is necessary for the first processing step perform a 3D reconstruction of Ir. from pixels into meshes. To achieve this single view 3D reconstruction images into their meshes, we employ the same architecture as proposed by (Kato et al., 2018; Wang et al., 2018). The 3D reconstruction model consists of two parts: an encoder network and a mesh decoder. Given the tool image and its silhouette in view i, i.c 77. and [5, we denote the latent variable encoding the tool computed by the encoder, 4/.
((r zr
The mesh decoder takes as input and synthesises the mesh by deforming template. A differen- tiable renderer (Liu et al., 2019) predicts the tool's silhoutte 13. in another view j. which is compared to the ground-truth silhouette 1s to compute the silhouette loss L,, This silhouette loss L, together with an auxiliary geometry loss Lg formulates the total 3D reconstruction loss:
Crecon C. phy:
where is the weight of the geometry loss. We refer the reader to Liu et al. (2019) regarding the exact parameer and training setup of the 3D reconstruction model.
Task images IG similarly represented in abstract latent space. For this employ task encoder, which consists of stack of convolutional layers.! á takes the task image la input and maps it into the task embedding zg.
3.2 TOOL. IMAGINATION
Task-driven learning The tool representation 2n contains task-relevant information such as tool length, width, and shape. In order to perform tool imagination, the sub-manifold the latent space that corresponds to the task relevvtlevnnt features needs to be accessed and traversed. This is achieved by adding layer MLP as classifier The classifier a takes input concatenation h of the task embedding zG and the tool representation Zy, and predicts the softmax over the binary task
Architecture details provided in the supplementary material.
4

---

Under review as a conference paper at ICIR 2021
success. The classifier learns to identify the lask-relevant sub-manifold the latent space by using the sparse success signal p and optimising the binary-cross entropy loss, such that
Liask (a (heat),> p) (p log (heas)) + (1 - p) log (1 a (heat))),
(3)
where E {0, } is a binary signal indicating whether not it is feasible to solve the task with the given tool. The whole system is trained end-to-end with loss given by
C (1c. 1's Ps..p) Crecon + Leask-
Note that the gradient from the task classifier a propagates through both the task encoder & and the toolkit encoder v., and therefore helps to shape the latent representations of the toolkit with respect to the requirements for task success.
Tool imagination Once trained, our model can synthesise new tools by traversing the latent man- ifold of individual tools following the trajectories that maximise classification success given tool image and its silhouette (fig. 2). To do this, we first pick a tool candidate and concatenate its representation žy with the task embedding zg- This warm-starts the imagination process. The con- catenated embedding beat is then fed into the performance predictor d to compute the gradient with respect to the tool embedding 2y. We then use activation maximisation (Erhan et al., 2009; Zeiler & Fergus, 2014; Simonyan et al., 2014) to optimise 2r with regard to Ltesk of the success estimation a (hewt) and a feasibility target p. = 1, such that
4r zr + Stask (a (ZT)
(5)
ÖzT
where 7) denotes the learning rate the update. Finally, we apply this gradient update for S steps or until the success estimation a (zy) reaches a threshold 7, and use (2(2))) to generate the imagined 3D tool mesh represented by Zy.
4 EXPERIMENTS
In this section we investigate our model's abilities in two experiments. First, we verify the func- tionality of the task performance predictor in tool selection experiment where only three tools is successfully applicable. Second, we examine our core hypothesis about task-driven tool synthesis in a tool imagination experiment where the model has to modify a tool shape to be successfully applicable in a given task. In both experiments, we compare our full task-driven model, in which the tool latent space was trained jointly with the task performance predictor, with task- unaware baseline, in which the 3D tool representation was trained first and the task performance predictor was fitted to the fixed tool latent space. We report our results in table as mean success performances within a 95% confidence interval around the estimated mean.
of
Tool Selection We verify that the classifier a correctly predicts whether or not a given tool can succeed at chosen lask. For each task, we create toolkit containing three tool candidates where exactly one satisfies the scenario constraints. The toolkits are sampled in the same way the remaining dataset and we refer the reader to fig. 3 again for illustrations of suitable and unsuitable tools. We check whether the classifier outputs the highest success probability for the suitable tool. Achieved accuracies for tool selection are reported in the left column of table 1.
Tool Imagination We evaluate whether our model can generate tools to succeed in the reaching tasks. For each instance the target signal for feasibility is to P. 1. i.e. success. Then, the latent vector of the tool modified via backpropagation using a learning rate of 0.01 for 10, 000 steps or until o o(h..)) reaches the threshold of 7 = 0.997. The imagined tool mesh is generated via the mesh decoder v!' This is then rendered into top-down view and evaluated using a feasibility test which checks whether all geometric constraints are satisfied, i.c. successful reaching from behind the workspace boundary while not colliding with any obstacle. We report the percentage of imagined tools that successfully pass this test in table
5

---

Under review as a conference paper at ICTR 2021
optimisation slegs
optimisation stps
optimistionn stago
Figure 4: Qualitative results of tool evolution during imagination process. Each row illustrates an example of how the imagination procedure can succeed at constructing tools that solve the task by: (left) increasing tool length, (middle) decreasing tool width, and (right) altering tool shape (creating an appropriately oriented hook). Each row in each grid represents different imagination experiment.
4.1 MODEL TRAINING
In order gauge the influence of the task feasibility signal on the latent space of the tools, train the model two different setups. A task-driven model trained with a curriculum: First, the 3D reconstruction module is trained on tool images alone. Then, the performance predictor trained jointly with this backbone, i.c. the gradient from the predictor is allowed to back-propagate into the encoder of the 3D reconstruction network. In task-unaware ablation, 3D reconstruction weights fixed during the predictor training removing any influence of the task performance on the latent space, All models are trained for 15,000 steps in total. The first 10,000 steps are spent on pre-training the 3D reconstruction part in isolation and the remaining 5,000 steps are spent training the task performance predictor. We select checkpoints that have the lowest task loss Ltast the validation split.
keep the pre-trained
4.2 QUANTITATIVE RESULTS
We start our evaluation by examining the task performance predictor in the tool selection experiment (TS). For each scenario type (A E) we present 250 tasks from the test set, cach paired with three tool images and their respective silhouettes. We encode each tool with the tool encoder 1 and concatenate representation 2y Zg obtained from the task image encoder d. Fach concatenated pair Aeat is passed through the feasibility predictor and the tool which scores highest selected. We report the results for this experiment in the left section of table 1. The results confirm that is able to identify suitable tools with almost perfect accuracy. Tool selection accuracy does not differ significantly between the task-driven and the task-unaware variations of the model. This suggests
Table 1: (Left) Tool Selection: Mean accuracy when predicting most useful tool among three possi- tools. (Right) Tool Imagination: Comparison imagination processes when artificially warm- starting from the same unsuitable tools in cach instance. Best results are highlighted in bold.
Tool Selection Success [%] Tnakare
Tool Imagination Success [%]
N
Task-Driven
Random Walk
Tasw-unaware
Task-Driven
A
250
88.8 + 3.9
90.8 4: 3.6
3.6 2.3
55.6 + 6.2
96.4 +: 2.3
B
250
96.4 + 2.3
97.6 + 1.9
5.6 = 2.9
42.0 + 6.1
78,8 + 5.1
250
96.4 + 2.3
97.2 2.1
23.6 5.3
56.8 + 6.1
76.4 + 5.3
D
250
96.8 + 2.2
98.4 + 1.6
2.4 1.9
75.2 + 5.4
81.2 + 4.8
E
250
87.2 + 4.1
87.6 + 4.1
13,6 4.3
88.4 + 4.0
86.4 t 4.3
Tot
1250
93.1 + 1.4
94.3 + 1.3
9.8 1.7
63.6 2.7
83.8 + 2.0
6

---

Under review as a conference paper at ICIR 2021
that the factors of tool shape variation are captured successfully in both cases and the feature vectors produced by y are discriminative enough to be separated efficiently via the MIP o.
After verifying the task performance predictor's functionality in the tool selection experiment, we investigate its ability to drive generative process in the next experiment. This is done to test our hypothesis about the nature and exploitability of the latent space. Given that the latent space captures factors of variation 3D tool geometry, we hypothesise that these factors can be actively leveraged to synthesise a new tool by focusing on the performance predictor. Specifically, we present our model with task image Ie and an unsuitable tool geometry. We then encode the tool via & and modify its latent representation 2y: by performing activation maximisation through the performance predictor a. As the feasibility prediction is pushed towards I, the tool geometry gradually evolves into one that is applicable to the given task image Ig. In addition to comparing the imagination outcomes for the driven and the task unaware model, we also include random walk baseline, where, in place of taking steps in the direction of the steepest gradient, we move in random direction in the task-driven latent space for 10, 000 steps. In this baseline the latent vector of the selected tool is updated by a sample drawn from an isotropic Gaussian with mean O, and, to match the step size of our approach, the absolute value of the ground-truth gradient derived by back- propagating from the predictor the variance.
For 250 instances per scenario type, warmstart each imagination attempt with the same infeasible tool across random walk, task-driven, and task-unaware models enable a like-for-like comparison, with the results presented in table 1, The performance of the random walk baseline reveals that a simple stochastic exploration of the latent space is not sufficient to find suitable tool geometries. However, following the gradient derived from the performance predictor leads to successful shaping of tool geometries in much more reliable way. While the task-unaware ablation provides a strong baseline, transforming tools successfully in 63.6% of the cases, the task-driven model significantly outperforms it, achieving a glohal success rate of 83.8% on the test cases. This implies that jointly training the 3D latent representation and task performance predictor significantly shapes the latent space in 'task-aware' way, encoding properties which are conducive task success (e.g. length, width, and configuration of tool) along smooth trajectories. Moreover, each these trajectories leads to higher reachability suggesting that these affordances can be seen as a set trajectories in a lask-aware latent space.
4.3 QUALITATIVE RESULTS
Qualitative examples the tool imagination process are provided in fig. and fig. 5. the right- middle example of fig. 4, a novel T-shape tool is created, suggesting that the model encodes the vertical stick-part and horizontal hook-pari distinct elements. The model also learns to interpolate novel tool. As
the direction of the hook part between pointing left and right, which leads shown in fig. 5, tools are modificd in a smooth manner, leading us to hypothesise that tools are embedded in continuous manifold of changing length, width and configuration. Optimising the latent embedding for the highest performance predictor score often drives the tools to evolve along these properties. This suggests that these geometric variables encoded trajectories in the structured latent space learnt by our model and deliberately traversed via high-level task objective in the form of the performance predictor.
5 CONCLUSION
In this paper we investigated the ability of an agent to synthesise tools via task-driven imagination within a set of simulated reaching tasks. Our approach explores hybrid architecture in which a high-level performance predictor drives optimisation process in a structured latent space. The resulting model successfully generates tools for unseen scenario types not in the training regime; it also learns to modify interpretable properties of tools such as length, width, and shape. Our experimental results suggest that these object affordances are encoded as trajectories in a learnt latent space, which we navigate through in deliberate way using a task predictor and activation maximisation, and interpret by decoding the updated latent representations. Ultimately, this may aid in our understanding of object affordances while offering novel way to disentangle interpretable factors of variation not only for 3D tool synthesis. To facilitate further work in this area, we plan to release both the reaching datasct and trained model to the community.
7

---

Under review as a conference paper at ICTR 2021
Intial tool
Task
Inibal tool
optimisation steps
Figure 5: Examples of tool synthesis progression during the imagination process. In the top row, stick tool morphs into hook. The middle row shows a left-facing hook transforming into a right-facing hook. the bottom row, the tool changes into a novel T-shape. Constraints on these optimisations are specified via task embeddings corresponding to the task images on the far left.
REFERENCES
Stanley Ambrose. Paleolithic technology and human evolution. Science, 291(5509):1748-1755 2001.
Christopher P. Burgess, Loic Matthey, Nicholas Watters, Rishabh Kabra, Irina Higgins, Matt Botvinick, and Alexander Lerchner. MONet: Unsupervised scene decomposition and representa- tion. arXiv preprint arXiv:191.111300 2019.
Thanh-Toar Do, Anh Nguyen, Ian D. Reid, Darwin G. Caldwell, and Nikos G. Tsagarakis. Af- fordanceNet: An end-to-end deep learning approach for object affordance detection. In JEEE International Conference Robotics and Automation (ICRA), 2018.
Nathan Emery and Nicola Clayton. Tool Current Opinion in Neurobiology, 19(1):27-33, 2009.
physical cognition in birds and mammals.
Martin Engelcke, Adam R. Kosiorek, Oiwi Parker Jones, and Ingmar Posner. GENESIS: Gen- crative scene inference and sampling with object-centric latent representations. arXiv preprint 1.Xi::1977.1352.. 2019.
Dumitru Erhan, Yoshua Bengio, Aaron Courville, and Pascal Vincent, Visualizing higher-layer features of deep network. Technical report, University of Montreal, 2009.
Kuan Fang, Yuke Zhu, Animesh Garg, Andrey Kurenkov, Viraj Mchta, Li Fei-Fei, and Silvio Savarese. Leamning task oriented grasping for tool manipulation from simulated superviiinv arXiv preprint 1806.0088002222 2018
James J. Gibson. The theory of affordances. In Perceiving, Acting, and Knowing: Toward an Ecological Psychology. Lawrence Erlbaum, Hilldale, USA, 1977.
R. Girdhar, D.F. Fouhey, M. Rodriguez, and A. Gupta. Learning predictable and generative vector representation for objects. ECCV, 2016.
H. Grabner, J. Gall, and L. Van Gool. What makes chair a chair? In Computer Vision and Pattern Recognition (CVPR), pp. 1529-1536, 2011.
Klaus Greff, Raphaël Lopez Kaufman, Rishabh Kabra, Nick Watters, Christopher Burgess, Daniel Zoran, Loic Matthey, Matthew Botvinick, and Alexander Lerchner. Multi-object representation lcarning with iterative variational inference. In International Conference on Machine Learning (ICML), 2019.
8

---

Under review as a conference paper at ICLR 2021
Trina Higgins, Loic Matthey, Arka Pal, Christopher Burgess, Xavier Glorot, Matthew Botvinick, Shakir Mohamed, and Alexander Lerchner. (-vae: Learning basic visual concepts with a con- straied variational framework. In ICLR, 2017.
Lorenzo Jamone, Giovanni Saponaro, Alexandre Antunes, Rodrigo Ventura, Alexandre Bernardino, and José Santos-Victor. Learning object affordances for tool and problem solving in cognitive robots. Proceedings of the 2nd Italian Workshop on Artificial Intelligence and Robotics, Pp. 68-82, 2015.
Hiroharu Kato, Yoshitaka Ushiku, and Tatsuya Harada. Neural 3D mesh renderer. In Proceedings of the IEEE Conference Computer Vision and Pattern Recognition, pp. 3907-3916, 2018.
H. Kjellström, J. Romero, and D. Kragie. Visual object-action recognition: Inferring object af- fordances from human demonstration. Computer Vision and Image Understanding, pp. 81-90, 2010.
Amit Kohli, Vincent Sitzmann, and Gordon Wetzstein. Inferring semantic information with 3D neural scene representations. arXiv preprint arXiv:2003.16673, 2020.
Shichen Liu, Tianye Li, Weikai Chen, and Hao Li. Soft rasterizer. A differentiable renderer for image-based baaed- 3D reasoning. Proceedings of the IEEE International Conference on Computer Vision, pp. 7708-7717, 2019.
Y. Liu, A. Gupta, P. Abbeel, and S. Levine. Imitation from observation: Learning to imitate behav- iors from raw video via context translation. In IEEE International Conference Robotics and Automation (ICRA), pp. 1118-1123 2018.
Tanis Mar, Vadim Tikhanoff, Giorgio Metta, and Lorenzo Natale. Self-supervised learning of grasp icub humanoid robot. In 2015 JEEE International Conference
dependent tool affordances on Robotics and Automation (ICRA). pp. 3200-3206. IEEE, 2015.
Alexander Mordvintsev, Christopher Olah, and Mike Tyka. Inceptionism: Going deeper into neural networks, 2015. URL inception going deeper into neural. html.
A. Myers, C. L. Teo, C. Fermüllerr and Aloimonos. Affordance detection of tool parts from geometric features. In IEEE International Conference on Robotics and Automation (ICRA), pp. 1374-13811 2015.
Namiko Saito, Kitae Kim, Shingo Murata, Tetsuya Ogata, and Shigeki Sugano. Tool-use model considering tool selection by robot using deep learning. In 2018 IEEE-RAS 18th International Conference on Humanoid Robots (Humanoids), pp. 270-276. IEEE, 2018
K. Simonyan, A. Vedaldi, and A. Zisserman. Deep inside convolutional networks: Visualising image classification models and saliency maps. In Workshop at International Conference on Learning Representations, 2014.
Alexander Stoytchev. Behavior-grounded representation of tool affordances. In IEEE International Conference on Robotics and Automation (ICRA), pp. 3071-3076, 2005.
Kuniyuki Takahashi, Kitae Kim, Tetsuya Ogata, Shigeki Sugano. Tool-body assimilation model considering grasping motion through deep learning. Robotics and Autonomous Systems, 91:115- 127, 2017.
Vadim Tikhanoff, Ugo Pattacini, Lorenzo Natale, and Giorgio Metta. Exploring affordances and tool the iCub. In IEEE-RAS International Conference on Humanoid Robots (Humanoids). pp. 130-137, 2013.
Mare Toussaint, Kelsey Allen, Kevin Smith, and Joshua Tenenbaum. Differentiable physics and stable modes for tool-use and manipulation planning. In Robotics: Science and Systems XIV.
Robotics: Science and Systems Foundation, 2018.
9

---

Under review as a conference paper at ICTR 2021
Nanyang Wang, Yinda Zhang, Zhuwen Li, Yanwei Fu, Wei Liu, and Yu-Gang Jiang. Pixel2mesh: Generating 3D mesh models from single RGB images. In Proceedings of the European Confer- ence on Computer Vision (ECCV), pp. 52-67, 2018.
Jiajun Wu, Chengkai Zhang. Tianfan Xue, Bill Freeman, and Josh Tenenbaum. Learning prob- abilistic latent space of object shapes via 3D generative-adversaiial modeling. In D. D. Lec, M. Sugiyama, U. V. Luxburg. I. Guyon, and R. Garnett (eds.), Advances in Neural Information Processing Systems 29, pp. 82-90. Curran Associates, Inc., 2016.
Annie Xie. Frederik Ebert, Sergey Levine, and Chelsea Finn. Improvisation through physical un- derstanding: Using novel objects as tools with visual foresight. arXiv preprint arXiv:1904.05538. 2019.
Wang Yifan, Noam Aigerman, Vladimir Kim, Siddhartha Chaudhuri, and Olga Sorkine- Horungg Neural cages for detail-preserving 3D deformations. arXiv preprint arXiv:1912.063355, 2019.
Matthew D Zeiler and Rob Fergus. Visualizing and understanding convolutional networks. In Proceedings of the IEFE Europen Conference on Computer Vision, pp- 818 833, 2014.
Yixin Zhu, Yibiao Zhao, and Song Chun Zhu. Understanding tools: Task-oriented object modeling, learning and recognition. In The IEEE Conference on Computer Vision and Pattern Recognition (CVPR), June 2015.
10


Reviews:
[{"review_id": "yfS-VAfmelL", "reviewer": "AnonReviewer3", "summary": "", "questions": "", "limitations": "", "rating": 4, "confidence": 5, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "The authors try to tackle the problem of tool synthesis by using a classifier to guide the learning and exploitation of a generative model through activation maximization. Experiments are conducted on the proposed simulated reaching dataset on tool selection and tool imagination.  While the idea of synthesizing tools step by step continuously is interesting, the technical and experimental design make the problem over-simplified and thus raise some concerns. \n\nThe overall concern for the paper is two-fold: 1) Some overclaims. Affordance is often used to describe the potential action that can be applied to certain objects. In this case only the shape of tool is changing however the concept of `\"emergent affordances\" is hardly touched. Also current proposed dataset only tackle reaching task in a static way. So the problem degenerates to a geometric constraint satisfaction problem. It would be interesting to add more different tasks and potentially extend the current binary label for task success.  2) The experiment design is weak.  For the tool selection task, the task-unaware and task-driven model achieve similar result, which means information from task predictor is not helpful in this case. In tool imagination task, the gradient doesn't flow back to the learned representation in the task-unaware case so suboptimal performance understandable. It would be better if the test setting is more diversified with more degree of freedom in the simulation dataset to showcase the capability of proposed model.\n\nThe paper is easy to follow despite some typos, e.g. the bold numbers in Table1. \n\nSome detail questions:\nWhat are the differences for 5 types of the scenarios in detail? \nHow many images of tools are used to train 3D reconstruction model?\nIs the position/orientation of the tool fixed in the tool selection and tool imagination evaluation? Is it possible that a tool that cannot finish the reaching task at one location can reach the target when positioned at another location/direction?\nAre there any failure cases in tool imagination and what might be the reason?\nIn tool selection task, how are the tools sampled apart from the ground truth?\n\n------\nPost rebuttal:\nAfter reading the author's response as well as the opinions from other reviewers, I will stick to my original rating. Although the authors resolve some of the concerns in the rebuttal, there are still limitations in the method and task design.\n", "extra_scores": {}, "extra_text": {}}, {"review_id": "wD0oamMkqKR", "reviewer": "AnonReviewer2", "summary": "", "questions": "", "limitations": "", "rating": 4, "confidence": 5, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "This paper explores the idea of tool synthesis in an unsupervised generative learning setting.\n\nPros:\n\n+ Tool use is one of the crucial aspects to demonstrate human-like intelligence. The authors' proposal to synthesize the tool's shapes for a given task is a fundamental research problem. This is different from most prior work that focuses on ranking/selection or simply categorization. Tool synthesis is particularly difficult; only one paper was published recently on this particular topic [1]. It has not yet passed the peer-review, but I recommend the authors consider citing it for future readers for completeness purposes.\n\n+ Different from classic object recognition, the tool use is intrinsically a task-oriented concept. This paper did a few things right: emphasis on the task-oriented view, no affordance label, etc. The authors are definitely heading in the right direction.\n\nCons:\n\n- What is the representation of the task or the tool? This is probably the most significant difference that could tell the present work from the prior work, especially if the results are promising. However, the authors only spend three short paragraphs in the paper (section 3.1), which does not really touch this point, despite the section title is called \"representing tasks and tools.\" It seems to be a simple, off-the-shelf encoder that has been published. Representing tools is significant as making/synthesizing tools requires a deep understanding of physics and causality (for instance, see the cited Zhu et al 2015 and Toussint et al 2018). Without a proper representation of these concepts or a clear disentanglement of the model, one simply cannot justify their model is learning how to synthesize tools. Instead, the model is more likely to learn an association between the given task and a trained shape (or 3D shape space if learned better). In this view, it really does not touch the core of the computational problem of tool-use: The problem of tool synthesis becomes 3D shape synthesis without really understanding why an object becomes a tool in that context, and the crucial concept of \"task-driven\" becomes a simple label.\n\n- What about action? A tool-use or task-oriented view cannot be separated from the actual action. This is one of the key ideas behind affordance in Gibson's theory. Given different embodiments of the agent with various capabilities of actions, the synthesized tool would be dramatically different. This is a very challenging problem in tool manipulation in robotics (e.g., see [2]).\n\n- Furthermore, what about trajectories? Even if an algorithm can synthesize the tool and chooses an action, one still needs to properly manipulate tools with the planned trajectory to complete the task. This seems to be completely left out in the paper.\n\n- Why not directly use 3D meshes as input for the algorithm, instead of using two views to reconstruct? The reconstructed one does not bring in any benefit for the overall goal of the paper.\n\n- The experiments are far too simple. The authors do have extra space on page 8, but did not include additional results. This makes me wonder how robust the algorithm is.\n\n[1] Tool MacGyvering: A Novel Framework for Combining Tool Substitution and Construction\n[2] Mirroring without Overimitation: Learning Functionally Equivalent Manipulation Actions", "extra_scores": {}, "extra_text": {}}, {"review_id": "QnqwFETJPw", "reviewer": "AnonReviewer4", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "Summary\n-------\nBy combining a task-success classifier with the latent space of a tool-shape generative model, this paper shows that an activation-maximization approach can generate tool shapes which can succeed at particular tasks.  \n\nPositives\n---------\nThe paper addresses the interesting topic of affordances, and how latent representations of tool shapes might be structured.\n\nThe methods and experiments are clean and well motivated, and the writing is clear.\n\nNegatives\n---------\nThe specific contribution of the paper is difficult to see at first glance.  The architecture is taken from previous work, as well as the method for maximizing activations.\n\nAlthough the task-unaware baseline is a reasonable one to compare against, I would like to see other comparisons to approaches present in the literature.  For example, how do the supervised affordance learning approaches perform in the tool imagination task when combined with task-success prediction?  Additionally, are other methods for latent disentangling sufficent to observe similar behavior?\n\nIn the tool imagination task, it would be useful to see a couple of things from the task-unaware approach to make sure the conclusions are sound.  First, how well does the activation-maximization step work?  Is the model reaching the same level of feasibility for task aware and unaware versions?  Second, what do the imagined tool trajectories look like for the task unaware approach?  If they are also smooth and encode length, width, etc., it would be hard to claim that task-aware approaches are required for that type of encoding.\n\nReasons for score\n-----------------\nThe topic explored is interesting, and the experiments are simple and illustrative, but there are remaining questions about the baseline comparisons required to make strong conclusions.\n\nPost-rebuttal response\n-----------\nThank you for the detailed rebuttal responses.  The rebuttal suggests that there are not many other baselines against which to compare.  If that is true, I would want to see much more detailed analysis of the comparisons offered in the paper.  However, I am still missing details of the latent space generated by the task-unaware approach.  I will leave my score as it is.", "extra_scores": {}, "extra_text": {}}, {"review_id": "AnA218efMpU", "reviewer": "AnonReviewer1", "summary": "", "questions": "", "limitations": "", "rating": 4, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "This paper presents a method with two goals: (1) estimate if a given tool can solve a given task, and (2) generate a tool that can solve a given task. One encoder maps a tool image and silhouette into a latent code, and a decoder maps this code into a mesh; another encoder maps a task image into a latent code; this code is concatenated with the tool code, then mapped to a task success probability. The networks are trained together, so that task success probability has some impact on the tool encoder's latent space. Results show that the model is largely successful in both tasks, but no challenging baselines are here. \n\nI think the idea overall is creative and well-executed, and the paper is well-written. \n\nI was happy to see that the supplemental includes information about how feasibility labels are generated. I think this should be in the main paper.   \n\nWhat is the geometry loss L_g? I looked in the cited paper and saw that details are bare there too -- it only says \"regularizes the Laplacian of both shape and color predictions\". What does this mean exactly, and why does it help?\n\nWhy is the success signal \\rho called \"sparse\" (top of page 5)? My understanding is that this label is available on every training example. \n\nWhen I think of \"imagination\", I think of interpolating or extrapolating into space that was not quite seen at training time. This leads to two ideas: \n\n1. The task is simple enough that imagination may not be necessary. How about a nearest neighbors baseline? Classify the task using nearest neighbor image retrieval (searching in the training set, using maybe the task encoding z_G), and use the tool that was successful in that training example. \n\n2. How about creating a setup where imagination is definitely necessary? e.g., generate a scenario that requires a tool in-between one of the tools seen at training time. Maybe train with long sticks and short L's, and create a task that demands a long L. This would show that the model has a useful latent space and that the imagination (i.e., latent space traversal) is beneficial. The paper notes a novel \"T-shape tool\" but I am not sure this counts, since in Figure 4 the T is only some intermediate stage of the optimization, and in Figure 5, a T is a strange choice for the task shown (in row 3), so it seems like traversal simply stopped when the horizontal part shifted far enough, and there is no purpose to the \"T\" tool per se.\n\n\n-------------\nPost-rebuttal: After reading the rebuttal and the other reviews, I have decided to lower my rating. The rebuttal appears to state that all baselines and evaluations suggested by the reviewers are irrelevant to the goals of the paper. I see the two main hypotheses clarified in the rebuttal, but I do not see a convincing argument for avoiding comparisons to related approaches to the task at hand. ", "extra_scores": {}, "extra_text": {}}]

Author rebuttal:

Meta-review: This paper proposes a method for tool synthesis by jointly training a generative model over meshes and a task success predictor. Gradient-based planning is then used to find a latent space tool representation which maximizes task success, given a starting tool and an input scene. The results indicate that this method can successfully generate simple tools, and that it performs better than either a random baseline or a version where the generative model and success predictor are trained independently.

The reviewers unanimously felt that this paper was not quite ready for publication at ICLR. While I'm a strong believer that unique and creative papers which tackle understudied problems---such as this one---ought to be encouraged, and that the authors' rebuttal satisfactorily addressed most of the reviewers' concerns, there was one major point that was not. In particular, all reviewers noted that the paper lacks comparison to convincing baselines and/or sufficiently extensive experiments. While I do not think baselines are necessary per se, especially in such a unconventional setting such as this, I believe what the reviewers are getting at (and I agree) is that the results as presented don't really help the reader understand the contours of the method and/or problem space, and as a result, the contributions of the paper feel thin. For example, here are some questions that the reviewers raised, which I do not feel were adequately addressed:

- R3: What are the failure cases of the model?
- R2: How important is the particular representation of the task and tool (i.e., visual for the task, meshes for the tool)?
- R4: How do the imagined tool trajectories compare between the task-aware and task-unaware cases?
- R4: Is the success classifier trained to the same level of performance in both task-aware and task-unaware settings? (In general, it would be helpful to include learning curves in the appendix.)
- R1: How important is the choice of the particular planning/optimization method (i.e. gradient descent)?
- R1: What is the generalization performance of the model along affordance directions (e.g. needing to synthesize longer/shorter tools than seen during training)?

Taken individually, such questions might not be an issue, but together they illustrate a larger concern that the paper has not done a thorough enough job of analyzing and evaluating the proposed method. Therefore, at this stage I recommend rejection. I think that by fleshing the paper out with some answers to the above questions, this could make an excellent submission to a future conference.
Decision: Reject