Venue: ICLR 2020 Conference

Reviews:

final_score: rating: 3: Weak Reject
review_content: This paper proposes an algorithm that learns to synthesize tools for the task of reaching. The main idea is to first use unsupervised segmentation algorithm to decompose an image of a set of tools into individual objects; then, with a trained feasibility model, search through the latent space of encoded tools to 'imagine' new ones for reaching. 

This paper has clear strengths and weaknesses. It's studying an important problem from a cognitive perspective; it also proposes a novel model for the task, building upon SOTA models. However, the current problem formulation and experiment setup are not well justified, and the experiments are quite limited. I lean toward rejection.

Most importantly, while this paper argues for the importance of an object-centric representation, it conducts most of its search in the pixel space (both as input to the model, and as the output of the imagination). This leads to some unnatural and unphysical results: in the teaser figure, it's true that the final, imagined tool reaches the target; however, the tool itself shouldn't be able to pass the gap/hole on the wall, due to its angular shape. Objects, in essence, have shapes and physical occupancy. Without modeling physics, it's unclear how useful the object-centric representation is.

Imagination is done by searching over the latent space, which limits the model's generalization power to novel tools or new configurations. This is revealed in the results on case H, where the model doesn't work at all.

The experimental results, as just mentioned, are not very impressive, especially given the simplified setup. There are no results on other tasks except reaching. In addition, comparisons with published methods are missing. For example, what if I just train a Pix2Pix from the inputs to successful reaches? That sounds like a reasonable baseline and should be compared with.

Due to all these limitations, I lean toward rejection.
review_title: Official Blind Review #1

final_score: rating: 1: Reject
review_content: This paper proposes an architecture for synthesizing tools to be used in a reaching task. Specifically, during training the agent jointly learns to segment an image of a set of three tools (via the MONet architecture) and to classify whether one the tools will solve the given scene. At test time, one of the three tools is selected based on which seems most feasible, and then gradient descent is used to modify the latent representation of the tool in order to synthesize a new tool to (hopefully) solve the scene. The paper demonstrates that this approach can achieve ok performance on familiar scenes with familiar tools, but that it fails to generalize when exposed to unfamiliar scenes or unfamiliar tools. The paper reports a combination of the quantitative results showing that optimizing the latent space can lead to successful synthesis in some cases, and qualitative results showing that the synthesized tools change along interpretable dimensions such as length, width, etc. The combination of these results suggest that the model has learned something about which tool dimensions are important for being able to solve the types of reaching tasks given in the paper.

While I think this paper tackles a very interesting, important, and challenging problem, I unfortunately feel it is not ready for publication at ICLR and thus recommend rejection. Specifically, (1) neither the particular task, results, or model are not very compelling, (2) there are no comparisons to meaningful alternatives, and (3) overall I am not quite sure what conclusions I should draw from the paper. However, given the coolness of the problem of tool synthesis, I definitely encourage the authors to continue working on this line of work!

1. The task, results, and model are not very compelling. Any of these three things alone would not necessarily be a problem, but given that all three are true the paper comes across as a bit underwhelming.
 
- First, while the task can be construed as a tool synthesis task, it doesn’t come across to me as very ecologically valid. In fact, the task seems to be more like a navigation task than a tool synthesis task: what’s required is simply to draw an unbroken line from one part of the scene to another, rather than actually generate a tool that has to be manipulated in an interesting way. Navigation has been studied extensively, while synthesis of tools that can be manipulated has not, which makes this task both not very novel and disappointing in comparison to what more ecologically-valid tool synthesis would look like. For example, consider a variation of the task where you would have to start the tool at the red region and move it to the green region. Many of the tools used here would become invalid since you wouldn’t actually be able to fit them through the gaps (e.g. Figure 2E).
 
- Second, given that the “synthesis” task is more like a navigation task, the results are somewhat disappointing. When provided with a feasible solution, the model actually gets *worse* even in some of the in-sample scenes that it has seen during training (e.g. scene types C and D) which suggests that it hasn’t actually learned a good generative model of tools. Generalization performance is pretty bad across the board and is only slightly better than random, which undermines the claim in the abstract that “Our experiments demonstrate that the synthesis process modifies emergent, task-relevant object affordances in a targeted and deliberate way”. While it’s clear there is successful synthesis in some cases, I am not sure that the results support the claim that the synthesis is “targeted” or “deliberate” given how poor the overall performance is.
 
- Third, the model/architecture is a relatively straightforward combination of existing components and is highly specialized to the particular task. As mentioned above, this wouldn’t necessarily be a problem if the task were more interesting (i.e. not just a navigation task) and if the results were better. I do think it is cool to see this use of MONet but I’m skeptical that the particular method of optimizing in the latent space is doing anything meaningful. While there is prior work that has optimized the latent space to achieve certain tasks (as is cited in the paper), there is also a large body of work on adversarial examples which demonstrate that optimizing in the latent space is also fraught with difficulty. I also suspect this is the reason why the results are not particularly good.
 
2. While I do appreciate the comparisons that are in the paper (to a “Random” version of TasMON that moves in a random direction in the latent space, and to “FroMON” agent which is not allowed to backpropagate gradients from the classification loss into MONet), these comparisons are not particularly meaningful. The difference between FroMON performance and TasMON tool imagination performance (I didn’t test tool utility) across tasks is not statistically significant (z(520, 544)=-0.8588, p=.38978), so I don’t think it is valid to claim that “a task-aware latent space can still provide benefits.” The Random baseline is a pretty weak baseline and it would be more interesting to compare to an alternative plausible architecture (for example, which doesn’t use a structured latent space, or which doesn’t have a perceptual frontend and operates directly on a symbolic representation of the tools/scene).
 
3. Overall, I am not quite sure what I am supposed to get out of the paper. Is it that “task relevant object affordances are implicitly encoded as directions in a structured latent space shaped by experience”? If so, then the results do not support this claim and so I am not sure what to take away. Is it that the latent space encodes information about what makes a tool feasible? If so, then this is a bit of a weak argument---of *course* it must encode this information if it is able to do the classification task at all. Is it that tool synthesis is a challenging problem? If so, then the lack of strong or canonical baselines makes it hard to evaluate whether this is true (and the navigation-only synthesis task also undermines this a bit).

Some additional suggestions:
 
It would be good to include a discussion of other recent work on tool use such as Allen et al. (2019) and Baker et al. (2019), as well as on other related synthesis tasks such as Ha (2018) or Ganin et al. (2018).
 
The introduction states that “tool selection and manufacture – especially once demonstrated – is a significantly easier task than tool innovation”. While this may be true, it is a bit misleading in the context of the paper as the agent is doing something more like tool selection and modification rather than tool innovation (and actually the in-sample scenes are more like “manufacture”, which the agent doesn’t always even do well on).
 
It would be helpful to more clearly explain scene types. Here is some suggested phrasings: in-sample = familiar scenes with familiar tools, interpolation = novel scenes with familiar tools, extrapolation = novel scenes with novel tools.
 
I was originally confused how psi’ knew where to actually place the tool and at what orientation, and whether the background part of the rendering process shown in Figure 1. I realized after reading the supplemental that this is not done by the agent itself but by separate code that tries to find the orientation and position of the tool. This should be explained more clearly in the main text.
 
In Table 1 it would be helpful to indicate which scene types are which (in-sample, interpolation, extrapolation).
 
Allen, K. R., Smith, K. A., & Tenenbaum, J. B. (2019). The Tools Challenge: Rapid Trial-and-Error Learning in Physical Problem Solving. arXiv preprint arXiv:1907.09620.
 
Baker, B., Kanitscheider, I., Markov, T., Wu, Y., Powell, G., McGrew, B., & Mordatch, I. (2019). Emergent tool use from multi-agent autocurricula. arXiv preprint arXiv:1909.07528.
 
Ganin, Y., Kulkarni, T., Babuschkin, I., Eslami, S. M., & Vinyals, O. (2018). Synthesizing programs for images using reinforced adversarial learning. arXiv preprint arXiv:1804.01118.
 
Ha, D. (2018). Reinforcement learning for improving agent design. arXiv preprint arXiv:1810.03779.
review_title: Official Blind Review #3

final_score: rating: 3: Weak Reject
review_content: The authors constructed an interesting dataset named reaching task, where the model need to predict if the given toolkit is able to solve the corresponding task or not. They showed that combining variational auto-encoding with an auxiliary loss (in this case, the predictor of solving the tasks, could help shaping a latent space where affordance of the tool is encoded as directions in such latent space.) Using the activation maximisation technique (which was phrased as the imagination process), they are able to modify the input tools into the ones that is suitable to solve the corresponding task. I found the idea of using an auxiliary loss when training a VAE may cause the latent space coding direction change novel and interesting. However, I do not find the authors has a strong case of proven it is the case in this manuscript.

1. The performance difference between FroMON and TasMON is not clear.
  The most critical control model in this paper is the FroMON (frozen MoNet). In this control model, the gradient from the success predictor is not flowing back into the VAE encoder. So, based on the author's assumption, it should not be benefit of having the tool affordance directions in the latent space. However, in the main results in Table 1. We found the performance between FroMON and TasMON is not quite clear. This is particularly true for the Scenario E, F, G (the interpolation tasks), which is more about generalization and is more important.

2. Are the affordance 'directions' in the latent space?
  The authors used activation maximisation approach to travel in the latent space. My understanding of the approach is it follow the gradient to maximise the predictor's success prediction in an iterative approach. So, at each optimization step, the z_im can move in different direction. This seems to not fit as a sense of 'direction', as I would assume it is moving along a particular line (not necessarily axis aligned.). Maybe this does explain whey FroMON and TasMon perform equally well. As long as the possible shapes is encoded in a smooth way in the latent space, the activation maximisation could find a path toward the target object. Unfortunately, is that a 'direction'? Would it be possible to train an optimization algorithm that is only allow to move in a linear direction, and see how well that work?

3. Why MoNet and multiple tools in the toolkit. A simplified version could drive the point as well.
Using MoNet to decompose tools from a toolkit is nice. However, is it really necessary to drive the main point (an auxillary loss of success prediction can shape the latent space of a VAE model) in this paper. In a simplified version, where there is only one tool in the toolkit, one may not need MoNet (maybe still need it for object-background separation?) May the authors comment why multiple tools in the toolkit is important?

Minor:
1. typo: page 1, (2nd to the last line). '...that habitual tool use cannot in and off itself ..' --> of
2. A simple video showing how the tool shape change sequentially during the activation maximisation process would be interesting. 
review_title: Official Blind Review #2

Meta-review: This paper investigates the task of learning to synthesize tools for specific tasks (in this case, a simulated reaching task). The paper was reviewed by 3 experts and received Reject, Weak Reject, and Weak Reject opinions. The reviews are very encouraging of the topic and general approach taken by the paper -- e.g. R3 commenting on the "coolness" of the problem and R1 calling it an "important problem from a cognitive perspective" -- but also identify a number of concerns about baselines, novelty of proposed techniques, underwhelming performance on the task, whether experiments support the conclusions, and some missing or unclear technical details. Overall, the feeling of the reviewers is that they're "not sure what I am supposed to get out of the paper" (R3). The authors posted responses that addressed some of these issues, in particular clarifying their terminology and contribution, and clearing up some of the technical details. However, in post-rebuttal discussions, the reviewers still have concerns with the claims of the papers. In light of these reviews, we are not able to recommend acceptance at this time, but I agree with reviewers that this is a "cool" task and that authors should revise and submit to another venue.
Decision: Reject