Venue: ICLR 2018 conference

Reviews:

final_score: rating: 8: Top 50% of accepted papers, clear accept
review_content: The authors propose an approach for zero-shot visual learning. The robot learns inverse and forward models through autonomous exploration. The robot then uses the learned parametric skill functions to reach goal states (images) provided by the demonstrator. The “zero-shot” refers to the fact that all learning is performed before the human defines the task. The proposed method was evaluated on a mobile indoor navigation task and a knot tying task. 

The proposed approach is well founded and the experimental evaluations are promising. The paper is well written and easy to follow. 

I was expecting the authors to mention “goal emulation” and “distal teacher learning” in their related work. These topics seem sufficiently related to the proposed approach that the authors should include them in their related work section, and explain the similarities and differences. 

Learning both inverse and forward models is very effective. How well does the framework scale to more complex scenarios, e.g., multiple types of manipulation together? Do you have any intuition for what kind of features or information the networks are capturing? For the mobile robot, is the robot learning some form of traversability affordances, e.g., recognizing actions for crossings, corners, and obstacles? The authors should consider a test where the robot remains stationary with a fixed goal, but obstacles are move around it to  see how it affects the selected action distributions.

How much can change between the goal images and the environment before the system fails? In the videos, it seems that the people and chairs are always in the same place. I could imagine a network learning to ignore features of objects that tend to wander over time. The authors should consider exploring and discussing the effects of adding/moving/removing objects on the performance. 

I am very happy to see experimental evaluations on real robots, and even in two different application domains. Including videos of failure cases is also appreciated. The evaluation with the sequence of checkpoints was created by using every fifth image. How does the performance change with the number of frames between checkpoints? In the videos, it seems like the robot could get a slightly better view if it took another couple of steps. I assume this is an artifact of the way the goal recognizer is trained. For the videos, it may be useful to indicate when the goal is detected, and then let it run a couple more steps and stop for a second. It is difficult to compare the goal image and the video otherwise. 
review_title: A good approach with some open questions on related work, scalability, and robustness

final_score: rating: 8: Top 50% of accepted papers, clear accept
review_content: Summary:
The authors present a paper about imitation of a task presented just during inference, where the learning is performed in a completely self-supervised manner.
During training, the agent explores by itself related (but different) tasks, learning a) how actions affect the world state, b) which action to perform given the previous action and the world state, and c) when to stop performing actions. This learning is done without any supervision, with a loss that tries to predict actions which result in the state achieved through self exploration (forward consistency loss).
During testing, the robot is presented with a sequence of goals in a related but different task. Experiments show that the system achieves a better performance than different subparts of the system (through an ablation study), state of the art and common open source systems.

Positive aspects:
The paper is well written and clear to understand. Since this is not my main area of research I cannot judge its originality in a completely fair way, but it is original AFAIK. The idea of learning the basic relations between actions and state through self exploration is definitely interesting.
This line of work is specially relevant since it attacks one of the main bottlenecks in learning complex tasks, which is the amount of supervised examples.
The experiments show clearly that a) the components of the proposed pipeline are important since they outperform ablated versions of it and b) the system is better than previous work in those tasks

Negative aspects:
My main criticism to the paper is that the task learning achieved through self exploration seems relatively shallow. From the navigation task, it seems like the system mainly learns a discover behavior that is better than random motion. It definitely does not seem able to learn higher level concepts like certain scenes being more likely to be close to each other than others (e.g. it is likely to find an oven in the same room as a kitchen sink but not in a toilet). It is not clear whether this is achievable by the current system even with more training data.
Another aspect that worries me about the system is how it can be extended to higher dimensional action spaces. Extending control laws through self-exploration under random disturbances has been studied in character control (e.g. "Domain of Attraction Expansion for Physics-based Character Control" by Borno et al.), but the dimensionality of the problem makes this exploration very expensive (even for short time frames, and even in simulation). I wonder if the presented ideas won't suffer from the same curse of dimensionality.
In terms of experiments, it is shown that the system is more effective than others but not so much *how* it achieves this efficiency. It would be good to show whether part of its efficiency comes from effective image-guided navigation: does a partial image match entail with targetted navigation (e.g. matches in the right side of the image make the robot turn right)?
A couple more specific comments:
- I think that dealing with multimodal distributions of actions with the forward consistency loss is effective for achieving the goal, but not necessarily good for modeling multimodality. Isn't it possible that the agent learns only one way of achieving such goal?
 - It is not clear how the authors achieve to avoid the problem of starting from scratch by "pre-train the forward model and PSF separately by blocking gradient flow". Isn't it still challenging to update them independently, given that at the beginning both components are probably not very accurate?


Conclusion:
I think the paper presents an interesting idea which should be exposed to the community. The paper is easy to read and its experiments show the effectiveness of the method. The relevance of the method to achieve a deeper sense of learning and performing more complex tasks is however unclear to me.
review_title: Interesting paper on imitation of (relatively simple) tasks enhanced by previous self-exploration

final_score: rating: 7: Good paper, accept
review_content: One of the main problems with imitation learning in general is the expense of expert demonstration. The authors here propose a method for sidestepping this issue by using the random exploration of an agent to learn generalizable skills which can then be applied without any specific pretraining on any new task. 

The proposed method has at its core a method for learning a parametric skill function (PSF) that takes as input a description of the initial state, goal state, parameters of the skill and outputs a sequence of actions (could be of varying length) which take the agent from initial state to goal state.

The skill function uses a RNN as function approximator and minimizes the sum of two losses i.e. the state mismatch loss over the trajectory (using an explicitly learnt forward model) and the action mismatch loss (using a model-free action prediction module) . This is hard to do in practice due to jointly learning both the forward model as well as the state mismatches. So first they are separately learnt and then fine-tuned together. 

In order to decide when to stop, an independent goal detector is trained which was found to be better than adding a 'goal-reached' action to the PSF.

Experiments on two domains are presented. 1. Visual navigation where images of start and goal states are given as input. 2. Robotic knot-tying with a loose rope where visual input of the initial and final rope states are given as input.

Comments:

- In the visual navigation task no numbers are presented on the comparison to slam-based techniques used as baselines although it is mentioned that it will be revisited.

- In the rope knot-tying task no slam-based or other classical baselines are mentioned.

- My main concern is that I am really trying to place this paper with respect to doing reinforcement learning first (either in simulation or in the real world itself, on-policy or off-policy) and then just using the learnt policy on test tasks. Or in other words why should we call this zero-shot imitation instead of simply reinforcement learnt policy being learnt and then used. The nice part of doing RL is that it provides ways of actively controlling the exploration. See this pretty relevant paper which attempts the same task and also claims to have the target state generalization ability. 

Target-driven Visual Navigation in Indoor Scenes using Deep Reinforcement Learning by Zhu et al.

I am genuinely curious and would love the authors' comments on this. It should help make it clearer in the paper as well.
 
Update:

After evaluating the response from the authors and ensuing discussion as well as the other reviews and their corresponding discussion, I am revising my rating for this paper up. This will be an interesting paper to have at the conference and will spur more ideas and follow-on work.
review_title: Why is this not simply doing RL in the real world and then using the learnt policy (with no exploration)?

Meta-review: The authors have proposed a method for imitating a given control trajectory even if it is sparsely sampled. The method relies on a parametrized skill function and uses a triplet loss for learning a stopping metric and for a dynamics consistency loss. The method is demonstrated with real robots on a navigation task and a knot-tying task. The reviewers agree that it is a novel and interesting alternative to pure RL which should inspire good discussion at the conference.
Decision: Accept (Oral)