Venue: ICLR 2020 Conference

Reviews:

final_score: rating: 6: Weak Accept
review_content: This paper tackles the problem of enabling robots to learn long-horizon, sparse-reward tasks. The proposed approach, the Curious Sample Planner (CSP), builds on insights in task and motion planning (TAMP), which is a standard approach for tackling these kinds of tasks. TAMP constructs a plan in the space of macro-actions (e.g., move object 1 to location (x,y)), and uses a motion planner to execute each macro-action. However, TAMP typically requires being able to describe macro-action effects and preconditions with logical predicates, which can be impossible in real-world environments, due to complex dynamics and interactions. CSP overcomes this limitation by planning in the space of macro-actions in a way that is biased toward novelty.

The core approach of CSP is to train a (macro-)action selection network to generate macro-actions that are both feasible and novel. The curiosity module is used in two ways: (1) to give reward to the action selection network for producing novel macro-actions, and (2) to expand states that are considered novel. Three state-of-the-art ways of computing novelty are compared -- state estimation (SE), forward dynamics (FD), and random network distillation (RND).

CSP is evaluated on a suite of simulated robotics tasks that require the robot to build simple machines from the objects in its environment, in order to achieve the specified objective. The experiments compare agents trained with CSP versus with deep RL (specifically, A2C, PPO, and PPO + RND). There is an ablation study, that compares against planning with uniform selection of macro-actions and uniform selection of states to expand.

Overall, this paper makes a significant contribution to improving robot learning of long-horizon, sparse reward tasks. The paper is clearly written and well-motivated, and the evaluations are thorough. The major downside is that CSP inherently requires knowing the dynamics of the environment (i.e., having a simulator), which means it cannot be directly run applied to real-world robotic systems. But it is a step in the right direction, and clearly outperforms vanilla deep RL.

I'm leaning toward accept, but I have a few concerns / questions about the paper. First, there are not enough details included for reproducibility (see list below). With regard to evaluation, I think another ablation should be run, where the action selection network is trained for only feasibility. This would be similar to CSP-No Curiosity, but with a reward of 0 for infeasible actions, and a small fixed positive reward for feasible ones. This would more clearly answer the question of how much it matters to include the curiosity module. Finally, I'm not convinced that this approach works well for transfer, and the evaluations seem inconclusive as well. I'm surprised that even for inter-task transfer, agents trained with action selection transfer and full transfer don't just learn to solve the task immediately. Am I missing something here about how the task is instantiated?

Reproducibility questions:
- In Algorithm 1, what are the inputs to the novelty metric that is used to compute L_\phi? Is it the batch of next states, S'?
- What is the form of the output of the action-selection network? And what exactly is the space of macro-actions? For instance, the number of possible RemoveConstraint macro-actions depends on how many objects are connected in the environment. But the dimension of the action-selection network's output must be fixed.
- What does the state vector input for FD and RND contain? Along these lines, why not also use image inputs for FD and RND, as is done for SE?
- What are the learning hyperparamters used to train the networks? (e.g., learning rate)
- How many perspectives (i.e., n_p) are used for the SE curiosity module?

Minor comments / typos:
- Avoid using the same variable with different meanings, e.g. using \phi to indicate both the curiosity module and the parameters of the value network.
- Page 3: "flexible", "a flexible"
- Page 5: "learnabe" --> "learnable"
- Page 8: "in which" --> "in which the"
- Page 9: "illustrate" --> "illustrated"
review_title: Official Blind Review #1

final_score: rating: 1: Reject
review_content: The idea of the paper is to augment a planner with a curiosity module to reduce the number of traversed paths, resulting in a speedup. The paper presents experiments where it is shown that deep RL methods are outperformed in that question.

I recommend to reject the paper.

The reason for this are threefold.
- The paper is crowded with text and ends up to be hard to follow. The actual contribution is hard to distill.
- The method compares to deep RL baselines. But these are *not* planning algorithms, instead these are RL methods. The paper does not compare to planners, which are tailored to solve the problem the paper adresses.
- The reader is left alone to place the work within the literature. The abstract and introduction do not have a single cite; terms like TAMP and multi-step planning are mentioned and certain properties of them are stated without resorting to where a reader could look those up. It is not the readers task to use a search engine to reverse engineer the authors writing!

I think that the authors need to reformulate what the contribution of their work is. That needs to be presented in a more abstract way, without focusing on the experimental setting prematurely. The experimental setup is ok if the method is restricted to robotic tasks, but too thin for the general setting of efficient planning with sparse costs. Planning algorithms that are tailored towards huge spaces shou;d be used as base lines instead of deep RL methods.
review_title: Official Blind Review #2

final_score: rating: 3: Weak Reject
review_content: ===== Summary =====
The paper introduces Curious Sample Planner (CSP) a long-horizon motion planning method that combines task and motion planning with deep reinforcement learning in order to solve simulated robotic tasks with sparse rewards. The CSP algorithm considers two different hierarchies of actions: primitive actions, which control the rotation of several joints in a robotic arm, and macro-actions corresponding to complex behaviours  such as moving from one position to another or linking two objects together. Macro-actions are selected using the actor-critic architecture PPO and then turned into primitive actions using geometric motion planning and inverse kinematics. Specifically, RRT-Connect is used for motion planning with the recursive Newton-Euler algorithm for inverse kinematics on a perfect model of the environment to determine the specific sequence of primitive actions necessary to execute the macro-action. As CSP is interacting with the environment, it also builds a tree of states in the environment connected by the macro-actions leading to each of them. Each vertex of the tree is assigned a curiosity score, which is used as an exploration bonus for PPO and to determine the probability with which each vertex is sampled from the tree for future exploration. The whole process is repeated until a feasible path from the initial state to the goal state is found. The paper provides empirical evaluations in four different tasks where it compares the performance of CSP with three different curiosity measures to the performance of PPO and A2C. The results show that CSP accomplishes each task while using significantly less samples. Moreover, a second set of experiments is presented that show the potential for transfer learning across tasks using CSP. 

Contributions:
1. The paper introduces CSP, a successful combination of task and motion planning and deep reinforcement learning that can discover temporally extended plans. 
2. The paper demonstrates a statistically significant improvement in performance over PPO and A2C in the four robotic tasks that the paper studies. 
3. The paper shows evidence that CSP might facilitate transfer learning across similar tasks. 

===== Decision ===== 
The paper represents a significant contribution to the reinforcement learning and task and motion planning literature. The main algorithm is well motivated from previous literature and demonstrate a significant improvement over previously proposed deep reinforcement learning methods. Moreover, the ideas are presented clearly and logically throughout the paper and the empirical evaluations clearly support the claims about the performance of CSP. However, I have concerns about the reproducibility of the results because of the little amount of details provided about the hyper-parameter selection and settings and about the network architectures and loss functions. Thus, I consider that the paper should be rejected, but I am willing to increase my score if my comments are properly addressed.

===== Questions and Comments =====

1. Although the ideas in the paper are presented clearly, the algorithms and methods are presented mostly at a very high level. There are no details about the losses used in lines 11 and 12 of Algorithm 1 and there are no specifications about the network architectures and the hyperparameter settings and selection for each algorithm. This raises two concerns. First, this hinders reproducibility and future work by other authors that might be interested in building upon the ideas presented in the paper; this would also decrease the impact of the paper. Two, it is difficult to determine if the comparisons against A2C and PPO were fair without any information about the hyper-parameter selection. Thus, I consider these details should be included and I would consider increasing my score to accept if this was properly addressed. Specifically, I think the paper should include this:
- The hyper-parameter settings for each different algorithm and an explanation about how they were selected. 
- A detailed description of the network architectures used in the experiments.
- A definition for the loss functions used for the policy network, the value network, and the curiosity module. 

2. As mentioned in the Decision section above, the paper clearly demonstrates an improvement over previously proposed deep reinforcement learning algorithm. However, there are no comparisons to any previously proposed Task and Motion Planning Methods. What motivated this decision?

3. An alternative to using separate networks for the policy and the value function, it could be possible to use a two-headed network with one head for the policy and another one for the value. What was the reason for using two separate networks over this alternative? 

4. Were any other alternatives tested for the activation functions of the network?

5. CSP was tested with three different curiosity and novelty metrics, none of which dominated over all the other ones. However, PPO was only tested with one of the measures and A2C had no curiosity measure added to it. Where there any preliminary results that justified this decision? In terms of computation and time, how difficult would it be to include this in the paper?

6. The paper already hinted at this, but macro-actions could be framed within the option framework from Sutton, Precup, & Singh (1999). This would open up the opportunity to apply some of the already proposed methods for option discovery such as the option-critic architecture from Bacon, Harb, & Precup (2016) or the Laplacian framework for option discovery from Machado, Bellemare, & Bowling (2017), which is cited on the paper. Could the authors provide more comments about this line of future work? 

===== References ===== 
Bacon, P., Harb, J., & Precup, D. (2016). The Option-Critic Architecture. Retrieved 17 October 2019, from https://arxiv.org/abs/1609.05140

Marlos C. Machado, Marc G. Bellemare, and Michael H. Bowling. A laplacian framework for option discovery in reinforcement learning. CoRR, abs/1703.00956, 2017. URL http://arxiv.org/abs/1703.00956.

Sutton, R., Precup, D., & Singh, S. (1999). Between MDPs and semi-MDPs: A framework for temporal abstraction in reinforcement learning. Artificial Intelligence, 112(1-2), 181-211. doi: 10.1016/s0004-3702(99)00052-1
review_title: Official Blind Review #3

Meta-review: The authors consider planning problems with sparse rewards.                                                                        
They propose an algorithm that performs planning based on an auxiliary reward                                                      
given by a curiosity score.                                                                                                        
They test they approach on a range of tasks in simulated robotics environments                                                     
and compare to model-free baselines.                                                                                               
                                                                                                                                   
The reviewers mainly criticize the lack of competitive baselines; it comes as now                                                  
surprise that the baselines presented in the paper do not perform well, as they                                                    
make use of strictly less information of the problem.                                                                              
The authors were very active in the rebuttal period, however eventually did not                                                    
fully manage to address the points raised by the reviewers.                                                                        
                                                                                                                                   
Although the paper proposes an interesting approach, I think this paper is below                                                   
acceptance threshold.                                                                                                              
The experimental results lack baselines,                                                                                           
Furthermore, critical details of the algorithm are missing / hard to find.
Decision: Reject