Flexible and Efficient Long-Range Planning Through Curious Exploration
Authors: Aidan Curtis
Minjian Xin
Kevin Feigelis
Dan Yamins
Venue:  (2020)
Keywords: Curiosity
Planning
Reinforcement Learning
Robotics
Exploration

Abstract: Identifying algorithms that flexibly and efficiently discover temporally-extended multi-phase plans is an essential next step for the advancement of robotics and model-based reinforcement learning. The core problem of long-range planning is finding an efficient way to search through the tree of possible action sequences — which, if left unchecked, grows exponentially with the length of the plan. Existing non-learned planning solutions from the Task and Motion Planning (TAMP) literature rely on the existence of logical descriptions for the effects and preconditions for actions. This constraint allows TAMP methods to efficiently reduce the tree search problem but limits their ability to generalize to unseen and complex physical environments. In contrast, deep reinforcement learning (DRL) methods use flexible neural-network-based function approximators to discover policies that generalize naturally to unseen circumstances. However, DRL methods have had trouble dealing with the very sparse reward landscapes inherent to long-range multi-step planning situations. Here, we propose the Curious Sample Planner (CSP), which fuses elements of TAMP and DRL by using a curiosity-guided sampling strategy to learn to efficiently explore the tree of action effects. We show that CSP can efficiently discover interesting and complex temporally-extended plans for solving a wide range of physically realistic 3D tasks. In contrast, standard DRL and random sampling methods often fail to solve these tasks at all or do so only with a huge and highly variable number of training samples. We explore the use of a variety of curiosity metrics with CSP and analyze the types of solutions that CSP discovers. Finally, we show that CSP supports task transfer so that the exploration policies learned during experience with one task can help improve efficiency on related tasks.

Under review as a conference paper at ICLR 2019
FLEXIBLE AND EFFICIENT LONG-RANGE PLANNING THROUGH CURIOUS EXPLORATION
Anonymous authors Paper under double-blind review
ABSTRACT
Identifying algorithms that flexibly and efficiently discover temporally-exter exxtendd multi-phase plans is an essential next step for the advancement of robotics and model-based reinforcement learning. The core problem of long-range planning is finding an efficient way to search through the tree of possible action sequences which, if left unchecked, grows exponentially with the length of the plan. Exist- ing non-learned planning solutions from the Task and Motion Planning (TAMP) literature rely on the existence of logical descriptions for the effects and precondi- tions for actions. This constraint allows TAMP methods to efficiently reduce the tree search problem but limits their ability to generalize to unseen and complex physical environments. In contrast, deep reinforcement learning (DRL) meth- ods use flexible neural-network-basee that generalize naturally unseen circumstances. However, DRL methods have had trouble dealing with the very sparse reward landscapes inherent to long-range multi-step planning situations. Here, we propose the Curious Sample Planner (CSP), which fuses elements of TAMP and DRL by using curiosity-guide sam- pling strategy learn efficiently explore the of action effects. We show that CSP can efficiently discover interesting and complex temporally-extende for solving wide range of physically realistic 3D tasks. In contrast, standard DRL and random sampling methods often fail to solve these tasks at all or do so only with a huge and highly variable number of training samples. We explore the use of variety of curiosity metrics with CSP and analyze the types of solutions that CSP discovers. Finally, we show that CSP supports task transfer so that the exploration policies learned during experience with one task can help improve efficiency on related tasks.
function approximators to discover policies
plans
INTRODUCTION
Many complex behaviors such as cleaning a kitchen, organizing a drawer, or cooking a meal require plans that are combination of low-level geometric manipulation and high-level action sequeneing. For example, boiling water requires sequencing high-level actions such as fetching a pot, pouring water into the pot, and turning the stove. In turn, each of these high-level steps consists of many low-level task-specific geometric action primitives. For instance, grabbing a pot requires intricate motor manipulation and physical considerations such friction, force, etc. The process of combining these low-level geometric decisions and high-level action sequences is often referred to as multi-step planning.
While high-level task planning and low-level geometric planning are difficult problems on their own, integrating them presents unique challenges that add further complexity. Task and Motion Planning (TAMP) is a powerful approach to the problem which constructs plans in logical terms that execute a sequence of macro-actions that are composed of geometric motion plans (Fikes Nilsson 1971 Dantam et al., 2016). While TAMP has been successful at generating temporally extended multi- step plans that conform to geometric constraints, requires actions to have defined preconditions and effects that modify the logical description of the world state. This is an unreasonable assumption for complex physical tasks because real-world effects and preconditions are often unknown or diffi- cult to describe with logical predicates. These assumptions limit the flexibility and robustness of the TAMP approach. In addition, TAMP is computationally costly because it requires geometric motion planning for each macro-action sample. In contrast, deep reinforcement learning (DRL. methods

---

Under review as a conference paper at ICLR 2019
have shown success learning flexible policies for variety of complex tasks in unstructured do- mains (Arulkumaran et al., 2017). However, DRL methods typically require very large sample sizes of successful trajectories to learn useful policies and thus have trouble in sparse reward landscapes (Choi et al. 2019; Riedmiller et al. 2018). Multi-step planning, by its very nature, involves sparse reward scapes, as highly specific action sequences must be executed before any positive rewards are observed. As such, flexible multi-step planning is an open challenge in artificial intelligence and algorithmic robotics.
In this paper we combine aspects of TAMP and DRL to achieve progress toward efficient and flexible multi-step planning. We introduce the Curious Sample Planner (CSP), which uses curiosity to bias the search through action space toward novel points in the state space. Our method combines the flexibility and transferability of DRI. with the temporally extended multi-step planning of TAMP. We illustrate the power of CSP in variety of qualitatively distinct problems in multi-step planning for which logical descriptions of action effects would be difficult to construct, including the build- ing of complex structures and the discovery of simple machines for achieving challenging physical goals. We show CSP dramatically improves sample complexity compared to random TAMP explo- ration and DRL baselines. We compare a variety of distinct curiosity metries for use with CSP and demonstrate that dynamics- based curiosity performs well on tasks which require many dynamic ob- ject interactions while state-based curiosity performs better for practure-building tasks. Finally, we show that CSP can be used to achieve substantial transfer between related but qualitatively distinct tasks, utilizing knowledge in one domain increase performance in a similar domain.
2 RELATED WORK
Finding policies in sparse-reward environments has been ever-present challenge for modern deep reinforcement learning. Model-free RL algorithms either require millions of samples in a sparse- reward setting (Mnih et al., 2015 Hessel al. (017) or precise reward curriculum that gives checkpoints or gradients toward the goal (Heess et | (017)). While these algorithms yield convinc- ing performance, humans can learn these tasks without strict reward curricula or massive sample
sizes.
Model-based RL presents promising approach for solving these sparse-reward tasks by allowing agents to create plans through virtual experimentation. Rather than learning direct policy or value function, model-based RL focuses on learning the local or global dynamics of the task to enable planning (Kaiser et al. 2019| Levine et al. 2015) or to create imagined environment roll-outs for more efficient policy training (Weher et al., 2017 Ha & Schmidhuber 2018). Once the structure of the task is known, the question becomes how to best plan under that structure. Our algorithm is a method for multi-step planning after the environmental dynamics and task structure are known.
A classical approach to planning is that of sample-based geometric motion planners. Rapidly Ex- ploring Random Trees (RRT) and Probabilistic Road Maps (PRM) combine goal-directed sampling with off- target sampling balance exploration of the state space with exploitation of knowledge about the goal configuration (Lavalle| [9988] Kavraki et al.) 1996). While these algorithms can work even for high dimensional configuration spaces, they are computationally intractable for tasks with the complex constraints that often exist in real world settings (Kingston et al.) 2018). For a robotic manipulator, grasping an object has a necessary condition that the agent's 's manipulator is in posi- tion to grasp that object. These constraint barriers in the configuration space render the goal-directed component of motion planning ineffective and necessitate random exploration the entire configura- tion space. So while sample-based based geometric motion planners are effective at simple tasks, they fail for temporally extended multi-step tasks with intricate constraints. We use sample-based geometric motion planners as part of our solution to the multi-step planning problem.
More recently, Task and Motion Planning has shown success in developing temporally extended multi-step plans under both fully known and partially observed environment dynamics. A number of TAMP algorithms (Kaelbling & Lozano- Pérez, 2013 Gravot et al.) 2005 Hertle et al., 2012 Srivastava et al. 2014) have solved tasks such as block stacking, object packing, table setting, and much more. TAMP algorithms generally iterate between a motion planning step (using RRT. PRM, etc) and symbolic planning step using task planning algorithms such Fast Downward Planning (Helmert) (006) or Fast Forward Planning (Hofimann & Nebel (000T. However, because geometric motion planning is relatively slow, Task and Motion Planning is computationally burdensome. Ad-
2

---

Under review as a conference paper at ICLR 2019
ditionally, defining logical predicates for the effects and preconditions of macro aiteror-cttons/ is manual undertaking which limits the flexibility of the solution. We take inspiration from Task and Motion planning by using geometric motion planners procedures used in executing macro-actions, but increase planning efficiency and flexibility by intelligently sampling macro-actions and avoiding logical descriptions of effects and preconditions.
Many other methods have been proposed that increase the efficiency and flexibility of Task and Motion Planning. Supervised learning of the preconditions and effects of certain macro-actions removes the need for manual specification, but requires constructing task-specific training datasets (Kroemer& & Sukhatme, 2016] WWng et al. [0118). Other work has focused on factoring the planning problem into submanifolds with analytic constraints in order to reduce the size of the search space. (Garrett al. 2018: Vega-Brown & Roy 2018). TAMP can be accelerated by generating a sampling distribution around the goal trajectory using GANs (Kim et al.) 2018) or by using reinforcement learning to learn search heuristics through expert examples or previously solved problem instances (Chitnis et al. 2016 Kim et al., 2019). In this work, we also aim to build an algorithm that can utilize information gained from previous problem instances to speed up planning and generalize to related tasks. However, instead of supervising expert examples or task- lask-specific training sets, aim to use a self-supervision signal to increase the efficiency of multi-step planning.
Along with model-based RL and planning, intrinsic motivation, also known curiosity, has been used to turn sparse reward spaces into dense ones by self-supervising on an auxiliary task. Some commonly used auxiliary tasks are forward dynamics (Burda al., 2019), random network distil- lation (Choi et al. 2019), and state estimation (Mitash et al. 2017). The error from these predictor networks are considered intrinsic reward for some reward optimization algorithm. These auxiliary tasks encourage state-space and action-space coverage and increase the probability of encountering sparse rewards over random policies with epsilon epsilon-greedy exploration (Jaderberg et al., (016). Our algorithm uses the principle of self-supervision on auxiliary tasks to accelerate goal discovery in multi-step planning-
MATHEMATICAL DESCRIPTION
3.1 PROBLEM DESCRIPTION
In its general formulation, our problem consists of a robot R in an environment with dynamic movable objects D1, Dr and static objects S2, The robot R. has an associated con- figuration space Q containing its possible configurations in its environment, so that dim( Q) the number of degrees of freedom of RR There is larger state-space S which, in addition to Q, includes includes the total description of the positions and orientations of the dynamic and static objects, as well as their velocities, masses, and shapes, together with any (possibly dynamic) state-space con- straints such as links or joints that may (permanently or temporarily) exist between objects in the environment.
The robot is equipped with a space A of primitive actions that cause reliable (though possibly stochastic) changes in the state. The robot is also equipped with set of macro-actions M that can be translated into sequences of motion primitives in A using geometric motion planning and in- verse kinematic procedures. Each macro-action is parameterized by specific continuous and discrete parameters, and has set of implicitly-defined feasible conditions.
In our specific experiments, we use mounted robot arm with seven bounded revolute joints result- ing in a seven dimensional configuration space, with fully-extended length r. Our robot's action primitives A consist of (1) a 7-dimensional continuous rotation space composed by linearly inter- polating between 0, and 6g where 0,, 02 are 7-vectors corresponding to bounds on each rotational degree of freedom, and (2) the ability to add a link constraint between any two objects in the envi- ronment, as long as the objects are touching and the robot arm is sufficiently close to both. We use Bullet (Coumans] 2015), a flexible physics simulation library, to execute robot actions in simulation. The robot has access to a P ckP lace macro-action which takes target object, a goal position, and a goal orientation as input, and moves the target block to the goal position and orientation. The robot also has access to AddConstraint macro which takes two target objects input, moves them into position for linking, and performs the link; and a corresponding RemoveConst raint that breaks the link if the objects are connected. While this setup is fairly typical of roboties appli-
3

---

Under review as a conference paper at ICLR 2019
Action Selection Networks
Planning
(c) Forward Dynamics
170
Sample Expansion
Nodes
Reward
my
St
Value
St
Novelty Signal
(e) Tree Expansion
(d) Curiosity Module
Figure 1: a.) The action selection networks select actions that maximize curiosity. b.) Parameterized macro-actions converted to motion primitives c.) The forward dynamics module predicts the effects of executing those motor primitives in a particular state, d.) The curiosity module represents the current world state and outputs one of the curiosity metrics (see section 433 expanded.
e.) The search tree
cations, the CSP algorithm we describe below makes no assumptions about the effects of actions on objects in the environment, and our method could be applied to complicated environments containing soft-body objects, cloths, and multiple robotic manipulators.
The robot must its actions and macro-actions achieve goal, defined by state-space subset G' C S, by manipulating the state from some specified initial conditions into the G subset. For example, for our robotic arm, the task of moving dynamic object D, outside the reachable radius of the robot would be defined by G {s SID;, 2 Diy > r²). Though this type of goal definition is very simple and natural, finding a solution to such a task require complex strategic planning e.g. perhaps requiring the discovery and construction of simple machine, such ramp that could be used to slide the block out of the robot's 's reach (see section[4.1]
Finally, the robot has access (deterministic) dynamics model, f A S, that maps states and actions to future states i.c. f(1,, the robot access directly the underlying Bullet physics simulator as black box, but future re- search could involve having f be a learned (possibly non-deterministic) forward dynamic prediction network that must be inferred from agent experience (such as e.g. (Mrowaa al. 2018)).
S2++r for some >> 0. In this work, f simply gives
3.2 CURIOUS SAMPLE PLANNER
Below, we describe the Curious Sample Planner. We first describe the CSP's system architecture, including its constituent neural networks and geometric planning module. We then describe the core curious tree-search algorithm by which CSP uses these modules construct multi-step plans.
System Architecture: CSP is comprised of four main modules (Fig. [] The action selection networks include an actor-network मe + M and critic-network V SS + IR (Fig. TT), which learn to select macro-actions and choose parameters of that macro-action given particular state. The action selection networks have two primary functions: maximizing curiosity in action selection and avoiding infeasible macro-actions. The networks are trained using actor-critie rein- forcement learning (PPO (Schulman et al. 2017)) where Ty has learnable parameters and V, has learnalle parameters db The networks select feasible actions which maximize the novelty signal,
4

---

Under review as a conference paper at ICLR 2019
leading to actions which result in novel configurations dynamics. The actor network outputs continuous (real-valued) vector which translated into a macro-action with both discrete and con- tinuous parameters. The forward dynamics module f S A + S (Fig. [Tt) takes a state and an action primitive, simulates forward fixed time т, and returns the resulting state. This forward dynamics module is used by geometric planning module (Fig. [T) convert macro-actions in M into feasible sequences of motor primitives in A. Finally, the curiosity module is neural network Hg (H stands for heuristic, Fig. ( that takes states inputs and returns a curiosity score, with learnable parameters B. The exact input and output of the curiosity module is dependent on the type of curiosity being used (see 4.3)
Input: Initial state so, Goal set G, dynamics Output: Path 40: $1), (8n-1, On-1, sn)) where 8n E G, G,f( 0,) = 81+1
T= (1s0).00 0) Randomly Initialize Te. Vg. Hg Initialize P(so) =
while V G = 0 do
S <- batch sample P(V);
// Expand novel states
M + (S)
A motion planning using f for each M
S <- f(S, A)
L +- novelty metrie;
11 Various metrics described in sec 4.3 // No reward for infeasible actions
10
C(A 0) = 0:
Train H3 minimize L(A + 0)
12
Update To. V to maximize L:
// PPO Training
13
V S UV Add each (St, S)) E softmax (novelty metric for each V)
14
15
end
16 return Puth in T from solo 8g E VNG
Algorithm 1: The CSP algorithm.
CSP Algorithm: its core, CSP an algorithm for efficiently building a search over the state space using parameterized macro-actions (see Algorithm []. The algorithm starts by initializing tree (V. E) in which the vertices are points in the continuous state space, edges are maero- actions with defined parameters, and paths are sequences of macro-actions which are guaranteed to transition between states at each end of the path under the dynamics model. The tree starts with single vertex so which is the start state of the multi-step planning problem. The algorithm also initializes a probability distribution over the tree vertices such that P(su) = 1. A batch size B states are then sampled from P and passed into na resulting in set of cate/macco-acttinn pairings.
The next step of the algorithm to convert the selected macro-actions into primitive action se- quences using combination of inverse kinematies and geometric motion planning. We RRT- Connect (Kuffner & Valle 2000) for motion planning and the recursive Newton Fuler algorithm (Luh al. 1980) for inverse kinematics. The exact routine for converting macro-actions to primi- tives is specific to the macro-action. In some cases it is infeasible to convert macro-actions to motor primitives. (For example, it is infeasible pick up object that is out of the robot's reach.) In such cases, the planning module returns an empty sequence of primitives. These feasibility conditions are not explicitly represented as logical preconditions, but are discovered from failed attempts at inverse kinematics motion planning. Over time, the action selection network learns to focus only on feasible macro-actions. The resulting sequence of action primitives are passed into the black-box dynamics module to get corresponding batch of future states.
In order to determine which states and actions should be further explored, the algorithm creates a curiosity score for each of the selected macro-actions. Passing the batch of states and future states through the curiosity module will give a prediction loss, or novelty score. subset the new states with high novelty scores in the batch are then added as vertices in the search tree and the probability distribution adjusted to give more weight to states with high novelty scores. Although is not strictly necessary for CSP to function, for increased computational efficiency we discard vertices
5

---

Under review as a conference paper at ICLR 2019
(a) STACK
(b)
PUSH-AWAY
Ce)
BOOKSHELF
(d) LAUNCH-BLOCK
Figure 2: Visualizations each of the four task categories used to test CSP. Top Row: representative initial state for each task. Bottom Row: representative final state for each task. The Block-Stack task requires the rohot to find way to cause one block to remain stably at a high z-position without touching the arm, necessitating the building of tower. The Push-Away task requires the robot to push the small (blue) block beyond the reach of the robot arm, which, depending on the circumstance might require the robot to build a ramp. The Bookshelf tasks requires the robot knock a book of shelf which is further away from the robot that it can reach, necessitating that the robot discover how to build a simple tool from the provided blue rods. The Launch-Bi Block task requires the robot to launch the small red block off the far end of the seesaw, but it can only do so (we think) by putting all five of the other blocks as counterweight into the bucket and detaching the pulley rope, causing the now-he heavy bucket to crash into the close end of the seesaw. Tasks are described in more detail in Section
with low probability (typically the bottom 90% of the distribution), as they are extremely unlikely to be sampled. After scores are calculated, the input states that have infeasible macro- macro-actions are given curiosity score of zero, strongly disincentivizing infeasible macro-action selection. The curiosity module is trained from the losses generated by the batch and the novelty score is used as the reward for the training of the action selection networks. This process repeats until the algorithm reaches state in the goal subset or is exogenously terminated.
EXPERIMENTS
4.1 TASKS
Each of the four task types below are extremely easy to specify in the nen-restrietive goal semantics accepted by CSP, but often require complex multi-step planning solve, and would be extremely statistically improbable to be solved through random exploration. From a DRI. point of view, such tasks correspond to extremely sparse reward landscapes. They also involve complex and fairly pre- cise continuous motor manipulation of both rigid and non-rigid objects, using sequences of macro- actions whose preconditions would be very challenging to specify using logical predicates. The task set also affords some natural opportunities for cross-task transfer (e.g. between tower-building in Block-Stack and ramp building for Push-Away).
Block-Stack: In this task, the robot is provided with of K cubic blocks of size h, and the robot's goal is to cause at least one block to stably remain position at z-position of greater than (K 1) h, without being in contact with the robot arm, for at least seconds. Of course, the

---

Under review as a conference paper at ICLR 2019
only feasible way for the robot to solve this problem is to stack the K blocks in stable tower, but the robot is not provided with any reward at intermediate unsolved conditions. This type of block stacking is commonly used task for evaluating planning algorithms because it requires detailed motor control and has a goal that is sparse in the state space.
Push-Away: In this task, the robot is provided with several large cubic blocks, flat elongated block, and one additional smaller "target" object that can be of variable shape (e.g. a cube or a sphere) and material (with e.g. high or low coefficient of friction). The objective is to somehow push the target object outside the directly reachable radius of the robotic arm. Depending on the situation, the solution to the task can be very simple or quite complex. For example, if the target object is spherical with low friction, the robot could solve the task simply by dropping it against one of the larger blocks, causing to roll away. However, if the target object cubic with high friction, it may be necessary for the robot to discover how to construct and use simple machine such as ramp- - e-g. consisting of the large blocks stacked up, with the elongated block placed at one end on the stack as an inclined plane down which the small object can be slid.
Bookshelf: this task, the environment contains a bookshelf with a single book on it. The robot is also provided with two elongated gectangular-prist locations in the environment. The goal is to knock the book off the bookshelf. However, the book and bookshelf are not only outside the reachable radius of the arm, but they are further than the combined length of the arm and a single rod. However, the robot can solve the task by (e.g.) combining the two rods in end-to-end configuration using the link macro-action, and then using the combined object to dislodge the book.
rods initial placed at random (reachable)
Launch-Block: In this task, the environment contains a pre-built rope-and-pulley with one end of the rope connected to an anchor block on the floor and the other attached to a bucket that is suspended in mid-air. A seesaw balances evenly below the bucket, with target block on the far end of the seesaw. The goal is to launch this target block into the air, above the seesaw surface. The robot could solve this task by (e.g.) gathering all blocks into the bucket and untying the anchor knot so that the bucket will descend onto the near end of the seesaw. However, due to the masses of the blocks and the friction the seesaw, this can only happen when the combined weight of all five blocks are used.
4.2 NEURAL NETWORK SETTINGS AND CURIOSITY METRICS
Throughout our experiments, the action selection networks and Vo are three-layer networks with 64 hidden units each, using the tanh activation function.
There is a wide range of potential curiosity and novelty metrics, and the optimal metric may be task-dependent. For this reason, we explored a range of such metrics from the recent literature, including: State Estimation (SE) (Mitash et al. 2017), Forward Dynamics (FD) (Burda al.. 2019), and Random Network Distillation (RND) (Choi et al. 2019). The neural network architecture of the curiosity module naturally needed vary curiosity module accepted images of the scene generated from multiple perspectives, in the form of 84 x 84 x (3 np)) pixel arrays (where np is the number of perspectives taken), while the architecture was a five-layer convolutional neural network with 3 convolution layers and two fully connected layers. For both FD and RND the architecture consisted of a four-layer MLP with 128 units in each layer. For FD, the inputs were the concatenated vector of system states and actions, while for RND the input simply consisted of the state vector. To enable fair comparison, we ensured that the number of trainable parameters in the curiosity modules was approximately even across all curiosity variants.
function of which metric chosen. For SE, the
4.3 QUANTITATIVE RESULTS
For each task, we tested CSP in comparison to a variety of baseline approaches. Simple random ex- ploration was tested by assigning uniform curiosity score each state and selecting macro-actions from a uniform distribution (CSP-No Curiosity). We also tested against vanilla implementations of vanilla PPO (Schulman et al. 2017) and A2C (Mnih et al. 2016) RL algorithms, using the same architecture as the curious action selector shown in figure [] but without the tree expansion and sam- pling module. Finally, tested non-planning based but still curiosity-driven DRL model, using
7

---

Under review as a conference paper at ICLR 2019
Solution Speed Comparisons
107
Vanille AZC
AND-PD With Resets
CSP-Nu Curiouity
104
CSP-RNE No AC
CSP-ID
kgg teamptes)
CSP-SE
30
161
3-Stack
Push-Waay
Launch
Solution speed for all tested algorithm variants across the five tested tasks. Within each group bars, the left-most bars represent baselines, while the right-most are the CSP models, including the control No-Curosity control and several CSP variants with different curiosity metrics (see text for details). The y-axis is on a log scale because the differences between CSP and non- CSP solutions is great that more subtle differences between various curiosity-metric variants of CSP would otherwise be too small to see. Each bar computed as the mean six independent trial runs, and error bars represent the standard error of the mean across trials. We set maximum number of samples (107) in order to ensure termination; in many case the baseline algorithms failed to terminate with a successful solution any trial.
Figure 3:
the PPO algorithm with the RND curiosity signal (PPO-RND). All metries and baselines are com- pared by measuring the number of macro-action samples needed to reach the goal over six different problem instances on five distinct tasks.
Overall, we found that CSP was dramatically more effective than the baselines at solving all four task types (Fig. 3. The control algorithms (including the CSP-No Curiosity baseline) were some- times able to discover solution in the simplest cases (e.g. the 3-Block task). However, they were substantially more variable in terms of the number of samples needed; and they largely failed to achieve solutions in more complex cases within 107-step maximum sample limit. The failure of random and vanilla PPO/A2C baseliness is not entirely surprising: the tasks we chose here are representative of the kind of rangerage planning problems that typically are extremely difficult for standard DRL approaches. The more sophisticated curiosity driven PPO-RND algorithm was able to make headway in few of the more complex circumstances, but was nonetheless substantially less powerful than the CSP variants We ran many additional actor critic and curiosity metric com- binations and discussed the results in Section D Comparing between curiosity metrics, we found that, while there is some task dependence, Random Network Distillation serves as a good general- purpose curiosity metric achieving the best performance four of the five tasks (see figure We also performed an ablation on the action selection networks (CSP-RND No AC) to see how ben- eficial they were to the planning process over random action selection. We found that all tasks except Bookshelf, performed slightly better with the action selection networks. While not vital to the planning process, these networks are key for transfer (see sec. 4.5
RL algorithms are often designed with multi-episodic environment in mind, in which the external con- troller repeatedly resets the system to known state (c.g. a video game level after the time limit has clapsed). Resetting can often help learning algorithms that would otherwise fail, since it rescues them from unrecov- erable states. In our case, such an unrecoverable state arises when objects needed to complete the task are prematurely pushed outside the available radius of the robot arm. The CSP algorithm easily able to handle this circumstance, because it can expand from previous explored nodes methods likely do not have this capacity. To give the baseline algorithms the best chance of working, we thus tested them in two situations: one without trial resetting (the "natural" circumstance for this work), and one with random trial resetting, characterized by set with probability lc-4 after each macro-action. Only the PPO-RND algorithm actually benefited from resets, however, and is the only one shown in Fig.
state space. However, standard
8

---

Under review as a conference paper at ICLR 2019
4.4 QUALITATIVE RESULTS
In the process of planning, CSP often found interesting state configurations and unexpected problem-solving techniques. A video illustrating some representative solutions found by CSP can be found here: http://yoouub///SssSyAAaa PUSH-AWAY task with a spherical ball as a target object, hoping CSP would build ramp and roll the ball out of the reachability zone. However, CSP instead found a simple solution consisting of dropping the ball directly next to another object in order to get enough horizontal velocity to roll out of the reachability zone. In an attempt to avoid this rather trivial solution, we then switched out the ball for a block with high coefficient of friction. However, again instead of always building ramp, CSP sometimes made use of its link macro-action to fix the block to one end of the plank and orient the plank so that the block was outsize the reachability zone. However, once the link macro-action was disabled, ramp-building behavior robustly emerged.
As an example, we initially developed the
4.5 TASK TRANSFER
A central aspect of human intelligence the ability to improve at solving problems over time and to use knowledge gained from solving one problem to more efficiently solve related problems. For this reason, we were curious whether CSP could transfer knowledge between problem instances with different initial conditions, and between different but related problems. We evaluated transfer both the within-task and between-task using 3-STACK, 5-STACK, and PUSH-AWAY tasks. We selected these tasks because they all share a common essential skill: stacking blocks.
We tested transfer from each learnable component of the system individually, to identify which components of the system can be effectively reused between problems, including (i) the baseline "no transfer" condition, where neither the curiosity module nor the action selection networks are transferred from another problem instance; (ii) condition where just the curiosity module or action selection networks were transferred; and (iii) a full transfer condition where both the action selection networks and the curiosity module are transferred between tasks.
Results (Fig. 4 show that transferring the curiosity module alone slightly improves planning ability between instances of the same task, but has minimal effect and can even be detrimental in the case of between-task transfer. On the other hand, transferring the action selection networks leads to a large increase in efficiency in all tasks except for 3-STACK. (The 3-STACK task is likely unaffected by transfer because it is solved so quickly that the networks have little time to train.) Full transfer results in transfer performance improvements similar that of transferring the action selection network alone. Gains arose for qualitatively identifiable reasons e.g. once CSP learned to solve the stacking problem, it tries stacking actions much more frequently as an initial guess, which is naturally useful for solving the Push-Away task. Transfer gains are reflected by the fact that the tree search procedure has learned to very effectively reduce the task tree, a particularly dramatic example of which is illustrated in Fig.
CONCLUSION
In this work, we introduce the CSP algorithm, fusion of classical task-planning and deep RL ap- proaches that is capable of flexible long-range multi-step planning in sparse reward settings. CSP consists of two learnable components which aim to maximize curiosity in action selection and state exploration. The first component is the action selector networks which reduces sampling of in- feasible or uninteresting effects in favor of macro- actiios and parameters which lead interesting and unseen effects. The second component is a curiosity module which guides exploration of the state tree toward novel state configurations by increasing the probability that those nodes will be ex- panded. We show that the addition of these components not only speeds up planning when compared to random action selection and reinforcement learning baselines but also has the ability to make oth- erwise intractable problems solvable. We also systematically examine the effect of different forms of curiosity on task performance, finding that the RND variant is the most effective overall. Finally, we look at the transferability of solutions to new instances of the same general problem and to lated but different problems, showing that networks trained to select actions states which result in interesting configurations perform better than those which are randomly initialized.
9

---

Under review as a conference paper at ICLR 2019
Task Transfor Resuts
(a)
No Transder
Curbeety Module Transter
Action action Trancler
Full Tranater
Ingiamplos
100
(b)
Reused Networks:
Curiosity Module
Actor
Critic
5-Stack Before Transfer, 187 Nodes
5-Stack After Transfer, Nodes
Figure 4: (a) Inter-task and between-task transfer efficiency measured by the number of samples needed to reach the goal, for each of several transfer policies (only curiosity module, only action selection module, and full network transfer). The CSP-RND variant used for all cases. Within each group of bars for given task, the difference between blue bar (No Transfer) and other bars represents transfer gain (note that the y-axis is logarithmic scale). Each condition
6 times and eтror bars represent standard error of the mean over runs. (b) Example of search trees for an instance of the 5-stack problem, before (left) and after (right) full network transfer.
Despite its initial successes, the CSP system has number of key limitations that will need to be addressed in future work. As a pure planning algorithm, CSP does not attempt to solve a directed learning problem over the course of many trials; when very large numbers of trials available for training, it plausible that Deep RL approaches would begin to catch up with and eventually exceed CSP performance. A very natural next step will be combine CSP with cross-trial learning algorithm to obtain the best of both worlds. will also be key to show that CSP is able to function effectively when only non-deterministic and noisy future prediction module is available, rather than the unrealistic perfect black-box predictor used in this work. It will be especially important determine if CSP compatible with learned future predictor, and of interest to determine JSP-generates plans could help accelerate the learning of such a predictor simultaneously while CSP modules are themselves learned. Another key direction for future work will be to apply CSP-like procedure to the discovery of macro-actions themselves (Barto et al. 2004 Machado talti-timescale procedure which more and more complex
ct al. 2017). resulting in a hicrarchical plans become routinized as macro-ate aactioss Finally, it will be of interest to compare the behavior sequences generated by CSP to data from experiments measuring patterns play and discovery in both human children and adults (Gopnik et al.) 2009 Begus et al.) 2014). both the similarities and differences between human decision making and exploration and algorithmic outputs will help inform the development of improved flexible and efficient planning algorithms.
REFERENCES
Kai Arulkumaran, Mare Peter Deisenroth, Miles Brundage, and Anil Anthony Bharath. Deep rein- forcement learning: A brief survey. IEEE Signal Processing Magazine, 34:26-38, 2017.
10

---

Under review as a conference paper at ICLR 2019
A. G. Barto, S. Singh, and N. Chentanez. Intrinsically motivated learning of hierarchical collections of skills. In Proceedings of International Conference on Developmental Learning (ICDL), MIT Press. Cambridge, MA, 2004.
Katarina Begus, Teodora Gliga, and Victoria Southgate. Infants learn what they want learn: Responding to infant pointing leads to superior learning. PLOS ONE, 9(10):1-4, 10 2014. doi: www.://ooaaa..ppo........
Yuri Burda, Harri Edwards, Deepak Pathak, Amos Storkey, Trevor Darrell, and Alexei A. Efros. Large-scale study of curiosity-driven learning. In ICLR, 2019.
Rohan Chitnis, Dylan Hadfield-Menele Abhishek Gupta, Siddharth Srivastava, Edward Groshev, Christopher Lin, and Pieter Abbeel. Guided search for task and motion plans using learned heuristics. In 2016 IEEE International Conference Robotics and Automation, ICRA 2016, volume 2016-June, pp. 447-454. Institute of Electrical and Electronics Engineers Inc., 6 2016. doi: 10. 11110//AA.22211.44499 7487165.
Jongwook Choi, Yijie Guo, Marein Moczulski, Junhyuk Oh, Neal Wu, Mohammad Norouzi, and Honglak Lee. Contingency-aware exploration in reinforcement learning. In International Confer- ence on Leurning Representations, 2019. URL htttp:://ppereviiee net/ forum?ide
HyxGB2AcY7
Erwin Coumans. Bullet physics simulation. In ACM SIGGRAPH 2015 Courses, SIGGRAPH '15, New York, NY, USA, 2015. ACM. ISBN 978-1-4503-3634-5. doi: 10.1145/27768802279970004 URL http: acm. /0. 1145/2776380
Neil T. Dantam, Zachary Kingston, Swarat Chaudhuri, and Lydia E. Kavraki. Incremental task and motion planning: A constraint-based approach. In Robotics: Science and Systems, 2016.
Richard E. Fikes and Nils J. Nilsson. Strips: A new approach to the application of theorem proving to problem solving. Artificial Intelligence, 2:189, 1971.
Caelan Reed Garrett, Tomás Lozano-Pérez, and Leslie Pack Kaelbling. Sampling-based methods for factored task and motion planning. CORR, 0as//800100665 2018. URI htttt:////axiii.
org/ hbs//ab//// 1801. 00680
A. Gopnik, A.N. Meltzoff, and P.K. Kuhl. The Scientist In The Crib: Minds, Bruins, And How Children Learn. HarperCollins, 2009. ISBN 9780061846111
Fabien Gravot, Stephane Cambon, and Rachid Alami. asymov: A planner that deals with intricate symbolic and geometric problems. In Paolo Dario and Raja Chatila (eds.), Robotics Research. The Eleventh International Symposium, pp. 100-110, Berlin, Heidelberg, 2005. Springer Berlin Heidelberg. ISBN 978-3-540-31508--
David Ha and Jürgen Schmidhuber. lution. In Advances in Neural Information Processing Systems 31, pp. 2451- 2463. Curran Associates, Inc.. 2018. URL htttp:://aapers nips cc/paper/
Recurrent world models facilitate policy evo-
7512-recurrent wor wor modee s- s-faciiiii wit.faiiiitteooolloooool worl dwwdelo wwt.ss..iittbbii.
ut inn https://
Nicolas Heess, Dhruva TB, Srinivasan Sriram, Jay Lemmon, Josh Merel, Greg Wayne, Yuval Tassa, Tom Erez, Ziyu Wang, S. M. Ali Eslami, Martin A. Riedmiller, and David Silver. Emergence of locomotion behaviours in rich environments. CoRR, abs/1707.022 022288,, 2017. URL http:
//arxiv..o warpiv..rg//as//1000000 02286
Malte Helmert. The fast downward planning system. Journal of Artificial Intelligence Research, 26:
191-246, 2006.
Andreas Hertle, Christian Dornhege, Thomas Keller, and Bernhard Nebel. Planning with seman- tic attachments: An
view. In Proceedings of the 20th European Conference Artificial Intelligence, ECAI' 12, pp. 402-407, Amsterdam, The Netherlands, The Nether- lands, 2012. IOS Press. ISBN 97881--61499-077-06 doi: 10.3233/978-141 1-1-79889--199999---7777 7-402. URL

---

Under review as a conference paper at ICLR 2019
Matteo Hessel, Joseph Modayil, Hado van Hasselt, Tom Schaul, Georg Ostrovski, Will Dabney, Daniel Horgan, Bilal Piot, Mohammad Gheshlaghi Azar, and David Silver. Rainbow: Combining improvements in deep reinforcement learning. CoRR, abx/1710.02298,, 2017. URL http://
arxiv. orgl abs 1710. 02298
Jörg Hoffmann and Bernhard Nebel. The ff planning system: Fast plan generation through heuristic search. J. Artif. Int. Res., 14(1):253-300 302, May 2001. ISSN 1076-9757. URL htttp://dl.
acm. org/ ci tat. on cfm? (223916622944. 12229944.622200 622404
Max Jaderberg, Volodymyr Mnih, Wojciech Marian Czamnekii, Tom Schaul, Joel Z. Leibo, David Silver, and Koray Kavukcuoglu. Reinforcement learning with unsupervised auxiliary tasks. CoRR, abs/1611.05397, 2016. (tttp:///aaxiis httgg/ab//61..... 05397
Leslie Pack Kaelbling and Tomás Lozano-Pérez. Integrated task and motion planning in belief space. Int. J. Rob. Res., 3209-10):11944-2227, August 2013. ISSN 0278-3649. doi: 10.1177/ 0278364913484077. JRLt://://dxx org. 117. 01111777777334413448888888
Lukasz. Kaiser, Mohammad Babaeizadeh, Piotr Milos, Blazej Osinski, Roy H. Campbell, Konrad Czechowski, Dumitru Erhan, Chelsea Finn, Piotr Kozakowski, Sergey Levine, Ryan Sepassi, George Tucker, and Henryk Michalewski. Model-based reinforcement learning for atari. CoRR, abss/1903.0037,, 2019, URL http: wweeiiivorr///ae abs /tt.a/////..00330/
Lydia Kavraki, Petr Svestka, J.C. Latombe, and M.H. Overmars. Probabilistic roadmaps for path planning in high-dimensional configuration spaces. Robotics and Automation, IEEE Transactions on, 12:566 580, 09 1996. doi: 109700997.0084439.
Beomjoon Kim, Leslie Kaelbling, and Toms Lozano-Peree Perez. Guiding search in continuous state- action spaces by learning an action sampler from targtt search experience. In Proceedings of the 32th AAAI Conference on Artificial Intelligence (AAAI). To appear. AAAI Press, 2018. URL
http: //ts//iii..../// csail. mit www. kim- aaail 18. pd
Beomjoon Kim, Leslie Pack Kaelbling, and Tomas Lozano-Perez. Adversarial actor-critic method for task and motion planning problems using planning experience. AAAI Confer- ence on Artificial Intelligence (AAAI), 2019. URL http://///lis csail wwt.scsaaaa...................ee edu/pubs/
kim-aaail 9, .dd
Zachary Kingston, Mark Moll, and Lydia E. Kavraki. Sampling based methods for motion planning with constraints. Annual Review of Control, Robotics, and Autonomous Systems, 1(1):159-185, 2018. doi: 10.1 anrurev cont 000111000000010::55
17-00127710226 URL htttts:///dd/ wttgg/..... 114//
Oliver Kroemer and Gaurav S. Sukhatme. Learning spatial preconditions of manipulation skills using random forests. 2016 JEEE-RAS 16th International Conference on Humanoid Robots (Hu- manoids), pp. 676-683, 2016.
James Kuffner and Steven Vallle Rit-connect: An efficient approach to single-query path plan- ning. volume 2, pp. 995-1001, 01 2000. doi: 10. 10.1 109/00001.200084473300
Steven M. Lavalle. Rapidly-exploring random trees: A new tool for path planning. 1998.
Sergey Levine, Nolan Wagener, and Pieter Abbeel. Learning contact-rich manipulation skills with guided policy search. CoRR, abs/1501.cn 151111111, 2015. UK//ittp.//arrrii.orr//////////
05611
J. Y. S. Luh, M. W. Walker, and R. P. C. Paul. On-Line Computational Scheme for Mechani- cal Manipulators. Journal of Dynamic Systems, Measurement, and Control, 102(2):69-75) 76, 06 1980. ISSN 0022-0434. doi: 10.1115/1.31499 3149599
URL hnttps://doi.. http://ooooooo110.111111..
Marlos C. Machado, Mare G. Bellemare, and Michael H. Bowling. A laplacian framework for option discovery in reinforcement learning. CoRR, abs/I 703.00966. 2017. URL httt:://arx: org/ Brg/abs//703.. 00956
12

---

Under review as a conference paper at ICLR 2019
C. Mitash, K. E. Bekris, and A. Boularias. A self-supervised learning system for object detection using physics simulation and multi-view, pose estimation, In JEEE/RSJ International Conference Intelligent Robots and Systems (IROS), Vancouver, Canada, 09/2017 2017. URL hetps:
7 //www. c8. www.gur..ee...... wwtpeeessedu//eee edd/
Volodymyr Mnih, Koray Kavukcuoglu, David Silver, Andrei A. Rusu, Joel Veness, Marc G. Belle- mare, Alex Graves, Martin Riedmiller, Andreas K. Fidjeland, Georg Ostrovski, Stig Petersen, Charles Beattie, Amir Sadik, Ioannis Antonoglou, Helen King, Dharshan Kumaran, Daan Wier- stra, Shane Legg, and Demis Hassabis. Human-level control through deep reinforcement learning- Nature, 518:529 EP 02 2015. URL htt.ttt::: /tt:::/dooooooooo org 10. 1 038 htt///aatrre122221
Volodymyr Mnih, Adrià Puigdomènech Badia, Mehdi Mirza, Alex Graves, Timothy P. Lillicrap, Tim Harley, David Silver, and Koray Kavukcuoglu. Asynchronous methods for deep reinforcement learning. CoRR, abs/1602.01783, 2016. URL URL http:tttp///iii...ott
abs / 0..02.07888
Damian Mrowca, Chengxu Zhuang, Elias Wang. Nick Haber, Li Fei-Fei, Joshua B. Tenenbaum, and Daniel L. K. Yamins. Flexible neural representation for physics prediction. In NeurIPS, 2018.
Martin A. Riedmiller, Roland Hafner, Thomas Lampe, Michael Neunert, Jonas Degrave, Tom Van de Wiele, Volodymyr Mnih, Nicolas Heess, and Jost Tobias Springenberg. Learning by playing solving spanse reward tasks from scratch. CoRR, abs/1 2018. URL http:://aaaxx
bxg/abs/1802. 10567
John Schulman, Filip Wolski, Prafulla Dhariwal, Alec Radford, and Oleg Klimov. Proximal policy optimization algorithms. CoRR, abs/1 707407677, 2017. URL http://arxxv..rr///25
1707. 06347
Siddharth Srivastava, Eugene Fang. Lorenzo Riano, Rohan Chitnis, Stuart Russell, and Pieter Abbeel. Combined task and motion planning through an extensible inneeer-nndeeeneentt interface layer. Proceedings IEEE International Conference on Robotics and Automation, pp. 639-646, 2014. ISSN 1050-4729. doi: 00.10090CCA.2014499996222
William Dega-Brown and Nicholas Roy. Asymptotically optimal planning under piecewise-analytic constraints. 2018.
Zi Wang, Caelan Reed Garrett, Leslie Pack Kaelbling, and Tomas Lozano-Perez. Active model learning and diverse action sampling for task and motion planning. In International Conference on Intelligent Robots and Systems (IROS), 2018. TtL:http://// /is. csail mit. wwl.eddu/ppp edt.pubs//ppp
wang-irosl8.pp
Theophane Weber, Sébastien Racanière, David P. Reichert, Lars Buesing, Arthur Guez, Danilo Jimenez Rezende, Adrià Puigdomênech Badia, Oriol Vinyals, Nicolas Heess, Yujia Li, Razvan Pascanu, Peter W. Battaglia, David Silver, and Daan Wierstra. Imagination-augmeente agents for deep reinforcement learning. CoRR, abbs/1707.06203 2017. URL http: orq/ org/abs/ abs/ 7 06203
13

---

Under review as a conference paper at ICLR 2019
A CURIOSITY METRICS
A. STATE ESTIMATION CURIOSITY
The objective of the state estimation curiosity module estimate the underlying state of the world through a set of visual observations. State configurations that are visually distinct from previously seen configurations will lead to high losses. The curiosity module and loss function are defined as follows.
HB Im + S
C. | (J
s:ll
Where Tm is the space of 84 84 x 3 ?p matrices containing dp images captured from various perspectives in the scene, & E denotes the state, and is an index into the batch. In our experiments, used single top-down perspective (np
A.2 FORWARD DYNAMICS CURIOSITY
The objective of the forward dynamics curiosity module is to estimate future states given current states and actions. Given deterministic physics, actions modify states deterministically. Therefore, it is theoretically possible to estimate future state from action and the current state. Rare or unpredictable object-object interaction will yield the highest loss dynamics prediction and therefore will be sought out by the curiosity module. The curiosity module and loss function described as follows.
S
1:8xA=2
C. = [H{(s)) - f(s;, a;)1
Where f is the deterministic dynamics model which maps state and action to state time t for some fixed 7.
A.3 RANDOM NETWORK DISTILLATION CURIOSITY
Random Network Distillation (RND) is a recently proposed method for obtaining state-space cover- age. This is achieved by training model to fit some fixed random transformation of the input data. The RND curiosity module achieves low loss when it matches the transformation. This results in high curiosity module loss for unseen states.
H3, 13 S S S
C, = H))))) - (8:))1
Where T the fixed mapping that is initialized with random parameters. Among other mappings including random linear and non-linear mappings with uniformly and gaussian distributed weights, we found that the best performing mapping was random permutation state vector.
B NETWORK STRUCTURES AND HYPERPARAMETERS
B. ACTOR CRITIC POLICY AND VALUE NETWORKS
The structure of the policy network is multilayer perceptron model with three layers. The input size the dimensionality of the state space and the output size is equivalent to the dimensionality of the action space. Action space and state space dimensionality The hidden layers each have hidden units with tanh activation functions. The the value network has equivalent architecture but with output size of one.
described in more detail below.
14

---

Under review as a conference paper at ICLR 2019
For PPO and A2C training, used the same hyperparameter: during action selection network training and baseline experimentation. For PPO we used a batch size of 128 with 1024 samples collected per update. Y 0, ir = 7e 4, dependent reasons. For the curiosity networks, we found that led to higher performance solution speed and selection of feasible actions. In curiosity-guided RI. baselines, we found difference between values of gamma. We also selected value and entropy coefficients of 0.5 and 0.0 respectively and restricted the magnitude of the gradient 0.5. For each batch, did PPO epoch updates. For A2C, we used the same learning rate and coefficients as PPO.
0.2. We chose ?
for multiple application- both
B.2 STATE ESTIMATION CONVOLUTION NETWORK
The state estimation network takes in visual input from perspectives in the and outputs estimated state. We achieved this image to state mapping using a convolutional neural network with three convolutional layers and two fully connected layers, all with ReLU nonlinearities. Because we used single perspective for all of our experiments, the input size is 84 x 84 x 3 and the output size the dimensionality of the state space. The convolutional layers have kernel sizes of 8, and strides of 4, 2, 1. The fully connected layers each have 128 hidden units.
B.3 FORWARD DYNAMICS NETWORK
The forward dynamics network maps 8t; Or 8t+T. This is achieved using a multilayer perceptron model with 3 layers, 64 hidden units per layer and tanh activation functions. The input size is the sum of the action and state dimensionality and the output size the dimensionality of the state.
B.4 RND NETWORK
The RND network maps states to transformed states. The size we used for the transformed state is equivalent to the size of the state. Therefore, the input and output sizes are both equivalent the dimensionality of the state. The architecture consisted of three layer multilayer perceptron model with lanh activation functions and 64 hidden units for each hidden layer.
B.5 UNIVERSAL CURIOSITY MODULE HYPERPARAMETER
For training the curiosity networks, we used an Adam optimizer with a learning rate of 5 x 10-5, batch size of 128, and 1024 samples per update. We also found that performance was higher when we used an adaptive number of samples per update. The number of samples is increased until the loss below threshold after training. This improves CSP by giving the curiosity network time train to baseline before adding nodes to the tree.
C TASK SPECIFIC STATE SPACE AND ACTION SPACE
C.I STACK STATE
This set of tasks consists of k cubes. Each cube has it's SF(() configuration parameterized by three positional degrees of freedom and three euclidean rotational degrees of freedom. When the linking macro-action is enabled, the state also contains indicator values for each pairwise object combination.
C.2 PUSHI-AWY STATE
This task contains two larger cubes, smaller cube, and elongated, flat rectangular prism. Again, the dimensionality of the state space is SE(3) for each object along with pairwise indicators when the linking macro-action is enabled.
15

---

Under review as a conference paper at ICLR 2019
Metric-Algorithm
3-Stack
5-Stack
Push-Away
Bookshelf
Push-Away
RND-PPO
3.34c6 + 1.92c6
1c7 +0
1c7 +0
1c7 =
1c7 = 0
Forward Dynamics-PPO
5.01e6 + 2.03e6
1e7 10
1e7 +0
1e7
le7 + 0
Effect Prediction-PPi
le7 0
1e7 0
1e7 +0
+
1e7 + 0
RND-A2C
2.72c6 + 1.92c6
1e7 +0 0
1e7 +0
1e7 1e7
1c7 + 0
Forward Synamics-A2C
Le7 =0
1e7 + 0
1e7 +
1e7
1e7 + 0
Effect Prediction-A2C
1e7 :0
1e7 1 0
1e7 1
le7
1e7 + 0
Table 1: Comprehensive List of Actor-Critic/Curiostty Metric Evaluations
C.3 BOOKSHELF STATE
This task contains two rods elongated rectangular prism rods and a rectangular prism book, each with (F) configuration space and pairwise linking. Linking is always enabled for this task because it is required to complete it.
C.4 LAUNCH-BLOCK STATE
The state space for this task contains five small cubes, one larger cube that sits the end of the seesaw, and the angle of the seesaw itself. Each of the cubes has an SE((3) configuration space and the seesaw has a bounded real configuration with a single degree of freedom.
C.5 TRANSFER STATES
When comparing between tasks, the action spaces and state spaces need matching dimensionality to ensure the networks could be substituted. We achieved this by making sure each problem instance had an equivalent number of objects, even if was unnecessary to use objects complete the task. For the transfer results stated in this paper, this equated to using five objects for each task.
C.6 ACTION SPACES
When both linking and pick-place macro-actions are enabled, the action space needs to contain information for choosing which object to move (k total), the goal pose of the object (dof), which of the object pairs link or unlink, and which macro-action to select (linking pick-place). Therefore, the dimensionality of the action space is (5) + (dof + 1) + 2. The macro-action, link, and object discrete variables are chosen by performing an argmax over the respective section of the action space. link macroaction is selected between two objects that are already linked, this is considered unlink action and the constraint between the objects is removed. If link chosen between two objects which aren't in contact, then the action considered to be infeasible. For implementation simplicity, we use 222 variables for the linking indicator rather than (6) and ignore any redundant pairs.
D ADDITIONAL BASELINES
We ran several other reinforcement learning baseline comparisons including Forward Dynamics- PPO, Effect Prediction-PPO, RND-A2C, Forward Dynamics-A2C and Effect Prediction A2C with- resets, and results are shown in Table[]] As expected, we found results similar to RND-PPO No Reset in Figure 3 Specifically, we found that all additional baselines solved 3-Stack in to two of the experiments, failed ever solving the other tasks.
16


Reviews:
[{"review_id": "Syl0y5O6Fr", "reviewer": "AnonReviewer2", "summary": "", "questions": "", "limitations": "", "rating": 1, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "The idea of the paper is to augment a planner with a curiosity module to reduce the number of traversed paths, resulting in a speedup. The paper presents experiments where it is shown that deep RL methods are outperformed in that question.\n\nI recommend to reject the paper.\n\nThe reason for this are threefold.\n- The paper is crowded with text and ends up to be hard to follow. The actual contribution is hard to distill.\n- The method compares to deep RL baselines. But these are *not* planning algorithms, instead these are RL methods. The paper does not compare to planners, which are tailored to solve the problem the paper adresses.\n- The reader is left alone to place the work within the literature. The abstract and introduction do not have a single cite; terms like TAMP and multi-step planning are mentioned and certain properties of them are stated without resorting to where a reader could look those up. It is not the readers task to use a search engine to reverse engineer the authors writing!\n\nI think that the authors need to reformulate what the contribution of their work is. That needs to be presented in a more abstract way, without focusing on the experimental setting prematurely. The experimental setup is ok if the method is restricted to robotic tasks, but too thin for the general setting of efficient planning with sparse costs. Planning algorithms that are tailored towards huge spaces shou;d be used as base lines instead of deep RL methods.", "extra_scores": {"experience_assessment": "I have read many papers in this area.", "review_assessment:_thoroughness_in_paper_reading": "I read the paper at least twice and used my best judgement in assessing the paper.", "review_assessment:_checking_correctness_of_experiments": "I assessed the sensibility of the experiments.", "review_assessment:_checking_correctness_of_derivations_and_theory": "I assessed the sensibility of the derivations and theory."}, "extra_text": {}}, {"review_id": "S1gqiLITFH", "reviewer": "AnonReviewer3", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "===== Summary =====\nThe paper introduces Curious Sample Planner (CSP) a long-horizon motion planning method that combines task and motion planning with deep reinforcement learning in order to solve simulated robotic tasks with sparse rewards. The CSP algorithm considers two different hierarchies of actions: primitive actions, which control the rotation of several joints in a robotic arm, and macro-actions corresponding to complex behaviours  such as moving from one position to another or linking two objects together. Macro-actions are selected using the actor-critic architecture PPO and then turned into primitive actions using geometric motion planning and inverse kinematics. Specifically, RRT-Connect is used for motion planning with the recursive Newton-Euler algorithm for inverse kinematics on a perfect model of the environment to determine the specific sequence of primitive actions necessary to execute the macro-action. As CSP is interacting with the environment, it also builds a tree of states in the environment connected by the macro-actions leading to each of them. Each vertex of the tree is assigned a curiosity score, which is used as an exploration bonus for PPO and to determine the probability with which each vertex is sampled from the tree for future exploration. The whole process is repeated until a feasible path from the initial state to the goal state is found. The paper provides empirical evaluations in four different tasks where it compares the performance of CSP with three different curiosity measures to the performance of PPO and A2C. The results show that CSP accomplishes each task while using significantly less samples. Moreover, a second set of experiments is presented that show the potential for transfer learning across tasks using CSP. \n\nContributions:\n1. The paper introduces CSP, a successful combination of task and motion planning and deep reinforcement learning that can discover temporally extended plans. \n2. The paper demonstrates a statistically significant improvement in performance over PPO and A2C in the four robotic tasks that the paper studies. \n3. The paper shows evidence that CSP might facilitate transfer learning across similar tasks. \n\n===== Decision ===== \nThe paper represents a significant contribution to the reinforcement learning and task and motion planning literature. The main algorithm is well motivated from previous literature and demonstrate a significant improvement over previously proposed deep reinforcement learning methods. Moreover, the ideas are presented clearly and logically throughout the paper and the empirical evaluations clearly support the claims about the performance of CSP. However, I have concerns about the reproducibility of the results because of the little amount of details provided about the hyper-parameter selection and settings and about the network architectures and loss functions. Thus, I consider that the paper should be rejected, but I am willing to increase my score if my comments are properly addressed.\n\n===== Questions and Comments =====\n\n1. Although the ideas in the paper are presented clearly, the algorithms and methods are presented mostly at a very high level. There are no details about the losses used in lines 11 and 12 of Algorithm 1 and there are no specifications about the network architectures and the hyperparameter settings and selection for each algorithm. This raises two concerns. First, this hinders reproducibility and future work by other authors that might be interested in building upon the ideas presented in the paper; this would also decrease the impact of the paper. Two, it is difficult to determine if the comparisons against A2C and PPO were fair without any information about the hyper-parameter selection. Thus, I consider these details should be included and I would consider increasing my score to accept if this was properly addressed. Specifically, I think the paper should include this:\n- The hyper-parameter settings for each different algorithm and an explanation about how they were selected. \n- A detailed description of the network architectures used in the experiments.\n- A definition for the loss functions used for the policy network, the value network, and the curiosity module. \n\n2. As mentioned in the Decision section above, the paper clearly demonstrates an improvement over previously proposed deep reinforcement learning algorithm. However, there are no comparisons to any previously proposed Task and Motion Planning Methods. What motivated this decision?\n\n3. An alternative to using separate networks for the policy and the value function, it could be possible to use a two-headed network with one head for the policy and another one for the value. What was the reason for using two separate networks over this alternative? \n\n4. Were any other alternatives tested for the activation functions of the network?\n\n5. CSP was tested with three different curiosity and novelty metrics, none of which dominated over all the other ones. However, PPO was only tested with one of the measures and A2C had no curiosity measure added to it. Where there any preliminary results that justified this decision? In terms of computation and time, how difficult would it be to include this in the paper?\n\n6. The paper already hinted at this, but macro-actions could be framed within the option framework from Sutton, Precup, & Singh (1999). This would open up the opportunity to apply some of the already proposed methods for option discovery such as the option-critic architecture from Bacon, Harb, & Precup (2016) or the Laplacian framework for option discovery from Machado, Bellemare, & Bowling (2017), which is cited on the paper. Could the authors provide more comments about this line of future work? \n\n===== References ===== \nBacon, P., Harb, J., & Precup, D. (2016). The Option-Critic Architecture. Retrieved 17 October 2019, from https://arxiv.org/abs/1609.05140\n\nMarlos C. Machado, Marc G. Bellemare, and Michael H. Bowling. A laplacian framework for option discovery in reinforcement learning. CoRR, abs/1703.00956, 2017. URL http://arxiv.org/abs/1703.00956.\n\nSutton, R., Precup, D., & Singh, S. (1999). Between MDPs and semi-MDPs: A framework for temporal abstraction in reinforcement learning. Artificial Intelligence, 112(1-2), 181-211. doi: 10.1016/s0004-3702(99)00052-1", "extra_scores": {"experience_assessment": "I do not know much about this area.", "review_assessment:_checking_correctness_of_experiments": "I carefully checked the experiments.", "review_assessment:_thoroughness_in_paper_reading": "I read the paper thoroughly.", "review_assessment:_checking_correctness_of_derivations_and_theory": "I carefully checked the derivations and theory."}, "extra_text": {}}, {"review_id": "BkxtxbUJqH", "reviewer": "AnonReviewer1", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "This paper tackles the problem of enabling robots to learn long-horizon, sparse-reward tasks. The proposed approach, the Curious Sample Planner (CSP), builds on insights in task and motion planning (TAMP), which is a standard approach for tackling these kinds of tasks. TAMP constructs a plan in the space of macro-actions (e.g., move object 1 to location (x,y)), and uses a motion planner to execute each macro-action. However, TAMP typically requires being able to describe macro-action effects and preconditions with logical predicates, which can be impossible in real-world environments, due to complex dynamics and interactions. CSP overcomes this limitation by planning in the space of macro-actions in a way that is biased toward novelty.\n\nThe core approach of CSP is to train a (macro-)action selection network to generate macro-actions that are both feasible and novel. The curiosity module is used in two ways: (1) to give reward to the action selection network for producing novel macro-actions, and (2) to expand states that are considered novel. Three state-of-the-art ways of computing novelty are compared -- state estimation (SE), forward dynamics (FD), and random network distillation (RND).\n\nCSP is evaluated on a suite of simulated robotics tasks that require the robot to build simple machines from the objects in its environment, in order to achieve the specified objective. The experiments compare agents trained with CSP versus with deep RL (specifically, A2C, PPO, and PPO + RND). There is an ablation study, that compares against planning with uniform selection of macro-actions and uniform selection of states to expand.\n\nOverall, this paper makes a significant contribution to improving robot learning of long-horizon, sparse reward tasks. The paper is clearly written and well-motivated, and the evaluations are thorough. The major downside is that CSP inherently requires knowing the dynamics of the environment (i.e., having a simulator), which means it cannot be directly run applied to real-world robotic systems. But it is a step in the right direction, and clearly outperforms vanilla deep RL.\n\nI'm leaning toward accept, but I have a few concerns / questions about the paper. First, there are not enough details included for reproducibility (see list below). With regard to evaluation, I think another ablation should be run, where the action selection network is trained for only feasibility. This would be similar to CSP-No Curiosity, but with a reward of 0 for infeasible actions, and a small fixed positive reward for feasible ones. This would more clearly answer the question of how much it matters to include the curiosity module. Finally, I'm not convinced that this approach works well for transfer, and the evaluations seem inconclusive as well. I'm surprised that even for inter-task transfer, agents trained with action selection transfer and full transfer don't just learn to solve the task immediately. Am I missing something here about how the task is instantiated?\n\nReproducibility questions:\n- In Algorithm 1, what are the inputs to the novelty metric that is used to compute L_\\phi? Is it the batch of next states, S'?\n- What is the form of the output of the action-selection network? And what exactly is the space of macro-actions? For instance, the number of possible RemoveConstraint macro-actions depends on how many objects are connected in the environment. But the dimension of the action-selection network's output must be fixed.\n- What does the state vector input for FD and RND contain? Along these lines, why not also use image inputs for FD and RND, as is done for SE?\n- What are the learning hyperparamters used to train the networks? (e.g., learning rate)\n- How many perspectives (i.e., n_p) are used for the SE curiosity module?\n\nMinor comments / typos:\n- Avoid using the same variable with different meanings, e.g. using \\phi to indicate both the curiosity module and the parameters of the value network.\n- Page 3: \"flexible\", \"a flexible\"\n- Page 5: \"learnabe\" --> \"learnable\"\n- Page 8: \"in which\" --> \"in which the\"\n- Page 9: \"illustrate\" --> \"illustrated\"", "extra_scores": {"experience_assessment": "I have published one or two papers in this area.", "review_assessment:_thoroughness_in_paper_reading": "I read the paper thoroughly.", "review_assessment:_checking_correctness_of_experiments": "I carefully checked the experiments.", "review_assessment:_checking_correctness_of_derivations_and_theory": "I carefully checked the derivations and theory."}, "extra_text": {}}]

Author rebuttal:

Meta-review: The authors consider planning problems with sparse rewards.                                                                        
They propose an algorithm that performs planning based on an auxiliary reward                                                      
given by a curiosity score.                                                                                                        
They test they approach on a range of tasks in simulated robotics environments                                                     
and compare to model-free baselines.                                                                                               
                                                                                                                                   
The reviewers mainly criticize the lack of competitive baselines; it comes as now                                                  
surprise that the baselines presented in the paper do not perform well, as they                                                    
make use of strictly less information of the problem.                                                                              
The authors were very active in the rebuttal period, however eventually did not                                                    
fully manage to address the points raised by the reviewers.                                                                        
                                                                                                                                   
Although the paper proposes an interesting approach, I think this paper is below                                                   
acceptance threshold.                                                                                                              
The experimental results lack baselines,                                                                                           
Furthermore, critical details of the algorithm are missing / hard to find.
Decision: Reject