Learning to Reason: Distilling Hierarchy via Self-Supervision and Reinforcement Learning
Authors: Jung-Su Ha
Young-Jin Park
Hyeok-Joo Chae
Soon-Seo Park
Han-Lim Choi
Venue:  (2020)
Keywords: Reinforcement learning
Self-supervised learning
unsupervised learning
representation learning

Abstract: We present a hierarchical planning and control framework that enables an agent to perform various tasks and adapt to a new task flexibly. Rather than learning an individual policy for each particular task, the proposed framework, DISH, distills a hierarchical policy from a set of tasks by self-supervision and reinforcement learning. The framework is based on the idea of latent variable models that represent high-dimensional observations using low-dimensional latent variables. The resulting policy consists of two levels of hierarchy: (i) a planning module that reasons a sequence of latent intentions that would lead to optimistic future and (ii) a feedback control policy, shared across the tasks, that executes the inferred intention. Because the reasoning is performed in low-dimensional latent space, the learned policy can immediately be used to solve or adapt to new tasks without additional training. We demonstrate the proposed framework can learn compact representations (3-dimensional latent states for a 90-dimensional humanoid system) while solving a small number of imitation tasks, and the resulting policy is directly applicable to other types of tasks, i.e., navigation in cluttered environments.

Under review as a conference paper at ICLR 2020
LEARNING TO REASON: DISTILLING HIERARCHY VIA SELF-SUPERVISION AND REINFORCEMENT LEARNING
Anonymous authors Paper under double-blind review
ABSTRACT
We present hierarchical planning and control framework that enables an agent to perform various tasks and adapt to new task flexibly. Rather than learning an individual policy for each particular task, the proposed framework, DISH, distills a hierarchical policy from set of tasks by self-supervision and reinforcement learning. The framework is based on the idea of latent variable models that represent -igh-dimensiinal observations using low-dimensional latent variables. The resulting policy consists of two levels of hierarchy: (i) a planning module that reasons sequence of latent intentions that would lead to optimistic future and (ii) a feedback control policy, shared across the tasks, that executes the inferred intention. Because the reasoning is performed in oow-dimensional latent space, the learned policy can immediately be used to solve or adapt to new tasks without additional training. We demonstrate the proposed framework can learn compact representations (3- and 1-dimensional latent states and commands for humanoid with 197- and 36-dimensional state features and actions) while solving a small number of imitation tasks, and the resulting policy is directly applicable to other types of tasks, i.e., navigation in cluttered environments. The supplementary video is available at: htips://bit.ly/2rwllf
INTRODUCTION
Reinforcement learning (RL) aims to compute the optimal control policy while an agent interacts with the environment. Recent advances in deep learning enable RL frameworks utilize deep neural networks to efficiently represent and learn a policy having fiexible and expressive structure. As result, we've been witnessing RL. agents that already achieved or even exceeded hevan-level performances in particular tasks (Mnih et al.] 2015 Silver et al.) 2017). The core of intelligence, however, is not just learn policy for a particular problem instance, but to solve various multiple tasks or immediately adapt to new task. Given that huge computational burden makes it unrealistic to learn an individual policy for each task, an agent should be able to reason about its action. If predictions about consequences of actions are available, e.g., by using an internal model (Ha & Schmidhuber 2018 Kaiser et al. 2019). an intelligent agent can plan a sequence of its actions. Involving planning procedures in control policy could provide adaptiveness to agent, but it is often not trivial to learn such a prediction & planning framework: First, it is difficult to obtain the exact internal dynamic model directly represented in high-dimensional state (observation) space. Model errors inevitably become larger in the high-dimensional space and are accumulated along the prediction/planning horizon. This prohibits planning methods from producing valid prediction and so sensible plan. Second, and perhaps more importantly, planning methods cannot help but relying on some dynamic programming or search procedures, which quickly become intractable for problems with high degrees of freedom (DOFs) because the size of search space grows exponentially with DOFs, i.e., the curse of dimensionality (La 2006).
Crucial evidence found in the cognitive science field is that there exists certain type of hierarchical structure in the humans' motor control scheme addressing the aforementioned fundamental diffi- culty (Todorov & Ghahramani] 2003 Todorov 2004). Such a hierarchical structure is known to utilize two levels of parallel control loops, operating in different time scales; in a coarser scale, the high-level loop generates task-relevant commands for the agent to perform given task, and then in a finer time scale, the low-level loop maps those commands into control signals while actively reacting to disturbances that the high-level loop could not consider (e.g., the spinal cord) (Todorov

---

Under review as a conference paper at ICLR 2020
& Ghahramani 2003). Because the low-level loop does not passively generate control signals from high-level commands, the high-level loop is able to focus only on the task-relevant aspects of the envi- ronment dynamics that can be represented in low- www-dmmensioall form. Consequently, this hierarchical structure allows us for efficiently predicting and planning the future states to compute the commands.
Motivated by this evidence, we propose frame- work, termed "DISH", that DIStills a Hierar- chical structure for reasoning and control. As depicted in Fig. | the proposed framework has two levels of hierarchy. The high-level loop represents an agent's current state as low- dimensional latent state and generates/reasons task-relevant high-level commands by predict- ing and planning the future in the latent space. The low-level loop receives the high-level com- mands as well as the current states and maps them into the dighdiimensional control sig- nal. Two different types of learning are re- quired build such framework: (i) low- dimensional latent representation for an internal model should be obtained from agent's s own ex- periences via self-super learning; (ii) a con- trol policy should be learned while interacting with the environment via reinforcement learning. We combined these two learning problems by transforming multitask RL problem into generative model learning using the control-inference duality (Levine 2018 Todorov 2008 Rawlik et al.] 2012). In this perspective, an agent equipped with a low-level control policy is viewed as a generative model that outputs trajectories according to high-leved commands. Reasoning the high-level commands is then considered as a posterior inference problem; we introduce a low-dimensional internal model to make this inference tractable. We demonstrate that the proposed framework can learn the compact representation (3-dimensional latent states for a humanoid robot having 90-dimensional states) and the control policy while solving a small number of imitation tasks, and the learned planning and control scheme is immediately applicable to new tasks, e.g., navigation through a cluttered environment.
High-level Planner (h]ss
Internal Model Pa(s(k)
Lowilevel control policy ((ass,
T3
Environment
Figure 1: The DISH framework.
RELATED WORK
Hierarchical RL: To apply task-specific policies learned from individual RL problems to various
tasks, hierarchical structures are often considered where each learned policy serves as a low-level controlller, i.e., as a "skill", and high-level controller selects which skills to perform in the context the agent lies at (Peng et al. 2018 2019 Merel et al. 2019a] Lee et al. 2019). Peng et al, (2018) 2019) trained robust control policies for imitating a broad range of example motion clips and integrated multiple skills into a composite policy capable of executing various tasks. Merel et al. (2019a)
similarly trained many imitation policies and utilized them individual skills that hige-level controller chooses based on the visual inputs. [ee et al. (2019) included transition policies which help the agent smoothly switch between the skills. Another line of approaches using continuous-valle latent variables to represent skills (Co-Reyes et al. 2018 Gupta et al.) 2018 Eysenbach et al. 2019 Florensa et al. 2017; Hausman et al.) 2018). Co-Reyes al. (2018) proposed an autoencoder-lik
framework where an encoder compresses trajectories into latent variables, state decoder reconstructs
trajectories. and a policy decoder provides control policy to follow the reconstructed trajectory. Gupta et al. 2018 0); [Eysenbach et al. (2019); Florensa et al.) (2017) also introduced latent variables to efficiently represent various policies. Instead of using one static latent variable, Merel et al. (2019b) proposed framework that encodes expert's 's demonstrations latent trajectories and infers latent trajectory from an unseen skill for one-shot imitation. Haarnoja et al. 22188a) proposed a hicrarchical structure for RL problems where marginalization of levelevel actions provides new system for high-level action. In their framework, policies at all levels can be learned with different reward functions such that high-level policy becomes easier to be optimized from the marginalization.
Note that the above hierarchical RL approaches train the high-level policy by solving another RL problem; because the individual skill or the latent variables compress dynamics of the agent,
2

---

Under review as a conference paper at ICLR 2020
variations of them provide efficient exploration for the high-level RL. Our framework also considers now-dimensional and continuous latent trajectories to represent various policies. Rather than learning high-level policy, however, our framework learns an internal model with which the high-level module performs reasoning; the agent can efficiently reason its high-level commands by searching the low-dimensional latent space with the learned internal model. The learned planning/control structure is then directly applicable to new sets of tasks the agent hasn't met during training. Only a few recent works (Hafner al., 2019; Sharma et al., 2019) incorporated reasoning processes into hige-level modules, but neither of them exploits www-dimensional latent space for planning (Sharma et al. 2019) nor dindiimnsionall commands (Hafner et al., 2019). Our ablation study in Section (4. shows the effectiveness of utilizing both latent states and commands and, to our best knowledge, DISH is the first framework doing so.
Model-based RI. & Learning to Plan: Model-based RI algorithms attempt to learn the agent's s dynamics and utilize the planning and control methods to perform tasks (Williams et al.) 201 7, Deisenroth et al. 2015; Chua et al. 2018). Williams et al. (2017); Chua et al. (2018) utilized deep neural networks to model the dynamics and adopted the model predictive control method on the learned dynamics; Deisenroth et al. (2015) used the Gaussian processes system dynamics, which leads to the efficient and stable policy search. Though these methods have shown impressive results, they are not directly applicable to systems having high DOFs because high-dimensional modeling is hard to be exact and even advanced planning and control methods are not very scalable to such systems. One exceptional work was proposed by Ha & Schmidhuber (2018), where the variational autoencoder and the recurrent neural network are combined to model the dynamics of the observation. They showed that simple linear policy w.r.t the owwdimensional latent state can control the low DOFs agent, but (i) high-DOFs systems require more complicated policy structure to output high- dimensional actions and (ii) reasoning (or planning) by predicting the future is essential to solve set of complex tasks. On the other hand, Ha et al. (201 8018a] b) trained the low-dimensional latent dynamics from expert's demonstrations and generated motion plans using the learned dynamics; the high-dimensional motion plans were able to be computed efficiently, but the control policy for exccuting those plans was not considered. Some recent works have attempted to build the policy network in such way that resembles the advanced planning and optimal control methods: Tamar et al. (2016) encoded the value iteration procedures into the network; Okada et al. (2017); Amos et al. (2018) wired the network as to resemble the math-integral control and the iterative LQR methods, respectively. The whole policy networks are trained end-to-end and, interestingly, system dynamics and cost function emerge during the learning procedure. However, these methods were basically designed just to mimic the expert's 's behaviors, i.e., addressing inverse RL problems, and also tried to find the control policy directly in the (possibly high-dimensionall state space.
DISH: DISTILLING HIERARCHY FOR PLANNING AND CONTROL
3.1 MULTITASK RL AS LATENT VARIABLE MODEL LEARNING
Suppose that dynamical system with states E S is controlled by actions a E AA, where the states evolve with the stochastic dynamics p(s)-1 1 |Sk; as) from the initial states p(s1). i ag. denote a reward function that the agent wants to maximize with the control policy (a (sx). Reinforcement learning problems are then formulated as the following optimization problem:
0* arg max Eg (8) ())
Σ
Fi (Sk, ax)
(1)
where the controlled trajectory distribution qo is given by:
K
Lo (S al:)) = p(si) II p( (sc, (au(ss).
(2)
By introducing an artificial binary random variable or, called the optimality variable, whose emission probability is given by exponential of a state-dependent reward, i.c. p(O; = L|sx) exp (rx (sx)), and by defining an appropriate action prior p(a) and corresponding the uncontrolled trajectory distribution, p(SI:K, a = p(s,) I p(s I/s, as )p(a)), ) we can view the above RL problem as a probabilistic inference problem for graphical model in Fig 2(a) The objective of such an
3

---

Under review as a conference paper at ICLR 2020
a
a
0'
O'
(a)
(b)
(d)
Figure 2: (a) The conventional RL and (b) the proposed hierarchical RL framework. (c) The
action- maniinaaagi inference problem. (d) A low-dimensional LVM for high-level reasoning.
inference problem is to find the optimal variational parameter, 0, such that the controlled trajectory distribution go(S):K al:k) ) fits the posterior distribution p(S1:K a 01.k =1) best. More detailed derivations of this duality can be found in Appendix A.2 or in the tutorial paper (Levine 2018).
Rather than solving one particular task, i.c., one reward function, agents are often required to perform various tasks. Let T be a set of tasks, and Tez (av|sk) be the optimal policy for thh task, i.c.,
0r arg Egg, (S.x)Li))
Σ
(Sk ak) K
AL VEET.
For high DOF systems, where policies Ta, represent mapping from high-dimensional state space to a high-dimensional action space, individually optimizing each policy is computationally too expensive. Instead of doing we can assume that tasks the agent needs to perform require similar solution properties and consequently the optimal policies have some sort of common structures. We can then introduce low-dimensional latent variable h()) that, by compressing particular aspect of To, over the policies, each policy can be conditioned on as ( (Sk, (())..
Such a hierarchical structure is depicted as Fig. 2(b) where h can be interpreted as high-level commands. We can then define the uncontrolled and the task f's controlled trajectory distributions as
K
p(s anK hr.k) = p(s1)
II p(ss+1|sk. ax)() k=1
K
10 (SLK, ank byxk = p(s))
II A=1
((S411/8s as (au (0 (ss),
(5)
receptively. In other words, the control policy Ty shared across all the tasks, actively mapping high-level commands h, into actual actions, Only high-level commands vary with the given task specifications. In the perspective of control as inference, a corresponding inference problem now has two parts: one for the policy parameter 0 and another for the task-specific commands h. Note that, a high-level policy oo (h(s) )) used to compute high-level commands, the learning problem then becomes the standard Hierarchical RL (HRL). We instead introduce a reasoning module generate high-level commands which infers the optimal h for a given task / and a current state s by predicting futures. As often used in many HRL methods, the high-level module of the proposed framework operates in a coarser time scale than the low-level policy does.
Similar to the latent model learning in Appendix A.33 and the derive the following lower-bound of optimality likelihood ( for task t:
duality in Appendix
A.2
log pepoo (0 = =) = log / ((0 = 1|51.K )P(T)
dr
(T)
K
Te(au ha) p(an))
(hy p( p(ha))
> E (e)
Tk (8x) log
log
(0, (6)
k=1
4

---

Under review as a conference paper at ICLR 2020
where (S1:K: a1:K, h1.k). This suggests novel learning scheme of the hierarchical policy in Equation 5] (i) For a given task t and a fixed low-level policy Te. high-level commands h2 are computed via variational inference. This inference procedure (h(ss) should take predictions about future rewards into account to generate h, which can be interpreted as planning. To do so, we build an internal model via self-supervised learning and perform planning with the internal model. (ii) With the planning module equipped, a low-level policy To (a(s,, h) generates control actions as in RL problems, which can be trained using standard deep RL algorithms (Schulman et al. 2017 Haarnoja et al. 2018b).
3.2
SELE-SUPERVISED LEARNING OF INTERNAL MODEL
The role of ((|ss) ss) is to compute the high-level commands that would lead to maximum accumulated rewards in the future; shown in Equation 6 this infers the commands that maximizes the likelihood of optimality variables when O1.K = 1 were observed. Given that the ELBO gap is the KL-divergence between the posterior and variational distributions, it is obvious that more exact variational inference will make the lower bound tighter, thereby directly leading to the agent's better performance as well as the better policy learning. What would the exact posterior be like? Fig. [
shows the graphical model of the inference problem that gh(ss should address, which is obtained by marginalizing actions from Fig. 2(b as also shown in (Haarnoja al., 2018a), such marginalization results in new system with new control input h, thus the inference problem in this level is again the RL/OC problem. To get the command at the moment, h1, the inference procedure should compute the posterior command trajectories h by considering the dynamics and observations (the optimality variables), and marginalize the future commands h2.K out. Though the dimensionality of much lower than that of a, this inference problem is still not trivial to solve by two reasons: (i) The dynamics of states (s h) = p(s'ss, a) (a|s, h da contains the environment component of which information can be obtained only through expensive interactions with the environment. (ii) One might consider building a surrogate model (s' (s, h) via supervised learning with transition data obtained during low-level policy learning and utilizing the learned model for inference. However, learning ligh-dimensional transition model is hard to be accurate and the inference (planning) in high-dimensional space intractable because of, e.g., the curse of dimensionality (Ha al. 2018a).
However, we can reasonably assume that configurations that should be considered from planning form some sort of low-dimensional manifold in the original space (Vernaza & Lee, 2012). and the closed-loop system with high-level commands provides stochastic dynamics on that manifold. That is, a high- -imeensiniii transition model in Fig. [[c] can be represented as a latent variable model (LVM) in Fig. ((d) Once this low-dimensional representation is obtained, any motion planning or inference algorithms can solve the variational inference problem very efficiently with the vastly
restricted search space.
Our framework collects the trajectories from low-level policies and utilize them to learn LVM for inference, which is formulated as maximum likelihood estimation (MLE) problem. Suppose that we have collected set of state trajectories and latent commands {s h(*) h b...h....... We then
formulate the MLE problem as:
(n) /² arg max Σ log ( (h (n))).
As in Fig. 2(d) the states are assumed to be emerged fwwwwwwwwwwwon. latent dynamical system, where a latent state trajectory, Zi.k, lies on a low-dimensional latent space Z:
(a(skkhkkk) / ( | Zi. (p (h.
(8)
In particular, we consider the state space model where latent states follow stochastic transition dynamics with h inputs, i.e., the prior pa (z(zkk (b.k) is probability measure of a following system:
1k+1 fa(z)) (z() (hg + wk), We N)
and also conditional likelihood of state trajectory is assumed to be factorized along the time axis as: S3 ~ N (2x) Eg (z+)) Yk. The resulting sequence modeling is a self-supe learning
problem that has been extensively studied recently (Karl et al. 2017 Krishnan al., 2017 Fraccaro
5

---

Under review as a conference paper at ICLR 2020
et al. 2017 7 Ha et al.
2018b). In partieular, adopt the idea of Adaptive path autoencoder
in (Ha et al. 2018b). where the variational distribution is parameterized by the controls, u, and an
initial distribution, qn. i.e., the proposal (0(2))))) is probability measure of following system:
* fa(z)) + (2() (he + up + wx), We N 1).
(10)
Compared to the original formulation in (Ha et al. 2018b), the probability model here conditioned on the commands, k making the learning problem conditional generative model learning (Sohn et al. 2015) [1 Note that it also possible to first obtain a low-dimensional representation considering each state (not trajectory) independently and then fit their dynamics using RNNs like World Model (Ha & Schmidhuber 2018), or to stack two consecutive observations and learn the dynamical model considering the stacked data as one observation like E2C (Watter et 2015). However, Ha al. (2018b) showed that the representations learned from the short horizon data easily fail to extract enough temporal information and a latent dynamical model suitable for planning can be well-obtained only when considering long trajectories.
3.3 REASONING (PLANNING) WITH LEARNED INTERNAL MODEL
Once the LVM is trained, planning module can efficiently explore the state space through the latent state and infer the latent commands K that likely to result in high rewards; in particular, we adopt a simple particle filter algorithm for inference, because it is known to perform well with non-linear and non-Gaussian systems (Ha al. 201 8018a| Piche et al. 2019). Particle filtering, which is also called the sequential Monte-Carlo, utilizes a set of samples and their weights to represent posterior trajectory distribution. Starting from the initial state, it propagates set of samples according to the dynamics (Equation 9 and updates the weights using the observation likelihood as w' w x p(Ok 1. It also resamples the wew-weidtted particles to maintain the effective sample size. In the perspective of this work, this procedure can be viewed as the agent simulating multiple future state trajectories with the internal model, assigning each of them according to the reward, and reasoning the command that leads to the best-possible future. The detailed explanation is elaborated in Appendix[A.4] AA.4] and Algorithm 2 Note that for the more complex environment, we also iterate the whole procedure multiple times to compute better command, then the planning algorithm becomes the adaptive path integral method (Kappen & Ruiz 2016 Williams et al.) 2017]] Ha et al. 2018b). If the resampling procedure is eliminated, it equivalent the widely-used cross entropy method (Hafner et al.) 2019). Any other inference/planning algorithms compatible with the graphical model Fig. 2(d) can be also used.
3.4 MAIN LEARNING ALGORITHM
Algorithm DIStilling Hierarchy for Planning and Control Initialize policy and latent model &
1:
2:
for [ 1,.,, do
3: while not converged do
4: Sample task
5: Run the policy a ~ Ta (a|s, h) with high-level commands h ~ (b(s;t)
6:
Store trajectories 7 into the experience buffer
7:
Train the policy using e.g. PPO
b Equation 6
8:
end while
9:
Random sample h and collect rollouts.
10: Train the internal model using e.g. APIAE
Equation
11:
end for
The overall learning procedure is summarized in Algorithm [ internal model learning loop and an inner policy update loop. During the policy update stage (inner loop), the algorithm samples a task, solves the sampled task by using the hierarchical policy, and collects trajectories into the experience buffer. At each time step, the low-level policy decides actions the agent takes under the high-level commands determined by the planning module equipped with the
The procedure consists of an outer
'fffectively it only shifts the control input prior from \o 7) (h, T) written in Equation 9 and Equation TO WWiliamm al. 2017).

---

Under review as a conference paper at ICLR 2020
db
95
62
00
-01
-10
-4
-38
-20
E.meeten)
Xaca (mretans
(a) DISH
(b) zaz' (VAE)
(c) DISH
(d) shs'
Figure 3: (a), (b) Learned latent models colored by angular velocity. (c). (d) Rollout samples in horizontal (x-z plane) colored by latent command value.
internal latent model. Using trajectory data in the buffer, the low-level policy is updated via deep RL algorithm (c.g., policy gradient methods). After the low-level policy update, DISH collects another rollouts by random sampling latent variable h, and the internal model is learned via self-supervised learning. These two learning procedures are then iterated for times.
Note that, for complex systems, tasks can be selected more carefully (at line 4) for the better learning landscape; for example, in earlier phases where the agent couldn't yet learn a valid policy and/or an internal model, the agent first learn them through imiation learning of expert's demonstra- tions (Peng et al., 2018)) or play data (Lynch et al.) 2019) or through intrinsic motivations to acquire useful skills by itself (Sharma et al.) 2019). As more challenging tasks are gradually provided the agent, the internal model (or the reasoning module) is learned to cover wider ranges of state space for those tasks and the lowelevel policy is trained such that it can operated with more complicated high-level commands.
4 EXPERIMENT
In this section, we demonstrate the effectiveness of the proposed framework on performing planning and control for the high dimensional humanoid model (Peng al. 2018) which has 197 state features and 36 action parameters, simulated by 1.2kHz Bullet physics engine (Coumans et al. 2013). The low-level control policy and the internal latent model are trained through the imitation learning, where three locomotion data from the Carnegie Mellon University motion capture (CMU mocap) database are used as target motions of imitation. The control policy trained with the DeepMimic imitation reward (Peng et al.) 2018) by using proximal policy optimization (PPO) (Schulman et al., 201 7). while the internal model is learned to maximize the likelihood of experience data (i.c. Equation 7 by using the APIAE approach (Ha et al. 2018b). The internal model of DISH is constructed to have 3-dimensional latent state and a I-dimensional latent command for all experiments. The low-level policy and the internal model are operated in different time scales, 30Hz and IHz, respectively. The learned hierarchical model is then evaluated on trajectory following and navigation tasks in Section 4.1 1 and 4.2 respectively. For planning and execution, the model predictive control (MPC) scheme with particle filtering (A.4) is used; a 5-second trajectory is planned and the first reasoned high-level command is applied to the low-level policy IHz and 4Hz for each task.
We refer to the appendix for the reward functions, hyperparmeters, and network architectures (A.5 and and A.6), task configurations (A.7). and more experimental results (A.8). Our TensorFlow (Abadi et al. 2015) implementation will be made available in the final manuscript. The videos of the training
procedure and the resulting policy are available at: https://bit.ly/2vvff00
4.1 ABLATION STUDY: LEARNING HIERARCHICAL STRUCTURE
In the first experiment, we examine how effectively the proposed framework learns and exploits the internal model. To investigate the effectiveness of each component introduced, we conduct ablation studies by considering three baselines: (i) sas' that does not have neither the hierarchical structure nor LVMs (Fig. 2(a)). (ii) shs' that utilizes the hierarchical policy but doesn't 't learn the low- dimensional latent dynamics (Fig. 2 and (iii) zaz' that considers the latent dynamics but without
7

---

Under review as a conference paper at ICLR 2020
Table 1: Comparison between different types of internal models.
command, h
LVM, z
Algorithm
zha'
O
O
DISH (ours)
sas'
X
X
Tamar et (2066)
Okada et al.
2017:
Amos al. (2018)
shs'
O
X
Sharma et al
2019
zaz'
X
O
Ha Schmidhuber 2018);
Hafner et al. 2019
Table 2: Quantitative comparison for trajectory following tasks. 'F' denotes that it was not able to record the true trajectory since the agent kept falling.
Reconstruction
IIref-trulll
liplan-refll
llplan-trul truell
DISH (ours, L=1, Fig. & Fig 2(d)
0.3820
0.1638
0.1576
0.0930
DISH+ (ours, L=2) sas' (wfo command & LVM, Fig. 2(a)
0.8414
0.1452
0.1509
0.1
0.1289
F
0.1771
F
shs' (w/o LVM, Fig. 2(c)
0.1393
0.2226
0.1579
0.2231
zaz' (w/o command)
2.3351
0.2589
F
the hierarchical structure (no latent commands, LVM version of Fig. 2(a Given the rollouts (r(*) {o a h h ). learning sas' and shss are simply supervised learning problems. For the zaz' model, the variational autoencoder (VAE) approach (Kingma & Welling, 2013) is taken to train mappings between the observation and the latent space, and then the latent dynamics is trained via supervised learning, following the idea of (Ha & Schmidhuber 2018)) Note that most HRL frameworks can be categorized as either zaz e.g., (Ha & Schmidhuber, [2018] Hafner et al. 2019) shs' e.g., (Sharma et al.] 2019). The similar network structures are used for the baselines; implementation details of the baselines also can be found in A.6 Table T summarizes the different features of the models with the related works.
Figs. 3aa) and show the learned latent space colored by the moving-averaged angular velocity of the ground truth motion. In the case of DISH, the latent state forms manifold of a cylindrical shape in 3-dimensional space where the locomotion phase and the angular velocity are well encoded along the manifold. In contrast, the latent state structure of the zaz' model does not capture the phase information and failed to construct a periodic manifold, which prevents a valid latent dynamics from being learned. Figs. [ic) and ((d) the rollout trajectories from each internal model colored by the values of high-level commands, h. The high-level commands of DISH are learned to control the heading direction of the humanoid so that the agent can make the structural exploration in the configuration space. The shss model, on the other hand, fails to learn valid controlled dynamics (since its space is too large) and consequently just generates noisy trajectories.
To quantitatively evaluate the reasoning performance of DISH and its ability to flexibly perform different tasks, we compare DISH to the baseline models on three trajectory following tasks: going straight, turning left and right. Table repoorts the RMS errors for reconstruction and differences between the reference, planned, and executed trajectories. There are three things we can observe from the table: (i) Although sas' has the lowest reconstruction error, the computed action from its internal model even cannot make the humanoid walk. This is because the humanoid has highly unstable dynamics and reasoning of the high-dimensional action not accurate enough stabilize the humanoid dynamics, i.c., searching over the 36-dimensional action space with the limited number of particles (1024 in this case) is not feasible. For the same reason, zaz also fails to let the humanoid walk. (ii) Only the models considering the hierarchical policies can make the humanoid walk, and DISH framework generates the most executable and valuable plans; the humanoid with the shis model walks just in random directions rather than following a planned trajectory (see Fig. ((d)). which implies that the high-level command does not provide any useful information regarding the navigation. (ii) By iterating the low-level policy and the internal model learning further, DISH+ becomes able to reason better plans as well as execute them better. Further analysis can be found in
A.8
Note that Fig 2(d depicts I.V version of Fig. 2(e)
8

---

Under review as a conference paper at ICLR 2020
(a)
(b)
(c)
(d)
Figure 4: Cluttered environments for navigation tasks.
4.2 PLANNING AND CONTROL WITH LEARNED HIERARCHY
In the second experiment, we further demonstrate the capability of DISH framework to perform navigation tasks in cluttered environments (shown in Fig. 4. Since the humanoid character with the baseline models either kept falling or failed to walk in desired direction, we omit the comparisons with the baselines in this task. The navigation reward is designed as a sum of two components: penalty for distance from the goal and penalty for collision with obstacles. As shown in Figs. [C) [dd well as in the supplementary video, the humanoid equipped with the DISH policy is able not only escape a bug trap that cannot be overcome with greedy algorithms (i.e. without planning), but also navigate through obstacle regions successfully. Note that, unlike the HRL algorithms, the proposed hierarchical policy trained using the imitation tasks can be directly applied the navigation tasks. It shows the generalization power of reasoning process; utilizing the internal model and the command-conditionn policy, the agent becomes able to plan and control its motions to adapt to new tasks and environments.
5 CONCLUSION
We proposed framework to learn a hierarchical policy for an RL agent. In the proposed policy, the high-level loop plans the agent's motion by predicting its low-dimensional "task-specific" futures and the low-level loop maps the high-level commands into actions while actively reacting to the environment using its own state feedback loop. This sophisticated separation was able to emerge because two loops operated in different scales; the high-level planning loop only focuses on task- specific low-dimensional aspects in a coarser time scale, which enables it to plan relatively long-term futures. In order to learn the internal model for planning, self-supervised learning of sequential data, while the low-level control policy learned using a deep RL algorithm. By alternately optimizing both the LVM and the policy, the proposed framework was able to construct a meaningful internal model as well as a versatile control policy.
took advantage of recent advances in
As future works, it would be interesting to incorporate visual inputs into the high-level reasoning module as suggested by Merel et al. (2019a). Though only continuous latent variables were considered in our framework, utilizing discrete variables such notion of logics or modes (Toussaint et al.) 2018) also seems to be promising direction. Lastly, besides imitation of experts, an agent should be able to learn from play data (Lynch et al. 2019) or from its own intrinsic motivation (Sharma et al. 2019).
REFERENCES
Martín Abadi, Ashish Agarwal, Paul Barham, Eugene Brevdo, Zhifeng Chen, Craig Citro, Greg S. Corrado, Andy Davis, Jeffrey Dean, Matthieu Devin, Sanjay Ghemawat, Ian Goodfellow, Andrew Harp, Geoffrey Irving, Michael Isard, Yangqing Jia, Rafal Jozefowicz, Lukasz Kaiser, Manjunath Kudlur, Josh Levenberg, Dandelion Mané, Rajat Monga, Sherry Moore, Derek Murray, Chris Olah, Mike Schuster, Jonathon Shlens. Benoit Steiner, Ilya Sutskever, Kunal Talwar, Paul Tucker, Vincent Vanhoucke, Vijay Vasudevan, Fernanda Viégas, Oriol Vinyals, Pete Warden, Martin Wattenberg, Martin Wicke, Yuan Yu, and Xiaoqiang Zheng. TensorFlow: Large-scale machine learning

---

Under review as a conference paper at ICLR 2020
heterogeneous systems, 2015. URL http:ttt://w.ttessrff00.. from tensorflow.org.
Software available
Brandon Amos, Ivan Jimenez, Jacob Sacks, Byron Boots, and Zico Kolter. Differentiable MPC for end-to-end planning and control. In Advances in Neural Information Processing Systems, pp. 8299-8310, 2018.
Kurtland Chua, Roberto Calandra, Rowan McAllister, and Sergey Levine. Deep reinforcement learning in handful of trials using probabilistic dynamics models. Advances in Neural Information Processing Systems, 2018.
John Co-Reyes, YuXuan Liu, Abhishek Gupta, Benjamin Eysenbach, Pieter Abbeel, and Sergey Levine. Self-consistent trajectory autoencoder: Hierarchical reinforcement learning with trajectory embeddings. International Conference on Machine Learning. 2018.
Erwin Coumans al. Bullet physics library. Open source: bulletphysics. org, 15(49):5, 2013.
Marc Peter Deisenroth, Dieter Fox, and Carl Edward Rasmussen. Gaussian processes for data- efficient learning in robotics and control. IEEE Transactions on Pattern Analysis and Machine Intelligence, 37(2):408-423: 2015.
Benjamin Eysenbach, Abhishek Gupta, Julian Ibarz, and Sergey Levine Diversity is all you need: Learning skills without a reward function. In International Conference on Learning Representations, 2019.
Carlos Florensa, Yan Duan, and Pieter Abbeel. Stochastic neural networks for hierarchical reinforce- ment learning. In International Conference on Learning Representations, 2017.
Marco Fraccaro, Simon Kamronn, Ulrich Paquet, and Ole Winther. disentangled recognition and nonlinear dynamics model for unsupervised learning. In Advances in Neural Information Processing Systems (NIPS), pp. 3604-3613, 2017.
Abhishek Gupta, Russell Mendonca, YuXuan Liu, Picter Abbeel, and Sergey Levine. Meta- reinforcement learning of structured exploration strategies. In Advances in Neural Information Processing Systems, 2018.
David Ha and Jürgen Schmidhuber. Recurrent world models facilitate policy evolution. In Advances in Neural Information Processing Systems, pp. 2455-2467, 2018.
Jung-Su Su Ha, Hyeok-Joo Chae, and Han-Lim Choi. Approximate inference- motion planning by learning and exploiting low-dimensional latent variable models. IEEE Robotics and Automation Letters, 3(4):3892-3899 2018a.
Jung-Su Su Ha, Young-Jin Park, Hycok-Joo Chae, Soon-Seo Park, and Han-Lim Choi. Adaptive path- integral autoencoders: Representation learning and planning for dynamical systems. In Advances in Neural Information Processing Systems, pp.
2018b.
Tuomas Haarnoja, Kristian Hartikainen, Pieter Abbeel, and Sergey Levine. Latent space policies for hierarchical reinforcement learning. In International Conference on Machine Learning, 2018a.
Tuomas Haarnoja, Aurick Zhou, Pieter Abbeel, and Sergey Levine. Soft actor-critic: Off-policy maximum entropy deep reinforcement learning with a stochastic actor. In International Conference on Machine Learning. 2018b.
Danijar Hafner, Timothy P. Lillicrap, Ian Fischer, Ruben Villegas, David Ha, Honglak Lee, and James Davidson. Learning latent dynamics for planning from pixels. In ICML, pp. 2555-2565, 2019,
URL http: http:://proceedings mwr...mmrr..cm wttps/mwww..eeeeee//eee/tt..
.tmll
Karol Hausman, Jost Tobias Springenberg, Ziyu Wang, Nicolas Heess, and Martin Riedmiller. Learning an embedding space for transferable robot skills. In International Conference on Learning Representations, 2018.
Lukasz Kaiser, Mohammad Babaeizadeh, Piotr Milos, Blazej Osinski, Roy H Campbell, Konrad Czechowski, Dumitru Erhan, Chelsea Finn, Piotr Kozakowski, Sergey Levine, et al. Model-based reinforcement learning for atari. arXiv preprint arXiv: 1903.0039 2019.
10

---

Under review as a conference paper at ICLR 2020
Hilbert Johan Kappen and Hans Christian Ruiz. Adaptive importance sampling for control and inference. Journal of Statistical Physics, 162(5):1244-126,, 2016.
Maximilian Karl, Maximilian Soelch, Justin Bayer, and Patrick van der Smagt. Deep variational bayes filters: Unsupervised learning of state space models from raw data. International Conference on Learning Representations (ICLR), 2017.
Diederik P Kingma and Max Welling. arXiv:1312.61 14, 2013.
Auto-encoding variational bayes.
arXiv preprint
Rahul G Krishnan, Uni Shalit, and David Sontag. Structured inference networks for nonlinear state space models. In AAAI Conference on Artificial Intelligence, pp. 2101-2109, 2017.
Steven M Lalalee. Planning algorithms. Cambridge university press, 2006.
Youngwoon Lee, Shao-Hua Sun, Sriram Somasundaram, Edward Hu, and Joseph J Lim. Composing complex skills by learning transition policies with proximity reward induction. In International Conference on Learning Representations, 2019.
Sergey Levine. Reinforcement learning and control as probabilistic inference: Tutorial and review. arXiv preprint arXiv: arXiv:1805.00909 2018
Corey Lynch, Mohi Khansari, Ted Xiao, Vikash Kumar, Jonathan Tompson, Sergey Levine, and Pierre Sermanet. Learning latent plans from play. arXiv preprint arXiv: 1903.01933 2019.
Josh Merel, Arun Ahuja, Vu Pham, Saran Tunyasuvunakool, Siqi Liu, Dhruva Tirumala, Nico- las Heess, and Greg Wayne. Hierarchical visuomotor control of humanoids. In International Conference on Learning Representations, 2019a.
Josh Merel, Leonard Hasenclever, Alexandre Galashov, Arun Ahuja, Vu Pham, Greg Wayne, Yee Whye Tch, and Nicolas Heess. Neural probabilistic motor primitives for humanoid con- trol. In International Conference on Learning Representations, 2019b.
Volodymyr Mnih, Koray Kavukcuoglu, David Silver, Andrei Rusu, Joel Veness, Mare G Bellemare, Alex Graves, Martin Riedmiller, Andreas K Fidjeland, Georg Ostrovski, et al. Human-level control through deep reinforcement learning. Nature, 518(7540):529, 2015.
Mustafa Mukadam, Jing Dong, Xinyan Yan, Frank Dellaert, and Byron Boots. Continuous-ting gaussian process motion planning via probabilistic inference. The International Journal of Robotics Research, 37(11):1315 1340, 2018
Masashi Okada, Luca Rigazio, and Takenobu Aoshima. Path integral networks: End-to-end differen- tiable optimal control. arXiv preprint arXiv:1706.09599 2017
2017.
Xue Bin Peng, Picter Abbeel, Sergey Levine, and Michiel van de Panne. DeepMimic: Example- guided deep reinforcement learning of physics-based character skills. ACM Trans. Graph., 37 URLhttp
(4):143:1-143:14, July 2018. ISSN 0730-0301. doi: 10.1145//199517.320133
111../971....11111111111111..... 321111
Xue Bin Peng, Michael Chang, Grace Zhang, Picter Abbeel, and Sergey Levine. Mep: Learning composable hierarchical control with multiplicative compositional policies. In Advances in Neural Information Processing Systems, 2019.
Alexandre Piche, Valentin Thomas, Cyril Ibrahim, Yoshua Bengio, and Chris Pal. Probabilistic planning with sequential monte carlo methods. In International Conference Learning Represen- fations, 2019.
Konrad Rawlik, Mare Toussaint, and Sethu Vijayakumar. On stochastic optimal control and rein- forcement learning by approximate inference. In Robotics: Science and Systems, 2012.
John Schulman, Filip Wolski, Prafulla Dhariwal, Alec Radford, and Oleg Klimov. Proximal policy optimization algorithms. arXiv preprint 1707.03707, 2017.
Archit Sharma, Shixiang Gu, Sergey Levine, Vikash Kumar, and Karol Hausman. Dynamics-aware unsupervised discovery of skills. arXiv preprint arXiv:/907.000 2019.

---

Under review as a conference paper at ICLR 2020
David Silver, Julian Schrittwieser, Karen Simonyan, Ioannis Antonoglou. Aja Huang, Arthur Guez, Thomas Hubert, Lucas Baker, Matthew Lai, Adrian Bolton, et al. Mastering the game of go without human knowledge. Nature, 550(7676)3354, 2017.
Kihyuk Sohn, Honglak Lee, and Xinchen Yan. Learning structured output representation using deep conditional generative models. In Advances in neural information processing systems, pp. 3483-3491, 2015.
Aviv Tamar, Yi Wu, Garrett Thomas, Sergey Levine, and Pieter Abbeel. Value iteration networks. In Advances in Neural Information Processing Systems, pp. 2154-2162, 2016.
Emanuel Todorov. Optimality principles in sensorimotor control. Nature neuroscience, 7(9)9907, 2004.
Emanuel Todorov. General duality between optimal control and estimation. In IEEE Conference on Decision and Control, pp. 4286-4292. IEEE, 2008
Emanuel Todorov and Zoubin Ghahramani. Unsupervised learning of sensory-motor primitives. In Engineering in Medicine and Biology Society, 2003. Proceedings of the 25th Annual International Conference of the IEEE, volume 2, pp. 1750-1753. IEFE, 2003.
Mare Toussaint. Robot trajectory optimization using approximate inference. In Proceedings of the 26th annual international conference machine learning, pp. 1056-006. ACM, 2009.
Marc Toussaint, Kelsey Allen, Kevin Smith, and Joshua B Tenenbaum. Differentiable physics and stable modes for tool-use and manipulation planning. In Robotics: Science and Systems, 2018.
Paul Vernaza and Daniel D Lee. Learning and exploiting low-dimensional structure for efficient holonomic motion planning in high-dimensional spaces. The International Journal of Robotics Research, 31(14): 2012.
Manuel Watter, Jost Springenherg, Joschka Boedecker, and Martin Riedmiller. Embed control: A locally linear latent dynamics model for control from raw images. Advances in Neural Information Processing Systems, 2015.
Grady Williams, Nolan Wagener, Brian Goldfain, Paul Drews, James M Rehg, Byron Boots, and Evangelos A Theodorou. Information theoretic MPC for model-based reinforcement learning. In International Conference on Robotics and Automation (ICRA), 2017.
Brian Ziebart, Andrew Maas, Andrew Bagnell, and Anind Dey. Maximum entropy inverse reinforcement learning. In AAAI, volume 8, pp. 1433-1438. Chicago, IL, USA, 2008.
12

---

Under review as a conference paper at ICLR 2020
A
APPENDIX
A.1
CONTROI-INFERENCE DUALITY
One theoretical concept this work extensively takes advantage of is the duality between optimal control (OC) and probabilistic inference (Leiine] 2018 Todorov 2008 Rawlik et al.] 2012) 2). The idea is that, if we consider artificial binary observation whose emission probability is given by the exponential of a negative cost, an OC problem can be reformulated as an equivalent inference problem. In this case, the objective is to find the trajectory control policy that maximizes the likelihood of the observations along the trajectory. One advantage of this perspective is that in order solve the OC or RL problems, we can adopt any powerful and flexible inference methods, e.g.. the expectation propagation (Toussaint 2009), the particle filtering (Ha et al. 2018a] Piche et al. (219)) or the inference for Gaussian processes (Mukadam et al., 2018). In addition utilizing efficient inference methods, this work also enjoys the duality to transform a multi-task RL problem into generative model learning problem, which enables an agent to distill a low-dimensional representation and versatile control policy in combined framework.
A.2 REINFORCEMENT LEARNING AS PROBABILISTIC INFERENCE
For easier reference, we restate the RL problem and the controlled trajectory distribution here:
0" arg max Bqu(snx.aa.
Σ (Sk; a
(11)
K
qo ( a1:k) )= = p(s1) II
p(sk+1 Sks (ar)
(12)
respectively. It is well known in the literature that the above optimization (Fquation 11 also be viewed as a probabilistic inference problem for a certain type of graphical models (Levine] 2018 Todorov (2088) Rawlik et al. 2012).. Suppose have artificial binary random variable ot, called
the optimality variable, whose emission probability is given by exponential of a state-dependent reward, i.e.,
p(o; = 1|s²) = exp (r*(s))),
(13)
and the action prior p(ax) defines the uncontrolled trajectory distribution (see also Fig. 2(a)
K
p(s al:k) = p(s:)
II
(S [Sk, a(as)
(14)
k=1
Then we can derive the evidence lower-bound (ELBO) for the variational inference:
logp(On.k) = log / p(01.x $1.. (p(Sk aik )ds
log /
Go (0(.) a1:k) (S a1.K)
(10.://1x)p(ss1) (5Lx)p((SIKK a1.k)
ds daj
To Ta(ak(sH) p(a:)
log logp(Ok(sz)) loog
To Tay [Sk) p(a:)
(gg a))
Tk(Sk) log
L(0).
(15)
The ELBO maximization in Equation 15 becomes equivalent to the reinforcement learning in
Equation [1 by choosing an action prior p ag) and parameterized policy family n (ag|ss (Sk) to match
Fs. Tk log > To resulting in the variational distribution that best approximates the posterior trajectory distribution when all the optimality variables were observed p(S1.K, a M1.KOO = L).
3d Similar Equation 19 the above maximization means to find the control policy
3 For example, when p(ax) and wo Faul(s))) are given as Gaussian with the same covariance, log encodes quadratic penalty on the control effort; when p(a) given as an uninformative uniform distribution, log ze hecomes the entropy regularization term in the maximum entropy reinforcement learning (Ziebart et al. 2008 Haarnoja al. 2018b)
13

---

Under review as a conference paper at ICLR 2020
A.3 SELF-SUPERVISSE LEARNING OF LATENT DYNAMICAL MODELS
Self-supervised learning is an essential approach that allows an agent to learn underlying dynamics from sequential high-dimessional sensory inputs. The learned dynamical model can be utilized to predict and plan the future state of the agent. By assuming that observations were emerged from the low-dimensional latent states, the learning problems are formulated as latent model learning, which inc intractable posterior inference of latent states for given input data (Karl et al. 2017 Krishnan et al. 2017
Fraccaro et al. 2017 Ha ct al. 2018b).
Suppose that a set of observation sequences {si:k I==I,...,, is given, where $1:K = {8k; Ak: 1,.., KJ ))(n) are i.i.d. sequences of observation that lie on (possibly high-dimensional) data space S. build probabilistic model that well describes the given observations. The problem is formulated as a maximum likelihood estimation (MLE) problem by parameterizing probabilistic model with di
The goal of the self-supervised learning problem of interest
arg max Σ log (a(s
(16)
For latent dynamic models, we assume that the observations are emerged from a latent dynamical system, where a latent state trajectory, Z1:K = {zE: Vk E 1,.., K}. lies on a (possibly low- dimensional) latent space Z:
Po(S1:k)= / po(SL: (i:k (p.k)) )dz.:;
(17)
where (S (1:K) and (z(2):) are called a conditional likelihood and prior distribution, respectively. Since the objective function (Fquation T6) contains the intractable integration, it cannot be optimized directly. To circumvent the intractable inference, a variational distribution g(-) is introduced and then a surrogate loss function C(g, $1.k), which is called the evidence lower bound (ELBO), can be considered alternatively:
log (S1:k) log / po(s1: 11. )p (21: )d21.
[log (s |1 no (2 (21.x)
2 Egtank)
q(1.k)
= E(g, $1.K),
(18)
where 7(-) can be any probabilistic distribution over 2 of which support includes that of pa(+). Note that the gap between the log-likelihhe ligeeikl and the ELBO the Kullback-Leibler (KL) divergence between (zz) and the posterior Po (Z ($1:k):
log - L(q. Φ: s1.kk) = D (xL(g(zxx))llpoxzx (sx
(19)
One of the most general approaches is the expectation- cnniriioommmiimizattion (EM) style optimization where, alternately, (i) E-step denotes an inference procedure where an optimal variational distribution q* is computed for given & and (ii) M-step maximizes the ELBO w.r.t. model parameter á for given g*.
Note that if we construct the whole inference and generative procedures as one computational graph. all the components can be learned by efficient end-to-end training (Karl et al. 2017); Krishnan jet al., 2017) FFaccarro et al. [017 Ha al. 2018b). In p articular, Ha et al. (201 (20T8b) proposed the adaptive path-integral autoencoder (APIAE), framework that utilizes the optimal control method; this framework suitable to this work because we want to perform the planning in the learned latent space. APIAE considers the state-space model in which the latent states governed by stochastic dynamical model, i.c., the prior p& (Z1:K) probability measure a following system:
Zk+1 fe (2)) a (Z))) WW,, Zo ~ po(.), We ~ NN(0 T).
(20)
Additionally, a conditional likelihood of sequential observations is factorized along the time axis:
K
Po (S (2.K)
II
(Sk zz))
(21)
14

---

Under review as a conference paper at ICLR 2020
If the variational distribution is parameterized by the control input u1:K-1 and the initial state distribution go as:
2x++1 [a(zz) + (x) (ux wk), Zo go(+), We ~ ~N(,
(22)
the ELBO can be written in the following form:
L - En log (2xx) + log po (Z)) To (zo)
un || u wa
(23)
Then, the problem of finding the optimal variational parameters u" and q (or equivalently, the best approximate posterior) can formulated as a stochastic optimal control (SOC) problem:
90 arg min E (x)
V(zi.) +
uu 112 + u Wk
(24)
1..00
E Pos (x) serves as a state cost of the SOC problem. Ha et al. 2018b) constructed the differentiable computational graph that resembles the path-integral control procedure to solve the above SOC problem, and trained the whole architecture including the
where VZz..) log (x())))
K
go (0))
latent dynamics, (z)),, (z) and o and the generative network, pa (s|z) through the end-to-end
training.
A.4 PLANNING BY PARTICLE FILTERING
Algorithm 2 Particle Filtering with Internal Model for Planning
1: Initialize Vi E {1, ..., Mparticle:: } z(( (-(--(s.ur) and w(() I/N particle
2: for k = 2, ..., Kplan do
3: for i 1, N partiele do
(i) (z (i)
t (7) (z
(0) (n(2 +w2l) w ~((.
4:
z
(1) N (Holzh) (0),, 2d(4 (1)
5:
Ss
w (0) wh1 exp(ra(s!)))
6:
7:
end for
8:
w w
E, wk Vi {l, . N }pantick
9: Resample f2 lski
(
w (1)
} if (u(
<Nprticle/3
lisk
10:
end for
return h Ee
b h E, w Kyon
Vk 1,.., KMPC for general MPC cases
Kptm
At each time step öt, the high-level planner takes the current state as an argument and required to output the commands by predicting the future trajectory and corresponding reward (-)))) We adopted the particle filter algorithm to perform such the reasoning and the pseudo code is shown in Algorithm 2 The particle filter algorithm attempts represent the posterior distribution using a of samples. The algorithm first samples N Nparticte initial latent states using the inference network (which a part of the learned internal model) and assigns the same weights for them. During the forward recursion, the particles are propagated using the latent dynamics of the internal model (line 4), and the corresponding configurations are generated through the learned model (line 5). The weights of all particles are then updated based on the reward of the generated configurations (line 6 and 8); i.e., the particles that induce higher reward values will get higher weights. If only few samples have weights effectively, i.e., if the weights collapse, the algorithm resamples the particles from the current approximate posterior distribution to maintain the effective sample size (line 9). After the forward recursion over the planning horizon, the optimal commands computed linear combination of the initial disturbances; i.c., is given by the expected disturbance under the posterior transition dynamics (Kappen & Ruiz 2016).
15

---

Under review as a conference paper at ICLR 2020
State
Action
1x|=197
1024
512
Latent
$12
256
la In/=36
elamenteise muliplization
|4|-1
1824
Figure 5: The network architecture of low-level policy network.
A.5 TRAINING LOW-LEVEL POLICY
For the training algorithm for low-level policy network (Ta), we extend motion imitation approach (Peng et al., 2018) to multi-task scheme; we construct value networks parameterized by neural network with size 1197, 1024, 512, 1] for each task (three in our experiments), and the low-level policy network (actor network) taking a state feature s and a latent variable as inputs to determine an action as illustrated in Fig. 5
The imitation reward is given following:
Ft
0.3r
0.2r pose + 0.15 + 0.15r 0.2r.
(25)
rt exp
0.5| p/:
- 2 0.5||p;
1
0.5| q
qi 1 0.05|l - d1113
rt pose
exp
(22 > |/ -
r exp
0.1 > |/ of 113
re exp
(-40) Σ
lp p(l1)
r = exp
( -Ipp; -p(1(
where qi and ps represent angle and global position while represent those of the reference As reference motion data, three motion capture clips, turning left (t [1. 0, 0J), going straight (t = [0, 1, OD, turning right (t [0, O, 1J) from the Carnegie Mellon University motion capture (CMU mocap) database are utilized. Following the reference, PPO with same hyperparameters is used for RL algorithm. Since the internal model does not exist at the first iteration (I = 1). the high-level planner is initialized by (b]s:t) wTt where = [-1, O, 1]. After the first iteration, high-level planner computes a command h, that makes the model to best follow the horizontal position the reference motion for seconds (ōt 0.1s and Kptan 50). The corresponding reward function is as following:
- -lP 11.
(26)
where Ph.& is the horizontal components of position vector time step k.
A.6 TRAINING INTERNAL MODELS
Internal models of DISH is trained maximize the ELBO in Equation 23 by using the APIAE
approach (Ha et al. 2018b) with hyperparameters as following: 3 adaptations R = 4). 10 time steps
(K 10), 32 samples (L = 32), and time interval of 0. (8l = 0.1). The network architectures of transition network and inference network are shown in Fig
For the baselines, the transition functions, fo (x++1 (k)),), were parameterized by neural networks having same architectures as DISH except for the input variables shown in Table 3 The loss
'Fach superscript denotes following: root (pelvis), j: local joints, end-effectors (hands and feet), center-of-mass.
16

---

Under review as a conference paper at ICLR 2020
Table 3: Input variables of baseline transition models
input variables
DISH
sa.'
shss
zaz'
X
Z
Sk
Zi:
Yk
h;
ag
hg
ag
State
Statee
Latent State
lxgl=3
Σ (Al' xl'xa)
M(s)
Loncal
Control
3r|=3
128
bral11
[44% ai learmable parameters
transition network
(b) generative network
Figure 6: The network architecture of internal model.
function for baseline is as following:
Leass (x)
|ss
86
12,
8g paas' ($k 1, a
(St
(27)
k=1
K
Lohe' (Ø)
Σ
||s: 51.12.
By paha' (s 12-1),
(28)
kml
Leas' (0)
Σ ||s:
g(x))l
Zg fzaz (ă au ax-1),
(29)
where $ $1, is latent state for 8 inferred by VAE, and g(-) is learned generative network of VAF.
A.7 TASK CONFIGURATIONS
Trajectory Following Tasks: Planning reward v. penalizes the distance between the horizontal position of the root of humanoid character P and the that of reference trajectory P
Fk - p5 P
(30)
Planning cost rt penalizes the distance between the horizontal position of the root of humanoid character P and the that of the goal Proal and the collision with obstacles, while giving a reward on arrival:
Navigation Tasks:
Fk -l/pg 1I. 102 (1S_ (S__CRASHE) 10* x (IS_ REACHED).
(31)
A.8 FURTHER RESULTS
Table 4 reports the RMS between reference, planned, and executed trajectories for each tasks. As illustrated in the table, DISHs showed the best performance. Although shs' sometimes showed smaller errors for the difference between the planed and reference trajectories, the emrors between the reference and executed trajectory of DISHs are always smallest. This demonstrates that DISHs best learn the internal dynamics of the humanoid, making the most valid predictions for future motion.
17

---

Under review as a conference paper at ICLR 2020
Table 4: Comparison of RMS errors between reference, planned, and executed trajectories for different types of internal models.
taskl (turn left)
Lank2 (go straight)
tack3 (turn right)
Iref-tract
Iplan-re/1
Upllan-cueel
Irrf--irul
Iplla--refi
Iplan-truell
Iree-truel
pplon-roff
Iplan-truell
0.1290 0.1466
0.1364
0,0743
0,2073
0.1705
0.1073
0.1550
0.1661
0.0974
DISH
0.1177
0.1711
sa.e" F
0.2474
F
0.0660
F
0.2178
0.2525
0.2767
0,2247
20.5
0.2731
0.1991
0.3014
Comparing DISH (Z 1) and DISH+ (L. = 2), we can observe that DISH outperforms the turning tasks while showing the worse performance in going straight. This is because the high-level planner initialized to output only one of {-1,0, 0, 1} (as shown in Appendix (A.5). so the corresponding low-level policy of DISH is trained only around h {-1, 0, 1} rather than along the continuous h values. As a result, the DISH agent is only good at radical turns (not smooth turns), making it difficult to stabilize its heading direction properly. The ability to turn smoothly is obtained in the next iteration where the proper reasoning module is equipped, thus, although lost some ability to turn fast, the DISH+ agent achieves the better ability to walk straight and the increased average performance (see Table 2)
Fig. shows rollout samples by varying the control values. Except for DISHs, the generated trajectories are very noisy, which indicates that the baseline internal models are not suitable for planning the future movements of the humanoid.
18

---

Under review as a conference paper at ICLR 2020
2
0.2
0.4
0.1
2-axis (meters)
0
0.2
(meteess)
1
0.0
0
-2
-0.2
-0.
-1
-0.4
-0.6
X (meters)
X-axis (meters)
(a) DISH
(b) shs
-2
-3
2 (meters)
6
0
2
4
(meters)
(c) sas'
(d) zaz'
Figure 7: Rollout samples from different types of internal models. (a) and (b) is colored by latent control value.
19


Reviews:
[{"review_id": "r1erI4tctH", "reviewer": "AnonReviewer3", "summary": "", "questions": "", "limitations": "", "rating": 1, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "This paper presents a framework for learning hierarchical policies using a latent variable conditioned policy operating at the low level, with model based planning at the high level. Unlike prior work which does hierarchical reinforcement learning, the key technical contribution of this work is that they use planning with a latent dynamics model as their high level policy. They demonstrate the method on a humanoid walking task in the DeepMimic [1] environment.\n\nWhile the idea is well motivated, this paper should be rejected, primarily due to a lack of experimental results. In particular, the experiments (1) are missing several critical details about the experimental setup, (2) the experimental setup differs significantly from the claims of self-supervision and multi-task RL made in the introduction/method, and (3) there is no comparison to any prior work. Without these, it is impossible to determine if any of the claims made about the proposed method are empirically true.\n\nFirst, no information is provided about the reward function used, the horizon of the tasks, or about the planning parameters or policy learning parameters. As currently stated, I don't think any of the results in the paper could be reproduced. Furthermore, the reward function and task horizon used are necessary to determine the difficulty of the proposed tasks. \n\nSecond, the title and method section would imply that the method is self-supervised, specifically in how the latent dynamics model is learned. While the samples used to train the latent dynamics model are taken from the agent's experience, the latent conditioned policy is trained (1) with ground truth task reward, and (2) is actually pre-trained on demonstrations of the tasks with ground truth skill labels. This suggests that much of the actual skill learning is done offline in this pre-training stage - with full supervision. As a result, this would make the learning of the latent dynamics model much easier, since the sub-policies have already converged to different behaviors for different values of h. Without this pre-training, learning the low level policies and the LVM jointly would be much more challenging. Hence it seems that the \"self-supervised\" learning of the LVM is actually heavily dependent on the full supervision used in the pre-training stage. Additionally, the demonstrations used for the pre-training correspond to the same 3 tasks that the agent is later evaluated on (moving forward, left, right). So the method receives full supervision on the test tasks, so the experiments do not actually reflect generalization in multi-task RL as claimed.\n\nLastly, and most importantly, there are no comparisons to prior work. The only result shown is the trajectory error against 3 ablations of the proposed method. The reported numbers are error between the reference trajectory and ground truth, predicted plan and reference, and predicted plan and ground truth. First, it seems like the most important number here is the difference between the predicted plan and ground truth, for which numbers are missing for 2/3 ablations. Why is task success or reward not the reported number, and why is performance for 2/3 ablations missing? Additionally, there should be comparisons to existing work both in terms of hierarchical model free RL (for example Nachum et al [2]) and model based RL with latent dynamics models (Hafner et al [3]). The results as presented do not actually support that the proposed method performs better than existing work. The final result of the video of the agent doing a long horizon task also has no quantitative numbers, so again does not support that the proposed method is better.\n\nSome other less significant points:\n- there are typos throughout (for example \"Figure 3: (a) Leanred latent model.\").\n- the tables and figures have very limited or no captions.\n- The method section is difficult to follow and could use some figures which demonstrate the key technical contribution.\n- Also from the method section it seems like the novelty is combining the latent conditioned policy learning from Haarnoja et al [4] and the latent dynamics learning/planning from Ha et al [5]. Is there an additional technical contribution beyond combining these two existing works? If so the method section should more clearly show it.\n- There is a recent work (Sharma et al [6]), which also learns skill conditioned low level policies, and does model based planning in the space of skills to reach previously unseen goals. In this work the skill discovery is also totally unsupervised. The authors should add citation to this paper and clarify the differences between their work and this work.\n\n[1] Xue Bin Peng, Pieter Abbeel, Sergey Levine, and Michiel van de Panne. DeepMimic: Example guided deep reinforcement learning of physics-based character skills.\n[2] Ofir Nachum, Shane Gu, Honglak Lee, and Sergey Levine. Data-efficient hierarchical reinforcement learning\n[3] Hafner, D., Lillicrap, T. P., Fischer, I., Villegas, R., Ha, D., Lee, H., and Davidson, J. Learning latent dynamics for planning from pixels\n[4] Tuomas Haarnoja, Kristian Hartikainen, Pieter Abbeel, and Sergey Levine. Latent space policies for hierarchical reinforcement learning.\n[5] Jung-Su Ha, Young-Jin Park, Hyeok-Joo Chae, Soon-Seo Park, and Han-Lim Choi. Adaptive path integral autoencoders: Representation learning and planning for dynamical systems.\n[6] Archit Sharma, Shixiang Gu, Sergey Levine, Vikash Kumar, and Karol Hausman. Dynamicsaware unsupervised discovery of skills.\n\n", "extra_scores": {"experience_assessment": "I have published one or two papers in this area.", "review_assessment:_checking_correctness_of_derivations_and_theory": "I assessed the sensibility of the derivations and theory.", "review_assessment:_checking_correctness_of_experiments": "I carefully checked the experiments.", "review_assessment:_thoroughness_in_paper_reading": "I read the paper thoroughly."}, "extra_text": {}}, {"review_id": "SJgWiAJCFr", "reviewer": "AnonReviewer1", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "This paper proposes a latent variable model to perform imitation learning. The authors propose the model in the control-as-inference framework and introduce two additional latent variables: one that represents a latent state (z) and another that represents a latent action (h). For the generative model, the authors use a sequence latent variable model. For inferring the latent action, the authors use a particle filter. For inferring the states, the authors use an \"Adaptive path-integral autoencoder,\" though it was unclear where the controls \"u\" come from. (I assume u is the same as the actions, at which point inferring the states amounts to rollout the policy in the sequence latent variable model). The authors compare to not having the latent states and/or not having the latent actions, and demonstrate that they get better imitation learning scores.\n\nOverall, I found the paper difficult to follow and some of the reasoning a bit unclear. The experiments seemed limited in scope, given that the authors discuss reinforcement learning in general, but only provide results on the reconstruction error when doing imitation learning. It is also unclear to me whether the gains from the experiments are from their model, or from the fact that their model probably has more parameters since it has more components. It would be good for the authors to compare to existing work that uses sequential latent variables models for deep RL, such as [1,2,3]\n\nMore detailed comments:\n\nIt would be good for the authors to substantiate statements like, \"Since training sas? is just a simple supervised learning problem, it had the lowest reconstruction error but the computed action from such the internal model couldn’t make the humanoid walk.\" with plots.\n\nThe statement, \"zaz' also failed to let the robot walk, because reasoning of the high-dimensional action can’t be accurate enough.\" seems similarly unjustified. If the authors wanted to test this, they could train a zaz' model with some low-dimensional action (e.g. left, straight right) and verify that this works.\n\nWhile the authors state that \"h can be interpreted as high-level commands,\" but if it is inferred at every time step, why is this a \"high-level\" command?\n\nNit-picks:\n- \"The procedure consists of outer internal\" --> \"The procedure consists of *an* outer internal\"\n- \"via via\"\n- \"Such the sophisticated separation was\"\n\n[1] Danijar Hafner et al. Learning Latent Dynamics for Planning from Pixels.\n[2] Maximilian Igl et al. Deep Variational Reinforcement Learning for POMDPs.\n[3] Alex Lee at al. Stochastic Latent Actor-Critic: Deep Reinforcement Learning with a Latent Variable Model.", "extra_scores": {"experience_assessment": "I have published one or two papers in this area.", "review_assessment:_thoroughness_in_paper_reading": "I read the paper at least twice and used my best judgement in assessing the paper.", "review_assessment:_checking_correctness_of_experiments": "I carefully checked the experiments.", "review_assessment:_checking_correctness_of_derivations_and_theory": "I assessed the sensibility of the derivations and theory."}, "extra_text": {}}, {"review_id": "Byl6ccPwFH", "reviewer": "AnonReviewer2", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "*Edit: original score was a weak reject (3), updating to a weak accept (6) in light of revisions.*\n\nThis work implements a hierarchical control scheme for a high-dimensional control problem (locomotion using a humanoid body).  The hierarchy consists of a high-level module that plans in an abstract space of \"intention\", and the intention variables serve as inputs, along with state, to a low-level controller that actually executes the movements.  The premise is that a lower-level controller should be usable for multiple tasks, and should be able to be commanded by a lower-dimensional intention input.  I find the basic ideas presented clear, the literature reviewed reasonably well, and the motivation and setting to be very interesting.  The video summary is valuable.\n\nMy main concerns have to do with presentation, but I think they are relatively significant concerns.  As the draft currently stands, I would, somewhat regrettably, be inclined to reject the submission (marginally).  I think revisions could seriously improve this paper and incline me towards acceptance.\n\nAlgorithm 1 indicates that the learning of the low-level controller will be done jointly with the learning of the latent model and planning using the high-level, learned intention space.  In the experiment, it is indicated that the low-level controller is pretrained.  This points to a couple issues that are not clear in the draft:\n(1) Presumably this pretraining is necessary and things do not work without it.  Indeed, it is hard to imagine that the movements will be well grounded to human motion capture movements without this pretraining.  Does the algorithm work as written or is pretraining a fundamentally essential step?  There are no settings, even toy settings, where the algorithm as written is shown to be effective.\n(2) The authors should be clearer how they conduct the pretraining which involves learning the low-level controller.  \n(3) I'm not clear how updating the low-level controller is effective in the algorithm.  While I understand why it makes sense to plan in the intention space of the pre-trained controller, and I understand why learning a model is a core part of planning, it would seem like fine-tuning the low-level controller could make the movements deviate considerably from the initial movement space and maybe even eliminate the ability of the low-level policy to express movements that are not used early in training. So essentially, while the planning in the low-D space makes sense and the learning of the model makes sense, the low-level controller update seems possibly to not make sense, and there aren't experiments showing that step helps. \n\nIs it just a coincidence that the intention space (h) is one-of-three and the low-d state space (z) is 3-dimensional as well?  Or are these both selected with sort of going straight vs turning left or right in mind?\n\nThe experiment section is generally very unclear, though details are made a little clearer from the video.  In the paper, there are a few points that need to be clearer: \n(1) \"ref\", \"plan\", and  \"true\" are not well defined and it is unclear what these distances in Table 1 refer to precisely.  Clearly introduce what each of these refers to.  The authors simply say that there are imitation tasks but do not walk through what these terms refer to. \n(2) In 4.1, the different structures are not adequately introduced.  The pointers to the figure 2 diagrams are essential, but there is no pointer for zaz', the pointers are only in the table (not in the text), there are grammar issues in the text and the text could be verbally clearer about the variants.\n(3) shs' setting is a bit unclear. Basically, clarify briefly how planning is performed in this case.   Is a forward model still trained, but the model opperates with the full state space?  If so, presumably the forward model is much worse and then the planning approach is correspondingly bad, hence the poor rollouts? \n(4) 4.2 is I think obviously inadequately described in the text and I can only assume was the result of rushing for the deadline?  The second experiment is essentially not presented in the text all aside from a still image.  \n(5) And for all of the experiments that rely on planning with a particle filter, details such as how reliable the filter is in generating useful control, how many samples are required, and possibly elements of compute speed would make much clearer how well the approach actually works.  Does the choice of planner matter at all?  A common, albeit relatively weak, baseline planning approach is CEM...would CEM work here?  I'd like to understand if the choice of particle filter is the author's default choice, which is fine if so, or if there is a positive assertion being made that the particle filter is particularly valuable.\n\nI hope the authors will generally improve the exposition in the experiment section (4) during the revisions.\n\nOverall, I find the paper well motivated in framing the problem (i.e. using model-based approaches to control the latent space of a low-level controller).  I also appreciate the scale of the problem (humanoid control is challenging, so this is not a toy problem).  I find the results a bit unclear, perhaps due to hurriedness in writing, so I find them a bit difficult to fully appreciate.  Nevertheless, the core contribution that I take away from this work is that there is a value to learning the low-dimensional state representation (z, via the LVM), relative to planning using a forward model on the full state (?...I'm still unclear on the presentation of this result, due to unclear exposition). Slightly more broadly, this is a good demonstration of using a planner jointly with a learned high-level command/intention representation, for a high-dimensional problem. \n\nIf I've understood this correctly, I'd be reasonably interested in this result. If the authors can both clarify the core results and communicate that the choices made in the algorithm are well thought through, I would be happy to adjust my score.\n\n\nRelatively minor:\n\nAbstract says 90-dimensional humanoid system, but later it is stated \"34 degrees of freedom,\n197 state features, and 36 action parameters\". Where the 90 dimensions comes from is unclear.  Often people refer to number of actuators or DoFs.  Please adjust this or be more explicit.\n\nIn equation 9, f() is not very clearly specified. Is f() a nonlinear function (e.g. a neural network) or is it a linear function? It seems like it might as well be a linear function, since the authors propose to learn a latent dynamics model that is nonlinearly related to the state.\n\nTypos in Fig 3 caption.\n", "extra_scores": {"experience_assessment": "I have published in this field for several years.", "review_assessment:_checking_correctness_of_experiments": "I assessed the sensibility of the experiments.", "review_assessment:_thoroughness_in_paper_reading": "I read the paper thoroughly.", "review_assessment:_checking_correctness_of_derivations_and_theory": "N/A"}, "extra_text": {}}]

Author rebuttal:

Meta-review: The authors present a self-supervised framework for learning a hierarchical policy in reinforcement learning tasks that combines a high-level planner over learned latent goals with a shared low-level goal-completing control policy.  The reviewers had significant concerns about both problem positioning (w.r.t. existing work) and writing clarity, as well as the fact that all comparative experiments were ablations, rather than comparisons to prior work.  While the reviewers agreed that the authors reasonably resolved issues of clarity, there was not agreement that concerns about positioning w.r.t. prior work and experimental comparisons were sufficiently resolved.  Thus, I recommend to reject this paper at this time.
Decision: Reject