Inducing Reusable Skills From Demonstrations with Option-Controller Network
Authors: Siyuan Zhou
Yikang Shen
Yuchen Lu
Aaron Courville
Joshua B. Tenenbaum
Chuang Gan
Venue: ICLR 2022 Submitted (2022)
Keywords: Reusable Skill
Option-Controller Network

Abstract: Humans can decompose previous experiences into skills and reuse them to enable fast learning in the future. Inspired by this process, we propose a new model called Option-Controller Network (OCN), which is a bi-level recurrent policy network composed of a high-level controller and a pool of low-level options. The options are disconnected from any task-specific information to model task-agnostic skills.
The controller use options to solve a given task, and it calls one option at a time and waits until the option return. With the isolation of information and the synchronous calling mechanism, we can impose a division of works between the controller and options in an end-to-end training regime. In experiments, we first perform behavior cloning from unstructured demonstrations coming from different tasks. We then freeze the learned options and learn a new controller with an RL algorithm to solve a new task. Extensive results on discrete and continuous environments show that OCN can jointly learn to decompose unstructured demonstrations into skills and model each skill with separate options. The learned options provide a good temporal abstraction, allowing OCN to quickly transfer to tasks with a novel combination of learned skills even with sparse reward, while previous methods either suffer from the delayed reward problem due to the lack of temporal
abstraction or a complicated option controlling mechanism that increases the complexity of exploration.

Under review as a conference paper at ICLR 2022
INDUCING REUSABLE SKILLS FROM DEMONSTRA- TIONS WITH OPTION-CONTROLLER NETWORK NEWWRR
Anonymous authors Paper under double-blind review
ABSTRACT
Humans can decompose previous experiences into skills and reuse them to enable fast learning in the future. Inspired by this process, we propose a new model called Option-Controle Network (OCN), which is a bi-level recurrent policy network composed of high-level controller and pool of low-level options. The options are disconnected from any task-specific information to model task-agnostic skills. The controller use options to solve a given task, and calls one option at time and waits until the option return. With the isolation of information and the synchronous calling mechanism, we can impose a division of works between the controller and options in end-to-end training regime. In experiments, behavior cloning from unstructured demonstrations coming from different tasks. We then freeze the learned options and learn new controller with RL algorithm to solve a new task. Extensive results on discrete and continuous environments show that OCN can jointly learn decompose unstructured demonstrations into skills and model each skill with separate options. The learned options provide good temporal abstraction, allowing OCN to quickly transfer tasks with a novel combination of learned skills even with sparse reward, while previous methods either suffer from the delayed reward problem due to the lack temporal abstraction or complicated option controlling mechanism that increases the complexity of exploration.
first perform
INTRODUCTION
Acquiring primitive skills from demonstrations and reusing them to solve novel long-horizon task is hallmark in human intelligence. For example, after learning the necessary skills (e.g., steering wheel, changing lanes) at a driving school, one could be capable of driving across the country by recombining the learned skills.
In hierarchical reinforcement learning, different temporal abstractions (Sutton et al., 1999; Dietterich, 2000; Parr & Russell, 1998; Nakanishi et al., 2004) are proposed to achieve structured exploration and transfer to a long-horizon task. However, when learning from scratch, these pure HRL methods share exploration problem: it takes significant amount of samples for random walk induce good temporal abstraction that leads to positive rewards at the beginning of training. To circumvent this issue, recent works (Le et al., 2018; Levy al., 2018: Gupta et al., 2019; Jiang et al., 2019) have focused on learning useful skills in pretraining phase first, and then reusing these skills when finetuning with HRL in the new environment. However, these methods either assume the existence of coal-condiiioned policies or require intensive interaction with environments. which limits the practical values of these approaches. One general approach demonstrations during pretraining, e.g., complLE (Kipf et al., 2019) pretrains VAE (Kingma & Welling, 2013) on the demonstrations and uses an action decoder for finetuning. Our work in this line of research.
leverage the additional unstructured
In this paper, we propose Option-Contren Network (OCN), Inspired by the option framework (Sutton et al., 1999). OCN is two-level recurrent policy network including a high-level controller and pool of low-level options. At each time step, a selected low-level option outputs an action and a termination probability, and the high-level controller selects new option whenever the old option is terminated. Inspired by Lu et al. (2021), we enforce a special recurrent hidden state updating rule to enforce the hierarchical constraint between the controller and the options that the high-level

---

Under review as a conference paper at ICLR 2022
controller is updated less frequently than the low-level options while keeping the model end-to-end differentiable. As shown in Figure 1, OCN can jointly learn options and controllers with multitask behavior cloning from unstructured demonstrations. When given new task, one could perform HRI finetuning by re-initializing the controller and freezing the options. This enables our model to generalize combinatorially to unforeseen conjunctions (Denil et al., 2017). Unlike previous works, our method does not require generative models (Eysenbach et al., 2018), goal-conditioned policies (Gupta al., 2019), pre-specified policy sketch (Shiarlis et al., 2018) or constraints on the number of segments (Kipf et al., 2019), making our approach conceptually simple and general.
Option
Option
GelWood
GetGrass
Controller
Controllar
Demonstrations
Option
Option
actorry
Geilron
Behavior Cloning
HRL Finetune
Re-intialize
GetWWood
GetGrass
GeltWood
GetGrass
Controllar
Controllar
CoFactory
Gellron
Go Factttrry
Geilron
The training pipeline of OCN. Our model is composed of a controller (circle) and options pool (rectangles). The controller and options are randomly initialized, which means each option does not correspond to a meaningful subtask. After behavior cloning, both options and controllers are induced (marked blue) and the options correspond to meaningful subtasks from demonstrations (e.g., get wood). Then we freeze the parameters in the options and re-initialize the controller. The controller is trained to adapt to the new environment with HRL (marked red).
Figure I:
We perform experiments in Craft (Andreas et al., 2017), a grid-world environment focusing on navigation and collecting objects, and Dial (Shiarlis et al., 2018). robotic setting where a JACO 6DoF manipulator interact with a large number pad. Our results show that with unstructured demonstrations, OCN can jointly learn to segment the trajectiries into meaningful skills as well as model this rich set of skills with our pool of low-level options. During HRL finetuning, we show that OCN achieves better performance in more complex long-horizon tasks with either sparse or dense reward compared with existing baselines. We also provide further visualization to show the discovered options are reused during finetuning. Our contributions can be concluded as:
1. We propose Option-Controller Controller Network, bi-level recurrent policy network that uses special recurrent hidden state updating rule to enforce the hierarchical constraints.
2. We show that the OCN can discover the optimum options decomposition from unstructured demonstration via multitasking behavior cloning.
3. We also demonstrate that the learned options can be coupled with new controller to solve unseen horg-horizon task via interaction with the environment.
2 RELATED WORK
Learning to solve temporally extended tasks is an important question for Hierarchical Reinforcement Learning (HRL), including option frameworks (Sutton et al., 1999), HAM (Parr Russell, 1998) and max-Q (Dietterich, 2000). With the popularity of neural nets, recent works propose to use bi-level neural network such as option critics (Bacon et al., 2017), feudal networks (Vezhnevets et al., 2017), generative models with latents (Nachum et al., 2018), and modulated networks (Pashevich et al., 2018). These models can be furthered combined with hindsight memory (Levy et al., 2018) to increase the sample efficiency. Our work can also be viewed designing specific neural architecture for HRL.
However, discussed in Section 1, pure HRL method suffers from serious exploration challenges when learning from introduce pretraining phase "warm up" the policy. Recent works propose to pretrain the policy
scratch (Gupta et al., 2019). A general approach to tackle this problem is to
2

---

Under review as a conference paper at ICLR 2022
with an intrinsic diversity reward (Eysenbach et al., 2018) or language abstraction (Jiang al., 2019), which is shown to be useful in the HRL. However, assuming access to an environment in the pretrain phase might be infeasible in many tasks. A principal approach is to leverage the additional unstructured expert demonstrations and performs imitation learning during pretraining. Our work can be viewed solving the "cold start" problem for the option framework (Sutton al., 1999).
Recent works build upon this "imitation finetune" paradigm. With the prevalence of goal-conditioned policies (Schaul et al., 2015; Kaelbling, 1993; Levy al., 2018) in robotics, these methods leverage demonstrations with relabelling technique to pretrain the low-level policy (Gupta et al., 2019) or a generative model (Lynch et al., 2020). However, they exploit the fact that the ending state of a trajectory segment can be described as a point in the goal space. Hence it is difficult to apply them beyond goal-conditionss
policies. ComplLE (Kipf et al., 2019) treats the segment boundaries as latent variables, and their model can be trained end-to-end with soft trajectory masking. However, ComplLE requires specifying the number of segments, while OCN only requires that the number of options to be large enough (Appendix A.5. Following similar VAE setting, SPiRL (Pertsch et al., 2020) includes a skill prior and a skill autoencoder. They encoder skills into latent variables and use a skill prior to generate high-level instructions sequence of latent variables. However, SPiRL define skills as a sequence of actions with a fix horizon H, which could prevent the model from learning skills with clear semantics. Modular policy networks (Andreas al., 2017; 77 Shiarlis et al., 2018) are also used in this paradigm, where each subtask corresponds to a single modular policy. However, in this setting, the demonstration needs to segmented beforchand, which requires additional human labor. On the contrary, our work focused on using unstructured demonstrations. OptionGAN (Henderson al., 2018) proposes a coctuure-of-Exerrr (MoE) formulation and performs IRL on the demonstration. However, without an explicit termination function, the learnt expert networks do not provide time-xttnndd actions for the high-level controller. As a result, this method still suffers from problems of exploration with sparse rewards (as also seen in our experimental comparison with an MoE baseline).
Extracting meaningful trajectory segments from the unstructured demonstration is the focus of Hicrarchical Imitation Learning (HIL). These works can be summarized as finding the optimal behavior hierarchy so that the behavior can be better predicted (Solway et al., 2014). DDO (Fox et al., 2017) proposes an iterative EM-like algorithm to discover multiple levels of options, and it is applied in the continuous action space (Krishnan et al., 2017) and program modelling (Fox et al., 2018). VALOR (Achiam et al., 2018) extends this idea by incorporating powerful inference methods like VAE (Kingma & Welling, 2013). Directed-Info GAIL (Sharma et al., 2018) extracts meaning segments by maximizing the mutual information between the subtask latent variables and the generated trajectory. Ordered Memory Policy Network (OMPN) (Lu et al.. 2021) proposes a hierarchical inductive bias to infer the skill boundaries. The above works mainly focus on skill extraction, so it is unclear how to use the segmented skills for RL finetuning. Although OCN shares a similar inductive bias with OMPN, OCN replaces the continuous hidden states communication with a softmax distribution over multiple low-level modules (options). This enables OCN to model different subtasks with different options and to effectively reuse them in a new task.
3 METHODOLOGY
An Controller Network (OCN) includes set of N options for, ON} and a controller c. As shown in figure 1, the OCN starts by using the controller to choose an option to execute the first subtask. Once the subtask is done, the controller will choose another option to execute the second subtask, and so on, until the goal of the task is achieved. OCN shares similar inductive bias with Ordered Memory Policy Network (OMPN) (Lu et al., 2021), that the lower-level components (options) execute more frequently than the higher-level components (controllers). The inductive bias enables OCN to induce the temporal hierarchical structure of unstructured demonstrations.
3.1 OPTION AND CONTROLLER
Option As shown in the middle of Figure 2, option o, models skill that can solve specific subtask, for example get wood, get iron or make at workbench. It can be described as:
Pi, ho eitt ox(x;, hi.-1)
(1)
3

---

Under review as a conference paper at ICLR 2022
where is the observation at time step t, and h is the hidden state of the respective option time step - 1; h is the hidden state of o, at time step f; e4, is a scalar between and 1, represents the probability that the current option is done; P is distribution of actions, including move up. move left and use. These actions are the smallest elementary operations that an agent can execute. During the execution of option, if probability 6i,t O, the option will keep executing the current subtask; if e is 1, the option will stop the execution and return to the controller for the next subtask. In our work, each option maintains a separate set of parameters.
Controller
Make bridge
Make bridge
Make bridge
Make at factory
Make at factory
Options
Get iron
Get iron
Get wood
Get wood
Actions
Down
Use
Right
Use
Down
Use
An example of OCN. The controller с models the task make bridge. separately model subtasks get iron, get wood make factory.
Figure 2:
Three options
Controller As shown at the top of Figure 2, controller models higher level task, like make bed, make axe, or make bridge. Each of these tasks can be decompose to a sequence of subtasks. For example, make bridge can be decompose 3 steps: 1) get iron, 2) get wood, 3) make at factory. Thus a controller can also be represented as:
P hū ef c(x, h
(2)
where Di is a distribution over the set of options {o,}. hy is the hidden state for controller, Ef is the probability that current task done. In this OCN architecture, we don't need the ef, since the environment will provide signal(reward) once the task is done. However, OCN can be easily expanded to multi-level model. In this lulti-levels model, a set of multiple controllers become options for a higher-level controller, and their respective tasks become subtasks for
complicated task.
Cell Network In OCN, options and controllers share the same format for input and output. Thus, we parameterize them with the same neural network architecture. To model the policy of controllers and options, proposed the following cell network:
h = MLP ([ h2=1))
(3)
P:: softmax (Waa Wacthy + bact)
(4)
h; tanh V hia + bbid)
(5)
e: sigmoid (wendhy + bend)
Ty is the raw observation, the shape of the vector depends on the environment. h2-1 is the recurrent hidden state of size dnia, it allows the model to remember important information from previous time steps. MLP is multi-layer neural network of Depth /ML and hidden size dMLr. We use tanh as activation function for MLP. he is a vector of size dya.P. Waet nct a matrix of size Plat dMM.P, where Naat is number of actions. Whia
matrix of size dhid dML.P. Wend is a vector of size dyLP. Following the fast and slow learning idea proposed in Madan al. (2021), we introduce a temperature term I' to controller's 's's softmax function:
W Waache + bact
pi softmax
T
large temperature T allows the option to output smoother distribution the beginning of training. It also reduces the scale of gradient backpropagated into the controller. This results in the controller changes and updates slower than options. We found makes OCN become more stable in imitation learning and converge to better hierarchical structure.
3.2 OPTION-CONTROLLER FRAMEWORK
Given the definition for options and controllers, we can further formulate OCN. As shown in Figure 3a, the first time step, the controller computes probability distribution over options for the first
4

---

Under review as a conference paper at ICLR 2022
step 1
step
step
Fc
Copy
O2
O:
hg
O,
Oj
(a) First time step
(b) Inside subtask
(c) Switching between subtasks
Figure 3: The three different phase of OCN: (a) At the first time step, the controller selects an option o; The option o, outputs the first action a1. (b) If the previous option o, predict that the subtask is not finish; The option o, then continue outputs action a;; The controller hidden state is copied from previous time step. (c) If the previous option o, predict that the subtask is done; The controller then selects new option 0, and updates the controller hidden state; The new option o, outputs action ag. Blue arrows represent probability distributions output by controller and options. Red arrows represent recurrent hidden states between time steps.
subtask, and options execute their first steps:
Pi h = c((1, hg)
(8)
P h 11 611 0x1(1, h1,0)
Pî Evi.po Pi.
(10)
where hn and h are initial hidden states for controller and options, p a distribution for actions. The output pf is formulated mixture of experts, where experts options and the gating model the controller.
At time steps t 1, the options first execute one step decide whether this subtask is done. If the subtask is unfinished, the option then outputs an action distribution, as shown in Figure 3b:
pi h. c, = -x((x, 0.(x,,hi,-) h
(11)
Et = Σ 1i
(12)
b4, EpiLL.PE.
(13)
where é; is the probability that the previous subtask is done and p. is the action distribution if the subtask is done. the previous subtask done. the controller need select a new option distribution for the next subtask and reinitialize the option, as shown in Figure 3c:
D h c(x,, h;_1)
(14)
p h f,, ox h%,0)
(15)
P la
Σ Pl.p..
(16)
where hyS h 2t and pt, are hidden states and distributions for the next subtask the previous subtask done. Thus, we can formulate the output time step as weighted sum of the two situations:
h D
h
h
P
he
C
h
(1
e))
ho
(17)
P
p
The equation 17 provides OCN internal hierarchical inductive bias, that higher-level component (c) only update its recurrent hidden state and output a new command (p/") when its current functioning subordinate (o:) reports "done".
5

---

Under review as a conference paper at ICLR 2022
3.3 INDUCING AND REUSING SKILLS
Imitation Learning and Inducing Skills OCN imitates and induces skills from unstructured demonstrations. In the rest of this paper, d represents an unstructured demonstration { at) | D represents of demonstrations of different tasks [(d1,T), m), (d2; 72); ...,, where 7; are task ids, belongs to a shared task set T.
Given demonstration d, OCN can perform behavior cloning with negative regllikel loss:
loss average, (NLLLoss(p) az))
(18)
For different tasks we can use two different methods to model their associated controllers. The first method is to assign one controller C, and a initial hidden state h u to each т. The second method is to share the controller c, but assign a different initial hidden state h o to each т. We choose the second method in this work because sharing с could avoid the risk that different controllers choose to model the same subtask with options. During the imitation learning, OCN allows gradient backpropagation through all probabilities p. Thus, the gradient descent will try to induce an optimal set of options that can best increase the likelihood of the data.
Reinforcement Learning and Reusing Skills Given the induced options from imitation learning. our model can learn to solve a new task by reusing these skills via reinforcement learning. For example, after training on demonstrations of task 1 and task 2, OCN induce N options {o..... ON) }. Given new task 3 without demonstrations, we can initialize new controller C3. that takes observations as input and outputs a probability distribution over N induced options. To learn C3, we freeze all options and use PPO (Schulman et al., 2017) algorithm to learn C3 from interactions with the environment. During the training, once the controller outputs an option distribution pS. OCN samples from the distribution, the sampled option will rollout until it's done, then the process will repeat until the task is solved. Thus, in the RL phase, our model only needs to explore at options space, which significantly reduces the number of interaction steps solve the new tasks. We outline the process in Appendix A.
4 EXPERIMENTS
We test OCN in two environments. For the discrete action space, we use a grid world environment called Craft adapted from Andreas al. (2017). For the continuous action space, we have a robotic setting called Dial (Shiarlis et al., 201 8), where JACO 6DoF manipulator interacts with a number pad. We compare OCN with three baselines including task decomposition methods and hierarchical methods: (1) compILE (Kipf et al., 2019), which leverages Variational Luto-Encoder to recover the subtask boundaries and models the subtasks with different options. (2) OMPN (Lu et al., 2021) which studies inductive bias and discovers hierarchical structure from demonstrations. (3) Mixture-of- Eff-xxe (MOE), which uses a similar architecture as OCN, but the controller predicts new distribution over options at every time step. This baseline is designed following the MoE framework proposed in Henderson et al. (2018). Implementation details for OCN and baselines can be found in Appendix A.2 and A.3.
4.1 CRAFT
In this environment, agent can move in 2D grid map with actions (up, down, left. right) and interact with the objects with the action use. The environment includes subtasks: A) get wood, B) get gold, C) get iron, D) get grass. They require the agent locate and collect specific type of object. For example, subtask A, get wood, requires the agent to first navigate to the block that contains wood, then execute use action to collect one unit of wood. A task requires the agent to finish a sequence of subtasks in the given order. The environment can provide either sparse reward or dense reward. In the dense reward, the agent receives rewards after completing each subtask while in the sparse reward, the agent only receives rewards after completing all subtasks.
S1: Transferring from Single Agent In this setting, the training task set is {AC, DA). Dur- ing the imitation phase, we pretrain OCN with controller с₁ and three options (o,, O2, 0s} to imitate these demonstrations. During the fine-tuning phase, the model needs to solve three new tasks: (ADC, CAD, DCA). We initialize new controller for each while freezing the parameters

---

Under review as a conference paper at ICLR 2022
ADC-dense
CAD-dense
DCA-dense
2.5
2.0
2.0
1.5
1.5
1.5
0.5
0.0
0.0
0.0
0.00
0.25 0.50 0.75 1.00 1.25
1.50 1.75 2.00
0.00
0.25
0.50 0.75 1.00 1.25
1.50 3.75 2,00
0.
0.25 0,50 0.75 1.00 1.25 150 1.75 2,00 Million Frames
Million Frames
Million Frames
ADC-sparse
CAD-sparse
DCA-sparse
0.8
0.6
0.6
0.6
0.4
0.2
1.25
1.00 Million Frames
Million
Million Frames
compile
Figure 4: The learning curve of different methods on three finetuning tasks of S1. dense means dense reward setting. sparse means sparse reward setting.
of options. This is the classical setting where an agent is required to learn skills from short expert demonstrations and to transfer to long- horizhn tasks.
As is shown in Figure 4, our method converges faster and achieves higher performance than baselines both dense and sparse reward settings. With dense rewards, our method achieves double the returns than the strongest baseline. In the sparse reward setting, our method can get average return of 0.7 with the maximum being 1, while other baselines struggle. We find that MoE fails to achieve similar performance even with very similar architecture OCN. The only difference is that MoE does not model the termination of an option and the controller selects a new option every time step. This result shows that exploring in option space is efficient than other schemes, provided the new task is expressible as a combination of previous observed subtasks.
BADC-dense
ACBD-dense
CABD-dense
3.5
3.0
2.5
2.0
2.0
2.0
1.5
1.5
1.0
0.5
0.0
0.5
1.0
1.5
2.0
2.5
0.0
0.5
1.0
1.5
2.0
2.5
0.0
0.5
1.0
3.5
2.0
2.5
Million Fromes
Million Frames
Million Frames
BADC-sparse
ACBD-sparse
CABD-sparse
0.5
0.4
0.4
0.3
0.2
0.2
0.1
0.1
3.5
Million Frames
Million Frames
Million
compile
Figure 5: The learning curve of different methods three finetuning tasks of S2. OMPN is included because it does not learn an explicit set of options.
7

---

Under review as a conference paper at ICLR 2022
S2: Transferring from Multiple Agents In this setting, have two disjoint task sets. The first set is {AB, BA] and the second task set is {CD, DCJ. We train two separate OCN models. Each model includes controller and two options. Thus, at the end of imitation phase, ob- tain four options (01,.....4) Then we initialize three new controllers to solve three new tasks: (BADC, ACBD, CABD)
This setting is related to the problem of data islands and federated learning (Yang et al., 2019), where companies could each pretrain models on their separate datasets, merge the induced options, and share the controller finetuned on more challenging tasks. This is made possible because of the highly modularized design of our architecture.
The results are shown in Figure 5, We show that OCN can still reuse merged option pools, while other baseline methods fail this setting. CompILE uses continuous latent variable for communication between the controller and the action decoder, which causes compatibility issues while merging skills from different models. The MoE method still suffers from the long horizon problem. Overall, this result highlights the flexibility of OCN and its promise in maintaining data privacy for collaborative machine learning applications.
4.2 DIAL
[1, 4, 8]-dense
[8, 4, 2]-dense
[4, 1, 2]-dense
2.00
1.4
2.00
1.75
1.2
1.75
1.0
1.50
0.8
1.25
1.00
0.6
1.00
0.75
0.4
0.75
0.50
0.2
0.50
0.0
10
10
10
Million Frames
Million Frames
Million Frames
[1, 4, 8]-sparse
[8, 4, 2]-sparse
[4, 1, 2J-sparse
0.20
0.35
0.30
0.15
0.25
0.10
0.20
0.15
0.05
0.10
0.05
0.00
0.00
10
10
10
Million Frames
Frames
Frames
MOF
OCN
OMPN
Figure 6: The learning curve of different methods on three finetuning tasks of Dial. dense means dense reward setting. sparse means sparse reward setting.
In this experiment, the task requires the agent to move the robotic arm dial a PIN sequence of digits that should be pressed in the given order. We choose set S (1, 2, 4, 8) of 4 digits as the environment. Demonstrations contain state-action trajectories that the robotic arm press 2-digits pin randomly sampled from S. Demonstrations are generated from PID controller that moves the robotic arm predefined joint angles for each digit. The fine-tuning task is dialing 3-digits PIN sampled from S. The environment can provide either sparse rewards or dense rewards. In the dense reward setting, the agent can receive reward after pressing each digit in the PIN. the sparse reward setting, the agent only receive reward if all the digits in the PIN are pressed. The state has 19-dimensions which containing the 9 joint angles and the distance from each digit on the dial pad to the finger of the robotic arm in three dimensions. We compare OCN with two baseline methods: OMPN and MoE.
Figure shows the learning curve of three different fine-tuning tasks. We find that OCN consistently outperforms baseline methods. It's worth noting that, although OCN is designed to learn a discrete transition between subtasks, is still capable of providing good solution for continuous settings.
8

---

Under review as a conference paper at ICLR 2022
4.3
MODEL ANALYSIS
Appendix A.6 shows several trajectories produced by OCN, including imitation learning trajectories and reinforcement learning trajectories. As shown in Figure 10 and 11, OCN can accurately segment the demonstration into different subtasks and unambiguously associate options with subtasks. At the reinforcement learning phase, the discovered options are reused to solve their assigned tasks when coupled with new controller to solve a long horizon task.
Visualization
F1_ Fo_T
Align_Suue Succ
NMI
1.00
0.95
0.8
6.6
0.75
0.2
0.70
0.0
1
Million Frames
Million Frames
Million Frames
OCN
OCN,T=1
DXPN
Comparison of unsupervised trajectory parsing results during the imitation phase with OMPN (Lu et al., 2021). The F1 score with tolerance (Left) and Task Alignment (Center) show the quality of learned task boundaries. The normalized mutual information (Right) between the emerged option selection P and the ground-truth shows that OCN learns associate each option one subtask. T=1 means that the temperature term in the controller is removed.
Figure 7:
Quantitative Analysis Figure 7 shows the perfor- mances of parsing and option-subtask correlation during the imitation phase. We find that OCN can converge faster and achieve better parsing perfor- mance than the OMPN model. The NMI figure in Figure 7 shows that, during the imitation phase, randomly initialized options slowly converged to model different subtasks. At the end of imitation, OCN shows strong alignment between options and subtasks. In 4 out of 5 runs, OCN actually achieves NMI=1, which means that the alignment between option and subtask perfect. On the other hand, if remove the temperature term (i.e. T'= = 1) in the controller, the NMI drops significantly. This result suggests that the fast and slow learning schema important for the model learn the correct alignment between options and subtasks. Furthermore, Table shows the success rate of using each option solve each subtask. We find that there is one-to-one correspondence between subtasks and learned options.
Table I: The success rate of each option when testing on different subtasks.
subtask
A
C
D
Option
1
0.96
0.07
0.03
2
0.00
0.02
0.95
3
0.01
0.98
0.01
In the Appendix A.5, we show that OCN is not sensitive to the number of options N, as long as it's larger than the intrinsic number of skills in demonstrations. If N too small, then the performance of skill transfer may decrease. So it's always recommended to choose larger N.
CONCLUSION
In this paper, we proposed novel framework: Coiito-Cn (CCN) It is composed of a controller and set of options. After training on a set of demonstrations, OCN can automatically discover the temporal hierarchical structure of training tasks and assign different options to solve different subtasks in the hierarchy. Experiment results show that our model can effectively associate subtasks with options. And a newly initialized controller can leverage previously learned options to solve complicated long horizon tasks via interaction with the environment. In this finetuning process, OCN shows strong combinatorial generalization. It outperforms previous baseline methods large margin. Overall, this method provides a simple and effective framework for hierarchical imitation learning and reinforcement learning in discrete space.

---

Under review as a conference paper at ICLR 2022
REFERENCES
Joshua Achiam, Harrison Edwards, Dario Amodei, and Pieter Abbeel. Variational option discovery algorithms. arXiv preprint arXiv: 10X.x:880.110290 2018. 3
Jacob Andreas, Dan Klein, and Sergey Levine. Modular multitask reinforcement learning with policy sketches. In International Conference on Machine Learning, pp. 166-175. PMLR, 2017. 2, 3,
Pierre-Lue luc Bacon, Jean Harb, and Doina Precup. The option-critic architecture. In Proceedings of the AAAI Conference Artificial Intelligence, 2017. 2
Misha Denil, Sergio Gómez Colmenarejo, Serkan Cabi. David Saxton, and Nando de Freitas. Pro- 2017.
grammable agents. arXiv preprint
Thomas Dietterich. Hierarchical reinforcement learning with the maxq value function decomposi- tion. Journal of artificial intelligence research, 13:227-303, 2000. 2
Benjamin Eysenbach, Abhishek Gupta, Julian Ibarz, and Sergey Levine. Diversity all you need: Learning skills without reward function. In International Conference Learning Representations, 2018. 2,3
Roy Fox, Sanjay Krishnan, Ion Stoica, and Ken Goldberg. Multi-level discovery of deep options.
arXiv preprint arXiv: 1703.08294, 2017. 3
Roy Fox, Richard Shin, Sanjay Krishnan, Ken Goldberg, Dawn Song, and Ion Stoica. Parametrized hierarchical procedures for neural programming. ICLR, 2018.
Abhishek Gupta, Vikash Kumar, Corey Lynch, Sergey Levine, and Karol Hausman. Relay policy learning: Solving long-horizon tasks via imitation and reinforcement learning. arXiv preprint 1910. 19966, 2019. 1,
Peter Henderson, Wei-Di Chang, Pierre-Luc Bacon, David Meger, Joelle Pineau, and Doina Precup. Optiongan: Learning joint reward-policy options using generative adversarial inverse reinforcement learning. In Proceedings the AAAI Conference on Artificial Intelligence, volume 32, 2018. 3,
Yiding Jiang, Shixiang Gu, Kevin Murphy, and Chelsea Finn. Language as an abstraction for hierarchical deep reinforcement learning. arXiv preprint arXiv:1906.0777 2019. 1.3
Leslie Pack Kaelbling. Learning to achieve goals. In IJCAI, pp. 1094-1099. Citeseer, 1993.
Diederik P Kingma and Max Welling, Auto-encoding variational bayes. arXiv:1312.61 14, 2013. 1,3
arXiv preprint
Thomas Kipf, Yujia Li, Hanjun Dai, Vinicius Zambaldi, Alvaro Sanchez-Gonzalez, Edward Grefen- stette, Pushmeet Kohli, and Peter Battaglia. Compile: Compositional imitation learning and execution. In International Conference on Machine Learning, pp. 1418-3428, PMLR, 2019, 1, 2, 3,
Sanjay Krishnan, Roy Fox, Ion Stoica, and Ken Goldberg. Ddco: options for robot learning from demonstrations. In Conference on Robot Learning, pp. 418 437. PMLR, 2017. 3
Discovery of deep continuous
Hoang Le, Nan Jiang, Alekh Agarwal, Miroslav Dudík, Yisong Yue, and Hal Daumé. Hierarchical imitation and reinforcement learning. In International Conference on Machine Learning, pp. 2917-2926. PMLR. 2018.
Andrew Levy, Robert Platt, and Kate Saenko. Hierarchical reinforcement learning with hindsight. arXiv preprini arXiv: 1805.08/85. 2018. 1,
Yuchen Lu, Yikang Shen, Siyuan Zhou, Aaron Courville, Joshua B. Tenenbaum, and Chuang Gan. Learning task decomposition with ordered memory policy network. In International Conference on Learning Representations, 2021. URL https://openzeetee net net./orum? i td=vcopnw77bc. 1,3, 6,9
10

---

Under review as a conference paper at ICLR 2022
Corey Lynch, Mohi Khansari, Ted Xiao, Vikash Kumar, Jonathan Tompson, Sergey Levine, and Pierre Sermanet. Learning latent plans from play. In Conference on Robot Learning, pp. 1113-1132. PMLR, 2020. 3
Kanika Madan, Rosemary Nan Ke, Anirudh Goyal, Bernhard Bernhard Schölkopf, and Yoshua Ben- gio. Fast and slow learning of recurrent independent mechanisms. arXiv preprinl arXiv:2 105.08870 2021. 4
Ofir Nachum, Shixiang Gu, Honglak Lee, and Sergey Levine. Data-efficient hierarchical reinforce- ment learning. In Proceedings of the 32nd International Conference Neural Information Processing Systems, pp. 3307-3317, 2018.
Jun Nakanishi, Jun Morimoto, Gen Endo, Gordon Cheng, Stefan Schaal, and Mitsuo Kawato. Learning from demonstration and adaptation of biped locomotion. Robotics and autonomous systems, 47(2-3):79-91, 2004.
Ronald Parr and Stuart Russell. Reinforcement learning with hierarchies of machines. In Advances in neural information processing systems, pp. 1043-1049, 1998. 1
Alexander Pashevich, Danijar Hafner, James Davidson, Rahul Sukthankar, and Cordelia Schmid. Modulated policy hierarchies. arXiv preprint arXiv:1812.00025, 2018. 2
Karl Pertsch, Youngwoon Lee, and Joseph Lim. Accelerating reinforcement learning with learned skill priors. arXiv preprint arXiv:2010. 1944, 2020. 3
Tom Schaul, Daniel Horgan, Karol Gregor, and David Silver. Universal value function approximators.
In International conference on machine learning, pp. 1312-1320. PMLR, 2015.
John Schulman, Filip Wolski, Prafulla Dhariwal, Alec Radford, and Oleg Klimov. Proximal policy optimization algorithms. arXiv preprint arXiv:1707.06347 2017. 6
Mohit Sharma, Arjun Sharma, Nicholas Rhinehart, and Kris M Kitani. Directed-info gail: Learning hierarchical policies from unsegmented demonstrations using directed information. In International Conference Learning Representations, 2018.
Kyriacos Shiarlis, Markus Wulfmeier, Sasha Salter, Shimon Whiteson, and Ingmar Posner. Taco: Learning task decomposition via temporal alignment for control. In International Conference on Machine Learning, pp. 4654 4663, 2018. 3,
Alec Solway, Carlos Diuk, Natalia Córdova, Debbie Yee, Andrew Barto, Yael Niv, and Matthew M Botvinick. Optimal behavioral hierarchy. PLOS Comput Biol, 10(8)e1003779) 2014. 3
Richard S Sutton, Doina Precup, and Satinder Singh. Between mdps and semi-mdps: A framework for temporal abstraction in reinforcement learning. Artificial intelligence, 112(1-2):811-111 1999. 2,3
Alexander Sasha Vezhnevets, Simon Osindero, Tom Schaul, Nicolas Heess, Max Jaderberg, David Silver, and Koray Kavukcuoglu. Feudal networks for hierarchical reinforcement learning, arXiv preprint arXiv: 01 161, 2017.
Qiang Yang, Yang Liu, Tianjian Chen, and Yongxin Tong. Federated machine learning: Concept and applications. ACM Transactions Intelligent Systems and Technology (TIST), 10(2):1-19, 2019.

---

Under review as a conference paper at ICLR 2022
A APPENDIX
A.1 REINFORCEMENT LEARNING ALGORITIMM
Algorithm 1: PPO, Adapt OCN to a new task
Initialize controller c;
Freeze all options (o1...Nv} for iterations= 1,2.. do for actor=1,2.... do
for step 1 do
p c 1f
1 = sample(p
Rollout o, until sample(es) = 1;
end
Compute advantage estimates Å Ai...., Ar;
end
Optimize surrogate L wrt with K epochs and minibatch size B;
end
A.2 IMPLEMENTATION DETAILS
We train all imitation learning methods by utilizing behaviour cloning with batch size of 512 and a learning rate of 0.001. For each task, sample 6000 demonstrations and split 80% for training and 20% for validation. For reinforcement learning, we use PPO algorithm with a batch size of 1024 and learning rate of 0.0003. We use Adam optimizer and a linear schedule to adjust the learning rate. The hidden state size dhid of OCN and baselines is 128. The depth MLP of cell network is 2. The temperature for controller's softmax function 10. The hyparameters used in IL and RL are listed in table 2 and 3. We perform imitation learning and reinforcement learning phase with Tesla V100 GPU.
Table 2: Imitation Learning Parameters
Table 3: Reinforcement Learning Parameters
Hyparameter
Value
Hyparameter
Value
batch size learning rate
512
batch size
256
0.001
learning rate
0.0003
train episodes
1500
hidden size
128
hidden size
128
entropy 0.001
optimizer
Adam
ppo epoch
temperature for Softmax
10
gamma 0.97
optimizer
Adam
A.3 BASELINES
complLE We modify the encoder and decoder complLE so that the model can adapt to the observation of our environment. We use the discrete latent as the original paper describes. We re-initialize the encoder to predict latent for the new tasks and freeze the decoder which predicts the actions in RL.
MoE This baseline has a similar architecture with OCN, but it doesn't 't have a terminate signal and has to predict options distribution and select options at each step. We pretrain MoE with controller and options in the imitation phase and re-initialize new controller for the new tasks while freezing options as we do with OCN.
We directly use this baseline in the imitation phase. However, since OMPN has strong connections(e.g, bottom-up and top-down recurrence) between the higher-level model and lower-level model, we don't 't re-initialize the higher-level model in RL.
OMPN
12

---

Under review as a conference paper at ICLR 2022
A.4 TASK ALIGNMENT EVALUATION
We three metrics evaluate the performance of parsing and option-subtask correlation: a) task align accuracy, b) F1 scores with tolerance and c) normalized mutual information scores(NMI). The first metrics mutual information score between two clusterings, which are subtasks and options in our experiment. We give the formulation of NMI between two clusterings U, V as:
the same as those defined in OMPN. The NMI is normalization of the
MT(U, V) mean (U), H())
NMI(U,V)
where MI(U, V) the mutual information and I/ the entropy. The values of N NM/ is from 1. The higher values mean the higher correlation between subtasks and options.
A.5 HYPERPARAMETERR, ANALYSIS
Our model does not require the assumption about the number of skills. We analyze the effect of the number of options K. As shown in Figure 8, when K is larger than or equal to the number of skills, which is in this experiment, our model basically remains similar results three metrics: Align Acc, F1 1ol1, and NMI and achieve almost 1. When K = 2. which means K is smaller than the number of skills, one of the options must execute two different skills, which contrary to our assumption and only achieves 0.4 at NMT. We also compare the prediction accuracy of the actions and the returns in Figure Our performance isn't influenced by the number of skills when K is larger than the number of skills.
F1 Toll
Align Acc
NMI
0.90
0.8
0.80
0.80
0.75
6.2
0.65
0.70
Million Frames
Million Frames
Million Frames
K-2
K=3
K=4
K=5
Figure 8: Comparison of parsing results during different K F1 scores with tolerance, task align accuracy and NMI.
Acc
Returns
2,00
1.75
1.50
0.7
1.25
0.6
0.5
0.75
0.4
0.50
0.25
0.3
0.00
Million Frames
Million Frames
K=2
K=3
K=4
K=5
Figure 9: Comparison of prediction accuracy of actions and the returns during different K
13

---

Under review as a conference paper at ICLR 2022
Iron
Way Grass
Gold
Wood
Agent
switch
option,
option;
optcon,
action
Iron
Way Grass
Gold
Wood
Agent
switch
optios; optiong option, action enr
Teee1
Figure 10: Visualizations of different tasks in S1. Different colored lines correspond to different subtasks.
Iron
Vay Grass
Gold
Wood
Agent
sattol
optios
optios,
aptions
aclion
mp
84
DC
AC3D
Figure 11: Visualization of a task in S2.
A.6 VISUALIZATION
We show more visualization results in Figure 10 and 11. swit eh indicates what time the previous option ends and switches to the new option. The options distribution is computed with pç. In Figure 10, the model trained on different tasks to learn skills(A,C,DD which can be reused in the new lasks(ADC.CAD), In Figure I, there are two separate OCN models trained on two disjoints tasks sets AB, BE..BA,,,DDC DD,DC). The four options can obtained and reused solve new task(.
14


Reviews:
[{"review_id": "UZ0S-wUVswj", "reviewer": "Reviewer_XgEE", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "\nStrengths:\n- Although this work extends the paradigm of imitation-finetune, its idea is still novel by freezing the options and fine-tuning the controller.\n- The topic is timely important, which is related to transfer learning and multi-tasking. \n- The paper is well-organized and its method is sound to me. Besides, the related work is quite clear and includes the relevant literature.\n\n\nWeakness:\n- There lacks the clarity. For example, why \"positive rewards at the beginning of training\" is a problem in HRL? what are structured exploration and unstructured exploration? \"During the execution of an option, if probability $e_{i,t}$ is 0, the option will keep executing\", what if the task is really hard and $e_{i,t}$ keeps 0, will this step into an infinite loop? According to Figure 1, Figure 3(e), and Table 1, does that mean each option corresponds to a specific task? If this is the case, there lacks analysis or discussion on the scalability and generalizability.\n- This work primarily offers an empirical contribution. However, the empirical evaluation is still weak. Regarding the transferability demonstrated in the experiment, the \"newly\" task is a similar task in the same domain, which is still limited. \n\nMinor:\n- typo: \"They encoder skills\" -> \"They encode skills\"\n", "summary_of_the_paper": "\nThis work targets the issues in learning from demonstration, in order to achieve the ability of skill reuse/ transferability and improve the sample efficiency. To address the issues, the authors propose an option-control network (OCN), including a high-level controller and a pool of low-level options, which is an \"imitation-finetune\" paradigm as well. Finally, the proposed method is evaluated on two domains, Craft (discrete action space) and Dial (continuous action space), with results demonstrating its effectiveness.", "summary_of_the_review": "I prefer to weak reject ( marginally below the acceptance threshold) the paper.\n- The clarity is an issue that makes it a little bit hard to follow.\n- The empirical evaluation is weak so that the transferability looks limited. The scalability and generalizability should be discussed in the paper as well.", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}, {"review_id": "R9Xvl1tmi2R", "reviewer": "Reviewer_2XVq", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "The overall ideas of task decomposition and using information from demonstrations to accelerate RL are important and this paper proposes a new architecture to address these ideas. It is overall well written and easy to follow. However, the particular network is quite similar to existing implementations and has contributions focused on an aspect that could fall under implementation details in other papers. A further big challenge is the limited evaluation and a missing connection to much of related work. In its current form, the submission is well worth discussion but fits more in the context of a workshop.\n\nRegarding the evaluation it is unclear why the authors do not compare to using options from known approaches like DDO (cited in paper), Option Critic [1] or HO2 [2] (where the latter two can naturally use BC pretrained options as it just requires a change of the initial parameters). The evaluation uses variations of two domains which are not compared against methods from the papers which propose them. In addition, one of the baselines, OMPN, uses an adaptation of the Craft domain in the paper but shows a very different level of performance which suggests that the variation of the domain is different for the submitted paper. The original reason for investigating the papers was that all baselines flatline at surprisingly low performance (most prominent in the Craft domain).\n\nOn the more positive side: Section 4.1 includes an example from transferring options from multiple previous experiments which provides an interesting perspective. A future iteration could benefit from more focus on this generally under-investigated idea. But the results for baselines are again surprisingly low.\n\n\nMinor:\nSection 1: the mentioned exploration problem is more a property of specific domains rather than algorithms \n‘High level controller is updated less frequently’ is likely supposed to mean acting. Updating can be confused with learning.\n‘each option does not correspond to a meaningful subtask’ It is unclear what this is supposed to mean. What is a ‘meaningful subtask’? And why is it good for options to not correspond to one?\n\n[1] Bacon, Pierre-Luc, Jean Harb, and Doina Precup. \"The option-critic architecture.\" Proceedings of the AAAI Conference on Artificial Intelligence. Vol. 31. No. 1. 2017.\n[2] Wulfmeier, Markus, et al. \"Data-efficient hindsight off-policy option learning.\" International Conference on Machine Learning. PMLR, 2021.\n\n", "summary_of_the_paper": "The paper proposes a specific neural network architecture for option learning with recurrency on both option and high-level controller. The final action outputs are determined from a mixture of experts of all options. The approach learns options offline from demonstrations (behavioural cloning) and combines them online by learning a new high-level controller via RL (PPO) while using frozen options. The submission uses variations of 2 domains from prior work for evaluation and shows good performance.\n", "summary_of_the_review": "Submission proposes a specific architecture for option learning. An aspect that in other work might fall under implementation details. The work has some interesting experiments on combining options from multiple experiments, but the evaluation and connection to previous work is very limited. ", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "1: The contributions are neither significant nor novel.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}, {"review_id": "P7aHX1oW8KX", "reviewer": "Reviewer_TfQH", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 5, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "Strengths:\n\nThe model proposed is interesting and builds on the recently proposed OMPN model as well as prior work [1, 2, 3, 4, 5] on incorporating inductive biases in RNNs to model temporal hierarchy. Baseline models compared against (CompILE, OMPN, MoE) are strong, appropriate and relevant. All models have been evaluated on the benchmark datasets such as Craft and Dial as in prior work. The pretrained options learned by OCN during the imitation phase lead to good performance gains when being reused later by the controller network. Implementation details for all models (including baselines) have been described well in the Appendix. \n\n\n\nWeaknesses:\n\nAssuming I understood the generation of training datasets in experiments S1 and S2 correctly. The training datasets (AC, CD, DA) or (AB, BA) or (CD, DC) generated for the imitation learning phase in S1 & S2 are overly simplistic. The associated decomposition problem involves just finding 1 boundary point to demarcate 2 skill primitives in an episode in the datasets for S1 and S2 experiments. It would be unreasonable to assume this training setup would directly transfer to any general/complex real-world offline datasets containing longer sequences involving execution of many options. Have I missed something in my interpretation? Could the authors clarify this point and/or provide some justification as to why they would expect their model to show similar performance in the more general case?\n\nIt would be beneficial to compare and discuss the OCN model in the context of prior work on RNN variants that model temporal hierarchy (Clockwork RNN[1], HM-RNN[2]) using various adaptations to hidden-state updates. Specifically, the HM-RNN update rule uses similar operators such as copy, update (and flush) to allow for slower updates to memory cells deeper in the hierarchy. Therefore it is important to delineate the differences in the update rules employed in HM-RNN [2] and the proposed OCN model to evaluate the novelty/originality of the contribution. Further, earlier work in literature on inducing hierarchical temporal structure in RNNs [3, 4, 5, 6] has not been referenced and compared/discussed in the related work. \n\nCan the hierarchical update rule (control flow between controller and option networks) proposed in the OCN model not be implemented/computed by an appropriately sized OMPN model? Or perhaps a HM-RNN [2] model? \n\n\n\nResults:\n\nI found it rather surprising to see that the OMPN seems to struggle to learn good decompositions of primitive skills in S1 which can be composed (reused) by the controller later. Several experiments show CompILE and MoE baselines outperforming the OMPN model quite significantly (Figure 4 and Figure 6). This is very surprising to me since OMPN essentially has a strong inductive bias for modelling temporal hierarchy which is clearly beneficial for these tasks and very similar to the proposed OCN model. Could the authors offer an explanation for why this is so? \n\nIt would be interesting to quantify how redundant (i.e. number of options learned for the same underlying skill) the learned options are and amount of reuse by the controller as we use more noisier/complex demonstration datasets for the imitation learning phase. \n\nThe authors claim that OCN does not require specifying the exact number of segments in an episode and only need to specify a safe upper bound on it. What do the “extra” slots capture/model? Are they left unused or does it lead to a slightly over-segmented decomposition of the episode? \n\n\n\nWriting/Presentation:\n\nIn general, the paper contains several typos and ambiguous or unclear phrasing in several sections. The papers’ readability and clarity would greatly benefit from a significant revision in this regard. I have highlighted some of the typos/inconsistent math notation/grammatical errors below:\n\n“model each skill with separate options” -> does this imply that there are several options used to model a single latent skill? \n\n“is a hallmark in human intelligence” -> “is a hallmark of human intelligence”. Further, the authors do not provide a reference to validate the claim in this sentence. \n\nThe authors introduce It is unclear whether this is a new problem introduced by the authors or an interpretation of an existing problem. It would help to cite relevant references that define/study this “exploration  problem” in pure HRL methods. \n\n“.. limits the practical values of these approaches” -> “ … limits the practical utility of these approaches”\n\n“ popularity of neural nets” -> “popularity of neural networks”\n\n“However assuming access to an environment in the pertain phase might be infeasible in many tasks” -> missing citation to relevant environments?\n\n“and performs imitation learning” -> “and perform imitation learning” \n\n“our work focused on” -> “our work focuses on”\n\n“performs IRL on the demonstration” -> “performs IRL on the demonstration data”; what does the abbreviation IRL stand for?\n\n“extracts meaning segments” -> “extracts meaningful segments”\n\n“temporal hierarchical structure” -> “hierarchical temporal structure” \n\n“Following the fast and slow learning idea proposed in Madan et. al 2021“ Please improve the citation, some suggestions for the same [6, 7]\n\n\nSome of the math notation in the “Methodology” section is confusing, difficult (unintuitive) to follow for the reader and also not consistent throughout the text. Using boldface for functions (i.e. RNN updates, example $o_i (x_t, h_{t-1}), c (x_t, h_{t-1})$) is confusing as a small letters with bold font is being used to denote vectors as well. Typo in equation 3? Should it be $\\hat{h_{t-1}}$ and not $h_{t-1}$? In Equations (3,4,5,6) several vectors with no bold font. It is also very confusing to have both $h_t$ and $\\hat{h_t}$ in the cell network equations. Further, the indexing notation (i, t) for activations of option models (ex: $h_{i, t}^{o}$) is also very confusing and reduces the clarity and readability. Please use another indexing scheme for activations of option models like $h_t^{o_i}$ or something similar. It is also incredibly hard to mentally keep track of 3 variants of each activation vector (for ex: $\\hat{p}$, $p’$ and $p$) throughout the description of the algorithm and their roles. Please use a more intuitive alternative to make the reading experience better and improve the clarity. The objective function to minimise (equation 18) is written in a pseudocode-like manner. Please remain consistent and define the objective function using a mathematical expression to avoid ambiguity.   \n\n\n[1] Koutnik et. al, “A Clockwork RNN.”, ICML 2014.\n\n[2] Chung et. al, “Hierarchical Multiscale Recurrent Neural Networks”, ICLR 2017. \n\n[3] Schmidhuber et. al, \"Learning Complex, Extended Sequences Using the Principle of History Compression,\", Neural Computation 1992,\n\n[4] Mozer et. al,  \"Induction of multiscale temporal structure.\", NIPS 1992.\n\n[5] El Hihi et. al, “Hierarchical recurrent neural networks for long-term dependencies.”, NIPS 1995. \n\n[6] Jaynes, Edwin T. (1957). Information Theory and Statistical Mechanics. II. Physical Review 108 (2):171.\n\n[7] Boltzmann, Ludwig,  \"Studien über das Gleichgewicht der lebendigen Kraft zwischen bewegten materiellen Punkten\", Wiener Berichte. 58: 517–560, 1868.\n", "summary_of_the_paper": "The paper proposes a new model called Option-Controller Network with requisite inductive biases to model temporal hierarchy. This model is used to learn temporal abstractions and the control policy in the space of options using demonstration data via imitation learning. The performance on discrete action (Craft) and continuous action (Dial) environments are promising compared to existing baselines such as OMPN, CompILE and MoE. ", "summary_of_the_review": "The proposed OCN model is an interesting extension to the recent OMPN. However, my main issue is that the demonstration dataset generation process has drastically reduced the full sequence decomposition task to an overly simplistic case. The writing can be improved significantly as suggested above to improve clarity of the proposed ideas. Further, some closely related prior work [2] on modelling temporal hierarchy in RNNs using similar ideas have not been compared/discussed and many others not referenced [1, 3, 4, 5].\n", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}, {"review_id": "9fWGwpYq_H1", "reviewer": "Reviewer_RMbd", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 5, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "### Strengths\n\n- The proposed approach is intuitive and easy to implement.\n- The paper is easy to follow and clearly written.\n- The experimental results in CRAFT show significant improvement over prior works.\n\n\n### Weaknesses\n\n- Comparison to prior latent skill learning approaches is required to understand the contribution of the proposed option extraction method. These approaches already have shown impressive results on complex robotic manipulation environments. It is unclear whether the proposed method can achieve better performance compared to these approaches.\n\t- Pertsch et al. Accelerating Reinforcement Learning with Learned Skill Priors, CoRL 2020\n\t- Shankar et al. Learning Robot Skills with Temporal Variational Inference, ICML 2020\n- In the approach section, \"OCN can be easily expanded to a multi-level model.\" does not seem true. Most hierarchical approaches so far have failed to learn more than two-level hierarchy, which implies challenges in extending to multi-level hierarchy.\n- When an option is continued, the hidden state of the controller is copied without any update. This results in missing task-specific information during the option execution. Given the fact that the proposed method relies heavily on recurrent models, this missing temporal information could make the agent decision not optimal.\n- There are discrepancies between pre-training and training OCN models, where the pre-training stage computes an action and termination signal from all options with soft weighting while the RL training stage chooses a single option at a time. Will this affect RL training badly? The options learn to be mixed together during skill learning without clear skill decomposition. But, when using these options, they need to work alone and have a clear skill boundary.\n- In the Dial task, the proposed method works better than the baseline methods. However, the variance in Figure 6 seems very high and the performance does not look significant. How many random seeds are used for the experiments? It definitely requires more random seeds to reach a meaningful conclusion. Another question from the learning curves is about the discrepancy between dense and sparse reward settings. OCN seems to succeed 20-30% in the sparse reward setting, but in the dense reward setting, it's not even close to solving 2 subtasks out of 3 subtasks. Moreover, this poor performance makes the scalability of the proposed method less convincing. Can OCN even learn the tasks it saw during pre-training (such as [1, 2])?\n- In the Dial task, the experiments focus on generalizing to new tasks with a longer sequence of subtasks. One interesting experiment would be learning skills from a large number of tasks (e.g. three digits pressing tasks with all numbers) and then learning a controller for a holdout task.\n\n\n\n### Minor comments\n\n- \"Option-Control Network\" is used instead of \"Option-Controller Network\" in the introduction.\n- \"option framework\" -> \"options framework\"\n- \"furthered combined\" -> \"further combined\"\n- In equation (13), LHS should be $\\hat{p}^a_{t}$, not $\\hat{p}^a_{i, t}$.\n- What happen if the chosen option outputs \"done\"?\n  ", "summary_of_the_paper": "This paper proposes to extract options from a dataset using an options framework with a recurrent controller and multiple recurrent option networks. During pre-training (i.e. skill extraction), the actions from the framework are computed by the weighted sum over options, which can be end-to-end differentiable, and trained using behavioral cloning. Once all options are extracted from the dataset, a new controller can be trained to solve a new task with the learned and frozen options via RL. The experimental results on the discrete CRAFT environment are impressive, outperforming all baselines. The results on the continuous DIAL environment show improved results over the baseline methods but the difference is marginal.", "summary_of_the_review": "This paper tackles an important problem of reusable skill (option) extraction from a large multi-task dataset. The proposed method is simple and works well in the discrete task, CRAFT. However, the training process is not intuitive (soft selection during pre-training and hard selection during RL). The experimental results on the continuous domain, DIAL, are noisy and show only marginal improvement over prior work. Moreover, comparisons to continuous latent skill approaches are required to show the technical contribution of the proposed method.", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}]

Author rebuttal:

Meta-review: Description of paper content:

The paper describes a technique to learn option policies using behavioral cloning and then recombine them using a high-level controller trained by RL. The underlying options are frozen. The method is tested in two published environments: a discrete grid world environment and a continuous action space robot. It is compared to three baselines.

Summary of paper discussion:

All reviewers moved to reject based on a lack of novelty and a lack of significant empirical results. No rebuttals were provided.
Decision: Reject