###### Abstract  
In open-ended continuous environments, robots need to learn multiple parameterised control tasks in hierarchical reinforcement learning. We hypothesise that the most complex tasks can be learned more easily by transferring knowledge from simpler tasks, and faster by adapting the complexity of the actions to the task. We propose a task-oriented representation of complex actions, called _procedures_, to learn online task relationships and unbounded sequences of action primitives to control the different observables of the environment. Combining both goal-babbling with imitation learning, and active learning with transfer of knowledge based on intrinsic motivation, our algorithm self-organises its learning process. It chooses at any given time a task to focus on; and what, how, when and from whom to transfer knowledge. We show with a simulation and a real industrial robot arm, in cross-task and cross-learner transfer settings, that task composition is key to tackle highly complex tasks. Task decomposition is also efficiently transferred across different embodied learners and by active imitation, where the robot requests just a small amount of demonstrations and the adequate type of information. The robot learns and exploits task dependencies so as to learn tasks of every complexity.  
curriculum learning; continual learning; hierarchical reinforcement learning; interactive reinforcement learning; imitation learning; multi-task learning; active imitation learning; hierarchical learning; intrinsic motivation +
Footnote †: journal:To solve this unbounded problem, the learning agent should be _starting small_ before trying to learn more complex tasks as theorised in [3]. Indeed, in multi-task learning problems, some tasks can be compositions of simpler tasks (which we call _'hierarchically organised tasks'_). This approach has been coined _'curriculum learning'_ in [4]. The idea of this approach is to use the knowledge acquired from simple tasks to solve more complex tasks or high level of hierarchy tasks, or in other words, to leverage transfer learning (TL) [5; 6; 7]. Uncovering the relationship between tasks is useful for transferring knowledge from a task to another. The insight behind TL is that generalization may occur not only within tasks, but also across tasks. This is relevant for compositional tasks. But how can the learning agent discover the decomposition of tasks and the relationship between tasks? Moreover, transfer of knowledge between tasks can also be completed by transfer of knowledge from teachers. Indeed, humans and many animals do not just learn a task by trial and error. Rather, they extract knowledge about how to approach a problem from watching other people performing a similar task. Behavioural psychology studies [8; 9] highlight the importance of social and instructed learning, _\"including learning about the consequences, sequential structure and hierarchical organisation of actions\"_[10]. Imitation is a mechanism for emerging representational capabilities [11]. How can imitation enable the task decomposition into subtasks, and which kind of information should be transferred from the teacher to the learning agent to enable effective hierarchical reinforcement learning? How robust is this transfer to correspondence problems? How can teachers avoid demonstrations that could correspond to behaviours the agent already masters or require pre-requisites the robot has not learned yet?  
This work addresses multi-task learning in open-ended environments by studying the role of transfer of knowledge across tasks with the hypothesis that some tasks are interrelated, and the role of transfer of knowledge from other learners or experts to determine how information is best transferred for hierarchical reinforcement learning: when, what and whom to imitate?