Bootstrapping Intrinsically Motivated Learning with Human Demonstrations

This paper studies the coupling of internally guided learning and social interaction, and more specifically the improvement owing to demonstrations of the learning by intrinsic motivation. We present Socially Guided Intrinsic Motivation by Demonstration (SGIM-D), an algorithm for learning in continuous, unbounded and non-preset environments. After introducing social learning and intrinsic motivation, we describe the design of our algorithm, before showing through a fishing experiment that SGIM-D efficiently combines the advantages of social learning and intrinsic motivation to gain a wide repertoire while being specialised in specific subspaces. I. APPROACHES FOR ADAPTIVE PERSONAL ROBOTS The promise of personal robots operating in human environments to interact with people on a daily basis points out the importance of adaptivity of the machine to its environment and users. The robot can no longer simply be all-programmed in advance by engineers, and reproduce only actions predesigned in factories. It needs to match its behaviour and learn new skills as the environment and users’ needs change. In order to learn an open-ended repertoire of skills, developmental robots, like animal or human infants, need to be endowed with task-independent mechanisms which push them to explore new activities and new situations [1], [2]. The set of skills that could be learnt is actually infinite, and can not be completely learnt within a life-time. Thus, deciding how to explore and what to learn becomes crucial. Exploration strategies, mechanisms and constraints in recent years can be classified into two broad interacting families: 1) socially guided exploration; 2) internally guided exploration and in particular intrinsically motivated exploration. A. Socially Guided Exploration In order to build a robot that can learn and adapt to human environment, the most straightforward way is probably to transfer knowledge about tasks or skills from a human into a machine. That is why several works incorporate human input to a machine learning process. Many prior systems are strongly dependent on human guidance, unable to learn in the absence of human interaction, such as in some examples of learning by demonstration [3]–[6] or learning by physical guidance [7]. In such systems, the learner scarcely explores on his own to learn tasks or skills beyond what it has observed with a human. Many prior works have given a human trainer control of the reinforcement learning reward [8], [9], provide advice [10], or tele-operate the agent during training [11]. However, the more dependent on the human the system, the more challenging learning from interactions with a human is, due to limitations like human patience, ambiguous human input, correspondence problems [12] etc. Increasing the learners autonomy from human guidance could address these limitations. This is the case of internally guided exploration methods. B. Intrinsically Motivated Exploration Intrinsic motivation, a particular example of internal mechanism for guiding exploration, has drawn a lot of attention recently, especially for open-ended cumulative learning of skills [1], [13]. The word intrinsic motivation was first used in psychology to describe the capability of humans to be attracted toward different activities for the pleasure that they experience intrinsically. These mechanisms have been shown crucial for humans to autonomously learn and discover new capabilities [14]–[16]. This inspired the creation of fully autonomous robots [17]–[22] with meta-exploration mechanisms monitoring the evolution of learning performances of the robot, in order to maximise informational gain, and with heuristics defining the notion of interest [23]–[25]. While driving an efficient progressive learning in numerous cases, most intrinsic motivation approaches address only partially the challenge of unlearnability and unboundedness [26]. Despite efforts in the case of continuous sensorimotor spaces, computing meaningful measures of interest still requires a sampling density which decreases the efficiency of those approaches as dimensionality grows. Even in bounded spaces, the measures of interest can be cast into a form of a nonstationary regression problem, which might face the curseof-dimensionality [27]. Thus, without additional mechanisms, the identification of learnable zones with knowledge or competence progress becomes inefficient in high-dimensions. The second limitation relates to unboundedness. Actually, whatever the measure of interest used, if it is only based on the evaluation of performances of predictive models or of skills, it is impossible to explore/sample inside all localities in a life time. Therefore, complementary developmental mechanisms need to constrain the growth of the size and complexity of practically explorable spaces, by introducing self-limits in the unbounded world and/or drive them rapidly toward learnable subspaces, such as motor synergies, morphological computation, maturational constraints as well as social guidance. C. Combining Internally Guided Exploration and Socially Guided Exploration Intrinsic motivation and socially guided learning are often studied separately in developmental robotics, and even in S. M. Nguyen, A. Baranes, P.-Y. Oudeyer (2011), Bootstrapping Intrinsically Motivated Learning with Human Demonstrations, in proceedings of the IEEE International Conference on Development and Learning. opposition to one another in psychology and educational theory. Indeed, many forms of socially guided learning can be seen as extrinsically driven learning. Yet, in the daily life of humans, the two strongly interact, and their combination could on the contrary push off the limitations we stated above. Social guidance can drive a learner into new intrinsically motivating spaces or activities which it may continue to explore alone and for their own sake, but might have discovered only due to social guidance. Robots may acquire new strategies for achieving those intrinsically motivated activities by observing others or by listening to their advice. Studies in robot learning by imitation and demonstration have already developed statistical inference mechanisms allowing the inference of new task constraints [4], [5], [7]. These techniques could be reused by intrinsically motivated learning architectures to efficiently expand the explored spaces. Inversely, as learning that depends highly on the teacher quickly shows limitations and would discourage the user from teaching to the robot, a need for autonomous exploration is needed. Integrating self-exploration to social learning methods could relieve the user from overly time-consuming teaching. For example, while self-exploration tends to result in a broader task repertoire of skills, guided-exploration with a human teacher tends to be more specialised, resulting in fewer tasks that are learnt faster. Combining both can thus bring out a system that acquires a wide range of knowledge which is necessary to scaffold future learning with a human teacher on specifically needed tasks. Initial work in this direction [28] and [29] proposes a symbolic representation of actions and environment for active learning, and stresses the importance of social dialogue through both the study of the human behaviour and transparency of the robot. The Socially Guided Exploration’s motivational drives, and social scaffolding from a human partner, bias behaviour to create learning opportunities for a hierarchical Reinforcement Learning mechanism. However, in this work, the representation of the continuous environment by the robot is discrete and the set up is a limited and preset world, with few primitive actions possible. We would like to address the learning in the case of an unbounded, non-preset and continuous environment. This paper introduces an algorithm to deal with such spaces, by merging socially guided exploration and intrinsic motivation, called Socially Guided Intrinsic Motivation (SGIM). The next section describes SGIM’s intrinsic motivation part before its social interaction part. Then, we present the fishing experiment and its results. II. INTRINSIC MOTIVATIONS : THE SAGG-RIAC ALGORITHM In this section we introduce Self-Adaptive Goal GenerationRobust Intelligent Adaptive Curiosity, an implementation of competence-based intrinsic motivations [30]. We chose this algorithm as the intrinsic motivation part of SGIM for its efficiency in learning a wide range of skills in high-dimensional space including both easy and unlearnable subparts. Moreover, its goal directedness allows bidirectional merging with socially guided methods based on feedback on either goal and/or means. Its ability to detect unreachable spaces also makes it suitable for unbounded spaces. A. Formalisation of the Problem Let us consider a robotic system whose configurations/states are described in both a state space X , and an operational/task space Y . For given configurations (x1, y1) ∈ X×Y , an action a ∈ A allows a transition towards the new states (x2, y2) ∈ X × Y . We define the action a as a parameterised dynamic motor primitive. While in classical reinforcement learning problems, a is usually defined as a sequence of micro-actions a = {a1, a2, . , an}, parameterised motor primitives consist of complex closed-loop dynamical policies which are actually temporally extended macro-actions, that include at the lowlevel long sequences of micro-actions, but have the advantage of being controlled at the high-level only through the setting of a few parameters. The association M : (x1, y1, a) 7→ (x2, y2) corresponds to a learning exemplar that will be memorised, and the goal of our system is to learn both the forward and inverse models of the mapping M . We can also describe the learning in terms of tasks, and consider y2 as a goal which the system reaches through the means a in a given context (x1, y1). In the following, both descriptions will be used interchangeably.