Intrinsically Motivated Learning in Natural and Artificial Systems

This chapter introduces the field of intrinsically motivated learning systems and illustrates the content, objectives, and organisation of the book. The chapter first expands the concept of intrinsic motivations, then introduces a taxonomy of three classes of intrinsic-motivation mechanisms (based on predictors, on novelty detection, and on competence), and finally introduces and reviews the various contributions of the book. The contributions are organised into six parts. The contributions of the first part provide general overviews on the concept of intrinsic motivations, the possible mechanisms that may implement them, and the functions that they can play. The contributions of the second, third, and fourth parts focus on the three classes of the aforementioned intrinsic-motivation mechanisms. The contributions of the fifth part discuss mechanisms that are complementary to intrinsic motivations. The contributions of the sixth part introduce tools and experimental paradigms that can be used to investigate intrinsic motivations. 1 Intrinsically Motivated Learning The capacity to learn autonomously and in a cumulative fashion is one of the hallmarks of intelligence. Higher mammals, especially when young, engage in a number of exploratory activities with the environment (e.g. think about children at play; von Hofsten 2007). These activities are not directed to pursue goals that are directly relevant for the survival and reproduction of the organism, but are driven by intrinsic motivations, that is, motivations such as curiosity, interest in novel stimuli or surprising events, and interest in learning new behaviours. The adaptive value of intrinsically motivated activities is that they allow the cumulative acquisition G. Baldassarre ( ) M. Mirolli Institute of Cognitive Sciences and Technologies, CNR, Roma, Italy e-mail: gianluca.baldassarre@istc.cnr.it; marco.mirolli@istc.cnr.it G. Baldassarre and M. Mirolli (eds.), Intrinsically Motivated Learning in Natural and Artificial Systems, DOI 10.1007/978-3-642-32375-1 1,  1 2 G. Baldassarre and M. Mirolli of knowledge and skills that can be later used (e.g. in adulthood) to accomplish fitness enhancing goals (Baldassarre 2011; Singh et al. 2010). Intrinsic motivations continue to operate also during adulthood, and indeed in humans they underlie lifelong learning and other typically human activities such as art and scientific discovery (Schmidhuber 2010). Interestingly, intrinsic motivations are also the basis of processes that strongly affect human well-being, such as the sense of competence, self-determination, and self-esteem (Ryan and Deci 2000). The concept of intrinsic motivation was introduced in the 1950s in animal psychology (Harlow 1950) and was further elaborated in human psychology (Deci 1975; Deci and Ryan 1985), where it is now widely applied, especially in the fields of educational psychology (e.g. Kohn 1993), developmental psychology (e.g. Harter 1981), and organisational psychology (Houkes et al. 2001). Furthermore, recent neuroscientific research on the learning signals used by real brains, in particular with respect to the learning of novel actions (e.g. Redgrave and Gurney 2006) and the memorisation of novel information (e.g. Lisman and Grace 2005), is starting to uncover the basic brain mechanisms underlying intrinsic motivations. Recently, the concept of intrinsic motivation was introduced in machine learning and developmental robotics (Barto et al. 2004; Oudeyer et al. 2007; Schembri et al. 2007c) as a means for developing artificial systems that can autonomously learn several different skills. The idea is that intelligent machines and robots could autonomously acquire skills and knowledge under the guidance of intrinsic motivations, and later exploit such knowledge and skills so to accomplish the tasks that are useful for the user in a more efficient and faster way than if they had to acquire them from scratch. This possibility would clearly enhance the utility and autonomy of intelligent artificial systems. For this reason, the investigation of intrinsically motivated learning systems is becoming a central research topic in machine learning and autonomous robotics. Notwithstanding the importance of this emerging field, up to now there has not been any attempt to draw a unifying picture. The present book provides a general and highly interdisciplinary overview of recent research on intrinsic motivations. While the principal target of this book is the community that tries to synthesise autonomous learning artificial systems, either for applicative purposes or as a means to understand real organisms, this book contains also a number of contributions from researchers in the empirical sciences, because we think that a close dialog between the natural and the artificial sciences is the most fruitful way to address the problems related to intrinsically motivated learning. As a result, this book contains contributions from several different fields of researchers, including robotics (Hart and Grupen 2012; Natale et al. 2012; Nehmzow et al. 2012; Oudeyer et al. 2012), machine learning (Barto 2012; Schmidhuber 2012), computational modelling (Baldassarre and Mirolli 2012; Dayan 2012; Gurney et al. 2012; Merrick 2012; Mirolli and Baldassarre 2012), psychology (Schlesinger 2012; Stafford et al. 2012; Taffoni et al. 2012), and neuroscience (Otmakova et al. 2012; Redgrave et al.