Intrinsically Motivated and Interactive Reinforcement Learning: a Developmental Approach. (Apprentissage par Renforcement Intrinsèquement Motivé et Interactif: une approche développementale)

Reinforcement learning is a learning paradigm in which the agent maximizes the accumulation of a reward signal. It has met with a growing interest since its combination with deep learning, but today, it confronts a paradox known as the Moravec’s paradox: reproducing child-level sensorimotor abilities, like locomotion or object manipulation, is more intricate than learning to play abstract games like go or poker. Many recent evolutions of reinforcement learning aim to change this status quo, and take inspiration from human learning behaviors. In doing so, they focus on challenges close to the objectives of another child of the quest for Artificial Intelligence: developmental robotics. The latter aims at modelling and reproducing the development of natural cognitive processes in artificial agents, and can be a rich source of inspiration for reinforcement learning. On the other hand, deep reinforcement learning provides a framework to combine these processes, with strong representational power to tackle high-dimensional environments. Following these observations, bridges between reinforcement learning and developmental sciences are explored more and more, but there is still much to do. In this dissertation, we study the intersection of two crucial topics in developmental sciences, and their application to reinforcement learning: social learning and intrinsic motivation. Interactive and intrinsically motivated reinforcement learning have already been studied, separately, but with a focus on improving existing agents performance rather than learning in a developmental fashion. Instead, we address this second objective. Precisely, we propose several agents allowing to study how reinforcement learning, intrinsic motivations and simple forms of social learning can interact to produce learning behaviors that are sound from a developmental standpoint, and richer than those produced by each mechanism alone. Developmental studies show that interactive capabilities emerge incrementally through the progressive interaction of simple attentional and exploratory mechanisms. The first agent that we propose combines two such mechanisms: novelty search and gaze-following. We show that these mechanisms expectedly speed up the learning of a task defined by standard reward signals. More unexpectedly, the two mechanisms together lead the agent to task completion in absence of such rewards, albeit sub-optimally. The agent thus recreates a typical interaction between a learner and a tutor, where the latter guides the former’s attention towards objects that pique its curiosity, in order to make it discover new action possibilities. However, this preliminary work shows that the agent lacks a form of intentionality to learn fully without rewards from the environment, as humans do. We thus go on to describe two agents endowed with goal-directed behaviors. We use goal-conditioned reinforcement learning as the base mechanism for building goaldirected agents, and we study how to interface goal selection with the learning mechanism to automatically produce a sequence of learning situations of growing difficulty, also known as a curriculum. Our survey of the literature on automatic curriculum generation in reinforcement learning suggests that the limited results obtained are due to the poorly understood generalization properties of the agents. From this analysis, we propose a strategy for curriculum generation based on two components: the Learning Progress Hypothesis described in developmental sciences, which suggests that humans organize their learning situations into trajectories by maximizing their learning progress; and a task definition and parametrization that fosters inter-task transfer.