Emergence of Functional Hierarchy in a Multiple Timescales Recurrent Neural Network Model : A Humanoid Robot Experiment

It is generally thought that skilled behavior in human beings results from functional hierarchy of the motor control system, within which reusable motor primitives are flexibly integrated into various complex sensori-motor sequence patterns. The underlying neural mechanisms governing the way in which continuous sensori-motor flows are segmented into primitives, and the way in which series of primitives are integrated into complex sequential behavior, have however not yet been clarified. In earlier studies, this functional hierarchy has been realized through the use of explicit hierarchical structure, with local modules representing motor primitives in the lower level and a higher module representing sequences of primitives switched via additional mechanisms such as gate-selecting. Complex sensori-motor sequences, however, are not easily handled in such earlier models due to a conflict, induced by this explicitly separated modular structure, between generalization and segmentation. To address this issue, we propose a different type of neural network model. The current model neither makes use of separate local modules to represent primitives, nor introduces explicit hierarchical structure. Rather than forcing architectural hierarchy onto the system, functional hierarchy emerges through a form of self-organization that is based on two distinct types of neurons, each with different time properties ("multiple timescales"). Such multiple timescales lead to complex sensori-motor flows of skilled behavior being segmented into reusable primitives, and the primitives, in turn, are flexibly integrated Emergence of functional hierarchy 2/18 into novel sequences. In experiments, the proposed network model, coordinating the physical body of a humanoid robot through high-dimensional sensori-motor control, also successfully situated itself within a noisy environment. The idea proposed here of a functional hierarchy which self-organizes through multiple timescales in neural activity, in addition to contributing to further neurophysiological investigations of the motor control system, could also contribute to the study of various regions of the brain where similar mechanisms may be observed. In addition the extension of the current model to bind behavior and linguistic modalities are also introduced. Emergence of functional hierarchy 3/18 Introduction Functional hierarchy, defined broadly as the principle that complex entities may be segmented into simpler elements and that simple elements may be integrated into a complex entity, is a ubiquitous feature of information processing in biological neural systems [1-4]. The motor control system governing skilled behavior is a representative example of a system with functional hierarchy. Humans acquire a number of skilled behaviors through the experience of repeatedly carrying out the same movements. Certain components of such movements, through repetitive experiences, are segmented into reusable elements referred to as "primitives". In adapting to various situations, series of motor primitives are in turn also integrated into complex sequential behavior [5, 6]. The action of drinking a cup of coffee, for example, may be broken down into a combination of motor primitives such as the motion of reaching for a cup on the table, and the motion of grasping the cup and bringing it to one's mouth. Ideally, these motor primitives should be represented in generalized manner, in the sense that the representation should be adaptive for differences of locations and shapes of the cup. Primitives must also be flexible with respect to changes in the sequence of actions; for example, after grasping a cup, one sometimes brings the cup to one's mouth to drink, but one also sometimes takes the cup off the table to wash up. It is this adaptability (intra-primitive level) and flexibility (inter-primitive level) of primitives that allow humans to generate countless patterns of complex behavior. The observations of human reaching motions [7][8] and animal muscle movements [9][10] strongly suggest that the complex movements of animals are made up of flexible combinations of reusable movement elements, i.e. motor primitives. What is not yet clear, however, is what underlying neural mechanisms govern the segmentation of continuous sensori-motor flows into primitives, and how series of primitives are combined into complex sequential behavior. To address this issue, we propose a neural network model for describing the neural mechanisms of segmentation and integration in complex sensori-motor flows. In experiments, the proposed network model was tested through the interaction of a humanoid robot with a noisy environment, the robot requiring high-dimensional complex sensori-motor control. The robotics experiment is important when one considers the idea of the embodied mind by Varela [11], who explained that cognitive functions of neural systems emerge not only in the brain, but also in dynamic interactions between the physical body and the environment. Emergence of functional hierarchy 4/18 Figure1. Schematic drawings of (A) Local representation model and (B) Multiple timescale model. Previous studies exist on the computational modeling of functional hierarchy in sequences of motor primitives [12, 13]. In these studies, functional hierarchy is realized through the use of explicit hierarchical structure, with local modules representing motor primitives in the lower level, and a higher module representing the order of motor primitives switched via additional mechanisms such as gate-selection (Figure 1A). We refer to this type of model as the "local representation" model. There are a number of possible advantages of the local representation. First, the learning of one module would seem not to affect other modules. Second, based on this independence in the learning process, it would seem that increasing the number of local modules would lead to an increase in the number of acquirable primitives. An earlier study using multiple complex sensori-motor sequences, however, demonstrated that difficult problems arise in the local representation model as a result of its local nature [14]. Similarities in learned sensori-motor sequences create competition in the learning process between corresponding modules. Generalization requires similar patterns to be represented in the same module as the same primitive, even if there are subtle differences between such patterns. On the other hand, for the purposes of achieving "crisp" segmentation of sensory-motor flow, different patterns must be represented as separate primitives in distinct modules. This conflict between generalization and segmentation poses serious problems for the treatment of complex sensori-motor sequences. Due to the difficulty of this problem, it is not possible to increase the number of acquirable primitives simply by increasing the number of local modules [14]. In addition, due to the explicit hierarchical structure of the local Emergence of functional hierarchy 5/18 representation, learning of the lower module (primitives) and learning of the higher module (sequences of primitives) have to be explicitly separated through subgoals arbitrarily set by the experimenter [12, 15]. In order to overcome difficulties associated with the local representation model, we introduce in the current study a different type of functional hierarchy representation. The representation we use neither makes use of separate local modules to represent primitives, nor introduces explicit hierarchical structure to manipulate these primitives. Instead of setting up an explicit hierarchy, we attempt to realize the self-organization of a functional hierarchy by means of neural activity with multiple timescales. This functional hierarchy is made possible through the use of two distinct types of neurons, each with different time properties. The first type of neuron is the "fast" unit, whose activity changes quickly over the short term. The second type of neuron is the "slow" unit, whose activity changes over the long term (Figure 1B). The use of timescale variation in representing different levels of function has been attempted in earlier computational studies. In a study of reinforcement learning among agents solving a maze game, for example, Sutton's group showed that learning speed can be increased by using not only short-term rewards, but also long-term rewards [16]. In a study of an evolutional neural network model using a mobile robot, Nolfi [17] showed that a model with differing timescales is superior to the normal model in cases in which the robot is required to achieve two different tasks: collision avoidance, which requires short-term sensori-motor control, and self-localization, which requires long-term sensory integration. Although advantages in learning were shown, these studies did not demonstrate how continuous sensori-motor experiences are segmented into primitives, and how series of primitives are integrated into complex behaviors. The current study focuses not only on demonstrating advantages in learning, but also on showing that hierarchical differentiation of functions can be self-organized through multiple timescales in neural activities. In the proposed model, through repetitive execution of skilled behavioral tasks, continuous sensori-motor flows are segmented into reusable motor primitives, and segmented primitives are integrated into new behavior sequences. The model does this without setting up an explicit sub-goal or functions such as gate-selection for manipulating primitives in the lower module. The main component of the current model is a continuous time recurrent neural network (RNN). Thanks to its capacity to preserve the internal state, which enables it to reproduce complex dynamics, the RNN is often used for modeling temporal sequence learning [18-20]. The continuous time RNN (CTRNN) is a type of RNN which Emergence of functional hierarchy 6/18 implements a