Adaptive Modular Architectures for Rich Motor Skills ICT-248311 D 5 . 2 March 2012 ( 24 months ) Technical report on Hierarchical Reservoir Computing architectures

One approach for building architectures (of which an overview was given in D.6.1) in AMARSi is to use reservoir computing. Here, untrained (or unsupervised trained) recurrent neural networks are used for motion control by learning simple readouts on the dynamic representation generated by the dynamic RNN system. Although single reservoirs are able to generate rich and tunable control patterns (as demonstrated in D.4.1), to allow composition of motion or high-level control, these modules need to be built in an architecture. An active research area in reservoir computing is to build hierarchical reservoir systems. The main reason for this is that reservoirs basically are band-pass systems and can only represent information in a limited frequency band. If information at both fast and slow timescales needs to be integrated, a natural approach is to build a hierarchical system where each layer operates at a different time scale. The big challenge in these hierarchies is how to learn intermediate representations that link the various layers, and especially how bottom-up and top-down information flows need to be organized. We believe that these hierarchical reservoir computing systems are good candidates to build (at least part of) architectures required in AMARSi for rich motor control. In this short deliverable we give an overview of and references to current approaches in hierarchical reservoir computing, several of which have been investigated on speech and handwriting recognition problems in the sister EU project ORGANIC (http://reservoircomputing.org/organic). Many of these hierarchical systems can be used to not only generate dynamical feature hierarchies, but are also able to learn a hierarchy of pattern controller, of special interest to the AMARSi project. Overview of hierarchical reservoir computing architectures in AMARSi In this overview section we discuss briefly a number of hierarchical reservoir computing based architectures which are developed and evaluated in the AMARSi project. Several of those have previously been proposed and used for speech and handwriting recognition in the ORGANIC project, but will be used as building block for motor control hierarchies in AMARSi. Hierarchical Echo State Networks (JAC) Although in biology hierarchical information processing is considered natural, in machine learning no system can be found that (1) can deal with high-dimensional temporal data, (2) is able to discover its own dynamical features and (3) uses a computationally efficient, online adaptive learning algorithm. In (Jaeger 2007) a hierarchical learning architecture was proposed that incorporates the three desired features mentioned above. This hierarchy can discover dynamical features in a high-dimensional stationary stochastic time series in such a way that each a feature itself is a time series, the dynamical features are organized hierarchically and the original time series can be reconstructed from all the dynamical features. In summary, on the lowest level a time series can be decomposed into a set of features and votes. The combination of features and votes should be such that the original signal can be reconstructed. The votes of the lowest level are obtained by the second layer and consists of the combination of second layer features and second layer votes. As previously, the second layer votes are derived by a third layer etc. A schematic overview of the proposed hierarchy can be seen in the figure on the next page. Learning within the hierarchy occurs by changing the output weights of the reservoir systems of each level. This is done on each time step by using stochastic gradient descent. Therefore the gradient is taken with respect to the squared prediction error (= difference between original external input and predicted input). A formal description of the hierarchy and its learning process can be found in (Jaeger 2007). A downside of this architecture is that it depends on backpropagation-based stochastic gradient descent through several layers of gating, which makes training the hierarchy in real world noisy datasets hard. Self-Organized Neural Hierarchies (JAC) A hierarchy of Recurrent Neural Networks (reservoirs) can be envisioned as a layered structure of reservoirs where the lowest level is driven by an external input, and each higher level is excited by the activation of its lower neighbouring layer: The internal reservoir weights are trained layer-wise in an unsupervised way by applying the recurrent self-organizing maps (recurrent SOMs) algorithm which is the classical SOM algorithm but generalized from the conventional static pattern domain to temporal patterns. For this, the reservoirs need to be constructed from radial basis function (RBF) units. The readout on the top layer of the hierarchy, is trained supervised by using linear regression. The idea behind this hierarchy is to achieve a sparse and localised representation of temporal invariances in the driving input at the readout layer. Thus making it easier to identify which representation is invariant. From bottom to top, the representation becomes gradually more sparse and localized. Tested configurations of this architecture can be found in (Lukosevicius 2010 and Lukosevicius 2012) RC-SFA architecture for unsupervised learning of a spatial representation of a robot environment (UGent) Experiments with rats in open fields have shown the existence of hippocampal place cells. These cells form an implicit spatial representation of an animal’s environment (Moser et al., 2008). Using a constrained environment for rats it is shown that hippocampal CA1 cells and entorhinal cortex cells code for spatial information on a way dependent on the rat’s path or behavior. This assumes that if the entorhinal cortex and the hippocampus are related to path planning, these structure should reflect where the animal intends to go (Frank et al., 2000). This can be mimicked by using a three level layered hierarchy which consists of a reservoir system, a slow feature analysis layer and an independent component analysis layer: Sensory information is mapped into a high-dimensional non-linear dynamical space. Because of the dynamical nature of the reservoir system, the entire hierarchy can rely on both current input changes as changes of the past. From the reservoir system slow varying signals are derived in an unsupervised way using Slow Feature Analysis (Wiskott and Sejnowski, 2002). This has the advantage that slow varying properties, such as environmental changes, can be tracked from (often) fast varying sensor information. From the obtained slow varying signals, a linear combination is taken using by the Independent Component Analysis layer. This learns in an unsupervised way a sparse coding of the slow varying signals, resulting in units that are activated only for a specific position in the environment. Or in other words, place cells can be derived from them. A more detailed description including results on robots can be found in (Antonelo and Schrauwen, 2012). This building block would allow to build hierarchical reservoir system by stacking these modules. Spatial SFA convolutional reservoir hierarchies (JAC) Signal processing of complex real world data is a difficult task for computational intelligence due to the many distorting variations in the data. In (Sakenas 2010) a method for learning temporal features which are invariant to the distortions in the input data was proposed. In the proposed approach a distorted signal together with its original example is first expanded by using a random non-linear dynamical system. Secondly the states are normalized by a whitening step such that all resulting signals has a unit variance in all directions. Thirdly, the difference between the whitened states obtained from the distorted signal and the original signal is taken. Finally, principal component analysis is applied on this whitened state difference The procedure is illustrated in the following Figure: The core idea of the proposed technique to find linear projections from a reservoir such that the reservoir states vary minimally when the same signal with a distortion is processed. The procedure is very comparable with Slow Feature Analysis (Berkes and Wiskott 2005), but instead of of finding the temporally slowest features, the features that are most invariant to spatial transformations are found. Examples on synthetic data and handwriting recognition can be found in (Sakenas 2010). Temporal Reservoir Machine (UGent) Guided by the advantageous and disadvantageous of previous approaches RBM-based approaches for time series modeling such as the CRBM and RTRBM, (Schrauwen and Busing, 2009) introduced a modification of these models called Temporal Reservoir Machine (TRM). The TRM is aimed at overcoming the limitations of a fixed model order (memory depth) of the CRBM and at the same time circumventing the problems associated with learning via BPTT which occur in the RTRBM, by using strategies from RC. The basic idea of the TRM is the following. In addition to the visible and hidden random variables we introduce bias variables for each time step. These determine the biases of the visible and hidden variables of the RBM via a linear mapping. Further, the biases are given by a deterministic function of the previous visible variables. Inspired by the ESN, we consider here the setup where the bias is given by the activation of a recurrent neural network that receives the previous visible variables as input. In the spirit of the ESN approach, the reservoir weight matrices are randomly generated and they remain untrained. Only the connections from the reservoir to the visible and hidden variables are adapted using contrastive divergence. Learning, sampling and inference in this model is tractable and efficient. Mimicking the hierarchical structure of a deep belief networks, TRMs can be stacked yielding a multi-layer model for sequential data analogous to t