Recurrent Neural Networks: from prediction to representation, adynamical systems perspective
Guillaume Pourcel1
1Bernoulli Institute & CogniGron, University of Groningen, Netherlands

Summary. A major advance in Cognitive Science was moving from the study of behavior(“behaviorism”) to the study of concepts in-between perception and action (“cognitivism”) interms of information processing concepts. Machine Learning also followed a similar trajectory bymoving from the study of input/output prediction to the learning of representation [1]. However,these studies are centrally focused on Feed Forward Neural Networks, which is inadequate formore general Recurrent Neural Networks and other Nonlinear Dynamical Systems where it’s notclear how abstract representations should be encoded. I present here a framework inspired byconceptors [2] to conceptualize the task of learning representation in Dynamical Systems by
carefully organizing neural trajectories instead of organizing atemporal neural activations (as doneclassically in Machine Learning). I discuss how this framework applies to Dynamical Systems ingeneral, where conceptors are just one entry point to a more general task.

Classically, the challenge of Learning Representation (LR) consists in extracting abstract/conceptlike encodings of a perceptual input without having access to supervision. The adequacy of theseencodings can be judged by their capacity to be decoded for regenerating the input. Additionally, theseencodings need to generalize. It should be possible to interpolate to generate novel meaningful examples.
The major difficulty for LR in Nonlinear Dynamical Systems (NDS) is that they don’t seem to offer
an identifiable, controllable neural code. The solution proposed by conceptors is to use the linearsubspaces of different input-driven neural trajectories as coarse-grained codes. Crucially, this code can beused bi-directionally: it can be extracted from input and decoded to regenerate the input.
To formalize the task, we start with a parametric family of time series 𝑢λ𝑖𝑛𝑝 :𝑢λ𝑖𝑛𝑝 = g(λ𝑖𝑛𝑝 , 𝑣, 𝑤)(1)
𝑖𝑛𝑝
where λis the input parameter, an abstract, unobserved concept that has to be inferred, and g is afunction that mixes the two signals 𝑣 and 𝑤 with λ𝑖𝑛𝑝 . Cognitively, λ𝑖𝑛𝑝 could represent an abstract speed
variable allowing an agent to perceive and produce complex speech signals along a continuum of
nonlinear distortions. For the cognitive system, I will use a discrete-time Recurrent Neural Network (RNN):𝑥(𝑛 + 1) = 𝑡𝑎𝑛ℎ (𝑊𝑥(𝑛) + 𝑊 𝑖𝑛 . 𝑢λ𝑖𝑛𝑝 (𝑛))(2)where 𝑥(𝑛) represents the N-dimensional neural activity at time n ∈ ℕ. W is the N×N connectivity matrix
and 𝑊 𝑖𝑛 is the N×p input matrix expanding the input into the reservoir. For pedagogical purposes, wewill use a simple linear morphing between two random signals of periodic 3, 𝑢λ𝑖𝑛𝑝 (𝑛) = (1 −λ𝑖𝑛𝑝 )𝑣(𝑛) + λ𝑖𝑛𝑝 𝑤(𝑛) sent to a 3-dimensional RNN (Fig.1).
Conceptor methods allow the input-driven system (2) to be transformed into an autonomous systemthat regenerates the input. First, each input signal from a set 𝐷 = {𝑢λ𝑖𝑛𝑝} of signals can be translated into𝑖𝑖∈𝐼a code 𝐶 𝑖 . For a signal 𝑖, this code captures the linear subspace spanned by its associated neural trajectory(2). Mathematically, this code is a projection matrix on this subspace and can be easily computed [2]. Forinstance, Fig1.D displays conceptors in a specific configuration where they correspond to 2D planes1
containing the neural trajectory. In higher dimensions, they can be visualized as ellipsoids (Fig1.A) [2].
The second step in the encoding phase (Fig.1A, left part) is to compute an additive weight matrix D(Fig.1A, green arrows) explicitly trained to reproduce the different trajectories created by the input [2].
While the D matrix entrenched2 a set of input-driven dynamics into an autonomous system, eachconceptor matrix 𝐶 𝑖 can be used to address and stabilize a specific one by projecting the dynamics with:𝑥(𝑛 + 1) = 𝑪𝒊 𝑡𝑎𝑛ℎ((𝑊 + 𝑫)𝑥(𝑛))(3)
1

For geometrical interpretability, the RNN has been optimized to produce visually intuitive geometries, where theconceptors are projecting on various z-rotations of the the xy plane. In high dimension, this optimization is notnecessary and interpolation with conceptors works with untrained RNNs (reservoir computers) [2].
2
D creates a represention of the input within the autonomous RNN. To verify that the representation is appropriate,
one can check if the input can be approximated by a time-invariant linear transformation on the neural trajectory (2).

Figure 1. A. Samples (blue crosses) from a 1D manifold of signals 𝑢(𝜆𝑖𝑛𝑝 ) are fed into a RNN which led
to the computation of an additive weight matrix (green arrows) and a sets of conceptors (ellipsoids) on amanifold 𝐶(𝜆𝑐 ). This manifold can be used to do interpolation (orange cross). B. Samples from theparametric family 𝑢(𝜆𝑖𝑛𝑝 ). C. 𝐶(𝜆𝑐 ) displayed in conceptor space (after PCA). D. Trajectories inducedby the inputs in the space of neural activity (the initial transients are not displayed). The 2D planes
correspond to conceptors associated with different trajectories.
We’ve seen that conceptor can be used to encode/decode signals, and we move now to the task of
LR by looking at their capacity to generalize. More precisely, after learning (the computation of D and the𝐶 𝑖 𝑠), we want the following neural trajectory of an unseen data sample close to the example i:(4)𝑥(𝑛 + 1) = 𝑡𝑎𝑛ℎ (𝑊𝑥(𝑛) + 𝑊 𝑖𝑛 . 𝑢λ𝑖𝑛𝑝 +∆λ𝑖𝑛𝑝 (𝑛))𝑖to be approximated by:(5)𝑥(𝑛 + 1) = 𝐶𝑖𝑛𝑡𝑒𝑟𝑝 𝑡𝑎𝑛ℎ((𝑊 + 𝐷)𝑥(𝑛))where 𝐶𝑖𝑛𝑡𝑒𝑟𝑝 is an appropriately interpolated code. Following ideas in Machine Learning [1], this code
can be extracted from compression principles. Indeed, the variability of real-world data as well as our toy
dataset 𝐷 can be assumed to be encodable with limited variability between trajectories. Work in progress[3] shows that dimensionality reduction algorithms (for instance, self-organizing maps) on conceptor
spaces are enough to uncover a parametrization of a manifold 𝐶(λ𝐶 ) that correspond to input manifold𝑢(λ𝑖𝑛𝑝 ) (Fig1.A). In our case the conceptor manifold is one-dimensional (Fig1.C).
This proposal extends the use cases of conceptors from the manipulation of memories to the learningof representations. Additionally, this work points to general challenges of LR with NDS when we analyze
two distinct roles for conceptors. First, they offer a space for compression to happen online without
perturbing the unfolding of the dynamic. This space can be used to replace the input-induced perturbationby ∆λ𝑖𝑛𝑝 (4) by a conceptor perturbation (5). These phenomena can be analyzed with perturbation
analysis [3]. The second role is to stabilize the interpolated trajectory. Indeed (5) pushes the NDS into aregion where it was not trained to behave autonomously, and the induced trajectory might be unstable.
This requirement is novel and should be further studied. Finally, this work opens new questions aroundthis new task of LR with NDS. As opposed to FeedForward Neural Networks, which fix the direction of
the flow of information (from encoding to decoding), RNNs and NDS don’t have this restriction:
encoding and decoding can happen simultaneously, opening the doors to new questions about cognitive
systems coupled with their environment.

References[1] Y. Bengio, et al., “Representation Learning: A Review and New Perspectives.” arXiv, Apr. 23, 2014.
[2] H. Jaeger, “Controlling Recurrent Neural Networks by Conceptors,” arXiv:1403.3369 [cs],Apr.2017,[3] G. Pourcel, “Learning Dynamical Representation with Conceptors” (manuscript in preparation)