arXiv:1502.04423v2 [] 26 Apr 2015

Exploring Transfer Function Nonlinearity in Echo
State Networks
Alireza Goudarzi

Alireza Shabani

Darko Stefanovic

Department of Computer Science
University of New Mexico
Albuquerque, NM 87131
Email: alirezag@cs.unm.edu

Google Inc.
Venice, CA 90291
Email: shabani@google.com

Department of Computer Science and
Center for Biomedical Engineering
University of New Mexico
Albuquerque, NM 87131
Email: darko@cs.unm.edu

Abstract—Supralinear and sublinear pre-synaptic and dendritic integration is considered to be responsible for nonlinear
computation power of biological neurons, emphasizing the role
of nonlinear integration as opposed to nonlinear output thresholding. How, why, and to what degree the transfer function
nonlinearity helps biologically inspired neural network models is
not fully understood. Here, we study these questions in the context
of echo state networks (ESN). ESN is a simple neural network
architecture in which a fixed recurrent network is driven with
an input signal, and the output is generated by a readout layer
from the measurements of the network states. ESN architecture
enjoys efficient training and good performance on certain signalprocessing tasks, such as system identification and time series
prediction. ESN performance has been analyzed with respect to
the connectivity pattern in the network structure and the input
bias. However, the effects of the transfer function in the network
have not been studied systematically. Here, we use an approach
tanh on the Taylor expansion of a frequently used transfer
function, the hyperbolic tangent function, to systematically study
the effect of increasing nonlinearity of the transfer function on the
memory, nonlinear capacity, and signal processing performance
of ESN. Interestingly, we find that a quadratic approximation is
enough to capture the computational power of ESN with tanh
function. The results of this study apply to both software and
hardware implementation of ESN.

I.

I NTRODUCTION

McCullough and Pitts [1] showed that the computational
power of the brain can be understood and modeled at the
level of a single neuron. Their simple model of the neuron
consisted of linear integration of synaptic inputs followed
by a threshold nonlinearity. Current understanding of neural
information processing reveals that the role of a single neuron
in processing input is much more complicated than a linear
integration-and-threshold process [2]. In fact, the morphology
and physiology of the synapses and dendrites create important
nonlinear effects on the spatial and temporal integration of
synaptic input into a single membrane potential [3]. Moreover,
dendritic input integration in certain neurons may adaptively
switch between supralinear and sublinear regimes [4]. From
a theoretical standpoint this nonlinear integration is directly
responsible for the ability of neurons to classify linearly
inseparable patterns [5]. The advantage of nonlinear processing
at the level of a single neuron has also been discussed in the
artificial neural network (ANN) community [6].
Historically, the ANN community has been concerned with
algorithms for finding the correct interaction pattern between

neurons for a specific task [7]–[9]. Some work in the field
has emphasized the importance of suitable collective behavior
of the neural network facilitated by macroscopic parameters
over microscopic degrees of freedom. Dominey et al. [10]
proposed a simple model for the context-dependent motor
control of eyes. In this model, the prefrontal cortex represents
a suitable high-dimensional mapping of visual input that is
adaptively projected onto basal ganglia, which in turn control
the eye movement. The only task-dependent learning in this
model occurs in the projection layer. This model has also been
used to explain higher-level cognitive tasks such as grammar
comprehension in the brain [11].
More abstract versions of this model, Liquid State Machines [12] and Echo State Networks [12], [13], were later
introduced in the neural network community and were subsequently unified under the name reservoir computing (RC) [14].
In RC, an easily tunable high-dimensional recurrent network,
called the reservoir, is driven by an input signal. An adaptive
readout layer then combines the reservoir states to produce a
desired output. Figure 1 provides a conceptual illustration of
RC. ESN implements this idea with a discrete-time recurrent
network with linear or tanh activation functions and a linear
readout layer trained using regression. Many variations of
ESN exist and have been successfully applied to different
engineering tasks, such as time series prediction and system
identification [15].
Owing to its fixed recurrent connections, training an ESN is
much more efficient than ordinary recurrent neural networks
(RNN), making it feasible to use its power in practical applications. ESN’s power in time series processing has been
attributed to the reservoir’s memory [16], [17] and highdimensional projection of the input which acts like a temporal
discriminant kernel [18] that is present in the critical dynamical
regime, where input perturbations in the reservoir dynamics
neither spread nor die out [19]–[21].
A major research direction in RC is to study how the
nonlinear dynamics of the reservoir may improve the performance in different tasks [15], [22]. In particular, the goal
is to understand and enhance the high-dimensional nonlinear
mapping created by the reservoir dynamics. In the case of ESN
architecture, the nonlinearity of the reservoir can be ascribed
to its connectivity pattern, transfer function, and the input
bias. While there have been some studies focusing on the
effect of connectivity and bias [21], [23], the transfer function
nonlinearity has never been systematically studied, to the best

input signal u(t)

input weights

output weights

ω

Ψ

actual output y(t)
E

reservoir state x(t)

target output b
y(t)

update old weights

Fig. 1: Computation in an ESN. The reservoir is an excitable recurrent network with N readable output states represented by the
vector X(t). The input signal u(t) is fed into one or more points i in the reservoir with a corresponding weight ω i , denoted by
the weight column vector ω = [ω i ].

of our knowledge.
Here, we examine what happens when we replace the tanh
function in the ESN reservoir with its partial Taylor series
expansion, varying the number of terms included. The addition
of each successive term will increase the order of nonlinearity
present in the transfer function, allowing us to gradually
interpolate between the linear and the tanh transfer functions.
In addition, we will explore the input weight scaling to study
the effect of sublinear integration on ESN performance, at each
level of nonlinearity. To control for other sources of variation,
we will restrict ourselves to the two most constrained reservoir
architectures that are known to preserve the computational
performance of the classical random reservoir, the simple cycle
reservoir (SCR) [24] and the Gaussian orthogonal reservoir
[16].
The main contribution of this work is a systematic study
of the role of the transfer function nonlinearity in the total
information processing capacity of recurrent neural networks.
Section II outlines the context and motivation of this work.
In Section III-A, we review the basic ESN formulation used
in this study. In Section III-B, we describe the details of our
Taylor expansion approach to quantify the degree of nonlinearity and its impact on the performance of tanh-neuron ESN.
The experimental study on information processing properties
of ESNs with Taylor expanded transfer functions is presented
in Section IV. We first study the memory and also the nonlinear
memory capacity of echo state networks with different transfer
function nonlinearity, then we evaluate the performance of
such networks against time-series tests of Mackey-Glass and
NARMA 10. In all cases, we find that the second order approximation of the tanh function provides all the nonlinear benefits
of the tanh with no significant improvement to the network
performance with increasing nonlinearity. Moreover, we show
that the region of the tanh function which is usually thought of
as linear is actually very nonlinear. RC has been suggested as a
suitable signal processing framework for hardware realizations
targeting unconventional substrates [25] and ultra-low power
implementations, due to its multitasking capability, robustness
to noise and variations, and a fixed computational core [26],
[27]. The result of this work can be used to simplify potential
hardware designs for RC while preserving their accuracy.

II.

BACKGROUND

Understanding the nature of computation and its properties
is an active subject of theoretical study in reservoir computing.
Hermans and Schrauwen [18] showed that the ESN reservoir
acts as a recursive kernel that generates a high-dimensional
mapping of an input signal that can be used by the readout
layer to reconstruct a target output. Büsing et al. [21] studied
the relationship between the reservoir and its performance and
found that while in continuous reservoirs the performance of
the system does not depend on the topology of the reservoir
network, coarse-graining the reservoir state will make the
dynamics and the performance of the system highly sensitive
to its topology. Verstraeten et al. [23] used a novel method
to quantify the nonlinearity of the reservoir as a function of
input weight magnitude. They used the ratio of the number
of frequencies in the input to the number of frequencies in
the dynamics of the input-driven reservoir as a proxy for
the reservoir nonlinearity. As a result of these studies the
growing consensus is that from a theoretical perspective one
would obtain more nonlinear computational power in the
reservoir by adjusting the input weight magnitudes such as
to project the input onto the more nonlinear regions of the
tanh transfer function [28] (see Figure 4a). This opens an
interesting research area; however, in existing approaches the
linear and nonlinear regions of the tanh function are not defined
precisely. Moreover, there is little evidence that using the
so-called nonlinear region of the tanh actually improves the
performance on nonlinear tasks [23], [24], [28].
To illustrate the effect of using the nonlinear parts of the
tanh function, we have included sensitivity analysis of reservoirs with linear and tanh transfer functions for solving four
different benchmarks, linear memory, nonlinear computation
capacity, Mackey-Glass chaotic prediction, and NARMA 10
computation (see Section IV for task details and Section III-A
for reservoir model). For memory and nonlinear computation
capacity a reservoir of N = 50 nodes was used, and for
Mackey-Glass and NARMA 10 tasks reservoirs of N = 500
and N = 100 nodes were used, respectively. The reservoirs are
generated by sampling the standard Gaussian distribution and
are rescaled to have spectral radius λ . Input weights are drawn
from the Bernoulli distribution over {−1, +1} and multiplied
by input weight coefficient v. The reservoir parameters v and λ
were swept on the interval [0.1, 1] with 0.1 increments and the

4
50

3

MC

40

ω

2

30
20

Ψ

1

10
0
1
0.8

0.8

0.6

0.6

0.4
0.2
v

0.2

0
1.2

1
0.8

0.4
λ

(a) tanh(x), memory

v 0.4

0

0

0.5
λ

u(t)

y(t)

(b) tanh(x), nonlinear computation
n=3

Ω

0

log10 (N M SE)

log 10(N M S E )

1

reservoir state x(t)

−2

−4

−6

−8
0 0.2
0.4 0.6
0.8

1 0

1
0.6 0.8
0.2 0.4
v

λ

(c) tanh(x), Mackey-Glass

0.5

(a) linear reservoir

0
-0.5
-1
0.2 0.4
λ 0.6 0.8

0.6
0.2 0.4v

ω

0.8 1

Tm

Ψ

Tm

(d) tanh(x), NARMA10

Fig. 2: Illustration of sensitivity of ESN performance to v and
λ . For linear memory and nonlinear capacity the highest values
are optimal and for Mackey-Glass prediction and NARMA 10
computation the lowest values are optimal. The optimal values
for all tasks occur for low v, where the input signal is mapped
onto the so-called linear region of the tanh(x) function.

u(t)

y(t)

Tm
Tm

Ω

Tm

reservoir state x(t)
(b) Taylor series reservoir

results were averaged over 10 runs. Figure 2 shows the results
of the sensitivity analysis. For all the tasks, the best results are
achieved for the lowest v values, which maps the inputs signals
well within the speculated linear region of the tanh function.
In this work, our goal is to decompose the nonlinearity of the
tanh function and study its effects as a function of the degree
of nonlinearity and input strength v.
III.

Ψ

u(t)

y(t)

Ω

M ODEL

reservoir state x(t)

A. Echo State Network
An ESN consists of an input-driven recurrent neural network, which acts as the reservoir, and a readout layer that reads
the reservoir states and produces the output. Mathematically,
the input driven reservoir is defined as follows. Let N be
the size of the reservoir. We represent the time-dependent
inputs as a column vector u(t), the reservoir state as a column
vector x(t), and the output as a column vector y(t). The input
connectivity is represented by the matrix ω and the reservoir
connectivity is represented by an N × N weight matrix Ω. For
simplicity, we assume that we have one input signal and one
output, but the notation can be extended to multiple inputs and
outputs. The time evolution of the reservoir is given by:
x(t + 1) = f (Ωx(t) + ωu(t)).

ω

(1)

where f is the transfer function of the reservoir nodes that is
applied element-wise to its operand. This function is usually
tanh or linear. The output is generated by the multiplication of
a readout weight matrix Ψ of length N + 1 and the reservoir
state vector x(t) extended by an optional constant 1 represented

(c) tanh reservoir

Fig. 3: (a) Schematic of a linear ESN. A time-varying input
signal u(t) drives a dynamical core called a reservoir. The
states of the reservoir x(t) are combined linearly to produce
the output y(t). The reservoir consists of N nodes. The input
and the reservoir connections are given by the vector ω and
the matrix Ω respectively. The reservoir states and the constant
are connected to the readout layer using the weight matrix
Ψ. (b) A Taylor series ESN with a similar structure to linear
ESN, but with Taylor series expansion of tanh tanh for the
transfer functions of the reservoir. (c) A tanh ESN with a
similar structure to linear ESN, but with tanh nonlinearity in
the transfer functions of the reservoir. Usually a tanh function
is used.

0.1
T1 (x)
T2 (x)
T3 (x)
T4 (x)
tanh(x)

1

0.5

0.5
0

0

tanh(x)
d tanh(x)
d2 tanh(x)

-0.5

-0.5
-1

-1
-2

-1

10 -1

0.08

0

1

2

0.06
0.04

10 -2
10 -3
10 -4
10 -5

1

3

-1.5
-1.5

5

7

9

m

0.02
0
-1

-0.5

x

(a) tanh function

RMSE

1.5

RMSE

1

0

0.5

1

1

1.5

3

x

5

7

9

m

(b) tanh Taylor series expansion

(c) approximating tanh

Fig. 4: (a) tanh and its first and second derivatives. (c) Taylor series approximation to tanh (c) Distance of Taylor series expansions
to tanh.

by x0 (t):

y(t) = Ψx0 (t).

(2)

The readout weights Ψ need to be trained using a teacher
input-output pair. A popular training technique is to use the
pseudo-inverse method [14]. One drives the ESN with a teacher
input and records the history of the reservoir states into a
matrix X, where the columns correspond to the reservoir nodes
and each row gives the states of all reservoir nodes at one time.
A constant column of 1s is added to X to serve as a bias. The
corresponding teacher output will be denoted by the column
vector b
y. The readout can be calculated as follows:
b 0 i,
Ψ = hXX0 i−1 hXY

(3)

where 0 indicates the transpose of a matrix. Figures 3a and 3c
show the architecture of ESNs with linear and tanh activation
functions, respectively. Figure 3b shows the architecture of an
ESN with the Taylor series approximation of tanh as transfer
function. In the next section, we will describe how we will
use these approximations to systematically study the transfer
function nonlinearity in the reservoir.
B. Transfer Function Nonlinearity
Our goal is to systematically explore the effect of nonlinearity of the reservoir transfer function on the ESN memory
and performance. Figure 4a illustrates the tanh(x) function and
its first and second derivatives, i.e., d tanh(x) and d 2 tanh(x).
The tanh(x) function is often considered to behave linearly for
|x| < 0.5, and nonlinearly otherwise. However, looking closely
at the curves of d tanh(x) and d 2 tanh(x), we see that the only
place where the tanh(x) behaves linearly (constant d tanh(x))
is when x → 0. As x increases in magnitude its first derivative
changes very rapidly with increasing rate, i.e., steep d 2 tanh(x),
until |x| = 0.65. This observation suggests that the so-called
linear region of tanh(x) function is where the function becomes
highly nonlinear very quickly as x increases.
We would like to decompose the nonlinearity of tanh
and study how much each additional degree of nonlinearity
affects the performance of the ESN. To this end, we use the
Taylor series expansion of the tanh function around x = 0 to
systematically interpolate the orders of nonlinearity between
the linear transfer function to tanh transfer function. We will
replace the tanh transfer function with the transfer functions

that we obtain by writing the tanh Taylor series to m terms,
denoted by Tm . Table I lists the first few expansions as well
as the exact Taylor series for tanh and Figure 3b illustrates the
architecture of the ESN with Taylor series expansions as the
reservoir transfer function.
m
T1 (x)
T2 (x)
T3 (x)
T4 (x)
.
.
.
tanh(x) = T∞ (x)

expansion
x
x − 13 x3
1 3
2 5
x − 3 x + 15
x
2 5
17 7
x − 13 x3 + 15
x − 315
x
.
.
.
B2m 4m (4m −1) 2m−1
x
∑∞
m=1
(2m)!

TABLE I: Example of Taylor series expansions of tanh(x) with
different orders m. Here, B2m is the number at the position 2m
in Bernoulli sequence.

Figure 4b shows the curves corresponding to the first four
expansions of the tanh(x) function. Although the Taylor series
expansion of tanh(x) is defined for |x| < π, it is only for |x| < 1
that the lowest order expansions do not rapidly diverge from
tanh(x). Figure 4c shows the root-mean-squared error (RMSE)
between the Taylor expansion m and the tanh(x) function
calculated for |x| < 1. With increasing number of terms in
the expansion, the approximation approaches the true tanh
exponentially (the inset plot). Understanding this exponential
behavior suggests most of the benefits of the tanh nonlinearity
may come from the first few orders of nonlinearity, and this
will help us to interpret the results in the later sections.
IV.

E XPERIMENTS

In this section we study the effect of nonlinearity of the
transfer function in ESNs using two parameters, the input
weight coefficient v and the order of the Taylor series expansions used as the transfer function m. We will evaluate the
performance of ESNs in linear memory capacity, nonlinear
capacity, Mackey-Glass chaotic time series prediction, and
NARMA 10 computation.
To make a fair comparison between systems, we adjust v
and the input signal scaling so that the the magnitude of the
reservoir states is less than 1. The next section will give the
details of ESN construction and evaluation.

The reservoirs are driven with task-dependent input ut for
2, 000 time steps and the readout weights Ψ are calculated as
described in Section III-A using MATLAB’s pinv() function.
For evaluation, the reservoir state is reinitialized and the
reservoir is driven for another T = 2, 000 time steps and the
output yt is generated. For brevity, throughout the experiments
section we adopt the subscript notation for the time index, e.g.,
yt instead of y(t). By convention, the system performance for
computational capacity tasks is evaluated using the capacity
function Cτ , which is the coefficient of determination between
the output yt and the desired output ybt :
Cov (yt , ybt )
,
Var(yt )Var(b
yt )

50
45

where yt is the network output and ybt is the desired output.

For all tasks we systematically explore v ∈
{10−5 , . . . , 10−1 } with quarter decade increments and
v ∈ {0.2, . . . , 0.35} with 0.05 increments. All results are
averaged over 10 runs. We chose this range for v in
preliminary runs in combination with appropriate input
scaling for each task to ensure that the magnitude of reservoir
states is always less than 1.
B. Linear Memory Capacity
The linear memory capacity is a standard measure of
memory in recurrent neural networks. The τ-delay memory

40
35
30
25

(4)

where τ is the memory length for the task (see Section IV-B
for details). For the chaotic prediction task, the performance
is evaluated by calculating the normalized mean-squared-error
NMSE as follows:
q
1 T
bt )2
T ∑t=0 (yt − y
NMSE =
,
(5)
Var(b
yt )

(6)

Figure 5a shows the total linear memory capacity surface as
a function of m and v. Consistent with previous theoretical and
experimental results the linear memory capacity does not show
any dependency on v for m = 1, i.e., for the linear network.
However, for large v > 0.05 and m > 1 we observe a deviation
from linear memory with no dependence on m. Figure 5b
shows the total memory capacity for the tanh transfer function
as a function of v on a linear-log scale, clearly showing
that for v < 0.05 the total memory capacity of the network
equals that of a linear network. Figure 5c shows the total
capacity for v = 0.1 for various m, confirming that for m > 1
the memory capacity does not vary with m, and suggesting
that all the relevant nonlinear characteristics of the network
stemming from tanh can be observed on the second-order
Taylor expansion m = 2.

0.05
0.15

v

0.25

20
tanh(x)

15

5

10

1

m

(a)
50

50
45

45

40

C

Cτ =

2

ybt = ut−τ .

C

To control for the variations that are due to topological factors, we will use very constrained reservoir architectures. For
the memory task, Mackey-Glass prediction, and NARMA 10
computation we will use the simple cycle reservoir [24]. This
topology compares well with random topology in memory and
signal-processing benchmark performance, while minimizing
the structural variations of the reservoir. In the simple cycle
reservoir, the reservoir is a simple ring topology with uniform
positive weights r. In this topology the weight r determines the
reservoir spectral radius: r = |λ | and no rescaling of the weight
matrix is needed. In initial experiments, we observed that the
simple cycle is unable to perform the nonlinear capacity task.
For this task we create the reservoir by sampling the Gaussian
orthogonal ensemble (GOE) [16]. The reservoir weight matrix
in this case is given by Ω = A + A0 , where A is a matrix with
the same dimensionality as Ω where the entries are sampled
from the standard Gaussian distribution N (0, 1). The reservoir
is then rescaled to have spectral radius λ . The number of
reservoir nodes N is adjusted for each task to get reasonably
good results in a reasonable amount of time. The input weights
are generated by sampling the Bernoulli distribution over
{−1, +1} and multiplying with the input weight coefficient
v. The reservoir nodes are initialized with 0s and a washout
period of 2N is used during training and testing.

function Cτ measures how long a network can remember
its inputs. These capacities are calculated by summing the
capacity function over τ: C = ∑τ Cτ . We use 1 ≤ τ ≤ 100 for
our empirical estimations. In these sets of experiments reservoirs of size N = 50 nodes are driven with a one-dimensional
input drawn from uniform distributions on [−0.5, 0.5]. We fix
λ = 0.9 for all experiments. The desired output for this task
is defined as:

C

A. Reservoir Construction and Evaluation

35

35

30
25
10-4

40

10-3

10-2

v

(b)

10-1

100

30

1

5

10

15

20

tanh(x)

m

(c)

Fig. 5: (a) Linear memory capacity for different v and m.
(b) For v < 0.05 the memory capacity of the tanh network is
similar to that of a linear network. (c) Increasing nonlinearity
beyond m > 2 there is no change in the memory capacity of
the network.

C. Nonlinear Computation Capacity

We must point out that unlike [17], here the network has
to reconstruct the output of a single polynomial and not the
product of several polynomials. In this work we only focus
on the case n = 3. For n = 1, the nonlinear capacity measure
reduces to linear memory and the tanh are unable to compute
the even orders because of the input-output symmetry.

Figure 6a shows the total nonlinear capacity surface as a
function of m and v. For v > 0.001 and m > 1 we observe a
deviation from the linear network capacity, with no dependence
on m. Figure 6b shows the nonlinear capacity for the tanh
transfer function as a function of v on a linear-log scale,
clearly showing that for v < 0.001 the nonlinear capacity
of the network equals that of a linear network. Figure 6c
shows the total capacity for v = 0.1 for various m, confirming
that for m > 1 the nonlinear capacity does not vary with
m, suggesting all the relevant nonlinear characteristics of the
network stemming from tanh can be observed on the secondorder Taylor expansion m = 2. We emphasize that we have
used a standard ESN implementation without reservoir bias for
simplicity. Applying a bias to the reservoir drastically changes
the nonlinear capacity and requires a more thorough analysis.
D. Mackey-Glass System Prediction
The Mackey-Glass system [29] is a delayed differential
equation with chaotic dynamics, commonly used as a benchmark for chaotic signal prediction. This system is described
by:
x
dxt
= β t−δn − γxt ,
dt
1 + xt−δ

(8)

where β = 0.2, n = 10, and γ = 0.1 are positive constants and
δ = 17 is the feedback delay. The reservoir consists of N = 500
nodes and λ = 0.9. The task is to predict the next τ integration
time steps given xt . We scaled the time series between [0, 0.5]
before feeding the network.
Figure 7a shows the NRMSE surface as a function of m and
v. For m > 1 we observe a deviation from the linear network
performance with no dependence on m. Figure 7b shows the
performance for the tanh transfer function as a function of v

4

C

3
2
1
0
0.05
0.15

v

0.25

9
tanh(x)

1

3

5

7

m

(a)
4

5
4
3

3

2

2

C

C

The nonlinear computation capacity measures the ability
of the system to reconstruct a nonlinear function of its past
inputs. Commonly, Legendre polynomials are used to calculate
the nonlinear computation capacity of the reservoir [17]; their
advantage is that Legendre polynomials of different orders
are orthogonal to each other, allowing one to measure the
reservoir’s capacity to compute functions of varying degrees of
nonlinearity independently from each other. These capacities
are calculated by summing the capacity function over τ:
C = ∑τ Cτ . We use 1 ≤ τ ≤ 100 for our empirical estimations.
In these sets of experiments reservoirs of size N = 50 nodes
are driven with a one-dimensional input drawn from uniform
distributions on [−1, 1]. We fix λ = 0.1 for all experiments.
We have previously observed this is the optimal λ for this
task. The desired output of the Legendre polynomial of order
n with delay τ is given by:
 
1 n n 2
yb(n, τ)t = n ∑
(ut−τ − 1)n−k (ut−τ + 1)k .
(7)
2 k=0 k

1
1

0
-1
10-4 10-3

10-2

10-1

0

100

1

3

5

7

v

m

(b)

(c)

9 tanh(x)

Fig. 6: (a) Nonlinear capacity for different v and m. (b) For
v < 0.001 the nonlinear capacity of tanh network is similar
to that of a linear network. (c) Increasing nonlinearity beyond
m > 2 there is no change on the nonlinear capacity of the
network.

on a linear-log scale, clearly showing that for v < 0.00075 the
performance of the network equals that of a linear network,
with no improvement for v > 0.1. Figure 7c shows the performance for v = 0.1 for various m, confirming that for m > 1 the
performance does not vary with m, suggesting all the relevant
nonlinear characteristics of the network stemming from tanh
can be observed on the second-order Taylor expansion m = 2.
In our experiments, we found that although applying a bias
to the reservoir improves its nonlinear capacity, it does not
improve the performance for Mackey-Glass tasks.
E. NARMA 10 Computation
NARMA 10 [24] is a highly non-linear auto-regressive task
with long lags that is frequently used to assess neural network
performance. This task is given by the following equation:
n

yt = αyt−1 + β yt−1 ∑ yt−i + γut−n ut−1 + δ ,

(9)

i=1

where n = 10, α = 0.3, β = 0.05, γ = 1.5, δ = 0.1. The input
ut is drawn from a uniform distribution in the interval [0, 0.5].
We use reservoir networks of size N = 100 and λ = 0.8.
Figure 8a shows the NRMSE surface as a function of m
and v. For m > 1 we observe a deviation from the linear
network performance, with no dependence on m. Figure 8b
shows the performance for the tanh transfer function as a
function of v on a linear-log scale, clearly showing that for

0.17

N RM SE

N RM SE

×10 -3
1.5
1
0.5

0.16
0.15
0.14

0
0.05

0.13

0.15
0.25

v

9
tanh(x)

7

0.05

1

3

5

0.15

v

0.25

m

×10-3

0.2
10-4 10-3 10-2 10-1 100

v

(b)

N RM SE

N RM SE

N RM SE

0.4

0

0.18

1

0.8

0.18

0.2

0.5

N RM SE

-3

0.6

m

(a)

(a)
1 ×10

7

9
tanh(x)

1

3

5

0.16
0.14
0.12

0

1

3

5

7

0.1
10-4 10-3 10-2 10-1 100

9 tanh(x)

0.14
0.12
0.1

1

v

m

3

5

7

9 tanh(x)

m

(b)

(c)

Fig. 7: (a) Mackey-Glass prediction performance for different
v and m. (b) The prediction performance for tanh network. For
v < 0.00075 the the performance of tanh network is similar to
that of a linear network. (c) Increasing nonlinearity beyond
m > 2 there is no change on the memory capacity of the
network.

0.16

(c)

Fig. 8: (a) NARMA 10 performance for different v and m.
(b) The performance for tanh network. For v < 0.01 the the
performance of the tanh network is similar to that of a linear
network. (c) Increasing nonlinearity beyond m > 2 there is no
change on the memory capacity of the network.

ACKNOWLEDGMENT
v < 0.01 the performance of the network equals that of a linear
network with, no improvement for v > 0.01. Figure 8c shows
the performance for v = 0.1 for various m. In this case because
of large standard deviation we cannot conclusively say that the
increasing nonlinearity in the transfer function is helpful.
V.

C ONCLUSION AND OUTLOOK

Nonlinearity of pre-synaptic and dendritic integration plays
an important role in the nonlinear computational ability of biological neurons. Similarly, nonlinearity of the transfer function
in neural networks is known to increase the capability of the
simple multi-layer perceptron to approximate any function. In
this work, we systematically studied the effect of increasing
nonlinearity on the memory, nonlinear capacity, and the signalprocessing performance of echo state networks (ESN), a class
of efficient recurrent neural network with state of the art performance in chaotic signal prediction. We found that the region of
the tanh function usually thought of as linear is actually quite
nonlinear. Moreover, we found that all the nonlinear power of
the tanh transfer function can be produced using its secondorder Taylor approximation. This finding suggests that ESN
performance will benefit from qualitative nonlinearity and not
from the degree to which the transfer function is nonlinear.
How and why small transfer function nonlinearity helps ESNs
will be the subject of our future research.

This material is tanh upon work supported by the National
Science Foundation under grants CDI-1028238 and CCF1318833.
R EFERENCES
[1]
[2]
[3]
[4]
[5]
[6]
[7]
[8]

W. McCulloch and W. Pitts, “A logical calculus of the ideas immanent
in nervous activity,” Bulletin of Mathematical Biology, vol. 5, pp. 115–
133, 1943.
C. Koch and I. Segev, “The role of single neurons in information
processing,” Nature Neuroscience, vol. 3, pp. 1171–1177, 2000.
J. C. Magee, “Dendritic integration of excitatory synaptic input,” Nature
Reviews Neuroscience, vol. 1, pp. 181–190, 2000.
M. Margulis and C.-M. Tang, “Temporal integration can readily switch
between sublinear and supralinear summation,” Journal of Neurophysiology, vol. 79, no. 5, pp. 2809–2813, 1998.
R. D. Caz, M. Humphries, and B. Gutkin, “Passive dendrites enable
single neurons to compute linearly non-separable functions,” PLoS
Comput Biol, vol. 9, no. 2, p. e1002867, 02 2013.
C. L. Giles and T. Maxwell, “Learning, invariance, and generalization in
high-order neural networks,” Appl. Opt., vol. 26, no. 23, pp. 4972–4978,
1987.
F. Rosenblatt, “The perceptron: A probabilistic model for information
storage and organization in the brain,” Psychological review, vol. 65,
no. 6, pp. 386–408, 1958.
D. E. Rumelhart, G. E. Hinton, and R. J. Williams, “Learning internal
representations by error propagation,” in Parallel Distributed Processing, D. E. Rumelhart and J. L. McClelland, Eds. Cambridge, MA:
MIT Press, 1986, vol. 1, ch. 8, pp. 318–362.

[9]
[10]
[11]

[12]

[13]
[14]
[15]
[16]
[17]
[18]
[19]
[20]
[21]
[22]
[23]
[24]
[25]
[26]

[27]

[28]
[29]

J. J. Hopfield, “Neural networks and physical systems with emergent
collective computational abilities,” Proc. Natl. Acad. Sci., vol. 79, no. 8,
pp. 2554–2558, 1982.
P. Dominey, M. Arbib, and J.-P. Joseph, “A model of corticostriatal
plasticity for learning oculomotor associations and sequences,” Journal
of Cognitive Neuroscience, vol. 7, no. 3, pp. 311–336, 1995.
X. Hinaut and P. F. Dominey, “Real-time parallel processing of grammatical structure in the fronto-striatal system: A recurrent network
simulation study using reservoir computing,” PLoS ONE, vol. 8, no. 2,
p. e52946, 2013.
W. Maass, T. Natschläger, and H. Markram, “Real-time computing
without stable states: a new framework for neural computation based
on perturbations,” Neural Computation, vol. 14, no. 11, pp. 2531–60,
2002.
H. Jaeger and H. Haas, “Harnessing nonlinearity: Predicting chaotic
systems and saving energy in wireless communication,” Science, vol.
304, no. 5667, pp. 78–80, 2004.
D. Verstraeten, B. Schrauwen, M. D’Haene and D. Stroobandt, “An
experimental unification of reservoir computing methods,” Neural Networks, vol. 20, no. 3, pp. 391–403, 2007.
M. Lukoševičius, H. Jaeger, and B. Schrauwen, “Reservoir computing
trends,” KI - Künstliche Intelligenz, vol. 26, no. 4, pp. 365–371, 2012.
O. L. White, D. D. Lee, and H. Sompolinsky, “Short-term memory in
orthogonal neural networks,” Phys. Rev. Lett., vol. 92, p. 148102, 2004.
J. Dambre, D. Verstraeten, B. Schrauwen, and S. Massar, “Information
processing capacity of dynamical systems,” Sci. Rep., vol. 2, 07 2012.
M. Hermans and B. Schrauwen, “Recurrent kernel machines: Computing with infinite echo state networks,” Neural Computation, vol. 24,
no. 1, pp. 104–133, 2011.
N. Bertschinger and T. Natschläger, “Real-time computation at the edge
of chaos in recurrent neural networks,” Neural Computation, vol. 16,
no. 7, pp. 1413–1436, 2004.
D. Snyder, A. Goudarzi, and C. Teuscher, “Computational capabilities
of random automata networks for reservoir computing,” Phys. Rev. E,
vol. 87, p. 042808, 2013.
L. Büsing, B. Schrauwen, and R. Legenstein, “Connectivity, dynamics,
and memory in reservoir computing with binary and analog neurons.”
Neural Computation, vol. 22, no. 5, pp. 1272–1311, 2010.
A. Goudarzi, A. Shabani, and D. Stefanovic, “Product reservoir
computing: Time-series computation with multiplicative neurons,”
arXiv:1502.00718 [], 2015.
D. Verstraeten, J. Dambre, X. Dutoit, and B. Schrauwen, “Memory
versus non-linearity in reservoirs,” in Neural Networks (IJCNN), The
2010 International Joint Conference on, July 2010, pp. 1–8.
A. Rodan and P. Tiňo, “Minimum complexity echo state network,”
Neural Networks, IEEE Transactions on, vol. 22, pp. 131–144, Jan.
2011.
D. S. C. T. Jens Brger, Alireza Goudarzi, “Hierarchical composition
of memristive networks for real-time computing,” arXiv:1504.02833
[cs.ET], 2015.
A. Goudarzi, M. R. Lakin, D. Stefanovic, and C. Teuscher, “A Model
for Variation- and Fault-Tolerant Digital Logic using Self-Assembled
Nanowire Architectures,” in Proceedings of the 2014 IEEE/ACM International Symposium on Nanoscale Architectures (NANOARCH). IEEE
Press, 2014, pp. 116–121.
A. Goudarzi, M. Lakin, and D. Stefanovic, “Reservoir computing
approach to robust computation using unreliable nanoscale networks,”
in Unconventional Computation and Natural Computation, ser. Lecture
Notes in Computer Science, O. H. Ibarra, L. Kari, and S. Kopecki, Eds.
Springer International Publishing, 2014, pp. 164–176.
J. Butcher, D. Verstraeten, B. Schrauwen, C. Day, and P. Haycock,
“Reservoir computing and extreme learning machines for non-linear
time-series data analysis,” Neural Networks, vol. 38, pp. 76 – 89, 2013.
M. C. Mackey and L. Glass, “Oscillation and chaos in physiological
control systems,” Science, vol. 197, no. 4300, pp. 287–289, 1977.

