Scale-Rule Selection of Affordable Neural Network
for Chaotic Time Series Learning
Yoko Uwate and Yoshifumi Nishio

Ruedi Stoop

Dept. of Electrical and Electronic Engineering,
Tokushima Univ., 2-1 Minami-Josanjima, Tokushima, Japan
Email: {uwate, nishio}@ee.tokushima-u.ac.jp

Institute of Neuroinfomatics, University / ETH Zurich
Winterthurerstrasse 190, 8057 Zurich, Switzerland
Email: ruedi@ini.phys.ethz.ch

Abstract— Scale-free networks are an important class of complex networks since many “real-world networks” fall into thiscategory. In our contribution we investigate the influence of thisproperty on the performance of an affordable neural network.
By means of computer simulations, we confirm that affordable
neural networks, when the affordable neurons are chosen ina scale-free manner, perform significantly better compared torandom selection.

# of nodes

Bell Curve

I. I NTRODUCTION
Since the scale-free networks were discovered by Barabasi
et al. [1], studies assessing the influence of this property on
the efficiency of networks have been carried out in various
fields. One way of how to characterize the difference between
random and scale-free networks is by means of the distributionof the number of links  a node has. From Fig. 1, where we
contrast the two network types, it is evident that in the scalefree network, although most nodes only have few connections,some nodes (marked in red) act as highly connected hubs.
This distinction is captured in a more quantitative way by the
distribution of the number of links vs. the number of nodes,
as shown in Figure 2. Random networks display a bell-shaped
curve, implying that most nodes have the same number oflinks, and no highly connected nodes (see Fig. 2 (a)). Scalefree networks, in contrast, often have many nodes with a
few links only, whereas quite a few hubs exist that have alarge number of links. Mathematically, scale-free networks arecharacterized by power law distributions (Fig. 2 (b)). Because
scale-rules emerge in many areas and disciplines of science(e.g. engineering, economics, social sciences and so on), we
expect that also in the development of the science of complex
networks, they will play an important role.

(a) Random network.
Fig. 1.

(b) Scale-free network.

# of links(a) Random network.
Fig. 2.

Power Law
distribution# of links(b) Scale-free network.

Comparing random and scale-free distribution.

hidden layer, for efficient BP-learning [2]. We christened thisnetwork “Affordable Neural Network.” In this network, someextra neurons are inserted into the hidden layer. When the
network operates, not all of the neurons present in the hidden
layer are updated; the affordable neurons are by-passed. By
using different sets of neurons for the update in the hidden
layer, the network is able to operate at a high performance.
By computer simulations [2], the affordable neural network
has been confirmed to achieve an improved performance overconventional networks for BP-learning, in terms of speed ofconvergence and of learning efficiency. Moreover, we haveinvestigated the performance of the affordable neural network
for noise-polluted input data. We found that the affordable neural network is able to generate noise-cleaned outputs, whichleads to the conclusion that the affordable neural network has
the generalization property. However, we believe that many
advantageous characteristics of the affordable neural network
are yet to be unveiled, and we also believe that the operationof the affordable neural network embodies important generalfeatures of the BP-learning process.
The first step in this programme deals with the performanceof the affordable neural network if the affordable neurons are
selected by a scale-rule, instead of making a random choice.
We will show by computer simulations, that the affordable
neural network under scale-rule selection achieves a better
performance, compared to random selection.
II. A FFORDABLE N EURAL N ETWORK
A. Network Model with Affordable Neurons

Example of random and scale-free network.

In a previous study on artificial neural networks, we proposed a new network structure with affordable neurons in the1-4244-1342-7/07/$25.00 ©2007 IEEE

# of nodes

811

In Ref. [2], we introduced the affordable neurons to reflect
important properties of the brain. During BP-learning, not all

of the neurons in the hidden layer are used at every updating:some of the neurons are selected for the learning and therest of the neurons are deactivated. The underlying networkmodel and the described mode of operation of the affordable
neurons in the hidden layer is sketched in Figs. 3(a) and (b),respectively.

Output layer
Input layer

Affordable
Neurons
Hidden layer

Selected times for operating neuron(a) Network model with affordable neurons.

the neurons in the hidden layer is set to be 100 and the number
of affordable neurons is 20, 40 and 60, respectively. Our
simulations will be based on 100000 updates. In this Figure,the horizontal axis indicates the neuron number, whereas thevertical axis displays the number of times the corresponding
neuron was in the set of operating neurons. By this Figure it isconfirmed that the operation time decreases gradually with theneuron number of the hidden layer. Furthermore, histogramsof the number of neurons that have a given operation time
are shown in Fig. 5. The resemblance with the scale-free
distribution of Fig. 2 (b) is evident, although, when inspected
in details, the distribution is not of a simple power law type.
2) Random selection: For comparison let us also consider
random selection of affordable neurons. The results we obtainare shown in Fig. 6. As can be expected, under random
selection, the operation time is similar for any neuron. Figure 7
shows again the distribution relationship between the selected
time and the number of neurons. This curve clearly reproduces
the bell-shaped distribution.

Hidden layer structure

Learning time
Operating neuron

100000
90000
80000

60000

40000
30000

Affordable neuron: 60

20000
10000
0
0th

(b) Operation of the affordable neurons in the hidden layer.

10th

20th

30th

40th

50th

60th

70th

80th

90th

100th

Neuron number of hidden layer

Affordable neural network.

30

30

25

25

20

15

10

5

812

20

15

10

5

0
70000

75000

80000

85000

90000

0
40000

95000

50000

60000

70000

80000

90000

Selected time for operating neuron

Selected time for operating neuron(a) Affordable neuron: 20(b) Affordable neuron: 40.

(1)

30

25

Number of neurons

where random() means the uniform random function producing values from 0.0 to 1.0. This implies that the neuron
with the highest index will generally have a small value,
whereas the first neuron will – unless the random function
states something different – have a larger entry. Note that thevalues of the entries follow a power-law distribution. Using
these values, we select in each update the set of active neurons
according to the values of S. From the s neurons in the hidden
layer, exactly the k neurons with the smallest entries arechosen as the affordable neurons. Figure 4 illustrates this scalerule selection of the affordable neurons, where the number of

Scale-rule selection (Hidden: 100).

Number of neurons

Number of neurons

Fig. 4.

B. Selection of Affordable Neurons
The two models of the selection of the affordable neurons
in the hidden layer are implemented as follows.
1) Scale-rule selection: Our scale-rule selection procedure
is described in terms of a parameter denoted by a vector S.
The dimension s of S equals the number of neurons present
in the hidden layer; each component of S corresponds to one
single neuron indexed by i. The values of the components areevaluated in each update by
Si = random()/i2

Affordable neuron: 40

50000

Affordable neuron (rest neuron)Fig. 3.

Affordable neuron: 20

70000

20

15

10

5

0
10000

30000

50000

70000

90000

Selected time for operating neuron(c) Affordable neuron: 60.
Fig. 5.

Distribution of scale-rule selection (Hidden: 100).

learning is to obtain weights between all layers of the networkthat minimize the total error E. In order to achieve this, theweights are adjusted according to the equation

Affordable neuron: 20

80000

Affordable neuron: 40

60000

k−1,k
k−1,k
(m + 1) = wi,j
(m) +
wi,j

Affordable neuron: 60

40000

k−1,k
∆p wi,j(m) = −η

20000

0

0

10

20

30

40

50

60

70

80

90

60

50

50

Number of neuron

Number of neurons

Random selection (Hidden: 100).

60

40

30

20

10

40

79500

80000

80500

20

0
59000

81000

59500

60000

60500

61000

Selected time for operating neuron

Selected time for operating neuron(a) Affordable neuron: 20(b) Affordable neuron: 40.

60

Number of neuron

50

40

30

20

10

0
39000

39500

40000

40500

41000

Selected time for operating neuron(c) Affordable neuron: 60.
Fig. 7.

p=1

k−1,k∆p wi,j
(m),

(3)

,
k−1,k

∂wi,j

where ζ denotes the inertia rate.
IV. S IMULATED R ESULTS
For our simulations we want to teach our network to
generate typical time series of the skew tent map. To this end,
the network is trained – using time series of the tent map –to output, starting from given initial conditions, the same timeseries as the tent map would have generated.
The skew tent map and an example of time series are shownin Fig. 8. The length of chaotic time series is set to 50 steps;
the size of the set of learning patterns is 20. In our approach,this requires the network to have 50 nodes in the input andthe output layers. Each data is inputted to each node in the
input layer. We carried out BP-learning by using the following
parameters. The parameter of the learning rate and the inertia
rate are fixed at η = 0.05 and ζ = 0.02, respectively. The
initial values of the weights are chosen between −1.0 and 1.0
at random. The learning time is set to 5000.

30

10

0
79000

P


k−1,k
where wi,jis the weight between the ith neuron of thek − 1th layer and the jth neuron of the layer k, m is thelearning time, and η is a proportionality factor known as thelearning rate. In this study, we also add an inertia term, whichchanges the second line of Eq.(2) into
∂Ep
k−1,k
k−1,k
(m) = −η k−1,k + ζ∆p wi,j(m − 1), (4)∆p wi,j∂wi,j100

Neuron number of hidden layer

Fig. 6.

∂Ep

Distribution of random selection (Hidden: 100).

1III. BP-L EARNING A LGORITHM

1

where P is the number of the input data, N is the number ofthe neurons in the output layer, t pi denotes the value of thedesired target data for the pth input data, and o pi denotes thevalue of the output data for the pth input data. The goal of the813

x(t+1)

BP is the most common learning algorithm for feedforward neural networks, and the effectiveness of BP-learning
has been confirmed in pattern recognition, system control,
signal processing, and so on [4]-[6]. The standard BP-learningalgorithm was introduced in [3]. In this study, we use the batch
BP-learning algorithm. The batch BP-learning algorithm canbe expressed similarly to the standard BP-learning algorithm,
with the difference lying in the timing of the weights. In
standard BP, the update is performed after each single data
input, whereas for batch BP, the update is performed after all
input data has been processed. The total error E of the networkis thus defined as

 N
P
P


1
Ep =
(tpi − opi )2 ) ,
(2)
E=
2
p=1
p=1
i=1

0.8

0.9
Chaotic time series

Selected time for operating neuron

100000

0.6
0.4
0.2

0

0.2

0.4

0.6

0.8

0.7
0.6
0.5
0.4
0.3
0.2

a=0.55
0

0.8

0.1

1

0

0

5

10

15

x(t)

(a) Skew tent map.
Fig. 8.

20

25 30
Time

35

40

45

50

(b) Time series.
Skew tent map.

A. Learning Ability
First, we investigate the learning efficiency as the averageof the total error between the output and the desired target,
when the network structure of the hidden layer is changed. The“Average Error E ave ” for this learning example is defined by
the following equation.

P 
1  1
2
Eave =(tp − op )(5)
P p=1 2

0.05

scaling rule. We conclude that with the scale-rule characteristics, the performance of affordable neural networks can beboosted considerably.

0.03
Number of neurons

Eave

0.04

random0.02

scale-rule

0.01

35

35

30

30

Number of neurons

scale-rule selection
random selection

25
20
15
10
5
0

25
20
15
10
5

0

1000

2000

3000

4000

5000

0

6000

0

1000

Total adjustment of weight

10

20

conventional
network

30

40

50

60

70

Number of affordable neurons

Fig. 10.

Fig. 9. Learning ability by changing the number of affordable neurons(Number of neurons in hidden layer: 100).

We consider the case that the hidden layer consists of 100
neurons. The number of the affordable neurons is varied from10 to 70. The results of this simulation are shown in Fig. 9,
where the horizontal axes are the number of the affordable
neurons and the vertical axes are E ave for the pattern learning.
From this Figure, we can confirm that the scale-rule selection
method achieves a better performance if compare to therandom selection. It is also seen that the difference between
the errors of the scale-rule and the random selection networks
increases with the number of affordable neurons. Even when
the number of affordable neurons becomes large, the scalerule selection network continues to show good learning ability.
From this result, we can conclude that the scale-rule selection
method of affordable neurons could play an important role forlearning processes, in particular in biological systems.
B. Network Characteristics
In this Section, we compare the characteristics of theaffordable neural network under the scale-rule selection to
the random selection methods. We pay attention to the total
adjustment of the weights of neurons in the hidden layer. The
total adjustment of the weights (T aw ) is defined as the sum of
the absolute values of the adjustments of the weights between
input and output. This can be written as
Taw =

R

r=1

1,2
2,3
|∆ωi,j+ ∆ωi,j|,(a) Affordable neurons: 20.

80

(6)

where R is the updating time. The results of the relationshipbetween total adjustment of the weight and the number of theneurons obtained in our simulations are shown in Figs. 10
and 11. In the case of scale-rule selection, the number of theneurons decreases gradually, by increasing the total adjustmentof the weight (Fig. 10). In contrast, the graph of the random
selection has a peak (Fig. 11). In this way, we have confirmedthat the operation of neurons in the hidden layer of affordable
neural networks with the scale-rule selection follows some

814

4000

5000

6000

Network characteristics of scale-rule selections (Hidden:100).

35

35

30

30

25
20
15
10
5
0

3000

(b) Affordable neurons: 60.

Number of neurons

0

Number of neurons

0

2000

Total adjustment of weight

25
20
15
10
5

0

1000

2000

3000

4000

5000

6000

0

0

1000

Total adjustment of weight(a) Affordable neurons: 20.
Fig. 11.

2000

3000

4000

5000

6000

Total adjustment of weight(b) Affordable neurons: 60.

Network characteristics of random selections (Hidden:100).

V. C ONCLUSIONSIn this study, we investigated the performance of affordable
neural networks when the affordable neurons are selected bya scale-rule. By computer simulations, we confirmed that theaffordable neural network with scale-rule selection achieves
a better performance compared to random selection. Furthermore, we have provided evidence that the operation ofneurons in the hidden layer of affordable neural network withscale-rule selection follows a scaling rule. We thus conclude
that in the context of affordable neural networks, scale-ruleselection of affordable neurons has an important impact on its
performance.
R EFERENCES[1] Barabasi, Albert-Laszlo and A. Reka, “Emergence of Scaling in Random
Networks,” Science, 286:509-512, Oct. 1999.
[2] Y. Uwate and Y. Nishio, “Performance of Affordable Neural Network for
Back Propagation Learning,” IEICE Transactions on Fundamentals, vol.
E89-A, no. 9, pp. 2374-2380, Sep. 2006.
[3] D.E. Rumelhart, G.E. Hinton and R.J. Williams, “Learning Internal
Representations by Error Propagation,” Parallel Distributed Processing,vol. 1, MIT Press, MA, pp. 318-362, 1986.
[4] Y. LeCun, B. Boser, J.S. Denker, et al, “Handwritten Digit Recognition
with a Back-Propagation Network,” Advances in Neural Information
Processing Systems, vol. 2, MIT Press, MA, 1990.
[5] M. Kawano, Y. Uno, M. Isobe and R. Suzuki, “A Hierarchical Neural
Network Model for Voluntary Movement with Application to Robotics,”
IEEE Control Systems Magazine, vol. 8, pp. 8-16, 1988.
[6] T.J. Sejnowski and C.R. Rosenberg, “Parallel Networks that Learn to
Pronounce English Text,” Complex Systems, vol. 1, pp. 145-168, 1987.