Durability of Aﬀordable Neural Networks
during Back Propagation Learning
Yoko Uwate† , Yoshifumi Nishio† and Ruedi Stoop‡† Dept. of Electrical and Electronics Engineering, Tokushima University

2-1 Minami-Josanjima, Tokushima, Japan
Email: {uwate, nishio}@ee.tokushima-u.ac.jp‡ ‡ Institute of Neuroinfomatics, University / ETH Zurich
Winterthurerstrasse 190, 8057 Zurich, Switzerland
Email: ruedi@ini.phys.ethz.ch
1

Introduction

Recently, studies on the brain have been carriedout actively on various levels. On the neuroscience level, many researchers investigate how
neuronal death occurs in the human brain. Researches of neuronal death are help to make clear
the mechanisms of unrevealed diseases such as
Alzheimer disease [1]. Neuronal death can be occurred by various but usual causes such as drinking a lot of alcohol, heavy smoking and concussions. Furthermore, some neurons die with apoptosis when the neuron can not receive any signals.
Some neurons often die in the human brain in
daily life. Even if some neurons die, the humanbrain is still able to operate normally by creating
new information processing from abundant experience and knowledge. This is because, the humanbrain has durability and ﬂexibility.
Many modelings of the human brain with thevisual or the audio sensation are reported [2]-[4]due to development of the brain researches. However, the investigation of modeling of higher functions in the human brain such as durability andﬂexibility is just getting started. We consider that
it is very important to apply these high functional
mechanisms of the human brain to novel artiﬁcial
neural networks.
In our previous research, we have proposed a
new network structure with aﬀordable neurons in
the hidden layer, for eﬃcient BP learning [5][6].
We named this network “Aﬀordable Neural Network.” In this network, we prepare some extra
neurons in the hidden layer. When the network
executes operating, all of the neurons in the hidden layer are not used at every updating. Namely,
this network is able to operate with high function

by using diﬀerent neurons in the hidden layer. By
computer simulations, the aﬀordable neural network has been conﬁrmed to gain better performance for the BP learning on both convergence
speed and learning eﬃciency. Further, we have investigated the performance of the aﬀordable neural network when noise-polluted data is inputted.
And we have conﬁrmed that the aﬀordable neuralnetwork is able to output noise-cleaned data. We
can say that the aﬀordable neural network has
generalization ability. However, we believe thatmany advantages of the aﬀordable neural network
are still veiled. We consider that the aﬀordable
neural network is able to self-generate network
durability.
In this study, we investigate the durability of
the aﬀordable neural network when some of theneurons in the hidden layer are damaged, after
the learning process. If a neuron is selected asa damaging neuron, the connections to the output layer from the neuron are cut. Namely, thedamaging neuron does not operate. By computer
simulations, we conﬁrm that the aﬀordable neural network keeps its eﬃciency. We conclude thatthe aﬀordable neurons exert an important inﬂuence on durability of the network.
2
2.1

Aﬀordable Neural Network
Network Model with Aﬀordable
Neurons

In Refs. [5][6], we have proposed a network withaﬀordable neurons in the hidden layer of the feedforward neural network structure for eﬃcient BP
learning. We introduced the aﬀordable neurons toreﬂect important properties of the brain. During
the BP learning, not all of the neurons in the hid-den layer are not used at every updating. Namely,
some of the neurons are selected for the learningand the rest of the neurons are deactivated. The
network model and the operation of the aﬀordable
neurons in the hidden layer are shown in Fig. 1(a)and Fig. 1(b), respectively.
Chaos

Chaos

Chaos

Chaos

Chaos

Chaos

Output layer

Input layer
Chaos

Chaos

simple chaotic map, to realize a chaotic selectionof operating neurons in the hidden layer. We prepare the skew tent maps with diﬀerent initial values, whose number is the same as the numberof the neurons in the hidden layer, so that eachskew tent map corresponds to one neuron. At
every update of the BP learning, the skew tent
maps are also updated, and their values are referred. By selecting a number of neurons in theorder of the values of their skew tent maps, theselection is made in a chaotic manner. Note thatchaos is not inputted to the neural network directly, but is used only for the selection of theoperating neurons. The skew tent map is deﬁned
by the following equation.

Affordable
neuronHidden layer

xn+1 =(a) Network model with aﬀordable neurons.

⎧
2xn + 1 − a⎪⎪⎨(−1 ≤ xn ≤ a)−2xn + 1 + a⎪⎪⎩(a < xn ≤ 1).

1+a

1−a

Learning time:1

Learning time:3

Learning time:2

Learning time:N
Operating neuron
Affordable neuron (rest neuron)(1)

One example of the temporal evolution underchaos selection is shown in Fig. 2. In this example, eight neurons in the hidden layer are preparedand the number of the aﬀordable neurons are set
to two. The horizontal axis is time and the vertical axis is the neuron number in the hidden layer.
The plotted marks indicate the neurons selected
as aﬀordable neurons at each updating. We cansee that the aﬀordable neurons are selected chaotically.

Neuron number in hidden layer(b) Operation of the aﬀordable neurons in thehidden layer.
Figure 1: Aﬀordable neural network.
2.2

Chaotic Selection

For the proposed network, at every update, someof the neurons have to be selected for the BP
learning. In Refs. [7]-[9], the authors have investigated the performance of the Hopﬁeld neuralnetwork solving combinatorial optimization problems when chaos is inputted to the neurons as
noise. By computer simulations, chaotic noise
has been conﬁrmed to improve performance by
being able to escape from local minima much better than random noise. Hence, we consider thatvarious features of chaos are eﬀective for neuralnetworks.
In this study, we use the skew tent map as a

8
7
6
5
4
3
2
1
0

50

100

150

200

time

Figure 2: Chaotic selection.
3

BP Learning Algorithm

The standard BP learning algorithm was introduced in [10]. The BP is the most common learning algorithm for feedforward neural networks. In
this study, we use the batch BP learning algorithm. The batch BP learning algorithm is expressed by a formula similar to the standard BP

learning algorithm. The diﬀerence lies in the timing of the weight. The update of the standard BP
is performed after each single input data, while forthe batch BP the update is performed after all input data has been processed. The total error Eof the network is deﬁned as
E=

P

p=1

Ep =

P

p=1



k−1,k
k−1,k
(m + 1) = wi,j
(m) +
wi,j

∂Ep

(2)

k−1,k
∂wi,j

P

p=1

k−1,k
∆p wi,j
(m),

,

(3)
k−1,k
is the weight between the ith neuron
where wi,jof the layer k−1 and the jth neuron of the layer k,
m is the learning time, and η is a proportionality
factor known as the learning rate. In this study,
we add to the second line of Eq.(3) an inertia
term, leading tok−1,k(m) = −η∆p wi,j

∂Ep

k−1,k
∂wi,j

DamagedneuronN1
(tpi − opi )2 ) ,2 i=1

where P is the number of the input data, N is
the number of the neurons in the output layer, tpi
denotes the value of the desired target data forthe pth input data, and opi denotes the value ofthe output data for the pth input data. The goalof the learning is to set weights between all layersof the network so as to minimize the total error E.
In order to minimize E, the weights are adjusted
according to the following equation:k−1,k(m) = −η∆p wi,jAfter learningk−1,k+ ζ∆p wi,j(m − 1),

DamagedneuronHidden layer

Figure 3: Damaging neurons.
5

Simulated Results

We consider the feedforward neural network forthe task to produce output x2 for input x, as one
learning example. The sampling range of the input data is [−1.0, 1.0] and the step size of the
input data is set to be 0.01. Our BP learning was
based on the fallowing parameters. The learning
rate and the inertia rate are ﬁxed as η = 0.1 andζ = 0.02, respectively, and initial values of theweights are given between −1.0 and 1.0 at random. The learning time is set to m = 20000.
We investigate the total error between the output and the desired target when some neurons
are damaged after learning. We deﬁne “Average
Error Eave ” by the following equation.

(4)



where ζ denotes the inertia rate.
4

Eave =

Damaging Neurons

Even if some neurons of the human brain die, it isstill able to operate normally in most cases. This
durability is one of the most important featuresof the brain. In the following, we numerically investigate the durability of the aﬀordable neuralnetwork.
We assume that some neurons in the hidden
layer are damaged by some causes after the BP
learning. The connections to the output layer ofthe damaged neurons are cut as shown in Fig. 3.
Namely, the damaged neurons do not operate. In
this situation, we investigate the performance ofthe network when the learning data are inputtedto the network.

Output layer

Input layerP
1 1(tpi − opi )2 ) .
P p=1 2

(5)

In this study, we prepare from 8 to 11 neurons
in the hidden layer of the network and the numberof the aﬀordable neurons is set to 1 to 3. For example, the number of neurons in the hidden layer
is 8 and the number of the aﬀordable neurons is
set to 2. In this case, only 6 neurons are operated
every time. This network is denoted as “aﬀordable neural network (8-2).” For comparison, weinvestigate the performance of the conventional
neural network without any aﬀordable neurons.
5.1

Eﬀect of Damaged Neurons

The simulation results of the aﬀordable neural network and the conventional neural network

0.35

0.35
Affordable NN (8-1)
Affordable NN (8-2)
Affordable NN (8-3)
Conventional NN (8-0)

0.3

0.25

0.2

Eave

Eave

0.25

0.2

0.15

0.15

0.1

0.1

0.05

0.05

0

0

1

2

Affordable NN (9-1)
Affordable NN (9-2)
Affordable NN (9-3)
Conventional NN (9-0)

0.3

0

3

0

Number of damaged neurons(a) Hidden layer: 8.
0.35
Affordable NN (10-1)
Affordable NN (10-2)
Affordable NN (10-3)
Conventional NN (10-0)

0.3
0.25

0.25

0.2

0.2

0.15

0.15

0.1

0.1

0.05

0.05
0

1
2
Number of damaged neurons

Affordable NN (11-1)
Affordable NN (11-2)
Affordable NN (11-3)
Conventional NN (11-0)

0.3

Eave

Eave

3

(b) Hidden layer: 9.

0.35

0

1
2
Number of damaged neurons

3(c) Hidden layer: 10.

0

0

1
2
Number of damaged neurons

3(d) Hidden layer: 11.

Figure 4: Eave when some neurons are damaged.
when some neurons in the hidden layer are damaged after learning are shown in Fig. 4. Figures
4 (a), (b), (c) and (d) show the simulation results
for the cases that the numbers of the neurons in
the hidden layer are prepared as 8, 9, 10 and 11,
respectively. The horizontal axis is the number ofthe damaged neuron in the hidden layer and thevertical axis is the average of Eave for all combination of the damaged neurons. The Eave ofboth the aﬀordable neural network and the conventional neural network becomes worse by increasing the number of the damaged neurons. By
comparison of two networks, the aﬀordable neuralnetworks gain better performance than the conventional neural network when some neurons aredamaged. Because the Eave of the aﬀordable neural networks is about half the conventional neuralnetworks. From these results, we conﬁrmed thatthe aﬀordable neural networks can operate welleven if some neurons are damaged.
Some examples of outputs of the aﬀordable neural network and the conventional neural network
before and after some neurons are damaged areshown in Figs. 5 and 6. The left ﬁgure shows theoutput before some neurons are damaged. The
right ﬁgure shows the output after some neurons
are damaged. In the case of the aﬀordable neuralnetwork, the eﬀect of damaging neuron is small
and the network gains good performance as before. While the case of the conventional neuralnetwork, the eﬀect of damaging neuron is verylarge and the network does not operate well.
We consider that the learning process using different patterns of operating neurons in the aﬀordable neural network prevents the neurons playing
ﬁxed roles and creates the durability of the wholenetwork.

Position of damaging neurons
1

1
Teach
Output (before damage)

0.8

x2

0.6

x2

0.6
0.4

0.4

0.2
0
-1

0.2

-0.5

0

0.5

-0.5

0

x

0.5

1

x

(1) Before damage.

(2) After damage.

(a) Aﬀordable NN (8-1).
1

1
Teach
Output (before damage)

0.8

0.8
Teach
Output (after damage)

0.6
x2

0.6
0.4

0.4

0.2
0
-1

0.2

-0.5

0
x

0.5

0
-1

1

(1) Before damage.

-0.5

0
x

0.5

1

(2) After damage.

(b) Conventional NN (8-0).
Figure 5: Output of the network before and after
one neuron is damaged (Hidden: 8).

1

1
Teach
Output (before damage)

0.8

Teach
Output (after damage)

0.8
0.6
x2

0.6
0.4

Conclusions

0.4

0
-1

0.2

-0.5

0
x

0.5

0
-1

1

(1) Before damage.

-0.5

0
x

0.5

1

(2) After damage.

(a) Aﬀordable NN (10-1).
1

1
Teach
Output (before damage)

0.8

0.8
Teach
Output (after damage)

0.6
x2

0.6
x2

In this study, we investigated the durability of
the aﬀordable neural network when some of theneurons in the hidden layer are damaged after
the learning process. By computer simulations,
we conﬁrmed that the aﬀordable neural network
keeps its good performance. It is obvious thatthe aﬀordable neurons exert an important inﬂuence on durability of the network. Further, we investigated the performance of chaos, regular andrandom selection methods of aﬀordable neurons
and found superior behavior of the chaotic andrandom choice over regular choice.
We believe thus the chaotic selection exerts animportant beneﬁcial eﬀect on the network’s learning in the hidden layer. In future work, we will investigate the eﬃcacy of chaotic selection for morediﬃcult problems in more complicated situations.

0
-1

1

0.2

6

Teach
Output (after damage)

0.8

x2

Next, we investigate the relationship between performance and position of the damaging neurons.
The simulated results when the damaged neurons
is only one are shown in Fig. 7. The horizontal axis is the position of the damaged neuron
in the hidden layer and the vertical axis is Eave .
The solid line denotes the aﬀordable neural network and the dashed line denotes the conventional
neural network. We emphasize that the verticalaxis in Fig. 7 is denoted in log scale. From theseﬁgures, Eave of the conventional neural network
is not good. And the performance of the conventional neural network does not depend on theposition of the damaged neurons. On the other
hand, Eave of the aﬀordable neural network is almost very small when any neuron in the hidden
layer is damaged. For example, the case of thenumber of the neurons in the hidden layer is set
to 11 (Fig. 7 (d)), the Eave of the aﬀordable neural network is less than one tenth or 1/100 of the
Eave of the conventional neural network for many
positions.
We conﬁrm that the conventional neural network is not able to operate when any neuron isdamaged. However, the aﬀordable neural network
keeps the same performance as the network before
some neurons damaged. This is because, the affordable neural network generates durability during BP learning.

x2

5.2

0.4

0.4

0.2
0
-1

0.2

-0.5

0
x

0.5

(1) Before damage.

1

0
-1

-0.5

0
x

0.5

1

(2) After damage.

(b) Conventional NN (10-0).
Figure 6: Output of the network before and after
one neuron is damaged (Hidden: 10).

References
1[1] H. Satoi, H. Tomimoto, R. Ohtani, T. Kitano,
T. Kondo, M. Watanabe, N. Oka, I. Akiguchi,
S. Furuta, Y. Hirabayashi and T. Okazaki, “Astroglial Expression of Ceramide in Alzheimer’s
Disease Brains: A Role During Neuronal Apoptosis,” Neuroscience, vol. 130, pp. 657-666, 2005.

Affordable NN (8-1)
Conventional NN (8-0)

Eave

0.1

0.01

0.001

0.0001

1

2

3

4
5
Position of damaged neuron

6

7

8

(a) Hidden layer: 8.

[3] R. Stoop and A. Kern, “Two-Tone Suppression
and Combination Tone Generation as Computations Performed by the Hopf Cochlea,” Phys.
Rev. Lett., 93 (26) 268103, 2004.

1
Affordable NN (9-1)
Conventional NN (9-0)
0.1

Eave

[2] R. P. Broussard, S. K. Rogers, M. E. Oxley and
G. L. Tarr, “Physiologically Motivated Image Fusion for Object Detection Using a Pulse Coupled
Neural Network,” IEEE Transactions on Neural
Networks, 10 (3), pp. 554-563, 1999.

[4] R. Stoop, W.-H. Steeb, J.C. Gallas and A. Kern,“Auditory Two-Tone Suppression from a Subcritical Hopf Cochlea,” Physica A, 351: pp. 175-183,
2005.

0.01

0.001

0.0001

1

2

3

4
5
6
Position of damaged neuron

7

8

9

(b) Hidden layer: 9.
1
Affordable NN (10-1)
Conventional NN (10-0)[6] Y. Uwate and Y. Nishio, “Back Propagation
Learning of Neural Networks with ChaoticallySelected Aﬀordable Neurons,” IEEE International Symposium on Circuits and Systems (ISCAS’05), pp. 1481-1484, May. 2005.

Eave

0.1

0.01

0.001

0.0001

1

2

3

4
5
6
7
Position of damaged neuron

8

9

10

(c) Hidden layer: 10.
1
Affordable NN (11-1)
Conventional NN (11-0)

Eave

0.1

[7] T. Ueta, Y. Nishio and T. Kawabe, “Comparison
between Chaotic Noise and Burst Noise on Solving Ability of Hopﬁeld Neural Networks,” Proc.
NOLTA’97, vol. 1, pp. 409-412, Nov. 1997.
[8] Y. Uwate, Y. Nishio and A. Ushida, ”Markov
Chain Modeling of Intermittency Chaos and its
Application to Hopﬁeld NN,” IEICE Transactions on Fundamentals, vol. E87-A, no. 4, pp.
774-779, Apr. 2004.
[9] Y. Uwate, Y. Nishio, T. Ueta, T. Kawabe andT. Ikeguchi, “Performance of Chaos and Burst
Noise Injected to the Hopﬁeld NN for Quadratic
Assignment Problems,” IEICE Transactions on
Fundamentals, vol. E87-A, no. 4, pp. 937-943,
Apr. 2004.

0.01

0.001

0.0001[5] Y. Uwate and Y. Nishio, “Back Propagation
Learning of an Aﬀordable Neural Network for
Chaotic Time Series,” Proceedings of RISP International Workshop on Nonlinear Circuits andSignal Processing (NCSP’05), pp. 139-142, Mar.
2005.

2

4

6
8
Position of damaged neuron

10(d) Hidden layer: 11.
Figure 7: Eave and position of damaged neurons.

[10] D.E. Rumelhart, G.E. Hinton and R.J. Williams,“Learning Internal Representations by Error
Propagation,” Parallel Distributed Processing,vol. 1, MIT Press, MA, pp. 318-362, 1986.