2006 International Joint Conference on Neural Networks
Sheraton Vancouver Wall Centre Hotel, Vancouver, BC, Canada
July 16-21, 2006

Durability of Affordable Neural Networks against Damages
Yoko Uwate, Student Member, IEEE, Yoshifumi Nishio, Member, IEEE and Ruedi Stoop, Member, IEEE
Abstract— In this study, we address the durability of thebrain, which is able to operate in various imperfect situations. In our previous research, we have proposed a new
network structure called the “Affordable Neural Network”,
where affordable neurons in the hidden layer of the feedforward
neural network reflect important aspects of the real brain
mechanism. We consider that the concept of the affordable
neural network embodies the important feature of durability. In
our contribution, we investigate the durability of the affordable
neural network when some of the neurons in the hidden layer
are damaged after the learning process.

I. I NTRODUCTION
Recently, studies on the brain have been carried out
actively on various levels. On the neuroscience level, many
researchers investigate how neuronal death occurs in thehuman brain. Researches of neuronal death are help tomake clear the mechanisms of unrevealed diseases such as
Alzheimer disease [1]. Neuronal death can be occurred byvarious but usual causes such as drinking a lot of alcohol,
heavy smoking and concussions. Furthermore, some neurons
die with apoptosis when the neuron can not receive any
signals. Figure 1 shows living neurons and dead neurons.
The living neurons form the multiple axon and dendrite byreceiving signals. However, the neurites are broken down by
the neuronal death and the neuron cells become very small
and dark.
Some neurons often die in the human brain in daily life.
Even if some neurons die, the human brain is still able tooperate normally by creating new information processing
Yoko Uwate and Yoshifumi Nishio are with Dept. of Electrical andElectronics Engineering, Tokushima University, 2-1 Minami-Josanjima,
Tokushima, Japan, (phone: +81-88-656-7470; fax: +81-88-656-7471; email:{uwate, nishio}@ee.tokushima-u.ac.jp).
Ruedi Stoop is with Institute of Neuroinfomatics, University /ETH Zurich, Winterthurerstrasse 190, 8057 Zurich, Switzerland, (email:ruedi@ini.phys.ethz.ch).

from abundant experience and knowledge. This is because,the human brain has durability and flexibility.
Many modelings of the human brain with the visual or theaudio sensation are reported [2]-[4] due to development of
the brain researches. However, the investigation of modelingof higher functions in the human brain such as durability andflexibility is just getting started. We consider that it is very
important to apply these high functional mechanisms of thehuman brain to novel artificial neural networks.
In our previous research, we have proposed a new network structure with affordable neurons in the hidden layer,
for efficient BP learning [5][6]. We named this network“Affordable Neural Network.” In this network, we prepare
some extra neurons in the hidden layer. When the network
executes operating, all of the neurons in the hidden layer are
not used at every updating. Namely, this network is able tooperate with high function by using different neurons in thehidden layer. By computer simulations, the affordable neuralnetwork has been confirmed to gain better performance for
the BP learning on both convergence speed and learning
efficiency. Further, we have investigated the performanceof the affordable neural network when noise-polluted data
is inputted. And we have confirmed that the affordable
neural network is able to output noise-cleaned data. We cansay that the affordable neural network has generalization
ability. However, we believe that many advantages of theaffordable neural network are still veiled. We consider thatthe affordable neural network is able to self-generate network
durability.
In this study, we investigate the durability of the affordable
neural network when some of the neurons in the hidden
layer are damaged, after the learning process. If a neuron is
selected as a damaging neuron, the connections to the output
layer from the neuron are cut. Namely, the damaging neuron
does not operate. By computer simulations, we confirm
that the affordable neural network keeps its efficiency. We
conclude that the affordable neurons exert an important
influence on durability of the network.
II. A FFORDABLE N EURAL N ETWORK
A. Network Model with Affordable Neurons(a) Living neurons.
Fig. 1.

(b) Dead neurons.

Neuron of culture microbe nervus sympathicus.

0-7803-9490-9/06/$20.00/©2006 IEEE

In Refs. [5][6], we have proposed a network with affordable neurons in the hidden layer of the feedforward neuralnetwork structure for efficient BP learning. We introduced
the affordable neurons to reflect important properties of thebrain. During the BP learning, not all of the neurons in thehidden layer are not used at every updating. Namely, someof the neurons are selected for the learning and the restof the neurons are deactivated. The network model and the8365

operation of the affordable neurons in the hidden layer are
shown in Fig. 2(a) and Fig. 2(b), respectively.
Chaos

Chaos

order of the values of their skew tent maps, the selection ismade in a chaotic manner. Note that chaos is not inputted to
the neural network directly, but is used only for the selectionof the operating neurons. The skew tent map is defined by
the following equation.

2xn + 1 − a(−1 ≤ xn ≤ a)1+axn+1 =
 −2xn + 1 + a (a < xn ≤ 1).
1−a

Chaos

Chaos

Chaos

Chaos

Output layer(1)

Input layer

One example of the temporal evolution under chaos selection is shown in Fig. 3. In this example, eight neurons in thehidden layer are prepared and the number of the affordable
neurons are set to two. The horizontal axis is time and thevertical axis is the neuron number in the hidden layer. The
plotted marks indicate the neurons selected as affordable
neurons at each updating. We can see that the affordable
neurons are selected chaotically.

Chaos

Chaos

Affordable
neuronHidden layer

Learning time:1

Neuron number in hidden layer(a) Network model with affordable neurons.

Learning time:2

8
7
6
5
4
3
2
1
0

50

100

150

200

time

Fig. 3.
Learning time:3

Chaotic selection.

Learning time:N
Operating neuron
Affordable neuron (rest neuron)(b) Operation of the affordable neurons in the hidden layer.
Fig. 2.

Affordable neural network.

B. Chaotic Selection
For the proposed network, at every update, some of theneurons have to be selected for the BP learning. In Refs. [7][9], the authors have investigated the performance of the
Hopfield neural network solving combinatorial optimization
problems when chaos is inputted to the neurons as noise. Bycomputer simulations, chaotic noise has been confirmed to
improve performance by being able to escape from local minima much better than random noise. Hence, we consider thatvarious features of chaos are effective for neural networks.
In this study, we use the skew tent map as a simple chaotic
map, to realize a chaotic selection of operating neurons in thehidden layer. We prepare the skew tent maps with different
initial values, whose number is the same as the numberof the neurons in the hidden layer, so that each skew tent
map corresponds to one neuron. At every update of the BP
learning, the skew tent maps are also updated, and theirvalues are referred. By selecting a number of neurons in theIII. BP L EARNING A LGORITHM
The standard BP learning algorithm was introduced in[10]. The BP is the most common learning algorithm for
feedforward neural networks. In this study, we use the batch
BP learning algorithm. The batch BP learning algorithm isexpressed by a formula similar to the standard BP learningalgorithm. The difference lies in the timing of the weight.
The update of the standard BP is performed after each single
input data, while for the batch BP the update is performed
after all input data has been processed. The total error E of
the network is defined as

 N
P
P


1
2
Ep =
(tpi − opi ) ) ,
(2)
E=
2 i=1
p=1
p=1
where P is the number of the input data, N is the number ofthe neurons in the output layer, tpi denotes the value of thedesired target data for the pth input data, and opi denotes thevalue of the output data for the pth input data. The goal of thelearning is to set weights between all layers of the network
so as to minimize the total error E. In order to minimize E,

8366

the weights are adjusted according to the following equation:
k−1,k
k−1,k
(m + 1) = wi,j
(m) +
wi,j
k−1,k
∆p wi,j(m) = −η

∂Ep

P

p=1

k−1,k
∆p wi,j
(m),

(3)

,
k−1,k

∂wi,j

k−1,k
where wi,jis the weight between the ith neuron of thelayer k−1 and the jth neuron of the layer k, m is the learning
time, and η is a proportionality factor known as the learning
rate. In this study, we add to the second line of Eq.(3) aninertia term, leading tok−1,k(m) = −η∆p wi,j

∂Ep
k−1,k
∂wi,j

k−1,k
+ ζ∆p wi,j(m − 1), (4)where ζ denotes the inertia rate.
IV. DAMAGING N EURONS
Even if some neurons of the human brain die, it is still
able to operate normally in most cases. This durability is oneof the most important features of the brain. In the following,
we numerically investigate the durability of the affordable
neural network.
We assume that some neurons in the hidden layer are damaged by some causes after the BP learning. The connections
to the output layer of the damaged neurons are cut as shownin Fig. 4. Namely, the damaged neurons do not operate. In
this situation, we investigate the performance of the network
when the learning data are inputted to the network.
After learning
DamagedneuronDamagedneuronOutput layer

Input layer
Hidden layer
Fig. 4.

Damaging neurons.

V. S IMULATED R ESULTS
We consider the feedforward neural network for the task to
produce output x2 for input x, as one learning example. The
sampling range of the input data is [−1.0, 1.0] and the step
size of the input data is set to be 0.01. Our BP learning was

based on the fallowing parameters. The learning rate and theinertia rate are fixed as η = 0.1 and ζ = 0.02, respectively,
and initial values of the weights are given between −1.0 and
1.0 at random. The learning time is set to m = 20000. We
investigate the total error between the output and the desired
target when some neurons are damaged after learning. We
define “Average Error Eave ” by the following equation.
P 
1  1
(tpi − opi )2 ) .
Eave =(5)
P p=1 2In this study, we prepare from 8 to 11 neurons in thehidden layer of the network and the number of the affordable
neurons is set to 1 to 3. For example, the number ofneurons in the hidden layer is 8 and the number of theaffordable neurons is set to 2. In this case, only 6 neurons areoperated every time. This network is denoted as “affordable
neural network (8-2).” For comparison, we investigate theperformance of the conventional neural network without any
affordable neurons.
A. Effect of Damaged Neurons
The simulation results of the affordable neural networkand the conventional neural network when some neurons in
the hidden layer are damaged after learning are shown in
Fig. 5. Figures 5 (a), (b), (c) and (d) show the simulation
results for the cases that the numbers of the neurons in thehidden layer are prepared as 8, 9, 10 and 11, respectively.
The horizontal axis is the number of the damaged neuron
in the hidden layer and the vertical axis is the average ofEave for all combination of the damaged neurons. The Eaveof both the affordable neural network and the conventional
neural network becomes worse by increasing the numberof the damaged neurons. By comparison of two networks,
the affordable neural networks gain better performance thanthe conventional neural network when some neurons aredamaged. Because the Eave of the affordable neural networks
is about half the conventional neural networks. From theseresults, we confirmed that the affordable neural networks canoperate well even if some neurons are damaged.
Some examples of outputs of the affordable neural networkand the conventional neural network before and after someneurons are damaged are shown in Figs. 6-9. The left figure
shows the output before some neurons are damaged. The
right figure shows the output after some neurons are damaged. In the case of the affordable neural network, the effectof damaging neuron is small and the network gains good
performance as before. While the case of the conventional
neural network, the effect of damaging neuron is very large
and the network does not operate well.
We consider that the learning process using different patterns of operating neurons in the affordable neural network
prevents the neurons playing fixed roles and creates the
durability of the whole network.
B. Position of damaging neurons
Next, we investigate the relationship between performanceand position of the damaging neurons. The simulated results

8367

0.35

0.35
Affordable NN (8-1)
Affordable NN (8-2)
Affordable NN (8-3)
Conventional NN (8-0)

0.3

0.25

0.2

Eave

Eave

0.25

0.2

0.15

0.15

0.1

0.1

0.05

0.05

0

0

1

0

3

2

Affordable NN (9-1)
Affordable NN (9-2)
Affordable NN (9-3)
Conventional NN (9-0)

0.30Number of damaged neurons(a) Hidden layer: 8.
0.35
Affordable NN (10-1)
Affordable NN (10-2)
Affordable NN (10-3)
Conventional NN (10-0)

0.3
0.25

0.25

0.2

0.2

0.15

0.15

0.1

0.1

0.05

0.05
0

1
2
Number of damaged neurons

Affordable NN (11-1)
Affordable NN (11-2)
Affordable NN (11-3)
Conventional NN (11-0)

0.3

Eave

Eave

3

(b) Hidden layer: 9.

0.35

0

1
2
Number of damaged neurons

0

3

0

(c) Hidden layer: 10.

1
2
Number of damaged neurons

3(d) Hidden layer: 11.
Fig. 5.

Eave when some neurons are damaged.

when the damaged neurons is only one are shown in Fig. 10.
The horizontal axis is the position of the damaged neuron in
the hidden layer and the vertical axis is Eave . The solid line
denotes the affordable neural network and the dashed line
denotes the conventional neural network. We emphasize thatthe vertical axis in Fig. 10 is denoted in log scale. Fromthese figures, Eave of the conventional neural network is
not good. And the performance of the conventional neuralnetwork does not depend on the position of the damaged
neurons. On the other hand, Eave of the affordable neuralnetwork is almost very small when any neuron in the hidden
layer is damaged. For example, the case of the number ofthe neurons in the hidden layer is set to 11 (Fig. 10 (d)), theEave of the affordable neural network is less than one tenthor 1/100 of the Eave of the conventional neural network formany positions.
We confirm that the conventional neural network is notable to operate when any neuron is damaged. However, theaffordable neural network keeps the same performance as
the network before some neurons damaged. This is because,

the affordable neural network generates durability during BP
learning.
VI. C ONCLUSIONSIn this study, we investigated the durability of the affordable neural network when some of the neurons in the hidden
layer are damaged after the learning process. By computer
simulations, we confirmed that the affordable neural network
keeps its good performance. It is obvious that the affordable
neurons exert an important influence on durability of the
network. Further, we investigated the performance of chaos,
regular and random selection methods of affordable neurons
and found superior behavior of the chaotic and random
choice over regular choice.
We believe thus the chaotic selection exerts an important
beneficial effect on the network’s learning in the hidden layer.
In future work, we will investigate the efficacy of chaotic
selection for more difficult problems in more complicatedsituations.

8368

1
Teach
Output (before damage)

0.8

1
Teach
Output (after damage)

0.8

0.8

0.6

x2

0.6

0.4

0.4

0.4

0.4

0.2

0.2

0.2

0.2

0
-1

-0.5

0

0.5

0
-1

1

-0.5

0

x

0.5

0
-1

1

-0.5

x

(1) Before damage.

(2) After damage.

0.4

0.2

0.2

0.2

0
x

0.5

1

(1) Before damage.

1

-0.5

0
x

0.5

0
-1

1

Teach
Output (after damage)

0.6
x2

x2

x2
0.4

-0.5

0.5

0.8

0.6
0.4

0
-1

0
x

1
Teach
Output (before damage)

0.8

Teach
Output (after damage)

0.6

0
-1

-0.5

(2) After damage.

1

0.8

0.6

0
-1

1

x2

0.8

0.5

(a) Affordable NN (10-1).

1
Teach
Output (before damage)0x(1) Before damage.

(a) Affordable NN (8-1).
1

Teach
Output (after damage)

0.8

0.6

x2

x2

0.6

1
Teach
Output (before damage)

x2

1

-0.5

0
x

0.5

0.4
0.2
0
-1

1

(1) Before damage.

(2) After damage.

-0.5

0
x

0.5

1

(2) After damage.

(b) Conventional NN (8-0).

(b) Conventional NN (10-0).

Fig. 6. Output of the network before and after one neuron is damaged(Hidden: 8).

Fig. 8. Output of the network before and after one neuron is damaged(Hidden: 10).

1

1
Teach
Output (before damage)

0.8

Teach
Output (after damage)

0.8

0.8

x2

x2

0.6

0.4

0.4

0.4

0.4

0.2

0.2

0.2

0.2

0
-1

-0.5

0
x

0.5

0
-1

1

(1) Before damage.

-0.5

0
x

0.5

0
-1

1

-0.5

1

1
0.8
0.6

x2

Teach
Output (after damage)

0.6
0.4

0.2

0.2

0.2

0.2

0.5

(1) Before damage.

1

0
-1

1

0.8

0.4

0
x

0.5

1
Teach
Output (before damage)

0.4

-0.5

0
x

(2) After damage.

0.4

0
-1

-0.5

x2

0.6

x2

0.6

0
-1

1

x2

Teach
Output (before damage)

0.5(a) Affordable NN (11-1).

0.8

1

0
x

(1) Before damage.

(2) After damage.

(a) Affordable NN (9-1).

0.8

Teach
Output (after damage)

0.8

0.6

0.6

x2

0.6

1
Teach
Output (before damage)

x2

1

-0.5

0
x

0.5

0
-1

1

-0.5

0
x

0.5

(1) Before damage.

(2) After damage.

1

0
-1

Teach
Output (after damage)

-0.5

0
x

0.5

1

(2) After damage.

(b) Conventional NN (9-0).

(b) Conventional NN (11-0).

Fig. 7. Output of the network before and after one neuron is damaged(Hidden: 9).

Fig. 9. Output of the network before and after one neuron is damaged(Hidden: 11).

8369

1

R EFERENCES

Affordable NN (8-1)
Conventional NN (8-0)

Eave

0.1

0.01

0.001

0.0001

1

2

3

4
5
Position of damaged neuron

6

7

8

(a) Hidden layer: 8.
1
Affordable NN (9-1)
Conventional NN (9-0)

Eave

0.1

0.01

0.001

0.0001

1

2

3

4
5
6
Position of damaged neuron

7

8

9

(b) Hidden layer: 9.
1
Affordable NN (10-1)
Conventional NN (10-0)[1] H. Satoi, H. Tomimoto, R. Ohtani, T. Kitano, T. Kondo, M. Watanabe,
N. Oka, I. Akiguchi, S. Furuta, Y. Hirabayashi and T. Okazaki,“Astroglial Expression of Ceramide in Alzheimer’s Disease Brains: A
Role During Neuronal Apoptosis,” Neuroscience, vol. 130, pp. 657-666,
2005.
[2] R. P. Broussard, S. K. Rogers, M. E. Oxley and G. L. Tarr, “Physiologically Motivated Image Fusion for Object Detection Using a Pulse
Coupled Neural Network,” IEEE Transactions on Neural Networks, 10(3), pp. 554-563, 1999.
[3] R. Stoop and A. Kern, “Two-Tone Suppression and Combination Tone
Generation as Computations Performed by the Hopf Cochlea,” Phys.
Rev. Lett., 93 (26) 268103, 2004.
[4] R. Stoop, W.-H. Steeb, J.C. Gallas and A. Kern, “Auditory Two-Tone
Suppression from a Subcritical Hopf Cochlea,” Physica A, 351: pp.
175-183, 2005.
[5] Y. Uwate and Y. Nishio, “Back Propagation Learning of an Affordable Neural Network for Chaotic Time Series,” Proceedings of RISPInternational Workshop on Nonlinear Circuits and Signal Processing(NCSP’05), pp. 139-142, Mar. 2005.
[6] Y. Uwate and Y. Nishio, “Back Propagation Learning of Neural Networks with Chaotically-Selected Affordable Neurons,” IEEE International Symposium on Circuits and Systems (ISCAS’05), pp. 1481-1484,
May. 2005.
[7] T. Ueta, Y. Nishio and T. Kawabe, “Comparison between Chaotic Noiseand Burst Noise on Solving Ability of Hopfield Neural Networks,” Proc.
NOLTA’97, vol. 1, pp. 409-412, Nov. 1997.
[8] Y. Uwate, Y. Nishio and A. Ushida, ”Markov Chain Modeling ofIntermittency Chaos and its Application to Hopfield NN,” IEICE
Transactions on Fundamentals, vol. E87-A, no. 4, pp. 774-779, Apr.
2004.
[9] Y. Uwate, Y. Nishio, T. Ueta, T. Kawabe and T. Ikeguchi, “Performanceof Chaos and Burst Noise Injected to the Hopfield NN for Quadratic
Assignment Problems,” IEICE Transactions on Fundamentals, vol. E87A, no. 4, pp. 937-943, Apr. 2004.
[10] D.E. Rumelhart, G.E. Hinton and R.J. Williams, “Learning Internal
Representations by Error Propagation,” Parallel Distributed Processing,vol. 1, MIT Press, MA, pp. 318-362, 1986.

Eave

0.1

0.01

0.001

0.0001

1

2

3

4
5
6
7
Position of damaged neuron

8

9

10

(c) Hidden layer: 10.
1
Affordable NN (11-1)
Conventional NN (11-0)

Eave

0.1

0.01

0.001

0.00012

4

6
8
Position of damaged neuron

10(d) Hidden layer: 11.
Fig. 10.

Eave and position of damaged neurons.

8370