IEICE TRANS. FUNDAMENTALS, VOL.E92–A, NO.2 FEBRUARY 2009

585

PAPER

Durability of Aﬀordable Neural Networks against Damaging
Neurons
Yoko UWATE†,††a) , Yoshifumi NISHIO† , Members, and Ruedi STOOP†† , Nonmember

SUMMARY
Durability describes the ability of a device to operateproperly in imperfect conditions. We have recently proposed a novel neural network structure called an “Aﬀordable Neural Network” (AfNN), in
which aﬀordable neurons of the hidden layer are considered as the elements
responsible for the robustness property as is observed in human brain function. Whereas earlier we have shown that AfNNs can still generalize andlearn, here we show that these networks are robust against damages occurring after the learning process has terminated. The results support the viewthat AfNNs embody the important feature of durability. In our contribution,
we investigate the durability of the AfNN when some of the neurons in thehidden layer are damaged after the learning process.
key words: aﬀordable neural networks, durability, back propagation
learning

1.

Introduction

An important issue (particularly in Neuroscience) is the understanding of how the mammalian brain is aﬀected in its
performance by the omnipresent death of some of their elements. This very question is likely to be relevant in understanding the main diﬀerences between the commonly usedartificial neural networks structures and biological neuralnetworks. Such an approach could be helpful in explaining
why the performance of artificial neural networks still lags
behind that of biological networks in many respects. Insights into how neuronal death aﬀects brain function in detail could be helpful in revealing the mechanisms underlying
brain diseases such as Alzheimers’ [1]. Biological neuronal
death can be triggered by diﬀerent reasons such as apoptosis(i.e., when the neuron does not receive a suﬃcient amountof signals), abuse of alcohol, nicotine and concussion.
When neurons die, the human brain is generally still
able to operate normally, because the human brain has boththe properties of durability and flexibility. However, few
attempts have been made in order to systematically incorporate the corresponding properties into the design of artificial neural networks. In order to explore this topic, werecently designed a new network paradigm that relies on affordable neurons in the hidden layer [2]. Aﬀordable neurons are extra neurons that are only occasionally included in
Manuscript received December 10, 2007.
Manuscript revised June 2, 2008.
†
The authors are with the Department of Dept. of Electricaland Electronics Engineering, Tokushima University, Tokushimashi, 770-8506 Japan.
††
The authors are with Institute of Neuroinfomatics, University/ETH Zurich, Winterthurerstrasse 190, 8057 Zurich, Switzerland.
a) E-mail: uwate@ee.tokushima-u.ac.jp
DOI: 10.1587/transfun.E92.A.585

the learning process (otherwise they are by-passed by temporally removing their connections). After learning the affoardable neurons are fully integrated into the network. In
a first step of exploring the potential of aﬀordable neurons,
we demonstrated that in a BP network, eﬃcient learning canbe achieved even in the presence of aﬀordable neurons [2].
In fact, it was observed that the performance of this AfNN
was generally superior to the performance of the normal BP
network, with respect to convergence speed and error size.
For such networks with aﬀordable neurons we coined theterm “Aﬀordable Neural Network.”
Here we extend these investigations with the hope toreveal some more advantages of AfNNs. In particular, we
explore to what extent AfNNs exhibit the property of durability after the learning has terminated (or: the major learning phase has terminated, if one insists on continuous learning). We study the eﬀect of two diﬀerent types of damage.
First, we consider that the connections of the damaged neurons in the hidden layer to the output layer are cut. As analternative, we consider that the damaged neurons relay random values to the output layer. As we will demonstrate, theAfNN maintains its eﬃciency with respect to both types ofdamage. From this we conclude that aﬀordable neurons are
an important element for the durability of artificial and biological neural networks.
2.

Aﬀordable Neural Network

2.1 The Network Model
For our investigations, we use [2] the standard three-layered
feedforward neural network. The output characteristics ofthe neurons are implemented by a sigmoid function and asubset of the neurons are tagged as aﬀordable neurons. For
convenience, the network model and the operation of theaﬀordable neurons in the hidden layer are shown in Fig. 1(a)and Fig. 1(b), respectively.
In the proposed BP learning network, at every input
pattern, a subset of the aﬀordable neurons is randomly chosen and silenced by cutting oﬀ their output. The temporal evolution that this network now undergoes is exhibitedin Fig. 2, where the hidden layer consists of eight neurons.
Out of these eight neurons, two are randomly selected andare silenced at each learning step (dots in Fig. 2).
In this study, we use the batch BP learning algorithm.
This learning algorithm can be cast in a formula similar tothe standard [3] BP learning algorithm. The diﬀerence lies

c 2009 The Institute of Electronics, Information and Communication Engineers
Copyright IEICE TRANS. FUNDAMENTALS, VOL.E92–A, NO.2 FEBRUARY 2009

586

E=

P

p=1

⎧ N
⎫
P ⎪ 
⎪

⎪
⎪
⎨1
2⎬
Ep =
(t
−
o
)
,
⎪
⎪
pi
pi
⎪2
⎪
⎩
⎭
p=1

(1)

i=1

where P is the number of the input data, N is the number ofthe neurons in the output layer, t pi denotes the value of thedesired target data for the pth input data, and o pi denotes thevalue of the output data for the pth input data. The goal of
the learning is to set the weights between the neurons so as
to minimize the total error E. In order to minimize E, theweights are adjusted according to:k−1,kwk−1,ki, j (m + 1) = wi, j (m)P+Δ p wk−1,ki, j (m),p=1

Δ p wk−1,ki, j (m) = −η∂E p∂wk−1,k
i, j

(2)

,

is the weight between the ith neuron of the layer
were wk−1,ki, j
k − 1 and the jth neuron of the layer k. m is the learning
time, and η is a proportionality factor known as the learning
rate. In this study, to the third line of Eq. (2) an inertia term
was added, which leads to
Δ p wk−1,ki, j (m) = −η∂E p∂wk−1,ki, j+ ζΔ p wk−1,ki, j (m − 1),(3)

where ζ denotes the inertia rate. The inertia term is introduced for eﬃcient learning convergence.
3.

Fig. 1

Aﬀordable neural network.

Damaging Neurons

We now assume that after the BP learning phase, some neurons in the hidden layer are damaged according to the following two paradigms.
3.1 Cutting-Oﬀ Damage
In this paradigm, the connections of the damaged neurons to
the output layer are temporally cut oﬀ, see Fig. 3.
3.2 Random Performance Damage
In this paradigm, the connections to output layer of thedamaged neurons take random values from the unit interval[0, 1]. The diagram of this model is shown in Fig. 4. Thissituation is similar to the case where some neurons operaterandomly, seemingly without any order or control.

Fig. 2

Random selection.

4.
in the timing of the weight. Whereas the update of the standard BP is performed after each single input data, in thebatch BP, the update is performed only after all input data
has been processed once. The total error E of the network is
defined as

Results

We first consider the task to learn the function y(x) = x2 as
one learning example (See Fig. 5). The sampling range ofthe input data is [−1.0, 1.0] and the step size of the input
data is set to be 0.01. Our BP learning used fixed learning and inertia rates of η = 0.4 and ζ = 0.02, respectively;

UWATE et al.: DURABILITY OF AFFORDABLE NEURAL NETWORKS AGAINST DAMAGING NEURONS

587

1 1
(t p − o p )2
P p=1 2
P

Eave =

Fig. 3

Damaging neurons: Cutting-oﬀ damage.

(4)

where P denotes the number of input data (P = 200). Givena hidden layer of h neurons we have discovered that it ispossible to choose up to h − 1 aﬀordable neurons for thisquadratic function problem. We believe that this is true inthe generic case as well. In this study we focused on simulations using hidden layers consisting of 8 to 11 neurons
with the number of the aﬀordable neurons ranging between
1 and 3. To give an example: If the number of neurons in
the hidden layer is 8 and the number of the aﬀordable neurons is set to 2, only 6 neurons operate at all times. Sucha network will be denoted as “AfNN (8 − 2).” Below wewill compare the performance of an AfNN of type (n − m)with a conventional BP network in which, after learning, thesame number of neurons of the hidden layer are damaged inexactly the same manner.
4.1 Cutting-Oﬀ Damage
4.1.1 Eﬀect of Damaged Neurons

Fig. 4

Damaging neurons: Random performance damage.

Fig. 5

Learning example: y = x2 .

the initial values of the weights were chosen uniformly random from the interval [−1, 1] and the learning time wasm = 2000. We investigate how after learning damaged neurons aﬀect the total error between the output and the desired
target. In order to measure this, we define “Average Error
Eave ” by the following equation:

For this damaging paradigm, the comparison between the
AfNN with the conventional network yields the results displayed in Fig. 6.
Figures 6(a), (b), (c) and (d) show the averaged resultsfor 8, 9, 10 and 11, respectively, neurons in the hidden layer,for 100 computer simulations using diﬀerent random initialcondition of the network. When increasing the number ofthe damaged neurons, Eave of both the AfNN and the conventional BP network worsen. From the comparison of theperformance of the two networks it is, however, seen that
the AfNNs achieves a clearly superior performance to theconventional neural network. Even for a large number ofdamaged neurons (e.g. 50 percent or more, the AfNN substantially outperforms the conventional BP network in thepresence of damage). From these results, it clearly emergesthat AfNNs provide an important element of reliability incases where neurons are aﬀected by damage.
Figures 7–10 show the detailed comparison of performance between the AfNN and the conventional BP network
before and after one neuron has been damaged. Only incomplete learning based on 2000 learning steps was considered.
Whereas in all panels the left hand side figures show theoutput after learning, the right hand side figures display theoutput after neuron damage. It is seen that in the case of
the AfNN, the eﬀect of a damaged neuron is small, whereas
in the case of the conventional BP network, the eﬀect of adamaged neuron is larger.
4.1.2 Eﬀect of the Position of a Damaged Neuron
Simulation results for trials with one damaged neuron are
shown in Fig. 11. The horizontal axis is the position of thedamaged neuron in the hidden layer and the vertical axis
is the error Eave . The solid line displays the performance

IEICE TRANS. FUNDAMENTALS, VOL.E92–A, NO.2 FEBRUARY 2009

588

Fig. 7 Output of the network before and after one neuron is damaged(Hidden: 8).

Fig. 8 Output of the network before and after one neuron is damaged(Hidden: 9).

Fig. 6 Eave dependant on the number of damaged neurons (Damaging
neurons: cutting-oﬀ damage).

of the AfNN, whereas the dashed line refers to the conventional BP network. We emphasize that the vertical axis in
Fig. 11 is in log scale. From these figures it can be inferredthat while the performance of the conventional BP network
is strongly aﬀected by damage, the loss of performance does
not depend on the position of the damage. In contrast, thedrop of performance for the AfNN is much smaller. OnlyUWATE et al.: DURABILITY OF AFFORDABLE NEURAL NETWORKS AGAINST DAMAGING NEURONS

589

Fig. 9 Output of the network before and after one neuron is damaged(Hidden: 10).

Fig. 10 Output of the network before and after one neuron is damaged(Hidden: 11).

occasionally, is the same drop in performance as the conventional BP network observed. For example, in the casewhere the number of the neurons in the hidden layer is set
to 11 (Fig. 11(d)), for many damage positions, the error Eaveof the AfNN is orders of magnitude smaller than that of theconventional BP network.
The results of Fig. 11, which were based on single tri-

Fig. 11

Eave and position of damaged neurons.

als, are insuﬃcient to make general statements on the behavior with respect to diﬀerent initial conditions. Therefore,
we calculated the distribution of Eave during 200 trails after

IEICE TRANS. FUNDAMENTALS, VOL.E92–A, NO.2 FEBRUARY 2009

590

Fig. 12 Distribution of Eave for each trial (Hidden layer: 8) (Dash line
indicates average value).

one neuron is damaged. Figures 12–15 show the simulated
results of distribution of Eave when the number of hidden
layer is set to 8 to 11, respectively. From these simulations, it emerges that for the AfNNs, although the value ofEave strongly depends on the initial conditions and generally
spreads across the interval [0, 1], it has a noticeable bias towards zero and is generally smaller than the corresponding
value of the conventional BP network. Moreover, since eachneuron has some characteristics or individuality regarding
to what performance drop its damage leads to, this situation
seems to be much closer to real biological network behavior
than that of the conventional BP network.
4.2 Random Performance Damage
We now investigate the characteristics of the second damaging paradigm. After learning, the connection to the output
layer of damaged neurons is modified to produce a random
value uniformly from the unit interval [0, 1].
The results obtained from networks of identical characteristics to before are shown in Fig. 16. They clearly indicate
the performance of the AfNN drops quicker towards that of
the conventional BP network, as the number of the damaged
neurons is increased. Thus, although the performance is still
superior to the conventional network, the changed damaging
paradigm is less eﬀective than in the case considered before.

Fig. 13 Distribution of Eave for each trial (Hidden layer: 9) (Dash line
indicates average value).

5.

Simulated Results for Pattern Recognition

As the last example, we consider a pattern recognition task,
where 4 diagrammic letters (B, U, C, S) are fed into theneural network for recognition (see Fig. 17).
In this case, the number of neurons in the input layer
is 35 The number of neurons in output layer is fixed to 4,
and we choose 16 hidden layer neurons. For recognition,a set of 25 patterns shifted 1 bit from each original pattern
was prepared, leading a set of 100 patterns to be recognized.
The learning rate and the inertia rate were set to η = 1.0 andζ = 0.8, respectively, and the initial weights were uniformly
randomly chosen from the interval [−1, 1]. The learning
time was set to m = 10000 steps. By using 100 diﬀerent network initial conditions, we obtained reliable averages characterizing the network performance in terms of the recognition rate R defined as
R=

Number o f sucsess.
T otal number o f input patterns(5)

5.1 Cutting-Oﬀ Damage
Figure 18 displays the recognition rate for the AfNN andconventional BP network. From this figure, we infer that forUWATE et al.: DURABILITY OF AFFORDABLE NEURAL NETWORKS AGAINST DAMAGING NEURONS

591

Fig. 14 Distribution of Eave for each trial (Hidden layer: 10) (Dash line
indicates average value).

this task, the recognition rates of the AfNN and the conventional network are similar, if only one neuron is damaged.
Upon increasing the number of damaged neurons, the difference between the recognition rates, however, increases,
leading to a noticeable higher recognition rate of the AfNN,
where the diﬀerence of recognition rate between the both
networks is larger than 10 percent.
5.2 Random Performance Damage
The results obtained for the pattern recognition problem inthe case of “random performance damage” are shown in
Fig. 19. Clearly, the influence of a damaged neuron in the
AfNN is again smaller than that of a damaged neuron in
the conventional BP network. Also for this task, the AfNN
demonstrates a superior performance to that of the conventional BP network.
6.

Conclusions

For two rather diﬀerent types of realistic neuron damages,
similar overall performance was observed, where the performance of the AfNN (at least in the statistical sense) strongly
outperformed the conventional BP network, which demonstrates the beneficial influence of aﬀordable neurons forFig. 15 Distribution of Eave for each trial (Hidden layer: 11) (Dash line
indicates average value).

the reliable performance of neural networks under realistic working conditions. It is thus obvious that the aﬀordable neurons exert an important influence on the durabilityof neural networks.
In future work, we will investigate the eﬀects bychaotic selection methods, that more realistically model theselection of aﬀordable neurons in the context of more complex problems. This view is motivated by investigationsof the Hopfield network solving combinatorial optimization
problems with the help of a chaotic input signal component,
designed in order to avoid local minima. It appears, fromcomputer simulations, that a chaotic input component may
substantially enhance the capability of avoiding these local
minima in comparison to random noise [4]–[6]. Hence, we
believe that chaotic selection may be used to further enhancethe eﬃciency of the proposed AfNN.
As a more general conclusion, the detailed modelingof biological neural networks may still reveal as yet undiscovered neural networks paradigms that have the capacityof substantially enhancing the performance of artificial neural networks. This should motivate an improved interaction
between the artificial and biological neural networks communities.

IEICE TRANS. FUNDAMENTALS, VOL.E92–A, NO.2 FEBRUARY 2009

592

Fig. 17

Fig. 18

Pattern recognition.

Recognition rate (Damaging neurons: Cutting-oﬀ damage).

Fig. 19 Recognition rate (Damaging neurons: Random performancedamage).

References

Fig. 16 Eave dependant on the number of damaged neurons (Damaging
neurons: random performance damage).

[1] H. Satoi, H. Tomimoto, R. Ohtani, T. Kitano, T. Kondo, M. Watanabe,
N. Oka, I. Akiguchi, S. Furuta, Y. Hirabayashi, and T. Okazaki,“Astroglial expression of ceramide in Alzheimer’s disease brains: A
role during neuronal apoptosis,” Neuroscience, vol.130, pp.657–666,
2005.
[2] Y. Uwate and Y. Nishio, “Performance of aﬀordable neural network
for back propagation learning,” IEICE Trans. Fundamentals, vol.E89A, no.9, pp.2374–2380, Sept. 2006.
[3] D.E. Rumelhart, G.E. Hinton, and R.J. Williams, “Learning internal
representations by error propagation,” Parallel Distributed Processing,vol.1, pp.318–362, 1986.
[4] Y. Hayakawa and Y. Sawada, “Eﬀects of the chaotic noise on theperformance of a neural network model for optimization problems,”
Phys. Rev. E, Stat. Phys. Plasmas Fluids Relat. Interdiscip. Top.,
vol.51, no.4, pp.2693–2696, April 1995.
[5] T. Ueta, Y. Nishio, and T. Kawabe, “Comparison between chaotic
noise and burst noise on solving ability of hopfield neural networks,”
Proc. NOLTA’97, vol.1, pp.409–412, Nov. 1997.
[6] Y. Uwate, Y. Nishio, T. Ueta, T. Kawabe, and T. Ikeguchi, “Per-UWATE et al.: DURABILITY OF AFFORDABLE NEURAL NETWORKS AGAINST DAMAGING NEURONS

593

formance of chaos and burst noise injected to the hopfield NN
for quadratic assignment problems,” IEICE Trans. Fundamentals,
vol.E87-A, no.4, pp.937–943, April 2004.

Yoko Uwate
was born in Tokushima, Japan,
in 1980. She received the B.E., M.E. and Ph.D.
degrees in Electrical and Electronic Engineeringfrom Tokushima University, Tokushima, Japan,
in 2003, 2005 and 2006, respectively. During
October 2006–March 2008, she was Post Doctral Resarch Fellow (PD) of the Japan Society
for the Promotion Science (JSPS) at the same
university, and she was also Visiting Post Doctoral Research Fellow, Institute of Neuroinformatics (INI), University and ETH Zurich. From
April 2008, she is currently working as Post Doctoral Fellow for Research
Abroad of Japan Society for the Promotion Science (JSPS) at Institute of
Neuroinformatics (INI), University and ETH Zurich. Her research interests
include complex phenomena in chaotic circuits and neural networks. Ms.
Uwate is a member of the IEEE.

Yoshifumi Nishio
received B.E., M.E.,
and Ph.D. degrees in electrical engineering from
Keio University, Yokohama Japan, in 1988,
1990, and 1993, respectively. In 1993, he joined
the Department of Electrical and Electronic Engineering at Tokushima University, Tokushima
Japan, where he is currently an Associate Professor. From May 2000 he spent a year in the
Laboratory of Nonlinear Systems (LANOS) at
the Swiss Federal Institute of Technology Lausanne (EPFL) as a Visiting Professor. His research interests include analysis and application of chaos in electrical circuits, analysis of synchronization in coupled oscillatory circuits, development of analyzing methods for nonlinear circuits, theory and applicationof cellular neural networks, and neural network architecture. He was the
Chair of the IEEE CAS Society Technical Committee on Nonlinear Circuits
and Systems (NCAS) during 2004–2005, the Steering Committee Secretaryof the IEICE Research Society of Nonlinear Theory and its Applications(NOLTA) during 2004–2007, and is currently the Secretary/Treasurer of
the IEEE CAS Society Shikoku Chapter. He was an Associate Editor of
the IEEE Transactions on Circuits and Systems–I: Regular Papers during
2004–2005, and is currently serving as an Associate Editor for the IEEE
CAS Magazine, the IEEE CAS Society Newsletter, and the RISP Journalof Signal Processing. He is also the Editor of the IEICE Fundamentals
Review and is a member of the Editorial Board of International Journal of
Circuit Theory and Applications. He is a senior member of the IEEE, anda member of the RISP.

Ruedi Stoop
was born 1952 in Davos,
Switzerland. He studied Mathematics and did a
Ph.D. in computer-aided physics at Zurich University. He did habilitations in the fields of nonlinear dynamics and biocomputation at the Universities of Berne, Zurich and the Federal Institute of Technology ETH of Zurich. As a professor, he lectures at the ETH and at the University of Zurich, where at the Institute of Neuroinformatics ETHZ/UNIZH, he leads a researchgroup concentrating on the application of nonlinear dynamics and statistical mechanics to problems in neuroscience. He
co-authored the book ‘Encounter with Chaos’ (Springer, 1992). He regularly organizes conferences and is an editorial board member of the journals“Nonlinear Analysis, Real World Applications” and “Discrete and Continuous Dynamical Systems.”