Damaging Neurons of Affordable Neural Network
for Pattern Recognition
Yoko Uwate

Yoshifumi Nishio

Ruedi Stoop

Institute of Neuroinfomatics,
University / ETH Zurich
Email: yu001@ini.phys.ethz.ch

Dept. Electrical and Electronic Eng.,
Tokushima University
Email: nishio@ee.tokushima-u.ac.jp

Institute of Neuroinfomatics,
University / ETH Zurich
Email: ruedi@ini.phys.ethz.ch

SUMMARY

Durability describes the ability of a device to operateproperly in imperfect conditions. We have recently proposeda novel neural network structure called an “Affordable NeuralNetwork” (AfNN), in which affordable neurons of the hiddenlayer are considered as the elements responsible for therobustness property as is observed in human brain function.
Whereas earlier we have shown that AfNNs can still generalize
and learn, here we show that these networks are robust against
damages occurring after the learning process has terminated.
The results support the view that AfNNs embody the importantfeature of durability. In our contribution, we investigate thedurability of the AfNN is applied to pattern recognition whensome of the neurons in the hidden layer are damaged after thelearning process.
I. I NTRODUCTION
Recently, studies on the brain have been carried out actively
on various levels. On the neuroscience level, many researchersinvestigate how neuronal death occurs in the human brain.
Neuronal death can be occurred by various but usual causessuch as drinking a lot of alcohol, heavy smoking and concussions. Furthermore, some neurons die with apoptosis when theneuron can not receive any signals. Even if some neurons diein daily life, the human brain is still able to operate normallyby creating new information processing from abundant experience and knowledge. This is because, the human brain hasdurability and flexibility.
One of most important application of neural networks ispattern recognition. Pattern recognition can be implemented
by using a feedforward neural network that has been trained
accordingly. During training, the network is trained to associate outputs with input patterns. When the network is used,
it identifies the input pattern and tries to output the associatedoutput pattern. The effectivenes of Back Propagation (BP)learning has been confirmed in a pattern recognition.
In our previous research, we have proposed a new network
structure with affordable neurons in the hidden layer, forefficient BP learning [1]. We named this network “Affordable
Neural Network” (AfNN). In this network, we prepare someextra neurons in the hidden layer. When the network executes
operating, all of the neurons in the hidden layer are not used atevery updating. By computer simulations, the AfNN has been

confirmed to gain better performance for the BP learning on
both convergence speed and learning efficiency. We can saythat the AfNN has generalization ability. However, we believethat many advantages of the AfNN are still veiled. We consider
that the AfNN is able to self-generate network durability.
In this study, we investigate the durability of the AfNN
when some of the neurons in the hidden layer are damaged,
after the learning process for a pattern recognition. Where thedurability means strong, tough and tenacious for any damages
from outside. Namely, the durability shows ability that network
can still operate as well as before the neurons are damaged.
We study the effect of two types of damaging. First, theconnections to the output layer of the damaged neurons arecut. Namely, the damaging neuron does not operate (cuttingoff damage). Second, the connections to the output layerof the damaged neurons produce random values (randomperformance damage). The learning example of the patternrecognition is 4 patterns of alphabet (B, U, C, S). Each patternis composed of 7x5 neurons. By computer simulations, weconfirm that the AfNN keeps its efficiency of recognition rate.
We conclude that the affordable neurons exert an important
influence on durability when the AfNN is applied to the patternrecognition.
II. A FFORDABLE N EURAL N ETWORKFor our investigations, we use [1] the standard three-layered
feedforward neural network. The output characteristics of theneurons are implemented by a sigmoid function and a subset of
the neurons are tagged as affordable neurons. For convenience,
the network model and the operation of the affordable neurons
in the hidden layer are shown in Fig. 1(a) and Fig. 1(b),respectively.
In the proposed BP learning network, at every input pattern,a subset of the affordable neurons is randomly chosen andsilenced by cutting off their output. The temporal evolutionthat this network now undergoes is exhibited in Fig. 2, wherethe hidden layer consists of eight neurons. Out of these eightneurons, two are randomly selected and are silenced at eachlearning step (dots in Fig. 2).
In this study, we use the batch BP learning algorithm. Thislearning algorithm can be cast in a formula similar to thestandard [2] BP learning algorithm. The difference lies in thetiming of the weight. Whereas the update of the standard BP- 72 -Neuron number in hidden layer

8Output layerInput layer

76543210102030

4050time

Affordable
NeuronsFig. 2.

Random selection.

Hidden layer(a) Network model with affordable neurons.

weights are adjusted according to:k−1,kk−1,kwi,j(m + 1) = wi,j(m)PX
k−1,k+∆p wi,j(m),
p=1

k−1,k
∆p wi,j(m) = −ηLearning time:2

Learning time:1∂Ep
k−1,k∂wi,j(2),k−1,kwere wi,jis the weight between the ith neuron of the layerk − 1 and the jth neuron of the layer k. m is the learning
time, and η is a proportionality factor known as the learningrate. In this study, to the third line of Eq.(2) an inertia term
was added, which leads toLearning time:3

Learning time:Nk−1,k∆p wi,j(m) = −ηOperating neuron
Affordable neuron (rest neuron)

k−1,k
∂wi,j

k−1,k+ ζ∆p wi,j(m − 1),(3)

where ζ denotes the inertia rate. The inertia term is introduced
for efficient learning convergence.

(b) Operation of the affordable neurons. In the proposed BP learning
network, at every input pattern, a subset of the affordable neurons is
randomly chosen and silenced by cutting off their output. The temporal
evolution that this network now undergoes is exhibited in Fig. 2, where thehidden layer consists of eight neurons. Out of these eight neurons, two are
randomly selected and the silenced at each learning step (dots in Fig. 2).
Fig. 1.

∂Ep

Affordable neural network.

III. DAMAGING N EURONS
We now assume that after the BP learning phase, someneurons in the hidden layer are damaged according to thefollowing two paradigms.
A. Cutting-off damageIn this paradigm, the connections of the damaged neurons
to the output layer are temporally cut off, see Fig. 3.

is performed after each single input data, in the batch BP,
the update is performed only after all input data has beenprocessed once. The total error E of the network is defined as

E=P
X
p=1

Ep =PX
p=1(N1X(tpi − opi )22 i=1),(1)

B. Random performance damage
In this paradigm, the connections to output layer of thedamaged neurons take random values from the unit interval[0, 1]. The diagram of this model is shown in Fig. 4. Thissituation is similar to the case where some neurons operaterandomly, seemingly without any order or control.
IV. S IMULATED R ESULTS

where P is the number of the input data, N is the numberof the neurons in the output layer, tpi denotes the value ofthe desired target data for the pth input data, and opi denotes
the value of the output data for the pth input data. The goalof the learning is to set the weights between the neurons soas to minimize the total error E. In order to minimize E, theIn this study, we consider a pattern recognition task, where
4 diagrammic letters (B, U, C, S) are fed into the neuralnetwork for recognition (see Fig. 5).
In this case, the number of neurons in the input layer is
35 The number of neurons in output layer is fixed to 4, andwe choose 16 hidden layer neurons. For recognition, a set- 73 -Output: zero

A. Cutting-off damage
DamagedneuronInput layer

Figure 6 displays the recognition rate for the AfNN and conventional BP network. From this figure, we infer that for thistask, the recognition rates of the AfNN and the conventional
network are similar, if only one neuron is damaged. Upon
increasing the number of damaged neurons, the difference
between the recognition rates, however, increases, leading toa noticeable higher recognition rate of the AfNN, where thedifference of recognition rate between the both networks islarger than 10 percent.

Output layer

Damagedneuron100
Affordable NN (16-10)
Conventional NN (16-0)

90

Hidden layer
Recognition rate [%]80Fig. 3.

Damaging neurons: cutting-off damage.

Output: random
Damagedneuron706050403020Fig. 6.

Input layer

123
Number of damaged neurons

45Recognition rate (Damaging neurons: cutting-off damage).

Output layer

DamagedneuronB. Random performance damage

Damaging neurons: random performance damage.

of 25 patterns shifted 1 bit from each original pattern wasprepared, leading a set of 100 patterns to be recognized. The
learning rate and the inertia rate were set to η = 1.0 andζ = 0.8, respectively, and the initial weights were uniformly
randomly chosen from the interval [−1, 1]. The learning time
was set to m = 10000 steps. By using 100 different network
initial conditions, we obtained reliable averages characterizing
the network performance in terms of the recognition rate R
defined as
N umber of sucsess.
R=(4)T otal number of input patterns

The results obtained for the pattern recognition problem inthe case of “random performance damage” are shown in Fig. 7.
Clearly, the influence of a damaged neuron in the AfNN isagain smaller than that of a damaged neuron in the conventional BP network. Also for this task, the AfNN demonstrates
a superior performance to that of the conventional BP network.
100AfNN (16-10)
Conventional NN (16-0)

9080
Recognition rate [%]Hidden layer

Fig. 4.

07060
50
4030
20
100123
Number of damaged neurons

45Fig. 7. Recognition rate (Damaging neurons: random performance damage).
pattern 1

pattern 2

pattern 3

pattern 4V. C ONCLUSIONS
Fig. 5.

Pattern recognition.

For two rather different types of realistic neuron damages, similar overall performance was observed, where the- 74 -performance of the AfNN (at least in the statistical sense)
strongly outperformed the conventional BP network, which
demonstrates the beneficial influence of affordable neurons forthe reliable performance of neural networks under realistic
working conditions. It is thus obvious that the affordable
neurons exert an important influence on the durability of neuralnetworks for pattern recognition task.
In future work, we will investigate the effects by chaotic
selection methods, that more realistically model the selectionof affordable neurons in the context of more complex problems. This view is motivated by investigations of the Hopfield
network solving combinatorial optimization problems with thehelp of a chaotic input signal component, designed in order
to avoid local minima. It appears, from computer simulations,that a chaotic input component may substantially enhance thecapability of avoiding these local minima in comparison torandom noise [3]-[5]. Hence, we believe that chaotic selection
may be used to further enhance the efficiency of the proposed
AfNN.
R EFERENCES[1] Y. Uwate and Y. Nishio, “Performance of Affordable Neural Network
for Back Propagation Learning,” IEICE Transactions on Fundamentals,
vol. E89-A, no. 9, pp. 473-478, Nov. 2005.
[2] D.E. Rumelhart, G.E. Hinton and R.J. Williams, “Learning Internal
Representations by Error Propagation,” Parallel Distributed Processing,vol. 1, MIT Press, MA, pp. 318-362, 1986.
[3] Y. Hayakawa and Y. Sawada, “Effects of the Chaotic Noise on thePerformance of a Neural Network Model for Optimization Problems,”
Phys. Rev. E, Stat. Phys. Plasmas Fluids Relat. Interdiscip. Top.,, vol.
51, no. 4, pp. 2693-2696, April. 1995.
[4] T. Ueta, Y. Nishio and T. Kawabe, “Comparison between Chaotic Noiseand Burst Noise on Solving Ability of Hopfield Neural Networks,” Proc.
NOLTA’97, vol. 1, pp. 409-412, Nov. 1997.
[5] Y. Uwate, Y. Nishio, T. Ueta, T. Kawabe and T. Ikeguchi, “Performanceof Chaos and Burst Noise Injected to the Hopfield NN for QuadraticAssignment Problems,” IEICE Transactions on Fundamentals, vol. E87A, no. 4, pp. 937-943, Apr. 2004.

- 75 -