Effectiveness of Artificial Neural Network with
Time-Varying Coupling System
Yoko Uwate and Yoshifumi Nishio
Dept. of Electrical and Electronic Engineering,Tokushima University
2-1 Minami-Josanjima, Tokushima, Japan
Email: {uwate, nishio}@ee.tokushima-u.ac.jp"A"

Abstract— We have recently proposed a novel neural network
structure called an “Affordable Neural Network” (AfNN), in
which affordable neurons of the hidden layer are consideredas the elements responsible for the robustness property as isobserved in human brain function.
In this study, we investigate the mechanism of learning processof the AfNN to make clear the reason of that the AfNNs canperform well for learning and generalization abilities and operate
as usually against damaging neurons.

"B""A""B"(a) Cell assembly-liked process.
"A""B""A""B"

I. I NTRODUCTION
The mechanisms of many kinds of higher brain functions
become clear with the advances in neuroscience technology.
Therefore, the studies of bio-inspired artificial neural networks
have been extensively reported to realize such intelligencesystems applying for the future engineering applications. However, the performance of artificial neural networks still lags
behind that of biological networks in many respects. In order
to fill the gap between biological neural systems and artificial
neural systems, it is important to apply these high functional
mechanisms of the human brain to novel artificial neuralnetworks.
In our previous work, we have proposed a new network
structure of the feedforward neural network with affordable
neurons in the hidden layer for Backpropagation (BP) learning [1]. We named this network “Affordable Neural Network(AfNN).” The AfNN was inspired by cell assembly [2][5]. which is one of explanations of multiple information
processing in the brain and an information is represented
by a firing space pattern of a group of plural neurons (seeFig 1(a)). In the AfNN, we prepare some extra neurons morethan requires in the hidden layer. When the AfNN executes
operating, all of the neurons in the hidden layer are not used
at every updating. Namely, the AfNN is able to operate withhigh function by using different neuron patterns in the hidden
layer. By computer simulations, the AfNN has been confirmed
to gain better performance for the BP learning on both learningability and generalization ability.
On the other hand, am important issue is the understanding
how the mammalian brain is affected in its performance by
the omnipresent death of their elements in the field of Neuroscience. It is also important to investigate the performance of
the artificial neural network when some neurons are damaged.
Several research groups have reported the studies of fault

978-1-4799-3432-4/14/$31.00 ©2014 IEEE(b) AfNN learning process.
Fig. 1.

Learning process of neural networks for each input pattern.

tolerance in the artificial neural networks [6]-[8]. In order toconfirm that the AfNN can generate a kind of fault tolerance,
we also have investigated the durability of the AfNN whensome of the neurons in the hidden layer are damaged after thelearning process [9]. We have confirmed that the AfNN canoperates with keeping its efficiency against damaging neurons.
However, the reason of what property of learning process of
the AfNN affect good performance for the learning ability anddurability has not yet understood in detail.
We focus on the learning process of AfNN which useddifferent neuron patterns for each input pattern. In the caseof the AfNN, the affordable neurons are selected “at everyupdating time.” The AfNN learns by using all neurons in thehidden layer with different pattern for every input patterns (seeFig. 1(b)). Therefore, we assume that the learning process of
the AfNN is more complex and it has a possibility to create
the high functions such as durability and flexibility. In order toconfirm the effectiveness of the learning process of the AfNN,
we consider different learning process with the AfNN. In thislearning process, the affordable neurons are selected for “foreach pattern.” The network learns by using specific neurons
in the hidden layer for each input pattern (see Fig. 1(a)).
In this study, we compare the performance of two typesof the learning process when the AfNNs are applied for thepattern recognition. The results support the view that AfNN
learned by all neurons with different pattern embody theimportant feature of durability.

2005

AfNN_std (8-2)

II. A FFORDABLE N EURAL N ETWORK (A F NN)
We have proposed a network with affordable neurons in thehidden layer of the three-layered feedforward neural network(one input layer, one hidden layer and one output layer) for
the BP learning. The output characteristics of the neurons
are implemented by a sigmoid function. We introduced theaffordable neurons to reflect important properties of the brain.
During the BP learning, all of neurons in the hidden layer
are not used at every updating. Namely, some of the neurons
are selected for the learning and the rest of the neurons aredeactivated. The affordable neurons are selected by random
at every updating time. The network model of the AfNN isshown in Fig. 2.

Input patternsiteration=1

iteration=2

ptn1 ptn2 ptn3 ptn4

ptn1 ptn2 ptn3 ptn4

ptn1 ptn2 ptn3 ptn4

BB

B

UCSU

iteration=3

CSUCS

Neurons patternin hidden layer

BUCS

Neurons pattern
for each input pattern

Operating neuron
Affordable neuron

Fig. 3.

Learning process of AfNN-std.

Output layer

Input layer

AfNN-ptn (8-2)
Operating neuron
Hidden layer

Fig. 2.

Affordable neuron

Input patterns

iteration=1

iteration=2

ptn1 ptn2 ptn3 ptn4

ptn1 ptn2 ptn3 ptn4

ptn1 ptn2 ptn3 ptn4BBBUCSU

iteration=3CSUCS

Neurons patternin hidden layer

Affordable neural network.

Here, we explain the learning process of the AfNN. In thecase of the standard AfNN (abbr. AfNN-std), the affordable
neurons are selected at every iteration time. Namely, theAfNN-std leans every input pattern by using the all neurons
in the hidden layer with different pattern as shown in Fig. 3.
We assume that this learning process is very effective toproduce the durability. In order to confirm the effectivenessof the AfNN-std, we consider a different learning process tothe AfNN. In a new learning process, same neurons patterns
are used with respect to each input pattern. We call such
AfNN “AfNN-ptn”. Figure 4 shows operating mechanism ofAfNN-ptn. In this study, we compare the the AfNN-std andthe AfNN-ptn when the AfNNs are applied for the patternrecognition.
The standard BP learning algorithm was introduced in[10]. The BP is the most common learning algorithm for thefeedforward neural networks. In this study, we use the batchBP learning algorithm. The batch BP learning algorithm isexpressed by a formula similar to the standard BP learningalgorithm. The difference lies in the timing of the weight. The
update of the standard BP is performed after each single input
data, while for the bath BP the update is performed after all
input data has been processed. The total error E of the network
is defined as
 N

PP


1
2
Ep =(tpi − opi ) ,
(1)
E=
2 i=1
p=1
p=1
where P is the number of the input data, N is the numberof the neurons in the output layer, tpi denotes the value ofBUCSNeurons pattern
for each input pattern

Operating neuron
Affordable neuron

Fig. 4.

Learning process of AfNN-ptn.

the desired target data for the pth input data, and opi denotes
the value of the output data for the pth input data. The goalof the learning is to set the weights between the neurons so
as to minimize the total error E. In order to minimize E, theweights are adjusted according to:k−1,kk−1,k(m + 1) = wi,j(m) +wi,jk−1,kΔp wi,j(m) = −η∂Epk−1,k∂wi,jPp=1

k−1,kΔp wi,j(m),(2),k−1,kis the weight between the ith neuron of the layer
were wi,jk − 1 and the jth neuron of the layer k. m is the learning
time, and η is a proportionality factor known as the learning

2006

Output of damaged neuron: Zerorate. In this study, to the third line of Eq.(2) an inertia term
was added, which leads tok−1,k(m) = −ηΔp wi,j∂Ep
k−1,k∂wi,jk−1,k+ ζΔp wi,j(m − 1),Damagedneuron(3)

where ζ denotes the inertia rate. The inertia term is introduced
for efficient learning convergence.
Output layer

Input layer

III. PATTERN R ECOGNITION

Damagedneuron

As more demanding example for the BP learning, weconsider a pattern recognition task, where 4 alphabetical letters(B, U, C, S) are fed into the neural network for recognition(see Fig. 5). The black and white box correspond to 1.0 and0.0. In this case, the number of neurons in the input layer
is 35 The number of neurons in output layer is fixed to 4,and we choose 20 hidden layer neurons. For recognition, a
set of 25 patterns shifted 1 bit from each original pattern wasprepared, leading a set of 100 patterns to be recognized. The
learning rate and the inertia rate were set to η = 0.8 andζ = 0.08, respectively, and the initial weights were uniformly
randomly chosen from the interval [−1, 1]. The learning time
was set to m = 10000 steps. By using 100 different network
initial conditions, we obtained reliable averages characterizing
the network performance in terms of the recognition rate R
defined as
R=N umber of success× 100[%].
T otal number of input patterns(4)

For culculating this equation, we use the following equation.
opi = 1.0

(opi > Tb )opi = 0.0

(opi > Tw )(5)

Hidden layer

Fig. 6.

Conceptual network model for damaging neurons (output: zero).

V. S IMULATION R ESULTS OF A F NN S
AGAINST DAMAGING N EURONSFirst, we explain the description of the network structureof AfNNs. For example, in the network (20-5), the neurons
in the hidden layer is fixed with 20 and the number of theaffordable neurons is 5.
We investigate the relationship between the performanceof the AfNNs (AfNN-std and AfNN-ptn) and recognition
rate when the number of affordable neurons is changed. We
consider that 2 neurons are damaged (Zero output) after
learning process. The simulation result of the AfNNs is shownin Fig. 7. The AfNN-std gains best performance when thenumber of the affordable neuron is 4. Also the AfNN-std
obtains better results than the conventional network with wide
range of the affordable neurons. While, the performance of theAfNN-ptn decrease by increasing the number of the affordable
neurons.
100Recognition rate [%]Where, we fix the threshold values (Tb = 0.9, Tw = 0.1).

AfNN-std
AfNN-ptn
Conventional NN

80604020

pattern 1

pattern 2
Fig. 5.

pattern 3

pattern 402

468
10
12
Affordable neurons

141618

Pattern recognition.
Fig. 7. Recognition rates by changing number of affordable neurons (Zerooutput).

IV. DAMAGING N EURONS (Z ERO OUTPUT )In this study, we investigate the generalization ability ofAfNNs when some of neurons in the hidden layer are damaged. In order to realize damaging neurons, we consider thatthe connections of the damaged neurons to the output layer are
temporally cut off (see Fig. 6). Namely, the damaged neurons
do not operate. In this situation, we calculate the recognition
rate for the several types of AfNNs.

Figure 8 shows the recognition rate of the several network
structure of AfNN-std, AfNN-ptn and the conventional BP
networks with the number of damaged neurons.
Figure 8(a) shows the recognition rate of the network
structure (20-1). We infer that for this task, the recognition
rates of the AfNNs and the conventional networks are similar,
if only one neuron is damaged. Upon increasing the number ofdamaged neurons, the difference between the recognition rates,

2007

Recognition rate [%]100AfNN-std (20-1)
AfNN-ptn (20-1)
Conventional NN

8060402000246Damaged neurons

810(a) network (20-1).

Recognition rate [%]100AfNN-std (20-2)
AfNN-ptn (20-2)
Conventional NN

80604020002

46
Damaged neurons

810however, increases, leading to a noticeable higher recognition
rate of the AfNNs, where the difference of recognition rate
between the both networks is larger than 10 percent. When thenumber of damaged neuron is larger than 5, the recognition
rate of AfNN-ptn is higher than AfNN-std.
Figures 8(b) and (c) show the recognition rate of thenetwork structure (20-2) and (20-3). In these cases, we confirm
that the AfNN-std can keep high recognition rate when thenumber of damaged neurons is smaller than 5. While, theperformance of the AfNN-ptn becomes worse by increasing
the number of affordable neurons.
The recognition rate of the network structure (20-5) and(20-10) is shown in Fig. 8(d) and (e). The AfNN-ptn can
not recognize the input data even if the number of damaged
neurons is small.
By comparison between the AfNN-std and the AfNN-ptn,
we confirm that the performance of the AfNN-std is better
than the AfNN-ptn and the conventional network. Namely,
the learning process of the AfNN-std is cause of producing
durability in the network.

(b) network (20-2).

VI. C ONCLUSIONS

Recognition rate [%]100
AfNN-std (20-3)
AfNN-ptn (20-3)
Conventional NN

8060402000246Damaged neurons

810(c) network (20-3).

In this study, we have investigated the mechanism of learning process of the AfNN-std to make clear the reason of that
the AfNN-std can perform well for learning and generalization
abilities and operate as usually against damaging neurons.
By computer simulations of the pattern recognition, we haveconfirmed that the learning process of the AfNN-std obtained
better recognition rate than the AfNN-ptn.
For the future works, we apply the theoretical or systematic
approaches to analyze the characteristics of the AfNNs anduse more practical task as the learning example.

Recognition rate [%]100R EFERENCES

AfNN-std (20-5)
AfNN-ptn (20-5)
Conventional NN

8060402000246Damaged neurons

810(d) network (20-5).

Recognition rate [%]100AfNN-std (20-10)
AfNN-ptn (20-10)
Conventional NN

8060402000246Damaged neurons

810(e) network (20-10).
Fig. 8.

Recognition rate against damaging neurons (Zero output).

[1] Y. Uwate and Y. Nishio, “Performance of Affordable Neural Network
for Back Propagation Learning,” IEICE Transactions on Fundamentals,
vol. E89-A, no. 9, pp. 473-478, Nov. 2005.
[2] Y. Sakurai, “Dependence of Functional Synaptic Connections of Hippocampal and Neocortical Neurons on Types of Memory,” Neuroscience
Letters, vol. 158, pp. 181-184, 1993.
[3] E. Vaadia, I. Haalman, M. Abeles, H. Bergman, Y. Prut, H. Slovin and A.
Aertsen, “Dynamics of Neural Interactions in Monkey Cortex in Relation
to Behavioral Events,” Nature, vol. 373, pp. 515-518, Feb. 1995.
[4] Y. Sakurai, “Hippocampal and Neocortical Cell Assemblies Encode
Memory Processes for Different Types of Stimuli in the Rat,” Journalof Neuroscience, vol. 16, pp. 2809-2819, 1996.
[5] Y. Sakurai, “How Do Cell Assemblies Encode Information in theBrain?,” Neuroscience and Biobehavioral Reviews, vol. 23, pp.785-796.
1999.
[6] C. H. Sequin and R. Clay, “Fault Tolerance in Artificial Neural Networks,” Proc. IEEE-INNS’90, pp. 703-708, June. 1990.
[7] W. Hsieh and B. Sher, “Fault Tolerant Capability of Multi-Layer
Perceptron Neural Network,” EUROMICRO’94, pp. 644-650, 1994.
[8] V. Piuri, “Analysis of Fault Tolerance in Artificial Neural Networks,”
Journal of Parallel and Distributed Computing, vol. 61, pp. 18-48, Jan.
2001.
[9] Y. Uwate, Y. Nishio and R. Stoop, “Durability of Affordable Neural
Networks against Damaging Neurons,” IEICE Transactions on Fundamentals, vol. E92-A, no. 2, pp. 585-593, Feb. 2009.
[10] D.E. Rumelhart, G.E. Hinton and R.J. Williams, “Learning Internal
Representations by Error Propagation,” Parallel Distributed Processing,vol. 1, MIT Press, MA, pp. 318-362, 1986.

2008