2010 International Symposium on Nonlinear Theory and its Applications
NOLTA2010, Krakow, Poland, September 5-8, 2010

Scale-Free Property of Aﬀordable Neural Network
Yoshifumi Nishio and Yoko Uwate
Dept. of Electrical and Electronic Engineering, Tokushima University
2-1 Minami-Josanjima, Tokushima, Japan
Email: {uwate, nishio}@ee.tokushima-u.ac.jp
Abstract—We have recently proposed a novel neuralnetwork structure called an Aﬀordable Neural Network(AfNN), in which aﬀordable neurons of the hidden layer
are considered as the elements responsible for the robustness property as is observed in human brain function. We
have confirmed that the AfNN gains good performanceboth of the generalization ability and the learning ability.
In this study, we focus on the firng frequency of neurons in
the hidden layer and the amount of weight changes during
the learning process. By computer simulations, we confirm
that the AfNN generates scale-free properties.

(a) Random network.

Figure 1: Example of random and scale-free network.
# of nodes

1. Introduction# of nodes

Bell Curve

Since the scale-free networks were discovered by
Barabasi et al. [1], studies assessing the influence of thisproperty on the eﬃciency of networks have been carriedout in various fields [2]-[4]. One way of how to characterize the diﬀerence between random and scale-free networks is by means of the distribution of the number oflinks which a node has. From Fig. 1, where we contrast
the two network types, it is evident that in the scale-freenetwork, although most nodes only have few connections,some nodes act as highly connected hubs. This distinction
is captured in a more quantitative way by the distributionof the number of links vs. the number of nodes, as shownin Fig. 2. Random networks display a bell-shaped curve,
implying that most nodes have the same number of links,
and no highly connected nodes (see Fig. 2(a)). Scale-freenetworks, in contrast, often have many nodes with a few
links only, whereas quite a few hubs exist that have a large
number of links. Mathematically, scale-free networks arecharacterized by power law distributions (Fig. 2(b)). Because scale-rules emerge in many areas and disciplines ofscience (e.g. engineering, economics, social sciences andso on), we expect that also in the development of the science of complex networks, they will play an important role.
In a previous study on artificial neural networks, we proposed a new network structure with aﬀordable neurons in
the hidden layer, for eﬃcient BP-learning [5]. We christened this network “Aﬀordable Neural Network (AfNN).”In this network, some extra neurons are inserted into thehidden layer. By computer simulations [5], the AfNN has
been confirmed to achieve an improved performance overconventional networks for BP-learning, in terms of speedof convergence and of learning eﬃciency. Moreover, we(b) Scale-free network.

# of links(a) Random network.

Power Law
distribution# of links(b) Scale-free network.

Figure 2: Comparing random and scale-free distribution.

have investigated the performance of the AfNN for noisepolluted input data. We found that the AfNN is able togenerate noise-cleaned outputs, which leads to the conclusion that the AfNN has the generalization property. Furthermore, the AfNN has a kind of durability, because theAfNN still performs well even if some of neurons in thehidden layer are damaged after learning process. We haveconfirmed that the AfNN can operates with keeping its efficiency against damaging neurons [6]. However, we believe that many advantageous characteristics of the AfNN
are yet to be unveiled, and we also believe that the operation of the AfNN embodies important general features ofthe BP-learning process.
In this article, we investigate the scale-free property ofthe AfNN, which can be regarded as a complex networkwith time-varying connections, in order to clarify the relationship between the network topology and its informationprocessing ability. First, we investigate the characteristicsof the firng frequency of neurons in the hidden layer andthe amount of weight changes during the learning process.
Next, we introduce the scale-rule selection of aﬀordable
neurons and its eﬀect is examined by computer simulations.

- 310 -

2. Aﬀordable Neural Network (AfNN)
In [5], we introduced the AfNN to reflect important properties of the brain. During BP-learning, not all of the neurons in the hidden layer are used at every updating: some of
the neurons are selected for the learning and the rest of theneurons are deactivated. The underlying network model issketched in Fig. 3. The aﬀordable neurons are selected byrandom every updating time (see Fig. 4). The definition of
the aﬀordable neurons is described as follows.
• The output of aﬀordable neurons is set to zero.
• The weight vectors connected to the aﬀordable neurons are not updated.

Output layer
Input layer

Affordable
Neurons

Figure 5: Pattern recognition.
In this case, the number of neurons in the input layer is25, and we choose 100 hidden layer neurons. The numberof neurons in the output layer is 10. The network learns
these 10 numeric characters with 100000 iterations and theerror curve converges to almost zero. The network parameters are fixed to η (learning rate)=0.05 and ξ (proportionality factor)=0.002. After learning process, the recognition
rate is investigated by using the set of 10 patterns shifted 1
bit from each original pattern. We confirm that the learned
networks can obtain around 70 to 90 percent recognition
rate.
Next, we investigate the following characteristics of theAfNN:
1. Firing frequency for each neuron in hidden layer.

Hidden layer

2. Amount of weight changes between input/hidden andhidden/output layers.

Figure 3: Aﬀordable neural network.

Figure 6 shows the simulation result of the frequency
of firing neurons in the hidden layer by using multi trial.
The threthold of firing is set to 0.5. This graph is shownwith double logarithmic plot. From Fig. 6, the graph curvealmost follows the straight line which means this graph hassome scale-free properties.

Hidden layer structure

Number of neurons

10000

Learning time
Operating neuron
Affordable neuron (rest neuron)

Figure 4: Random selection of aﬀordable neurons.

10001001012

Firing frequency

1e+06

Figure 6: Frequency distribution for firing neurons in hidden layer.

3. Pattern Recognition by Random Selection
In this study, we use the batch BP-learning algorithm.
The batch BP-learning algorithm can be expressed similarly to the standard BP-learning algorithm [7], with thediﬀerence lying in the timing of the weights updating.
As the first task, we consider the pattern recognition
where 10 numeric characters (Fig. 5) are fed into the neuralnetwork for recognition.

Figure 7 shows the frequency distribution for total adjustment of weight vectors between input and hidden layers. In this case, the graph curves does not have any rules.
While in the case of he total adjustmen of weight vectors
between hidden and output layers as shown in Fig. 8, thegraph curve shows the straight line.

- 311 -

The total adjustment of weight vectors between input and hidden layers corresponds with the input pattern
deeply. Then, the scale-free property does not occur forthe total adjustment of weight vectors between input andhidden layers.

Number of neurons

100000

10000

1000

100

1011

10

Amount of weight changes

Selected times for operating neuron

Figure 7: Frequency distribution for total adjustment ofweight vectors between input and hidden layers.

that the values of the entries follow a power-law distribution. Using these values, we select in each update the setof active neurons according to the values of S . From the sneurons in the hidden layer, exactly the k neurons with thesmallest entries are chosen as the aﬀordable neurons. Figure 9 illustrates this scale-rule selection of the aﬀordable
neurons, where the number of the neurons in the hidden
layer is set to be 100 and the number of aﬀordable neurons is 20, 40 and 60, respectively. Our simulations will
be based on 100000 updates. In this Figure, the horizontal axis indicates the neuron number, whereas the verticalaxis displays the number of times the corresponding neuron was in the set of operating neurons. By this Figure
it is confirmed that the operation time decreases graduallywith the neuron number of the hidden layer. Furthermore,
histograms of the number of neurons that have a given operation time are shown in Fig. 10. The resemblance withthe scale-free distribution of Fig. 2(b) is evident, although,when inspected in details, the distribution is not of a simple power law type. For comparison let us also consider
random selection of aﬀordable neurons.

Number of neurons

100000

10000

1000

100

10000090000
80000

Affordable neuron: 2070000
60000
Affordable neuron: 40

50000
40000
30000

Affordable neuron: 60

20000
10000
00th

101

10th

20th

30th

40th

50th

60th

70th

80th90th

100th

Neuron number of hidden layer

10

Amount of weight changes

Figure 9: Scale-rule selection (Hidden: 100).

Figure 8: Frequency distribution for total adjustment ofweight vectors between hidden and output layers.

30

Number of neurons

25

4. Scale-Rule Selection of Aﬀordable Neurons [7]In this section, we introduce the simulation results ofpast study in [7]. The scale-rule selection of aﬀordable neurons has been investigated. Our scale-rule selection procedure is described in terms of a parameter denoted by a vector S . The dimension s of S equals the number of neurons
present in the hidden layer; each component of S corresponds to one single neuron indexed by i. The values of thecomponents are evaluated in each update by
S i = random()/i2(1)

where random() means the uniform random function producing values from 0.0 to 1.0. This implies that the neuron
with the highest index will generally have a small value,
whereas the first neuron will – unless the random function states something diﬀerent – have a larger entry. Note2015105040000

50000

60000

70000

80000

90000

Selected time for operating neuron

Figure 10: Distribution of scale-rule selection (Aﬀordable
neuron: 40, Hidden: 100).
For our simulations we want to teach our network to generate typical time series of the skew tent map. To this end,
the network is trained – using time series of the tent map– to output, starting from given initial conditions, the same- 312 -0.05

time series as the tent map would have generated.
The skew tent map and an example of time series areshown in Fig. 11. The length of chaotic time series is set
to 50 steps; the size of the set of learning patterns is 20. In
our approach, this requires the network to have 50 nodes
in the input and the output layers. Each data is inputted toeach node in the input layer. We carried out BP-learning
by using the following parameters. The parameter of thelearning rate and the inertia rate are fixed at η = 0.05 andζ = 0.02, respectively. The initial values of the weights arechosen between −1.0 and 1.0 at random. The learning timeis set to 5000.

scale-rule selection
random selection

Eave

0.04

0.03

random0.02

scale-rule
0.01

0010

conventional
network

2030

40

50

60

7080

Number of affordable neurons

110.9
Chaotic time seriesx(t+1)

0.8
0.6
0.4
0.2

00.2

0.4

0.6

0.8

0.7
0.6
0.5
0.4
0.3
0.2

a=0.550

Figure 12: Learning ability by changing the number
of aﬀordable neurons (Number of neurons in hidden
layer: 100).

0.8

0.1
0

1

05

10

15

20

x(t)

(a) Skew tent map.

25 30
Time3540

45

computer simulations, we have confirmed that the AfNNcould generate scale-free properties.

50(b) Time series.

Acknowledgment

Figure 11: Skew tent map.
We investigate the learning eﬃciency as the average ofthe total error between the output and the desired target,
when the network structure of the hidden layer is changed.
The “Average Error Eave ” for this learning example is defined by the following equation.
Eave =P {∑11(t p − o p )2
P p=1 2}
(2)

We consider the case that the hidden layer consists of 100
neurons. The number of the aﬀordable neurons is variedfrom 10 to 70. The results of this simulation are shown inFig. 12, where the horizontal axes are the number of theaﬀordable neurons and the vertical axes are Eave for thepattern learning. From this Figure, we can confirm that thescale-rule selection method achieves a better performance
if compare to the random selection. It is also seen thatthe diﬀerence between the errors of the scale-rule and therandom selection networks increases with the number ofaﬀordable neurons. Even when the number of aﬀordable
neurons becomes large, the scale-rule selection networkcontinues to show good learning ability. From this result,
we can conclude that the scale-rule selection method of affordable neurons could play an important role for learning
processes, in particular in biological systems.
5. Conclusions
In this study, we have investigated the characteristics ofthe firng frequency of neurons in the hidden layer and theamount of weight changes during the learning process. By

This work was partly supported by JSPS Grant-in-Aidfor Scientific Research 22500203.
References[1] Barabasi, Albert-Laszlo and A. Reka, “Emergence of Scalingin Random Networks,” Science, 286:509-512, Oct. 1999.
[2] W.-X. Wang, B.-H. Wand, C.-Y. Yin, Y.-B. Xie and T. Zhou,“Traﬃc Dynamics Based on Local Routing Protocol on a
Scale-Free Network,” Physical Review E, vol. 73, no. 2,026111, Feb. 2006. [3] W. Tam, F.C.M. Lau and C.K. Tse, “Complex-Network Modeling of a Call Network,” IEEE Transactions on Circuits
Syst. I, vol. 56, no. 2, pp. 416-429, Feb. 2009. [4] X. Zheng, F.C.M. Lau and C.K. Tse, “Constructing HighRate Scale-Free LDPC Codes,” Proc. ISCAS’10, pp. 37813784, May 2010.
[5] Y. Uwate and Y. Nishio, “Performance of Aﬀordable Neural
Network for Back Propagation Learning,” IEICE Transactions on Fundamentals, vol. E89-A, no. 9, pp. 2374-2380,
Sep. 2006.
[6] Y. Uwate, Y. Nishio and R. Stoop, “Durability of Aﬀordable
Neural Networks against Damaging Neurons,” IEICE Transactions on Fundamentals, vol. E92-A, no. 2, pp. 585-593,
Feb. 2009.
[7] D.E. Rumelhart, G.E. Hinton and R.J. Williams, “Learning
Internal Representations by Error Propagation,” Parallel Distributed Processing, vol. 1, MIT Press, MA, pp. 318-362,
1986.
[8] Y. Uwate, Y. Nishio and R. Stoop, “Scale-Rule Selection ofAﬀordable Neural Network for Chaotic Time Series Learning,” Proc. ECCTD’07, pp. 811-814, Aug. 2007.

- 313 -