2011 International Workshop on Nonlinear Circuits,Communication and Signal Processing
NCSP'11, Tianjin, China, March 1-3, 2011

Improvement of Neural Network Learning Performanceby Resting and Working State
Tatsuya Ide, Chihiro Ikuta, Yoko Uwate and Yoshifumi Nishio
Tokushima University
2–1 Minami-Josanjima, Tokushima, Japan
Phone:+81-88-656-7470
Email: {tatsuya, ikuta, uwate, nishio} @ee.tokushima-u.ac.jp

Abstract
Currently, many researchers research about relationships of arest to a work in a field of biomechanical science. Because
taking the rests improve eﬃciency of our works. In fact, wesometime decrease our concentration power by hard works.
Then, we must take a rest, because we become tired by asame work for a long time. However, if we take too muchrest, the work eﬃciency is decreased. We should consider thebalance between the rest and the work.
In this study, we propose a Multi-Layer Perceptron withResting State (RSMLP). The RSMLP has two diﬀerent states,
one of the state is a resting state, on the other state is aworking state. By computer simulations, we confirm that theRSMLP has better performance than the conventional MLP
and the MLP with random noise by learning a step function.

work eﬃciency is decreased. We should consider the balance
between the rest and the work.
In this study, we propose a Multi-Layer Perceptron withResting State (RSMLP). The RSMLP has two diﬀerent states,
one of the state is a resting state, on the other state is a working state. The RSMLP changes the two states during thelearning process and learn to tasks. The working state is thestandard BP learning. In the resting state, the RSMLP cuts theinput of learning task and continues the learning. We expectthat the RSMLP obtains a good performance by repeating theworking and the resting states. We confirm that the RSMLP
has better performance than the conventional MLP by learning a step function. Furthermore, we investigate the performance of the RSMLP when the ratio between the working
and the resting states is changed.
2. MLP with Resting State

1. Introduction

A Multi-Layer Percetron (MLP) is one of a feed forward
neural network. The MLP is known to be able to solve somekinds of tasks, for example, the pattern recognition, the pattern classification, the data maiming and so on [5][6]. This
network is composed of some neuron’s layers and learns tothe tasks by changing the weight parameters. The performance of the MLP is changed by the number of neurons andthe number of layers.
In this study, we consider that the RSMLP is composed ofthree layers (connected 1-5-1). The updating rule of neuron
is given by Eq. (1).
 n∑
xi (t + 1) = f  wi j (t)x j (t) − θi (t) ,(1)Back Propagation (BP) was proposed by D. J. Rumelhart
in 1987 [1]. The BP algorithm is used for learning of several artificial neural networks. In this algorithm, the network
calculate the error from the output and learning points. After
that, this error is propagating backward in the network. The
network can learn to tasks by the repeating this process. However, the BP algorithm often falls into the local minimum,
because this algorithm uses the steepest descend method. Inorder to avoid this problem, we propose a new scheme for thisalgorithm.
Currently, many researchers research about relationships
of a rest to a work in field of biomechanical science. Because
taking the rests improve eﬃciency of our works. In fact, wej=1
sometime decrease our concentration power by hard works.
Then, we must take a rest, because we become tired by a same where x: input or output, w: weight parameter and θ: threshwork for a long time. In general, we can refresh by taking old. We use sigmoidal function for output function as Eq. (2).
the short time rest. Some research groups have reported that1
our works grow in eﬃciency by taking appropriate amountf (a) =(2)1 + e−aof the rests [2]-[4]. However, if we take too much rest, our- 191 -1

Learning point

0.8

START

Input

0.6

0.4

0.2

Internal processingof MLP
00

0.2

0.4

0.6

0.8

1

Input

OutputFigure 2: Step function.
BPRestNO

Learning
time=10000

RestorWork

YES

Work

We use Mean Square Error (MSE) as the error function.
The MSE is given by Eq. (4).
1 ∑(ti − Oi )2 ,N iNENDMS E =(4)where ti is learning point, Oi is the MLP’s output and N is thenumber of learning points.
Figure 1: Flow chart of RSMLP.

3.1. Performance of RSMLP as Diﬀerent Ratio of Rest toWorkFirst, we compare the learning ability of the RSMLP bychanging the ratio of the working state and the resting state.
We change the time span of the working state and the restingstate little by little. The MLPs learn to the 21 point and theiteration time is fixed as 10000. Table 1 shows the simulation
results of the average MSE when the ratio of the working andthe resting states is changed. We obtain the average MSE of100 trials by changing the initial condition.

We show a flow chart of the learning process using theRSMLP in Fig. 1. The RSMLP has two diﬀerent states during the learning process. One of the state is the working state
which is learned by the BP in the same way as the conventional method. On the other hand, it is the resting state. During the resting state, the RSMLP is cut the input which islearning task, moreover, we give the random oscillation tothe RSMLP’s input. The two states are changed periodically.
The neuron updating in the input layer of the resting state is Table 1: Performance of the RSMLP as diﬀerent ratio of thedefined as Eq. (3).
rest to the work. (a) is the span of the resting state. (b) is thespan of the working state.
xi (t + 1) = f (ψ),(3)(b)

where ψ is the random oscillation as the noise.
3. Simulation Results

(a)

1020
40
60
80
100

100
0.004
0.030
0.119
0.152
0.157
0.160

200
0.004
0.004
0.034
0.057
0.067
0.100

400
0.003
0.005
0.008
0.017
0.030
0.034

600
0.004
0.004
0.005
0.011
0.014
0.019

8000.006
0.006
0.005
0.006
0.010
0.009

1000
0.009
0.004
0.005
0.005
0.007
0.007

We compare the learning performance of the RSMLP, theconventional MLP and the MLP with a random noise (whichis given random oscillation to neurons’ threshold) by learninga step function as shown in Fig. 2. The MLPs learn to eachIn Tab. 1, the errors of the upper triangular matrix are
point of vertical axis’s value from the input value (horizontal
smaller than the errors of the lower triangular matrix. Thus,
axis’s value).

- 192 -0.3
RSMLP
0.25

Random Noise
Conventional

0.2
MSE

the learning performance of the RSMLP becomes worse
when the ratio of the resting state to the working state becomes high. Moreover, the error of the upper right in Tab. 1
is worst than around the center of Tab. 1. We consider that
the MLP should be taken the enough rest, however, too much
rest gives bad influence to the learning ability of the RSMLP.
Figure 3 shows the relationship between the ratio of theresting state to the working state and the error. We can seethat if the ratio of resting state become high, the error increase
exponentially. The error is the smallest value when the ratioof the resting state is fixed from 0.02 to 0.1. Thus, we shouldchoose the appropriate ratio of the resting state to obtain thegood learning ability.

0.15
0.1
0.05
002500

5000

7500

10000

Learning Time

0.18

Figure 4: Learning curve of each MLP.

0.16
Average of error

0.14
0.12

0.018

0.1

0.016

0.08

0.014

0.06

0.012
MSE

0.04
0.02

Resting state

0.01
0.008
0.006

00.004

00.2

0.4
0.6
Ratio of rest to work

0.8

10.00206500

Figure 3: Average of error as each ratio of resting and working.

70007500

8000

8500

9000

Learning time

Figure 5: Learning curve of RSMLP as the local time.
3.2. Performance of the MLPs
Next, we compare the learning performance of the threekinds of MLPs which are the RSMLP, the conventional MLP
and the MLP with the random noise.
Figure 4 is an example of the error curves of the MLPs.
We can see that every MLP can reduce the error to around 0.
However, the RSMLP and the MLP with the random noisereduce error earlier than the conventional MLP. In general,
solution searching performance of the MLP becomes high byadding the noise. We consider that the RSMLP has similar
ability to the MLP with the random noise.
In Fig. 4, the learning curve of the RSMLP converges
smooth. Actually, the RSMLP is learned to false data during the resting state. We show the scale up learning curve as
ratio = 0.1 in Fig. 5. From this figure, we can see that theresting state has the certain period of time. The error of theRSMLP becomes large at the resting state, however, the error
is decreased more at the next working state.

Table 2 shows the average of the error (Avg. Err.), the minimum error (Min.) and the maximum error (Max.)) of the 100
trials. We used the RSMLP as the ratio to 0.1 for comparing
performance of the MLPs. From Tab. 2, the RSMLP is thebest of all for the average of the error and the maximum error. We consider that the RSMLP could reduce the influenceof the local minima from the maximum error to be became
small value by repeating the resting and the working states.
Finally, we show the recalling of the learning point by using the three MLPs in Fig. 6. We plot the learning data andthe output data of the MLPs after learning process. In thisresult, the recalling point of the conventional MLP is as sameas the recalling point of the MLP with random noise. They
could not learn enough to the learning data. In the RSMLP,
the recalling result was similar to the learning data than theothers. Thus, we consider that the resting state can give good
influence to the MLP learning.

- 193 -[3] P. M. Maki and S. M. Resnick, “Eﬀects of Estrogen on
Patterns of Brain Activity at Rest and during Cognitive
Activity: A Review of Neuroimaging Studies”, Neuroimage vol. 14, no. 4 pp. 789-801, 2001.

Table 2: Performance of the MLPs
Ave. Err.
0.00360
0.007640.00988

RSMLP (Ratio=0.1)
Random noise
ConventionalMin.
0.002440.001640.00177Max.
0.011910.172490.17321[4] E. Marx, A. Deutschländer, T. Stephan, M. Dieterich, M.
Wiesmann and T. Brandt, “Eyes Open and Eyes Closed
as Rest Conditions: Impact on Brain Activation Patterns,”Neurimage vol. 21, no. 4, pp. 1818-1824, 2004.
[5] H. Lappalainen and A.Honkela, “Bayesian Nonlinear Independent Component Analysis by Multi-Layer Perceptrons,” Advances in Independent Component Analysis,
Springer-Verlag, pp. 93-121, 2000.

Output10.8

Leaning data

0.6

RSMLP

0.4

MLP additionalrandom noise

0.2

Conventional MLP[6] H. C. Leung and V. W.Zue, “Phonetic Classification Using Multi-Layer Perceptrons,” Proc. ICASSP’90, vol. 1,
pp. 525-528, 1990.

000.2

0.4

0.6

0.8

1

Input

Figure 6: Recalling step function by MLPs.

4. Conclusion
In this study, we have proposed MLP with the resting state.
The RSMLP has two diﬀerent states which are the restingstate and the working state. During the working state, theRSMLP learned by standard BP algorithm. While, in the caseof the resting state, the input data was cut. The two states
were switched during the learning process.
By computer simulations, we confirmed that the RSMLP
was better than the conventional MLP and the MLP with random noise when the ratio of the resting state to the working
state was small. If the ratio of the resting state and the working state was large, the learning ability of RSMLP became
worse. Further, we showed the result of pattern recalling by
the RSMLP and the recalling result was close to the learning
data. Thus, we consider that the resting state can give good
influence to the MLP learning.
References[1] D. E. Rumelhart, G. E. Hinton and R. J. Williams,“Learning Representations by Back-Propagating Errors,”
Nature, vol. 323-9, pp. 533-536, 1986.
[2] M. E. Raichle, A. M. Macleod, A. Z. Snyder, W. J. Powers, D. A. Gusnard and G. L. Shulman, “A Default Modeof Brain Function,” Proc. National Academy of Sciencesof the U. S. A, vol. 98, no. 2, pp. 676-682, 2001.

- 194 -