2013 International Workshop on Nonlinear Circuits,Communications and Signal Processing
NCSP'13, Isrand of Hawaii, Hawaii, USA. March 4-7, 2013

Copying Weight Parameters in Back Propagation
Naohiro Shibuya, Yuta Yokoyama, Tomoya Shima, Chihiro Ikuta, Yoko Uwate and Yoshifumi Nishio
Tokushima University
2-1 Minami-Josanjima, Tokushima, Japan
E-mail: {shibuya, yuta, s-tomoya, ikuta, uwate, nishio}@ee.tokushima-u.ac.jp

Abstract

2. Back Propagation

The MLP is one of a feed-forward neural network. Thisnetwork is used for the function approximation [1], patternrecognition, pattern classification and pattern learning. TheMLP is composed some layers which are input layer, hidden
layer and output layer.
The BP is the learning algorithm of the parameter study in
the MLP [2]-[4]. The BP was introduced by D. J. Rumelhart
in 1986. The algorithm of the BP is listed below. First, theteaching signal is provided to the neural network for learning.
Second, the network calculates the error from the output andteaching signal. Finally, this error is propagating backward inthe network. The network can learn to tasks by the repeatingthis process. BP algorithm changes the value of weights to1. Introduction
obtain smaller error than before.
The following are equations of BP. The output function isThe human brain is classified into the cerebrum, cerebeldescribedby Eq. (1). Moreover, the internal state and sigmoid
lum and brain stem. It is able to process the complex inforfunction
aredescribed by Eqs. (2) and (3).
mation because different parts of the brain have various functions. One of the reason is that the cerebellum has a particularxi (t + 1) = f (ui (t + 1)) ,(1)function. This function is that the cerebellum copies information in the cerebrum. For example, in the case of motion ofhuman, the cerebellum copies rough information of motion∑ui (t + 1) =wij xj (t),(2)
in the cerebrum and learns more detail motion. Thereby, thej
human can do detailed motion.
We apply such function to the artificial neural network operating the Back Propagation (BP). The neural network is the1,(3)f (a) =
mathematical model and be able to actualize brain function by
1 + e−acomputation simulation. The BP is the technique of the parameter study in the neural network. When we actualize the where x is the input or output, u is the internal state and w isfunction of the cerebellum, we add two processing to conven- the weight parameter. The square error used for error evaluational BP. First, the hidden layer in the Multi-Layer Percep- tion is described by Eq. (4).
tron (MLP) is divided into two groups. Second, the connecn
tion weight parameter of neurons in the one group are copied1∑E=(ti − Oi )2 ,(4)into neurons in the other group. We hope the contraction of
2 i=1
the learning time, high efficiency and accuracy learning byapplying this function. Because, the MLP learns by the cerewhere E is the error, n is the number of output, t is the target
brum group. After that, the MLP learns particularly by thevalue and O is the output value.
cerebellum group. In this study, we prove that the learning
performance of the proposed network is better than the conventional network by applying the cerebellum function.
The human brain is able to process the complex information.
One of the reason is that the cerebellum has a particular function. This function is that the cerebellum copies informationin the cerebrum. We focus on the function of the cerebellum.
In this study, we apply such function to the artificial neuralnetwork operating the Back Propagation (BP). We actualize
the function of the cerebellum by dividing the hidden layer
into two groups. The weight parameter of neurons in one
group are copied into neurons in the other group. We confirm that the learning performance of the proposed network isbetter than the conventional network.

- 552 -3. Structure and Algorithm
We consider a feed-forward neural network with three layers. The composition of the proposed network is shown inFig. 1. The hidden layer is divided into two groups. The one
group is the cerebrum group, and the other group is the cerebellum group. The cerebrum and the cerebellum group has
4 neurons. The algorithm of the proposed network is listed
below. First, the MLP learns several times by the cerebrum
group in the hidden layer and updates the weight parameter. Before copying, the weight parameters in the cerebellum
group are shown in Fig. 2. Second, the connection weightparameter of neurons in the cerebrum group are copied into
the cerebellum group. By copying, the weight parameters in
the cerebellum group are shown in Fig. 3. Finally, the MLP
learns by the cerebellum group in the hidden layer and updates the weight parameter.

Figure 3: After copying the weight parameter.

4. Simulations
We apply the function approximation to the proposed network. We input the 2 dimensional and 3 dimensional Chebyshev polynomial to the network. We consider the 2 dimensional and 3 dimensional Chebyshev polynomial. The functions of the 2 dimensional and 3 dimensional Chebyshevpolynomial are described by Eqs. (5) and (6).
T2 (x) = 2x2 − 1,(5)T3 (x) = 4x3 − 3x.

(6)

Figure 1: Composition of the proposed network.
We compare the learning curve of the proposed networkand the conventional network. In this study, we consider thatthe number of neurons in the hidden layer is 12. The numberof neurons in the cerebrum group is set to 4. The number ofneurons in the cerebellum group is set to 8. The number ofneurons in the input layer and the output layer is 1. The learning time is 20000. The learning late is 0.005. The starting
value of the weight parameter is randomly set for composing
the network.
4.1 2 dimensional Chebyshev Polynomial
4.1.1 Comparison between Proposed and Conventional Networks

Figure 2: Before copying the weight parameter.

In this section, we compare the proposed network and theconventional network. We investigate effects on the error of
the network if the weight parameter of the neurons in the cerebrum group are copied into the cerebellum group. Here, theerror of the network is average of 10 random initial values.

- 553 -

We input the 2 dimensional Chebyshev polynomial. The algorithm of the proposed network is listed below. From 0 until
4000 learning times, the MLP learns by the cerebrum groupin the hidden layer and updates the weight parameter. At
400 learning times, the weight parameter of the neurons in
the cerebrum group are copied into neurons in the cerebellum group. From 4001 learning times, the MLP learns by thecerebellum group in the hidden layer and updates the weightparameter. The simulation result is shown in Fig. 4.

Figure 5: Comparison with three proposed networks.

4.1.3 Average of Results
We give 100 random initial values. The results are averageof error and minimum value in the final learning time. The
results are shown in Table 1.
Table 1: Average of results

Figure 4: Comparison between proposed and conventional network.

Conventional network

From 0 until 4000 learning times, the conventional network is better than the proposed network. At 4000 learning
times, the error of the proposed network increases because
the weight parameter of neurons in the cerebrum group arecopied into neurons in the cerebellum group. From 4001
learning times, suddenly the error decreases. Finally, thelearning performance of the proposed network is better thanthe conventional network. However, the convergence speedof the proposed network is slower than the conventional network.

Number of neurons

4812

Average of error

8.02E − 03

1.78E − 04

1.64E − 04

Minimum value

8.27E − 03

5.42E − 05

2.29E − 05

Timing of copying

500

4000

8000

Average of error

1.42E − 06

1.54E − 06

2.55E − 06

Minimum value

1.82E − 09

2.12E − 08

5.08E − 09

Proposed network

From Table 1, we can see that the error of the conventional
network is the worst. The learning performance of the copy
in 500 learning times is the best of all.

4.1.2 Comparison with Three Proposed Networks
In this section, we compare three proposed networks. The
timing of copy of the weight parameter is set to 500, 4000
and 8000 learning times. We investigate effects on the errorof network if the timing of copy of the weight parameter ischanged. The error of network is average of 10 random initialvalues. The simulation result is shown in Fig. 5.
From Fig. 5, we can see that the error of the proposednetwork (500) is the best. Moreover, the proposed network(8000) is the worst. We consider that the timing of copyof the weight parameter relates to convergence of the error
curve. Therefore, we assume that there is the optimal timingof copy of the weight parameter at before convergence of theerror curve.

4.2 3 dimensional Chebyshev Polynomial
4.2.1 Comparison between Proposed and Conventional Networks
In this section, we compare the proposed network and theconventional network. Here, the error of network is the average of 10 random initial values. We input the 3 dimensional
Chebyshev polynomial. The simulation result is shown inFig. 6.
From this result, the learning performance of the proposed
network is better than the conventional network in the final
learning time. If we give the complex input, the learning performance is better by copying the weight parameter.

- 554 -

Table 2: Average of result
Conventional network
Number of neurons

48

12

Average of error

1.58E − 04

1.17E − 04

2.67E − 04

Minimum value

6.60E − 05

4.29E − 05

5.91E − 05

Timing of copying

500

40008000

Average of error

1.34E − 06

1.63E − 06

2.15E − 06

Minimum value

2.52E − 09

5.30E − 08

1.58E − 08

Proposed network

Figure 6: Comparison between proposed and conventional network.
ing at 500 learning times is the best of all.
4.2.2 Comparison with Three Proposed Networks
In this section, we compare the three types proposed network. The timing of copy of the weight parameter is set to 5. Conclusions
500, 4000 and 8000 learning times. The error of network is
In this study, we applied the function of the cerebellum toaverage of 10 random initial values. The simulation result isthe BP. When we actualize the function of the cerebellum, weshown in Fig. 7.
add two processing to the conventional BP. First, the hidden
layer in the MLP is divided into two groups. The one groupis the cerebrum group, and the other group is the cerebellum
group. Second, the connection weight parameter of neurons
in the cerebrum group are copied into neurons in the cerebellum group.
We applied the function approximation to the proposed network. From these results, the learning performance of theproposed network is better than the conventional network.
However the convergence speed of the proposed network isslower than the conventional network. Moreover, we considered that the timing of copy of the weight parameter relate to
the convergence of error curve.
Figure 7: Comparison with three proposed networks.
Acknowledgment
From this result, if we give the complex input, the error of This work was partly supported by MEXT/JSPS Grant-in-Aid
the proposed network (500) is the best. Moreover, the pro- for JSPS Fellows (24·10018).
posed network (8000) is the worst. Because, the timing ofReferences
copy of the weight parameter of proposed network (8000) isthe slowest of all. In the case of 3 dimensional Chebyshev [1] Y. Uwate and Y. Nishio: ”Durability of Affordable Neuralpolynomial, the timing of copy of the weight parameter reNetworks against Damages”, International Joint Conference onlates to convergence of the error curve .
Neural Networks (IJCNN’06), pp.8365-8370, July. 2006.
4.2.3 Average of Results[2] D.E. Rumelhart, G.E. Hinton, and R.J. Williams: ”Learning
representations by back-propagating errors”, Nature, vol.323-9,pp.533-536, 1986.

We give 100 random initial values. The results are average[3] D.E. Rumelhart, J.L. McClelland, and the PDP Research Group:of error and minimum value in the final learning time. The”Parallel distributed processing”, MIT Press, 1986.
results are shown in Table 2.
[4] Takeshi Agui, Hiroshi Nagahashi, Hiroki Takahashi: Neural
From Table 2, we can see that the error of the conventional
Program, Shokoudou corp, pp.11-18, 20-42, 1995.
network is the worst. The learning performance of the copy-- 555 -