RISP

NCSP’16

International Workshop on
Nonlinear Circuits, Communications and Signal Processing

Honolulu, Hawaii, USA, March 6-9, 2016

Improving Back Propagation of Feed-Forward Neural Networkwith Changing Sigmoid Functions
Shinsaburo Kittaka, Yoko Uwate, and Yoshifumi Nishio
Dept. of Electrical and Electronic Engineering, Tokushima University
2-1 Minami-Josanjima, Tokushima 770–8506, Japan
Email: {kittaka, uwate, nishio}@ee.tokushima-u.ac.jp

Abstract
Our study is about feed-forward neural network’s learning
method. Generally, the method of improving its learning isfocused on learning rate and moment term. We focus on sigmoid functions. Sigmoid functions are used for converting input signal into output signal and adjusting connection weightof learning in feed-forward neural network. We change gradient of sigmoid functions and investigate our method’s effect.

1. Introduction
Neural network is constituted by some element that iscalled neuron. This network is based on animal brain. With
Figure 1: Feed-forward neural networks composition .
similarity brain, it is appropriate in recognize written characters and recognize voice. Neural network’s features are paralSignal runs only one direction. Neuron converts input siglel processing and learning. We focus on learning. Neuron inneural network transforms input signal to output signal with nal to output signal that is from 0 to 1 with sigmoid funcitons.
sigmoid functions and adjusts connection weight from error
between actual value and estimated value.
1f (x) =(1)We regard that learning system is appropriate in recognize
1 + exp(−x)
brain signal, because brain signal has difference with each
person and condition. In other research, the recognition brain
signal rate is 86 percent [1] [2]. We submit that we should
increase this rate. In this study, we focus on Feed-forward
neural network’s learning method. Feed-forward neural network’s learning method is back propagation. We change sigmoid functions and improve back propagation.

2. Feed-forward neural network
Feed-forward neural network is one of neural network. It
consists of input layer, hidden layer and output layer. Figure
1 shows feed-forward neural network’s composition.

- 614 -

Figure 2: Sigmoid functions.

2.1. Back propagation“k” by 0.001 at iteration. Finally, we change the decreasing
value of “k”.

Back propagation is learning method of feed-forward neural network. Estimated value is inputed into output layer.
Output layer calculates the difference of actual value and estimated value.(2) The connection weight that is between neuron and neuron adjusts based on this value.(3) fn (x) is actualvalue and Yn is estimated value. W is the value of connection weight, η is learning rate and netj is input signal fromall connecting neurons.
δ = fn (x) − Yn0∆W = ηδn f (netj )(2)

4. Simulation result
In this simulation, we investigate 2 simulations. One is Iris
plants and another one is sin function. We define parameters
as learning iteration = 1000 and the number of hidden layer’s
neurons = 4.

4.1. Iris plants(3)

We divide Iris plants into 3 types. There are total of 150
data entries. The data records 4 attributes and 3 classifications. The 4 attributes are sepal length, sepal width, petal
length and petal width. The 3 types of iris flower are Iris Setosa, Iris Versicolor and Iris Virginica. First, we search the3. Proposed method
learning rate ( a ) that has the smallest error. Table I shows
In this study, we change gradient of sigmoid functions. We the error with changing learning rate from 0.1 to 1.0.
decrease the parameter “k” that is coefficient added into natTable 1: The error with changing aural logarithm’s multiplier.
Repeating this calculate makes output signal perfect.

f (x) =1
1 + exp(−kx)

aave
aave

(4)

Figure 3 shows sigmoid functions with changing “k” andFig. 4 shows differential form of sigmoid functions.

0.1
0.028960.6
0.02144

0.20.02438
0.7
0.02079

0.3
0.02974
0.8
0.02137

0.40.02261
0.9
0.021120.5
0.02144
1.00.02144

We define as the learning rate = 0.7, because it has thesmallest error. We pursue simulation with this learning late.
Next, we search the “k” that has the smallest error. Table IIshows the error with changing the “k” from 0.6 to 1.5.
Table 2: The error with changing “k”
kave
kave

Figure 3: Sigmoid functions with “k”.

0.6
0.02295
1.10.02076

0.7
0.022311.2
0.022370.8
0.02177
1.3
0.022670.9
0.022211.4
0.023641.00.02079
1.50.02365

In this simulation, we obtain smaller error than the original
method. However, it is very small difference. We consider
that this simulation does not obtain good results. Next, wedecrease “k” by 0.001 at iteration and change the starting decreasing time. Table III shows the error with decreasing “k”.
In this simulation, we obtain very small error. This method
is effective to improve back propagation. In Table III, bestparameters are “k” = 0.9, the decreasing value = 0.001 andthe starting decreasing time = 500. Finally, we change the decreasing value of “k”. We set best parameters in Table III andinvestigate the error with changing decreasing value of “k” atiteration from 0.0006 to 0.0015. Table IV shows results.

Figure 4: Differencial form of sigmoid functions.
First, we search learning late that has the smallest error.
Second, the parameter “k” is changed. Third, we decrease- 615 -Table 4: The error with changing decreasing valuevalue
ave
value
ave

0.00060.017480.00110.015440.00070.015470.00120.017110.00080.014900.00130.012140.00090.012710.00140.01693

Table 8: The error with changing decreasing value

0.001
0.010780.00150.01285

The smallest error is 0.01078 with the decreasing value of“k” = 0.001. In this simulation, we obtain half of the original
method’s error.

value
ave
value
ave

0.00060.010350.00110.01198

0.00070.00906
0.00120.012730.00080.011770.00130.013030.00090.010440.00140.01672

0.001
0.011180.00150.01688

The smallest error is 0.0906 with the decreasing value =0.007. We obtain smaller error than the original method.
5. Conclusion

4.2. Sin function
Next, we transform input signal from angle into sin function. There are total of 180 data entries. The data records
angle from 0 to 180 and sin function from 0 to 1. First, wesearch the learning rate ( a ) that has the smallest error. In
this time, the learning late that has smallest error is big. So,table V shows the error with changing learning rate from 2.1
to 3.0.

Our learning method is changing gradient of sigmoid functions by decreasing “k” at iteration for back propagation in
feed-forward neural network. In these simulations, we obtainsmaller error than the usual method. However, if the “k” issmall and starting decreasing time is early, our method makes
error very large. So, this method requires attention to set parameters.
In the future work, we resolution that why the error becomes small with our method.

Table 5: The error with changing aaave
aave

2.10.01716
2.60.01651

2.20.01667
2.70.01632

2.30.01678
2.80.01527

2.40.01592
2.90.01503

2.50.01687
3.0
0.01541

References

We define the learning rate = 2.9, because it has the smallest error. We pursue simulation with this learning late. Next,
we search the “k” that has the smallest error. Table VI showsthe error with changing the“k” from 1.1 to 2.0.
Table 6: The error with changing “k”kavek
ave

1.10.01604
1.60.01383

1.20.01415
1.70.01331

1.30.01493
1.80.01422

1.40.01349
1.90.01311

1.50.01511
2.00.01335[1] A. Hiraiwa, N Uchida, K Shimohara, N Sonehara“EMG Recognition with a Neural Network Model for
Cyber Finger Control,” Transactions of the Society ofInstrument and control Engineers, vol. 30, no.2, pp.216224, 1994.
[2] S. Kitayama, M. Sasaki, S. Ito, “Limb motion estimation by Bereitschaffs Potentian,” Journal of the Japan
Society of Applied Electromagnetics and Mechanics,
vol. 22, no. 2, pp. 318-323, 2014.

We obtain smaller error than the original learning. 1.9 isthe best parameter of “k” in table VI. Next, we decrease “k”
by 0.001 at iteration and change the starting decreasing time.
Table VII shows the error with decreasing “k”.
This method has smaller error than the orginal method.
However, we can not obtain effective results like first simulation. In this simulation, best parameters are “k” = 2.0,
the decreasing value = 0.001 and the starting decreasing time= 200. Finally, we change decreasing value of “k”. We set
best parameters in Table VII and investigate the error withchanging decreasing value of “k” at iteration from 0.0006 to0.0015. Table VIII shows results.

- 616 -Table 3: The error with changing “k”
kstarting decreasing time

iteration

0100
200
300
400
500
600
700
800
900

0.6
0.33406
0.227130.135940.108160.02578
0.01394
0.10579
0.02457
0.01371
0.01378

0.7
0.199770.17337
0.164100.094960.03072
0.01325
0.02672
0.022900.01937
0.012280.8
0.182390.213370.178440.083140.05036
0.011220.01150
0.03057
0.026130.011940.9
0.013150.012440.01165
0.01304
0.011800.010710.01072
0.010900.01386
0.01336

1.0
0.01150
0.01225
0.011430.01106
0.01395
0.01101
0.011480.01183
0.013360.012501.1
0.01282
0.01301
0.013030.01465
0.01356
0.01301
0.012190.013780.014090.01498

1.2
0.015760.015090.01492
0.01575
0.01587
0.013250.015890.016420.014710.017351.3
0.01674
0.016220.015910.015140.01697
0.01556
0.01560
0.016100.01558
0.015281.4
0.017800.01732
0.01598
0.01659
0.016440.016170.01574
0.01682
0.016440.016101.5
0.017470.017430.017070.016900.01697
0.015840.016390.016240.016640.016481.6
0.01684
0.017440.01657
0.014410.015620.01749
0.01752
0.01876
0.02429
0.02658

1.7
0.01526
0.014850.014700.01495
0.01487
0.01547
0.01601
0.01899
0.022360.027411.8
0.01552
0.01508
0.013290.01199
0.013220.014220.015210.01753
0.01889
0.023931.9
0.015470.01467
0.01146
0.01204
0.012370.013210.015200.01568
0.01892
0.02168

2.0
0.01291
0.01205
0.01097
0.013070.012410.013070.01392
0.015400.016420.01897

Table 7: The error with changing “k”
kstarting decreasing time

iteration0100
200
300
400
500
600
700
800
900

1.1
0.029620.025280.024240.031760.029250.03032
0.03175
0.03351
0.03773
0.060381.2
0.02434
0.02261
0.02661
0.02379
0.02362
0.02692
0.030120.02595
0.030350.04008

1.3
0.02279
0.021450.022400.02006
0.022130.02504
0.023930.025900.02523
0.03369

1.4
0.018340.02059
0.01832
0.01779
0.020020.022030.02003
0.02278
0.02360
0.02365- 617 -

1.5
0.018290.01926
0.01699
0.01650
0.02130
0.01798
0.017910.01905
0.01987
0.02586