2011 International Workshop on Nonlinear Circuits,
Communication and Signal Processing
NCSP'11, Tianjin, China, March 1-3, 2011

Modified Back Propagation Algorithm for CNN Template Design
Masashi Nakagawa† , Yoko Uwate† and Yoshifumi Nishio‡†Tokushima University
2-1 Minami-Josanjima, Tokushima, Japan
Phone:+81-88-656-7470, Fax:+81-88-656-7471
Email: {miyabi, uwate, nishio}@ee.tokushima-u.ac.jp

dition, we examine the performance of this learning algorithm
In the previous study, we have proposed a template design for two-dimensional binary image.
method of cellular neural networks with back propagation al- 2. Cellular Neural Networks
gorithm. In that method, template learns by using the average
In this section, we explain the basic structure of the CNN.
error which corresponds to the diﬀerence between the out- The CNN has M by N processing unit circuits called cells.
put image and the desired image. In this study, we modify The cells are arranged in a reticular pattern to M line N row.
the back propagation algorithm for cellular neural networks We represent a cell C(i, j) using a variable i which denotes
template design. We inspect the performance of this learning vertical position and a variable j which denotes horizontal
algorithm for two-dimensional binary image.
position. The cell contains linear and nonlinear circuit eleAbstract

1. Introduction
Cellular neural networks (CNN) were introduced by Chua
and Yang in 1998 [1]. The idea of the CNN is inspired fromthe architecture of the cellular automata and the neural networks. Unlike the conventional neural networks, the CNN
has local connectivity property. Since the structure of the
CNN resembles the structure of animals retina, the CNN canbe used for various image processing applications.
Wiring weights of the cells of the CNN are established byparameters called the template. The template is most important parameter, because performance of the CNN is determined by the template. Thus, some template design methods, e.g., template learning using genetic algorithm (GA),have been proposed. These works are important subject inthe studies of the CNN.
In the previous study, we have proposed a template learning method of CNN using the back propagation (BP) algorithm [2]. As well known, BP is one of the supervised learning methods [3]. In that method, the template of CNN is dynamically updated from the error between an output image
and a desired image as BP neural networks. We have confirmed that the method could work for a simple example using one-dimensional binary image by computer simulations.
Furthermore, we have investigated the method for gray scale
image. However, in the case of applying for gray scale images, we have not obtained good results. This is because theprevious method was a technique of imitating the process of
conventional BP. Therefore, even if the method was able toprocess binary images, the method was not able to processgray scale images. Thereupon, in this study, we modify theback propagation algorithm for CNN template design. In ad-ments. The CNN is an array of cells. Each cell is connected
to its neighboring cells according to a template. Usually, thetemplate is the same for all cells except for boundary cells.
The CNN has the features of time continuity, spatial discreteness, nonlinearity and parallel processing capability.
The state equation and the output equation of the cell are
shown as the following equations.
State equation:
j+r
i+r ∑
∑
dv xi j (t)= −v xi j (t) +A(i, j;k,l) vykl (t)dtk=i−r k= j−r
+

j+r
i+r ∑
∑

B(i, j;k,l) vukl (t) + I(1)

k=i−r k= j−r

Output equation:
vyi j (t) =1(|v xi j + 1| − |v xi j − 1|)2(2)

where v x , vy and vu represent a state, an output and an inputof cell, respectively. In the equation (1), A is the feedback
template and B is the control template. These and bias I arecollectively called general template.
3. Proposed Template Learning Algorithm
In this section, we explain the proposed template learningalgorithm based on BP algorithm. Our template learning algorithm is considered based on the dynamic and flexible pointin BP algorithm. In particular, in BP neural networks, parameter is changed by using error between an output signal and adesired signal, in order to bring an output close to a desired
value. From this learning algorithm, the template of CNN ischanged by using the error between an output image and a- 56 -

∂E pdesired image, and we can obtain a desired image. Moreover,= −(t p j − o p j ).
(13)
∂o p jsince the template in this study is always continuing learning,
we can refer that the template is dynamical.
In the previous study, we have proposed some learning In conclude of all, δ p j is expressed as follows,δ p j = (t p j − o p j ) f j0 (x p j ).
(14)
methods [4]. However, these learning methods have imitatedthe process of the original BP algorithm. Because of this,better results have not been obtained in one-dimensional gray In this way, in the neural networks, the error is able to calculate.
scale images.
When we use sigmoidal function as output function in neu3.1. BP Algorithm
ral networks, the rate of change δ p j is expressed as:
In the neural networks, the weight is adjusted according to
the following equation.
∆ p w ji ∝ −

∂E p
.
∂w ji

(3)

(15)

3.2. Approximation of Output Equation
First, we approximate the output equation of CNN. Output
equation of CNN is shown as follows.

Equation (3) is taken apart as follows,∂E p ∂x p j∂E p=,∂w ji ∂x p j ∂w ji∑(x p j =w ji o pi ).

vy p =(4)(5)

i

The second diﬀerential calculus of the right side of Eq. (4) is
calculated as follows,∂x p j∂ ∑=w jk o pk = o pi .
(6)
∂w ji ∂w ji k
We define an error direction δ p j as follows,δp j = −δ p j = β(t p j − o p j )o p j (1 − o p j ).

∂E p.
∂x p j(7)

1(|v x + 1| − |v x p − 1|)
2 p

(16)

Since the output equation of CNN is piece-wise linear function, we cannot diﬀerentiate it. So, we divide output equation
to 3 parts as follows.
Part 1 : When v x ≤ −1, vy = −1
Part 2 : When 1 ≤ v x , vy = 1
Part 3 : When −1 ≤ v x ≤ 1, vy = v x
Especially, we pay attention to [Part 3]. Because, in this part,the values of v x and vy change according to y = x. We consider that this part is more important for learning template
than the other parts.
3.3. Modified BP Algorithm for Template Learning

First, we consider templates A and B as variables as folIn conclusion, we adjust the weight according to the follow- lows,ing equation,∂Ei j∂Ei j ∂v xi j
∆Ai j = −=−.
(17)
∆ p w ji = ηδ p j o pi .
(8)
∂Ai j
∂v xi j ∂Ai j
Next, we explain how to decide the rate of change δ p j .In orderto calculate Eq. (7), we divide as follows,δp j = −∂E p ∂o p j.
∂o p j ∂x p j

The first diﬀerential calculus of Eq. (17) is calculated as follows,
δi j = −

(9)

(18)

The each diﬀerential calculus of Eq. (18) is calculated as follows,

Then, in the neural networks, o p j is expressed as follows,o p j = f j (x p j ),(10)1
( f(x) =),1 + e−βx

(11)

∂Ei j
= −(ti j − oi j ),∂vyi j(19)∂vyi j= 1.
∂v xi j(20)

the second diﬀerential calculus of the right side of Eq. (9) is
calculated as follows,∂o p j= f j0 (x p j ).
∂x p j∂Ei j ∂vyi j
∂Ei j
=−
.
∂v xi j∂vyi j ∂v xi j

(12)

In conclude of all,
δi j = −Next, the first diﬀerential calculus of the left side of Eq. (9) is
calculated as follows,- 57 -∂Ei j= (ti j − oi j ) × 1.
∂v xi j(21)

Next, the second diﬀerential calculus of Eq. (17) is calcuWhen we use a certain two-dimensional image, a12 and b12
lated as follows. To begin with, we consider the state equation influence a pixel from (0, 1) to (255, 256), a21 and b21 influof CNN,
ence a pixel from (1, 0) to (256, 255), a23 and b23 influencea pixel from (1, 2) to (256, 257) and a32 and b32 influence a∂v xi j= −v xi j + Σ(Avyi j ) + Σ(Bvui j ) + I.
(22) pixel from (2, 1) to (257, 256). (Then, 0 and 257 are neigh∂tborhood pixels.) Namely, when the elements of a template
are updated, a12 and b12 are added the average error from (0,We exchange an element of Eq. (22),1) to (255, 256) between the desired image and the output∂v xi jv xi j = −+ Σ(Avyi j ) + Σ(Bvui j ) + I.
(23) image, a21 and b21 are added the average error from (1, 0) to∂t(256, 255), a23 and b23 are added the average error from (1,We diﬀerentiate Eq. (23) with templates A and B,∂v xi j∂2 v xi j=−+ vyi j ,∂Ai j∂t × ∂Ai j∂2 v xand we assume that − ∂t×∂Ai ji j is 0.
In conclude, in the case of template A,∂v xi j= vyi j ,∂Ai jSimilarly for template B,∂v xi j= vui j .
∂Bi j2) to (256, 257), and a32 and b32 are added the average error
from (2, 1) to (257, 256), respectively.
In Eq. (27) and Eq. (28), we show the error in each tem(24)plate. According to these, we calculate the error in each element of template A and B,
0 (i f m = n = 2)


 1
∑

∆amn (t) = ∆Ai+m−2, j+n−2 (t), (29)


 MN

1≤i≤M,1≤ j≤N(25)∑
1∆bmn (t) =∆Bi+m−2, j+n−2 (t), (30)
MN 1≤i≤M,1≤ j≤N(26)

In conclude of all,∂Ei j
= η × (ti j − oi j ) × vyi j ,∂Ai j

amn (t + 1) =amn (t) + ∆amn (t),(31)

bmn (t + 1) =bmn (t) + ∆bmn (t),(32)

where m, n ∈ {1,2,3}.
In addition, in the case of template A, we have to keep thestability condition. Then, we add the average error of two∂Ei j∆Bi j = −(28) elements to symmetric part of elements.
= η × (ti j − oi j ) × vui j .
∂Bi j


 ∆a11 + ∆a33 ∆a12 + ∆a32 ∆a13 + ∆a31 
1 
As a result, we can refer that we can express equation of cor ∆a21 + ∆a23
0
∆a21 + ∆a23 
(33)


2
rection using the error in CNN based on BP.
∆a13 + ∆a31 ∆a12 + ∆a32 ∆a11 + ∆a33
∆Ai j = −(27)

3.4. Updating Elements of Template

3.5. Flow of Learning

In this study, since we use two-dimensional images, theelements of a template are expressed as follows, a11 a12 a13 
A =  a21 1 a23  ,a31 a32 a33
 b11 b12 b13 
B =  b21 b22 b23  .

b31 b32 b33

CNN processes an image using Runge-Kutta method. So,
we define learning as process per “Step size” in Runge-Kutta
method. Learning flow is as follows.

Step 1 : An image is inputted.
Step 2 : An image is processed by CNN in “Step size” offirst time.
Step 3 : After process with “Step size” of first time, being
processed image is compared with the desired image.
Step 4 : Calculating the error.
Template has a roll which influence neighborhood pixel of Step 5 : Updating a template.
a certain pixel. For example, we consider a relation with the Step 6 : Being processed image is processed by CNN using
top, bottom, left and right of a certain element. a12 and b12
updated a template in “Step size” of second time.
influence the top pixel of a certain pixel and a32 and b32 influ- Step 7 : Step 2 - Step 6 are repeated until reaching “Maxience the bottom pixel of a certain pixel. a21 and b21 influence
mum iteration number”.
the left pixel of a certain pixel, a22 and b22 influence a certain pixel itself and a23 and b23 influence the right pixel of a This method is able to update a template more dynamically.
Namely, we can bring dynamically the being processed image
certain pixel.
close to the desired image while processing.

- 58 -

4. Simulation Result of Two-Dimensional Binary Image
In this study, we examine performance of proposed methodwith two-dimensional binary image. In this simulation, we
examine noise reduction in CNN processing. First, we prepare an input image with noise, and the desired image corresponding to the input image. We have CNN learns the template which can process noise reduction.
The following parameters are used in this simulation (seeTable 1).
Table 1: Parameters.
Name of parametervalue
Step size
0.005
Maximum iteration number 10000
Element of initial template −1 ∼ 1
Learning rate η
0.1

(a)

(b)

(c)

(d)

(e)

(f)

Figure 2: The images of simulation results. (a) Input image of
CIRCLE. (b) Desired image of CIRCLE. (c) Output image of
CIRCLE. (d) Input image of TRIANGLE. (e) Desired image“Step size” and “Maximum iteration number” are parameters of TRIANGLE. (f) Output image of TRIANGLE.
of Runge-Kutta method. Learning rate is η of equation (27)
output image using learned template. Figure 2(d) shows theand (28). Finally, the input image and the desired image are
input image of TRIANGLE. Figure 2(e) shows the desired
256 × 256 pixels, and the initial template is determined atimage of TRIANGLE. Figure 2(f) shows the output image
random.
using learned template.
From these results, we can refer that CNN using learned
4.1. Template Learning
template can remove the noise. Because we cannot distinWe prepare the following input image. The image is binary
guish output image using learned template from the desired
image with noise and consisted of simple figures; triangle,
image. When we use the learned template for similar images
circle and square.
to first inputted image, CNN can process images correctly.
5. Conclusions

(a)
(b)
(c)
Figure 1: The images of simulation results. (a) Input image.
(b) Desired image. (c) Output image.

In this study, we modified the BP algorithm for CNN template design. As a result, we obtained learned template fortwo-dimensional image, and showed the performance of the
proposed algorithm. However, in this study, we used simple
binary images and simple tasks. As future works, we apply
the proposed algorithm for more diﬃcult tasks and use grayscale images.
References[1] L.O. Chua and L. Yang, “Cellular Neural Networks: Theory,” IEEE Trans. Circuits Syst., vol. 32, pp. 1257-1272,
Oct. 1988.

Figure 1(a) shows the input image with noise. Figure 1(b)shows the desired image. Figure 1(c) shows the output image using the proposed method. Comparing Fig. 1(c) with [2] M. Nakagawa, T. Inoue and Y. Nishio, “CNN Template
Fig. 1(b), some noises remain. However, we can refer that
Design Using Back Propagation Algorithm,” Proc. ofthe template in this case is learned and we can reduce the
CNNA’10, pp. 47-51, Feb. 2010.
noises. Therefore, we can refer that this method has a good[3] D.E. Rumelhart, G.E. Hinton and R.J. Williams, “Learnperformance for simple binary images.
ing Internal Representations by Error Propagation,” Par4.2. Applying Learned Template to Other Images
allel Distributed Processing, vol. 1, pp. 318-362, 1986.
From this result, we obtain learned template which can donoise reduction. Next, we process other images using the [4] M. Nakagawa, Y. Uwate and Y. Nishio, “Template Design of CNN by BP with Annealing Noise,” IEICE Tech.
learned template in order to confirm whether being able toRep., vol. NLP2010-67, & CAS2010-51, pp. 93-98, Aug.
apply learned template to other images or not.
2010 (in Japanese).
Figure 2(a) shows the input image of CIRCLE. Figure 2(b)shows the desired image of CIRCLE. Figure 2(c) shows the- 59 -