2010 12th International Workshop on Cellular Nanoscale
Networks and their Applications (CNNA)

CNN Template Design
Using Back Propagation Algorithm
Masashi Nakagawa, Takashi Inoue and Yoshifumi Nishio
Department of Electrical and Electronic Engineering, Tokushima University
2-1 Minami-Josanjima, Tokushima, 770-8506, Japan
Email: {miyabi, takashi, nishio}@ee.tokushima-u.ac.jp

Abstract— In this article, detailed investigation of the template
design method of cellular neural networks with back propagation
algorithm, which was proposed by the authors before, is carriedout. The performance of the method is also evaluated by using
the average error which corresponds to the difference between
the output produced by the designed template and the ideal
output. Furthermore, the method is applied for gray scale image
applications.

I. I NTRODUCTION
Cellular neural networks (CNN) were introduced by Chua
and Yang in 1998 [1]. The idea of the CNN is inspired
from the architecture of the cellular automata and the neuralnetworks. Unlike the conventional neural networks, the CNN
has local connectivity property. Since the structure of the CNN
resembles the structure of animals retina, the CNN can be usedfor various image processing applications [2]-[4].
Wiring weights of the cells in CNN are established byparameters called template. The template is most importantparameter in the CNN, because performance of the CNN
is decided by the template. Thus, some template learningschedule as template learning method using GA algorithm areproposed [5]. These works are important subject in the studiesof the CNN.
In the previous study, we have proposed a template learning
method of cellular neural networks using the back propagation(BP) algorithm [6]. As well known, BP is one of the supervised
learning methods [7]. In that method, the template of CNN isdynamically updated from the error between an output image
and an ideal image as BP neural networks. We have confirmedthat the method could work for a simple example using binary
image by computer simulation. However, in the previous study,
since we carried out the simulation only for a set of one input
image and its corresponding teacher image, more detailed
investigation of the method should be done in order to design
better templates for a given tasks.
In this study, we carry out more detailed investigation of
our template design method with various learning rates anddifferent learning schedules. We also evaluate the performanceby using the average error which corresponds to the difference
between the output produced by the designed template and theideal output. Furthermore, we try to use our method for grayscale image applications.
In the Sec. 2, we review the basic of the standard CNN. In
the Sec. 3, we review the basic of BP neural networks. In the

978-1-4244-6678-8/10/$26.00 ©2010 IEEE

Sec. 4, we show the proposed template learning method using
BP. In the Sec. 5, we show the simulation results. Section 6
concludes the article.
II. C ELLULAR N EURAL N ETWORKSIn this section, we explain the basic structure of the CNN.
The CNN has M by N processing unit circuits called cells.
The cells are arranged in a reticular pattern to M line N row.
We represent a cell C(i, j) using a variable i which denotes
vertical position and a variable j which denotes horizontal
position. The cell contains linear and nonlinear circuit elements. The CNN is an array of cells. Each cell is connected
to its neighboring cells according to a template. Usually, thetemplate is the same for all cells except for boundary cells. The
CNN has the features of time continuity, spatial discreteness,
nonlinearity and parallel processing capability.
The state equation and the output equation of the cell are
shown as follows.
State equation:
dvxij (t)dt=−vxij (t) +

i+r


j+r


A(i,j;k,l) vykl (t)k=i−r k=j−r
i+r


+

j+r


B(i,j;k,l) vukl (t) + I,(1)

k=i−r k=j−r

Output equation:
1(|vxij + 1| − |vxij − 1|),(2)
2where vx , vy and vu represent a state, an output and an inputof cell, respectively. In the equation (1), A is the feedback
template and B is the control template. These and bias I arecollectively called general template.
vyij (t)=III. BACK P ROPAGATION A LGORITHM
BP is the most common learning algorithm for feed forward
neural networks. BP uses the error of an output signal and ateacher signal for learning. The total error E of the network
is defined as the following equation.
 N

P
P


1
E=
(3)
Ep =
(tpi − opi )2 ,
2
p=1
p=1
i=1
where P is the number of the input data, N is the number ofthe neurons in the output layer, tpi denotes the value of thedesired target data for the pth input data, and opi denotes thevalue of the output data for the pth input data. The goal of thelearning is to set weights between all layers of the network
to minimize the total error E. In order to minimize E, theweights are adjusted according to the following equation.
k−1,k(m + 1) =
wi,j

k−1,k
wi,j
(m) +

P


k−1,k
δp wi,j
(m),

p=1
k−1,k
δp wi,j
(m)

=

−η

∂Ep
k−1,k
∂wi,j

,

(4)

k−1,k
where wi,jis the weight between the ith neuron of the layerk −1 and the jth neuron of the layer k, m is the learning time,and η is a proportionality factor known as the learning rate.
Thus, BP brings an output close to the ideal output bychanging weight.

IV. P ROPOSED T EMPLATE L EARNING M ETHOD
In this section, we describe the proposed template learning
method using BP algorithm. Our template learning method is
inspired from the flexibility of BP algorithm. In particular, in
the BP neural networks, parameter is changed by using error
between an output and an ideal value, in order to bring an
output close to an ideal value. From this perception of learning,
template of CNN is changed by using error between an output
image and an ideal image. Additionally, this template learning
carry out dynamically.
First, we describe the structure of template learning method
using BP algorithm. Second, we present four patterns template
learning schedules for investigation of an output characteristics.
A. Procedure of Template Learning Method
Processing procedure for learning is explained as follows.
Figure. 1 show the proposed template learning method using
BP algorithm.
Step 1: An input image and an ideal image (a teacher image)are prepared. An initial template is decided at random, in order to be able to learn the template for any
initial values. However, in the template learning, weshould mention the stability of template in the CNN.
Because the output value of each cell in the CNN
may become unstable with the value of a template.
The condition of stability is that the center elementof template A maintains 1 and the symmetry of eachelement.
Step 2: CNN using Runge-Kutta method process an input
image. At this time, unit width of Runge-Kutta
method is made small, and only a few processes. It is
for using the image currently processed not using theprocessed image in order to make learning dynamic.
Step 3: The error between an output image and an ideal
image is calculated for each pixel. An error Ep isFig. 1.

Proposed template learning method.

calculated as the following equation:
Ep = (tp − op ),(5)

where p is the number of the pixel in input image,tp and op are represent the value of the ideal image
and output value of the pth pixel respectively. This
step is shown in “Calculate error” of Fig. 1.
Step 4: The corrected direction δp is decided using error Ep
as the following equation:
δp = Ep × (V xnow
− V xpast
),
p
p

(6)

and V xnow
are represent past value
where V xpastppof V xp and present value of V xp respectively.
Step 5: The correction quantity Δp for each pixel is calculated as the following equation:Δp = η × δp × V ypnow ,(7)

TABLE I

where η is a proportionality factor known as thelearning rate. This step is shown in “Calculation thecorrection quantity” of Fig. 1.
Step 6: The errors of all the pixels are added, hence theaverage of the all errors D is calculated as thefollowing equation.
1
D=
Δi ,
p

L EARNING SCHEDULE

Pattern 1
Pattern 2
Pattern 3
Pattern 4

Number oflearnings L
10
25
50
100

Set ofimages20
20
20
20

Number ofgroup repetitions λ
10
4
2
1

The total oflearning times
2000
2000
2000
2000

p

(8)

i=1

In this article, each element of template are decided
as equation (9), because of an input image is only
one-dimensional image.
⎤
⎡
0
0
0
A = ⎣ a21 a22 a23 ⎦ ,
0
0
0
⎤
⎡
0
0
0
B = ⎣ b21 b22 b23 ⎦ ,
0
0
0
I = i
(9)

“number of learnings” L shows the number of times oflearning of the BP algorithm, and λ shows the number oftimes of a repetition group.
The each block of Fig. 2 is explained as follows.

The update equation of template is shown as follow.
aRenewed21ore= aBef+ D,
21

aRenewed23ore= aBef+ D,
23

bRenewed21ore= bBef+ D,
21bRenewed22ore= bBef+ D,
22bRenewed23ore= bBef
+ D.
23

(10)

The average D adds one row of the middle elementsin the template A and B. However, it does notinclude the center element of the template A, andthe bias I. It is because that the center element oftemplate A is fixed with 1 in order to protect thestability condition of a template. Namely, the reasonof the above update method is that the stability ofthe template can be kept. This step is shown in “Addto the value of template” of Fig. 1.
Step 7: Step. 2 - Step. 6 are repeated until it reaches the
set-up number of learning times L.
By using this learning algorithm, template of each cell is
updated for bringing close to an ideal image.
B. Four patterns of learning schedule
In this study, we prepare the four patterns of leaningschedule for investigation of output characteristics. Figure 2
show the flow-chart of template learning method. In Fig. 2, we
focus on the two parameters as follow. The number of learningin the BP algorithm and number of repetitions of group aredescribed L and λ, respectively.
The parameter of four patterns of learning schedule areshown in TABLE I.

Fig. 2.

Procedure for learning.

Step 1. P reparation of input and ideal image
First, 20 sets of input and ideal images are prepared.
Step 2. Decision the learning rate and intial template
Second, the learning rate is chosen 0.01, 0.05 and0.1. Each values of an initial template are decided at
the random as Step 1 of previous section.

Step 3. CN N
Third, one of the 20 sets of an input and an ideal
image is chosen. CNN processes the input image byusing a learning rate and an initial template. An input
image is processed by using the learning rate decided
at Step 2.
Step 4. BP
Forth, an initial template is updated by using BP. In
the four patterns learning, the number of learnings L
is chosen 10, 25, 50 or 100.

Rule 1. If the colors of the neighbors of a certain cell is thesame color, the cell’s color turns into its neighboringcolor.
Rule 2. If the colors of the neighbors of a certain cell isdifferent, the cell’s color does not change.
That is to say, this rule is like “Reversi”.
First, ten kinds of different images as an input image areprepared. These pixels are also arranged at random. These
images are processed as an input image using the template of
twelve patterns. The average error when processing ten images
in each template is shown in TABLE III.
TABLE III

Step 5. Changing the input and ideal images
Sixth, after template learning using BP algorithm,
input and ideal images are changed to another set ofimages.
Step 6. Repetition
Step 3 - Step 5 are repeated until 20th set of images
finished. In this time, updated template using BP iscontinued to the next processing using next input andideal image.
Step 7. Group Repetition
Step 3 - Step 6 are repeated until the number of set
comes λ. where, λ is the whole number of times ofa repetition.
Step 8. Obtained learned template
After all learning is finished, we can obtain the final
learned template.
Although the schedule of learning is different, they can bereferred to have the same total learning times of 2000.
V. S IMULATION RESULTS
A. Parameters for Learning
The following parameters are used in this simulation.
TABLE IIPARAMETERS
Name of parameter
Step size
Maximum iteration number
Number of pixels
Learning rate

value
0.01
1000
40
0.01
0.05
0.1

“Step size” and “Maximum iteration number” are parameterof Runge-Kutta method. Learning rate is η of equation (7)B. For binary image
First, we show the four patterns of template learning resultsfor one-dimensional binary image. The number of pixels is 40
and they are placed at random. The black pixels correspond tothe value +1, while the white pixels correspond to the value-1. An ideal image is made from the following two rules.

AVERAGE ERROR

Learning rate

0.01 [%]
0.05 [%]
0.1 [%]
0.25 [%]
0.5 [%]
0.75 [%]
1 [%]Number of learnings L
10
25
50
100
15.75
30.75
35.75
25
21.5
21.75
11.75
23
0
18.5
14.75
27
8.75
18.5
11.75
21.75
4
12.75
9.5
21.75
4
8.75
13.5
12.75
3.25
3.75
5.5
14

An error is so low that a learning rate is high. Because, if alearning rate becomes large, the updating width of a template
becomes large and more dynamic learning can be carried out.
Moreover, an error is so low that the number of times oflearning is lessened and a group is learned repeatedly. If there
is much number of times of learning, it will be completed
by updating and will not become dynamic in many cases.
Then, there is little number of times which learns a group,and dynamic updating is not carried out. That is, if there ismuch number of times of learning, that dynamic renewal ofa template is carried out will decrease. From this result, alearning rate is large and there should be just little number oftimes of learning moderately.
C. For gray-scale image
Second, we show learning result for one-dimensional grayscale image. The number of pixels is 40 and they are put thevalue of pixels at random. The value of range is from +1 to-1. We consider the task of gradient detection. An ideal imageof this task is made from the following three rules.
Rule 1. The absolute value between a certain pixel and its
right-hand pixel is calculated.
Rule 2. If the absolute value is one or larger, the pixelchanges to black.
Rule 3. If the absolute value is less than one, the pixelchanges to white.
We prepare 20 one-dimensional input images and the corresponding ideal images and one example is shown in Figs. 3
and 4.

Input image:[6] M. Nakagawa, T. Inoue and Y. Nishio, “Template Learning of CNN Using
Back Propagation Algorithm,” (being submitted to a conference), 2009.
[7] D.E. Rumelhart, G.E. Hinton and R.J. Williams, “Learning Internal
Representations by Error Propagation,” Parallel Distributed Processing,vol. 1, pp. 318-362, 1986.
Fig. 3.

Input image.

Fig. 4.

Ideal image.

Ideal image:Template learning is carried out to gray scale on these
condition. However, only the following a result have comeout.
TABLE IVPARAMETER AND R ESULT
Learning rate
The number of times of BP
The whole number of a repetition times
Error Rate [%]

0.01
50
2
40.75

Only the result in which an error is very high is obtained.
The error of each pixel of an input image and a teacher imageis large, and it seems by the present proposed method thatlearning was not carried out. It is necessary to change theproposal technique itself into the technique of the ability to respond also to a big error. Since most image processing cannot
be carried out if a gray scale image cannot be processed, theproposed technique should be improved for gray scale images.
VI. C ONCLUSIONSIn this article, we have investigated the template design
method of cellular neural networks with back propagation
algorithm in more detail. The performance of the method wasevaluated by using the average error. Furthermore, the method
was applied for a gray scale image application.
At the moment, we do not say that the proposed template
learning method exhibited a superior performance than theexisting other template learning methods. However, we feelthat we obtained some results to broaden the research on thetemplate learning of CNN.
Improving the method for gray scale images or/and 2dimensional images is our important future works.
R EFERENCES[1] L.O. Chua and L. Yang, “Cellular Neural Networks: Theory,” IEEE Trans.
Circuits Syst., vol. 32, pp. 1257-1272, Oct. 1988.
[2] F. Dirk and T. Ronald, “Coding of Binary Image Data using Cellular
Neural Networks and Iterative Annealing,” Prof. of ECCLES’03, vol. 1,
pp. 229-232, Sep. 2003.
[3] M. Samba and Z. Hang, “Cellular Neural Network for Associative
Memory and Its Application to Braille Image Recognition,” Prof. of
IJCNN’06, pp. 4716-4721, Jul. 2006.
[4] H. Koeppl and L.O. Chua, “An Adaptive Cellular Nonlinear Network andits Application,” Proc. of NOLTA’07, pp. 15-18, Sep. 2007.
[5] T. Kozek, T. Roska and L.O. Chua, “Genetic Algorithm for CNN
Template Learning,” IEEE Trans. Circuits Syst., vol. 40, pp. 392-401,
Jun. 1993.