2008 International Workshop on Nonlinear Circuits and Signal Processing
NCSP'08, Gold Coast, Australia, March 6-8, 2008

Neural Gas Containing Two Kinds of Neurons and its Behaviors
Keiko Kanda† , Haruna Matsushita† and Yoshifumi Nishio††Tokushima University, JAPAN
Email: {kanda, haruna, nishio}@ee.tokushima-u.ac.jp

Abstract - In this study, we propose two new Neural Gas algorithms which contains two kinds of neurons (called TN-NG) andcontains two kinds of update functions (called TF-NG). In TN-NG,
all neurons learn according to own character at each learning. On theother hand, in TF-NG, all neurons learn according to winner’s character at each learning. We confirm that two proposed methods canobtain more eﬀective learning results than the conventional NeuralGas.

1. Introduction
In the real world, living things with diﬀerent various features are coexistent. For example, an elephant, which has aslow heart rate, and a mouse, which has a rapid heart rate,
have diﬀerent senses of time. However, in the learning ofNeural Gas [1], the network contains only one type of neurons.
In this study, we propose two new Neural Gas algorithms.
One method is a Neural Gas Containing Two Kinds of Neurons (called TN-NG). The other method is a Neural Gas Containing Two Kinds of Update Functions (called TF-NG). BothTN-NG and TF-NG have two kinds of neurons. In TN-NG,
all neurons learn according to own character at each updating.
On the other hand, in TF-NG, all neurons learn according towinner’s character at each updating.
The learning behaviors of TN-NG and TF-NG are investigated with the computer simulation. In addition, TN-NG isapplied to noise reduction and TF-NG is applied to feature
extraction, by using each feature of the learning results. We
confirm that TN-NG and TF-NG can obtain the more eﬀective learning results than the conventional Neural Gas.
2. Neural Gas

same time in parallel.
(NG2) Euclid distances d ji between the input vector x j andall the weight vectors wi are calculated;d ji = kx j − wi k.

(1)(NG3) Each neuron ci is assigned a rank k = 0, · · · , (M − 1)
depending on their d ji . ki denotes the rank of neuron ci . The
rank of 0 indicates the neuron closest to x j and the rank of(M − 1) indicates the neuron farthest from x j .
(NG4) The weight vectors of the neurons are updated according towi (t + 1) = wi (t) + E(t)hλ (t, k)(x j − wi (t)),(2)

where E(t) is the learning rate;()t/TE(t) = E0 E0 /E f,(3)

where t is the time step and T is the total number of training
steps. E0 and E f are the initial and the final values of E(t),respectively. hλ (t, k) is the neighborhood ranking function described ashλ (t, k) = exp (−ki /λ(t)) ,(4)where the parameter λ(t) controls the convergence rate of theneighborhood ranking function;()t/Tλ(t) = λ0 λ0 /λ f,(5)where λ0 and λ f are the initial and the final values of λ(t),respectively.
(NG5) If t < T , return to (NG1).
3. Proposed Methods

We explain the learning algorithm of the conventional
Neural Gas. The set A containing M neurons; A ={c1 , c2 , · · · , c M }. Each neuron ci (i = 1, 2, · · · , M) has a ddimensional weight vector wi = (wi1 , wi2 , · · · , wid ). The initial values of the weight vectors are given at random. Therange of the elements of d-dimensional input data x j =(x j1 , x j2 , · · · , x jd ) ( j = 1, 2, · · · , N) are assumed to be from0 to 1.
(NG1) An input vector x j is inputted to all the neurons at the3.1. NG Containing Two Kinds of Neurons (TN-NG)
We explain the algorithm of the proposed NG which has
two kinds of neurons whose features are diﬀerent (called TNNG). The network consists of a set A containing M neurons;
A = {c1 , c2 , · · · , c M }. The all neurons are classified into twokinds of sets; A1 and A2 at random. A1 contains m1 neurons,and A2 contains m2 neurons, namely M = m1 + m2 . Theneurons belonging to A1 converges rapidly and the neurons- 263 -

belonging to A2 converges slowly. To be specific, the convergence rate of the neighborhood ranking function is diﬀerentbetween A1 and A2 . The initial value of the weight vectors
are given at the same method of the conventional NG.
(TN-NG1) An input vector x j is inputted to all the neurons at
the same time in parallel.
(TN-NG2) Euclid distances d ji between an input vector x j
and all the weight vectors wi are calculated according toEq. (1).
(TN-NG3) Each neuron ci is assigned a rank k = 0, · · · , (M −1) depending on their d ji .
(TN-NG4) The weight vectors of the neurons are updated according towi (t + 1) = wi (t) + E(t)hλN (t, k)(x j − wi (t)).

(6)

hλN (t, k) is the neighborhood ranking function of TN-NG described as;hλN (t, k) = exp (−ki /λN (t)) ,(7)where λN (t) is decided by neuron’s character;( ) Ttλfλ0,ci ∈ A1 ,

λ0



() Tt

λN (t) = p(p+λ−λ)0f
p + λ0 − λ f +λ − p,ci ∈ A2 ,f(8)(TF-NG3) Each neuron ci is assigned a rank k = 0, · · · , (M −1) depending on their d ji . The closest neuron to x j , namelyki = 0, is defined as a winner.
(TF-NG4) The weight vectors of the neurons are updated according towi (t + 1) = wi (t) + E F (t)hλ (t)(x j − wi (t)),(9)

where E F (t) is decided by winner’s character;( ) TtEfE,winner ∈ A1 ,0
E0


() Tt

E F (t) = q(q + E0 − E f )
q + E0 − E f
 + E − q,winner ∈ A2 ,f(10)where the fixed parameter q controls the convergence rate of
the learning rate E F (t) and q ≥ E f . In this way, all neurons
learn according to winner’s character. As a result of Eq. (10),the learning rate E F (t) of A1 converges rapidly, and E F (t) ofA2 converges slowly. In other words, if winner belongs to A1 ,all neurons learn based on the character of A1 , otherwise, allneurons learn based on the character of A2 .
(TF-NG5) If t < T , return to (TF-NG1).
4. Simulation Results

where the fixed parameter p controls the convergence rate ofthe λN (t). As a result of Eq. (8), λN of A1 decreases rapidly,and λN of A2 decreases slowly. In other words, the neurons of
A1 converge rapidly, and the neurons of A2 converge slowly.
(TN-NG5) If t < T , return to (TN-NG1).
3.2. NG Containing Two Kinds of Update Functions (TFNG)
We explain the learning algorithm of the proposed NGwhich has two kinds of update functions whose features arediﬀerent (called TF-NG). The network consists of a set A containing M neurons; A = {c1 , c2 , · · · , c M }. The all neurons areclassified into two kinds of sets; A1 and A2 at random. A1contains m1 neurons, and A2 contains m2 neurons, namely
M = m1 +m2 . The neurons belonging to A1 converges rapidly
and the neurons belonging to A2 converges slowly. To be specific, the convergence rate of the learning rate is diﬀerent between A1 and A2 . The initial value of the weight vectors are
given at the same method of the conventional NG.
(TF-NG1) An input vector x j is inputted to all the neurons at
the same time in parallel.
(TF-NG2) Euclid distances d ji between an input vector x j
and all the weight vectors wi are calculated according toEq. (1).

We carry out learning simulation for the 2-dimensional input data, shown in Fig. 1(a). The total number of the input
data N is 1000, and the input data has a square cluster. Allthe input data are sorted at random.
The leaning conditions are as follows. The conventional
NG, TN-NG and TF-NG have 200 neurons. In TN-NG andTF-NG, the numbers of neurons included in A1 and A2 areequal, namely m1 = m2 = 100. We repeat the learning 5times for all input data, namely T = 5000. The parameters ofthe learning are chosen as follows;(NG)
E0 = 0.7, E f = 0.01, λ0 = 70, λ f = 0.2(TN-NG) E0 = 0.7, E f = 0.01, λ0 = 70, λ f = 0.2, p = 50
(TF-NG) E0 = 0.7, E f = 0.01, λ0 = 70, λ f = 0.2, q = 50
where we use the same E0 , E f , λ0 , and λ f to the conventional
NG, TN-NG and TF-NG for the comparison.
The learning results of the conventional NG, TN-NG andTF-NG are shown in Figs. 1(b), (c) and (d), respectively. InFig. 1(c) and (d), we can see that TN-NG and TF-NG has two
kinds of neurons. In Fig. 1(c), we can see that the input dataon the interior of the cluster are learned by the neurons of A2 ,and the exteriors are learned by the neurons of A1 . In otherwords, in TN-NG, the neurons of A1 and A2 do not mingle andrespective neurons have own area by each group. This is because the convergence rate to the input data is diﬀerent in two- 264 -

1

1

1

1

0.9

0.9

0.9

0.9

0.8

0.8

0.8

0.8

0.7

0.7

0.7

0.7

0.6

0.6

0.6

0.6

0.5

0.5

0.5

0.5

0.4

0.4

0.4

0.4

0.3

0.3

0.3

0.3

0.2

0.2

0.2

0.2

0.1

0.1

0.1

0.1

00

0
0

0
0

0.2

0.4

0.6

0.8

10.2

(a)

0.4

0.6

0.8

1

0.2

(b)

0.4

0.6

0.8

100

0.2

0.4

(c)

0.6

0.8

1(d)

Figure 1: Computer simulation for 2-dimensional data. (a) Input data. (b) Learning result of conventional NG．(c) Learning
result of TN-NG. ● and ★ denote neurons belonging to A1 and A2 , respectively. (d) Learning result of TF-NG. ● and ★
denote neurons belonging to A1 and A2 , respectively．
111

1

0.9

0.9

0.9

0.9

0.8

0.8

0.8

0.8

0.7

0.7

0.7

0.7

0.6

0.6

0.6

0.6

0.5

0.5

0.5

0.5

0.4

0.4

0.40.4

0.3

0.3

0.3

0.3

0.2

0.2

0.2

0.2

0.1

0.1

0.1

0.1

000000

0.2

0.4

(a)

0.6

0.8

1

0.2

0.4

0.6

0.8

1(b)

0.2

0.4

(c)

0.6

0.8

1

00

0.2

0.4

0.6

0.8

1

(d)

Figure 2: Simulation for data including noises．(a) Input data． (b) Learning result by conventional NG．(c) Learning result
by TN-NG．(d) Extracted cluster by TN-NG.

kinds of neurons. Meanwhile, in Fig. 1(b) and (d), it is difficult to see diﬀerences of the simulation results between theconventional NG and TF-NG. Therefore, we use a following
measurement to evaluate the learning performances of threealgorithms.
Quantization Error Qe: This measures the average distance
between each input vector and its winner [3]. Therefore, thesmall value Qe is more desirable.
In Table 1, the quantization error Qe of TF-NG is the smallest in three algorithms and has improved 5.47% from the conventional NG. However, Qe of TN-NG is larger than the conventional NG and has not improved.
Table 1: Quantization error Qe for 2-dimensional data
Conventional NG TN-NG TF-NG
Qe0.0146
0.0155 0.0138

5. Application of TN-NG to Noise Reduction

In the simulation for Fig. 1, the quantization error Qe has
not improved from the conventional NG. However, because
we can confirm that the each neuron group of A1 and A2 has
own area, we apply TN-NG to noise reduction.
We consider 2-dimensional input data as shown inFig. 2(a). The input data has 1 cluster and noises. There are
700 points in the cluster and 300 points for noises. All the input data are sorted at random. We repeat the learning 5 times
for all input data, namely T = 5000. The leaning conditions
are the same used in Fig. 1.
The simulation results of the conventional NG and TN-NG
are shown in Figs. 2(b) and (c), respectively. We can see thatsome neurons of the conventional NG learn the input dataof not only the cluster but also the noises. The other side,in the result of the TN-NG, the A2 neurons learn the cluster
and the A1 neurons learn the noises. In other words, TNNG learns by using diﬀerent kind of neurons in the cluster
and noises while the conventional NG uses only one kindsof neurons. Furthermore, we classify each input data into
the group corresponding to its winner’s character. Fig. 2(d)shows the extracted input data corresponding to the group of- 265 -

1

1

1

0.9

0.9

0.9

0.8

0.8

0.8

0.7

0.7

0.7

0.6

0.6

0.6

0.5

0.5

0.5

0.4

0.4

0.4

0.3

0.3

0.3

0.2

0.2

0.2

0.1

0.1

0.1

0

0
0

0.2

0.4

0.6

0.8

1

00(a)

0.2

0.4

0.6

0.8

1

0

0.2

0.4

(b)

0.6

0.8

1(c)

Figure 3: Simulation for Target data．(a) Input data． (b) Learning result by conventional NG．(c) Learning result by TF-NG．

0.022

A2 . As a result, we can see that noises are removed from theinput data and only cluster is extracted. Therefore, we can saythat TN-NG can be used for noise reduction and it is eﬀective.

Conventional NG
TF−NG

0.021
0.02
0.019

6. Application of TF-NG to Feature Extraction

0.018

Because TF-NG has more eﬀective learning result withsmaller the quantization error Qe than the conventional NG
in the simulation for Fig. 1, we apply TF-NG to feature extraction.
We consider the 2-dimensional input data; Target data
set [2] shown in Fig. 3(a). The total number of the input data
N is 770, and the input data has six clusters which include 4outliers. All the input data are sorted at random. We repeat
the learning 5 times for all input data, namely T = 3850. The
leaning conditions are the same used in Fig. 1.
The learning results of the conventional NG and TF-NG
are shown in Figs. 3(b) and (c), respectively. We can see thatsome neurons of TF-NG are attracted to the corner input data
more strongly than the conventional NG. Thus, TF-NG canobtain more eﬀective result reflecting the features of the input
data than the conventional NG.
The calculated quantization error Qe are summarised in Table 2. Qe of TF-NG is smaller than the conventional NG and
has improved 25.7% from the conventional NG.
Table 2: Quantization error Qe for Target data
Conventional NG TF-NG
Qe
0.01710.0127

Qe 0.017
0.016
0.015
0.014
0.013
0.012
1

1.5

2

2.5

3

3.5

4

4.5

5Tmax

Figure 4: Quantization error Qe versus maximum repeatingtimes T max .

7. Conclusions
In this study, we have proposed two new Neural Gas algorithm; Neural Gas containing two kinds of update Neurons(TN-NG) and Neural Gas Containing Two Kinds of Update
Functions (TF-NG). From each learning behavior, we haveconfirmed that TN-NG is eﬀective for application to noise
reduction and TF-NG is eﬀective for application to feature
extraction.
Acknowledgment
This work was partly supported by Yamaha Music Foundation.

Fig. 4 shows the quantization error Qe of the learning result References
when changing the maximum number of the repeating times [1] T. M. Martinetz and K. J. Schulten, “A “neural-gas” network
T max of all the input data (T = N × T max ). In this figure, alllearns topologies. In T. Kohonen, K. Mäkisara, O. Simula, and
J. Kangas, editors,” Artificial Neural Networks, pp. 397–402,
Qe of TF-NG are smaller than conventional NG. This implies
1991.
that TF-NG can converge much faster than the conventional[2] A. Ultsch, “Clustering with SOM: U*C”, Proc. Workshop on
NG. From these results, we can say that TF-NG can obtain
Self-Organizing Maps, pp. 75–82, 2005.
more eﬀective result than the conventional NG and can be [3] T. Kohonen, Self-organizing Maps, 2nd ed., Berlin, Springer,
1995.
used for feature extraction.

- 266 -