# Multifocus Color Image Fusion Based on NSST and PCNN

Authors: Xin Jin; Rencan Nie; Dongming Zhou; Quan Wang; Kangjian He
Journal: Journal of Sensors
 (2016)

---

## Abstract

This paper proposed an effective multifocus color image fusion algorithm based on nonsubsampled shearlet transform (NSST) and pulse coupled neural networks (PCNN); the algorithm can be used in different color spaces. In this paper, we take HSV color space as an example, H component is clustered by adaptive simplified PCNN (S-PCNN), and then the H component is fused according to oscillation frequency graph (OFG) of S-PCNN; at the same time, S and V components are decomposed by NSST, and different fusion rules are utilized to fuse the obtained results. Finally, inverse HSV transform is performed to get the RGB color image. The experimental results indicate that the proposed color image fusion algorithm is more efficient than other common color image fusion algorithms.


---

## Body

## 1. Introduction

Video technology is one of the important technologies for coastal monitoring, and image fusion is the basis of video technology. Color images contain color information and brightness information, so the color images are more suitable for coastal monitoring than gray images [1]. Besides, the identifiable degree of human vision to color information is higher than the gray image [2]. The whole procedure of the image fusion is to extract the significant and representative information from the source images of the same scene, which may come from different types of image sensors or the same one acting in different modes, and then attempt to fuse it into the final composite image with a better description of the scene than any of the individual source images. Thus, the study of suitable fusion technology in multisensor image is necessary and valuable [3].Color image is the combination of different brightness and colors. Because color image is comprised of several components and the fusion image is the fusion of each color space component, there are some common algorithms, such as average, Intensity, Hue and Saturation (HIS), and principal component analysis (PCA) [4, 5], which are easy to implement but the performances are not good. Recently, image fusion methods based on multiresolution analysis have been widely studied; the first step is image transform and then recombining the coefficients of the transformed image; at last the fused image can be obtained by inverse transform. According to the different ways of decomposition, these algorithms can be divided into pyramid transform, wavelet transform [6], curvelet [7], and contourlet [8]. In 2005, Labate et al. proposed a new multidimensional representation algorithm, which is called shearlet [9]. One advantage of this algorithm is that it can be constructed using generalized multiresolution analysis and efficiently implemented using a classical cascade algorithm. So shearlet has good performance in both time domain and frequency domain [10]. In order to combine the superiorities and overcome the defects of nonsubsampled contourlet transform (NSCT) and shearlet transform (ST), [11] proposed the theory of nonsubsampled shearlet transform (NSST) combining the nonsubsampled Laplacian pyramid transform with several different shearing filters. In comparison with current multiresolution geometric analysis (MGA) tools, NSST absorbs some recent developments in the MGA field and shows satisfactory fusion performance such as the better sparse representation ability and much lower computational costs. Besides, NSST also has the requirement of the shift-invariance property ST lacks. Therefore, it is hoped that further research on the area of image fusion based on NSST domain is promising and competitive [12]. In recent years, the image fusion method based on PCNN is getting more and more attention by many experts and scholars with PCNN’s characters in biological background. Compared with other artificial neural networks, PCNN has an incomparable advantage over other traditional artificial neural networks. So PCNN has been widely used in image processing fields and shows extremely superior performances [12–15].In this paper, a new multifocus color image fusion algorithm is proposed based on NSST and PCNN. The paper absorbs some advantages of NSST and PCNN; it firstly converts RGB color image to HSV color image, and then H component is input into adaptive simplified PCNN (S-PCNN) model to get oscillation frequency graph (OFG) of S-PCNN; a new fused H component is obtained by comparing the OFG; S and V components are decomposed into low frequency subband and high frequency subband by NSST, and these subbands are fused by different methods to get new fused S and V components. At last, inverse HSV transform is performed to obtain a new fused RGB color image. The experimental results indicate that the proposed algorithm is more effective to save the color information of the source color images than other common algorithms; and the fused image contains more edges, texture, and detail.This paper is arranged as follows. Section2 introduces related theories of NSST and PCNN model. Section 3 explains the proposed algorithm, including framework and workflow. Section 4 presents the experimental results and analysis. Section 5 concludes this paper.


## 2. Related Theories

### 2.1. PCNN

PCNN model has three fundamental parts: the receptive field, the modulation field, and the pulse generator [13, 14]. In the receptive field, which consists of L and F channels and is described by (1), the neuron receives neighboring neurons’ coupling input Y and external stimulus input S. In L and F channels of the neuron, the neuron links with its neighborhood neurons via the synaptic linking weights W and M, respectively; the two channels accumulate input and exponential decay changes at the same time; the decay exponentials are α
L and α
F, respectively, while the channel amplitudes are V
L and V
F, respectively:(1)
F
i
j
n
=
V
F
∑
k
l
M
i
j
k
l
Y
k
l
n
-
1
+
e
-
α
F
F
i
j
n
-
1
+
S
i
j
,
L
i
j
n
=
V
L
∑
k
l
W
i
j
k
l
Y
k
l
n
-
1
+
L
i
j
n
-
1
e
-
α
L
.In the modulation field, the linking inputL
i
j
(
n
) made by adding a bias to the linking; then, it is multiplied by the feeding input; the bias is unitary, β is the linking strength, and the total internal activity U is the result of modulation, which is described by(2)
U
i
j
n
=
F
i
j
n
1
+
β
L
i
j
n
.Pulse generator consists of a threshold adjuster, a comparison organ, and a pulse generator, which is described by (3). Its function is to generate the pulse output Y, and θ
i
j is adjustment threshold; V
i
j
θ is threshold coefficient. When the internal state U is larger than the threshold θ, that is, the neuron satisfies the condition U
i
j
n
>
θ
i
j
n, a pulse would be produced by the neuron; we call an ignition, which is described by (4):(3)
θ
i
j
n
=
e
-
α
θ
θ
i
j
n
-
1
+
V
i
j
θ
Y
i
j
n
-
1
,
(4)
Y
i
j
n
=
1
,
U
i
j
n
>
θ
i
j
n
0
,
otherwise
,where the subscripts i and j represent the neuron location in PCNN and n denotes the current iteration (discrete time step), where n varies from 1 to N (N is the total number of iterations). In particular, “a neuron ignition” means a PCNN’s neuron generates a pulse. The total times of ignitions represent image information of the corresponding code sequences after N iterations.When PCNN is used for image processing, a pixel is connected to unique neuron. The number of neurons in the network is equal to the pixel number of the input image; namely, there exists one-to-one correspondence between the image and neurons network, and the pixel value is taken as the external input stimulus of the neuron inF channel. A neuron outputs results in two states, namely, pulse (status 1) and nonpulse (status 0), so the output status of neurons composes a binary image. More information about PCNN will be found in [12–15].


### 2.2. S-PCNN

Simplified PCNN (S-PCNN) model [15] is composed the same as the original PCNN model, but the input of F channel is only related to image gray value and has no relationship with external coupling and exponential decay characteristics, and its parameters are less than original PCNN model, and the input channel of the receptive field is simple and effective. In S-PCNN model, the variables of a neuron N
i
j satisfy the following:(5)
F
i
j
n
=
S
i
j
,
L
i
j
n
=
V
i
j
L
∑
k
l
W
i
j
k
l
Y
k
l
n
-
1
+
L
i
j
n
-
1
e
-
α
L
,
U
i
j
n
=
F
i
j
n
1
+
β
L
i
j
n
,
θ
i
j
n
=
e
-
α
θ
θ
i
j
n
-
1
+
V
i
j
θ
Y
i
j
n
-
1
,
Y
i
j
n
=
1
,
U
i
j
n
>
θ
i
j
n
0
,
otherwise
.


### 2.3. The OFG of PCNN

Capture characters of PCNN neurons will cause a similar brightness to the surrounding neurons to capture the ignition.. The capture characters can be automatically coupled to transmit information. In this paper, we use PCNN to extract image features; PCNN also can extract the information of the image’s texture, edge, and regional distribution and has a good effect on image processing. In an iteration of PCNN, a binary image will be obtained by recording the neuron fires or not. The binary images effectively express the features of the image such as texture, edge, and regional distribution; the binary map and OFG are shown in Figures1(b) and 1(c). After the global statistics of the binary image of the neurons, we get an oscillation frequency graph (OFG), which is shown in (6) and Figures 1(d) and 1(e):(6)
O
F
G
N
i
,
j
=
∑
n
=
1
N
Y
i
j
n
,where N denotes the iteration times, Y
i
j denotes the pulse output of the neuron (
i
,
j
), and n is the current iteration.Figure 1
An illustration of the binary map and OFG. (a) Source image. (b) Binary map at 4th time iteration. (c) Binary map at 8th time iteration. (d) OFG at 10th time iteration. (e) OFG at 15th time iteration.
(a)
(b)
(c)
(d)
(e)


### 2.4. Nonsubsampled Shearlet Transform

Shearlet transform was proposed based on wavelet by Labate et al. [9, 11]. In dimension n
=
2, affine system(7)
ψ
A
S
ψ
=
ψ
j
,
l
,
k
x
=
det
⁡
A
j
/
2
ψ
S
l
A
j
x
-
k
:
l
,
j
∈
Z
,
k
∈
Z
2
,where ψ is a collection of basis functions and satisfies ψ
∈
L
2
(
R
2
); A represents anisotropy matrix for multiscale partitions; S is a shear matrix for directional analysis. j, l, and k are scale, direction, and shift parameter, respectively. A and S are both 2
×
2 invertible matrices and det
⁡
S
=
1. For ∀
a
>
0 and s
∈
R, the matrices of A and S are given by(8)
A
=
a
0
0
a
,
S
=
1
s
0
1
.Let a
=
4 and s
=
1, from (8), be further modified:(9)
A
=
4
0
0
2
,
S
=
1
1
0
1
.For ∀
ξ
=
(
ξ
1
,
ξ
2
)
∈
R
^
2, ξ
1
≠
0, the mathematical expression of basic function ψ
^
(
0
) for shearlet transform can be given according to [11](10)
ψ
^
0
ξ
=
ψ
^
0
ξ
1
,
ξ
1
=
ψ
^
1
ξ
1
ψ
^
2
ξ
2
ξ
1
,where ψ
^ is the Fourier transform of ψ, ψ
^
1
∈
C
∞
R and ψ
^
2
∈
C
∞
R are both wavelets, and supplement ψ
2
⊂
-
1
/
2
,
-
1
/
16
∪
1
/
16,1
/
2, supp
⁡
ψ
^
2
⊂
-
1,1. It implies that ψ
^
0
=
C
∞
(
R
) and compactly supported with supp
⁡
ψ
0
⊂
-
1
/
2,1
/
2
2. In addition, we assume that(11)
∑
j
≥
0
ψ
^
1
2
-
2
j
ω
2
=
1
,
ω
≥
1
8
,and for each j
≥
0, ψ
^
2 satisfies (12)
∑
l
=
-
2
j
2
j
-
1
ψ
^
2
2
j
ω
-
l
=
1
,
ω
≤
1
.From the conditions on the support of ψ
^
1, and ψ
^
2, one can obtain that function ψ
j
,
l
,
k has the frequency support listed in(13)
supp
⁡
ψ
^
0
j
,
l
,
k
⊂
ξ
1
,
ξ
2
ξ
1
∈
-
2
2
j
-
1
,
-
2
2
j
-
4
∪
-
2
2
j
-
4
,
-
2
2
j
-
1
,
ξ
2
ξ
1
+
l
2
-
j
≤
2
-
j
.That is, each elementψ
^
j
,
l
,
k is supported on a pair of trapeziform zones, whose sizes all approximate to 2
2
j
×
2
j. The tiling of the frequency by shearlet and the size of the frequency support ψ
j
,
l
,
k.In NSST algorithm, in order to remove the influence of upsampling and subsampling, nonsubsampled Laplacian pyramid filters are used as a substitute in the shearlet transform, so it has excellent performance in terms of shift-invariance, multiscale, and multidirectional properties. The discretization process of NSST has two phases: multi-scale factorization and multi-directional factorization. Nonsubsampled Laplacian pyramid filters complete multiscale factorization. The first phase usesk classes two-channel nonsubsampled filter to get one low frequency image and k high frequency images. The multidirectional factorization in NSST is realized via improved shearlet transform. These filters are formed by avoiding the subsampling to satisfy the property of shift-invariance. Shearlet transform allows the direction decomposition with l stages in high frequency images from nonsubsampled Laplacian pyramid at each level and produces 2
l directional subimages with the same size as the source image [16, 17]. Figure 2 shows the two-level NSST decomposition of an image.Figure 2
An illustration of NSST. (a) Source image. (b) The approximate NSST coefficients. (c) Images of the detailed coefficients at level 1. (d) Images of the detailed coefficients at level 2.
(a)
(b)
(c)
(d)


## 2.1. PCNN

PCNN model has three fundamental parts: the receptive field, the modulation field, and the pulse generator [13, 14]. In the receptive field, which consists of L and F channels and is described by (1), the neuron receives neighboring neurons’ coupling input Y and external stimulus input S. In L and F channels of the neuron, the neuron links with its neighborhood neurons via the synaptic linking weights W and M, respectively; the two channels accumulate input and exponential decay changes at the same time; the decay exponentials are α
L and α
F, respectively, while the channel amplitudes are V
L and V
F, respectively:(1)
F
i
j
n
=
V
F
∑
k
l
M
i
j
k
l
Y
k
l
n
-
1
+
e
-
α
F
F
i
j
n
-
1
+
S
i
j
,
L
i
j
n
=
V
L
∑
k
l
W
i
j
k
l
Y
k
l
n
-
1
+
L
i
j
n
-
1
e
-
α
L
.In the modulation field, the linking inputL
i
j
(
n
) made by adding a bias to the linking; then, it is multiplied by the feeding input; the bias is unitary, β is the linking strength, and the total internal activity U is the result of modulation, which is described by(2)
U
i
j
n
=
F
i
j
n
1
+
β
L
i
j
n
.Pulse generator consists of a threshold adjuster, a comparison organ, and a pulse generator, which is described by (3). Its function is to generate the pulse output Y, and θ
i
j is adjustment threshold; V
i
j
θ is threshold coefficient. When the internal state U is larger than the threshold θ, that is, the neuron satisfies the condition U
i
j
n
>
θ
i
j
n, a pulse would be produced by the neuron; we call an ignition, which is described by (4):(3)
θ
i
j
n
=
e
-
α
θ
θ
i
j
n
-
1
+
V
i
j
θ
Y
i
j
n
-
1
,
(4)
Y
i
j
n
=
1
,
U
i
j
n
>
θ
i
j
n
0
,
otherwise
,where the subscripts i and j represent the neuron location in PCNN and n denotes the current iteration (discrete time step), where n varies from 1 to N (N is the total number of iterations). In particular, “a neuron ignition” means a PCNN’s neuron generates a pulse. The total times of ignitions represent image information of the corresponding code sequences after N iterations.When PCNN is used for image processing, a pixel is connected to unique neuron. The number of neurons in the network is equal to the pixel number of the input image; namely, there exists one-to-one correspondence between the image and neurons network, and the pixel value is taken as the external input stimulus of the neuron inF channel. A neuron outputs results in two states, namely, pulse (status 1) and nonpulse (status 0), so the output status of neurons composes a binary image. More information about PCNN will be found in [12–15].


## 2.2. S-PCNN

Simplified PCNN (S-PCNN) model [15] is composed the same as the original PCNN model, but the input of F channel is only related to image gray value and has no relationship with external coupling and exponential decay characteristics, and its parameters are less than original PCNN model, and the input channel of the receptive field is simple and effective. In S-PCNN model, the variables of a neuron N
i
j satisfy the following:(5)
F
i
j
n
=
S
i
j
,
L
i
j
n
=
V
i
j
L
∑
k
l
W
i
j
k
l
Y
k
l
n
-
1
+
L
i
j
n
-
1
e
-
α
L
,
U
i
j
n
=
F
i
j
n
1
+
β
L
i
j
n
,
θ
i
j
n
=
e
-
α
θ
θ
i
j
n
-
1
+
V
i
j
θ
Y
i
j
n
-
1
,
Y
i
j
n
=
1
,
U
i
j
n
>
θ
i
j
n
0
,
otherwise
.


## 2.3. The OFG of PCNN

Capture characters of PCNN neurons will cause a similar brightness to the surrounding neurons to capture the ignition.. The capture characters can be automatically coupled to transmit information. In this paper, we use PCNN to extract image features; PCNN also can extract the information of the image’s texture, edge, and regional distribution and has a good effect on image processing. In an iteration of PCNN, a binary image will be obtained by recording the neuron fires or not. The binary images effectively express the features of the image such as texture, edge, and regional distribution; the binary map and OFG are shown in Figures1(b) and 1(c). After the global statistics of the binary image of the neurons, we get an oscillation frequency graph (OFG), which is shown in (6) and Figures 1(d) and 1(e):(6)
O
F
G
N
i
,
j
=
∑
n
=
1
N
Y
i
j
n
,where N denotes the iteration times, Y
i
j denotes the pulse output of the neuron (
i
,
j
), and n is the current iteration.Figure 1
An illustration of the binary map and OFG. (a) Source image. (b) Binary map at 4th time iteration. (c) Binary map at 8th time iteration. (d) OFG at 10th time iteration. (e) OFG at 15th time iteration.
(a)
(b)
(c)
(d)
(e)


## 2.4. Nonsubsampled Shearlet Transform

Shearlet transform was proposed based on wavelet by Labate et al. [9, 11]. In dimension n
=
2, affine system(7)
ψ
A
S
ψ
=
ψ
j
,
l
,
k
x
=
det
⁡
A
j
/
2
ψ
S
l
A
j
x
-
k
:
l
,
j
∈
Z
,
k
∈
Z
2
,where ψ is a collection of basis functions and satisfies ψ
∈
L
2
(
R
2
); A represents anisotropy matrix for multiscale partitions; S is a shear matrix for directional analysis. j, l, and k are scale, direction, and shift parameter, respectively. A and S are both 2
×
2 invertible matrices and det
⁡
S
=
1. For ∀
a
>
0 and s
∈
R, the matrices of A and S are given by(8)
A
=
a
0
0
a
,
S
=
1
s
0
1
.Let a
=
4 and s
=
1, from (8), be further modified:(9)
A
=
4
0
0
2
,
S
=
1
1
0
1
.For ∀
ξ
=
(
ξ
1
,
ξ
2
)
∈
R
^
2, ξ
1
≠
0, the mathematical expression of basic function ψ
^
(
0
) for shearlet transform can be given according to [11](10)
ψ
^
0
ξ
=
ψ
^
0
ξ
1
,
ξ
1
=
ψ
^
1
ξ
1
ψ
^
2
ξ
2
ξ
1
,where ψ
^ is the Fourier transform of ψ, ψ
^
1
∈
C
∞
R and ψ
^
2
∈
C
∞
R are both wavelets, and supplement ψ
2
⊂
-
1
/
2
,
-
1
/
16
∪
1
/
16,1
/
2, supp
⁡
ψ
^
2
⊂
-
1,1. It implies that ψ
^
0
=
C
∞
(
R
) and compactly supported with supp
⁡
ψ
0
⊂
-
1
/
2,1
/
2
2. In addition, we assume that(11)
∑
j
≥
0
ψ
^
1
2
-
2
j
ω
2
=
1
,
ω
≥
1
8
,and for each j
≥
0, ψ
^
2 satisfies (12)
∑
l
=
-
2
j
2
j
-
1
ψ
^
2
2
j
ω
-
l
=
1
,
ω
≤
1
.From the conditions on the support of ψ
^
1, and ψ
^
2, one can obtain that function ψ
j
,
l
,
k has the frequency support listed in(13)
supp
⁡
ψ
^
0
j
,
l
,
k
⊂
ξ
1
,
ξ
2
ξ
1
∈
-
2
2
j
-
1
,
-
2
2
j
-
4
∪
-
2
2
j
-
4
,
-
2
2
j
-
1
,
ξ
2
ξ
1
+
l
2
-
j
≤
2
-
j
.That is, each elementψ
^
j
,
l
,
k is supported on a pair of trapeziform zones, whose sizes all approximate to 2
2
j
×
2
j. The tiling of the frequency by shearlet and the size of the frequency support ψ
j
,
l
,
k.In NSST algorithm, in order to remove the influence of upsampling and subsampling, nonsubsampled Laplacian pyramid filters are used as a substitute in the shearlet transform, so it has excellent performance in terms of shift-invariance, multiscale, and multidirectional properties. The discretization process of NSST has two phases: multi-scale factorization and multi-directional factorization. Nonsubsampled Laplacian pyramid filters complete multiscale factorization. The first phase usesk classes two-channel nonsubsampled filter to get one low frequency image and k high frequency images. The multidirectional factorization in NSST is realized via improved shearlet transform. These filters are formed by avoiding the subsampling to satisfy the property of shift-invariance. Shearlet transform allows the direction decomposition with l stages in high frequency images from nonsubsampled Laplacian pyramid at each level and produces 2
l directional subimages with the same size as the source image [16, 17]. Figure 2 shows the two-level NSST decomposition of an image.Figure 2
An illustration of NSST. (a) Source image. (b) The approximate NSST coefficients. (c) Images of the detailed coefficients at level 1. (d) Images of the detailed coefficients at level 2.
(a)
(b)
(c)
(d)


## 3. The Proposed Algorithm

In this section, the proposed multifocus color image fusion algorithm is presented in detail. The framework of the proposed algorithm is shown in Figure3. In this algorithm, the RGB color images are transformed into HSV color space [18], NSST is used to decompose the image and PCNN is used to extract the features and fuse these features using different rules. Besides, it is important to note that two kinds of PCNN model are used in the algorithm; adaptive S-PCNN is used to fuse H component and original PCNN is used to fuse high frequency coefficients of NSST.Figure 3
The schematic diagram of the proposed image fusion method.


### 3.1. RGB Color Image Transform to HSV

RGB color image contains almost all basic colors that can be perceived by human vision; however, the correlation among the components is very strong; it is shown in Figures4(b), 4(c), and 4(d); this makes it difficult for RGB color image to deal with the fact that the color of the image will be changed if a component changes. HSV image can be obtained by the RGB transform. The values of R, G, and B correspond to unique H, S, and V values, as the values of H, S, and V components depend on the values of R, G, and B in RGB color space. This color system is closer than the RGB color system to human experience and perception of the color; this is shown in Figures 4(e), 4(f), and 4(g).Figure 4
An illustration of HSV. (a) Source image. (b) R component. (c) G component. (d) B component. (e) H component. (f) S component. (g) V component. (h) Histogram of H. (i) Histogram of S. (j) Histogram of V.
(a)
(b)
(c)
(d)
(e)
(f)
(g)
(h)
(i)
(j)The distribution of H component is concentrated, and its pixel value generally is small; see Figure4(h). So its edge is obvious, and S-PCNN is sensitive to edge and regional distribution of the image. In this algorithm, H component is partitioned into blocks and input into adaptive S-PCNN to get OFG of the H component; we fuse the H component according to the OFG. The distribution of S and V components is dispersive; see Figures 4(i) and 4(j), which contain lots of details of the image in different grayscale, so S and V components are decomposed into multiscale and multidirectional subband by NSST to get different detailed information of the images, and then according to the characters of new S and V components, new S and V components are fused using different rules.


### 3.2. H Component Fusion Using Adaptive S-PCNN

β is the linking strength of S-PCNN, and it is a key determinant to the ignition behavior of S-PCNN. Spatial frequency (SF) and average gradient (AG) are very important indicators of the image definition, which represent the quality of the image. So S-PCNN should be adaptively adjusted by SF and AG to make it work well. In S-PCNN model, H component is divided into several blocks, and then SF and AG of the blocks are calculated as linking strength β of S-PCNN, respectively, so this can adaptively adjust β of S-PCNN; see (14) and (15). The block images are input into S-PCNN to get two kinds of OFG according to different β, which can effectively express the quality of the block images. The two types of indexes will change with the content of the image and produce a certain range of changes. The index itself and its change range are suitable for adjusting the parameter β of S-PCNN, which can affect the OFG of S-PCNN, so that the two types of OFG can reflect different details of the image. If one type of OFG is only used, as the size of OFG for different quality of pixel is the same, this cannot select appropriate pixels. However, the combination of two types OFG can reflect the image information from different angles; thus, this reduces the possibility of different pixels with the same OFG. It can make the algorithm more effective.SF and AG of the image are described in (16) to (19):(14)
β
1
=
η
1
∗
S
F
,
(15)
β
2
=
η
2
∗
A
G
,where η is adjusted factor. Consider(16)
S
F
=
R
F
2
+
C
F
2
,
(17)
R
F
=
1
M
×
N
∑
i
=
1
M
∑
j
=
2
N
F
i
,
j
-
F
i
,
j
-
1
2
,
(18)
C
F
=
1
M
×
N
∑
i
=
1
N
∑
j
=
2
M
F
i
,
j
-
F
i
,
j
-
1
2
.SF is composed of row frequency (RF) and column frequency (CF), whereM is the row of the image, N is the column of the image, and F
(
i
,
j
) is grey level of the image F at pixel (
i
,
j
):(19)
A
G
=
1
M
×
N
∑
i
=
1
M
∑
j
=
1
N
1
2
F
i
,
j
-
F
i
+
1
,
j
2
+
F
i
,
j
-
F
i
,
j
+
1
2
.


### 3.3. S and V Components Fusion Using NSST and PCNN

S and V components contain lots of details of the image, and the gray values of S and V components are dispersive, respectively. S and V components are decomposed by NSST to make it easy to extract, and then we will get one low frequency subband and several high frequency subbands in multiscale and multidirection. The higher values of the low frequency coefficients are used into the new fused low frequency subband. The high frequency subbands contain abundant detailed information of the S and V components; these components are input into original PCNN model to get OFG, which shows the statistics of the ignition times of the pixels. Compared with the OFG of the high frequency subbands from different S and V components, we will get the new fused high frequency subbands. At last, inverse NSST is performed to get the fused S and V components according to the new fused low frequency subbands and high frequency subbands.


### 3.4. Algorithm Steps

The proposed fusion algorithm processes are shown in Figure3, and the steps are as follows.Step 0. Given source images A and B.Step 1. The color image in RGB color space is converted to HSV color space to get three components H, S, and V.Step 2. PCNN is utilized to deal with H component:(a)
The H components of imagesA
i
j and B
i
j are divided into subblocks, and then the SF and AG of the subblocks are calculated using (16) to (19); the linking strengths β
1 and β
2 are gotten using (14) and (15); β
1 and β
2 acted as β value of S-PCNN, respectively.
(b)
The subblocks are input into S-PCNN model twice with differentβ to get two OFGs of the H components.
(c)
Get the average value of two OFGs using (20), and then F
i
j of the fused H component can be decided by (21):(20)
A
O
F
G
=
O
F
G
β
1
+
O
F
G
β
2
2
,
(21)
F
i
j
=
A
i
j
,
A
O
F
G
A
i
j
>
A
O
F
G
B
i
j
,
B
i
j
,
A
O
F
G
A
i
j
<
A
O
F
G
B
i
j
,
A
i
j
+
B
i
j
2
,
A
O
F
G
A
i
j
=
A
O
F
G
B
i
j
,
whereA
O
F
G
A
i
j and A
O
F
G
B
i
j are the average OFG of images A and B.Step 3. Perform decomposition of source S and V components of image A
i
j and B
i
j using NSST to obtain the low-pass subband coefficients and the high-pass directional subband coefficients; different rules are utilized to deal with the S and V components.(a)
The fused low-pass subband coefficientsF
C
i
j can be decided by (22)
F
C
i
j
=
C
A
i
j
,
C
A
i
j
>
C
B
i
j
,
C
B
i
j
,
C
A
i
j
<
C
B
i
j
,
C
A
i
j
+
C
B
i
j
2
,
C
A
i
j
=
C
B
i
j
,
whereC
A
i
j and C
B
i
j is the low-pass subband coefficients of images A and B.
(b)
The high-pass directional subband coefficients are input into PCNN model to get corresponding OFGs (O
F
G
A
i
j, O
F
G
B
i
j) according to the statistics of the ignition times of the pixels; the fusion rule of S and H components can be decided by(23)
F
i
j
=
A
i
j
,
O
F
G
A
i
j
>
O
F
G
B
i
j
,
B
i
j
,
O
F
G
A
i
j
<
O
F
G
B
i
j
,
A
i
j
+
B
i
j
2
,
O
F
G
A
i
j
=
O
F
G
B
i
j
.Step 4. Reconstruct the fused HSV image by an inverse NSST.Step 5. Obtain the fused RGB color image by inverse HSV.


## 3.1. RGB Color Image Transform to HSV

RGB color image contains almost all basic colors that can be perceived by human vision; however, the correlation among the components is very strong; it is shown in Figures4(b), 4(c), and 4(d); this makes it difficult for RGB color image to deal with the fact that the color of the image will be changed if a component changes. HSV image can be obtained by the RGB transform. The values of R, G, and B correspond to unique H, S, and V values, as the values of H, S, and V components depend on the values of R, G, and B in RGB color space. This color system is closer than the RGB color system to human experience and perception of the color; this is shown in Figures 4(e), 4(f), and 4(g).Figure 4
An illustration of HSV. (a) Source image. (b) R component. (c) G component. (d) B component. (e) H component. (f) S component. (g) V component. (h) Histogram of H. (i) Histogram of S. (j) Histogram of V.
(a)
(b)
(c)
(d)
(e)
(f)
(g)
(h)
(i)
(j)The distribution of H component is concentrated, and its pixel value generally is small; see Figure4(h). So its edge is obvious, and S-PCNN is sensitive to edge and regional distribution of the image. In this algorithm, H component is partitioned into blocks and input into adaptive S-PCNN to get OFG of the H component; we fuse the H component according to the OFG. The distribution of S and V components is dispersive; see Figures 4(i) and 4(j), which contain lots of details of the image in different grayscale, so S and V components are decomposed into multiscale and multidirectional subband by NSST to get different detailed information of the images, and then according to the characters of new S and V components, new S and V components are fused using different rules.


## 3.2. H Component Fusion Using Adaptive S-PCNN

β is the linking strength of S-PCNN, and it is a key determinant to the ignition behavior of S-PCNN. Spatial frequency (SF) and average gradient (AG) are very important indicators of the image definition, which represent the quality of the image. So S-PCNN should be adaptively adjusted by SF and AG to make it work well. In S-PCNN model, H component is divided into several blocks, and then SF and AG of the blocks are calculated as linking strength β of S-PCNN, respectively, so this can adaptively adjust β of S-PCNN; see (14) and (15). The block images are input into S-PCNN to get two kinds of OFG according to different β, which can effectively express the quality of the block images. The two types of indexes will change with the content of the image and produce a certain range of changes. The index itself and its change range are suitable for adjusting the parameter β of S-PCNN, which can affect the OFG of S-PCNN, so that the two types of OFG can reflect different details of the image. If one type of OFG is only used, as the size of OFG for different quality of pixel is the same, this cannot select appropriate pixels. However, the combination of two types OFG can reflect the image information from different angles; thus, this reduces the possibility of different pixels with the same OFG. It can make the algorithm more effective.SF and AG of the image are described in (16) to (19):(14)
β
1
=
η
1
∗
S
F
,
(15)
β
2
=
η
2
∗
A
G
,where η is adjusted factor. Consider(16)
S
F
=
R
F
2
+
C
F
2
,
(17)
R
F
=
1
M
×
N
∑
i
=
1
M
∑
j
=
2
N
F
i
,
j
-
F
i
,
j
-
1
2
,
(18)
C
F
=
1
M
×
N
∑
i
=
1
N
∑
j
=
2
M
F
i
,
j
-
F
i
,
j
-
1
2
.SF is composed of row frequency (RF) and column frequency (CF), whereM is the row of the image, N is the column of the image, and F
(
i
,
j
) is grey level of the image F at pixel (
i
,
j
):(19)
A
G
=
1
M
×
N
∑
i
=
1
M
∑
j
=
1
N
1
2
F
i
,
j
-
F
i
+
1
,
j
2
+
F
i
,
j
-
F
i
,
j
+
1
2
.


## 3.3. S and V Components Fusion Using NSST and PCNN

S and V components contain lots of details of the image, and the gray values of S and V components are dispersive, respectively. S and V components are decomposed by NSST to make it easy to extract, and then we will get one low frequency subband and several high frequency subbands in multiscale and multidirection. The higher values of the low frequency coefficients are used into the new fused low frequency subband. The high frequency subbands contain abundant detailed information of the S and V components; these components are input into original PCNN model to get OFG, which shows the statistics of the ignition times of the pixels. Compared with the OFG of the high frequency subbands from different S and V components, we will get the new fused high frequency subbands. At last, inverse NSST is performed to get the fused S and V components according to the new fused low frequency subbands and high frequency subbands.


## 3.4. Algorithm Steps

The proposed fusion algorithm processes are shown in Figure3, and the steps are as follows.Step 0. Given source images A and B.Step 1. The color image in RGB color space is converted to HSV color space to get three components H, S, and V.Step 2. PCNN is utilized to deal with H component:(a)
The H components of imagesA
i
j and B
i
j are divided into subblocks, and then the SF and AG of the subblocks are calculated using (16) to (19); the linking strengths β
1 and β
2 are gotten using (14) and (15); β
1 and β
2 acted as β value of S-PCNN, respectively.
(b)
The subblocks are input into S-PCNN model twice with differentβ to get two OFGs of the H components.
(c)
Get the average value of two OFGs using (20), and then F
i
j of the fused H component can be decided by (21):(20)
A
O
F
G
=
O
F
G
β
1
+
O
F
G
β
2
2
,
(21)
F
i
j
=
A
i
j
,
A
O
F
G
A
i
j
>
A
O
F
G
B
i
j
,
B
i
j
,
A
O
F
G
A
i
j
<
A
O
F
G
B
i
j
,
A
i
j
+
B
i
j
2
,
A
O
F
G
A
i
j
=
A
O
F
G
B
i
j
,
whereA
O
F
G
A
i
j and A
O
F
G
B
i
j are the average OFG of images A and B.Step 3. Perform decomposition of source S and V components of image A
i
j and B
i
j using NSST to obtain the low-pass subband coefficients and the high-pass directional subband coefficients; different rules are utilized to deal with the S and V components.(a)
The fused low-pass subband coefficientsF
C
i
j can be decided by (22)
F
C
i
j
=
C
A
i
j
,
C
A
i
j
>
C
B
i
j
,
C
B
i
j
,
C
A
i
j
<
C
B
i
j
,
C
A
i
j
+
C
B
i
j
2
,
C
A
i
j
=
C
B
i
j
,
whereC
A
i
j and C
B
i
j is the low-pass subband coefficients of images A and B.
(b)
The high-pass directional subband coefficients are input into PCNN model to get corresponding OFGs (O
F
G
A
i
j, O
F
G
B
i
j) according to the statistics of the ignition times of the pixels; the fusion rule of S and H components can be decided by(23)
F
i
j
=
A
i
j
,
O
F
G
A
i
j
>
O
F
G
B
i
j
,
B
i
j
,
O
F
G
A
i
j
<
O
F
G
B
i
j
,
A
i
j
+
B
i
j
2
,
O
F
G
A
i
j
=
O
F
G
B
i
j
.Step 4. Reconstruct the fused HSV image by an inverse NSST.Step 5. Obtain the fused RGB color image by inverse HSV.


## 4. Experimental Results and Analysis

To verify the validity of the algorithm presented in this paper, we take several groups of experimental color images with different focus position test. The first group of color image is cups, which is shown in Figure5. Image A focuses on the left and image B focuses on the right, and there are many words as details.Figure 5
Source images. (a) Source imageA. (b) Source image B.
(a)
(b)


### 4.1. Evaluation Index System

In order to verify the effectiveness of this method, we consider the quantitative assessment of the fused images. For evaluation of the proposed fusion method, we have considered several common fusion performance metrics defined in this section. The final fused color images are composed of three components such as R, G, and B. Each component can be regarded as a grayscale image, and the quality of the fused color image strongly depends on the grayscale image quality. In this paper, we take the average of the three component’s evaluation indexes as the final color image evaluation index, which is the basic indicators of image fusion quality. Tables1 and 2 show the evaluation of the fused image quality with space frequency (SF), average gradient (AG), entropy (EN), mean value (MV), standard deviation (SD), and mutual information (MI), and Q
A
B
F indicates how much edge information is reserved in the fused image [19–23].Table 1
Fusion quality using different methods.
Method
SF
AG
EN
MV
SD
MI
Q
A
B
F
Weighted
12.7859
4.2004
7.4416
166.7102
65.1893
5.4941
0.5449
PCA
13.0306
4.2311
7.4408
166.5136
65.3166
5.5144
0.5525
PCNN
16.8538
4.9683
7.4449
167.3324
65.8106
5.5454
0.5986
PCNN + WT
17.7381
5.2384
7.2349
143.0606
57.6579
5.1952
0.6696
PCNN + LP
18.8702
5.4816
7.3187
150.6954
61.1962
5.2172
0.6795
This paper (HSV)
20.3181
5.8530
7.4487
175.4014
61.6361
5.6377
0.6886Table 2
Fusion quality under different color spaces.
Color spaces
SF
AG
EN
MV
SD
MI
Q
A
B
F
NTSC
19.1768
5.5980
7.4530
171.1844
66.1808
5.8862
0.6547
YUV
19.1937
5.5983
7.4382
171.1177
66.2103
5.9109
0.6555
YCbCr
19.1805
5.6076
7.4743
171.6032
65.4123
5.7668
0.6496
HSV
20.3181
5.8530
7.4487
175.4014
61.6361
5.6377
0.6886
HIS
20.4130
5.8660
7.3967
172.0959
68.2812
5.8896
0.6989
LAB
20.5100
5.9280
7.42820
172.4920
67.6792
6.0364
0.7018The space frequency (SF) is defined by (16) to (18), and the average gradient is defined by (19).


#### 4.1.1. Entropy

The entropy (EN) of an image is defined by(24)
H
=
-
∑
i
=
0
L
-
1
P
i
log
2
⁡
P
i
,where P
i is the probability of the gray level i in the image and L is the gray level of the image from 0 to 255.


#### 4.1.2. Mean Value

The mean value (MV) of the image is defined by(25)
μ
=
1
M
×
N
∑
i
=
1
M
∑
j
=
1
N
F
i
,
j
,where F
(
i
,
j
) is the pixel value of the fused image at the position (
i
,
j
). MV represents the average brightness of the whole image.


#### 4.1.3. Standard Deviation

The standard deviation (SD) of an image is defined by(26)
S
D
=
1
M
×
N
∑
i
=
1
M
∑
j
=
1
N
F
i
,
j
-
μ
2
,where F
(
i
,
j
) is the pixel value of the fused image at the position (
i
,
j
) and μ is the mean value of the image. The larger the SD is, the better the result is.


#### 4.1.4. Mutual Information

The mutual informationM
I between the source images A and B and the fused image F is defined by(27)
M
I
=
∑
i
=
0
L
-
1
∑
j
=
0
L
-
1
∑
k
=
0
L
-
1
P
A
B
F
i
,
j
,
k
log
2
⁡
P
A
B
F
i
,
j
,
k
P
A
B
i
,
j
P
F
k
,where P
A
B
F
(
i
,
j
,
k
) is the normalization union grey level histogram of images A and B and F, P
A
B
(
i
,
j
) is the normalization union grey level histogram of images A and B, P
F
(
k
) is the normalization grey level histogram of F, L is the gray level of the image, and (
i
,
j
,
k
) represents the pixel value of images A and B and F, respectively.


#### 4.1.5. Edge Based on Similarity Measure

TheQ
A
B
F metric evaluates the sum of edge information preservation values and is defined by(28)
Q
A
B
F
=
∑
i
=
1
M
∑
j
=
1
N
Q
A
F
i
,
j
×
ω
A
i
,
j
+
Q
B
F
i
,
j
×
ω
B
i
,
j
∑
i
=
1
M
∑
j
=
1
N
ω
A
i
,
j
+
ω
B
i
,
j
-
1
,where Q
A
F
(
i
,
j
)
=
Q
β
A
F
(
i
,
j
)
Q
α
A
F
(
i
,
j
), Q
β
A
F
(
i
,
j
), Q
β
A
F
(
i
,
j
), and Q
α
A
F
(
i
,
j
) are the edge strength and orientation preservation values, respectively, Q
B
F
(
i
,
j
) is similar to Q
β
A
F
(
i
,
j
), and ω
A
(
i
,
j
) and ω
B
(
i
,
j
) are weights to measure the importance of Q
A
F
(
i
,
j
) and Q
B
F
(
i
,
j
), respectively. The dynamic range Q
A
B
F is [
0
,
1
], and it should be close to 1 as far as possible for the best fusion, as Q
A
B
F
=
1. In addition, (
i
,
j
) represents the pixel location, and M and N are the sizes of images, respectively.


### 4.2. Experiment One

For comparison, this paper proposes the fusion scheme of the algorithm and presents several common fusion algorithms; for instance, take the average of the source images pixel by pixel (average), principal component analysis (PCA), pulse coupled neural network (PCNN), pulse coupled neural network and Laplacian pyramid transform (PCNN + LP), and pulse coupled neural network and discrete wavelet transform (PCNN + WT). The fusion images using different methods are shown in Figure6. According the contrast experiments, the fusion image of this paper is better than the others; see Figure 6. The algorithm of this paper does well in extracting the characteristics of the source images, and the fused image is closer to the natural color, which contains more edges, texture, and detail, so it is the closest to the source images. We can conclude that the method in this paper is an effective method.Figure 6
Fusion images from different methods. (a) Average method. (b) PCA method. (c) PCNN method. (d) PCNN + WT. (e) PCNN + LP. (f) This paper.
(a)
(b)
(c)
(d)
(e)
(f)From Table1, the fusion image of this paper contains much more information. SF, AG, MV, and Q
A
B
F of this paper are larger than other methods, EN and MI are slightly better than other methods, and only SD is less than other methods.


### 4.3. Experiment Two

In other color spaces such as NTSC, YUV, YCbCr, HIS, and LAB, all have the similar histogram distribution as the distribution of H, S, and V components in HSV color space, so all can use the proposed algorithm for color image fusion. In order to confirm the method that works best under the color space of fusion, we carry out the following experiments. The fusion images under different color space are shown in Figure7; all can achieve the goal of the image fusion, but the fusion effects on HSV, HIS, and LAB color spaces are better than others.Figure 7
Fusion images under different color spaces. (a) NTSC. (b) YUV. (c) YCbCr. (d) HSV. (e) HIS. (f) LAB.
(a)
(b)
(c)
(d)
(e)
(f)From Table2, it is indicated that the fusion image of this paper in LAB color space contains more information, and SF, AG, MI, and Q
A
B
F are better than others. Overall, the evaluation indexes in HSV, HIS, and LAB are better than others. In practical application, we suggest that the color image fusion of the proposed algorithm should focus on HSV, HIS, and LAB color spaces.


### 4.4. Experiment Three

More experimental results of coastal images in HSV color space are shown in Figure8, the source images with different focus position, and there are a lot of textures in the source images.Figure 8
The fusion results of the coastal images. (a) Source imageA. (b) Source image B. (c) Fusion image. (d) Source image A. (e) Source image B. (f) Fusion image. (g) Source image A. (h) Source image B. (i) Fusion image. (j) Source image A. (k) Source image B. (l) Fusion image.
(a)
(b)
(c)
(d)
(e)
(f)
(g)
(h)
(i)
(j)
(k)
(l)It can be seen in Figure8 that the edge of the fusion image is clear, and it retains most of the textures in the source images; besides, the details are also well preserved. This method can extract the main features from the source images; it shows that the method in this paper also achieved effective results in these groups of color images.The same conclusion as in Figure6 can be concluded. Overall, the method presented in this paper is better than the traditional methods obviously. Compared with the other methods, this method reflects better performance and visual effect in terms of definition and detail.


## 4.1. Evaluation Index System

In order to verify the effectiveness of this method, we consider the quantitative assessment of the fused images. For evaluation of the proposed fusion method, we have considered several common fusion performance metrics defined in this section. The final fused color images are composed of three components such as R, G, and B. Each component can be regarded as a grayscale image, and the quality of the fused color image strongly depends on the grayscale image quality. In this paper, we take the average of the three component’s evaluation indexes as the final color image evaluation index, which is the basic indicators of image fusion quality. Tables1 and 2 show the evaluation of the fused image quality with space frequency (SF), average gradient (AG), entropy (EN), mean value (MV), standard deviation (SD), and mutual information (MI), and Q
A
B
F indicates how much edge information is reserved in the fused image [19–23].Table 1
Fusion quality using different methods.
Method
SF
AG
EN
MV
SD
MI
Q
A
B
F
Weighted
12.7859
4.2004
7.4416
166.7102
65.1893
5.4941
0.5449
PCA
13.0306
4.2311
7.4408
166.5136
65.3166
5.5144
0.5525
PCNN
16.8538
4.9683
7.4449
167.3324
65.8106
5.5454
0.5986
PCNN + WT
17.7381
5.2384
7.2349
143.0606
57.6579
5.1952
0.6696
PCNN + LP
18.8702
5.4816
7.3187
150.6954
61.1962
5.2172
0.6795
This paper (HSV)
20.3181
5.8530
7.4487
175.4014
61.6361
5.6377
0.6886Table 2
Fusion quality under different color spaces.
Color spaces
SF
AG
EN
MV
SD
MI
Q
A
B
F
NTSC
19.1768
5.5980
7.4530
171.1844
66.1808
5.8862
0.6547
YUV
19.1937
5.5983
7.4382
171.1177
66.2103
5.9109
0.6555
YCbCr
19.1805
5.6076
7.4743
171.6032
65.4123
5.7668
0.6496
HSV
20.3181
5.8530
7.4487
175.4014
61.6361
5.6377
0.6886
HIS
20.4130
5.8660
7.3967
172.0959
68.2812
5.8896
0.6989
LAB
20.5100
5.9280
7.42820
172.4920
67.6792
6.0364
0.7018The space frequency (SF) is defined by (16) to (18), and the average gradient is defined by (19).


### 4.1.1. Entropy

The entropy (EN) of an image is defined by(24)
H
=
-
∑
i
=
0
L
-
1
P
i
log
2
⁡
P
i
,where P
i is the probability of the gray level i in the image and L is the gray level of the image from 0 to 255.


### 4.1.2. Mean Value

The mean value (MV) of the image is defined by(25)
μ
=
1
M
×
N
∑
i
=
1
M
∑
j
=
1
N
F
i
,
j
,where F
(
i
,
j
) is the pixel value of the fused image at the position (
i
,
j
). MV represents the average brightness of the whole image.


### 4.1.3. Standard Deviation

The standard deviation (SD) of an image is defined by(26)
S
D
=
1
M
×
N
∑
i
=
1
M
∑
j
=
1
N
F
i
,
j
-
μ
2
,where F
(
i
,
j
) is the pixel value of the fused image at the position (
i
,
j
) and μ is the mean value of the image. The larger the SD is, the better the result is.


### 4.1.4. Mutual Information

The mutual informationM
I between the source images A and B and the fused image F is defined by(27)
M
I
=
∑
i
=
0
L
-
1
∑
j
=
0
L
-
1
∑
k
=
0
L
-
1
P
A
B
F
i
,
j
,
k
log
2
⁡
P
A
B
F
i
,
j
,
k
P
A
B
i
,
j
P
F
k
,where P
A
B
F
(
i
,
j
,
k
) is the normalization union grey level histogram of images A and B and F, P
A
B
(
i
,
j
) is the normalization union grey level histogram of images A and B, P
F
(
k
) is the normalization grey level histogram of F, L is the gray level of the image, and (
i
,
j
,
k
) represents the pixel value of images A and B and F, respectively.


### 4.1.5. Edge Based on Similarity Measure

TheQ
A
B
F metric evaluates the sum of edge information preservation values and is defined by(28)
Q
A
B
F
=
∑
i
=
1
M
∑
j
=
1
N
Q
A
F
i
,
j
×
ω
A
i
,
j
+
Q
B
F
i
,
j
×
ω
B
i
,
j
∑
i
=
1
M
∑
j
=
1
N
ω
A
i
,
j
+
ω
B
i
,
j
-
1
,where Q
A
F
(
i
,
j
)
=
Q
β
A
F
(
i
,
j
)
Q
α
A
F
(
i
,
j
), Q
β
A
F
(
i
,
j
), Q
β
A
F
(
i
,
j
), and Q
α
A
F
(
i
,
j
) are the edge strength and orientation preservation values, respectively, Q
B
F
(
i
,
j
) is similar to Q
β
A
F
(
i
,
j
), and ω
A
(
i
,
j
) and ω
B
(
i
,
j
) are weights to measure the importance of Q
A
F
(
i
,
j
) and Q
B
F
(
i
,
j
), respectively. The dynamic range Q
A
B
F is [
0
,
1
], and it should be close to 1 as far as possible for the best fusion, as Q
A
B
F
=
1. In addition, (
i
,
j
) represents the pixel location, and M and N are the sizes of images, respectively.


## 4.1.1. Entropy

The entropy (EN) of an image is defined by(24)
H
=
-
∑
i
=
0
L
-
1
P
i
log
2
⁡
P
i
,where P
i is the probability of the gray level i in the image and L is the gray level of the image from 0 to 255.


## 4.1.2. Mean Value

The mean value (MV) of the image is defined by(25)
μ
=
1
M
×
N
∑
i
=
1
M
∑
j
=
1
N
F
i
,
j
,where F
(
i
,
j
) is the pixel value of the fused image at the position (
i
,
j
). MV represents the average brightness of the whole image.


## 4.1.3. Standard Deviation

The standard deviation (SD) of an image is defined by(26)
S
D
=
1
M
×
N
∑
i
=
1
M
∑
j
=
1
N
F
i
,
j
-
μ
2
,where F
(
i
,
j
) is the pixel value of the fused image at the position (
i
,
j
) and μ is the mean value of the image. The larger the SD is, the better the result is.


## 4.1.4. Mutual Information

The mutual informationM
I between the source images A and B and the fused image F is defined by(27)
M
I
=
∑
i
=
0
L
-
1
∑
j
=
0
L
-
1
∑
k
=
0
L
-
1
P
A
B
F
i
,
j
,
k
log
2
⁡
P
A
B
F
i
,
j
,
k
P
A
B
i
,
j
P
F
k
,where P
A
B
F
(
i
,
j
,
k
) is the normalization union grey level histogram of images A and B and F, P
A
B
(
i
,
j
) is the normalization union grey level histogram of images A and B, P
F
(
k
) is the normalization grey level histogram of F, L is the gray level of the image, and (
i
,
j
,
k
) represents the pixel value of images A and B and F, respectively.


## 4.1.5. Edge Based on Similarity Measure

TheQ
A
B
F metric evaluates the sum of edge information preservation values and is defined by(28)
Q
A
B
F
=
∑
i
=
1
M
∑
j
=
1
N
Q
A
F
i
,
j
×
ω
A
i
,
j
+
Q
B
F
i
,
j
×
ω
B
i
,
j
∑
i
=
1
M
∑
j
=
1
N
ω
A
i
,
j
+
ω
B
i
,
j
-
1
,where Q
A
F
(
i
,
j
)
=
Q
β
A
F
(
i
,
j
)
Q
α
A
F
(
i
,
j
), Q
β
A
F
(
i
,
j
), Q
β
A
F
(
i
,
j
), and Q
α
A
F
(
i
,
j
) are the edge strength and orientation preservation values, respectively, Q
B
F
(
i
,
j
) is similar to Q
β
A
F
(
i
,
j
), and ω
A
(
i
,
j
) and ω
B
(
i
,
j
) are weights to measure the importance of Q
A
F
(
i
,
j
) and Q
B
F
(
i
,
j
), respectively. The dynamic range Q
A
B
F is [
0
,
1
], and it should be close to 1 as far as possible for the best fusion, as Q
A
B
F
=
1. In addition, (
i
,
j
) represents the pixel location, and M and N are the sizes of images, respectively.


## 4.2. Experiment One

For comparison, this paper proposes the fusion scheme of the algorithm and presents several common fusion algorithms; for instance, take the average of the source images pixel by pixel (average), principal component analysis (PCA), pulse coupled neural network (PCNN), pulse coupled neural network and Laplacian pyramid transform (PCNN + LP), and pulse coupled neural network and discrete wavelet transform (PCNN + WT). The fusion images using different methods are shown in Figure6. According the contrast experiments, the fusion image of this paper is better than the others; see Figure 6. The algorithm of this paper does well in extracting the characteristics of the source images, and the fused image is closer to the natural color, which contains more edges, texture, and detail, so it is the closest to the source images. We can conclude that the method in this paper is an effective method.Figure 6
Fusion images from different methods. (a) Average method. (b) PCA method. (c) PCNN method. (d) PCNN + WT. (e) PCNN + LP. (f) This paper.
(a)
(b)
(c)
(d)
(e)
(f)From Table1, the fusion image of this paper contains much more information. SF, AG, MV, and Q
A
B
F of this paper are larger than other methods, EN and MI are slightly better than other methods, and only SD is less than other methods.


## 4.3. Experiment Two

In other color spaces such as NTSC, YUV, YCbCr, HIS, and LAB, all have the similar histogram distribution as the distribution of H, S, and V components in HSV color space, so all can use the proposed algorithm for color image fusion. In order to confirm the method that works best under the color space of fusion, we carry out the following experiments. The fusion images under different color space are shown in Figure7; all can achieve the goal of the image fusion, but the fusion effects on HSV, HIS, and LAB color spaces are better than others.Figure 7
Fusion images under different color spaces. (a) NTSC. (b) YUV. (c) YCbCr. (d) HSV. (e) HIS. (f) LAB.
(a)
(b)
(c)
(d)
(e)
(f)From Table2, it is indicated that the fusion image of this paper in LAB color space contains more information, and SF, AG, MI, and Q
A
B
F are better than others. Overall, the evaluation indexes in HSV, HIS, and LAB are better than others. In practical application, we suggest that the color image fusion of the proposed algorithm should focus on HSV, HIS, and LAB color spaces.


## 4.4. Experiment Three

More experimental results of coastal images in HSV color space are shown in Figure8, the source images with different focus position, and there are a lot of textures in the source images.Figure 8
The fusion results of the coastal images. (a) Source imageA. (b) Source image B. (c) Fusion image. (d) Source image A. (e) Source image B. (f) Fusion image. (g) Source image A. (h) Source image B. (i) Fusion image. (j) Source image A. (k) Source image B. (l) Fusion image.
(a)
(b)
(c)
(d)
(e)
(f)
(g)
(h)
(i)
(j)
(k)
(l)It can be seen in Figure8 that the edge of the fusion image is clear, and it retains most of the textures in the source images; besides, the details are also well preserved. This method can extract the main features from the source images; it shows that the method in this paper also achieved effective results in these groups of color images.The same conclusion as in Figure6 can be concluded. Overall, the method presented in this paper is better than the traditional methods obviously. Compared with the other methods, this method reflects better performance and visual effect in terms of definition and detail.


## 5. Conclusions

We propose an effective multisensor color image fusion algorithm based on NSST and PCNN. PCNN’s neurons capture character will cause the similar brightness of the surrounding neurons to capture the ignition. This character can be automatically coupled and transmitted information. Nonsubsampled Laplacian pyramid filters are used in NSST to remove the influence of upsampling and subsampling. NSST has excellent performance in terms of shift-invariance, multiscale, and multidirectional properties. In the proposed algorithm, RGB color image is converted into HSV color image. H component is fused by adaptive S-PCNN; S and V components are decomposed into different frequency subbands according to different scales and direction by NSST and are fused by different rules. The experimental results show that the proposed color image fusion algorithm can fuse different focus position of the color images, and the fused image contains more information about color, texture, and detail. Compared with the traditional algorithms, this method embodies better fusion performance in many aspects. The paper also discusses the effect of the proposed algorithm on other color spaces, and the experiments show that the algorithm achieved better effects under HSV, HIS, and LAB color spaces, and we recommend three kinds of color spaces as the practical application color space.



---

*Source: 8359602-2015-12-28.xml*