# Infrared and Visible Image Fusion Combining Interesting Region Detection and Nonsubsampled Contourlet Transform

Authors: Kangjian He; Dongming Zhou; Xuejie Zhang; Rencan Nie
Journal: Journal of Sensors
 (2018)

---

## Abstract

The most fundamental purpose of infrared (IR) and visible (VI) image fusion is to integrate the useful information and produce a new image which has higher reliability and understandability for human or computer vision. In order to better preserve the interesting region and its corresponding detail information, a novel multiscale fusion scheme based on interesting region detection is proposed in this paper. Firstly, the MeanShift is used to detect the interesting region with the salient objects and the background region of IR and VI. Then the interesting regions are processed by the guided filter. Next, the nonsubsampled contourlet transform (NSCT) is used for background region decomposition of IR and VI to get a low-frequency and a series of high-frequency layers. An improved weighted average method based on per-pixel weighted average is used to fuse the low-frequency layer. The pulse-coupled neural network (PCNN) is used to fuse each high-frequency layer. Finally, the fused image is obtained by fusing the fused interesting region and the fused background region. Experimental results demonstrate that the proposed algorithm can integrate more background details as well as highlight the interesting region with the salient objects, which is superior to the conventional methods in objective quality evaluations and visual inspection.


---

## Body

## 1. Introduction

Image fusion is an important branch of information science, which has been widely used in many fields, such as bioinformatics, medical image processing, and military target visualization. Especially in military field, infrared (IR) and visible (VI) image fusion is important to military science technology, such as automatic military target detection and localization. As a hot image fusion field, it has attracted the attention of many researchers [1–7]. The key problem of IR and VI image fusion is to integrate and extract the feature information of the source images to produce a new image which is more reliable and understandable, and the fused image not only has the detailed texture information of VI image but also can highlight the target area in an IR image.There are many different algorithms for the IR and VI image fusion that have been proposed and developed over the past few decades. The early fusion methods such as intensity-hue-saturation (IHS) and principal component analysis (PCA) were to process pixel values on spatial domain, which were traditional classical methods, but the fusion effect was limited compared with other excellent fusion methods [8–10]. Many fusion methods based on multiscale transform (MST) have become popular in recent years, such as Laplacian pyramid (LP), wavelet transform (WT), discrete wavelet transform (DWT), and nonsubsampled contourlet transform (NSCT) [11–16]. Due to the excellent characteristics of the multiscale decomposition method, the MST-based method could get a good fusion effect compared with early fusion methods, such as NSCT-PCNN [17]. However, these methods usually failed to highlight the target information in the fused image. IR image target detection-based method is another popular IR and VI image fusion method; these methods detected the target region of the IR image firstly, then fused the background regions using other methods to get the fused background image, and finally fused the target region and background regions directly to get a new image. The advantages of these methods can fully retain the infrared target information in the fused image, but commonly, these infrared target regions of the fused image will lack the corresponding detail information in the VI image. Our previous work proposed a fusion algorithm which was based on target extraction; it was useful to highlight the target in the infrared image due to the target region which was directly fused into the final image [8]. Taking into account the shortcomings of these algorithms, in order to overcome these problems, a novel IR and VI image fusion method is proposed in this paper. Compared with our previous work, we improved the accuracy of interesting region detection where it contains highlighting target and heat sources. In addition, in order to enrich the visible information in the interesting region, we also adopt fusion strategy to fuse them.The first step of the proposed method is to detect the interesting region which contains significant target in the IR image by the MeanShift method. The MeanShift has many applications, such as clustering, discontinuity preserving smoothing, object contour detection, image segmentation, and nonrigid object tracking [15, 18]. We will use it to detect the interesting region with a significant infrared target from the background regions in the IR image. In order to fully retain and highlight the interesting region and significant target information in the fused image, the interesting region will be taken as a separate component and directly fused into the finally image. But the interesting region extracted from the IR image will lose the details of the corresponding region of the VI image. To solve this problem, we use the guided filter to fuse the interesting regions of IR and its corresponding VI image [19–21]. The interesting region of the IR image serves as the guidance image and the interesting region of VI image as the input image. The guided image filter was proposed in 2013 by He et al. [19]; the guided filter has many good characteristics, such as edge-preserving and image smoothing. So we use it to preserve the edge of the VI image; the produced interesting region contains the significant target information as well as the detail information.Next, the background regions will be decomposed by nonsubsampled contourlet transform to get a low-frequency and a series of high-frequency layers. NSCT as an effective decomposition tool was proposed by Da Cunha et al. [16]. NSCT has many good properties of time-frequency localization, multidirection, and multiscale; therefore, it has been widely used in image fusion compared with other multiscale-based methods [22–24]. For the low-frequency layer, we proposed an improved weighted average method based on per-pixel weighted average. Due to the characteristics of the low-frequency layer (hazy image), the per-pixel weighted average based method is effective and will be detailedly described in the fusion rule section. For the high-frequency layers, the pulse-coupled neural network (PCNN) will be used to process each high-frequency layer. PCNN was proposed by Eckhorn et al. [25]. Since it was introduced, it has been widely used in the field of image processing, such as image segmentation, image enhancement, image edge detection, and image fusion [26, 27]. In the proposed method, the spatial frequency (SF) metric of the high-frequency layers will be used as external incentive information of the PCNN model, which makes it better to deal with overexposed or weak exposure images and make the fusion result more suitable for human visual inspection.The remaining sections of this paper are organized as follows: the related work and proposed methods are introduced in Section2, including the interesting region detection and fusion, the background region fusion, and concrete fusion steps. Experimental result comparisons and analysis are given in Section 3. The conclusions are shown in Section 4.


## 2. Related Work and Proposed Methods

### 2.1. Related Work

#### 2.1.1. MeanShift Algorithm

The most important function of the MeanShift is as a tool for computing probability density function in a set of data samples [28]. It has been widely used in discontinuity preserving smoothing, object contour detection, and image segmentation.Given a finite number of data pointsx1,…,xn in the d-dimensional space Rd, a multivariate kernel density function is defined as
(1)Px=1n∑i=1nKx−xi,where
(2)Kx=c∏i=1dkxi.With the kernelK(x) being a bounded function with the following properties,
(3)∫RdKxdx=1,∫RdxKxdx=0,limx→∞xdKx=0,∫RdxxTKxdx=cI,where c is a constant and those of equations mean normalized, symmetric, and exponential weight decay, respectively. The normal kernel K(x) is computed by
(4)KNx=c×exp−12x2.Estimate the kernel density gradient by(5)∇Px=1n∑i=1n∇Kx−xi,and using the normal kernel form, (5) can be rewritten as
(6)∇Px=cn∑i=1n∇ki=cn∑i=1ngi·∑i=1nxigi∑i=1ngi−x,where gx=−k′x. We use the MeanShift to process the IR image, cluster the infrared target pixels, and obtain the interesting region of IR. For image clustering and segmentation, we treat the image as data points in the spatial and gray level domains; two radially symmetric kernels will be used which are defined as follows:
(7)Kx=C⋅ksxshs⋅krxrhr,where xs is the spatial coordinate, xr is the range of a feature vector in color space, and hs and hr are the employed kernel bandwidths. An example of the interesting region detection of IR with different bandwidths is given in Figure 1.Figure 1
Interesting region detection with different bandwidths.We can see from Figure1 that the MeanShift method can effectively extract the IR image region which we interested in and the infrared target information accurately. Compared with IR, the interesting region of VI contains more detailed information. In order to highlight the interesting region of IR and enrich the details of the corresponding region of VI in the fused image, when the interesting regions of IR and VI are determined, the guided filter is used to fuse ones.


#### 2.1.2. Guided Filter

The guided filter is an edge-preserving filter and can compute the filter output by considering the content of the guidance image. There are many good characteristics of the guided filter, especially in edge detail preservation [18, 19, 29]. The filtered output image is very similar to the input image, and it also contains both texture and detail information of the guidance image, as shown in Figure 2.Figure 2
Two examples of the guided filter.
(a)
Guidance image
(b)
Input image
(c)
Filtered imageSupposing that the guidance image isI, the detail description of the guided filter is given as follows:
(8)Oi=akIi+bk,∀i∈ωk,where O is the linear transformation of I, ωk is a local window, in which the pixel k is the center, and the coefficients ak and bk are constant, to make the input image and the output image as similar as possible; we minimize the variance between the output image O and the input image P as follows:
(9)Eak,bk=∑i∈ωkakIi+bk−P2+εak2,where ε is the regularization parameter and ak and bk are computed by
(10)ak=x1/ω∑i∈ωkIiPi−μkP¯Kδk+ε,bk=P¯K−akμk,where ak is the mean and bk is the variance of the local window ωk in the image I, ω is the total number of pixels in the local window ωk, and P¯K is the mean of the input image P in the local window ωk. Figure 2 shows a set of examples of the guided filter.It can be seen from Figure2 that the guidance image contains a large number of detail and texture information. And the input image just contains significant regional information but lacks of detail, texture, and edge information. As can be seen from Figure 2(c), the output image of the guided filter is consistent with the input image, but it only contains detailed texture information in the corresponding region of the guidance image. This is also suitable to process the saliency target in the IR image and its corresponding region in the VI image. Through the guided filter, we can fuse the detail information into the interesting region in the IR image; in this way, the produced new interesting region contains both salient object and detail information.


#### 2.1.3. Nonsubsampled Contourlet Transform (NSCT)

NSCT is a new two-dimensional image decomposition and analysis tool, which is derived from contourlet transform (CT) [16]. The construction of NSCT contains nonsubsampled pyramid filter banks (NSPFB) and nonsubsampled directional filter banks (NSDFB), which are shown in Figure 3.Figure 3
The construction of NSCT.
(a)
The decomposition framework of NSCT
(b)
Ideal frequency partitioningIt can be seen from Figure3 that the source image can be decomposed by NSCT to get a low-frequency layer and a series of high-frequency layers. All obtained layers are the same size with the source image. Figure 4 shows an example of NSCT decomposition. In Figure 4, we decompose the source image into four layers, each of which is decomposed into four images in four directions; we select two images from each layer as shown in Figure 4. Figure 4(b) is the low-frequency layer; it can be seen that it contains only the low frequency information of the source image without high frequency details. From level 1 to level 4 are the high-frequency layers, which show the detail information from different levels to different levels.Figure 4
An example of NSCT decomposition. (a) Source image. (b) Low-frequency layer. (c)-(d) Level 1. (e)-(f) Level 2. (g)-(h) Level 3. (i)-(j) Level 4.
(a)
(b)
(c)
(d)
(e)
(f)
(g)
(h)
(i)
(j)


#### 2.1.4. Pulse-Coupled Neural Network (PCNN)

The PCNN is a single-layered artificial neural network [25]. A basic neuron of PCNN contains the receptive field, the modulation field, and the pulse generator, which are shown in Figure 5.Figure 5
The typical structure of PCNN model.The receptive field of PCNN can be described in detail as follows:(11)Fijn=VF∑k,lWkjYkln−1+Fijne−αF+Sij,Lijn=e−αLLijn+VL∑klMkjYijkln−1,where Sij is the input stimulus at pixel (i, j) of the source image, Fij is the feeding input of it, matrices W and B are the constant synaptic weight, αF and αL are the time constants, and VF and VL are normalizing constants.In modulation field, the internal state is controlled by linking strengthβ, which is given by
(12)Uijn=Fijn1+βLijn,where Uij is the internal state of the neuron, which is created by modulating the feeding and linking channels.The pulse generator field can be described as(13)θijn=e−αθθijn−1+VTYijn−1,(14)Yijn=1,Uijn>θijn,0,otherwise,where Yij is the output of input Sij and θij is the dynamic threshold of the neuron, which is used to compare with Uij. It can be seen from (14), if Uij is larger than θij, the output Yij of the neuron at (i, j) is 1, which we call the neuron is fired. The time matrix T of the neuron fired can be described as follows:
(15)Tijn=n,ifYijn=1,forthefirsttime,Tijn−1,otherwise.


### 2.2. Proposed Method

The proposed fusion algorithm framework is depicted in Figure6. The first step in the proposed method is to detect the interesting region which contains the significant target areas and then fuse the interesting regions of the IR and VI image. In our method, the MeanShift and guided filter are used to perform the first step in our algorithm.Figure 6
Schematic diagram of the proposed fusion method.The background region is obtained by removing the interesting region from the source image. For the background region, the multiscale transform-based method is used to process it. Firstly, nonsubsampled contourlet transform (NSCT) is used to decompose the background region of two source images and then to get a low-frequency layer and a series of high-frequency layers for each image. Next, we use an improved weighted average method based on per-pixel weighted average and pulse-coupled neural network (PCNN) to process the low-frequency and high-frequency layers, respectively.


#### 2.2.1. Low-Frequency Layer Fusion Rules

In nature images, low-frequency information is the main component of an image; on the contrary, high-frequency information contains the details of the image [30]. It can be seen in Figure 6 that, compared with the image B1 and B2, the low-frequency layers L1 and L2 are the main components without the details. Most low-frequency layer fusion methods are weighted averaging based methods, which do not consider the membership relationship between pixels and only weigh the independent pixel values. These methods cannot fully fuse the details of the low-frequency layers. In order to have a better fusion effect, we proposed an improved weighted average method which is based on per-pixel weighted average, which can be described as follows:
(16)CFLi,j=wAi,j×CALi,j+1−wAi,j×CBLi,j,where CFLi,j denotes the final result of the low-frequency layer, CALi,j is the low-frequency layer of the background region in the source IR image A, and CBLi,j is the low-frequency layer of the background region in source VI image B.
(17)wAi,j=exp−CBLi,j−μ22τσ2,where μ and σ are the mean and variance of the background regions in source VI image B and τ is the adjustment factor of Gaussian function. The Gaussian function curve and an example are shown in Figure 7. In the proposed method, we set τ=1. It can be seen in Figure 7(d) that, after the source image is processed by per-pixel weighted average, only the low-frequency information of the source image is reserved; to some extent, it is a low pass filter, and similar as Figure 4(b), the low-frequency layer is obtained by NSCT. Therefore, it is effective to process the low-frequency layer by the weighted average method based on per-pixel weighted average.Figure 7
Gaussian function. (a) Gaussian curve with differentτ. (b) Gaussian surface. (c) Source image. (d) Produced image by Gaussian.
(a)
(b)
(c)
(d)


#### 2.2.2. High-Frequency Layer Fusion Rules

From Figure4, it can be seen that most details of information, texture, and edge are included in the high-frequency layers. For the high-frequency layers, PCNN is used in the proposed method. In the modulation field, the linking coefficient β is a key parameter which value can directly affect the weighting of the linking channel. We use the spatial frequency (SF) of the high-frequency layer as the linking coefficient β in our proposed method. In Section 2.1.4, we have analyzed the PCNN model. The spatial frequency (SF) can reflect the overall definition level of an image; the SF of the source image is used to determine the linking strength β, which can be described as follows:
(18)SF=RF2+CF2,where RF is the spatial row frequency and CF is the spatial column frequency, which can be computed by
(19)RF=1M×N∑i=1M∑j=2NFi,j−Fi,j−12,CF=1M×N∑i=1N∑j=2MFi,j−Fi,j−12.The fused high-frequency layerCF,ij can be determined as follows:
(20)CF,ij=CA,ij,TA,ijn≥TB,ijn,CB,ij,TA,ijn<TB,ijn,where TA,ij(n) and TB,ij(n) denote time matrices of each neuron obtained by (15) and CA,ij and CB,ij are the high-frequency layers of the background regions in source IR image A and VI image B.


#### 2.2.3. Fusion Steps

The framework of the proposed method in this paper is shown in detail in Figure6, and the concrete fusion steps are summarized as follows: input: source IR image A and VI image B.
Step 1:
detect the interesting region which contains the salient infrared objects of IR and corresponding VI image by the MeanShift, to get the interesting region and the background regions.
Step 2:
for the interesting regions of the source image, fuse them by the guided filter method which is described in Section2.1.2, to produce the fused interesting region.
Step 3:
perform NSCT in the background regions and then obtain a low-frequency layer and a series of high-frequency layers for each source image.
Step 4:
for the low-frequency layer, an improved weighted average method based on per-pixel weighted average algorithm is used to produce the fused low-frequency layer, which is shown in (16) and (17).
Step 5:
for the high-frequency layers, SF-PCNN-based method is used to produce the fused high-frequency layers, which are described in Section2.2.2 in detail.
Step 6:
the fused background region is produced by NSCT reconstruction.
Step 7:
fuse the interesting region and the fused background region to produce the final fusion image.


## 2.1. Related Work

### 2.1.1. MeanShift Algorithm

The most important function of the MeanShift is as a tool for computing probability density function in a set of data samples [28]. It has been widely used in discontinuity preserving smoothing, object contour detection, and image segmentation.Given a finite number of data pointsx1,…,xn in the d-dimensional space Rd, a multivariate kernel density function is defined as
(1)Px=1n∑i=1nKx−xi,where
(2)Kx=c∏i=1dkxi.With the kernelK(x) being a bounded function with the following properties,
(3)∫RdKxdx=1,∫RdxKxdx=0,limx→∞xdKx=0,∫RdxxTKxdx=cI,where c is a constant and those of equations mean normalized, symmetric, and exponential weight decay, respectively. The normal kernel K(x) is computed by
(4)KNx=c×exp−12x2.Estimate the kernel density gradient by(5)∇Px=1n∑i=1n∇Kx−xi,and using the normal kernel form, (5) can be rewritten as
(6)∇Px=cn∑i=1n∇ki=cn∑i=1ngi·∑i=1nxigi∑i=1ngi−x,where gx=−k′x. We use the MeanShift to process the IR image, cluster the infrared target pixels, and obtain the interesting region of IR. For image clustering and segmentation, we treat the image as data points in the spatial and gray level domains; two radially symmetric kernels will be used which are defined as follows:
(7)Kx=C⋅ksxshs⋅krxrhr,where xs is the spatial coordinate, xr is the range of a feature vector in color space, and hs and hr are the employed kernel bandwidths. An example of the interesting region detection of IR with different bandwidths is given in Figure 1.Figure 1
Interesting region detection with different bandwidths.We can see from Figure1 that the MeanShift method can effectively extract the IR image region which we interested in and the infrared target information accurately. Compared with IR, the interesting region of VI contains more detailed information. In order to highlight the interesting region of IR and enrich the details of the corresponding region of VI in the fused image, when the interesting regions of IR and VI are determined, the guided filter is used to fuse ones.


### 2.1.2. Guided Filter

The guided filter is an edge-preserving filter and can compute the filter output by considering the content of the guidance image. There are many good characteristics of the guided filter, especially in edge detail preservation [18, 19, 29]. The filtered output image is very similar to the input image, and it also contains both texture and detail information of the guidance image, as shown in Figure 2.Figure 2
Two examples of the guided filter.
(a)
Guidance image
(b)
Input image
(c)
Filtered imageSupposing that the guidance image isI, the detail description of the guided filter is given as follows:
(8)Oi=akIi+bk,∀i∈ωk,where O is the linear transformation of I, ωk is a local window, in which the pixel k is the center, and the coefficients ak and bk are constant, to make the input image and the output image as similar as possible; we minimize the variance between the output image O and the input image P as follows:
(9)Eak,bk=∑i∈ωkakIi+bk−P2+εak2,where ε is the regularization parameter and ak and bk are computed by
(10)ak=x1/ω∑i∈ωkIiPi−μkP¯Kδk+ε,bk=P¯K−akμk,where ak is the mean and bk is the variance of the local window ωk in the image I, ω is the total number of pixels in the local window ωk, and P¯K is the mean of the input image P in the local window ωk. Figure 2 shows a set of examples of the guided filter.It can be seen from Figure2 that the guidance image contains a large number of detail and texture information. And the input image just contains significant regional information but lacks of detail, texture, and edge information. As can be seen from Figure 2(c), the output image of the guided filter is consistent with the input image, but it only contains detailed texture information in the corresponding region of the guidance image. This is also suitable to process the saliency target in the IR image and its corresponding region in the VI image. Through the guided filter, we can fuse the detail information into the interesting region in the IR image; in this way, the produced new interesting region contains both salient object and detail information.


### 2.1.3. Nonsubsampled Contourlet Transform (NSCT)

NSCT is a new two-dimensional image decomposition and analysis tool, which is derived from contourlet transform (CT) [16]. The construction of NSCT contains nonsubsampled pyramid filter banks (NSPFB) and nonsubsampled directional filter banks (NSDFB), which are shown in Figure 3.Figure 3
The construction of NSCT.
(a)
The decomposition framework of NSCT
(b)
Ideal frequency partitioningIt can be seen from Figure3 that the source image can be decomposed by NSCT to get a low-frequency layer and a series of high-frequency layers. All obtained layers are the same size with the source image. Figure 4 shows an example of NSCT decomposition. In Figure 4, we decompose the source image into four layers, each of which is decomposed into four images in four directions; we select two images from each layer as shown in Figure 4. Figure 4(b) is the low-frequency layer; it can be seen that it contains only the low frequency information of the source image without high frequency details. From level 1 to level 4 are the high-frequency layers, which show the detail information from different levels to different levels.Figure 4
An example of NSCT decomposition. (a) Source image. (b) Low-frequency layer. (c)-(d) Level 1. (e)-(f) Level 2. (g)-(h) Level 3. (i)-(j) Level 4.
(a)
(b)
(c)
(d)
(e)
(f)
(g)
(h)
(i)
(j)


### 2.1.4. Pulse-Coupled Neural Network (PCNN)

The PCNN is a single-layered artificial neural network [25]. A basic neuron of PCNN contains the receptive field, the modulation field, and the pulse generator, which are shown in Figure 5.Figure 5
The typical structure of PCNN model.The receptive field of PCNN can be described in detail as follows:(11)Fijn=VF∑k,lWkjYkln−1+Fijne−αF+Sij,Lijn=e−αLLijn+VL∑klMkjYijkln−1,where Sij is the input stimulus at pixel (i, j) of the source image, Fij is the feeding input of it, matrices W and B are the constant synaptic weight, αF and αL are the time constants, and VF and VL are normalizing constants.In modulation field, the internal state is controlled by linking strengthβ, which is given by
(12)Uijn=Fijn1+βLijn,where Uij is the internal state of the neuron, which is created by modulating the feeding and linking channels.The pulse generator field can be described as(13)θijn=e−αθθijn−1+VTYijn−1,(14)Yijn=1,Uijn>θijn,0,otherwise,where Yij is the output of input Sij and θij is the dynamic threshold of the neuron, which is used to compare with Uij. It can be seen from (14), if Uij is larger than θij, the output Yij of the neuron at (i, j) is 1, which we call the neuron is fired. The time matrix T of the neuron fired can be described as follows:
(15)Tijn=n,ifYijn=1,forthefirsttime,Tijn−1,otherwise.


## 2.1.1. MeanShift Algorithm

The most important function of the MeanShift is as a tool for computing probability density function in a set of data samples [28]. It has been widely used in discontinuity preserving smoothing, object contour detection, and image segmentation.Given a finite number of data pointsx1,…,xn in the d-dimensional space Rd, a multivariate kernel density function is defined as
(1)Px=1n∑i=1nKx−xi,where
(2)Kx=c∏i=1dkxi.With the kernelK(x) being a bounded function with the following properties,
(3)∫RdKxdx=1,∫RdxKxdx=0,limx→∞xdKx=0,∫RdxxTKxdx=cI,where c is a constant and those of equations mean normalized, symmetric, and exponential weight decay, respectively. The normal kernel K(x) is computed by
(4)KNx=c×exp−12x2.Estimate the kernel density gradient by(5)∇Px=1n∑i=1n∇Kx−xi,and using the normal kernel form, (5) can be rewritten as
(6)∇Px=cn∑i=1n∇ki=cn∑i=1ngi·∑i=1nxigi∑i=1ngi−x,where gx=−k′x. We use the MeanShift to process the IR image, cluster the infrared target pixels, and obtain the interesting region of IR. For image clustering and segmentation, we treat the image as data points in the spatial and gray level domains; two radially symmetric kernels will be used which are defined as follows:
(7)Kx=C⋅ksxshs⋅krxrhr,where xs is the spatial coordinate, xr is the range of a feature vector in color space, and hs and hr are the employed kernel bandwidths. An example of the interesting region detection of IR with different bandwidths is given in Figure 1.Figure 1
Interesting region detection with different bandwidths.We can see from Figure1 that the MeanShift method can effectively extract the IR image region which we interested in and the infrared target information accurately. Compared with IR, the interesting region of VI contains more detailed information. In order to highlight the interesting region of IR and enrich the details of the corresponding region of VI in the fused image, when the interesting regions of IR and VI are determined, the guided filter is used to fuse ones.


## 2.1.2. Guided Filter

The guided filter is an edge-preserving filter and can compute the filter output by considering the content of the guidance image. There are many good characteristics of the guided filter, especially in edge detail preservation [18, 19, 29]. The filtered output image is very similar to the input image, and it also contains both texture and detail information of the guidance image, as shown in Figure 2.Figure 2
Two examples of the guided filter.
(a)
Guidance image
(b)
Input image
(c)
Filtered imageSupposing that the guidance image isI, the detail description of the guided filter is given as follows:
(8)Oi=akIi+bk,∀i∈ωk,where O is the linear transformation of I, ωk is a local window, in which the pixel k is the center, and the coefficients ak and bk are constant, to make the input image and the output image as similar as possible; we minimize the variance between the output image O and the input image P as follows:
(9)Eak,bk=∑i∈ωkakIi+bk−P2+εak2,where ε is the regularization parameter and ak and bk are computed by
(10)ak=x1/ω∑i∈ωkIiPi−μkP¯Kδk+ε,bk=P¯K−akμk,where ak is the mean and bk is the variance of the local window ωk in the image I, ω is the total number of pixels in the local window ωk, and P¯K is the mean of the input image P in the local window ωk. Figure 2 shows a set of examples of the guided filter.It can be seen from Figure2 that the guidance image contains a large number of detail and texture information. And the input image just contains significant regional information but lacks of detail, texture, and edge information. As can be seen from Figure 2(c), the output image of the guided filter is consistent with the input image, but it only contains detailed texture information in the corresponding region of the guidance image. This is also suitable to process the saliency target in the IR image and its corresponding region in the VI image. Through the guided filter, we can fuse the detail information into the interesting region in the IR image; in this way, the produced new interesting region contains both salient object and detail information.


## 2.1.3. Nonsubsampled Contourlet Transform (NSCT)

NSCT is a new two-dimensional image decomposition and analysis tool, which is derived from contourlet transform (CT) [16]. The construction of NSCT contains nonsubsampled pyramid filter banks (NSPFB) and nonsubsampled directional filter banks (NSDFB), which are shown in Figure 3.Figure 3
The construction of NSCT.
(a)
The decomposition framework of NSCT
(b)
Ideal frequency partitioningIt can be seen from Figure3 that the source image can be decomposed by NSCT to get a low-frequency layer and a series of high-frequency layers. All obtained layers are the same size with the source image. Figure 4 shows an example of NSCT decomposition. In Figure 4, we decompose the source image into four layers, each of which is decomposed into four images in four directions; we select two images from each layer as shown in Figure 4. Figure 4(b) is the low-frequency layer; it can be seen that it contains only the low frequency information of the source image without high frequency details. From level 1 to level 4 are the high-frequency layers, which show the detail information from different levels to different levels.Figure 4
An example of NSCT decomposition. (a) Source image. (b) Low-frequency layer. (c)-(d) Level 1. (e)-(f) Level 2. (g)-(h) Level 3. (i)-(j) Level 4.
(a)
(b)
(c)
(d)
(e)
(f)
(g)
(h)
(i)
(j)


## 2.1.4. Pulse-Coupled Neural Network (PCNN)

The PCNN is a single-layered artificial neural network [25]. A basic neuron of PCNN contains the receptive field, the modulation field, and the pulse generator, which are shown in Figure 5.Figure 5
The typical structure of PCNN model.The receptive field of PCNN can be described in detail as follows:(11)Fijn=VF∑k,lWkjYkln−1+Fijne−αF+Sij,Lijn=e−αLLijn+VL∑klMkjYijkln−1,where Sij is the input stimulus at pixel (i, j) of the source image, Fij is the feeding input of it, matrices W and B are the constant synaptic weight, αF and αL are the time constants, and VF and VL are normalizing constants.In modulation field, the internal state is controlled by linking strengthβ, which is given by
(12)Uijn=Fijn1+βLijn,where Uij is the internal state of the neuron, which is created by modulating the feeding and linking channels.The pulse generator field can be described as(13)θijn=e−αθθijn−1+VTYijn−1,(14)Yijn=1,Uijn>θijn,0,otherwise,where Yij is the output of input Sij and θij is the dynamic threshold of the neuron, which is used to compare with Uij. It can be seen from (14), if Uij is larger than θij, the output Yij of the neuron at (i, j) is 1, which we call the neuron is fired. The time matrix T of the neuron fired can be described as follows:
(15)Tijn=n,ifYijn=1,forthefirsttime,Tijn−1,otherwise.


## 2.2. Proposed Method

The proposed fusion algorithm framework is depicted in Figure6. The first step in the proposed method is to detect the interesting region which contains the significant target areas and then fuse the interesting regions of the IR and VI image. In our method, the MeanShift and guided filter are used to perform the first step in our algorithm.Figure 6
Schematic diagram of the proposed fusion method.The background region is obtained by removing the interesting region from the source image. For the background region, the multiscale transform-based method is used to process it. Firstly, nonsubsampled contourlet transform (NSCT) is used to decompose the background region of two source images and then to get a low-frequency layer and a series of high-frequency layers for each image. Next, we use an improved weighted average method based on per-pixel weighted average and pulse-coupled neural network (PCNN) to process the low-frequency and high-frequency layers, respectively.


### 2.2.1. Low-Frequency Layer Fusion Rules

In nature images, low-frequency information is the main component of an image; on the contrary, high-frequency information contains the details of the image [30]. It can be seen in Figure 6 that, compared with the image B1 and B2, the low-frequency layers L1 and L2 are the main components without the details. Most low-frequency layer fusion methods are weighted averaging based methods, which do not consider the membership relationship between pixels and only weigh the independent pixel values. These methods cannot fully fuse the details of the low-frequency layers. In order to have a better fusion effect, we proposed an improved weighted average method which is based on per-pixel weighted average, which can be described as follows:
(16)CFLi,j=wAi,j×CALi,j+1−wAi,j×CBLi,j,where CFLi,j denotes the final result of the low-frequency layer, CALi,j is the low-frequency layer of the background region in the source IR image A, and CBLi,j is the low-frequency layer of the background region in source VI image B.
(17)wAi,j=exp−CBLi,j−μ22τσ2,where μ and σ are the mean and variance of the background regions in source VI image B and τ is the adjustment factor of Gaussian function. The Gaussian function curve and an example are shown in Figure 7. In the proposed method, we set τ=1. It can be seen in Figure 7(d) that, after the source image is processed by per-pixel weighted average, only the low-frequency information of the source image is reserved; to some extent, it is a low pass filter, and similar as Figure 4(b), the low-frequency layer is obtained by NSCT. Therefore, it is effective to process the low-frequency layer by the weighted average method based on per-pixel weighted average.Figure 7
Gaussian function. (a) Gaussian curve with differentτ. (b) Gaussian surface. (c) Source image. (d) Produced image by Gaussian.
(a)
(b)
(c)
(d)


### 2.2.2. High-Frequency Layer Fusion Rules

From Figure4, it can be seen that most details of information, texture, and edge are included in the high-frequency layers. For the high-frequency layers, PCNN is used in the proposed method. In the modulation field, the linking coefficient β is a key parameter which value can directly affect the weighting of the linking channel. We use the spatial frequency (SF) of the high-frequency layer as the linking coefficient β in our proposed method. In Section 2.1.4, we have analyzed the PCNN model. The spatial frequency (SF) can reflect the overall definition level of an image; the SF of the source image is used to determine the linking strength β, which can be described as follows:
(18)SF=RF2+CF2,where RF is the spatial row frequency and CF is the spatial column frequency, which can be computed by
(19)RF=1M×N∑i=1M∑j=2NFi,j−Fi,j−12,CF=1M×N∑i=1N∑j=2MFi,j−Fi,j−12.The fused high-frequency layerCF,ij can be determined as follows:
(20)CF,ij=CA,ij,TA,ijn≥TB,ijn,CB,ij,TA,ijn<TB,ijn,where TA,ij(n) and TB,ij(n) denote time matrices of each neuron obtained by (15) and CA,ij and CB,ij are the high-frequency layers of the background regions in source IR image A and VI image B.


### 2.2.3. Fusion Steps

The framework of the proposed method in this paper is shown in detail in Figure6, and the concrete fusion steps are summarized as follows: input: source IR image A and VI image B.
Step 1:
detect the interesting region which contains the salient infrared objects of IR and corresponding VI image by the MeanShift, to get the interesting region and the background regions.
Step 2:
for the interesting regions of the source image, fuse them by the guided filter method which is described in Section2.1.2, to produce the fused interesting region.
Step 3:
perform NSCT in the background regions and then obtain a low-frequency layer and a series of high-frequency layers for each source image.
Step 4:
for the low-frequency layer, an improved weighted average method based on per-pixel weighted average algorithm is used to produce the fused low-frequency layer, which is shown in (16) and (17).
Step 5:
for the high-frequency layers, SF-PCNN-based method is used to produce the fused high-frequency layers, which are described in Section2.2.2 in detail.
Step 6:
the fused background region is produced by NSCT reconstruction.
Step 7:
fuse the interesting region and the fused background region to produce the final fusion image.


## 2.2.1. Low-Frequency Layer Fusion Rules

In nature images, low-frequency information is the main component of an image; on the contrary, high-frequency information contains the details of the image [30]. It can be seen in Figure 6 that, compared with the image B1 and B2, the low-frequency layers L1 and L2 are the main components without the details. Most low-frequency layer fusion methods are weighted averaging based methods, which do not consider the membership relationship between pixels and only weigh the independent pixel values. These methods cannot fully fuse the details of the low-frequency layers. In order to have a better fusion effect, we proposed an improved weighted average method which is based on per-pixel weighted average, which can be described as follows:
(16)CFLi,j=wAi,j×CALi,j+1−wAi,j×CBLi,j,where CFLi,j denotes the final result of the low-frequency layer, CALi,j is the low-frequency layer of the background region in the source IR image A, and CBLi,j is the low-frequency layer of the background region in source VI image B.
(17)wAi,j=exp−CBLi,j−μ22τσ2,where μ and σ are the mean and variance of the background regions in source VI image B and τ is the adjustment factor of Gaussian function. The Gaussian function curve and an example are shown in Figure 7. In the proposed method, we set τ=1. It can be seen in Figure 7(d) that, after the source image is processed by per-pixel weighted average, only the low-frequency information of the source image is reserved; to some extent, it is a low pass filter, and similar as Figure 4(b), the low-frequency layer is obtained by NSCT. Therefore, it is effective to process the low-frequency layer by the weighted average method based on per-pixel weighted average.Figure 7
Gaussian function. (a) Gaussian curve with differentτ. (b) Gaussian surface. (c) Source image. (d) Produced image by Gaussian.
(a)
(b)
(c)
(d)


## 2.2.2. High-Frequency Layer Fusion Rules

From Figure4, it can be seen that most details of information, texture, and edge are included in the high-frequency layers. For the high-frequency layers, PCNN is used in the proposed method. In the modulation field, the linking coefficient β is a key parameter which value can directly affect the weighting of the linking channel. We use the spatial frequency (SF) of the high-frequency layer as the linking coefficient β in our proposed method. In Section 2.1.4, we have analyzed the PCNN model. The spatial frequency (SF) can reflect the overall definition level of an image; the SF of the source image is used to determine the linking strength β, which can be described as follows:
(18)SF=RF2+CF2,where RF is the spatial row frequency and CF is the spatial column frequency, which can be computed by
(19)RF=1M×N∑i=1M∑j=2NFi,j−Fi,j−12,CF=1M×N∑i=1N∑j=2MFi,j−Fi,j−12.The fused high-frequency layerCF,ij can be determined as follows:
(20)CF,ij=CA,ij,TA,ijn≥TB,ijn,CB,ij,TA,ijn<TB,ijn,where TA,ij(n) and TB,ij(n) denote time matrices of each neuron obtained by (15) and CA,ij and CB,ij are the high-frequency layers of the background regions in source IR image A and VI image B.


## 2.2.3. Fusion Steps

The framework of the proposed method in this paper is shown in detail in Figure6, and the concrete fusion steps are summarized as follows: input: source IR image A and VI image B.
Step 1:
detect the interesting region which contains the salient infrared objects of IR and corresponding VI image by the MeanShift, to get the interesting region and the background regions.
Step 2:
for the interesting regions of the source image, fuse them by the guided filter method which is described in Section2.1.2, to produce the fused interesting region.
Step 3:
perform NSCT in the background regions and then obtain a low-frequency layer and a series of high-frequency layers for each source image.
Step 4:
for the low-frequency layer, an improved weighted average method based on per-pixel weighted average algorithm is used to produce the fused low-frequency layer, which is shown in (16) and (17).
Step 5:
for the high-frequency layers, SF-PCNN-based method is used to produce the fused high-frequency layers, which are described in Section2.2.2 in detail.
Step 6:
the fused background region is produced by NSCT reconstruction.
Step 7:
fuse the interesting region and the fused background region to produce the final fusion image.


## 3. Experimental Results and Analysis

In order to illustrate the effectiveness of the proposed fusion algorithm, several groups of IR and VI images fusion experiments will be described in detail in this section. These images are available athttp://figshare.com/articles/TNO_Image_Fusion_Dataset/1008029. All simulations are conducted in MATLAB 2014a, on an Intel(R) Core (TM) i5-6400 @2.7GHz PC with 16GB RAM. Firstly, the experimental parameter setting is introduced; then the discussion of fusion results compared with other methods will be given.


### 3.1. Experimental Introduction

To show the improvement of the proposed method, the fusion results of “Jeep” by the proposed method and the method of [8] are shown in Figure 8. In [8], the target region was directly fused into the final image to highlight the target in an infrared image. And in this paper, we improve the accuracy of interesting region detection where it contains highlighting target and heat sources. In addition, we integrate the interesting regions of VI and IR for enriching the visible information in the interesting region. As shown in Figure 8 and Figure 9, the fusion result by the proposed method contains rich details while highlighting the target of IR image.Figure 8
Experiment with “Jeep.”
(a)
Visible image jeep
(b)
Infrared image jeep
(c)
Fused Jeep by [8]
(d)
Fused jeep by the proposed methodFigure 9
Detail with enlarged scale “Jeep.”
(a)
Visible image jeep
(b)
Infrared image jeep
(c)
Fused by [8]
(d)
Fused by the proposed methodThe proposed method will be compared with eight current fusion methods: principal component analysis- (PCA-) based method [10], discrete wavelet transform- (DWT-) based method [11], PCNN-based method [15], NSCT-based method [23], Laplacian pyramid transform- (LP-) PCNN-based method [14], NSCT-PCNN-based method [17], IFM based method [31], and MWGF-based method [32]. In all PCNN-based method experiments, through a large number of verification and comparison in experiments, the parameters of PCNN are set as αθ=0.2, αL=0.05, VL=0.02, Vθ=40, N=200, and M=W=0.707,1,0.707;1,0,1;0.707,1,0.707, where N is the number of iterations of PCNN. All NSCT-based methods, “pkva,” and “9–7” are set as the pyramid and the direction filter. For all multiscale decomposition methods, the decomposition level is set to 3, and “averaging” is used to fuse the low-frequency layer, and the high-frequency layers are fused by “absolute maximum choosing.”In order to evaluate the fusion results with different methods objectively, three most commonly used objective indicators will be used as the evaluation index: mutual information (MI), pixel of visual information (VIF), and edge gradient operator (QAB/F). MI is used to measure the amount of the source images’ information retained in the fused image. VIF is an evaluation index for human visual system, which is based on natural scenes and image distortion [33]. QAB/F is used to measure the edge information based on edge strength and orientation preserving from the source images. Commonly, the greater value of these evaluation metrics indicates that the fused image has a better quality [34].


### 3.2. Fusion Results and Discussions

The experimental images consisted of six pairs of IR and VI images, which are shown in Figure10. The first line in Figure 10 is VI images, and the second line is IR images. A large number of details and texture are included in the VI images, while the IR image contains only significant information.Figure 10
Experimental images. (a) “Sand path.” (b) “Bristol Queens Road.” (c) “UN Camp.” (d) “Trees.” (e)“Jeep.” (f) “Kaptein.”
(a)
(b)
(c)
(d)
(e)
(f)The fusion results obtained by different fusion algorithms of “Sand path” are given in Figure11. Figure 11(a) is the fused image by PCA, Figure 11(b) is the fused image by DWT, Figure 11(c) is the fused image by PCNN, Figure 11(d) is the fused image by NSCT, Figure 11(e) is the fused image by LP-NSCT, Figure 11(f) is the fused image by PCNN-NSCT, Figure 11(g) is the fused image by IFM, Figure 11(h) is the fused image by MWGF, and Figure 11(i) is the fused image by the proposed method. From Figure 11, we can see that the fused image by the proposed method contains more detail information of the VI image, as well as the highlighted infrared target information compared with other methods. In addition, the fused image by our method has advantages in visual effects and it is also superior to other algorithms in objective evaluation, which are shown in Table 1.Figure 11
Experimental results of “Sand path”. (a) PCA. (b) DWT. (c) PCNN. (d) NSCT. (e) LP-PCNN. (f) NSCT-PCNN. (g) IFM. (h) MWGF. (i) The proposed method.
(a)
(b)
(c)
(d)
(e)
(f)
(g)
(h)
(i)Table 1
Objective results for various fusion results.
Evaluation
Experiment
Fusion methods
PCA
DWT
PCNN
NSCT
LP-PCNN
NSCT-PCNN
IFM
MWGF
Reference [8]
Proposed
MI
Sand path
1.1117
0.5578
1.4473
1.5081
0.5468
1.0629
1.6325
1.7830
1.7935
1.8874
Bristol Queens Road
2.4788
1.2387
2.3390
1.9813
1.1924
2.2035
1.6665
2.5665
2.3513
2.6238
UN Camp
2.1191
0.7973
2.0342
1.5648
0.8595
1.0644
2.7810
2.9160
2.3461
2.1219
Trees
1.3114
0.8653
0.8844
1.1424
0.7757
1.3761
1.0500
1.3480
1.3418
1.3598
Jeep
2.1219
1.0000
3.1090
2.4130
1.9150
2.4980
1.8704
2.9550
2.9721
2.9613
Kaptein
2.0190
1.0910
2.5090
1.8650
1.1960
1.7100
3.469
3.3830
3.2045
3.3921
Average MI
1.8603
0.9250
2.0538
1.7457
1.0809
1.6524
2.0782
2.4919
2.3349
2.3910
VIF
Sand path
0.4201
0.2566
0.3048
0.3284
0.2788
0.3886
0.3599
0.4813
0.4821
0.5230
Bristol Queens Road
0.4991
0.3797
0.4986
0.4906
0.4687
0.5196
0.4947
0.5715
0.4949
0.5202
UN Camp
0.4599
0.2899
0.3107
0.3538
0.3156
0.3743
0.4419
0.4937
0.3873
0.4656
Trees
0.3776
0.3238
0.3145
0.4128
0.3648
0.4372
0.5141
0.5013
0.5243
0.4311
Jeep
0.4575
0.4100
0.5440
0.4841
0.5544
0.4788
0.4572
0.6023
0.5528
0.6107
Kaptein
0.4182
0.3637
0.4332
0.4488
0.4308
0.4359
0.4871
0.5189
0.4947
0.5103
Average VIF
0.4387
0.3372
0.4009
0.4197
0.4021
0.4390
0.4591
0.5281
0.4894
0.5101
QAB/F
Sand path
0.5013
0.3934
0.4188
0.4331
0.4136
0.5218
0.5104
0.5161
0.5072
0.5295
Bristol Queens Road
0.5678
0.3841
0.6708
0.5963
0.5889
0.6661
0.6066
0.6746
0.6693
0.6719
UN Camp
0.5038
0.3442
0.4785
0.4169
0.4356
0.4650
0.5265
0.5262
0.5296
0.5330
Trees
0.4533
0.3909
0.3804
0.4306
0.4476
0.4423
0.6004
0.5815
0.4832
0.4524
Jeep
0.4252
0.3436
0.6453
0.3022
0.5467
0.3142
0.3892
0.6769
0.6348
0.6818
Kaptein
0.3249
0.3442
0.5280
0.3692
0.4759
0.4240
0.5853
0.6109
0.6118
0.6189
AverageQAB/F
0.4627
0.3667
0.5203
0.4247
0.4847
0.4722
0.5364
0.5977
0.5726
0.5813In order to illustrate the applicability of the proposed method, other groups of experiments are performed, which are given as follows.It can be seen from Figure12 and Figure 13 that the proposed method has more advantages in detail information integration. In order to better reflect their differences of the fused images obtained by different fusion methods, Figure 13 shows the detail with an enlarged scale of Figure 12. In Figure 13(i), the red frame region is more suitable for human visual system, with more visible detail information. Compared with the same position in two source images, the fused region has higher readability and reliability.Figure 12
Second example experimental results of “Bristol Queens Road.” (a) PCA. (b) DWT. (c) PCNN. (d) NSCT. (e) LP-PCNN. (f) NSCT-PCNN. (g) IFM. (h) MWGF. (i) The proposed method.
(a)
(b)
(c)
(d)
(e)
(f)
(g)
(h)
(i)Figure 13
Detail with enlarged scale. (a) PCA. (b) DWT. (c) PCNN. (d) NSCT. (e) LP-PCNN. (f) NSCT-PCNN. (g) IFM. (h) MWGF. (i) The proposed method.
(a)
(b)
(c)
(d)
(e)
(f)
(g)
(h)
(i)Figure14 and Figure 15 are the third and fourth groups of experiment of “UN Camp” and “Trees,” respectively. The objective evaluation matrices are given in Table 1. In order to reflect more directly out of their difference, line chart comparison of MI, VIF, and QAB/F values of the experiments is given in Figure 18.Figure 14
Experimental results of “UN Camp.” (a) PCA. (b) DWT. (c) PCNN. (d) NSCT. (e) LP-PCNN. (f) NSCT-PCNN. (g) IFM. (h) MWGF. (i) The proposed method.
(a)
(b)
(c)
(d)
(e)
(f)
(g)
(h)
(i)Figure 15
Experimental results of “Trees.” (a) PCA. (b) DWT. (c) PCNN. (d) NSCT. (e) LP-PCNN. (f) NSCT-PCNN. (g) IFM. (h) MWGF. (i) The proposed method.
(a)
(b)
(c)
(d)
(e)
(f)
(g)
(h)
(i)Figure16 and Figure 17 are the experimental results of “Jeep” and “Kaptein.” The objective evaluation matrices are given in Table 1. Line chart comparison of MI, VIF, and QAB/F values of the experiments is given in Figure 19. It can be seen that the fusion results of the proposed method have a better visual effect. Compared with the same position in two source images, the fused region has higher readability and reliability.Figure 16
Experimental results of “Jeep.” (a) PCA. (b) DWT. (c) PCNN. (d) NSCT. (e) LP-PCNN. (f) NSCT-PCNN. (g) IFM. (h) MWGF. (i) The proposed method.
(a)
(b)
(c)
(d)
(e)
(f)
(g)
(h)
(i)Figure 17
Experimental results of “Kaptein.” (a) PCA. (b) DWT. (c) PCNN. (d) NSCT. (e) LP-PCNN. (f) NSCT-PCNN. (g) IFM. (h) MWGF. (i) The proposed method.
(a)
(b)
(c)
(d)
(e)
(f)
(g)
(h)
(i)All fusion results by the proposed method and the method of [8] are shown in Figure 18. The first line in Figure 18 is the results of [8], and the second line is the results of the proposed method. The objective evaluation matrices are given in Table 1. It can be seen from Figure 18 and Table 1 that the proposed method has better results in objective evaluation and visual effect. The average values of MI, VIF, and QAB/F for six pairs of IR and VI images are listed in Table 1. It can be seen that the average value of the proposed method is only less than MWGF and greater than the other comparison algorithms, because the MWGF is more effective for some special images, so as to increase the average value, such as the MI of “UN Camp”, but the proposed method has higher evaluation metrics on most images than MWGF.Figure 18
Experimental results by [8] and the proposed method. (a) “Sand path.” (b) “Bristol Queens Road.” (c) “UN Camp.” (d) “Trees.” (e)“Jeep.” (f) “Kaptein.”
(a)
(b)
(c)
(d)
(e)
(f)Figure 19
Line chart comparison of MI, VIF, andQAB/F values for four groups of the experiments.
(a)
(b)
(c)


## 3.1. Experimental Introduction

To show the improvement of the proposed method, the fusion results of “Jeep” by the proposed method and the method of [8] are shown in Figure 8. In [8], the target region was directly fused into the final image to highlight the target in an infrared image. And in this paper, we improve the accuracy of interesting region detection where it contains highlighting target and heat sources. In addition, we integrate the interesting regions of VI and IR for enriching the visible information in the interesting region. As shown in Figure 8 and Figure 9, the fusion result by the proposed method contains rich details while highlighting the target of IR image.Figure 8
Experiment with “Jeep.”
(a)
Visible image jeep
(b)
Infrared image jeep
(c)
Fused Jeep by [8]
(d)
Fused jeep by the proposed methodFigure 9
Detail with enlarged scale “Jeep.”
(a)
Visible image jeep
(b)
Infrared image jeep
(c)
Fused by [8]
(d)
Fused by the proposed methodThe proposed method will be compared with eight current fusion methods: principal component analysis- (PCA-) based method [10], discrete wavelet transform- (DWT-) based method [11], PCNN-based method [15], NSCT-based method [23], Laplacian pyramid transform- (LP-) PCNN-based method [14], NSCT-PCNN-based method [17], IFM based method [31], and MWGF-based method [32]. In all PCNN-based method experiments, through a large number of verification and comparison in experiments, the parameters of PCNN are set as αθ=0.2, αL=0.05, VL=0.02, Vθ=40, N=200, and M=W=0.707,1,0.707;1,0,1;0.707,1,0.707, where N is the number of iterations of PCNN. All NSCT-based methods, “pkva,” and “9–7” are set as the pyramid and the direction filter. For all multiscale decomposition methods, the decomposition level is set to 3, and “averaging” is used to fuse the low-frequency layer, and the high-frequency layers are fused by “absolute maximum choosing.”In order to evaluate the fusion results with different methods objectively, three most commonly used objective indicators will be used as the evaluation index: mutual information (MI), pixel of visual information (VIF), and edge gradient operator (QAB/F). MI is used to measure the amount of the source images’ information retained in the fused image. VIF is an evaluation index for human visual system, which is based on natural scenes and image distortion [33]. QAB/F is used to measure the edge information based on edge strength and orientation preserving from the source images. Commonly, the greater value of these evaluation metrics indicates that the fused image has a better quality [34].


## 3.2. Fusion Results and Discussions

The experimental images consisted of six pairs of IR and VI images, which are shown in Figure10. The first line in Figure 10 is VI images, and the second line is IR images. A large number of details and texture are included in the VI images, while the IR image contains only significant information.Figure 10
Experimental images. (a) “Sand path.” (b) “Bristol Queens Road.” (c) “UN Camp.” (d) “Trees.” (e)“Jeep.” (f) “Kaptein.”
(a)
(b)
(c)
(d)
(e)
(f)The fusion results obtained by different fusion algorithms of “Sand path” are given in Figure11. Figure 11(a) is the fused image by PCA, Figure 11(b) is the fused image by DWT, Figure 11(c) is the fused image by PCNN, Figure 11(d) is the fused image by NSCT, Figure 11(e) is the fused image by LP-NSCT, Figure 11(f) is the fused image by PCNN-NSCT, Figure 11(g) is the fused image by IFM, Figure 11(h) is the fused image by MWGF, and Figure 11(i) is the fused image by the proposed method. From Figure 11, we can see that the fused image by the proposed method contains more detail information of the VI image, as well as the highlighted infrared target information compared with other methods. In addition, the fused image by our method has advantages in visual effects and it is also superior to other algorithms in objective evaluation, which are shown in Table 1.Figure 11
Experimental results of “Sand path”. (a) PCA. (b) DWT. (c) PCNN. (d) NSCT. (e) LP-PCNN. (f) NSCT-PCNN. (g) IFM. (h) MWGF. (i) The proposed method.
(a)
(b)
(c)
(d)
(e)
(f)
(g)
(h)
(i)Table 1
Objective results for various fusion results.
Evaluation
Experiment
Fusion methods
PCA
DWT
PCNN
NSCT
LP-PCNN
NSCT-PCNN
IFM
MWGF
Reference [8]
Proposed
MI
Sand path
1.1117
0.5578
1.4473
1.5081
0.5468
1.0629
1.6325
1.7830
1.7935
1.8874
Bristol Queens Road
2.4788
1.2387
2.3390
1.9813
1.1924
2.2035
1.6665
2.5665
2.3513
2.6238
UN Camp
2.1191
0.7973
2.0342
1.5648
0.8595
1.0644
2.7810
2.9160
2.3461
2.1219
Trees
1.3114
0.8653
0.8844
1.1424
0.7757
1.3761
1.0500
1.3480
1.3418
1.3598
Jeep
2.1219
1.0000
3.1090
2.4130
1.9150
2.4980
1.8704
2.9550
2.9721
2.9613
Kaptein
2.0190
1.0910
2.5090
1.8650
1.1960
1.7100
3.469
3.3830
3.2045
3.3921
Average MI
1.8603
0.9250
2.0538
1.7457
1.0809
1.6524
2.0782
2.4919
2.3349
2.3910
VIF
Sand path
0.4201
0.2566
0.3048
0.3284
0.2788
0.3886
0.3599
0.4813
0.4821
0.5230
Bristol Queens Road
0.4991
0.3797
0.4986
0.4906
0.4687
0.5196
0.4947
0.5715
0.4949
0.5202
UN Camp
0.4599
0.2899
0.3107
0.3538
0.3156
0.3743
0.4419
0.4937
0.3873
0.4656
Trees
0.3776
0.3238
0.3145
0.4128
0.3648
0.4372
0.5141
0.5013
0.5243
0.4311
Jeep
0.4575
0.4100
0.5440
0.4841
0.5544
0.4788
0.4572
0.6023
0.5528
0.6107
Kaptein
0.4182
0.3637
0.4332
0.4488
0.4308
0.4359
0.4871
0.5189
0.4947
0.5103
Average VIF
0.4387
0.3372
0.4009
0.4197
0.4021
0.4390
0.4591
0.5281
0.4894
0.5101
QAB/F
Sand path
0.5013
0.3934
0.4188
0.4331
0.4136
0.5218
0.5104
0.5161
0.5072
0.5295
Bristol Queens Road
0.5678
0.3841
0.6708
0.5963
0.5889
0.6661
0.6066
0.6746
0.6693
0.6719
UN Camp
0.5038
0.3442
0.4785
0.4169
0.4356
0.4650
0.5265
0.5262
0.5296
0.5330
Trees
0.4533
0.3909
0.3804
0.4306
0.4476
0.4423
0.6004
0.5815
0.4832
0.4524
Jeep
0.4252
0.3436
0.6453
0.3022
0.5467
0.3142
0.3892
0.6769
0.6348
0.6818
Kaptein
0.3249
0.3442
0.5280
0.3692
0.4759
0.4240
0.5853
0.6109
0.6118
0.6189
AverageQAB/F
0.4627
0.3667
0.5203
0.4247
0.4847
0.4722
0.5364
0.5977
0.5726
0.5813In order to illustrate the applicability of the proposed method, other groups of experiments are performed, which are given as follows.It can be seen from Figure12 and Figure 13 that the proposed method has more advantages in detail information integration. In order to better reflect their differences of the fused images obtained by different fusion methods, Figure 13 shows the detail with an enlarged scale of Figure 12. In Figure 13(i), the red frame region is more suitable for human visual system, with more visible detail information. Compared with the same position in two source images, the fused region has higher readability and reliability.Figure 12
Second example experimental results of “Bristol Queens Road.” (a) PCA. (b) DWT. (c) PCNN. (d) NSCT. (e) LP-PCNN. (f) NSCT-PCNN. (g) IFM. (h) MWGF. (i) The proposed method.
(a)
(b)
(c)
(d)
(e)
(f)
(g)
(h)
(i)Figure 13
Detail with enlarged scale. (a) PCA. (b) DWT. (c) PCNN. (d) NSCT. (e) LP-PCNN. (f) NSCT-PCNN. (g) IFM. (h) MWGF. (i) The proposed method.
(a)
(b)
(c)
(d)
(e)
(f)
(g)
(h)
(i)Figure14 and Figure 15 are the third and fourth groups of experiment of “UN Camp” and “Trees,” respectively. The objective evaluation matrices are given in Table 1. In order to reflect more directly out of their difference, line chart comparison of MI, VIF, and QAB/F values of the experiments is given in Figure 18.Figure 14
Experimental results of “UN Camp.” (a) PCA. (b) DWT. (c) PCNN. (d) NSCT. (e) LP-PCNN. (f) NSCT-PCNN. (g) IFM. (h) MWGF. (i) The proposed method.
(a)
(b)
(c)
(d)
(e)
(f)
(g)
(h)
(i)Figure 15
Experimental results of “Trees.” (a) PCA. (b) DWT. (c) PCNN. (d) NSCT. (e) LP-PCNN. (f) NSCT-PCNN. (g) IFM. (h) MWGF. (i) The proposed method.
(a)
(b)
(c)
(d)
(e)
(f)
(g)
(h)
(i)Figure16 and Figure 17 are the experimental results of “Jeep” and “Kaptein.” The objective evaluation matrices are given in Table 1. Line chart comparison of MI, VIF, and QAB/F values of the experiments is given in Figure 19. It can be seen that the fusion results of the proposed method have a better visual effect. Compared with the same position in two source images, the fused region has higher readability and reliability.Figure 16
Experimental results of “Jeep.” (a) PCA. (b) DWT. (c) PCNN. (d) NSCT. (e) LP-PCNN. (f) NSCT-PCNN. (g) IFM. (h) MWGF. (i) The proposed method.
(a)
(b)
(c)
(d)
(e)
(f)
(g)
(h)
(i)Figure 17
Experimental results of “Kaptein.” (a) PCA. (b) DWT. (c) PCNN. (d) NSCT. (e) LP-PCNN. (f) NSCT-PCNN. (g) IFM. (h) MWGF. (i) The proposed method.
(a)
(b)
(c)
(d)
(e)
(f)
(g)
(h)
(i)All fusion results by the proposed method and the method of [8] are shown in Figure 18. The first line in Figure 18 is the results of [8], and the second line is the results of the proposed method. The objective evaluation matrices are given in Table 1. It can be seen from Figure 18 and Table 1 that the proposed method has better results in objective evaluation and visual effect. The average values of MI, VIF, and QAB/F for six pairs of IR and VI images are listed in Table 1. It can be seen that the average value of the proposed method is only less than MWGF and greater than the other comparison algorithms, because the MWGF is more effective for some special images, so as to increase the average value, such as the MI of “UN Camp”, but the proposed method has higher evaluation metrics on most images than MWGF.Figure 18
Experimental results by [8] and the proposed method. (a) “Sand path.” (b) “Bristol Queens Road.” (c) “UN Camp.” (d) “Trees.” (e)“Jeep.” (f) “Kaptein.”
(a)
(b)
(c)
(d)
(e)
(f)Figure 19
Line chart comparison of MI, VIF, andQAB/F values for four groups of the experiments.
(a)
(b)
(c)


## 4. Conclusion

A novel multiscale fusion scheme for the IR and VI image based on the interesting region detection is proposed in this paper, which can integrate more background details as well as highlight the interesting region with the salient objects. This method combines the advantages of the MeanShift and guided filter, which are used to detect the interesting significant target region and fuse the interesting regions of the IR and VI image. Next, the background regions are fused in the NSCT domain. An improved weighted average method based on per-pixel weighted average is used to fuse the low-frequency layers, and for the high-frequency layers, SF-PCNN-based method is used to produce the fused new layers. Then the fused background regions are produced by NSCT reconstruction. The fused image is produced by fusing the fused interesting and background regions. Experimental results show that the proposed fusion scheme can achieve superior results in visual inspection and objective evaluations.



---

*Source: 5754702-2018-04-05.xml*