# Fusion of IR and Visual Images Based on Gaussian and Laplacian Decomposition Using Histogram Distributions and Edge Selection

Authors: Seohyung Lee; Daeho Lee
Journal: Mathematical Problems in Engineering
 (2016)

---

## Abstract

We propose a novel fusion method of IR (infrared) and visual images to combine distinct information from two sources. To decompose an image into its low and high frequency components, we use Gaussian and Laplacian decomposition. The strong high frequency information in the two sources can be easily fused by selecting the large magnitude of Laplacian images. The distinct low frequency information, however, is not as easily determined. As such, we use histogram distributions of the two sources. Therefore, experimental results show that the fused images can contain the dominant characteristics of both sources.


---

## Body

## 1. Introduction

Multisensor fusion is widely used for signal, image, feature, and symbol combination [1]. To fuse images, variable images, such as visual image, infrared (IR) image, millimeter wave (MMW) image, X-ray image, and depth image are used for concealed weapon detection [2, 3], remote sensing [4, 5], multifocus imaging [6], and so forth [1, 7, 8].Forward looking infrared (FLIR) cameras can sense IR radiation (i.e., thermal radiation). Therefore, IR images can contain useful information that is not apparent in visual images. Alternatively, detailed information within the visual band is not included in IR images. Therefore, the fusion of IR and visual images can provide the advantages of both types of images. Also, the fusion of IR and visual images can apply various research fields, such as night vision [9, 10], face recognition [11], human detection [12], and detecting concealed object [13].Visual and IR images can be easily fused by averaging; however, in this scheme, the advantages of the two sources are eliminated and the details are annihilated in the worst case.To preserve the dominant advantages in the fused images, two methods can be utilized. Various methods using multiscale decomposition, such as Laplacian pyramid [14, 15], discrete wavelet transform (DWT) [16, 17], and contrast pyramid [18] can be used. One stage of Laplacian pyramid decomposition is shown in Figure 1. Additionally, some methods using region-segmentation [19–21] have also been proposed.Figure 1
One stage of Laplacian pyramid decomposition.The multiscale decomposition-based methods are performed with low computation; however, selecting the correct and distinct values of the low frequency information is not easily determined. So, many stages of pyramid and DWT are used to select significant values, and distinct values are only detected from Laplacian images or high frequency bands [22]. Although distinct values can be selected from both low and high frequency images, most of all methods used only strong intensity and predetermined weights.On the other hand, the latter methods can simply select the distinct values of the low frequency information; accurate segmentation is not guaranteed, and segmentation methods have a higher computation complexity. In addition, seam boundary regions should be blended by two image sources to prevent discontinuities.In this paper we are aimed at developing a fusion method that has the advantages of the methods mentioned above. To do this, we use simple Gaussian and Laplacian decomposition and utilize histogram distributions and edge selection to determine the distinct values of the low and high frequency information, respectively. Because we use histogram distributions, significant low frequency information can be selected, such as locally hot or cold regions of IR images and locally bright or dark regions of visual images. In addition, only one decomposition is used and fast averaging filters are used, so processing speed is fast enough for real-time applications.


## 2. Fusion of IR and Visual Images

The proposed fusion method is similar to methods that use the Laplacian pyramid or DWT. We use Gaussian smoothing, as opposed to Gaussian or wavelet scaling, so our method is identical to methods that perform single scaling. A Gaussian imageX- is the filtered image of an original image X by the Gaussian convolution, and a Laplacian image ∇2X is calculated by X-X- (X=X-+∇2X) as shown in Figure 2.Figure 2
Gaussian and Laplacian decomposition.To obtain a fused imageF=F-+∇2F, the distinct information of ∇2F is related to the magnitude; a larger ∇2F indicates a strong edge. To determine the distinct value of F-, we use the histogram distributions of the visual and IR Gaussian images.To fuse a visual imageV and an IR image I, we first compute the Gaussian images V- and I- and decompose the Laplacian images ∇2V=V-V- and ∇2I=I-I-, and then the Gaussian and Laplacian image of fused image are generated by selecting distinct values using histogram distributions and edge selection as shown in Figure 3.Figure 3
The proposed fusion method of IR and visual images.The Laplacian component of the fused image∇2F is easily determined by comparing edge strength such as the absolute Laplacian values at each pixel. If the large absolute Laplacian pixels are directly used, boundary discontinuities have an evil effect in terms of visuality. This can cause the fused images to contain discontinuous artefacts. Accordingly, we use weight map of the absolute Laplacian values. To compute this weight map L-w, we compute the binary weight map Lw calculated by(1)Lwx,y=255,if∇2Ix,y>∇2Vx,y,0,otherwise.Finally, we determine∇2F using L-w or Lw as follows:(2a)∇2Fx,y=∇2Ix,yL-wx,y255+∇2Vx,y255-L-wx,y255,(2b)∇2Fx,y=∇2Ix,y,ifLwx,y>127,∇2Vx,y,otherwise,where (2a) can be used for the fusion image having smooth boundaries of objects, and (2b) can be used for the fusion image having distinct boundaries of objects.The distinct intensity values of most images have a low population. In particular, intensities with a low population in the IR images show the highest or the lowest temperature as shown in Figure4. Therefore, we use the histogram distribution to select F-.Figure 4
Intensities with a low population: (a) IR image and its histogram, (b) the binary imageI(x,y)>τ1 of the IR image (a), and (c) the binary image I(x,y)<τ2 of the IR image (a).
(a)
(b)
(c)To select the Gaussian component of the fusion image, we use the low population mapMh(x,y) given by(3)Mhx,y=I-x,y,ifHisV-V-x,y>HisI-I-x,y,V-x,y,otherwise,where HisV-() and HisI-() denote the histogram distribution functions for V- and I-, respectively. However, Mh has extreme discontinuities, so we substitute M-h for F-.In all processes for Gaussian filter, we use a fast mean filter using spatial buffers to reduce computation complexities.


## 3. Experimental Result

The proposed fusion method was tested with three image sequences of TNO image fusion dataset [23]: “UN Camp,” “Dune,” and “Trees” (360 × 270, grayscale). These test images have small intensity rages, so we tested modified images by linear histogram normalization instead of original images. In this experiment, we used two parameter sets as shown in Table 1, where the first three parameters are the mask sizes. The optimal parameters are experimentally determined to make visually nature fusion images not having artificial discontinuities.Table 1
Parameter sets.
Parameter sets
V
-
(
I
-
)
L
-
w
M
-
h
∇
2
F condition
Optimal parameters
31
×
31
3
×
3
31
×
31
(2a)
Large parameters
131
×
131
5
×
5
51
×
51
(2b)Examples of our results using the optimal parameters are shown in Figures5, 6, and 7. Edge discontinuities are observed in Lw and Mh, while they are blurred in L-w and F-. It seems that the distinct information of V and I is well fused in F. In ∇2F images, strong high frequency components are well selected. In F- images, the blurred distinct values are well revealed regardless of ∇2F images. Particularly, isolated dark regions having low frequency are distinctly shown in F images.Figure 5
An example of the fusion result (“UN Camp,” the intensity range of∇2F is [-128,128]).Figure 6
An example of the fusion result (“Dune,” the intensity range of∇2F is [-128,128]).Figure 7
An example of the fusion result (“Trees,” the intensity range of∇2F is [-128,128]).More results for the three image sequences using optimal parameters are shown in Figure8; the red fused regions are more influenced by the visual images, while orange fused regions are more influenced by the IR images.Figure 8
Examples of the proposed fusion method (from top to bottom: “UN Camp,” “Trees,” and “Dune”): (a) visual images, (b) IR images, and (c) fused images using the proposed method.To objectively evaluate the proposed method, we consider two evaluation metrics: entropy (E) and the Xydeas and Petrovic index (QpVI/F) [24]. The performance comparison with the averaging method is shown in Table 2. To compare our results with those reported in [12], we compared the improvement in the ratio of the averaging method as shown in Table 3, because the existing methods used a different image enhancement methods which were not appeared in literature.Table 2
Comparison with the averaging method.
Methods
UN Camp
Dune
Trees
Q
p
V
I
/
F
Averaging
0.312
0.315
0.297
Proposed (large parameters)
0.487
0.497
0.550
Proposed (optimal parameters)
0.437
0.430
0.476
E
Averaging
6.275
6.769
6.413
Proposed (large parameters)
7.191
7.453
7.086
Proposed (optimal parameters)
7.093
7.400
6.973Table 3
Comparison with the existing methods (the improved ratio of the averaging method; the performance ranking is shown in parentheses).
Methods
UN Camp
Dune
Trees
Q
p
V
I
/
F
Li’s [16]
1.277 (5)
1.154 (5)
1.402 (5)
Lewis’s [19]
1.515 (3)
1.435 (3)
1.575 (4)
Saeedi’s [21]
1.663 (
1)
1.568 (2)
1.795 (2)
Proposed (large parameters)
1.562 (2)
1.578 (
1)
1.851 (
1)
Proposed(optimal parameters)
1.400 (4)
1.365 (4)
1.603 (3)
E
Li’s [16]
1.075 (4)
1.033 (4)
1.015 (4)
Lewis’s [19]
1.059 (5)
1.019 (5)
1.011 (5)
Saeedi’s [21]
1.178 (
1)
1.120 (
1)
1.099 (2)
Proposed(large parameters)
1.146 (2)
1.101 (2)
1.105 (
1)
Proposed(optimal parameters)
1.130 (3)
1.093 (3)
1.087 (3)Although our method using the optimal parameters does not yield the best objective performance, these metrics do not completely agree with human subjective evaluation as shown in Figure9. The performance of Figure 9(b) is higher than Figure 9(a); however, Figure 9(b) has many discontinuities as indicated by the circles. In addition, in Figure 9(c), the stitching of the two source images (after clipping and mean filtering) yields the highest E compared to the other results. Therefore, QpVI/F and E for segmentation-based fusion methods may be overestimated to the human subjective evaluation.Figure 9
Limitations of objective evaluation metrics: (a) result of the proposed method (optimal parameters):QpVI/F=0.479, E=7.100, (b) result of the proposed method (large parameters): QpVI/F=0.556, E=7.161, and (c) stitching result: QpVI/F=0.550, E=7.214.
(a)
(b)
(c)Visual comparison of two results using two parameter sets is shown in Figure10. As shown in Table 3, QpVI/F and E of the results using the large parameters are higher than the results using the optimal parameters; however the results using the optimal parameters show better visuality.Figure 10
Comparison of the proposed method with averaging; (a–c) results of the proposed method (optimal parameters), results of the proposed method (large parameters), and results of the averaging method.
(a)
(b)
(c)Using single thread processing with a 3.60 GHz CPU, the average computation time of all of the sequences is 5.237 msec. This processing time is inconceivable for segmentation-based methods. Therefore, the proposed method can be used for real-time fusion applications.


## 4. Conclusion

In this paper, we have proposed a novel fusion method for IR and visual images based on Gaussian and Laplacian decomposition using histogram distributions and edge selection. This method can easily determine the distinct values of Gaussian and Laplacian images. The distinct values of Laplacian images are selected by edge strength, and the distinct values of Gaussian images are selected by using histogram distributions. So, the fused images can contain the dominant characteristics of two source images and can be obtained via relatively simple computation. In addition, we showed that the object evaluation, entropy and Xydeas and Petrovic index, does not completely agree with human visual evaluation by showing the results using two different parameter sets. Therefore, the proposed method can be used for image fusion and blending instead of other existing methods.



---

*Source: 3130681-2016-02-07.xml*