# Infrared and Visible Image Fusion with Hybrid Image Filtering

Authors: Yongxin Zhang; Deguang Li; WenPeng Zhu
Journal: Mathematical Problems in Engineering
 (2020)

---

## Abstract

Image fusion is an important technique aiming to generate a composite image from multiple images of the same scene. Infrared and visible images can provide the same scene information from different aspects, which is useful for target recognition. But the existing fusion methods cannot well preserve the thermal radiation and appearance information simultaneously. Thus, we propose an infrared and visible image fusion method by hybrid image filtering. We represent the fusion problem with a divide and conquer strategy. A Gaussian filter is used to decompose the source images into base layers and detail layers. An improved co-occurrence filter fuses the detail layers for preserving the thermal radiation of the source images. A guided filter fuses the base layers for retaining the background appearance information of the source images. Superposition of the fused base layer and fused detail layer generates the final fusion image. Subjective visual and objective quantitative evaluations comparing with other fusion algorithms demonstrate the better performance of the proposed method.


---

## Body

## 1. Introduction

Image fusion is an important technique of image enhancement that extracts different salient feature information from numerous images into one full enhanced image for increasing the amount of information and utilization of the image. In recent years, image fusion technology has been applied in several aspects such as multifocus, medical, remote sensing, infrared, and visible images [1], especially in the merging of infrared and visible images. The image generated by the infrared image according to the principle of thermal imaging has high contrast and mainly provides the saliency target information of the fused image, and the visible image mainly includes the accurate background information. The saliency target in the infrared image is important for target recognition, while the background texture data in the visible image are the key to environmental analysis and detail judgment. Infrared and visible image fusion provides more comprehensive information, which has important practical significance in military and civilian fields [2].At present, researchers have proposed large amount of approaches for infrared and visible image fusion [3]. Spatial and transform domain-based approaches are two popular approaches in this area. The spatial domain-based approaches mainly generate weights according to the characteristics of the original spatial information of the pixels or regions in the source image. These methods are usually simple and fast, but the edge blur is easy to occur. The transform domain-based method mainly includes two parts: image decomposition and fusion rules. Multiscale decomposition tools decompose the source images into different scale spaces for obtaining layers with different feature information. Fusion rules depending on the information characteristics of each layer guide the fusion of different layers. Then, the fusion result is obtained by the inverse operation of multiscale decomposition [2]. The transform domain-based methods have become a research hotspot for its good adaptability to the human visual system. Traditional multiscale transform methods have achieved good fusion results, such as pyramid [4], discrete wavelet [5], contourlet, non-subsampled contourlet [6], and non-subsampled shearlet transforms [7]. However, a large number of transform coefficients resulting in the complexity of parameter optimization compromise the fusion performance.In recent years, deep learning has been utilized in the modeling of complicated relationship between data and extraction of distinctive features [2, 3]. The methods based on deep learning such as convolutional neural networks [8–10], adversarial network [11–14], and dictionary learning-based sparse representation [15] have achieved better fusion performance. Ma et al. proposed a new end-to-end model, termed as the dual-discriminator conditional generative adversarial network, for fusing infrared and visible images of different resolutions [16]. Chen et al. proposed a target-enhanced multiscale transform decomposition model for infrared and visible image fusion to simultaneously enhance the thermal target in infrared images and preserve the texture details in visible images [17]. Xu et al. presented a new unsupervised and unified densely connected network for infrared and visible image fusion [18]. Zhang et al. proposed a fast unified image fusion network for infrared and visible images based on proportional maintenance of gradient and intensity [19]. They are good at feature extraction and data reproduction. But the difficulties of ideal image selection and training, learning parameter settings, and the domain knowledge may compromise the fusion quality [8]. Despite the flexibility, rigidity, and robustness of the conventional infrared and visible image fusion approaches, some improvements could be attained in this area, yet. The current study concentrates on these improvements.As is known, conventional transform domain-based methods suffer from complication of parameter optimization and cost in coefficient processing. Thus, an edge-preserving filter with spatial consistency and edge retention are introduced to image fusion. Edge-preserving filtering is an effective tool for image fusion, which enhances the edge information of the image and reduces artifacts around the edges. Local filter, global optimization, and hybrid filter based techniques are the three main methods in this area [2].Bilateral filter [20], cross bilateral filter [21], iterative guided filtering [22], guided filtering (GFF) [23], gradient domain-guided filtering (GDGF) [24], and co-occurrence filter [25] (CoF) methods are the popular local filter-based methods. The visual quality of the fused images could be significantly enhanced through the mentioned filtering methods. The mean filter can well remove noise by averaging value of the neighboring pixels, so it may cause reduction of visual quality. The bilateral filter method can well smooth images and maintain their details except for the results in gradient reversal artifacts. To overcome the mentioned deficiencies, the guided filtering approach presented in [23] could be effectively employed. However, the near edges of the image could not be accurately described via the local liner model in a guided filtering approach. This approach may cause several halos in the fused images that degrade the fusion efficiency. The gradient domain-guided filtering method significantly improves the visual quality except for much time computation of three visual features. The CoF method can well extract the detail information of the original images and enhance the visual quality of the fused image, but the iterative co-occurrence filtering on base layers for sharping boundaries between textures costs much time.Weighted least squares filtering approach [26, 27], L1 fidelity with L0 gradient [28], anisotropic diffusion [29], and fourth-order partial differential equation [30] are some well-known global optimization-based methods. The image smoothing could be achieved via the mentioned methods by solving an optimization problem with various fidelity terms or regularization. Although superior results and considerable enhancement of the fusion quality are always attained through these approaches, some iteration rules performed in them are time consuming. Besides, several parameters such as regularized factors, scale level, and synthetic weights should be adjusted in these methods that could be considered as another deficiency. On the other hand, various layers fused through the L1 fidelity with the L0 gradient method [27] by combining several levels with various weighted values may cause blocking artifacts during the fusion procedure.The algorithms based on hybrid filter smooth the source images with two or more filters, such as Gaussian filter, bilateral filter [31], and rolling guidance filters [22]. These methods can well suppress the halos and retain details except for the lack in robust of adaptability.As mentioned above, GFF can obtain better results with high computational efficiency except for representation of the image well near some edges. The local linear model used in the guided image filter improves the computational efficiency and gives the guided image filter the superiority in representation of background information of the source image. Unlike the across texture, the edges could be smoothed through CoF, which enables the extraction of texture data of the original images by CoF. Inspired by the advantage of GFF and CoF, a fusion method is presented for infrared and visible image with the guided filter and co-occurrence filter. The advantage of the guided filter in background information extraction and that of the co-occurrence filter in edge structure information extraction are combined to improve the fusion performance of infrared and visible image. The contributions of this study can be concluded as the following four aspects: (1) a novel infrared and visible image fusion approach using the guided filter and co-occurrence filter is proposed. (2) The guided filtering in base layers and co-occurrence filtering in detail layers enhance the fusion efficiency of the source images. (3) The base and detail layers are fused with the saliency maps constructed by the guided filter and co-occurrence filter, respectively. (4) The range filter of the normalized co-occurrence matrix is removed for improving the filtering speed.The remaining parts of the current work are given as follows. Section2 provides a detailed description of the guided filter and fast co-occurrence filter. Section 3 describes the presented approach. Section 4 presents the experimental results and their related discussions. Finally, Section 5 is devoted to conclusions and future work aspects.


## 2. Guided Filter and Improved Co-Occurrence Filter

### 2.1. Guided Filter

The output imageYi of the guided filter is a local linear transformation of the guided imageIi:(1)Yi=apIi+bp,∀i∈ωp,where i denotes the position of the image, ωpis a partial rectangular window of (2r + 1) ∗ (2r + 1) at center pixel p, r is an input parameter of the GF, and ap and bp are obtained by minimizing the output image Yi and the input image Xi:(2)E=∑i∈ωpapIi+bp−Xi2+λap2,where λ is a regularization parameter given by the user and the optimal value is obtained by linear ridge regression:(3)ap=1/ω∑i∈ωpIiXi−μpXp¯δp+λ,bp=Xp¯−apμp,where μp and δp represent the mean and variance of the guided image I in ωp, respectively, ω is the total number of pixels in ωp, and Xp¯ is the mean of the input image X in ωp. The filtered output image is(4)Yi=ai¯Ii+bi¯,where ai and bi are the average of the different matrix windows:(5)ai=1ω∑p∈ωiap,bi=1ω∑p∈ωibp.Large windows and regularization parameters can be used to fuse smooth regions containing background information, and the fusion efficiency is fast [32].


### 2.2. Improved Co-Occurrence Filter

The co-occurrence filter places the value-domain Gaussian filter of the BF into the normalized co-occurrence matrix [33]. The pixel values of high co-occurrences are assigned with larger weights and are smoothed, and pixels of low co-occurrences have smaller weights and are not smoothed. The CoF is defined as follows:(6)Yp=∑q∈Npwp,q∗Xp∑q∈Npwp,q,where Yp and Xp are the pixel values of the output and the input images, respectively, p and q denote indices of the pixels, and wp,q is the weight contributed by the pixel q to the output pixel p.In a Gaussian filter,wp,q could be described as follows:(7)wp,q=e−dp,q2/2δs2≜Gδsp,q,where Gδsp,q denotes a Gaussian filter, dp,q denotes the Euclidean distance of the pixels p and q, and δs is the specific parameter.In the BF,wp,q is defined as follows:(8)wp,q=Gδsp,q×e−Xp−Xq2/2δr2,where δr is a certain parameter.CoF combines the normalized co-occurrence matrix with the range filter in BF to extend BF to handle the boundary. The formula is described as follows:(9)wp,q=Gδsp,q∗MXp,Xq,MXp,Xq=Ca,bhahb,where M is a 256∗256 matrix and Ca,b denotes the co-occurrence of the counts a and b. ha and hb represent the corresponding frequencies of a and b in the image:(10)Ca,b=∑p,qe−dp,q2/2δs2Xp=aXq=b,ha=∑pXp=a,where δ is the Gaussian filter parameter (15 by default) and [] is a logical operation, and if the item inside is true, the result is 1, otherwise 0.CoF gathers co-occurrence data from the image and filters out noise while maintaining sharp boundaries between different textures. However, a co-occurrence filter is utilized to calculate the co-occurrence value by means of local window combined with the Gaussian filter, which increases the filter rate. The time complexity of the co-occurrence matrix in the original co-occurrence filter isOn×r2, where n denotes the number of pixel points and r is the size of the local window.To increase the filtering procedure speed, we remove the range filter of the normalized co-occurrence matrix and globally count the number of co-occurrences of the image. TheCa,b of the improved co-occurrence filter (ICoF) is calculated as follows:(11)Ca,b=∑p,qXp=aXq=b.The number of intervals of the statistical pixel pair is determined experimentally to be 6. The simplified co-occurrence matrix time complexity isOn, which improves the filter speed.The comparison of the filter results ofCoF and ICoF is shown in Figure 1.Figure 1
Comparison of filtering results using CoF and ICoF: (a) source image, (b) CoF, and (c) ICoF.
(a)(b)(c)Due to the removal of the range filter, ICoF will produce spots in the texture, but ICoF still has the ability to maintain the edges. To further verify the edge retention capability of ICoF, the normalized image is input into two filters, and the results are presented in Figure2.Figure 2
Comparison of normalized filtering results using CoF and ICoF: (a) source image, (b) CoF, and (c) ICoF.
(a)(b)(c)As shown in Figure2, CoF is smooth for the target person’s head, leaves, grass edges, etc., while the ICoF edge retains more stringent. For the experimental comparison of the filter speeds of CoF and ICoF, the time consumption of the ten groups of experiments is averaged to attain the final time consumption. The experimental image size is 270∗360, and the results are given in Table 1. It is evident from the table that the filter speed of ICoF is about 50% smaller than that of CoF.Table 1
CoF and ICoF time consumption comparison (unit: second).
FilterCoFICoFTime consuming1.13080.5086As mentioned above, the guided filter is good at background information extraction, and ICoF is good at filtering out noise while maintaining sharp boundaries between different textures. Thus, we make the most of their advantages to construct a novel fusion method for improving the fusion performance of the infrared and visible image.


## 2.1. Guided Filter

The output imageYi of the guided filter is a local linear transformation of the guided imageIi:(1)Yi=apIi+bp,∀i∈ωp,where i denotes the position of the image, ωpis a partial rectangular window of (2r + 1) ∗ (2r + 1) at center pixel p, r is an input parameter of the GF, and ap and bp are obtained by minimizing the output image Yi and the input image Xi:(2)E=∑i∈ωpapIi+bp−Xi2+λap2,where λ is a regularization parameter given by the user and the optimal value is obtained by linear ridge regression:(3)ap=1/ω∑i∈ωpIiXi−μpXp¯δp+λ,bp=Xp¯−apμp,where μp and δp represent the mean and variance of the guided image I in ωp, respectively, ω is the total number of pixels in ωp, and Xp¯ is the mean of the input image X in ωp. The filtered output image is(4)Yi=ai¯Ii+bi¯,where ai and bi are the average of the different matrix windows:(5)ai=1ω∑p∈ωiap,bi=1ω∑p∈ωibp.Large windows and regularization parameters can be used to fuse smooth regions containing background information, and the fusion efficiency is fast [32].


## 2.2. Improved Co-Occurrence Filter

The co-occurrence filter places the value-domain Gaussian filter of the BF into the normalized co-occurrence matrix [33]. The pixel values of high co-occurrences are assigned with larger weights and are smoothed, and pixels of low co-occurrences have smaller weights and are not smoothed. The CoF is defined as follows:(6)Yp=∑q∈Npwp,q∗Xp∑q∈Npwp,q,where Yp and Xp are the pixel values of the output and the input images, respectively, p and q denote indices of the pixels, and wp,q is the weight contributed by the pixel q to the output pixel p.In a Gaussian filter,wp,q could be described as follows:(7)wp,q=e−dp,q2/2δs2≜Gδsp,q,where Gδsp,q denotes a Gaussian filter, dp,q denotes the Euclidean distance of the pixels p and q, and δs is the specific parameter.In the BF,wp,q is defined as follows:(8)wp,q=Gδsp,q×e−Xp−Xq2/2δr2,where δr is a certain parameter.CoF combines the normalized co-occurrence matrix with the range filter in BF to extend BF to handle the boundary. The formula is described as follows:(9)wp,q=Gδsp,q∗MXp,Xq,MXp,Xq=Ca,bhahb,where M is a 256∗256 matrix and Ca,b denotes the co-occurrence of the counts a and b. ha and hb represent the corresponding frequencies of a and b in the image:(10)Ca,b=∑p,qe−dp,q2/2δs2Xp=aXq=b,ha=∑pXp=a,where δ is the Gaussian filter parameter (15 by default) and [] is a logical operation, and if the item inside is true, the result is 1, otherwise 0.CoF gathers co-occurrence data from the image and filters out noise while maintaining sharp boundaries between different textures. However, a co-occurrence filter is utilized to calculate the co-occurrence value by means of local window combined with the Gaussian filter, which increases the filter rate. The time complexity of the co-occurrence matrix in the original co-occurrence filter isOn×r2, where n denotes the number of pixel points and r is the size of the local window.To increase the filtering procedure speed, we remove the range filter of the normalized co-occurrence matrix and globally count the number of co-occurrences of the image. TheCa,b of the improved co-occurrence filter (ICoF) is calculated as follows:(11)Ca,b=∑p,qXp=aXq=b.The number of intervals of the statistical pixel pair is determined experimentally to be 6. The simplified co-occurrence matrix time complexity isOn, which improves the filter speed.The comparison of the filter results ofCoF and ICoF is shown in Figure 1.Figure 1
Comparison of filtering results using CoF and ICoF: (a) source image, (b) CoF, and (c) ICoF.
(a)(b)(c)Due to the removal of the range filter, ICoF will produce spots in the texture, but ICoF still has the ability to maintain the edges. To further verify the edge retention capability of ICoF, the normalized image is input into two filters, and the results are presented in Figure2.Figure 2
Comparison of normalized filtering results using CoF and ICoF: (a) source image, (b) CoF, and (c) ICoF.
(a)(b)(c)As shown in Figure2, CoF is smooth for the target person’s head, leaves, grass edges, etc., while the ICoF edge retains more stringent. For the experimental comparison of the filter speeds of CoF and ICoF, the time consumption of the ten groups of experiments is averaged to attain the final time consumption. The experimental image size is 270∗360, and the results are given in Table 1. It is evident from the table that the filter speed of ICoF is about 50% smaller than that of CoF.Table 1
CoF and ICoF time consumption comparison (unit: second).
FilterCoFICoFTime consuming1.13080.5086As mentioned above, the guided filter is good at background information extraction, and ICoF is good at filtering out noise while maintaining sharp boundaries between different textures. Thus, we make the most of their advantages to construct a novel fusion method for improving the fusion performance of the infrared and visible image.


## 3. Proposed Fusion Method

We propose a hybrid filtering-based fusion approach for infrared and visible images. The proposed fusion framework is described in Figure3. We use the Gaussian filter to perform two-scale decomposition on the source image. The detail and the base layers are, respectively, fused by a weighted averaging method of the improved co-occurrence filter and guided filter, and finally the fusion result is obtained by superimposition.Figure 3
Block diagram of the image fusion algorithm based on GF-ICoF.


### 3.1. Two-Scale Decomposition

The mean filter and Gaussian filter are usually used for two-scale decomposition. The Gaussian filter decomposition is sharper, and the edge information obtained by the Gaussian filtering is more significant. Thus, the Gaussian filter is applied to our fusion framework for obtaining the base layer containing a large amount of background information. Assume thatIi represents the registered input image, where i = 1, 2. The base layer Bi of the two source images is calculated as follows:(12)Bi=Ii∗G,where G is the Gaussian filter with the standard deviation 5 while the filter window is adjusted as 11×11. The detail layer Di including the edge structure can be obtained by subtraction of the base layer Bi containing the background information from the source image Ii:(13)Di=Ii−Bi.


### 3.2. Fusion of Base Layers

It is easy to see that background information of the base layer is smooth. The Laplacian filter is applied to extract the saliency of the layer for obtaining a contrast map. Then, the Gaussian low-pass filter is applied to filter the high-frequency noise for a final saliency mapBSi. The multiple filtering is performed as follows:(14)BiH=LapBi,BSi=GBiH,where Lap represents the Laplacian filter and the filtered operator is [0 1 0; 1 −4 1; 0 1 0]. G denotes the Gaussian filter, and the filter window size is 11×11, and the standard deviation is 5. The initial binary weight maps BPik of the base layer are obtained by comparing the saliency maps of the base layers. The pixels with high saliency have high weight:(15)BPik=1,ifSik=maxBS1k,BS2k,0,otherwise,where k represents a pixel point, and a binary weight map is obtained by comparing pixel by pixel. The binary weight maps BPik directly applied to the base layer fusion may produce artifacts and blurring because of spatial inconsistency. As is known, the guided filter can well preserve the edges between textures while smoothing the surrounding pixel values. Optimized weight maps with the guided filter can preserve texture information more while maintaining spatial consistency. Optimized weight maps are defined as follows:(16)WiB=GIFr,εBPi,Bi,where WiB represents the final base layer weight map, the input of the guided filter is BPi, and the guided image is the original base layer Bi.The multiple base layer weight maps are merged with the base layer to obtain the fused base layerFB:(17)FB=WiB∗Bi.


### 3.3. Fusion of Detail Layers

The detail layer mainly includes the edge structure data of source images. Five classical saliency detection algorithms: spectral residual (SR) [34], frequency-tuned (FT) [35], maximum symmetric surround (MSS) [36], median mean difference (MMD) [37], and visual weight map (VWM) [22] are compared to search for a better measurement of the saliency of the edge structure in detail layers. The results are shown in Figure 4. It is evident that FT is better for the significant extraction of character and leaves except for the road and the grass. The MMD has improved the saliency of the road, but the contrast of the grass is insufficient. SR has a lot of saliency positions, but not very continuous. The saliency of VWM is obvious, but there is still some information lost. The saliency of MSS is obvious in representing roads and grasses, and it is more comprehensive. Thus, we use MSS as the saliency extraction method of the detail layer containing a lot of edge structures.Figure 4
Significant comparison results: (a) source image, (b) FT, (c) SR, (d) MMD, (e) VWM, and (f) MSS.
(a)(b)(c)(d)(e)(f)MSS is defined as follows:(18)Si,j=Xμi,j−Xfi,j,where Si,j is the saliency value of the output at position i,j. Xμi,j is the average of the CIELAB vectors of all pixels of the input image Xi,j. Xfi,j is the corresponding CIELAB pixel vector of the input image after the Gaussian filter. The value range of the Gaussian filtered gray image in this paper is adjusted to [0, 100] for obtaining the brightness value as Xfi,j.  indicates the L2 norm. Xμi,j is as follows:(19)Xμi,j=1H∑n=i−i0i+i0∑m=j−j0j+j0Xn,m,where i0 and j0 represent the offset and H is a subgraph, which is defined as follows:(20)i0=mini,r−i,j0=minj,r−j,where r and c represent the width and height of the input image, and H=2×i0+12×j0+1. The initial weight map DPik is obtained by comparing the obtained saliency map. It is defined as follows:(21)DPik=1,ifSik=maxDS1k,DS2k,0,otherwise.Binary weight maps directly applied for the fusion of detail layers may produce artifacts and blurring. Considering the advantage of CoF in texture feature extraction and the visual feature of the detail layer, we preserve the intertexture edge with the co-occurrence filter smoothing binary weight map. In order to improve the speed, we optimized CoF by removing the range filter of the normalized co-occurrence matrix.ICoF is defined as follows:(22)WiD=ICoFδsDPi,where δs denotes the standard deviation of the spatial Gaussian filter in ICoF. Parameter settings are consist with those of the original co-occurrence filter, and δs2=2∗15+1. The weight maps obtained guide the fusion of the detail layer for obtaining the fused detail layer FD:(23)FD=∑i=12WiD∗Di.


### 3.4. Two-Scale Image Reconstruction

Since the detail layer and the base layer are obtained by subtraction, the reconstruction is obtained by merging the fusion results of the detail and the base layers:(24)F=FB+FD.


## 3.1. Two-Scale Decomposition

The mean filter and Gaussian filter are usually used for two-scale decomposition. The Gaussian filter decomposition is sharper, and the edge information obtained by the Gaussian filtering is more significant. Thus, the Gaussian filter is applied to our fusion framework for obtaining the base layer containing a large amount of background information. Assume thatIi represents the registered input image, where i = 1, 2. The base layer Bi of the two source images is calculated as follows:(12)Bi=Ii∗G,where G is the Gaussian filter with the standard deviation 5 while the filter window is adjusted as 11×11. The detail layer Di including the edge structure can be obtained by subtraction of the base layer Bi containing the background information from the source image Ii:(13)Di=Ii−Bi.


## 3.2. Fusion of Base Layers

It is easy to see that background information of the base layer is smooth. The Laplacian filter is applied to extract the saliency of the layer for obtaining a contrast map. Then, the Gaussian low-pass filter is applied to filter the high-frequency noise for a final saliency mapBSi. The multiple filtering is performed as follows:(14)BiH=LapBi,BSi=GBiH,where Lap represents the Laplacian filter and the filtered operator is [0 1 0; 1 −4 1; 0 1 0]. G denotes the Gaussian filter, and the filter window size is 11×11, and the standard deviation is 5. The initial binary weight maps BPik of the base layer are obtained by comparing the saliency maps of the base layers. The pixels with high saliency have high weight:(15)BPik=1,ifSik=maxBS1k,BS2k,0,otherwise,where k represents a pixel point, and a binary weight map is obtained by comparing pixel by pixel. The binary weight maps BPik directly applied to the base layer fusion may produce artifacts and blurring because of spatial inconsistency. As is known, the guided filter can well preserve the edges between textures while smoothing the surrounding pixel values. Optimized weight maps with the guided filter can preserve texture information more while maintaining spatial consistency. Optimized weight maps are defined as follows:(16)WiB=GIFr,εBPi,Bi,where WiB represents the final base layer weight map, the input of the guided filter is BPi, and the guided image is the original base layer Bi.The multiple base layer weight maps are merged with the base layer to obtain the fused base layerFB:(17)FB=WiB∗Bi.


## 3.3. Fusion of Detail Layers

The detail layer mainly includes the edge structure data of source images. Five classical saliency detection algorithms: spectral residual (SR) [34], frequency-tuned (FT) [35], maximum symmetric surround (MSS) [36], median mean difference (MMD) [37], and visual weight map (VWM) [22] are compared to search for a better measurement of the saliency of the edge structure in detail layers. The results are shown in Figure 4. It is evident that FT is better for the significant extraction of character and leaves except for the road and the grass. The MMD has improved the saliency of the road, but the contrast of the grass is insufficient. SR has a lot of saliency positions, but not very continuous. The saliency of VWM is obvious, but there is still some information lost. The saliency of MSS is obvious in representing roads and grasses, and it is more comprehensive. Thus, we use MSS as the saliency extraction method of the detail layer containing a lot of edge structures.Figure 4
Significant comparison results: (a) source image, (b) FT, (c) SR, (d) MMD, (e) VWM, and (f) MSS.
(a)(b)(c)(d)(e)(f)MSS is defined as follows:(18)Si,j=Xμi,j−Xfi,j,where Si,j is the saliency value of the output at position i,j. Xμi,j is the average of the CIELAB vectors of all pixels of the input image Xi,j. Xfi,j is the corresponding CIELAB pixel vector of the input image after the Gaussian filter. The value range of the Gaussian filtered gray image in this paper is adjusted to [0, 100] for obtaining the brightness value as Xfi,j.  indicates the L2 norm. Xμi,j is as follows:(19)Xμi,j=1H∑n=i−i0i+i0∑m=j−j0j+j0Xn,m,where i0 and j0 represent the offset and H is a subgraph, which is defined as follows:(20)i0=mini,r−i,j0=minj,r−j,where r and c represent the width and height of the input image, and H=2×i0+12×j0+1. The initial weight map DPik is obtained by comparing the obtained saliency map. It is defined as follows:(21)DPik=1,ifSik=maxDS1k,DS2k,0,otherwise.Binary weight maps directly applied for the fusion of detail layers may produce artifacts and blurring. Considering the advantage of CoF in texture feature extraction and the visual feature of the detail layer, we preserve the intertexture edge with the co-occurrence filter smoothing binary weight map. In order to improve the speed, we optimized CoF by removing the range filter of the normalized co-occurrence matrix.ICoF is defined as follows:(22)WiD=ICoFδsDPi,where δs denotes the standard deviation of the spatial Gaussian filter in ICoF. Parameter settings are consist with those of the original co-occurrence filter, and δs2=2∗15+1. The weight maps obtained guide the fusion of the detail layer for obtaining the fused detail layer FD:(23)FD=∑i=12WiD∗Di.


## 3.4. Two-Scale Image Reconstruction

Since the detail layer and the base layer are obtained by subtraction, the reconstruction is obtained by merging the fusion results of the detail and the base layers:(24)F=FB+FD.


## 4. Experimental Results and Discussion

In order to evaluate the performance of the proposed method, the classical and recent proposed fusion algorithms are used for comparison. The experimental results are assessed with subjective evaluation and objective evaluation.


### 4.1. Experimental Settings

#### 4.1.1. Image Sets

Figure5 shows eight pairs of the test image used for the experiment of infrared and visible image fusion derived from three public datasets [37–39]. The three image datasets used to support the findings of this study are included within the open data collection in http://www.imagefusion.org/, https://figshare.com/articles/TNO_Image_Fusion_Dataset/1008, and https://github.com/hanna-xu/RoadScene. The upper images are infrared image, and the lower images are corresponding visible images.Figure 5
Eight sets of images to be fused shown in the experiment: (a) NATO camp, (b) Kaptein man, (c) sandpath, (d) Kaptein rain, (e) soldiers with jeep, (f) bike with car, (g) crowds, and (h) backpacker.
(a)(b)(c)(d)(e)(f)(g)(h)


#### 4.1.2. Compared Methods

The fusion methods used for experiment in this paper are as follows: guided filter algorithm (GFF) [23], two-scale using saliency detection (TSD) [37], contrast pyramid algorithm (CP) [40], convolution sparse representation (CSR) algorithm [41], gradient-based transfer and total variation algorithm (GTTV) [42], weighted least squares optimization (WLS) algorithm [22], multiple visual feature measurement-based fusion (MVFMF) approach [24], co-occurrence filter-based fusion method (CoF) [25], cross bilateral filter (CBF) [21], target-enhanced multiscale transform decomposition algorithm (TEMTD) [17], and latent low-rank representation algorithm (LLRR) [43].


#### 4.1.3. Parameter Setting

The parameters of the fast co-occurrence filter have been introduced in the previous section. For the setting of the two parametersr and ε of the guided filter, through multiple groups of experiments combined with subjective and objective evaluation analysis, the value of the two parameters and the size of the fused image are positively correlated. This paper introduces a window factor t which divides the sum of image rows and columns to obtain the final window size r:(25)r=m+nt,where m and n represent the corresponding rows and columns of the image, rounded up, and the value is integer. It is known that not all objective evaluation indicators can effectively reflect the fusion effect. In order to avoid different evaluation indicators for determining the value of the window factor, we use QAB/F as the objective feedback value of fusion results with different t. The final value of r is determined based on the trend of QAB/F of the fusion results going along with different t. The experimental results are shown in Figure 6. It can be seen from Figure 6 that QAB/F has reached a peak when t is set to 34. Thus, the window factor is determined to be 34. ε is obtained by dividing r by a constant g, ε=r/g. According to the parameters of the guided filter fusion base layer in [19], g=150.Figure 6
QAB/F values of different t fusion results.


#### 4.1.4. Evaluation Metrics

To verify the performance of the presented algorithm comparing with existing approaches, we apply five evaluation metrics to the objective fusion assessment. They are quality assessment based on structural similarity entropy (EN), standard deviation (SD), multiscale structural similarity (MS-SSIM) [44], normalized feature mutual information (NFMI) [45], edge retention fusion quality indicator QAB/F [46], and visual information fidelity degree VIFF [47]. These metrics describe the fusion results from different perspectives, and the larger of those values indicate that more source image information is retained, and better fusion performance is achieved [48].


### 4.2. Qualitative Assessments

Using qualitative result analysis, Figures7–11 show the fusion results with different algorithms for the infrared and visible images of Figure 5.Figure 7
Fusion results of “NATO camp” with different fusion methods: (a) GFF, (b) CP, (c) TSSD, (d) GTTV, (e) CSR, (f) WLS, (g) MVFMF, (h) CBF, (i) TEMTD, (j) LLRR, (k) CoF, and (l) GF-ICoF.
(a)(b)(c)(d)(e)(f)(g)(h)(i)(j)(k)(l)Figure 8
Fusion results of “Kaptein man” with different fusion methods: (a) GFF, (b) CP, (c) TSSD, (d) GTTV, (e) CSR, (f) WLS, (g) MVFMF, (h) CBF, (i) TEMTD, (j) LLRR, (k) CoF, and (l) GF-ICoF.
(a)(b)(c)(d)(e)(f)(g)(h)(i)(j)(k)(l)Figure 9
Fusion results of “sandpath” with different fusion methods: (a) GFF, (b) CP, (c) TSSD, (d) GTTV, (e) CSR, (f) WLS, (g) MVFMF, (h) CBF, (i) TEMTD, (j) LLRR, (k) CoF, and (l) GF-ICoF.
(a)(b)(c)(d)(e)(f)(g)(h)(i)(j)(k)(l)Figure 10
Fusion results of “Kaptein rain” with different fusion methods: (a) GFF, (b) CP, (c) TSSD, (d) GTTV, (e) CSR, (f) WLS, (g) MVFMF, (h) CBF, (i) TEMTD, (j) LLRR, (k) CoF, and (l) GF-ICoF.
(a)(b)(c)(d)(e)(f)(g)(h)(i)(j)(k)(l)Figure 11
Fusion results of “soldiers with jeep” with different fusion methods: (a) GFF, (b) CP, (c) TSSD, (d) GTTV, (e) CSR, (f) WLS, (g) MVFMF, (h) CBF, (i) TEMTD, (j) LLRR, (k) CoF, and (l) GF-ICoF.
(a)(b)(c)(d)(e)(f)(g)(h)(i)(j)(k)(l)The GFF algorithm can enhance the visual qualities of the fusion image by employing appropriate weights on the corresponding regions. Although the additional structure and detailed data could be transferred from the guidance image to the fused one via the GFF method, the images around certain edges could not be described appropriately through the local liner model in this method. Although no explicit constraint occurs from the edge-aware factor to treat edges, it may generate specific halos in the fused images. A few blurring artifacts could be observed due to the GFF fusion such as the lost data of grass (boxes II and III) in Figure7(a), the blurred branches (box I) in Figure 8(a), the reduced brightness of the trunk portion (box II) of the tree in Figure 9(a), the reduced brightness of the tent (box II) in Figure 10(a), the lost cloud information of the sky background (box I) in Figure 11(a), and the blurred street lamp (box I) and blurred building (box II) in Figure 12(a).Figure 12
Fusion results of “soldiers with jeep” with different fusion methods: (a) GFF, (b) CP, (c) TSSD, (d) GTTV, (e) CSR, (f) WLS, (g) MVFMF, (h) CBF, (i) TEMTD, (j) LLRR, (k) CoF, and (l) GF-ICoF.
(a)(b)(c)(d)(e)(f)(g)(h)(i)(j)(k)(l)CP decomposed the source image into some decomposition layers with various resolution and spatial frequencies. CP has achieved a good visual effect except for the lack in adaptability. The CP fusion results show some blurring, such as the low contrast in Figure7(b), blurred branches (box I) in Figure 8(b), the black block appearing in the lower right corner (box III) in Figure 9(b), the low brightness of the jeep and soldiers in Figure 11(b), and the blurred street lamp (box I) and the blurred leg of the crowds (box II) in Figure 12(b). TSSD used a mean and median filter to extract visual saliency extraction. The weight map obtained from the saliency map guided the fusion of the base and detail layers, which greatly improves the transfer ability of complementary data from the original images effectively. But the ability of the mean and median filter in saliency extraction compromised the fusion performance, such as the lost surrounding fence and grass information (boxes II and III) in Figure 7(c), the blurred grass around (box II) in Figure 8(c), the blurred trunk portion (box II) of the tree in Figure 9(c), the low brightness of the tent (box II) in Figure 10(c), the low brightness of the jeep and soldiers (box II) in Figure 11(c), and the low brightness of the crowds (box II) and street lamp (box I) in Figure 12(c).The fusion process is converted to al1-TV minimization problem through the GTTV in which the fundamental intensity distribution in the infrared image and the gradient change in the visible image are preserved through the data fidelity and regularization terms, respectively. Moreover, the thermal radiation and the appearance data in the infrared and visible images could be preserved, respectively (except for the lack of adaptability). The blur appears such as the blurred branches (box I) in Figure 8(d), the blurred trunk (box II) of the tree in Figure 9(d), the lost information of the tent in Figure 10(d), the blurred jeep and soldiers (box II) in Figure 11(d), and the blurred crowds (box II) in Figure 12(d).CSR used convolutional sparse representation to overcome detail preservation and strong sensitivity to misregistration in image fusion. It has obtained better fusion performance for infrared and visible image. But the regularization parameters in the CSR model and the spatial size of dictionary filters lack in self-adaptability, which compromise the fusion efficiency such as the brightness of the target person (box I) in Figure7(e), the dark background in Figure 8(e), the low brightness of the trunk portion (box II) of the tree in Figure 9(e), the low brightness of the jeep and soldiers (box II) in Figure 11(e), and the low brightness of the street lamp (box I) in Figure 12(e).WLS employs the RGF and Gaussian filter to decompose input images to base and detail layers. WLS constructs weight maps by choosing various features of the IR and visible image. Effective visual information could be appropriately transformed into the fused image through the WLS while reducing the noise from the IR image. But the lack of parameter flexibility compromises the fusion quality, such as the low brightness of the target person (box I) in Figure7(f), the dark background in Figure 8(f), the blurred trunk portion (box II) of the tree in Figure 9(f), the low-contrast vehicle and soldiers (box II) in Figure 11(f), and the low-contrast street lamp (box I) and crowds (box II) in Figure 12(f).Three visual property measurements are utilized by the MVFMF to generate decision maps that are optimized through gradient domain-guided filtering. The visual description of the detailed data of the original images was provided through it. Although the fusion efficiency was significantly enhanced via the MVFMF decision map construction model, the feasibility of the constant parameters for all original images with various values is not satisfactory in some instances. Slight blurs and artifacts could be observed in the fusion images such as the lost information of the grass (boxes II and III) in Figure7(g), the blurred branches (box I) in Figure 8(g), the low brightness of the trunk portion (box II) of the tree in Figure 9(g), the information of the tent (box II) in Figure 10(g), the lost cloud information of the sky background (box I) in Figure 11(g), and the lost building information of the street lamp (box I) in Figure 12(g).The CBF extracts the detail information by using joint bilateral filtering and avoids the gradient reversal artifacts of the bilateral filter. It uses weighted average of pixel values to the fuse source image and improves the visual quality of the fused image. However, the weight construction may compromise the contrast of the fused image. The little blurs and the corresponding artifacts appear in the fused images such as the blurred target person (box I) in Figure7(h), the blurred edge of the tree (box I) and the blurred figure of the man in Figure 8(h), the blurred trunk portion (box II) of the tree in Figure 9(h), the vehicle and soldiers (box II) in Figure 11(h), and the street lamp (box I) and crowds (box II) in Figure 12(h).TEMTD is a novel infrared and visible image fusion method based on target-enhanced MST decomposition. It deals with problem regarding the preservation of thermal radiation features in the traditional MST-based methods. It can simultaneously maintain thermal radiation characteristics in the infrared image and texture details in the visible image by using a specific fusion rule design. It uses the decomposed infrared low-frequency information to determine the fusion weight of low-frequency bands and highlight the target. The common “max-absolute” fusion rule is performed for fusion for high-frequency bands. The common “max-absolute” fusion rule may compromise the fusion performance, such as the blurred bush (boxes II and III) in Figure7(i), the blurred branches (box II) in Figure 9(i), the blurred bush in the left bottom of Figure 10(i), the blurred sky in Figure 11(i), and the street lamp in Figure 12(i).LLRR is a novel fusion framework for fusing infrared and visible images. A projection matrixL learned by latent low-rank representation is applied to extract detail parts and base parts of the input images at several representation levels. It extracts multilevel salient features by using latent low-rank representation. The final fused image is reconstructed by adaptive fusion strategies designed specifically for dealing with the detail parts and the base parts, respectively. The LLRR framework can be used to provide an efficient decomposition approach for extracting multilevel features for an arbitrary number of input images. The adaptive fusion rules improve the ability in texture and structural information extraction. The bush in Figure 7(j), the branch in Figure 9(j), the bush in Figure 10(j), and the soldiers and jeep in Figure 11(j) all demonstrate high contrast. But there are information losses appear in the fused image, such as the black edge around the target man in Figures 7(j), 8(j), and 12(j).The co-occurrence filter features of boundary detection and edge preservation could be utilized by the CoF for weight optimization, which improves the fusion quality of the base layers and detail layers. The same fusion rule of base and detail layers compromises the fusion performance, such as the lost information of the grass (boxes II and III) in Figure7(k), the blurred grass (box II) in Figure 8(k), the blurred details of the branches (box I) in Figure 9(k), the blurred grass at the lower right corner (box II) in Figure 10(k), the lost cloud information of the sky background in Figure 11(k), and the lost information of the street lamp in Figure 12(k). We perform different image filtering for saliency extraction based on the visual feature of base layers and detail layers. By and large, compared with other algorithms, the proposed algorithm not only retains the saliency desired data of the infrared image but also preserves the background data in the visible image.


### 4.3. Quantitative Assessments

In the current subsection, seven quality measurement indices (as discussed in the previous subsection) are applied to the test images. The obtained results are presented in Tables2–6. Tables 2–6 demonstrate that the metric values of the presented approach are higher than the corresponding ones obtained with existing approaches, which demonstrates that further data could be preserved from the original images, and superior fusion quality could be obtained.Table 2
Fusion evaluation indicators for different algorithms for “NATO camp” and “Kaptein man.”
MethodNATO campKaptein manQAB/FNFMIVIFFENMS-SSIMSDQAB/FNFMIVIFFENMS-SSIMSDGFF0.5330.9030.1736.6630.70427.4330.5450.9420.2806.8580.87932.898CP0.4810.8860.4036.8240.87032.7760.4550.9350.3336.6320.91934.143TSSD0.4340.8890.3986.4690.92726.3800.4480.9320.3716.6480.93333.347GTTV0.4050.8850.2186.6940.78727.3800.4080.9060.3166.6830.83638.173CSR0.4630.8980.3316.3350.89024.4620.3140.9370.1846.9570.79832.259WLS0.4170.8910.3506.3740.90624.4790.5800.9280.2136.9670.83132.254MVFMF0.5110.9040.2226.5630.72126.4820.5110.9420.3086.5930.91035.139CBF0.3890.8710.2736.4500.74026.0160.3880.8970.1996.7770.73333.193TDMTD0.4020.8850.1756.6000.75226.7270.4680.9380.2256.7660.84335.712LLRR0.3120.8860.4146.7790.84931.3430.3280.9340.3507.1540.85845.318CoF0.3900.8900.3306.7330.83130.1300.4060.9380.2877.1390.89250.986GF-ICoF0.5470.9160.4246.7900.92632.0200.5620.9420.3577.1720.92351.693Table 3
Fusion evaluation indicators for different algorithms for “sandpath” and “Kaptein rain.”
MethodSandpathKaptein rainQAB/FNFMIVIFFENMS-SSIMSDQAB/FNFMIVIFFENMS-SSIMSDGFF0.5300.8530.2566.8400.88633.6540.5850.9110.3767.0270.88948.277CP0.4700.8300.3186.7350.88327.1180.5130.8920.4416.8070.86337.091TSSD0.4160.8390.3546.3420.93121.7790.4930.8910.4166.4420.92530.622GTTV0.4950.8410.1446.5100.82522.3120.4000.8810.1606.5240.74528.765CSR0.4710.8550.2576.2240.89520.1240.5250.9000.3406.3540.89628.560WLS0.3850.8450.2966.2440.89920.1880.4700.8930.3476.4000.89728.664MVFMF0.5900.8800.2607.0460.89838.2810.5830.9060.4197.1550.88047.532CBF0.3560.8310.2816.5200.80827.5600.4320.8640.3286.8780.65136.413TDMTD0.3150.8460.0966.1480.67517.4980.3630.8880.1316.1390.69526.014LLRR0.2760.8500.3117.1630.85538.7710.3130.9170.4397.1920.83248.288CoF0.3440.8460.2306.6500.80130.4920.4520.8980.3416.6710.84639.839GF-ICoF0.6020.8830.3247.1640.94738.9650.5980.9200.4897.2620.93748.362Table 4
Fusion evaluation indicators for different algorithms for “soldiers with jeep” and “bike with car.”
MethodSoldiers with jeepBike with carQAB/FNFMIVIFFENMS-SSIMSDQAB/FNFMIVIFFENMS-SSIMSDGFF0.5850.9140.3666.9340.85730.8730.5810.8620.2397.4650.85145.442CP0.2940.8730.4116.7240.79530.3930.5040.8270.3317.3320.81041.419TSSD0.4950.9040.4606.5520.94723.5230.4660.8350.3096.6550.90225.477GTTV0.3240.9060.3957.5360.87556.7950.3210.8500.1337.6000.64748.860CSR0.4900.9130.3476.4710.91922.1470.5130.8560.2506.5480.86123.755WLS0.4580.8970.3886.4940.89626.2530.4700.8400.2596.5520.87023.618MVFMF0.6430.9150.4687.1870.85339.6170.5800.8610.2277.5650.84249.134CBF0.3520.8340.3847.0180.59534.1690.4630.8080.2317.5100.75848.269TDMTD0.5580.9170.5667.2270.84143.1750.4500.8410.1557.1080.68839.293LLRR0.2960.9070.5607.0940.85037.8290.3610.8310.2497.3390.85444.219CoF0.4480.9100.5077.1480.89138.9040.4240.8340.1777.0740.77840.135GF-ICoF0.6110.9200.5667.6280.94858.9190.5690.8560.2587.5360.87749.241Table 5
Fusion evaluation indicators for different algorithms for “crowds” and “backpacker.”
MethodCrowdsBackpackerQAB/FNFMIVIFFENMS-SSIMSDQAB/FNFMIVIFFENMS-SSIMSDGFF0.6270.9150.4267.3570.88647.6920.5930.8690.2917.2450.84437.960CP0.5870.9100.6107.1260.93443.3940.5250.8510.3587.4900.78644.898TSSD0.5130.9010.5306.8840.94239.8690.4840.8520.3676.9380.85631.128GTTV0.3910.9110.2677.2770.81347.8540.4230.8510.2147.3190.71443.100CSR0.5550.9120.4606.8140.92338.9980.5430.8660.3136.8510.86329.101WLS0.5310.9040.4586.8290.92338.8540.4900.8560.3186.8320.86328.546MVFMF0.6380.9150.4667.4120.89949.0640.5970.8680.2737.5050.81844.921CBF0.5260.9060.5047.2800.83944.9990.4470.8080.2737.5440.60645.049TDMTD0.5480.9100.3167.1480.83945.5780.4680.8590.1766.8740.71035.952LLRR0.3840.8990.5077.2920.85049.5980.3950.8450.3267.4220.83945.265CoF0.5030.9090.5446.9890.90341.9520.4610.8510.2186.9780.73144.146GF-ICoF0.6510.9160.5967.4500.92349.6010.5860.8650.3327.4590.86445.591Table 6
Time consumption of nine algorithms (unit: second).
MethodGFFCPTSSDGTTVCSRWLSMVFMFCBFTDMTDLLRRCoFGF-I CoFTime0.6380.1060.1516.651211.0023.1900.6344.0232.963159.16533.8753.158As is known,Q(AB/F) shows the edge data of the fused image retained from the source image. NFMI measures the feature information of the fused image retention from the source one based on mutual information. MS-SSIM is based on structural similarity to measure the retention degree of the structural data of the fused image from the original one. EN indicates the information value contained in the fused image as a whole. SD reflects the contrast of the image with the difference degree between each pixel and the average value of the pixels. VIFF measures the degree of fusion of the effective visual information of each region.Histogram of the average metric (QAB/F, NFMI, VIFF, and MS-SSIM) amounts of the fusion approaches is presented in Figure 13. It is easy to see that the average QAB/F values of GFF, MVFMF, and GF-ICoF are obviously higher than that of other methods, which demonstrates that GFF, MVFMF, and GF-ICoF can well preserve the edge data from original images to fused images. The average NFMI value of GFF, CSR, LLRR, MVFMF, CoF, and GF-ICoF are obviously higher than that of other fusion methods. It demonstrates that more significant feature data could be transformed from the source image to the fused one through the five methods transfer compared with existing fusion approaches. Moreover, the mean VIFF and MS-SSIM values of TSSD and GF-ICoF are obviously greater than that of existing fusion approaches. It shows that TSSD and GF-ICoF can transfer more significant structure data from original images compared with existing fusion techniques.Figure 13
Histogram of the average metric (QAB/F, NFMI, VIFF, and MS-SSIM) values of the fusion methods.Histogram of the mean metric (EN, SD) amounts of the fusion approaches is depicted in Figure14. It is easy to see that the average SD values of MVFMF, LLRR, CoF, and GF-ICoF are obviously greater than the corresponding values obtained by other fusion approaches. It demonstrates that the fusion results of MVFMF, LLRR, CoF, and GF-ICoF have higher contrast with better visual quality compared with existing fusion algorithms. The mean EN values of GFF, MVFMF, CoF, and GF-ICoF are greater than that of existing fusion techniques. The comparison results of the fused image quality evaluation indices demonstrate the superiority of the developed fusion approach. The proposed method has sufficient retention for the edge and structure data from the original image. The fused image obtained with the presented approach has the best visual quality in terms of contrast. Overall, the approach is preferred to existing algorithms in accordance with different evaluation metrics.Figure 14
Histogram of the average metric (EN and SD) values of the fusion methods.The time complexity of each algorithm is obtained on the running time of 10 tests. The selected image size is270∗360. The results are shown in Table 6. It is evident that the time consumption of CSR is highest. The parameter training and dictionary learning cost much time. The extraction of detail parts and base parts of the source images is implemented by learning with latent low-rank representation, which results in higher time consumption of LLRR. The algorithm models of CP, TSSD, and GFF are simple and small time consumptive. The time consumption of WLS, CBF, TDMTD, and MVFMF is a little more than the previously mentioned methods. CSR spends a lot of time in sparse representation, which leads to excessive time overhead. The time cost of the propose method and the GTTV algorithm tends to be similar. Because the time consumption of the CoF is a little more, the CoF fusion algorithm uses an iterative way to fuse the base layer, which further increases the time spent for the fusion. Compared with the CoF fusion algorithm, the time efficiency of this work is increased by about 90%. In terms of the subjective and objective results, we can conclude that the algorithm is an effective infrared and visible image fusion algorithm.


## 4.1. Experimental Settings

### 4.1.1. Image Sets

Figure5 shows eight pairs of the test image used for the experiment of infrared and visible image fusion derived from three public datasets [37–39]. The three image datasets used to support the findings of this study are included within the open data collection in http://www.imagefusion.org/, https://figshare.com/articles/TNO_Image_Fusion_Dataset/1008, and https://github.com/hanna-xu/RoadScene. The upper images are infrared image, and the lower images are corresponding visible images.Figure 5
Eight sets of images to be fused shown in the experiment: (a) NATO camp, (b) Kaptein man, (c) sandpath, (d) Kaptein rain, (e) soldiers with jeep, (f) bike with car, (g) crowds, and (h) backpacker.
(a)(b)(c)(d)(e)(f)(g)(h)


### 4.1.2. Compared Methods

The fusion methods used for experiment in this paper are as follows: guided filter algorithm (GFF) [23], two-scale using saliency detection (TSD) [37], contrast pyramid algorithm (CP) [40], convolution sparse representation (CSR) algorithm [41], gradient-based transfer and total variation algorithm (GTTV) [42], weighted least squares optimization (WLS) algorithm [22], multiple visual feature measurement-based fusion (MVFMF) approach [24], co-occurrence filter-based fusion method (CoF) [25], cross bilateral filter (CBF) [21], target-enhanced multiscale transform decomposition algorithm (TEMTD) [17], and latent low-rank representation algorithm (LLRR) [43].


### 4.1.3. Parameter Setting

The parameters of the fast co-occurrence filter have been introduced in the previous section. For the setting of the two parametersr and ε of the guided filter, through multiple groups of experiments combined with subjective and objective evaluation analysis, the value of the two parameters and the size of the fused image are positively correlated. This paper introduces a window factor t which divides the sum of image rows and columns to obtain the final window size r:(25)r=m+nt,where m and n represent the corresponding rows and columns of the image, rounded up, and the value is integer. It is known that not all objective evaluation indicators can effectively reflect the fusion effect. In order to avoid different evaluation indicators for determining the value of the window factor, we use QAB/F as the objective feedback value of fusion results with different t. The final value of r is determined based on the trend of QAB/F of the fusion results going along with different t. The experimental results are shown in Figure 6. It can be seen from Figure 6 that QAB/F has reached a peak when t is set to 34. Thus, the window factor is determined to be 34. ε is obtained by dividing r by a constant g, ε=r/g. According to the parameters of the guided filter fusion base layer in [19], g=150.Figure 6
QAB/F values of different t fusion results.


### 4.1.4. Evaluation Metrics

To verify the performance of the presented algorithm comparing with existing approaches, we apply five evaluation metrics to the objective fusion assessment. They are quality assessment based on structural similarity entropy (EN), standard deviation (SD), multiscale structural similarity (MS-SSIM) [44], normalized feature mutual information (NFMI) [45], edge retention fusion quality indicator QAB/F [46], and visual information fidelity degree VIFF [47]. These metrics describe the fusion results from different perspectives, and the larger of those values indicate that more source image information is retained, and better fusion performance is achieved [48].


## 4.1.1. Image Sets

Figure5 shows eight pairs of the test image used for the experiment of infrared and visible image fusion derived from three public datasets [37–39]. The three image datasets used to support the findings of this study are included within the open data collection in http://www.imagefusion.org/, https://figshare.com/articles/TNO_Image_Fusion_Dataset/1008, and https://github.com/hanna-xu/RoadScene. The upper images are infrared image, and the lower images are corresponding visible images.Figure 5
Eight sets of images to be fused shown in the experiment: (a) NATO camp, (b) Kaptein man, (c) sandpath, (d) Kaptein rain, (e) soldiers with jeep, (f) bike with car, (g) crowds, and (h) backpacker.
(a)(b)(c)(d)(e)(f)(g)(h)


## 4.1.2. Compared Methods

The fusion methods used for experiment in this paper are as follows: guided filter algorithm (GFF) [23], two-scale using saliency detection (TSD) [37], contrast pyramid algorithm (CP) [40], convolution sparse representation (CSR) algorithm [41], gradient-based transfer and total variation algorithm (GTTV) [42], weighted least squares optimization (WLS) algorithm [22], multiple visual feature measurement-based fusion (MVFMF) approach [24], co-occurrence filter-based fusion method (CoF) [25], cross bilateral filter (CBF) [21], target-enhanced multiscale transform decomposition algorithm (TEMTD) [17], and latent low-rank representation algorithm (LLRR) [43].


## 4.1.3. Parameter Setting

The parameters of the fast co-occurrence filter have been introduced in the previous section. For the setting of the two parametersr and ε of the guided filter, through multiple groups of experiments combined with subjective and objective evaluation analysis, the value of the two parameters and the size of the fused image are positively correlated. This paper introduces a window factor t which divides the sum of image rows and columns to obtain the final window size r:(25)r=m+nt,where m and n represent the corresponding rows and columns of the image, rounded up, and the value is integer. It is known that not all objective evaluation indicators can effectively reflect the fusion effect. In order to avoid different evaluation indicators for determining the value of the window factor, we use QAB/F as the objective feedback value of fusion results with different t. The final value of r is determined based on the trend of QAB/F of the fusion results going along with different t. The experimental results are shown in Figure 6. It can be seen from Figure 6 that QAB/F has reached a peak when t is set to 34. Thus, the window factor is determined to be 34. ε is obtained by dividing r by a constant g, ε=r/g. According to the parameters of the guided filter fusion base layer in [19], g=150.Figure 6
QAB/F values of different t fusion results.


## 4.1.4. Evaluation Metrics

To verify the performance of the presented algorithm comparing with existing approaches, we apply five evaluation metrics to the objective fusion assessment. They are quality assessment based on structural similarity entropy (EN), standard deviation (SD), multiscale structural similarity (MS-SSIM) [44], normalized feature mutual information (NFMI) [45], edge retention fusion quality indicator QAB/F [46], and visual information fidelity degree VIFF [47]. These metrics describe the fusion results from different perspectives, and the larger of those values indicate that more source image information is retained, and better fusion performance is achieved [48].


## 4.2. Qualitative Assessments

Using qualitative result analysis, Figures7–11 show the fusion results with different algorithms for the infrared and visible images of Figure 5.Figure 7
Fusion results of “NATO camp” with different fusion methods: (a) GFF, (b) CP, (c) TSSD, (d) GTTV, (e) CSR, (f) WLS, (g) MVFMF, (h) CBF, (i) TEMTD, (j) LLRR, (k) CoF, and (l) GF-ICoF.
(a)(b)(c)(d)(e)(f)(g)(h)(i)(j)(k)(l)Figure 8
Fusion results of “Kaptein man” with different fusion methods: (a) GFF, (b) CP, (c) TSSD, (d) GTTV, (e) CSR, (f) WLS, (g) MVFMF, (h) CBF, (i) TEMTD, (j) LLRR, (k) CoF, and (l) GF-ICoF.
(a)(b)(c)(d)(e)(f)(g)(h)(i)(j)(k)(l)Figure 9
Fusion results of “sandpath” with different fusion methods: (a) GFF, (b) CP, (c) TSSD, (d) GTTV, (e) CSR, (f) WLS, (g) MVFMF, (h) CBF, (i) TEMTD, (j) LLRR, (k) CoF, and (l) GF-ICoF.
(a)(b)(c)(d)(e)(f)(g)(h)(i)(j)(k)(l)Figure 10
Fusion results of “Kaptein rain” with different fusion methods: (a) GFF, (b) CP, (c) TSSD, (d) GTTV, (e) CSR, (f) WLS, (g) MVFMF, (h) CBF, (i) TEMTD, (j) LLRR, (k) CoF, and (l) GF-ICoF.
(a)(b)(c)(d)(e)(f)(g)(h)(i)(j)(k)(l)Figure 11
Fusion results of “soldiers with jeep” with different fusion methods: (a) GFF, (b) CP, (c) TSSD, (d) GTTV, (e) CSR, (f) WLS, (g) MVFMF, (h) CBF, (i) TEMTD, (j) LLRR, (k) CoF, and (l) GF-ICoF.
(a)(b)(c)(d)(e)(f)(g)(h)(i)(j)(k)(l)The GFF algorithm can enhance the visual qualities of the fusion image by employing appropriate weights on the corresponding regions. Although the additional structure and detailed data could be transferred from the guidance image to the fused one via the GFF method, the images around certain edges could not be described appropriately through the local liner model in this method. Although no explicit constraint occurs from the edge-aware factor to treat edges, it may generate specific halos in the fused images. A few blurring artifacts could be observed due to the GFF fusion such as the lost data of grass (boxes II and III) in Figure7(a), the blurred branches (box I) in Figure 8(a), the reduced brightness of the trunk portion (box II) of the tree in Figure 9(a), the reduced brightness of the tent (box II) in Figure 10(a), the lost cloud information of the sky background (box I) in Figure 11(a), and the blurred street lamp (box I) and blurred building (box II) in Figure 12(a).Figure 12
Fusion results of “soldiers with jeep” with different fusion methods: (a) GFF, (b) CP, (c) TSSD, (d) GTTV, (e) CSR, (f) WLS, (g) MVFMF, (h) CBF, (i) TEMTD, (j) LLRR, (k) CoF, and (l) GF-ICoF.
(a)(b)(c)(d)(e)(f)(g)(h)(i)(j)(k)(l)CP decomposed the source image into some decomposition layers with various resolution and spatial frequencies. CP has achieved a good visual effect except for the lack in adaptability. The CP fusion results show some blurring, such as the low contrast in Figure7(b), blurred branches (box I) in Figure 8(b), the black block appearing in the lower right corner (box III) in Figure 9(b), the low brightness of the jeep and soldiers in Figure 11(b), and the blurred street lamp (box I) and the blurred leg of the crowds (box II) in Figure 12(b). TSSD used a mean and median filter to extract visual saliency extraction. The weight map obtained from the saliency map guided the fusion of the base and detail layers, which greatly improves the transfer ability of complementary data from the original images effectively. But the ability of the mean and median filter in saliency extraction compromised the fusion performance, such as the lost surrounding fence and grass information (boxes II and III) in Figure 7(c), the blurred grass around (box II) in Figure 8(c), the blurred trunk portion (box II) of the tree in Figure 9(c), the low brightness of the tent (box II) in Figure 10(c), the low brightness of the jeep and soldiers (box II) in Figure 11(c), and the low brightness of the crowds (box II) and street lamp (box I) in Figure 12(c).The fusion process is converted to al1-TV minimization problem through the GTTV in which the fundamental intensity distribution in the infrared image and the gradient change in the visible image are preserved through the data fidelity and regularization terms, respectively. Moreover, the thermal radiation and the appearance data in the infrared and visible images could be preserved, respectively (except for the lack of adaptability). The blur appears such as the blurred branches (box I) in Figure 8(d), the blurred trunk (box II) of the tree in Figure 9(d), the lost information of the tent in Figure 10(d), the blurred jeep and soldiers (box II) in Figure 11(d), and the blurred crowds (box II) in Figure 12(d).CSR used convolutional sparse representation to overcome detail preservation and strong sensitivity to misregistration in image fusion. It has obtained better fusion performance for infrared and visible image. But the regularization parameters in the CSR model and the spatial size of dictionary filters lack in self-adaptability, which compromise the fusion efficiency such as the brightness of the target person (box I) in Figure7(e), the dark background in Figure 8(e), the low brightness of the trunk portion (box II) of the tree in Figure 9(e), the low brightness of the jeep and soldiers (box II) in Figure 11(e), and the low brightness of the street lamp (box I) in Figure 12(e).WLS employs the RGF and Gaussian filter to decompose input images to base and detail layers. WLS constructs weight maps by choosing various features of the IR and visible image. Effective visual information could be appropriately transformed into the fused image through the WLS while reducing the noise from the IR image. But the lack of parameter flexibility compromises the fusion quality, such as the low brightness of the target person (box I) in Figure7(f), the dark background in Figure 8(f), the blurred trunk portion (box II) of the tree in Figure 9(f), the low-contrast vehicle and soldiers (box II) in Figure 11(f), and the low-contrast street lamp (box I) and crowds (box II) in Figure 12(f).Three visual property measurements are utilized by the MVFMF to generate decision maps that are optimized through gradient domain-guided filtering. The visual description of the detailed data of the original images was provided through it. Although the fusion efficiency was significantly enhanced via the MVFMF decision map construction model, the feasibility of the constant parameters for all original images with various values is not satisfactory in some instances. Slight blurs and artifacts could be observed in the fusion images such as the lost information of the grass (boxes II and III) in Figure7(g), the blurred branches (box I) in Figure 8(g), the low brightness of the trunk portion (box II) of the tree in Figure 9(g), the information of the tent (box II) in Figure 10(g), the lost cloud information of the sky background (box I) in Figure 11(g), and the lost building information of the street lamp (box I) in Figure 12(g).The CBF extracts the detail information by using joint bilateral filtering and avoids the gradient reversal artifacts of the bilateral filter. It uses weighted average of pixel values to the fuse source image and improves the visual quality of the fused image. However, the weight construction may compromise the contrast of the fused image. The little blurs and the corresponding artifacts appear in the fused images such as the blurred target person (box I) in Figure7(h), the blurred edge of the tree (box I) and the blurred figure of the man in Figure 8(h), the blurred trunk portion (box II) of the tree in Figure 9(h), the vehicle and soldiers (box II) in Figure 11(h), and the street lamp (box I) and crowds (box II) in Figure 12(h).TEMTD is a novel infrared and visible image fusion method based on target-enhanced MST decomposition. It deals with problem regarding the preservation of thermal radiation features in the traditional MST-based methods. It can simultaneously maintain thermal radiation characteristics in the infrared image and texture details in the visible image by using a specific fusion rule design. It uses the decomposed infrared low-frequency information to determine the fusion weight of low-frequency bands and highlight the target. The common “max-absolute” fusion rule is performed for fusion for high-frequency bands. The common “max-absolute” fusion rule may compromise the fusion performance, such as the blurred bush (boxes II and III) in Figure7(i), the blurred branches (box II) in Figure 9(i), the blurred bush in the left bottom of Figure 10(i), the blurred sky in Figure 11(i), and the street lamp in Figure 12(i).LLRR is a novel fusion framework for fusing infrared and visible images. A projection matrixL learned by latent low-rank representation is applied to extract detail parts and base parts of the input images at several representation levels. It extracts multilevel salient features by using latent low-rank representation. The final fused image is reconstructed by adaptive fusion strategies designed specifically for dealing with the detail parts and the base parts, respectively. The LLRR framework can be used to provide an efficient decomposition approach for extracting multilevel features for an arbitrary number of input images. The adaptive fusion rules improve the ability in texture and structural information extraction. The bush in Figure 7(j), the branch in Figure 9(j), the bush in Figure 10(j), and the soldiers and jeep in Figure 11(j) all demonstrate high contrast. But there are information losses appear in the fused image, such as the black edge around the target man in Figures 7(j), 8(j), and 12(j).The co-occurrence filter features of boundary detection and edge preservation could be utilized by the CoF for weight optimization, which improves the fusion quality of the base layers and detail layers. The same fusion rule of base and detail layers compromises the fusion performance, such as the lost information of the grass (boxes II and III) in Figure7(k), the blurred grass (box II) in Figure 8(k), the blurred details of the branches (box I) in Figure 9(k), the blurred grass at the lower right corner (box II) in Figure 10(k), the lost cloud information of the sky background in Figure 11(k), and the lost information of the street lamp in Figure 12(k). We perform different image filtering for saliency extraction based on the visual feature of base layers and detail layers. By and large, compared with other algorithms, the proposed algorithm not only retains the saliency desired data of the infrared image but also preserves the background data in the visible image.


## 4.3. Quantitative Assessments

In the current subsection, seven quality measurement indices (as discussed in the previous subsection) are applied to the test images. The obtained results are presented in Tables2–6. Tables 2–6 demonstrate that the metric values of the presented approach are higher than the corresponding ones obtained with existing approaches, which demonstrates that further data could be preserved from the original images, and superior fusion quality could be obtained.Table 2
Fusion evaluation indicators for different algorithms for “NATO camp” and “Kaptein man.”
MethodNATO campKaptein manQAB/FNFMIVIFFENMS-SSIMSDQAB/FNFMIVIFFENMS-SSIMSDGFF0.5330.9030.1736.6630.70427.4330.5450.9420.2806.8580.87932.898CP0.4810.8860.4036.8240.87032.7760.4550.9350.3336.6320.91934.143TSSD0.4340.8890.3986.4690.92726.3800.4480.9320.3716.6480.93333.347GTTV0.4050.8850.2186.6940.78727.3800.4080.9060.3166.6830.83638.173CSR0.4630.8980.3316.3350.89024.4620.3140.9370.1846.9570.79832.259WLS0.4170.8910.3506.3740.90624.4790.5800.9280.2136.9670.83132.254MVFMF0.5110.9040.2226.5630.72126.4820.5110.9420.3086.5930.91035.139CBF0.3890.8710.2736.4500.74026.0160.3880.8970.1996.7770.73333.193TDMTD0.4020.8850.1756.6000.75226.7270.4680.9380.2256.7660.84335.712LLRR0.3120.8860.4146.7790.84931.3430.3280.9340.3507.1540.85845.318CoF0.3900.8900.3306.7330.83130.1300.4060.9380.2877.1390.89250.986GF-ICoF0.5470.9160.4246.7900.92632.0200.5620.9420.3577.1720.92351.693Table 3
Fusion evaluation indicators for different algorithms for “sandpath” and “Kaptein rain.”
MethodSandpathKaptein rainQAB/FNFMIVIFFENMS-SSIMSDQAB/FNFMIVIFFENMS-SSIMSDGFF0.5300.8530.2566.8400.88633.6540.5850.9110.3767.0270.88948.277CP0.4700.8300.3186.7350.88327.1180.5130.8920.4416.8070.86337.091TSSD0.4160.8390.3546.3420.93121.7790.4930.8910.4166.4420.92530.622GTTV0.4950.8410.1446.5100.82522.3120.4000.8810.1606.5240.74528.765CSR0.4710.8550.2576.2240.89520.1240.5250.9000.3406.3540.89628.560WLS0.3850.8450.2966.2440.89920.1880.4700.8930.3476.4000.89728.664MVFMF0.5900.8800.2607.0460.89838.2810.5830.9060.4197.1550.88047.532CBF0.3560.8310.2816.5200.80827.5600.4320.8640.3286.8780.65136.413TDMTD0.3150.8460.0966.1480.67517.4980.3630.8880.1316.1390.69526.014LLRR0.2760.8500.3117.1630.85538.7710.3130.9170.4397.1920.83248.288CoF0.3440.8460.2306.6500.80130.4920.4520.8980.3416.6710.84639.839GF-ICoF0.6020.8830.3247.1640.94738.9650.5980.9200.4897.2620.93748.362Table 4
Fusion evaluation indicators for different algorithms for “soldiers with jeep” and “bike with car.”
MethodSoldiers with jeepBike with carQAB/FNFMIVIFFENMS-SSIMSDQAB/FNFMIVIFFENMS-SSIMSDGFF0.5850.9140.3666.9340.85730.8730.5810.8620.2397.4650.85145.442CP0.2940.8730.4116.7240.79530.3930.5040.8270.3317.3320.81041.419TSSD0.4950.9040.4606.5520.94723.5230.4660.8350.3096.6550.90225.477GTTV0.3240.9060.3957.5360.87556.7950.3210.8500.1337.6000.64748.860CSR0.4900.9130.3476.4710.91922.1470.5130.8560.2506.5480.86123.755WLS0.4580.8970.3886.4940.89626.2530.4700.8400.2596.5520.87023.618MVFMF0.6430.9150.4687.1870.85339.6170.5800.8610.2277.5650.84249.134CBF0.3520.8340.3847.0180.59534.1690.4630.8080.2317.5100.75848.269TDMTD0.5580.9170.5667.2270.84143.1750.4500.8410.1557.1080.68839.293LLRR0.2960.9070.5607.0940.85037.8290.3610.8310.2497.3390.85444.219CoF0.4480.9100.5077.1480.89138.9040.4240.8340.1777.0740.77840.135GF-ICoF0.6110.9200.5667.6280.94858.9190.5690.8560.2587.5360.87749.241Table 5
Fusion evaluation indicators for different algorithms for “crowds” and “backpacker.”
MethodCrowdsBackpackerQAB/FNFMIVIFFENMS-SSIMSDQAB/FNFMIVIFFENMS-SSIMSDGFF0.6270.9150.4267.3570.88647.6920.5930.8690.2917.2450.84437.960CP0.5870.9100.6107.1260.93443.3940.5250.8510.3587.4900.78644.898TSSD0.5130.9010.5306.8840.94239.8690.4840.8520.3676.9380.85631.128GTTV0.3910.9110.2677.2770.81347.8540.4230.8510.2147.3190.71443.100CSR0.5550.9120.4606.8140.92338.9980.5430.8660.3136.8510.86329.101WLS0.5310.9040.4586.8290.92338.8540.4900.8560.3186.8320.86328.546MVFMF0.6380.9150.4667.4120.89949.0640.5970.8680.2737.5050.81844.921CBF0.5260.9060.5047.2800.83944.9990.4470.8080.2737.5440.60645.049TDMTD0.5480.9100.3167.1480.83945.5780.4680.8590.1766.8740.71035.952LLRR0.3840.8990.5077.2920.85049.5980.3950.8450.3267.4220.83945.265CoF0.5030.9090.5446.9890.90341.9520.4610.8510.2186.9780.73144.146GF-ICoF0.6510.9160.5967.4500.92349.6010.5860.8650.3327.4590.86445.591Table 6
Time consumption of nine algorithms (unit: second).
MethodGFFCPTSSDGTTVCSRWLSMVFMFCBFTDMTDLLRRCoFGF-I CoFTime0.6380.1060.1516.651211.0023.1900.6344.0232.963159.16533.8753.158As is known,Q(AB/F) shows the edge data of the fused image retained from the source image. NFMI measures the feature information of the fused image retention from the source one based on mutual information. MS-SSIM is based on structural similarity to measure the retention degree of the structural data of the fused image from the original one. EN indicates the information value contained in the fused image as a whole. SD reflects the contrast of the image with the difference degree between each pixel and the average value of the pixels. VIFF measures the degree of fusion of the effective visual information of each region.Histogram of the average metric (QAB/F, NFMI, VIFF, and MS-SSIM) amounts of the fusion approaches is presented in Figure 13. It is easy to see that the average QAB/F values of GFF, MVFMF, and GF-ICoF are obviously higher than that of other methods, which demonstrates that GFF, MVFMF, and GF-ICoF can well preserve the edge data from original images to fused images. The average NFMI value of GFF, CSR, LLRR, MVFMF, CoF, and GF-ICoF are obviously higher than that of other fusion methods. It demonstrates that more significant feature data could be transformed from the source image to the fused one through the five methods transfer compared with existing fusion approaches. Moreover, the mean VIFF and MS-SSIM values of TSSD and GF-ICoF are obviously greater than that of existing fusion approaches. It shows that TSSD and GF-ICoF can transfer more significant structure data from original images compared with existing fusion techniques.Figure 13
Histogram of the average metric (QAB/F, NFMI, VIFF, and MS-SSIM) values of the fusion methods.Histogram of the mean metric (EN, SD) amounts of the fusion approaches is depicted in Figure14. It is easy to see that the average SD values of MVFMF, LLRR, CoF, and GF-ICoF are obviously greater than the corresponding values obtained by other fusion approaches. It demonstrates that the fusion results of MVFMF, LLRR, CoF, and GF-ICoF have higher contrast with better visual quality compared with existing fusion algorithms. The mean EN values of GFF, MVFMF, CoF, and GF-ICoF are greater than that of existing fusion techniques. The comparison results of the fused image quality evaluation indices demonstrate the superiority of the developed fusion approach. The proposed method has sufficient retention for the edge and structure data from the original image. The fused image obtained with the presented approach has the best visual quality in terms of contrast. Overall, the approach is preferred to existing algorithms in accordance with different evaluation metrics.Figure 14
Histogram of the average metric (EN and SD) values of the fusion methods.The time complexity of each algorithm is obtained on the running time of 10 tests. The selected image size is270∗360. The results are shown in Table 6. It is evident that the time consumption of CSR is highest. The parameter training and dictionary learning cost much time. The extraction of detail parts and base parts of the source images is implemented by learning with latent low-rank representation, which results in higher time consumption of LLRR. The algorithm models of CP, TSSD, and GFF are simple and small time consumptive. The time consumption of WLS, CBF, TDMTD, and MVFMF is a little more than the previously mentioned methods. CSR spends a lot of time in sparse representation, which leads to excessive time overhead. The time cost of the propose method and the GTTV algorithm tends to be similar. Because the time consumption of the CoF is a little more, the CoF fusion algorithm uses an iterative way to fuse the base layer, which further increases the time spent for the fusion. Compared with the CoF fusion algorithm, the time efficiency of this work is increased by about 90%. In terms of the subjective and objective results, we can conclude that the algorithm is an effective infrared and visible image fusion algorithm.


## 5. Conclusions

As conventional infrared and visible image fusion methods suffer from low contrast and background texture loss, a novel fusion approach is presented using guided and improved co-occurrence filters. The advantage of the guided filter in background information extraction and the advantage of the co-occurrence filter in edge structure information extraction are combined to improve the fusion performance of the infrared and visible image. The co-occurrence filter is improved by removing the range filter and globally synthesizing the co-occurrence information. The filtering time of the co-occurrence filter is reduced by half while preserving across texture edges. The qualitative assessments demonstrate that the fusion results of the proposed method can retain the thermal radiation and appearance data in the infrared and visible images, respectively. The quantitative comparisons on seven metrics with recent fusion approaches indicate that more significant edge and structure data could be transformed from the original image to the fused one through the presented approach. Future work will further improve the speed of the proposed method and apply it to image fusion applications such as medical and remote sensing.



---

*Source: 1757214-2020-07-29.xml*