A Novel Fusion Algorithm for Visible and Infrared Image using
Non-subsampled Contourlet Transform and Pulse-coupled Neural
Network
Chihiro Ikuta1 , Songjun Zhang2 , Yoko Uwate1 , Guoan Yang3 and Yoshifumi Nishio1
1 Department of Electrical and Electronic Engineering, Tokushima University, 2-1 Minami-Josanjima, Tokushima, Japan
2 Department of Computing Mathematics School of Science, Xi’an Jiaotong University,

No.28 Xianning West Road, Xi’an City, Shaanxi Province, China
3 Department of Automation Science and Technology, School of Electronic and Information Engineering,

Xi’an Jiaotong University, No.28 Xianning West Road, Xi’an City, Shaanxi Province, China{ikuta, uwate, nishio}@ee.tokushima-u.ac.jp, youyi000@gmail.com, gayang@mail.xjtu.edu.cn

Keywords:

Image Fusion, Visible Image, Infrared Image, Pulse Coupled Neural Network, Non-subsampled Contourlet
Transform.

Abstract:

An image fusion algorithm between visible and infrared images is significant task for computer vision applications such as multi-sensor systems. Among them, although a visible image is clear perfectly able to be
seen through the naked eyes, it is often suffers with noise; while an infrared image is unclear but it has highanti-noise property. In this paper, we propose a novel image fusion algorithm for visible and infrared images
using a non-subsampled contourlet transform (NSCT) and a pulse-coupled neural network (PCNN). First, we
decompose two original images above mentioned into low and high frequency coefficients based on the NSCT.
Moreover, each low frequency coefficients for both images are duplicated at multiple scales, and are processedby laplacian filter and average filter respectively. Finally, we can fuse the normalized coefficients by using the
PCNN. Conversely, we can reconstruct a fused image based on the low and high frequency coefficients, which
are fused by using the inverse NSCT. Experimental results show that the proposed image fusion algorithm
surpasses the conventional and state-of-art image fusion algorithm.

1

INTRODUCTION

Image fusion plays an important role in the computer
vision and image processing fields. In recent years,
many image fusion algorithm are applied to computer vision, pattern recognition and image processing fields such as multi-focus and multi-sensors image fusion, and so on (Xu and Chen, 2004) (Wanget al., 2008) (Qu et al., 2008). Especially, an image
fusion algorithm between visible and infrared images
is significant for computer vision and image processing applications. The contourlet transform is a new
two-dimensional extension of the wavelet transformusing multi-scale and directional filter banks (Yanget al., 2010). And then, a non-subsampled contourlet
transform (NSCT) is developed by Da Cunha, Zhou
and Do (da Cunha et al., 2006). The NSCT has afully shift invariant property than the contourlet, leads
to better frequency selectivity, directivity and regularity (Zhou et al., 2005). On the other hand, we knowthat a pulse-coupled neural network (PCNN) is pre-160

sented by Eckhorn in 1990 (Eckhorn, 1990). Thismethod is developed based on the experimental observations of synchronous pulse bursts in cat cortex.
It is characterized by the global coupling and pulse
synchronization of neurons. And the PCNN has excellent performance in image edge detection applications. Recently, several image fusion algorithm basedon the NSCT and PCNN have been developed, for example, based on spatial frequency-motivated PCNN
in NSCT domain of Qu (Qu et al., 2008), stationary
wavelet-based NSCT and PCNN of Yang (Yang et al.,2009), based on NSCT-PCNN of Ge for visible andinfrared image (Ge and Li, 2010), a simplified PCNN
in NSCT domain of Liu (Liu et al., 2012), and so on.
These image fusion algorithm implemented better fusion performance for various image processing applications.
However, hardly any work based on NSCT-PCNN
algorithm for the visible and infrared image. Therefore, in this paper, we consider to utilize the NSCT for
implementing multi-scale decomposition, and PCNN

ANovelFusionAlgorithmforVisibleandInfraredImageusingNon-subsampledContourletTransformandPulse-coupled
NeuralNetwork

for implementing image fusion. Furthermore, we propose a novel images fusion algorithm for visible andinfrared image based on the NSCT and PCNN. And
the quality of a fusion image is improved by using
the PCNN for the low frequency coefficients in NSCT
domain. In order to the purpose, we employed two
different filters, which are average filter and laplacian
filter before image fusion using the PCNN, and better
image fusion performance is demonstrated than basedon the PCNN only.
This paper is organized as follows. In section
2, we describe an image fusion algorithm used the
NSCT and PCNN. In section 3, we propose the newimage fusion algorithm for visible and infrared images. In section 4, we show the experimental results,
and compare with the conventional method about image fusion. Finally, in section 5, we state the conclusion and future works.

2

Image
Multi-scale
Decomposition

Figure 1: The non-subsampled contourlet transform for theimage decomposition.

fusion (Jhonson and Padgett, 1999). When we apply
the PCNN to an image processing, one pixel data in
an image inputs the one neuron. Accepters of neuron
are composed of a feeding part and a linking part, as
follows. The feeding part is described by Eq. (1).
Filkj (n) = Silkj ,IMAGE FUSION ALGORITHM

Generally, the image fusion algorithm above mentioned mainly has three steps. In the first step, images
are decomposed into some coefficients by the NSCT
decomposition algorithm. In the next step, the coefficients from two images are compared and fused by
the PCNN. Finally, the fused coefficients reconstruct
into one new image by the NSCT reconstruction algorithm.

2.1 NSCTThe NSCT is compatible with the image fusion byusing the PCNN, because this method does not exist the up sampling and down sampling (Zhou et al.,2005). The NSCT use pyramid filter banks to implement a multi-scale decomposition and a directional
decomposition as shown in Fig. 1 (da Cunha et al.,2006). In the case of multi-scale decomposition, an
image is decomposed into the low frequency coefficients and high frequency coefficients, and these operations are repeated at different scale. After that,the coefficients is decomposed into directional components in the high frequency banks.

2.2 PCNN
Eckhorn proposed a neural network model from cat’svisual cortex in 1990 (Eckhorn, 1990). The PCNN is
proposed by Thomas for the application of this mechanism in 1998 (Lindblad and Kinser, 2005). It is
known that this method can be applied to the image
processing, in addition can also be applied to image

Directional
Decomposition(1)

where F is an output of feeding part and S is anexternal stimulus. Eq. (2) is the linking part.
Llk
i j (n) =e−αL Llk
i j (n − 1)
+VL ∑ pq Wilkj,pqYilkj,pq (n − 1),(2)

where L is an output of linking part, V is a normalization coefficient and W is a weight to connect otherneurons. An internal state is calculated by outputs ofthe feeding and linking part as shown in Eq. (3).
Uilkj (n) = Filkj (n){1 + βLlk
i j (n)},(3)where U is an internal state. The output of neuron is
calculated from comparison between the internal state
and a threshold. The threshold is described by Eq. (4).
−αθ θlk (n − 1) + V Y lk (n − 1),
θlk
θ ij
ij
i j (n) = e(4)where θ is a threshold. In the PCNN, an activation
function is a step function. The output of the neuron
is described by Eq. (5).
n1,
Uilkj (n) > θlk
i j (n)
Yilkj =
(5)0,otherwise.
We choose these parameters by heuristic. In thiscondition, we tune that the conventional method has
become more high performance, and the model of
PCNN shown in Fig. 2.
In the simulation, the PCNN is inputted in the decomposition coefficients of two images. In the application for the image processing, the one neuron receive one pixel of the coefficients. Thus, the number of neurons depend on the image size. The PCNN
iterates process to the number of times. After that,161

VISAPP2014-InternationalConferenceonComputerVisionTheoryandApplicationsܹ ௡ ௜௝,௞௟Linking

…

ܻ௜௝






0.11 0.11 0.11
Average f ilter =  0.11 0.12 0.11 
0.11 0.11 0.11

1 + ߚ‫ܮ‬௜௝

ߙ௅

ܸఛ
ߙఛ

1
ܻ௜௝

‫ܯ‬௡ ௜௝,௞௟ߠ௜௝ －Feeding…

Step function

‫ܨ‬௜௝

ܻ௜௝

ܷ௜௝ ＋

ߙி
ܵ௜௝

Figure 2: Block diagram of a single neuron of the PCNN.

the two images which are processed by the PCNN arecompared and are fused by Eq. (6).
n Alk ,
Yilkj,A > Yilkj,Bij
Iilkj =
(6)lk
Bi j ,
otherwise,
where I is an fused image, A is coefficients of the image A, and B is coefficients of the image B.

3

PROPOSED IMAGE FUSIONALGORITHM

In this paper, we introduce a novel image fusion processing for the low frequency coefficients. Generally, the high frequency region of an image include
the edge and texture information, the low frequency
region concentrate almost of power for an image.
Whereas image energy of the most natural scenes
is mainly concentrated in the low frequency region.
Hence, we consider that the quality of an image canbe improved by tuning the low frequency of image.
Process of proposed image fusion algorithm is illustrated in Fig. 3. Firstly, we decompose two image
into the low frequency coefficients and the high frequency coefficients by the NSCT. In addition, each
low frequency coefficients for both images are duplicated at multiple scales. And then, on one hand,the coefficients is filtering by the laplacian filter expressed in Eq. (7). The laplacian filter is an edge detection operator, it can enhance the edge effects of animage. On the other hand, the coefficient is filtered by
the average filter shown in Eq. (8). The average filter
simplified the all of low coefficients, and can fuse theboth coefficients which is normalized from the laplacian and average filter for increasing the intensity ofan image in the low frequency domain.
−1 −1 −1
Laplacian f ilter =  −1 9 −1 (7)
−1 −1 −1162

(8)After filter process, we normalize the two coefficients.
The two coefficients are fused by the PCNN image
fusion algorithm. We process coefficients of image A
and image B, and the two fused coefficients are fusedby using Eq. (6). As shown in Fig. 3, the images aredecomposed five levels by the pyramidal filter bankof NSCT. Thereby, the coefficient of five different
frequency bands are generated, respectively. Amongthem, we mainly used filtering technique for the low
frequency bands, which are two lower frequency levels. There realized the fusion processing for visible
image and infrared image using the algorithm above.
Finally, the both fused coefficient above mentioned
from two different images are fused by the PCNN,
and the high frequency coefficients are further decomposed to be directional band coefficient which fusedby the PCNN, it is called as the second fusion for visible image and infrared image separately. Therefore,
we can reconstruct the visible and infrared image using all the low and high frequency coefficient by the
inverse NSCT. We investigated that the quality ofa fused image is improved by filtering the low frequency coefficients. It is because that laplacian filter
and average filter increased an intensity of image inthe low frequency.

4

SIMULATIONSIn this section, we show the computer simulation of
proposed image fusion algorithm. Here use two kindsof image sets, and investigate the image fusion performance. Furthermore, we analyze the quality of image
fusion for the proposed algorithm with the low frequency coefficients to the PCNN and high frequency
coefficients to the PCNN. And that parameters of the
PCNN are set as p × q = 3 × 3, αL = 1.0, αθ = 0.2,
β = 3, VL = 1.0, Vθ = 20,
0.707 1 0.707
01 ,
W = 10.707 1 0.707
and the maximal iteration number is n = 200. Here,
we adopted three kinds of methods exist for evaluating image fusion quality, which are the Mutual Information (MI), entropy, and Standard Deviation (St.
Dev.). The MI shows that the fused image might havevaluable information of both original images. The entropy shows that the fused image carries average information. The St. Dev. shows the statistical distribution of fused image. In the first simulation, we use

ANovelFusionAlgorithmforVisibleandInfraredImageusingNon-subsampledContourletTransformandPulse-coupled
NeuralNetwork

Figure 3: The proposed image fusion algorithm.

the original images in Fig. 4, and the fused images
are shown in the Fig. 5. From the Fig. 5, we can seethat the original images are fused well and its visualperformance is as good as original images. .

(a) Visible image.

(b) Infrared image.

Figure 4: Target images for image fusion.

hancements of the original coefficients by using theboth filters, hence the mutual information, entropy
and standard deviation are much improved, respectively. Especially, the proposed method has a highMI performance and a high standard deviation of the
fused image. In the proposed model, the coefficientsare preprocessed by the filters, thereby the original
image is varied. However, the result of the proposed
method obtains the high MI performance. The standard deviation generally shows the edge performance.
The PCNN responds the edge. The proposed methodenhances the edge by the laplacian filter from the original image coefficients. In the conventional method,
the fused image is made from original images. Thus,
the fused image has only characteristics of the original image.
Table 1: Fusion performance index of different methods.
PCNN+NSCT
Proposed(a) PCNN with NSCT.

(b) Proposed algorithm.

Figure 5: Fused image by different methods.

Table 1 shows the experimental results and comparisons for the image fusion performance in theFig. 4. From this results we can see that the proposed image fusion algorithm has better quality of
fused image than the PCNN. It means that the original images held the characteristics even if the original
images are processed by two filters, namely laplacianand average filters. We focus on the low frequency
coefficients for the localized property and edge en-MI
2.49562.8358

Entropy
7.03007.1275

St. Dev.
32.71334.865In the next simulation, we use the test images andcan obtain the fused images as shown in Fig. 6 andFig. 7 respectively. From the Fig. 7, we can see thatthe original images are fused well and its visual performance is as good as original images.
From Table 2, the proposed image fusion algorithm has better quality of fused image than the image
fusion of the PCNN. The mutual information and thestandard deviation are improved well. This result’s
trend is similar to the Table. 1. Thereby, we can saythat the proposed method has a better image fusion
performance than the conventional method.
Table 2: Fusion performance index of different .
PCNN+NSCT
ProposedMI
5.58016.0821

Entropy
7.48627.5491

St. Dev.
53.45454.669163

VISAPP2014-InternationalConferenceonComputerVisionTheoryandApplications(a) Visible image.

(b) Infrared image.

Figure 6: Target images for image fusion.

(a) PCNN with NSCT.

(b) Proposed algorithm.

Figure 7: Fused image by different methods.

5

CONCLUSIONSIn this paper, we have proposed the novel image fusion algorithm for visible and images by using NSCTand PCNN. We processed the low frequency coefficients which are decomposed by the NSCT, in addition filtered by the laplacian and the average filter.
There, the coefficients were fused by the PCNN. Finally, the fused image is reconstructed using the low
frequency and high frequency coefficient by the inverse NSCT. From experimental results, the proposed
image fusion algorithm has better quality than the image fusion performance for using the PCNN only.

REFERENCES
da Cunha, A., Zhou, J., and Do, M. (2006). The nonsubsampled contourlet transform: Theory, design, and applications. IEEE Transaction on Image Processing,15(10):3089–3101.
Eckhorn, R. (1990). Feature linking via synchronization
among distributed assemblies: Simulations of result
from cat visual cortex. Neural Computation, 2:293–
307.
Ge, Y. and Li, X. (2010). Image fusion algorithm basedon pulse coupled neural networks and nonsubsampled contourlet transform. Proc. Second International
Workshop on Education Technology and Computer
Science, 3:27–30.

164

Jhonson, J. and Padgett, M. (1999). Pcnn models and applications. IEEE Transactions of Neural Network,
10(3):480–498.
Lindblad, T. and Kinser, J. (2005). Image Processing using
Pulse-Coupled Neural Networks. Springer Publisher.
Liu, F., Li, J., and Huang, C. (2012). Image fusion algorithm based on simplified pcnn in nonsubsampled
contourlet transform domain. International Workshop on Information and Electronics Engineering,
29:1434–1438.
Qu, X., Yan, J., Xiao, H., and Zhu, Z. (2008). Image fusion algorithm based on spatial frequency-motivatedpulse coupled neural networks in nonsubsampled contourlet transform domain. Acta Automatica Sinica,
34(12):1508–1514.
Wang, M., Peng, D., and Yang, S. (2008). Fusion of multiband sar images based on nonsubsampled contourlet
and pcnn. Proc. 4th International Conference on Natural Computation, pages 529–533.
Xu, B. and Chen, Z. (2004). A multisensor image fusion algorithm based on pcnn. Proc. the 5th World Congresson Intelligent Control and Automation, pages 3679–
3682.
Yang, G., Wetering, H. V. D., Hou, M., Ikuta, C., and Liu, Y.
(2010). A novel design approach for contourlet filterbanks. IEICE Transactions on Information and Systems, E93-D(7):2009–2011.
Yang, S., M. Wang, Y. L., Qi, W., and Jiao, L. (2009).
Fusion of multiparametric sar images based on swnonsubsampled contourlet and pcnn. Signal Processing, 89(12):2596–2608.
Zhou, J., da Cunha, A., and Do, M. (2005). Nonsubsampled
contourlet transform: Construction and application inenhancement. Proc. International Conference on Image Processing, pages 469–472.