# PCNN-Based Image Fusion in Compressed Domain

Authors: Yang Chen; Zheng Qin
Journal: Mathematical Problems in Engineering
 (2015)

---

## Abstract

This paper addresses a novel method of image fusion problem for different application scenarios, employing compressive sensing (CS) as the image sparse representation method and pulse-coupled neural network (PCNN) as the fusion rule. Firstly, source images are compressed through scrambled block Hadamard ensemble (SBHE) for its compression capability and computational simplicity on the sensor side. Local standard variance is input to motivate PCNN and coefficients with large firing times are selected as the fusion coefficients in compressed domain. Fusion coefficients are smoothed by sliding window in order to avoid blocking effect. Experimental results demonstrate that the proposed fusion method outperforms other fusion methods in compressed domain and is effective and adaptive in different image fusion applications.


---

## Body

## 1. Introduction

Image fusion is the processing of combining the complementary information from multiple source images into a fused image which provides more accuracy than any of the source images. Image fusion is widely used in civil, military, and medical image processing. For example, fusion of multifocus images [1, 2] can provide a better view for human or machine perception. Fusion of infrared and visible light [3, 4] can provide a strong ability of discovering important targets with detailed texture expression. In addition, fusion of computed tomography (CT) image and magnetic resonance imaging (MRI) images [5, 6] can provide detailed information on bones structures and soft tissue for diagnosis.Many image fusion methods have been proposed and they can be classified into three levels of information representation, namely, pixel level [7, 8], feature level [9], and decision level [10]. Among these categories of image fusion, the pixel level image fusion is the most effective in terms of conveying more information from multimodal images. Particularly, multiscale transform based methods are the most widely used, such as pyramid [11], gradient [12], wavelet [13–15], and contourlet [16–18].In recent years, CS [19, 20] has become a much preferred algorithm for image fusion and other image processings due to its compression capability in the sampling procedure on the sensor side. Sampling is performed on source images to obtain their linear measurements in the compressed domain. There are different CS sampling patterns addressed in previous CS literature [20–23], such as discrete cosine transform (DCT) [21], star shape, double-star shape, star-circle shape [22], and scrambled block Hadamard ensemble (SBHE) [24]. In the fusion rules of compressive image fusion, weighted fusion factors are calculated by mathematical combinations of image channels. The calculation rules of fusion weighted factors are mainly based on average [25], mean, variance, PCA [26, 27], and mutual information [28, 29]. In addition, fused image in compressed domain can be reconstructed from the measurement according to a recovery algorithm such as gradient projection for sparse reconstruction (GPSR) [30], basis pursuit [31], total variation minimization [32], orthogonal matching pursuit [33], and L1-norm minimization [34, 35].Different from existed compressive image fusion methods, this paper addresses a novel image fusion method by using PCNN in compressed domain. PCNN is a biologically inspired neural network algorithm developed by Eckhorn et al. [36], which has been used in image segmentation, image fusion [2], image enhancement, and pattern recognition. It is characterized by the global coupling and pulse synchronization of neurons. These characteristics benefit image fusion which makes use of local image information. Generally, humans are sensitive to edges or salient information. Therefore, local standard variance, which stands for gradient activity in the local neighbourhood, is used to motivate PCNN neurons in this paper.The remainder of the paper is organized as follows. Section2 provides a brief description of compressive sensing theory. Proposed image fusion methods in compressed domain are illustrated in Section 3. Experimental results and discussions are presented in Section 4 and conclusions are provided in Section 5.


## 2. Compressive Sensing and Sampling Pattern

Compressive sensing theory [19, 20] enables a sparse or compressible signal to be reconstructed from a small number of nonadaptive linear projections, thus significantly reducing the sampling and computation costs.


### 2.1. Background on Compressive Sensing

Consider an unknown signalx∈RN; it is expressed in the following form on the orthogonal base: x=Ψα. For the coefficient vector α, if only K elements are not zero, we say K is sparse when x is in the substrate Ψ. After the measurement matrix M∈RM×N projection, there is y=Mx=MΨα. Since M≪N, the projection process combines the traditional signal sampling and compression process. Through y and M, direct recovery of x is morbid inverse problem. But sparse theory suggests that this problem can be transformed into an optimization problem of sparse vectors to be solved:
(1)min⁡⁡α1s.t.y=Φα,
where *1 represents L1 norm. Thus the signal x can be restored from the formula x=Ψα.Subsequently, measurement matrix is required to satisfy the following properties (RIP) criterion in order to recover signalx accurately:
(2)1-ε≤Φv2v2≤1+ε,
where ε>0, *2 represents norm. v is a N dimensional vector with strictly K sparseness. That is to say, matrix M and substrate Ψ are not related.


### 2.2. SBHE Sampling

In the block CS, an image is divided into small blocks with the size ofNB×B. Sampling operator Φi with size of mi×N is formed by the partial block Hadamard transform with its columns randomly permuted as SBHE operator [24]. The sampling operator Φ is a block diagonal matrix, which is expressed as
(3)H=H1H2⋱HB,
where Hj represents the Hadamard matrices of B×B. Let xi denote the vectorized signal of the ith block. The corresponding measurement output vector is expressed as
(4)yi=Φixi.
Then, the measurement output vector of the entire image is determined by
(5)y=y1y2⋮yI.SBHE is adopted as the sampling pattern in this paper, which has been shown to satisfy the five requirements, including near optimal performance, university, fast computation, being memory efficient, and being Hardware friendly.


## 2.1. Background on Compressive Sensing

Consider an unknown signalx∈RN; it is expressed in the following form on the orthogonal base: x=Ψα. For the coefficient vector α, if only K elements are not zero, we say K is sparse when x is in the substrate Ψ. After the measurement matrix M∈RM×N projection, there is y=Mx=MΨα. Since M≪N, the projection process combines the traditional signal sampling and compression process. Through y and M, direct recovery of x is morbid inverse problem. But sparse theory suggests that this problem can be transformed into an optimization problem of sparse vectors to be solved:
(1)min⁡⁡α1s.t.y=Φα,
where *1 represents L1 norm. Thus the signal x can be restored from the formula x=Ψα.Subsequently, measurement matrix is required to satisfy the following properties (RIP) criterion in order to recover signalx accurately:
(2)1-ε≤Φv2v2≤1+ε,
where ε>0, *2 represents norm. v is a N dimensional vector with strictly K sparseness. That is to say, matrix M and substrate Ψ are not related.


## 2.2. SBHE Sampling

In the block CS, an image is divided into small blocks with the size ofNB×B. Sampling operator Φi with size of mi×N is formed by the partial block Hadamard transform with its columns randomly permuted as SBHE operator [24]. The sampling operator Φ is a block diagonal matrix, which is expressed as
(3)H=H1H2⋱HB,
where Hj represents the Hadamard matrices of B×B. Let xi denote the vectorized signal of the ith block. The corresponding measurement output vector is expressed as
(4)yi=Φixi.
Then, the measurement output vector of the entire image is determined by
(5)y=y1y2⋮yI.SBHE is adopted as the sampling pattern in this paper, which has been shown to satisfy the five requirements, including near optimal performance, university, fast computation, being memory efficient, and being Hardware friendly.


## 3. Proposed Medical Image Fusion Method

### 3.1. Image Fusion Framework in Compressed Domain

In image fusion, pixel level image fusion offers effective fused information at the cost of higher computational complexity. Compressed domain approaches, on the other hand, are more promising due to their compression capability and computational simplicity on the sensor side. Thus, it is important to explore the compressive image fusion method, and the flowchart of image fusion in compressed domain is illustrated as Figure1.Figure 1
The flowchart of image fusion in compressed domain.Firstly, source images are compressed through compressive sensing so as to facilitate the transmission of the sensor. In the fusion phase, fusion rule is used to combine the compressive sensing coefficients. And then inverse transformation is applied to the coefficients derived from the fusion, and the fused image is obtained eventually. In this paper, SBHE is adopted as the sampling pattern and GPSR reconstruction algorithm is used as well.


### 3.2. PCNN Fusion Rule

PCNN model consists of three parts: the dendritic tree, the linking modulation, and the pulse generator. The role of the dendritic tree is to receive the inputs from two kinds of receptive fields, the linking and the feeding. The linking receives local stimulus from the output of surrounding neurons, while the feeding receives local stimulus and external stimulus. We adopt PCNN as the fusion rule and the flowchart of the proposed PCNN-based image fusion in compressed domain is shown in Figure2. The procedure of the proposed method is summarized in Algorithm 1.Figure 2
The flowchart of PCNN-based image fusion in compressed domain.Algorithm 1 (PCNN-based image fusion in compressed domain).
(1) Decompose the source images by SBHE sampling pattern.
(2) Measure the local standard varianceS in sliding window with (6).
(3) Motivate PCNN with local standard varianceS, generate pulse of neurons with (8), and calculate the firing times T with (9).
(4) Calculate the decision mapD with (10), and weighted factors are smoothed by the sliding window with (11) and (12).
(5) Fusion coefficients are obtained with weighted factors of the compressed image.
(6) Reconstruct the fused image from the fusion coefficients by GPSR algorithm.We use the local standard variance to motivate the PCNN model. The local standard varianceSr,c at the point (r,c) of the source image is calculated as
(6)Sr,c=sqrt∑l,k∈Ωr,cIr-l,c-k-Ir,c-2,(7)Ir,c-=∑l,k∈Ωr,cIr-l,c-kM,
where Ir,c represents the gray value of the point (r,c) at the jth block of the image and M denotes the number of elements inΩ(r,c), which denotes the neighbourhood of the point (r,c). Sr,c measures the local standard variance in Ω(r,c). We use the local standard variance to motivate PCNN, and corresponding PCNN model can be expressed as
(8)Frc=Sr,c,Lrc=e-αLLrcn-1+VL∑pqWrc,pqYrc,pqn-1,Urcn=Frcn*1+βLrcn,θrcn=e-αθθrcn-1+VθYrcn-1,Yrcn=1,ifUrcn>θrcn0,otherwise,(9)Trcn=Trcn-1+Yrcn,
where the feeding input Frc equals the local standard variance Sr,c. The linking input Lrc equals the sum of neurons firing times in the linking range with the range size of (p,q). Wrc,pq is the synaptic gain, and αL denotes the decay constants. VL and Vθ are the amplitude gain. β is the linking strength. Urc is total internal activity and θrc is the threshold. n denotes the iteration times. Trc(n) denotes firing times at iteration n at the point (r,c). The neuron generates a pulse, that is, Yrc = 1, when Urc is larger than θrc. Due to the reason that neighbouring coefficients with similar features represent similar firing times in given iteration times, firing times, that is, Trc(n), are used to represent image information, which is defined as (9).The decision mapD is used to decide weighted factors of the source images in the fused image, which can be calculated in (10). The weighted factors at the jth block are smoothed by the sliding window, which are calculated in (11) and (12). Coefficients with larger firing times have more weighted factors in the fused image:
(10)Drc=1,ifTrc1n≥Trc2n0,otherwise,(11)w1=∑r,cDrc∑r,c1,(12)w2=1-w1.


## 3.1. Image Fusion Framework in Compressed Domain

In image fusion, pixel level image fusion offers effective fused information at the cost of higher computational complexity. Compressed domain approaches, on the other hand, are more promising due to their compression capability and computational simplicity on the sensor side. Thus, it is important to explore the compressive image fusion method, and the flowchart of image fusion in compressed domain is illustrated as Figure1.Figure 1
The flowchart of image fusion in compressed domain.Firstly, source images are compressed through compressive sensing so as to facilitate the transmission of the sensor. In the fusion phase, fusion rule is used to combine the compressive sensing coefficients. And then inverse transformation is applied to the coefficients derived from the fusion, and the fused image is obtained eventually. In this paper, SBHE is adopted as the sampling pattern and GPSR reconstruction algorithm is used as well.


## 3.2. PCNN Fusion Rule

PCNN model consists of three parts: the dendritic tree, the linking modulation, and the pulse generator. The role of the dendritic tree is to receive the inputs from two kinds of receptive fields, the linking and the feeding. The linking receives local stimulus from the output of surrounding neurons, while the feeding receives local stimulus and external stimulus. We adopt PCNN as the fusion rule and the flowchart of the proposed PCNN-based image fusion in compressed domain is shown in Figure2. The procedure of the proposed method is summarized in Algorithm 1.Figure 2
The flowchart of PCNN-based image fusion in compressed domain.Algorithm 1 (PCNN-based image fusion in compressed domain).
(1) Decompose the source images by SBHE sampling pattern.
(2) Measure the local standard varianceS in sliding window with (6).
(3) Motivate PCNN with local standard varianceS, generate pulse of neurons with (8), and calculate the firing times T with (9).
(4) Calculate the decision mapD with (10), and weighted factors are smoothed by the sliding window with (11) and (12).
(5) Fusion coefficients are obtained with weighted factors of the compressed image.
(6) Reconstruct the fused image from the fusion coefficients by GPSR algorithm.We use the local standard variance to motivate the PCNN model. The local standard varianceSr,c at the point (r,c) of the source image is calculated as
(6)Sr,c=sqrt∑l,k∈Ωr,cIr-l,c-k-Ir,c-2,(7)Ir,c-=∑l,k∈Ωr,cIr-l,c-kM,
where Ir,c represents the gray value of the point (r,c) at the jth block of the image and M denotes the number of elements inΩ(r,c), which denotes the neighbourhood of the point (r,c). Sr,c measures the local standard variance in Ω(r,c). We use the local standard variance to motivate PCNN, and corresponding PCNN model can be expressed as
(8)Frc=Sr,c,Lrc=e-αLLrcn-1+VL∑pqWrc,pqYrc,pqn-1,Urcn=Frcn*1+βLrcn,θrcn=e-αθθrcn-1+VθYrcn-1,Yrcn=1,ifUrcn>θrcn0,otherwise,(9)Trcn=Trcn-1+Yrcn,
where the feeding input Frc equals the local standard variance Sr,c. The linking input Lrc equals the sum of neurons firing times in the linking range with the range size of (p,q). Wrc,pq is the synaptic gain, and αL denotes the decay constants. VL and Vθ are the amplitude gain. β is the linking strength. Urc is total internal activity and θrc is the threshold. n denotes the iteration times. Trc(n) denotes firing times at iteration n at the point (r,c). The neuron generates a pulse, that is, Yrc = 1, when Urc is larger than θrc. Due to the reason that neighbouring coefficients with similar features represent similar firing times in given iteration times, firing times, that is, Trc(n), are used to represent image information, which is defined as (9).The decision mapD is used to decide weighted factors of the source images in the fused image, which can be calculated in (10). The weighted factors at the jth block are smoothed by the sliding window, which are calculated in (11) and (12). Coefficients with larger firing times have more weighted factors in the fused image:
(10)Drc=1,ifTrc1n≥Trc2n0,otherwise,(11)w1=∑r,cDrc∑r,c1,(12)w2=1-w1.


## 4. Experimental Results

### 4.1. Experiment Setup

We conduct the experiment on six image datasets with three application scenarios. These fusion applications are multifocus image fusion, infrared and visible light image fusion, and medical image fusion. In addition to visual comparison, we also present fusion results with three objective metrics: information entropy (IE), mutual information (MI) [29], and QAB/F [37]. We compare these image fusion methods in compressed domain, with sampling rate set at 0.5. In addition, we use SBHE as the sampling pattern and adopt GPSR as the CS reconstruction algorithm. The SBHE block size is set at 32×32. In this way, the block Hadamard sampling matrix can encode pixels with 32×32. All the source images can be obtained at http://www.imagefusion.org and http://www.med.harvard.edu/aanlib/home.html.


### 4.2. Experimental Result

#### 4.2.1. Multifocus Image Fusion

Due to the limited focus depth of the optical lens, it is not possible to get an image that contains all relevant objects in focus. In this way, multifocus image fusion is required to fuse the images taken from the same view point under different focal settings, in order to provide a fused image with a better description of the scene than any individual image. The obtained fusion image shows all the details of the source image, making it more conducive to the follow-up treatment. We implement CS image fusion based on six fusion methods on pixel level combined with compressive sensing coefficients and compare these fusion methods on two datasets of multifocus image fusion. Experimental results are shown in Figure3, and objective measurements are provided in Table 1, with the best results marked in bold.Table 1
Comparison of fusion methods for Clock and Pepsi.
Image
Method
Average
Mean
Variance
MI
PCA
Proposed
Clock
IE
7.3939
7.3814
7.3818
7.3843
7.3812
7.4060
MI
6.3917
6.4224
6.4202
6.4129
6.4242
6.4550
Q
A
B
/
F
0.4174
0.4200
0.4182
0.4227
0.4183
0.5920
Pepsi
IE
7.0311
7.0311
7.0312
7.0314
7.0312
7.0452
MI
5.4329
5.4379
5.4341
5.4343
5.4342
5.4938
Q
A
B
/
F
0.1937
0.1939
0.1932
0.1927
0.1932
0.2377Comparison result of the multifocus image fusion.
(a)
Image A
(b)
Image B
(c)
Average
(d)
Mean
(e)
Variance
(f)
MI
(g)
PCA
(h)
Proposed
(i)
Image A
(j)
Image B
(k)
Average
(l)
Mean
(m)
Variance
(n)
MI
(o)
PCA
(p)
ProposedWe have proposed the compressive image fusion method based on PCNN. It is difficult to evaluate multisensor image fusion results from subjective evaluation for the reason that the fusion methods adopt the same CS theory as the image decomposition method. Therefore, fusion results are evaluated by means of IE, MI, andQAB/F metrics that can provide objective measurements although minor differences exist. As shown in Table 1, it is demonstrated that, in the fusion scenario of multifocus images, our proposed method has the best result from the perspective of both objective assessment and visual perception.


#### 4.2.2. Infrared and Visible Light Image Fusion

The infrared (IR) image and visible image are different kinds of images where IR image has low definition but a strong ability of discovering important military targets, while the visible image has higher definition and provides more detail information of texture expression. Thus, the IR and visible image fusion can help in target detection and recognition. We implement CS image fusion based on six fusion methods combined with compressive sensing coefficients and conduct the experiments in “UNcamp” image sets (two frames are selected randomly). Experimental results and objective measurements are provided as in Figure4 and Table 2.Table 2
Comparison of fusion methods for “UNcamp” dataset (frames 1807 and 1810).
Image
Method
Average
Mean
Variance
MI
PCA
Proposed
Frame1807
IE
6.2244
6.2106
6.2262
6.2168
6.2297
6.3605
MI
1.4523
1.3853
1.3758
1.4119
1.3712
1.7758
Q
A
B
/
F
0.1129
0.1165
0.1174
0.1191
0.1187
0.2276
Frame1810
IE
6.2280
6.2269
6.2411
6.2233
6.2444
6.3781
MI
1.4642
1.4202
1.4204
1.4422
1.4049
1.9327
Q
A
B
/
F
0.1077
0.1145
0.1142
0.1129
0.1152
0.2129Comparison result of the infrared and visible light image fusion.
(a)
Image A
(b)
Image B
(c)
Average
(d)
Mean
(e)
Variance
(f)
MI
(g)
PCA
(h)
Proposed
(i)
Image A
(j)
Image B
(k)
Average
(l)
Mean
(m)
Variance
(n)
MI
(o)
PCA
(p)
ProposedIn infrared and visible light images fusion scenarios, as shown in Figure4, the proposed fusion method has better object detection ability. In addition, Table 2 shows that the proposed method has the better performance in IE, MI, and QAB/F metrics than any of the other five fusion methods. It also demonstrates that the fused image reserved the edge and salient information from source images.


#### 4.2.3. Medical Image Fusion Application

Medical image fusion plays an important role in clinical applications. We apply the proposed method to medical image fusion on two image datasets. These datasets include CT-MRI image fusion and T1-weighted MR image (MR-T1) and T2-weighted MR image (MR-T2) fusion. Fusion of CT and MRI images can preserve bone structures and soft tissues information at the same time. In addition, MR-T1 and MR-T2 image fusion can provide the details of an anatomical structure of tissues and the information about normal and pathological tissues at the same time. Experiments are conducted to evaluate the fusion method in compressed domain, shown in Figure5. The objective measurements are provided in Table 3.Table 3
Comparison of fusion methods for CT-MRI and MRT1-MRT2.
Image
Method
Average
Mean
Variance
MI
PCA
Proposed
CT-MRI
IE
5.8369
6.6259
6.5445
6.4765
6.6467
6.5692
MI
2.9354
2.5375
2.2429
2.4872
2.4367
2.9798
Q
A
B
/
F
0.2720
0.5762
0.5116
0.5250
0.5870
0.6711
MRT1-MRT2
IE
3.9925
4.1990
4.1803
4.0174
4.2195
4.4534
MI
2.7466
2.9481
2.9445
2.7193
2.9971
3.6100
Q
A
B
/
F
0.3686
0.4864
0.4845
0.3732
0.5092
0.7362Comparison result of the medical image fusion.
(a)
Image A
(b)
Image B
(c)
Average
(d)
Mean
(e)
Variance
(f)
MI
(g)
PCA
(h)
Proposed
(i)
Image A
(j)
Image B
(k)
Average
(l)
Mean
(m)
Variance
(n)
MI
(o)
PCA
(p)
ProposedAs Figure5 shows, the fused image in Figure 5(h) obtained by our proposed method contains both the bone structure from the CT image and the soft tissue information from the MRI image. In addition, the fusion result of the proposed method is smooth while the result of mean, variance, and PCA causes the blocking effect. Table 3 demonstrates that the proposed method has the best result in terms of IE, MI, and QAB/F.


## 4.1. Experiment Setup

We conduct the experiment on six image datasets with three application scenarios. These fusion applications are multifocus image fusion, infrared and visible light image fusion, and medical image fusion. In addition to visual comparison, we also present fusion results with three objective metrics: information entropy (IE), mutual information (MI) [29], and QAB/F [37]. We compare these image fusion methods in compressed domain, with sampling rate set at 0.5. In addition, we use SBHE as the sampling pattern and adopt GPSR as the CS reconstruction algorithm. The SBHE block size is set at 32×32. In this way, the block Hadamard sampling matrix can encode pixels with 32×32. All the source images can be obtained at http://www.imagefusion.org and http://www.med.harvard.edu/aanlib/home.html.


## 4.2. Experimental Result

### 4.2.1. Multifocus Image Fusion

Due to the limited focus depth of the optical lens, it is not possible to get an image that contains all relevant objects in focus. In this way, multifocus image fusion is required to fuse the images taken from the same view point under different focal settings, in order to provide a fused image with a better description of the scene than any individual image. The obtained fusion image shows all the details of the source image, making it more conducive to the follow-up treatment. We implement CS image fusion based on six fusion methods on pixel level combined with compressive sensing coefficients and compare these fusion methods on two datasets of multifocus image fusion. Experimental results are shown in Figure3, and objective measurements are provided in Table 1, with the best results marked in bold.Table 1
Comparison of fusion methods for Clock and Pepsi.
Image
Method
Average
Mean
Variance
MI
PCA
Proposed
Clock
IE
7.3939
7.3814
7.3818
7.3843
7.3812
7.4060
MI
6.3917
6.4224
6.4202
6.4129
6.4242
6.4550
Q
A
B
/
F
0.4174
0.4200
0.4182
0.4227
0.4183
0.5920
Pepsi
IE
7.0311
7.0311
7.0312
7.0314
7.0312
7.0452
MI
5.4329
5.4379
5.4341
5.4343
5.4342
5.4938
Q
A
B
/
F
0.1937
0.1939
0.1932
0.1927
0.1932
0.2377Comparison result of the multifocus image fusion.
(a)
Image A
(b)
Image B
(c)
Average
(d)
Mean
(e)
Variance
(f)
MI
(g)
PCA
(h)
Proposed
(i)
Image A
(j)
Image B
(k)
Average
(l)
Mean
(m)
Variance
(n)
MI
(o)
PCA
(p)
ProposedWe have proposed the compressive image fusion method based on PCNN. It is difficult to evaluate multisensor image fusion results from subjective evaluation for the reason that the fusion methods adopt the same CS theory as the image decomposition method. Therefore, fusion results are evaluated by means of IE, MI, andQAB/F metrics that can provide objective measurements although minor differences exist. As shown in Table 1, it is demonstrated that, in the fusion scenario of multifocus images, our proposed method has the best result from the perspective of both objective assessment and visual perception.


### 4.2.2. Infrared and Visible Light Image Fusion

The infrared (IR) image and visible image are different kinds of images where IR image has low definition but a strong ability of discovering important military targets, while the visible image has higher definition and provides more detail information of texture expression. Thus, the IR and visible image fusion can help in target detection and recognition. We implement CS image fusion based on six fusion methods combined with compressive sensing coefficients and conduct the experiments in “UNcamp” image sets (two frames are selected randomly). Experimental results and objective measurements are provided as in Figure4 and Table 2.Table 2
Comparison of fusion methods for “UNcamp” dataset (frames 1807 and 1810).
Image
Method
Average
Mean
Variance
MI
PCA
Proposed
Frame1807
IE
6.2244
6.2106
6.2262
6.2168
6.2297
6.3605
MI
1.4523
1.3853
1.3758
1.4119
1.3712
1.7758
Q
A
B
/
F
0.1129
0.1165
0.1174
0.1191
0.1187
0.2276
Frame1810
IE
6.2280
6.2269
6.2411
6.2233
6.2444
6.3781
MI
1.4642
1.4202
1.4204
1.4422
1.4049
1.9327
Q
A
B
/
F
0.1077
0.1145
0.1142
0.1129
0.1152
0.2129Comparison result of the infrared and visible light image fusion.
(a)
Image A
(b)
Image B
(c)
Average
(d)
Mean
(e)
Variance
(f)
MI
(g)
PCA
(h)
Proposed
(i)
Image A
(j)
Image B
(k)
Average
(l)
Mean
(m)
Variance
(n)
MI
(o)
PCA
(p)
ProposedIn infrared and visible light images fusion scenarios, as shown in Figure4, the proposed fusion method has better object detection ability. In addition, Table 2 shows that the proposed method has the better performance in IE, MI, and QAB/F metrics than any of the other five fusion methods. It also demonstrates that the fused image reserved the edge and salient information from source images.


### 4.2.3. Medical Image Fusion Application

Medical image fusion plays an important role in clinical applications. We apply the proposed method to medical image fusion on two image datasets. These datasets include CT-MRI image fusion and T1-weighted MR image (MR-T1) and T2-weighted MR image (MR-T2) fusion. Fusion of CT and MRI images can preserve bone structures and soft tissues information at the same time. In addition, MR-T1 and MR-T2 image fusion can provide the details of an anatomical structure of tissues and the information about normal and pathological tissues at the same time. Experiments are conducted to evaluate the fusion method in compressed domain, shown in Figure5. The objective measurements are provided in Table 3.Table 3
Comparison of fusion methods for CT-MRI and MRT1-MRT2.
Image
Method
Average
Mean
Variance
MI
PCA
Proposed
CT-MRI
IE
5.8369
6.6259
6.5445
6.4765
6.6467
6.5692
MI
2.9354
2.5375
2.2429
2.4872
2.4367
2.9798
Q
A
B
/
F
0.2720
0.5762
0.5116
0.5250
0.5870
0.6711
MRT1-MRT2
IE
3.9925
4.1990
4.1803
4.0174
4.2195
4.4534
MI
2.7466
2.9481
2.9445
2.7193
2.9971
3.6100
Q
A
B
/
F
0.3686
0.4864
0.4845
0.3732
0.5092
0.7362Comparison result of the medical image fusion.
(a)
Image A
(b)
Image B
(c)
Average
(d)
Mean
(e)
Variance
(f)
MI
(g)
PCA
(h)
Proposed
(i)
Image A
(j)
Image B
(k)
Average
(l)
Mean
(m)
Variance
(n)
MI
(o)
PCA
(p)
ProposedAs Figure5 shows, the fused image in Figure 5(h) obtained by our proposed method contains both the bone structure from the CT image and the soft tissue information from the MRI image. In addition, the fusion result of the proposed method is smooth while the result of mean, variance, and PCA causes the blocking effect. Table 3 demonstrates that the proposed method has the best result in terms of IE, MI, and QAB/F.


## 4.2.1. Multifocus Image Fusion

Due to the limited focus depth of the optical lens, it is not possible to get an image that contains all relevant objects in focus. In this way, multifocus image fusion is required to fuse the images taken from the same view point under different focal settings, in order to provide a fused image with a better description of the scene than any individual image. The obtained fusion image shows all the details of the source image, making it more conducive to the follow-up treatment. We implement CS image fusion based on six fusion methods on pixel level combined with compressive sensing coefficients and compare these fusion methods on two datasets of multifocus image fusion. Experimental results are shown in Figure3, and objective measurements are provided in Table 1, with the best results marked in bold.Table 1
Comparison of fusion methods for Clock and Pepsi.
Image
Method
Average
Mean
Variance
MI
PCA
Proposed
Clock
IE
7.3939
7.3814
7.3818
7.3843
7.3812
7.4060
MI
6.3917
6.4224
6.4202
6.4129
6.4242
6.4550
Q
A
B
/
F
0.4174
0.4200
0.4182
0.4227
0.4183
0.5920
Pepsi
IE
7.0311
7.0311
7.0312
7.0314
7.0312
7.0452
MI
5.4329
5.4379
5.4341
5.4343
5.4342
5.4938
Q
A
B
/
F
0.1937
0.1939
0.1932
0.1927
0.1932
0.2377Comparison result of the multifocus image fusion.
(a)
Image A
(b)
Image B
(c)
Average
(d)
Mean
(e)
Variance
(f)
MI
(g)
PCA
(h)
Proposed
(i)
Image A
(j)
Image B
(k)
Average
(l)
Mean
(m)
Variance
(n)
MI
(o)
PCA
(p)
ProposedWe have proposed the compressive image fusion method based on PCNN. It is difficult to evaluate multisensor image fusion results from subjective evaluation for the reason that the fusion methods adopt the same CS theory as the image decomposition method. Therefore, fusion results are evaluated by means of IE, MI, andQAB/F metrics that can provide objective measurements although minor differences exist. As shown in Table 1, it is demonstrated that, in the fusion scenario of multifocus images, our proposed method has the best result from the perspective of both objective assessment and visual perception.


## 4.2.2. Infrared and Visible Light Image Fusion

The infrared (IR) image and visible image are different kinds of images where IR image has low definition but a strong ability of discovering important military targets, while the visible image has higher definition and provides more detail information of texture expression. Thus, the IR and visible image fusion can help in target detection and recognition. We implement CS image fusion based on six fusion methods combined with compressive sensing coefficients and conduct the experiments in “UNcamp” image sets (two frames are selected randomly). Experimental results and objective measurements are provided as in Figure4 and Table 2.Table 2
Comparison of fusion methods for “UNcamp” dataset (frames 1807 and 1810).
Image
Method
Average
Mean
Variance
MI
PCA
Proposed
Frame1807
IE
6.2244
6.2106
6.2262
6.2168
6.2297
6.3605
MI
1.4523
1.3853
1.3758
1.4119
1.3712
1.7758
Q
A
B
/
F
0.1129
0.1165
0.1174
0.1191
0.1187
0.2276
Frame1810
IE
6.2280
6.2269
6.2411
6.2233
6.2444
6.3781
MI
1.4642
1.4202
1.4204
1.4422
1.4049
1.9327
Q
A
B
/
F
0.1077
0.1145
0.1142
0.1129
0.1152
0.2129Comparison result of the infrared and visible light image fusion.
(a)
Image A
(b)
Image B
(c)
Average
(d)
Mean
(e)
Variance
(f)
MI
(g)
PCA
(h)
Proposed
(i)
Image A
(j)
Image B
(k)
Average
(l)
Mean
(m)
Variance
(n)
MI
(o)
PCA
(p)
ProposedIn infrared and visible light images fusion scenarios, as shown in Figure4, the proposed fusion method has better object detection ability. In addition, Table 2 shows that the proposed method has the better performance in IE, MI, and QAB/F metrics than any of the other five fusion methods. It also demonstrates that the fused image reserved the edge and salient information from source images.


## 4.2.3. Medical Image Fusion Application

Medical image fusion plays an important role in clinical applications. We apply the proposed method to medical image fusion on two image datasets. These datasets include CT-MRI image fusion and T1-weighted MR image (MR-T1) and T2-weighted MR image (MR-T2) fusion. Fusion of CT and MRI images can preserve bone structures and soft tissues information at the same time. In addition, MR-T1 and MR-T2 image fusion can provide the details of an anatomical structure of tissues and the information about normal and pathological tissues at the same time. Experiments are conducted to evaluate the fusion method in compressed domain, shown in Figure5. The objective measurements are provided in Table 3.Table 3
Comparison of fusion methods for CT-MRI and MRT1-MRT2.
Image
Method
Average
Mean
Variance
MI
PCA
Proposed
CT-MRI
IE
5.8369
6.6259
6.5445
6.4765
6.6467
6.5692
MI
2.9354
2.5375
2.2429
2.4872
2.4367
2.9798
Q
A
B
/
F
0.2720
0.5762
0.5116
0.5250
0.5870
0.6711
MRT1-MRT2
IE
3.9925
4.1990
4.1803
4.0174
4.2195
4.4534
MI
2.7466
2.9481
2.9445
2.7193
2.9971
3.6100
Q
A
B
/
F
0.3686
0.4864
0.4845
0.3732
0.5092
0.7362Comparison result of the medical image fusion.
(a)
Image A
(b)
Image B
(c)
Average
(d)
Mean
(e)
Variance
(f)
MI
(g)
PCA
(h)
Proposed
(i)
Image A
(j)
Image B
(k)
Average
(l)
Mean
(m)
Variance
(n)
MI
(o)
PCA
(p)
ProposedAs Figure5 shows, the fused image in Figure 5(h) obtained by our proposed method contains both the bone structure from the CT image and the soft tissue information from the MRI image. In addition, the fusion result of the proposed method is smooth while the result of mean, variance, and PCA causes the blocking effect. Table 3 demonstrates that the proposed method has the best result in terms of IE, MI, and QAB/F.


## 5. Conclusions

In this paper, we present a PCNN-based image fusion framework in compressed domain. In this framework, the local standard variance is calculated as the input to motivate PCNN, which guides the calculation of weighted factors in the fusion rule. Experiments are conducted on six image datasets with three fusion application scenarios to validate the performance and adaptability of the proposed method. Experimental results demonstrate that the proposed method has better performance in terms of objective evaluation metrics compared with other image fusion methods in compressed domain. Moreover, the proposed method is adaptive for different application scenarios illustrated in this paper.It is significant to explore the image fusion method in compressed domain due to the reason that compressive sensing has excellent ability of compression capability and computational simplicity on the sensor side. In the future, we plan to extend the proposed method by integrating more sampling patterns and develop more advanced fusion rules.



---

*Source: 536215-2015-01-26.xml*