# Compressed Sensing and Low-Rank Matrix Decomposition in Multisource Images Fusion

Authors: Kan Ren; Fuyuan Xu; Guohua Gu
Journal: Mathematical Problems in Engineering
 (2014)

---

## Abstract

We propose a novel super-resolution multisource images fusion scheme via compressive sensing and dictionary learning theory. Under the sparsity prior of images patches and the framework of the compressive sensing theory, the multisource images fusion is reduced to a signal recovery problem from the compressive measurements. Then, a set of multiscale dictionaries are learned from several groups of  high-resolution sample image’s patches via a nonlinear optimization algorithm. Moreover, a new linear weights fusion rule is proposed to obtain the high-resolution image. Some experiments are taken to investigate the performance of our proposed method, and the results prove its superiority to its counterparts.


---

## Body

## 1. Introduction

Fusion of multisource images that came from different modalities is very useful for obtaining a better understanding of the environmental conditions, for example, the fusion of multifocus images, the infrared (IR) images and visible images, the medical CT images and MRI images, and the multispectrum images and panchromatic images. Nowadays multiresolution based fusion approaches have been one of the popular techniques that is investigated by many researches and proves to present state-of-the-art result [1–4], including pyramid-based methods and discrete wavelet transform- (DWT-) based methods. In recent years, a new developed compressive sensing (CS) [5–8] theory is introduced into image fusion. It is well known that the compressive sensing theory provides a possible way of recovering sparse signals from their projection onto a small number of random vectors, so compressive sensing indicated a possible way of recovering high-resolution signals from their low-resolution version.Assume that a signalx
∈
R
N is compressible under a dictionary Ψ
∈
R
N
×
N
:
x
=
Ψ
θ, where ∥
θ
∥
0
=
K is the number of nonzero components of θ. The main idea of CS is to recover the original signal x from its compressive measurements y
=
Φ
x
∈
R
M, where N
≫
M. Under the condition that the matrix Φ
Ψ satisfies the restricted isometry property (RIP), the signal x can be accurately recovered from only M
≥
K measurements [5], by solving such an optimization problem,
(1)
min
⁡
θ
∥
θ
∥
0
s
.
t
.
c
y
=
Φ
x
=
Φ
Ψ
θ
.
Therefore, there are many advantages of combining the CS technique and image fusion application [9–14].Nowadays the applications of compressive sensing technology into image processing can be classified into three categories: compressive sensing based imaging [15–21], compressive sensing based image processing [22–26], and “compressive sensing” form applications [27–29]. Imaging is one of the most successful applications of compressive sensing theory, where a few sensors or low-resolution sensors are employed to achieve high-resolution imaging, such as optical imaging [16, 17], medical imaging [18, 19], and hyperspectral imaging [20, 21]. Compressive sensing is also used to transform images to other spaces to obtain more efficient analysis, such as the texture classification [22] and super-resolution image construction [23]. Numerous works are of the “compressive sensing” form applications; that is, if the task can be reduced to the optimization problem shown in (1), these works are also called compressive sensing based applications.In image fusion, most of the available compressive sensing based fusion schemes are of “compressive sensing” form [9–14, 27, 28]; that is, they did not consider the simultaneous fusion and super-resolution of multisource images. In this paper, we indicate another solution for simultaneous fusion and super-resolution of multisource images via the recent developed compressive sampling theory. Under the sparsity prior of images patches and the framework of the compressive sensing theory, the multisource images fusion is reduced to a signal recovery problem from the compressive measurements. A set of multiscale dictionaries are learned from some groups of high-resolution sample image’s patches via a nonlinear optimization algorithm. Moreover, a new linear weights fusion rule is proposed. Some experiments are taken to investigate the performance of our proposed method, and the results prove its superiority to its counterparts.The rest of this paper is organized as follows. Our proposed simultaneous fusion and super-resolution scheme of multisource images is expounded in Section2. In Section 3, some experiments are made to compare the proposed method with other related segmentation approaches. The conclusions are finally summarized in Section 4.


## 2. Simultaneous Fusion and Super-Resolution Scheme of Multisource Images

In this section, the foundations of our proposed method are illustrated, including the super-resolution multisource images fusion, the super-resolution multisource images fusion via compressive sensing, and the dictionary learning algorithm used in our approach.


### 2.1. Super-Resolution Multisource Images Fusion

Assume that the multisource images{
Y
i
low
,
i
=
1,2
,
…
S
} to be fused are low-resolution images; that is, the ith source images Y
i
low are a low-resolution version of X
i
high:
(2)
Y
i
low
=
M
X
i
high
+
v
i
(
i
=
1
,
…
,
S
)
,
where S is the number of source images, M is the down-resolution operator, and v
i is the measurement noise of the ith source image. We aim to recover a high-resolution image X
high from the multisource low-resolution images {
Y
i
low
,
i
=
1,2
,
…
S
}.The patches based fusion is adopted in our method; that is,X
high is processed in raster-scan order, from left to right and top to bottom, and then sequentially recovered. Let x
j
high
∈
R
n denote the jth n
×
n local patch vector extracted from a high-resolution fusion image X
high at the spatial location j
:
x
j
high
=
R
j
X
high, where R
j denotes a rectangular windowing operator and the overlapping is allowed. Given a set of p
×
p (p
∈
Z
+) LR patches taken from Y
i
low
:
{
y
i
,
1
low
,
y
i
,
2
low
,
…
,
y
i
,
Q
low
}
∈
R
m (m
=
p
2), (
i
=
1
,
…
,
S
;
j
=
1
,
…
,
Q
), we have
(3)
y
i
,
1
low
=
H
x
i
,
1
high
+
n
i
,
1
y
i
,
2
low
=
H
x
i
,
2
high
+
n
i
,
2
⋮
y
i
,
Q
low
=
H
x
i
,
Q
high
+
n
i
,
Q
.
A simple example of the matrix H
∈
R
4
×
16 is as follows:
(4)
H
=
[
1
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
1
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
1
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
1
0
0
0
0
0
]
.
Our aim is to reconstruct the fusion image X
high from the high-resolution patches x
i
high (i
=
1
,
…
,
S) from a set of the corresponding q
×
q (q
∈
Z
+) HR patches x
i
,
j
high
(
i
=
1
,
…
,
S
;
j
=
1
,
…
,
Q
)
∈
R
n (n
=
q
2). This is a simultaneous fusion and super-resolution problem of multisource images.


### 2.2. Super-Resolution Multisource Images Fusion via Compressive Sensing

According to the recent developed compressive sampling theory [5, 6], it is capable of recovering x
i
,
j
high (i
=
1
,
…
,
S; j
=
1
,
…
,
Q) from y
i
,
j
low under the sparsity prior of x
i
,
j
high; that is, x
i
,
j
high can be represented as a sparse linear combination by an overcomplete dictionary D
i
high
∈
R
n
×
K that is not coherent with the measurement (or sampling) matrix H; that is,
(5)
x
i
,
j
high
=
D
i
high
α
i
,
j
.
Here the “sparsity” of the decomposition coefficient α
i
,
j
∈
R
K means ∥
α
i
,
j
∥
0
=
S
≪
n
<
K, and K is the number of elements (or atoms) in the dictionary D
i
high. Under this sparsity assumption, x
i
,
j
high can thus be reconstructed by taking only m
≥
O
(
S
log
⁡
n
) measurements. As soon as the sparse coefficient α
i
,
j is determined by
(6)
min
⁡
α
i
,
j
∥
α
i
,
j
∥
0
s.t.
c
x
y
i
,
j
low
=
H
x
i
,
j
high
=
H
D
i
high
α
i
,
j
,
estimation of x
i
,
j
high can be obtained using (5).In our method, a linear fusion rule is performed on the{
Y
i
low
,
Y
2
low
,
…
,
Y
S
low
} with each Y
i
low
=
{
y
i
,
1
low
,
y
i
,
2
low
,
…
,
y
i
,
Q
low
}. Considering the patch by patch processing pattern, we write the low-resolution fusion patch as
(7)
y
j
low
=
∑
i
=
1
S
w
i
,
j
y
i
,
j
low
.
Therefore, its high-resolution version can be written as
(8)
x
j
high
=
∑
i
=
1
S
w
i
,
j
x
i
,
j
high
.
In our proposed method, we determine the weights according to the following formula:
(9)
w
i
,
j
=
1
S
-
1
×
(
∑
k
=
1
,
k
≠
i
S
∥
y
i
,
j
low
-
H
D
i
high
α
i
,
j
∥
2
2
∑
k
=
1
S
∥
y
k
,
j
low
-
H
D
k
,
j
high
α
k
,
j
∥
2
2
)
=
1
S
-
1
×
(
1
-
∥
y
i
,
j
low
-
H
D
i
high
α
i
,
j
∥
2
2
∑
k
=
1
S
∥
y
k
,
j
low
-
H
D
k
,
j
high
α
k
,
j
∥
2
2
)
.
Because patches {
x
j
high
} are highly redundant and the recovery of X from {
x
j
high
} becomes an overdetermined system, it is straightforward to obtain the following least-square solution in the patch aggregation:
(10)
X
=
(
∑
j
R
j
T
R
j
)
-
1
(
∑
j
R
j
T
x
j
high
)
.


### 2.3. Dictionary Learning Algorithm

The compressibility of patches shown in (5) is the sparsity prior used in our method. In order to generate several overcomplete dictionaries D
i
high (i
=
1,2
,
…
,
S) that can represent well the underlying HR patches, we propose an algorithm to adaptively tune the dictionary from a set of High-Resolution multisource sample image’s patches. In this section, we will reduce the learning of dictionary D
i
high  (i
=
1,2
,
…
,
S) as another sparsity-oriented optimization problem. Recent research on image statistics suggests that image patches can be well represented as a sparse linear combination of elements from an appropriately chosen overcomplete dictionary. Under this assumption, the HR image patches set Q
i
high
=
{
q
i
1
,
q
i
2
,
…
q
i
Q
∣
q
i
j
∈
R
n
,
j
=
1
,
…
,
Q
} (i
=
1
,
…
,
S) sampled from some training HR images can be represented as a sparse linear combination in a dictionary D
i
high
=
[
d
i
1
,
…
,
d
i
K
]
∈
R
n
×
K (i
=
1,2
,
…
,
S) (∥
d
i
j
∥
2
=
1
,
j
=
1
,
…
,
K); that is,
(11)
q
i
j
=
D
i
high
β
i
j
,
with the sparse coefficient vectors β
i
j
∈
R
K and ∥
β
i
j
∥
0
≪
K. The objective of designing D
i
high is to make the reconstruction error over Q
i
high (i
=
1
,
…
,
S) be minimal under the sparsity assumption; that is,
(12)
min
⁡
β
i
j
,
D
i
high
∥
β
i
j
∥
0
s
.
t
.
mc
q
i
j
=
D
i
high
β
i
j
.
We reformulate (12) as follows:
(13)
min
⁡
Β
i
,
D
i
high
∥
Β
i
∥
0,1
s
.
t
.
mc
Q
i
high
=
D
i
high
Β
i
,
where Β
i
=
[
β
i
1
,
β
i
2
,
…
,
β
i
Q
] is the coefficients matrix. In order to solve it, the KSVD dictionary learning algorithm is used to train the dictionaries D
i
high (i
=
1
,
…
,
S) from Q
i
high (i
=
1
,
…
,
S) [30, 31].


## 2.1. Super-Resolution Multisource Images Fusion

Assume that the multisource images{
Y
i
low
,
i
=
1,2
,
…
S
} to be fused are low-resolution images; that is, the ith source images Y
i
low are a low-resolution version of X
i
high:
(2)
Y
i
low
=
M
X
i
high
+
v
i
(
i
=
1
,
…
,
S
)
,
where S is the number of source images, M is the down-resolution operator, and v
i is the measurement noise of the ith source image. We aim to recover a high-resolution image X
high from the multisource low-resolution images {
Y
i
low
,
i
=
1,2
,
…
S
}.The patches based fusion is adopted in our method; that is,X
high is processed in raster-scan order, from left to right and top to bottom, and then sequentially recovered. Let x
j
high
∈
R
n denote the jth n
×
n local patch vector extracted from a high-resolution fusion image X
high at the spatial location j
:
x
j
high
=
R
j
X
high, where R
j denotes a rectangular windowing operator and the overlapping is allowed. Given a set of p
×
p (p
∈
Z
+) LR patches taken from Y
i
low
:
{
y
i
,
1
low
,
y
i
,
2
low
,
…
,
y
i
,
Q
low
}
∈
R
m (m
=
p
2), (
i
=
1
,
…
,
S
;
j
=
1
,
…
,
Q
), we have
(3)
y
i
,
1
low
=
H
x
i
,
1
high
+
n
i
,
1
y
i
,
2
low
=
H
x
i
,
2
high
+
n
i
,
2
⋮
y
i
,
Q
low
=
H
x
i
,
Q
high
+
n
i
,
Q
.
A simple example of the matrix H
∈
R
4
×
16 is as follows:
(4)
H
=
[
1
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
1
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
1
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
1
0
0
0
0
0
]
.
Our aim is to reconstruct the fusion image X
high from the high-resolution patches x
i
high (i
=
1
,
…
,
S) from a set of the corresponding q
×
q (q
∈
Z
+) HR patches x
i
,
j
high
(
i
=
1
,
…
,
S
;
j
=
1
,
…
,
Q
)
∈
R
n (n
=
q
2). This is a simultaneous fusion and super-resolution problem of multisource images.


## 2.2. Super-Resolution Multisource Images Fusion via Compressive Sensing

According to the recent developed compressive sampling theory [5, 6], it is capable of recovering x
i
,
j
high (i
=
1
,
…
,
S; j
=
1
,
…
,
Q) from y
i
,
j
low under the sparsity prior of x
i
,
j
high; that is, x
i
,
j
high can be represented as a sparse linear combination by an overcomplete dictionary D
i
high
∈
R
n
×
K that is not coherent with the measurement (or sampling) matrix H; that is,
(5)
x
i
,
j
high
=
D
i
high
α
i
,
j
.
Here the “sparsity” of the decomposition coefficient α
i
,
j
∈
R
K means ∥
α
i
,
j
∥
0
=
S
≪
n
<
K, and K is the number of elements (or atoms) in the dictionary D
i
high. Under this sparsity assumption, x
i
,
j
high can thus be reconstructed by taking only m
≥
O
(
S
log
⁡
n
) measurements. As soon as the sparse coefficient α
i
,
j is determined by
(6)
min
⁡
α
i
,
j
∥
α
i
,
j
∥
0
s.t.
c
x
y
i
,
j
low
=
H
x
i
,
j
high
=
H
D
i
high
α
i
,
j
,
estimation of x
i
,
j
high can be obtained using (5).In our method, a linear fusion rule is performed on the{
Y
i
low
,
Y
2
low
,
…
,
Y
S
low
} with each Y
i
low
=
{
y
i
,
1
low
,
y
i
,
2
low
,
…
,
y
i
,
Q
low
}. Considering the patch by patch processing pattern, we write the low-resolution fusion patch as
(7)
y
j
low
=
∑
i
=
1
S
w
i
,
j
y
i
,
j
low
.
Therefore, its high-resolution version can be written as
(8)
x
j
high
=
∑
i
=
1
S
w
i
,
j
x
i
,
j
high
.
In our proposed method, we determine the weights according to the following formula:
(9)
w
i
,
j
=
1
S
-
1
×
(
∑
k
=
1
,
k
≠
i
S
∥
y
i
,
j
low
-
H
D
i
high
α
i
,
j
∥
2
2
∑
k
=
1
S
∥
y
k
,
j
low
-
H
D
k
,
j
high
α
k
,
j
∥
2
2
)
=
1
S
-
1
×
(
1
-
∥
y
i
,
j
low
-
H
D
i
high
α
i
,
j
∥
2
2
∑
k
=
1
S
∥
y
k
,
j
low
-
H
D
k
,
j
high
α
k
,
j
∥
2
2
)
.
Because patches {
x
j
high
} are highly redundant and the recovery of X from {
x
j
high
} becomes an overdetermined system, it is straightforward to obtain the following least-square solution in the patch aggregation:
(10)
X
=
(
∑
j
R
j
T
R
j
)
-
1
(
∑
j
R
j
T
x
j
high
)
.


## 2.3. Dictionary Learning Algorithm

The compressibility of patches shown in (5) is the sparsity prior used in our method. In order to generate several overcomplete dictionaries D
i
high (i
=
1,2
,
…
,
S) that can represent well the underlying HR patches, we propose an algorithm to adaptively tune the dictionary from a set of High-Resolution multisource sample image’s patches. In this section, we will reduce the learning of dictionary D
i
high  (i
=
1,2
,
…
,
S) as another sparsity-oriented optimization problem. Recent research on image statistics suggests that image patches can be well represented as a sparse linear combination of elements from an appropriately chosen overcomplete dictionary. Under this assumption, the HR image patches set Q
i
high
=
{
q
i
1
,
q
i
2
,
…
q
i
Q
∣
q
i
j
∈
R
n
,
j
=
1
,
…
,
Q
} (i
=
1
,
…
,
S) sampled from some training HR images can be represented as a sparse linear combination in a dictionary D
i
high
=
[
d
i
1
,
…
,
d
i
K
]
∈
R
n
×
K (i
=
1,2
,
…
,
S) (∥
d
i
j
∥
2
=
1
,
j
=
1
,
…
,
K); that is,
(11)
q
i
j
=
D
i
high
β
i
j
,
with the sparse coefficient vectors β
i
j
∈
R
K and ∥
β
i
j
∥
0
≪
K. The objective of designing D
i
high is to make the reconstruction error over Q
i
high (i
=
1
,
…
,
S) be minimal under the sparsity assumption; that is,
(12)
min
⁡
β
i
j
,
D
i
high
∥
β
i
j
∥
0
s
.
t
.
mc
q
i
j
=
D
i
high
β
i
j
.
We reformulate (12) as follows:
(13)
min
⁡
Β
i
,
D
i
high
∥
Β
i
∥
0,1
s
.
t
.
mc
Q
i
high
=
D
i
high
Β
i
,
where Β
i
=
[
β
i
1
,
β
i
2
,
…
,
β
i
Q
] is the coefficients matrix. In order to solve it, the KSVD dictionary learning algorithm is used to train the dictionaries D
i
high (i
=
1
,
…
,
S) from Q
i
high (i
=
1
,
…
,
S) [30, 31].


## 3. Experiment Results

For evaluating the performance of the proposed fusion algorithm, in this section we have implemented them on some multisource images, including the multifocus images, infrared (IR) images, and visual images, as shown in Figure1. The size of all images used in the test is 256 lines × 256 columns and we aim to recover the 512 lines × 512 columns images. We compare our method with the following two related methods.Multisource images.
(a)
(b)
(c)
(d)
(e)
(f)
(g)
(h)Method (
1). Consider the multimodal image fusion with joint sparsity model [32].Method (
2). Consider the image features extraction and fusion based on joint sparse representation [33].For evaluating the performance of the proposed algorithm, the computed results are compared by visual quality subjectively and by some guidelines in fusion. The simulations are conducted in MATLAB R2009 on PC with Intel Core 2/1.8 G/1 G.The fusion results of three methods are shown in Figures2, 3, 4, and 5, and from left to right are Method 1, our method, and Method 2. From them we can see that the fusion images of our method can get higher-resolution images at the same time of fusing the multisource images. Compared with the rules in [32, 33], our proposed method has better preservation of directional information. The numerical guidelines are shown in Table 1, where some measures including the entropy (
E
), mutual information (MI), average gradient (AG), standard deviation (SD), cross entropy (
J
), universal image quality index (UIQI) [34], and correlation coefficient (CC) are calculated from the fusion images derived by different methods.Table 1
The fusion result of different methods.
Images
Measures
Our method
Method 1 [17]
Method 2 [18]
Figures1(a) and 1(b)
E
7.3026
7.2996
7.3012
MI
6.7899
6.7741
6.7560
AG
2.9150
2.8183
2.8250
SD
10.6438
10.6607
10.6643
J
0.3438
0.3428
0.3432
UIQI
0.9832
0.9901
0.9943
CC
0.9881
0.9884
0.9882
Figures1(c) and 1(d)
E
6.1440
6.4647
6.5612
MI
5.0346
3.6546
3.9025
AG
7.3064
6.1164
7.1345
SD
8.7785
8.7267
8.9131
J
6.0068
0.5881
6.9546
UIQI
0.5446
0.5328
0.5421
CC
0.6547
0.6459
0.6425
Figures1(e) and 1(f)
E
7.6970
7.5243
7.6133
MI
7.3204
5.7659
4.3234
AG
10.1423
7.4218
9.3781
SD
11.3244
10.6344
11.2825
J
0.6622
0.6729
0.7545
UIQI
0.7144
0.7204
0.7131
CC
0.8395
0.8782
0.8179
Figures1(g) and 1(h)
E
6.6827
6.2301
7.0025
MI
2.1098
1.4961
1.5836
AG
5.3866
3.5715
6.6887
SD
8.5479
7.8916
8.7738
J
0.5097
0.8127
0.2107
UIQI
0.9116
0.9244
0.9079
CC
0.5242
0.6333
0.4697Figure 2
Fusion results of Figures1(a) and 1(b).Figure 3
Fusion results of Figures1(c) and 1(d).Figure 4
Fusion results of Figures1(e) and 1(f).Figure 5
Fusion results of Figures1(g) and 1(h).Here UIQI is used to estimate the subjective vision effect, which combined the spatial correlation, wrap of mean, and variance together, and it can embody the comparability between the fused image and original images. It is defined as(14)
Q
=
σ
A
B
σ
A
σ
B
•
2
μ
A
μ
B
μ
A
2
+
μ
B
2
•
2
σ
A
σ
B
σ
A
2
+
σ
B
2
,
where σ
A
B is the covariance of the fused source images A and B and σ
x and μ
x are the standard variance and the mean of the image x, respectively. From it we can see that the numerical result accords with the subjective result.


## 4. Conclusions

In this paper we propose a novel super-resolution multisource images fusion scheme based on compressive sensing and dictionary learning. Under the sparsity prior of images patches and the framework of the compressive sensing theory, the multisource images fusion is reduced to a signal recovery problem from the compressive measurements. A new linear weights fusion rule is proposed. A set of multiscale dictionaries are learned from several groups of high-resolution (HR) sample image’s patches, and a higher resolution fusion image can be obtained from multisource images. Some experiments are taken and the results prove its superiority to its counterparts.



---

*Source: 278945-2014-07-07.xml*