# Block-Based MAP Superresolution Using Feature-Driven Prior Model

Authors: Feng Xu; Tanghuai Fan; Chenrong Huang; Xin Wang; Lizhong Xu
Journal: Mathematical Problems in Engineering
 (2014)

---

## Abstract

In the field of image superresolution reconstruction (SRR), the prior can be employed to solve the ill-posed problem. However, the prior model is selected empirically and characterizes the entire image so that the local feature of image cannot be represented accurately. This paper proposes a feature-driven prior model relying on feature of the image and introduces a block-based maximum a posteriori (MAP) framework under which the image is split into several blocks to perform SRR. Therefore, the local feature of image can be characterized more accurately, which results in a better SRR. In process of recombining superresolution blocks, we still design a border-expansion strategy to remove a byproduct, namely, cross artifacts. Experimental results show that the proposed method is effective.


---

## Body

## 1. Introduction

The superresolution reconstruction (SRR) is a technique that constructs one high-resolution image from low-resolution degraded images [1]. SRR plays a preprocessing role in image engineering, and it is important to provide accurate data for the subsequent image analysis, testing, understanding, and application [2, 3]. Many algorithms for SRR have been proposed and can be categorized into frequency domain methods and spatial domain methods. Frequency domain methods rely on the property of space-frequency transformation and the aliasing on low-resolution images [4, 5]. However, due to motion restriction and lack of prior, the frequency domain methods are not developed widely and spatial domain methods are preferred.The major spatial domain methods include maximum likelihood (ML) [6], projection onto convex sets (POCS) [7], and maximum a posteriori (MAP) [8, 9]. In MAP method, the SR task is cast as the inverse problem of recovering the original high-resolution image by fusing the low-resolution images, based upon the observation model and prior about the recovered image [10]. As MAP method introduces prior of image and follows rigorous reasoning, it has become one of the most frequently used SRR methods.In order to improve the performance of MAP method, many image prior models have been proposed. Gaussian Markov random field (GMRF) prior model [9, 11, 12] assumes a Gaussian distribution over the image; it not only penalizes the gradient of SRR solution but also preserves edge. Tikhonov prior model [13] uses certain high-pass operator to restrict the total energy of image grayscale and make image smooth. Total variation (TV) prior model [14, 15] adopts the idea of heavy tails which avoids oversmoothing in discontinuities. Bilateral TV (BTV) prior model [16, 17] is an updated version of TV prior model, which considers not only the intensity difference of pixels but also the distance between pixels and can remove more artifacts. Particularly, prior models above can be reduced to two fundamental models according to the exponent of norm [18]. Specifically, when the value of exponent is two, the prior model is a Gaussian model, and when the value of exponent is one the prior model is a Laplacian model. In this paper, these two models are named constant prior.The conventional MAP with constant prior is widely used, whereas theconstant prior has two main problems to solve.(
1
) Many papers discuss the learning of prior parameter from the observed data, but the selection from both models of constant prior is completely determined by experience and even is random. Although the recent example-based method [19] tries to overcome this drawback, it cannot get rid of the restriction of using external training set which is not the reliable evidence. (
2
) The aboveconstant prior characterizes the total pixels’ statistic feature of entire image, so the distinction of different regions of the image is not considered.In order to overcome the first drawback, inspired by the learning mechanism of Baker’s hallucination algorithm [20], this paper proposes a variable prior model which is essentially a mixture of two models of constant prior. The formulation of this variable prior model is similar to the elastic net prior model by Mohammad-Djafari [21]. But, in the prior model proposed in this paper, the weight assigned to each component model is obtained by learning and the formulation of this variable prior model completely lies on the pixels’ statistic feature of self-image, so the proposed prior model is also named feature-driven priormodel by us.In order to overcome the second drawback, this paper introduces a block-based MAP framework under which the image is split into many blocks. According to the feature of block, a proper feature-driven prior is assigned to the block. When each block has its own feature-driven prior, MAP SRR is performed for each block. And finally all reconstructed blocks are recombined into a whole reconstructed image. This framework considers local distinction of image feature, so it can improve the reconstruction result. It is still worth noticing that in reconstruction image by above method there are usually some cross artifacts. In order to eliminate these byproducts, this paper designs a border-expansion strategy for each block to achieve satisfied result.The rest of the paper is organized as follows. Section2 briefly reviews the MAP framework for SRR and introduces the formulation of constant priormodel. Section 3 analyizes and reveals the drawback of the constant prior, proposes the variable prior model which is called feature-driven prior model, presents a block-based MAP SSR framework, and designs a border-expansion strategy for each block to eliminate cross artifacts. In Section 4, experiments are conducted to validate the effectiveness of proposed algorithm. Finally, conclusion will be given in Section 5.


## 2. MAP Framework for SRR

In order to formulate the MAP framework for SRR,K observed images are denoted by {
y
t
}
t
=
1
K and the desired high-resolution image is represented by x. Then generative model of low-resolution images can be expressed as
(1)
y
t
=
F
t
x
+
n
t
for
t
=
1,2
,
…
,
K
,
where F
t represents the degraded matrix which includes warp, blur, and decimation, and n
t represents zero-mean Gaussian white noise with variance 1
/
β.In fact, MAP solution is derived from Bayes’ theorem. That is, given{
y
t
}
t
=
1
K and {
F
t
}
t
=
1
K, the posterior distribution of x can be formulated as
(2)
p
(
x
∣
{
y
t
}
t
=
1
K
)
=
p
(
{
y
t
}
t
=
1
K
∣
x
)
p
(
x
)
p
(
{
y
t
}
t
=
1
K
)
.
As {
y
t
}
t
=
1
K is a constant value, maximizing (2) is equal to maximizing the logarithm of numerator of (2) with respect to x. So the MAP solution for SRR is calculated iteratively by minimizing the following cost function [18]:
(3)
C
=
β
2
∑
t
=
1
K
∥
y
t
-
F
t
x
∥
2
2
-
log
⁡
⁡
(
p
(
x
)
)
.Equation (3) is fusion of ML cost function and regularization which, respectively, correspond to the first term and second term of (3). The multivariate probability distribution p
(
x
) in regularization is the exact image prior which is generally formulated from the following two models or their modifications:
(4)
Gaussian
model:
p
G
(
x
)
=
(
2
π
)
-
N
/
2
|
Γ
T
Γ
|
1
/
2
exp
⁡
⁡
{
-
∥
Γ
x
∥
2
2
2
}
,
(5)
Laplacian
model:
p
L
(
x
)
=
2
-
N
|
Γ
T
Γ
|
1
/
2
exp
⁡
⁡
{
-
∥
Γ
x
∥
1
1
}
,
where N is the size of image x and Γ is the operator matrix which computes the gradient of the image x in multidirection. In the past work [9–17], either the models above or their modifications were used to serve as the constant prior named in this paper.


## 3. Block MAP SRR with Feature-Driven Prior

### 3.1. The Drawback of Constant Prior Model

In fact, it is difficult to characterize images accurately by constant prior because the selection of such two formulated models is by experience. In addition, the formulation of constant prior makes it rigescent and inapplicable, so it still has some flaws.Through comprehensive analysis, we find that above two models (4) and (5) not only are multivariate distributions of x, but also correspond to multivariate distributions in which the random vector is the gradient (i.e., difference in multidirection) of image, denoted by x
g (i.e., x
g
=
Γ
x) with size of N
g. Elements of x
g are approximately independent identically distributed with zero-mean and standard deviation σ
G (Gaussian model) or σ
L (Laplacian model). Thus, with the theorem of random variables function, the distributions of x
g can be obtained from (4) and (5) as follows:
(6)
Gaussian
model:
p
g
G
(
x
g
)
=
(
2
π
)
-
N
g
/
2
exp
⁡
⁡
{
-
∥
x
g
∥
2
2
2
}
,
(7)
Laplacian
model:
p
g
L
(
x
g
)
=
2
-
N
g
exp
⁡
⁡
{
-
∥
x
g
∥
1
1
}
.The flaw with the two models lies at that the standard deviations ofx
g; that is, σ
G in (6) and σ
L in (7) are identical and equal to 1. This reveals that σ
G in (6) and σ
L in (7) cannot reflect the pixels’ statistic feature of image. Certainly, σ
G and σ
L in the conventional MAP-based SRR can be adjusted manually and through adding regulation parameter a satisfying result can be achieved, so the flaw is not exposed. However, σ
G and σ
L are both equal to 1 and the regulation parameter is not available when (6) or (7) is strictly substituted into (3). In this situation, when the prior model cannot characterize the feature of image the annoying superresolved image will probably emerge.In fact, it is impossible for image thatσ
G or σ
L is always equal to one, and it is not reasonable that σ
G is always equal to σ
L. Hence, a more practical prior of x
g is given by
(8)
Gaussian
model:
p
P
g
G
(
x
g
)
=
(
2
π
σ
G
2
)
-
N
g
/
2
exp
⁡
⁡
{
-
∥
x
g
∥
2
2
2
σ
G
2
}
,
Laplacian
model:
p
P
g
L
(
x
g
)
=
(
2
σ
L
)
-
N
g
exp
⁡
⁡
{
-
∥
x
g
∥
1
1
σ
L
}
.


### 3.2. Feature-Driven Prior

Now it is confirmed that the standard deviation (σ
G or σ
L) of variable x
g is just the core feature of prior which can be used later to determine the model of prior. Here, we firstly denote the samples of x and x
g by x
S and x
g
S, respectively. Then, given x
g
S (i.e., Γ
x
s), maximum likelihood estimations of the standard deviations (σ
G and σ
L) in the models of (8) are as follows:
(9)
σ
^
G
=
∥
x
g
S
∥
2
2
N
g
,
σ
^
L
=
∥
x
g
S
∥
1
1
N
g
.In this paper, the initial estimation of SRR iteration is selected as image samplex
S, and then the grade sample x
g
S is obtained. This sample selection is considered to be better than the image with similar texture [19] because the initial estimation is theoretically related to its true image and visually more similar to its true image than image with similar texture.To identify which model is better for prior, the model of prior is defined asM
∈
{
M
G
,
M
L
}, where M
G represents Gaussian prior model and M
L represents Laplacian prior model. Then the posterior probability of M is
(10)
p
(
M
∣
x
g
S
)
=
p
(
x
g
S
∣
M
)
p
(
M
)
p
(
x
g
S
)
,
where p
(
x
g
S
) is a normalization constant and p
(
M
) is prior of M without any additional condition. Simple and reasonable hypothesis is that M follows the uniform distribution. Thus
(11)
p
(
x
g
S
)
=
constant
;
p
(
M
G
)
=
p
(
M
L
)
=
1
2
.
With (10) and (11), the better prior model is obtained by
(12)
M
^
=
arg
max
⁡
M
p
(
M
∣
x
g
S
)
=
arg
max
⁡
M
p
(
x
g
S
∣
M
)
.AsM only has two options, (12) can be converted into a criterion in the form of ratio
(13)
R
=
p
P
g
G
(
x
g
S
)
p
P
g
L
(
x
g
S
)
,
where p
P
g
G
(
x
g
S
)
=
p
(
x
g
S
∣
M
G
) and p
P
g
L
(
x
g
S
)
=
p
(
x
g
S
∣
M
L
). Combining (8)–(9), (13) can be simplified as
(14)
R
=
(
2
e
σ
^
L
π
σ
^
G
)
N
g
.WhenR
>
1, that is, probability of Gaussian model is larger than Laplacian model, Gaussian model will be fitter for the image than Laplacian model and vice versa. Additionally, R not only indicates which model is fitter for the image but also can construct a more flexible variable prior model (i.e., the feature-driven prior model).The formulation of feature-driven prior model is a proportional mixture of Gaussian model and Laplacian model which is similar to the elastic net prior model [21]. (Inspired from elastic net regression [22], the elastic net prior model is a combination of a Gaussian model and a Laplacian model.)Difference in the elastic net prior model is that the feature-driven prior model assigns the weight to constant prior model (i.e., Gaussian model and Laplacian model) according to the possibility and applicability of either model, which makes this mixture prior characterize the desired image more accurately.The weight (i.e., proportion) assigned to the Laplacian model in the feature-driven prior can be formulated as(15)
W
L
=
p
P
g
L
(
x
g
S
)
p
P
g
G
(
x
g
S
)
+
p
P
g
L
(
x
g
S
)
=
1
p
P
g
G
(
x
g
S
)
/
p
P
g
L
(
x
g
S
)
+
1
=
1
R
+
1
.
And the weight (i.e., proportion) assigned to the Gaussian model in the feature-driven prior can be formulated as
(16)
W
G
=
p
P
g
G
(
x
g
S
)
p
P
g
G
(
x
g
S
)
+
p
P
g
L
(
x
g
S
)
=
p
P
g
G
(
x
g
S
)
/
p
P
g
L
(
x
g
S
)
p
P
g
G
(
x
g
S
)
/
p
P
g
L
(
x
g
S
)
+
1
=
R
R
+
1
.Referring to (3) and (8), a new cost function is constructed as follows:
(17)
C
=
β
2
∑
t
=
1
T
∥
y
t
-
F
t
x
∥
2
2
+
W
L
×
∥
Γ
x
∥
1
1
σ
^
L
+
W
G
×
∥
Γ
x
∥
2
2
2
σ
^
G
2
=
β
2
∑
t
=
1
T
∥
y
t
-
F
t
x
∥
2
2
+
1
R
+
1
×
∥
Γ
x
∥
1
1
σ
^
L
+
R
R
+
1
×
∥
Γ
x
∥
2
2
2
σ
^
G
2
.In (17), the last two terms correspond to the variable prior model which is flexible and named asfeature-driven prior. When R approaches 0 the prior is Laplacian, when R approaches positive infinity the prior is Gaussian, and when R is modest the prior is mixture.


### 3.3. Block-Based MAP Framework for SRR

In Section3.2, the feature-driven prior characterizes the entire image ignoring the difference between each region because the prior is derived from the entire image data. In order to take different region features into account, a block-based MAP framework is proposed for SRR in this subsection, in which the image is split into a tessellation with many blocks and for each block an exclusive prior is assigned.Applying the above block-based framework to the SSR, the block-based MAP superresolution using feature-driven prior (BMSFP) can be constructed. Procedure of BMSFP is as follows:(1)
the initial image for SRR iteration is estimated by shift and add algorithm (the shift and add algorithm, which is proposed by Farsiu et al. [16], is a simple and effective initial estimation method for SRR. After motion compensation according to F
t for all LR images, there may be several measurements at each pixel point in high-resolution grid. Then shift and add algorithm can be achieved by computing the pixelwise median of all measurements.) [16];
(2)
this initial image is split into a tessellation withp
×
q blocks (p is rows and q is columns of blocks;
(3)
the statistics (σ
G, σ
L, and R) of each block can be calculated, which determine the feature-driven prior model for each block. An illustration of tessellation and its block images with their assigned feature-driven prior is shown in Figure 1;
(4)
each block cost function can be formulated by (17) based on its statistics; that is;
(18)
C
i
=
β
2
∑
t
=
1
T
∥
y
t
-
F
t
i
x
i
∥
2
2
+
1
R
i
+
1
×
∥
Γ
x
i
∥
1
1
σ
^
L
i
+
R
i
R
i
+
1
×
∥
Γ
x
i
∥
2
2
2
σ
^
G
i
2
for
i
=
1,2
,
…
,
p
×
q
;
whereF
t
i is the compressed version of F
t of which some unrelated columns are deleted;
(5)
under the block-based MAP framework described in steps(
1
)
–
(
4
), a block superresolution can be conducted for each block by optimizing (18);
(6)
a perfect entire superresolution image is obtained by recombining all the superresolution results of each block.Illustration of an initial image and its split version with feature-driven priors. (a) Initial image. (b) Initial image split into2
×
2 blocks (each block is indicated by a fitted feature-driven prior model).
(a)
(b)


### 3.4. The Cross-Artifact Problem and Solution

Theoretically, the method we proposed in Section3.3 should be prominent in SRR. Whereas, in practice, the reconstructed image using the above method has obvious and annoying appearance containing cross between adjacent blocks. We call this phenomenon as cross artifact which reduces the final image’s quality even lower than initial image’s sometimes. In order to visualize this phenomenon, we implement an experiment and the result is shown in Figure 2.The comparison between initial image and reconstructed image with cross artifact. (a) Initial image (MSE = 101.5). (b) Reconstructed image with cross (MSE = 115).
(a)
(b)From Figure2 we can see that the cross artifact in (b) is very obvious, which makes the reconstructed image worse than initial image in (a) from not only vision but also performance index. Here, the performance index is mean square error (MSE) which calculates the mean square error between original image and its estimation. The MSE is formulated as follows:
(19)
MSE
=
25
5
2
×
1
N
∥
x
-
x
^
∥
2
2
,
where N is the number of pixels in image x.It is necessary to remove this byproduct, that is, cross artifact. However, where and how does this artifact come from? By deep exploration, we find that, for any SRR result, the border of image is usually different from other areas of image in the mean gray level, which we call as border effect. This is because the less neighbor pixels (only including two or three orientations) participate in the estimation of border, while, unlike the border case, for other regions, the neighbor pixels include entire four orientations. Based on the above analysis, there is no exception for each block in Figure2; that is, each block has the border effect. It is certainly reasonable that the cross artifact will emerge in final superresolved image between adjacent blocks due to the border effect when all reconstructed blocks are recombined. The cross artifact, in fact, comes from the border effect of each block.As the origin of cross effect is anatomized, the path to removing it is clear. Actually, the point lies in the modification on the size of block. When splitting the initial image we should expand each block several pixels widthw, which lead to the fact that adjacent blocks have twice-w-wide overlapped border. As such, these expanded blocks should be superresolved. But before recombining, we need to cut each block w pixels width border. Thus, the border effect of block can be removed effectively, so as to eventually avoid the cross-effect emerging in the final superresolved image after recombining.So far, when the SRR result may emerge cross artifact, we give the entire and effective artifact removal BMSFP (AR-BMSFP) algorithm shown in Algorithm1.Algorithm 1:The summary of AR-BMSFP Algorithm.
Objective: Estimate super-resolution x.
Input:
Low-resolution images sequence {
y
t
}
t
=
1
T, Motion matrix F
t, Border-expansion width w, Total rows of block p, Total
columns of block q
Initialization: Estimate initial image x
0 with shift and add Algorithm
Splitting:
Split the initial image x
0 into p
×
q blocks
Expand each block w pixels width and obtain x
i
0
Estimate σ
L
i, σ
G
i and R
i for each block x
i
0
Compress F
t into F
t
i according to x
i
0
Optimization:
Form the block cost function C
i using (18)
Optimize C
i with Scaled Conjugate Gradients (SCG) and initial block x
i
0 obtaining the overlapped super-resolution
block x
i
∞
Recombination:
Cut border of x
i
∞ with w pixels width, obtaining super-resolution block x
i; Combine all super-resolution blocks obtaining
the result image x
Result:The output is x.


## 3.1. The Drawback of Constant Prior Model

In fact, it is difficult to characterize images accurately by constant prior because the selection of such two formulated models is by experience. In addition, the formulation of constant prior makes it rigescent and inapplicable, so it still has some flaws.Through comprehensive analysis, we find that above two models (4) and (5) not only are multivariate distributions of x, but also correspond to multivariate distributions in which the random vector is the gradient (i.e., difference in multidirection) of image, denoted by x
g (i.e., x
g
=
Γ
x) with size of N
g. Elements of x
g are approximately independent identically distributed with zero-mean and standard deviation σ
G (Gaussian model) or σ
L (Laplacian model). Thus, with the theorem of random variables function, the distributions of x
g can be obtained from (4) and (5) as follows:
(6)
Gaussian
model:
p
g
G
(
x
g
)
=
(
2
π
)
-
N
g
/
2
exp
⁡
⁡
{
-
∥
x
g
∥
2
2
2
}
,
(7)
Laplacian
model:
p
g
L
(
x
g
)
=
2
-
N
g
exp
⁡
⁡
{
-
∥
x
g
∥
1
1
}
.The flaw with the two models lies at that the standard deviations ofx
g; that is, σ
G in (6) and σ
L in (7) are identical and equal to 1. This reveals that σ
G in (6) and σ
L in (7) cannot reflect the pixels’ statistic feature of image. Certainly, σ
G and σ
L in the conventional MAP-based SRR can be adjusted manually and through adding regulation parameter a satisfying result can be achieved, so the flaw is not exposed. However, σ
G and σ
L are both equal to 1 and the regulation parameter is not available when (6) or (7) is strictly substituted into (3). In this situation, when the prior model cannot characterize the feature of image the annoying superresolved image will probably emerge.In fact, it is impossible for image thatσ
G or σ
L is always equal to one, and it is not reasonable that σ
G is always equal to σ
L. Hence, a more practical prior of x
g is given by
(8)
Gaussian
model:
p
P
g
G
(
x
g
)
=
(
2
π
σ
G
2
)
-
N
g
/
2
exp
⁡
⁡
{
-
∥
x
g
∥
2
2
2
σ
G
2
}
,
Laplacian
model:
p
P
g
L
(
x
g
)
=
(
2
σ
L
)
-
N
g
exp
⁡
⁡
{
-
∥
x
g
∥
1
1
σ
L
}
.


## 3.2. Feature-Driven Prior

Now it is confirmed that the standard deviation (σ
G or σ
L) of variable x
g is just the core feature of prior which can be used later to determine the model of prior. Here, we firstly denote the samples of x and x
g by x
S and x
g
S, respectively. Then, given x
g
S (i.e., Γ
x
s), maximum likelihood estimations of the standard deviations (σ
G and σ
L) in the models of (8) are as follows:
(9)
σ
^
G
=
∥
x
g
S
∥
2
2
N
g
,
σ
^
L
=
∥
x
g
S
∥
1
1
N
g
.In this paper, the initial estimation of SRR iteration is selected as image samplex
S, and then the grade sample x
g
S is obtained. This sample selection is considered to be better than the image with similar texture [19] because the initial estimation is theoretically related to its true image and visually more similar to its true image than image with similar texture.To identify which model is better for prior, the model of prior is defined asM
∈
{
M
G
,
M
L
}, where M
G represents Gaussian prior model and M
L represents Laplacian prior model. Then the posterior probability of M is
(10)
p
(
M
∣
x
g
S
)
=
p
(
x
g
S
∣
M
)
p
(
M
)
p
(
x
g
S
)
,
where p
(
x
g
S
) is a normalization constant and p
(
M
) is prior of M without any additional condition. Simple and reasonable hypothesis is that M follows the uniform distribution. Thus
(11)
p
(
x
g
S
)
=
constant
;
p
(
M
G
)
=
p
(
M
L
)
=
1
2
.
With (10) and (11), the better prior model is obtained by
(12)
M
^
=
arg
max
⁡
M
p
(
M
∣
x
g
S
)
=
arg
max
⁡
M
p
(
x
g
S
∣
M
)
.AsM only has two options, (12) can be converted into a criterion in the form of ratio
(13)
R
=
p
P
g
G
(
x
g
S
)
p
P
g
L
(
x
g
S
)
,
where p
P
g
G
(
x
g
S
)
=
p
(
x
g
S
∣
M
G
) and p
P
g
L
(
x
g
S
)
=
p
(
x
g
S
∣
M
L
). Combining (8)–(9), (13) can be simplified as
(14)
R
=
(
2
e
σ
^
L
π
σ
^
G
)
N
g
.WhenR
>
1, that is, probability of Gaussian model is larger than Laplacian model, Gaussian model will be fitter for the image than Laplacian model and vice versa. Additionally, R not only indicates which model is fitter for the image but also can construct a more flexible variable prior model (i.e., the feature-driven prior model).The formulation of feature-driven prior model is a proportional mixture of Gaussian model and Laplacian model which is similar to the elastic net prior model [21]. (Inspired from elastic net regression [22], the elastic net prior model is a combination of a Gaussian model and a Laplacian model.)Difference in the elastic net prior model is that the feature-driven prior model assigns the weight to constant prior model (i.e., Gaussian model and Laplacian model) according to the possibility and applicability of either model, which makes this mixture prior characterize the desired image more accurately.The weight (i.e., proportion) assigned to the Laplacian model in the feature-driven prior can be formulated as(15)
W
L
=
p
P
g
L
(
x
g
S
)
p
P
g
G
(
x
g
S
)
+
p
P
g
L
(
x
g
S
)
=
1
p
P
g
G
(
x
g
S
)
/
p
P
g
L
(
x
g
S
)
+
1
=
1
R
+
1
.
And the weight (i.e., proportion) assigned to the Gaussian model in the feature-driven prior can be formulated as
(16)
W
G
=
p
P
g
G
(
x
g
S
)
p
P
g
G
(
x
g
S
)
+
p
P
g
L
(
x
g
S
)
=
p
P
g
G
(
x
g
S
)
/
p
P
g
L
(
x
g
S
)
p
P
g
G
(
x
g
S
)
/
p
P
g
L
(
x
g
S
)
+
1
=
R
R
+
1
.Referring to (3) and (8), a new cost function is constructed as follows:
(17)
C
=
β
2
∑
t
=
1
T
∥
y
t
-
F
t
x
∥
2
2
+
W
L
×
∥
Γ
x
∥
1
1
σ
^
L
+
W
G
×
∥
Γ
x
∥
2
2
2
σ
^
G
2
=
β
2
∑
t
=
1
T
∥
y
t
-
F
t
x
∥
2
2
+
1
R
+
1
×
∥
Γ
x
∥
1
1
σ
^
L
+
R
R
+
1
×
∥
Γ
x
∥
2
2
2
σ
^
G
2
.In (17), the last two terms correspond to the variable prior model which is flexible and named asfeature-driven prior. When R approaches 0 the prior is Laplacian, when R approaches positive infinity the prior is Gaussian, and when R is modest the prior is mixture.


## 3.3. Block-Based MAP Framework for SRR

In Section3.2, the feature-driven prior characterizes the entire image ignoring the difference between each region because the prior is derived from the entire image data. In order to take different region features into account, a block-based MAP framework is proposed for SRR in this subsection, in which the image is split into a tessellation with many blocks and for each block an exclusive prior is assigned.Applying the above block-based framework to the SSR, the block-based MAP superresolution using feature-driven prior (BMSFP) can be constructed. Procedure of BMSFP is as follows:(1)
the initial image for SRR iteration is estimated by shift and add algorithm (the shift and add algorithm, which is proposed by Farsiu et al. [16], is a simple and effective initial estimation method for SRR. After motion compensation according to F
t for all LR images, there may be several measurements at each pixel point in high-resolution grid. Then shift and add algorithm can be achieved by computing the pixelwise median of all measurements.) [16];
(2)
this initial image is split into a tessellation withp
×
q blocks (p is rows and q is columns of blocks;
(3)
the statistics (σ
G, σ
L, and R) of each block can be calculated, which determine the feature-driven prior model for each block. An illustration of tessellation and its block images with their assigned feature-driven prior is shown in Figure 1;
(4)
each block cost function can be formulated by (17) based on its statistics; that is;
(18)
C
i
=
β
2
∑
t
=
1
T
∥
y
t
-
F
t
i
x
i
∥
2
2
+
1
R
i
+
1
×
∥
Γ
x
i
∥
1
1
σ
^
L
i
+
R
i
R
i
+
1
×
∥
Γ
x
i
∥
2
2
2
σ
^
G
i
2
for
i
=
1,2
,
…
,
p
×
q
;
whereF
t
i is the compressed version of F
t of which some unrelated columns are deleted;
(5)
under the block-based MAP framework described in steps(
1
)
–
(
4
), a block superresolution can be conducted for each block by optimizing (18);
(6)
a perfect entire superresolution image is obtained by recombining all the superresolution results of each block.Illustration of an initial image and its split version with feature-driven priors. (a) Initial image. (b) Initial image split into2
×
2 blocks (each block is indicated by a fitted feature-driven prior model).
(a)
(b)


## 3.4. The Cross-Artifact Problem and Solution

Theoretically, the method we proposed in Section3.3 should be prominent in SRR. Whereas, in practice, the reconstructed image using the above method has obvious and annoying appearance containing cross between adjacent blocks. We call this phenomenon as cross artifact which reduces the final image’s quality even lower than initial image’s sometimes. In order to visualize this phenomenon, we implement an experiment and the result is shown in Figure 2.The comparison between initial image and reconstructed image with cross artifact. (a) Initial image (MSE = 101.5). (b) Reconstructed image with cross (MSE = 115).
(a)
(b)From Figure2 we can see that the cross artifact in (b) is very obvious, which makes the reconstructed image worse than initial image in (a) from not only vision but also performance index. Here, the performance index is mean square error (MSE) which calculates the mean square error between original image and its estimation. The MSE is formulated as follows:
(19)
MSE
=
25
5
2
×
1
N
∥
x
-
x
^
∥
2
2
,
where N is the number of pixels in image x.It is necessary to remove this byproduct, that is, cross artifact. However, where and how does this artifact come from? By deep exploration, we find that, for any SRR result, the border of image is usually different from other areas of image in the mean gray level, which we call as border effect. This is because the less neighbor pixels (only including two or three orientations) participate in the estimation of border, while, unlike the border case, for other regions, the neighbor pixels include entire four orientations. Based on the above analysis, there is no exception for each block in Figure2; that is, each block has the border effect. It is certainly reasonable that the cross artifact will emerge in final superresolved image between adjacent blocks due to the border effect when all reconstructed blocks are recombined. The cross artifact, in fact, comes from the border effect of each block.As the origin of cross effect is anatomized, the path to removing it is clear. Actually, the point lies in the modification on the size of block. When splitting the initial image we should expand each block several pixels widthw, which lead to the fact that adjacent blocks have twice-w-wide overlapped border. As such, these expanded blocks should be superresolved. But before recombining, we need to cut each block w pixels width border. Thus, the border effect of block can be removed effectively, so as to eventually avoid the cross-effect emerging in the final superresolved image after recombining.So far, when the SRR result may emerge cross artifact, we give the entire and effective artifact removal BMSFP (AR-BMSFP) algorithm shown in Algorithm1.Algorithm 1:The summary of AR-BMSFP Algorithm.
Objective: Estimate super-resolution x.
Input:
Low-resolution images sequence {
y
t
}
t
=
1
T, Motion matrix F
t, Border-expansion width w, Total rows of block p, Total
columns of block q
Initialization: Estimate initial image x
0 with shift and add Algorithm
Splitting:
Split the initial image x
0 into p
×
q blocks
Expand each block w pixels width and obtain x
i
0
Estimate σ
L
i, σ
G
i and R
i for each block x
i
0
Compress F
t into F
t
i according to x
i
0
Optimization:
Form the block cost function C
i using (18)
Optimize C
i with Scaled Conjugate Gradients (SCG) and initial block x
i
0 obtaining the overlapped super-resolution
block x
i
∞
Recombination:
Cut border of x
i
∞ with w pixels width, obtaining super-resolution block x
i; Combine all super-resolution blocks obtaining
the result image x
Result:The output is x.


## 4. Experimental Results

In this section, three tests are conducted to validate the utility of the proposed prior and algorithm. The performance of algorithm is evaluated by mean square error (MSE).


### 4.1. Comparison Experiments for Different Prior Models

In the first test, flexibility and superiority of the feature-driven prior are demonstrated. Firstly, the degraded low-resolution (LR) image sequence is generated from an original remote sensing image which is shown in Figure3(a). In the degradation process, original image is warped by homography and blurred by Gaussian point spread function with standard deviation of 0.8 high-resolution (HR) pixels, and then the warped and blurred image is decimated by factor 2 : 1 in horizontal and vertical directions, and finally the white Gaussian noise with standard deviation of 5 is added. Note that the intensity range of image is from 0 to 255. Two of the five degraded LR images are shown in Figures 3(b) and 3(c) which have been zoomed in the same size as original image for observation (all LR images in the following tests are also shown in this mode).SRR with different prior models (remote sensing image): (a) the original image, (b) one of the degraded LR images, (c) another of the degraded LR images, (d) SRR with Tikhonov prior model (MSE: 372.3448), (e) SRR with Farsiu BTV prior model (MSE: 292.9566), (f) SRR with Gaussian prior model (MSE: 68.8928), (g) SRR with Laplacian prior model (MSE: 68.7965), (h) SRR with elastic net prior model (MSE: 68.0170), and (i) SRR with feature-driven prior model (MSE: 67.5259).
(a)
(b)
(c)
(d)
(e)
(f)
(g)
(h)
(i)In the reconstruction process, the conventional preparation such as motion estimation [23] is omitted in order to focus on our algorithm. The information about warp, blurring, and decimation is supposed to be available; that is, the degraded matrix used in degradation can also be employed for reconstruction. In order to compare withthe feature-driven prior, firstly we reconstruct image using LR sequence based on Tikhonov prior and Farsiu BTV prior, respectively. The results are shown in Figures 3(d) and 3(e). Then two cost functions are formed, respectively, by substituting Gaussian prior model (4) into (3) and Laplacian prior model (5) into (3). Optimizing two cost functions above by scaled conjugate gradients (SCG) [24] yields two results of superresolution as shown in Figures 3(f) and 3(g). In order to convince that the feature-driven prior is a little bit better than the state-of-the-art elastic net prior model, we reconstruct image based on elastic net prior, the result of which is shown in Figure 3(h). Finally, another cost function is formed with the feature-driven prior based on (17). Optimizing this cost function by SCG yields the result of superresolution as shown in Figure 3(i).In order to demonstrate the applicability of the proposed prior model for different images, another standard image Lena is used to repeat this experiment. Figure4(a) is the part of the original Lena image. Figures 4(b) and 4(c) are two of the five degraded LR images. Figures 4(d)–4(i) are results of SRR with, in order, Tikhonov prior, Farsiu BTV prior, Gaussian prior, Laplacian prior, elastic net prior, and feature-driven prior.SRR with different prior models (Lena image): (a) the original image, (b) one of the degraded LR image, (c) another of the degraded LR image, (d) SRR with Tikhonov prior model (MSE: 226.4613), (e) SRR with Farsiu BTV prior model (MSE: 173.5646), (f) SRR with Gaussian prior model (MSE: 40.5469), (g) SRR with Laplacian prior model (MSE: 40.1860), (h) SRR with elastic net prior model (MSE: 39.3338), and (i) SRR with feature-driven prior model (MSE: 38.6370).
(a)
(b)
(c)
(d)
(e)
(f)
(g)
(h)
(i)It is important to note that the goal of Section4.1 is to compare different prior models, so the image is not split into blocks to perform SRR (i.e., the image-splitting scheme and border expansion are not required). All the experiments are performed for entire image. Therefore, there is no block artifact, that is, the cross artifact, in any results.The MSE results for remote sensing image and Lena image are listed in Table1. Figures 3 and 4 and Table 1 illustrate the advantage of the proposed prior compared with the other priors.Table 1
MSE of SSR with different prior models.
Prior model
Remote sensing image
Lena image
Tikhonov prior model
372.3448
226.4613
Farsiu BTV prior model
292.9566
173.5646
Gaussian prior model
68.8928
40.5469
Laplacian prior model
68.7965
40.1860
Elastic Net prior model
68.0170
39.3338
Proposed prior model
67.5259
38.6370


### 4.2. Comparison Experiments for Different Image-Splitting Schemes

In order to show that image-splitting scheme proposed in Section3.3 is effective, the second test is conducted. Certainly, the detailed algorithm to be validated is AR-BMSFP proposed in Section 3.4 because the cross artifact must be removed. So the border expansion must be utilized in image-splitting experiments. In the test, AR-BMSFP with different image-splitting schemes and several other algorithms without image-splitting scheme are compared with each other. The second test also includes two groups of experiments which use Peppers image and remote sensing image, respectively.The original Peppers image is shown in Figure5(a). In order to show the robustness and applicability of the algorithm, we made some adjustments on degradation process used in Section 4.1. To be specific, Gaussian point spread function is changed with standard deviation of 1.0 HR pixels, the decimation factor is changed with 4 : 1, and white Gaussian noise is amplified with standard deviation of 10.SRR with different image-splitting schemes (Peppers image). (a) Original image. (b) SRR with Tikhonov prior model (MSE: 576.640). (c) SRR with Farsiu BTV prior model (MSE: 554.615). (d) SRR with feature-driven prior model (MSE: 86.710). (e)2
×
2 block-based AR-BMSFP (MSE: 85.337). (f) 4
×
4 block-based AR-BMSFP (MSE: 84.456). (g) 8
×
8 block-based AR-BMSFP (MSE: 83.611).
(a)
(b)
(c)
(d)
(e)
(f)
(g)Then the AR-BMSFP SRR proposed in Section3.4 is compared with the method without image-splitting scheme to validate its superiority. Figure 5(b) shows the result of MAP SRR based on Tikhonov prior model without image-splitting scheme. Figure 5(c) shows the result of MAP SRR based on Farsiu BTV prior model without image-splitting scheme. Figure 5(d) shows the result of MAP SRR based on feature-driven prior model without image-splitting scheme.For AR-BMSFP algorithm, three experiments are conducted with2
×
2, 4
×
4, and 8
×
8 blocks, of which the results are shown in Figures 5(e), 5(f), and 5(g). It is worth noting that all these three results are obtained through several different border-expansion trials. Each final result is the best result selected from several trials. Therefore, the block artifact, that is, cross artifact, is not displayed in Figure 5.To prove the robustness of proposed method, the second group of experiments is conducted using remote sensing image. The degradation process is also repeated as in the first test, based on which the reconstruction process of first group of experiments is repeated. The results are shown in Figure6. It is also worth noting that the cross artifact has been removed in results through several border-expansion trials. So Figures 6(e), 6(f), and 6(g) have no artifacts.SRR with different image-splitting schemes (remote sensing image). (a) Original image. (b) SRR with Tikhonov prior model (MSE: 452.343). (c) SRR with Farsiu BTV prior model (MSE: 426.456). (d) SRR with feature-driven prior model (MSE: 73.703). (e)2
×
2 block-based AR-BMSFP (MSE: 72.967). (f) 4
×
4 block-based AR-BMSFP (MSE: 72.055). (g) 8
×
8 block-based AR-BMSFP (MSE: 71.980).
(a)
(b)
(c)
(d)
(e)
(f)
(g)The MSEs of two groups of experiments above are also listed in Table2. Both visual quality and MSE show that the AR-BMSFP algorithm is better than algorithm without image-splitting framework. Furthermore, MSE improves as the number of blocks increases.Table 2
MSE of SSR with different image-splitting schemes.
Algorithm
Peppers image
Remote sensing image
SRR with Tikhonov prior model
576.640
452.343
SRR with Farsiu BTV prior model
554.615
426.456
SRR with feature-driven prior model
86.710
73.703
2
×
2 block-based AR-BMSFP
85.337
72.967
4
×
4 block-based AR-BMSFP
84.456
72.055
8
×
8 block-based AR-BMSFP
83.611
71.980


### 4.3. Comparison Experiments for Different Border Expansions in Cross-Artifact Removal

The third test is a final test in this paper, which validates the capability of our solution for cross-artifact emerging in result. There are three groups of experiments in this test to show the robustness.As the cross artifact is only about 3 to 5 pixels width, the first two groups of experiments use small size images to make the cross artifact more visible. In the first group of experiments, a256
×
256 remote sensing image shown in Figure 7(a) is used. In the second group of experiments, a 128
×
128 magnetic resonance imaging (MRI) image shown in Figure 8(a) is used.AR-BMSFP based on8
×
8 blocks with different border expansions (remote sensing image): (a) the original image, (b) the degraded LR image, (c) the result without border expansion (MSE: 396.43), (d) the result with 1-pixel border expansion (MSE: 266.43), (e) the result with 2-pixel border expansion (MSE: 233.07), (f) the result with 3-pixel border expansion (MSE: 222.11), and (g) the result with 4-pixel border expansion (MSE: 218.48).
(a)
(b)
(c)
(d)
(e)
(f)
(g)AR-BMSFP based on8
×
8 blocks with different border expansions (MRI image): (a) the original image, (b) the degraded LR image, (c) the result without border expansion (MSE: 244.26), (d) the result with 1-pixel border expansion (MSE: 191.38), (e) the result with 3-pixel border expansion (MSE: 157.15), (f) the result with 4-pixel border expansion (MSE: 145.51), and (g) the result with 5-pixel border expansion (MSE: 143.29).
(a)
(b)
(c)
(d)
(e)
(f)
(g)The degradation of them is determined by the same method in Figure5. Figures 7(b) and 8(b) show one of the degraded LR images, respectively. According to the algorithm present in Algorithm 1, we begin to reconstruct image from LR sequence. For comparison, we select the same strategy to split the image; that is, the splitting index is fixed as 8
×
8.In order to illustrate effect with different border-expansion widths for blocks, we repeat the reconstruction with 0-, 1-, 2-, 3-, and 4-pixel border expansions for blocks in remote sensing image experiments. These results are shown in Figures7(c)–7(g). From result images, we can see that the cross artifact in Figure 7(c) is severe as no expansion, and cross artifact in Figures 7(d) and 7(e) is still obvious as thin border expansion. Due to the thick border expansion, Figures 7(f) and 7(g) appear without cross artifact. But in vision, Figure 7(g) is a little better than Figure 7(f), especially for the river in image. Besides, the MSEs also prove the judgments in vision. To make it clear, MSE values are illustrated in the graph of Figure 9(a). It is also obvious that as the border becomes wider the MSE will decrease.MSE values of AR-BMSFP based on8
×
8 blocks with different border expansions: (a) remote sensing image and (b) MIR image.
(a)
(b)Keeping all the conditions and conducts in the first group of experiments, we carry out the second group of experiments for a MRI image. The results and graph are shown in Figures8 and 9(b). The difference from previous is that we repeat the reconstruction with 0-, 1-, 2-, 3-, 4-, and 5-pixel border expansions. Note that due to need of layout, Figure 8 only includes results with 0-, 1-, 3-, 4-, and 5-pixel border expansions. But Figure 9(b) shows all results. From Figure 8, the effect of reconstruction is not as good as that of Figure 7. It is perhaps because of the fact that (
1
) the size of image is smaller than that in Figure 7 and (
2
) the texture of image is more monotonous than that in Figure 7. Even so, all results and graph still give the same conclusion as Figure 7. It also proves the robustness of our algorithm.No doubt it is time consuming to do the above experiment using bigger size image. But the capability to remove cross artifact is still effective for bigger size image. In order to prove this claim, a1024
×
768 image shown in Figure 10(a) is selected to repeat the above experiment.AR-BMSFP based on8
×
8 blocks with different border expansions (1024
×
768 image): (a) the original image, (b) the degraded LR image, (c) the result without border expansion (MSE: 785.74), (d) the result with 1-pixel border expansion (MSE: 259.49), (e) the result with 2-pixel border expansion (MSE: 249.29), and (f) the result with 3-pixel border expansion (MSE: 244.39).
(a)
(b)
(c)
(d)
(e)
(f)The degradation of the1024
×
768 image is the same as the method of the first two groups of experiments. A degraded LR image is shown in Figure 10(b). The AR-BMSFP reconstruction is conducted with 0-, 1-, 2-, and 3-pixel border expansions. The reconstructed results are shown in Figures 10(c)–10(f). Figure 10 verifies that the AR-BMSFP algorithm is effective for bigger size image. Additionally, the computed MSEs of all results are listed in Table 3 to prove that the MSE decreases as the border is expanded.Table 3
MSE of AR-BMSFP based on8
×
8 blocks (1024
×
768 image).
Border expansion
Zero pixel
One pixel
Two pixels
Three pixels
MSE
785.74
259.49
249.29
244.39


## 4.1. Comparison Experiments for Different Prior Models

In the first test, flexibility and superiority of the feature-driven prior are demonstrated. Firstly, the degraded low-resolution (LR) image sequence is generated from an original remote sensing image which is shown in Figure3(a). In the degradation process, original image is warped by homography and blurred by Gaussian point spread function with standard deviation of 0.8 high-resolution (HR) pixels, and then the warped and blurred image is decimated by factor 2 : 1 in horizontal and vertical directions, and finally the white Gaussian noise with standard deviation of 5 is added. Note that the intensity range of image is from 0 to 255. Two of the five degraded LR images are shown in Figures 3(b) and 3(c) which have been zoomed in the same size as original image for observation (all LR images in the following tests are also shown in this mode).SRR with different prior models (remote sensing image): (a) the original image, (b) one of the degraded LR images, (c) another of the degraded LR images, (d) SRR with Tikhonov prior model (MSE: 372.3448), (e) SRR with Farsiu BTV prior model (MSE: 292.9566), (f) SRR with Gaussian prior model (MSE: 68.8928), (g) SRR with Laplacian prior model (MSE: 68.7965), (h) SRR with elastic net prior model (MSE: 68.0170), and (i) SRR with feature-driven prior model (MSE: 67.5259).
(a)
(b)
(c)
(d)
(e)
(f)
(g)
(h)
(i)In the reconstruction process, the conventional preparation such as motion estimation [23] is omitted in order to focus on our algorithm. The information about warp, blurring, and decimation is supposed to be available; that is, the degraded matrix used in degradation can also be employed for reconstruction. In order to compare withthe feature-driven prior, firstly we reconstruct image using LR sequence based on Tikhonov prior and Farsiu BTV prior, respectively. The results are shown in Figures 3(d) and 3(e). Then two cost functions are formed, respectively, by substituting Gaussian prior model (4) into (3) and Laplacian prior model (5) into (3). Optimizing two cost functions above by scaled conjugate gradients (SCG) [24] yields two results of superresolution as shown in Figures 3(f) and 3(g). In order to convince that the feature-driven prior is a little bit better than the state-of-the-art elastic net prior model, we reconstruct image based on elastic net prior, the result of which is shown in Figure 3(h). Finally, another cost function is formed with the feature-driven prior based on (17). Optimizing this cost function by SCG yields the result of superresolution as shown in Figure 3(i).In order to demonstrate the applicability of the proposed prior model for different images, another standard image Lena is used to repeat this experiment. Figure4(a) is the part of the original Lena image. Figures 4(b) and 4(c) are two of the five degraded LR images. Figures 4(d)–4(i) are results of SRR with, in order, Tikhonov prior, Farsiu BTV prior, Gaussian prior, Laplacian prior, elastic net prior, and feature-driven prior.SRR with different prior models (Lena image): (a) the original image, (b) one of the degraded LR image, (c) another of the degraded LR image, (d) SRR with Tikhonov prior model (MSE: 226.4613), (e) SRR with Farsiu BTV prior model (MSE: 173.5646), (f) SRR with Gaussian prior model (MSE: 40.5469), (g) SRR with Laplacian prior model (MSE: 40.1860), (h) SRR with elastic net prior model (MSE: 39.3338), and (i) SRR with feature-driven prior model (MSE: 38.6370).
(a)
(b)
(c)
(d)
(e)
(f)
(g)
(h)
(i)It is important to note that the goal of Section4.1 is to compare different prior models, so the image is not split into blocks to perform SRR (i.e., the image-splitting scheme and border expansion are not required). All the experiments are performed for entire image. Therefore, there is no block artifact, that is, the cross artifact, in any results.The MSE results for remote sensing image and Lena image are listed in Table1. Figures 3 and 4 and Table 1 illustrate the advantage of the proposed prior compared with the other priors.Table 1
MSE of SSR with different prior models.
Prior model
Remote sensing image
Lena image
Tikhonov prior model
372.3448
226.4613
Farsiu BTV prior model
292.9566
173.5646
Gaussian prior model
68.8928
40.5469
Laplacian prior model
68.7965
40.1860
Elastic Net prior model
68.0170
39.3338
Proposed prior model
67.5259
38.6370


## 4.2. Comparison Experiments for Different Image-Splitting Schemes

In order to show that image-splitting scheme proposed in Section3.3 is effective, the second test is conducted. Certainly, the detailed algorithm to be validated is AR-BMSFP proposed in Section 3.4 because the cross artifact must be removed. So the border expansion must be utilized in image-splitting experiments. In the test, AR-BMSFP with different image-splitting schemes and several other algorithms without image-splitting scheme are compared with each other. The second test also includes two groups of experiments which use Peppers image and remote sensing image, respectively.The original Peppers image is shown in Figure5(a). In order to show the robustness and applicability of the algorithm, we made some adjustments on degradation process used in Section 4.1. To be specific, Gaussian point spread function is changed with standard deviation of 1.0 HR pixels, the decimation factor is changed with 4 : 1, and white Gaussian noise is amplified with standard deviation of 10.SRR with different image-splitting schemes (Peppers image). (a) Original image. (b) SRR with Tikhonov prior model (MSE: 576.640). (c) SRR with Farsiu BTV prior model (MSE: 554.615). (d) SRR with feature-driven prior model (MSE: 86.710). (e)2
×
2 block-based AR-BMSFP (MSE: 85.337). (f) 4
×
4 block-based AR-BMSFP (MSE: 84.456). (g) 8
×
8 block-based AR-BMSFP (MSE: 83.611).
(a)
(b)
(c)
(d)
(e)
(f)
(g)Then the AR-BMSFP SRR proposed in Section3.4 is compared with the method without image-splitting scheme to validate its superiority. Figure 5(b) shows the result of MAP SRR based on Tikhonov prior model without image-splitting scheme. Figure 5(c) shows the result of MAP SRR based on Farsiu BTV prior model without image-splitting scheme. Figure 5(d) shows the result of MAP SRR based on feature-driven prior model without image-splitting scheme.For AR-BMSFP algorithm, three experiments are conducted with2
×
2, 4
×
4, and 8
×
8 blocks, of which the results are shown in Figures 5(e), 5(f), and 5(g). It is worth noting that all these three results are obtained through several different border-expansion trials. Each final result is the best result selected from several trials. Therefore, the block artifact, that is, cross artifact, is not displayed in Figure 5.To prove the robustness of proposed method, the second group of experiments is conducted using remote sensing image. The degradation process is also repeated as in the first test, based on which the reconstruction process of first group of experiments is repeated. The results are shown in Figure6. It is also worth noting that the cross artifact has been removed in results through several border-expansion trials. So Figures 6(e), 6(f), and 6(g) have no artifacts.SRR with different image-splitting schemes (remote sensing image). (a) Original image. (b) SRR with Tikhonov prior model (MSE: 452.343). (c) SRR with Farsiu BTV prior model (MSE: 426.456). (d) SRR with feature-driven prior model (MSE: 73.703). (e)2
×
2 block-based AR-BMSFP (MSE: 72.967). (f) 4
×
4 block-based AR-BMSFP (MSE: 72.055). (g) 8
×
8 block-based AR-BMSFP (MSE: 71.980).
(a)
(b)
(c)
(d)
(e)
(f)
(g)The MSEs of two groups of experiments above are also listed in Table2. Both visual quality and MSE show that the AR-BMSFP algorithm is better than algorithm without image-splitting framework. Furthermore, MSE improves as the number of blocks increases.Table 2
MSE of SSR with different image-splitting schemes.
Algorithm
Peppers image
Remote sensing image
SRR with Tikhonov prior model
576.640
452.343
SRR with Farsiu BTV prior model
554.615
426.456
SRR with feature-driven prior model
86.710
73.703
2
×
2 block-based AR-BMSFP
85.337
72.967
4
×
4 block-based AR-BMSFP
84.456
72.055
8
×
8 block-based AR-BMSFP
83.611
71.980


## 4.3. Comparison Experiments for Different Border Expansions in Cross-Artifact Removal

The third test is a final test in this paper, which validates the capability of our solution for cross-artifact emerging in result. There are three groups of experiments in this test to show the robustness.As the cross artifact is only about 3 to 5 pixels width, the first two groups of experiments use small size images to make the cross artifact more visible. In the first group of experiments, a256
×
256 remote sensing image shown in Figure 7(a) is used. In the second group of experiments, a 128
×
128 magnetic resonance imaging (MRI) image shown in Figure 8(a) is used.AR-BMSFP based on8
×
8 blocks with different border expansions (remote sensing image): (a) the original image, (b) the degraded LR image, (c) the result without border expansion (MSE: 396.43), (d) the result with 1-pixel border expansion (MSE: 266.43), (e) the result with 2-pixel border expansion (MSE: 233.07), (f) the result with 3-pixel border expansion (MSE: 222.11), and (g) the result with 4-pixel border expansion (MSE: 218.48).
(a)
(b)
(c)
(d)
(e)
(f)
(g)AR-BMSFP based on8
×
8 blocks with different border expansions (MRI image): (a) the original image, (b) the degraded LR image, (c) the result without border expansion (MSE: 244.26), (d) the result with 1-pixel border expansion (MSE: 191.38), (e) the result with 3-pixel border expansion (MSE: 157.15), (f) the result with 4-pixel border expansion (MSE: 145.51), and (g) the result with 5-pixel border expansion (MSE: 143.29).
(a)
(b)
(c)
(d)
(e)
(f)
(g)The degradation of them is determined by the same method in Figure5. Figures 7(b) and 8(b) show one of the degraded LR images, respectively. According to the algorithm present in Algorithm 1, we begin to reconstruct image from LR sequence. For comparison, we select the same strategy to split the image; that is, the splitting index is fixed as 8
×
8.In order to illustrate effect with different border-expansion widths for blocks, we repeat the reconstruction with 0-, 1-, 2-, 3-, and 4-pixel border expansions for blocks in remote sensing image experiments. These results are shown in Figures7(c)–7(g). From result images, we can see that the cross artifact in Figure 7(c) is severe as no expansion, and cross artifact in Figures 7(d) and 7(e) is still obvious as thin border expansion. Due to the thick border expansion, Figures 7(f) and 7(g) appear without cross artifact. But in vision, Figure 7(g) is a little better than Figure 7(f), especially for the river in image. Besides, the MSEs also prove the judgments in vision. To make it clear, MSE values are illustrated in the graph of Figure 9(a). It is also obvious that as the border becomes wider the MSE will decrease.MSE values of AR-BMSFP based on8
×
8 blocks with different border expansions: (a) remote sensing image and (b) MIR image.
(a)
(b)Keeping all the conditions and conducts in the first group of experiments, we carry out the second group of experiments for a MRI image. The results and graph are shown in Figures8 and 9(b). The difference from previous is that we repeat the reconstruction with 0-, 1-, 2-, 3-, 4-, and 5-pixel border expansions. Note that due to need of layout, Figure 8 only includes results with 0-, 1-, 3-, 4-, and 5-pixel border expansions. But Figure 9(b) shows all results. From Figure 8, the effect of reconstruction is not as good as that of Figure 7. It is perhaps because of the fact that (
1
) the size of image is smaller than that in Figure 7 and (
2
) the texture of image is more monotonous than that in Figure 7. Even so, all results and graph still give the same conclusion as Figure 7. It also proves the robustness of our algorithm.No doubt it is time consuming to do the above experiment using bigger size image. But the capability to remove cross artifact is still effective for bigger size image. In order to prove this claim, a1024
×
768 image shown in Figure 10(a) is selected to repeat the above experiment.AR-BMSFP based on8
×
8 blocks with different border expansions (1024
×
768 image): (a) the original image, (b) the degraded LR image, (c) the result without border expansion (MSE: 785.74), (d) the result with 1-pixel border expansion (MSE: 259.49), (e) the result with 2-pixel border expansion (MSE: 249.29), and (f) the result with 3-pixel border expansion (MSE: 244.39).
(a)
(b)
(c)
(d)
(e)
(f)The degradation of the1024
×
768 image is the same as the method of the first two groups of experiments. A degraded LR image is shown in Figure 10(b). The AR-BMSFP reconstruction is conducted with 0-, 1-, 2-, and 3-pixel border expansions. The reconstructed results are shown in Figures 10(c)–10(f). Figure 10 verifies that the AR-BMSFP algorithm is effective for bigger size image. Additionally, the computed MSEs of all results are listed in Table 3 to prove that the MSE decreases as the border is expanded.Table 3
MSE of AR-BMSFP based on8
×
8 blocks (1024
×
768 image).
Border expansion
Zero pixel
One pixel
Two pixels
Three pixels
MSE
785.74
259.49
249.29
244.39


## 5. Conclusion

This paper introduces an effective algorithm to achieve superresolution. To make the prior model more accurate, the proposed algorithm supposes that the image prior is a variable prior based on feature instead of constant prior. Additionally, the proposed block-based framework enables the prior to characterize the local image rather than the entire image. By analysis and experiment we find that for each block the result has border effect which leads to the cross artifact as a result of superresolution. In order to remove this byproduct, we add a strategy in proposed algorithm, which expands the block several pixels before optimization and cuts the expanded border when recombining all blocks. This strategy is simple but effective for our algorithm.Experimental results validate the effectiveness of the proposed algorithm. In future, the focus of research will be on speeding up the algorithm and combining registration with reconstruction in SRR.



---

*Source: 508357-2014-03-18.xml*