# Multiframe Superresolution Reconstruction Based on Self-Learning Method

Authors: Shao-Shuo Mu; Ye Zhang; Ping Jia; Xun Yang; Xiao-Feng Qiu
Journal: Mathematical Problems in Engineering
 (2015)

---

## Abstract

One category of the superresolution algorithms widely used in practical applications is dictionary-based superresolution algorithms, which constructs a single high-resolution (HR) and high-clarity image from multiple low-resolution (LR) images. Despite the fact that general dictionary-based superresolution algorithms obtain redundant dictionaries from numerous HR-LR images, HR image distortion is unavoidable. To solve this problem, this paper proposes a multiframe superresolution reconstruction based on self-learning methods. First, multiple images from the same scene are selected to be both input and training images, and larger-scale images, which are also involved in the training set, are constructed from the learning dictionary. Then, different larger-scale images are constructed via repetition of the first step and the initial HR sets whose scale closely approximates that of the target HR image are finally obtained. Lastly, initial HR images are fused into one target HR image under the NLM idea, while the IBP idea is adopted to meet the global constraint. The simulation results demonstrate that the proposed algorithm produces more accurate reconstructions than those produced by other general superresolution algorithms, while, in real scene experiments, the proposed algorithm can run well and create clearer HR images from input images captured by cameras.


---

## Body

## 1. Introduction

In modern engineering, high-resolution (HR) images are always desirable which provides important information for making important decisions in various practical applications, for example, biometrics, airborne detection system, medical diagnosis, high definition television (HDTV), and remote surveillance. One way to obtain HR images is to increase sensor pixel resolution, which, however, leads to increased cost in sensors. A promising alternative is the superresolution technique, which reconstructs one HR image from one or more low-resolution (LR) images shot from the same scene, providing sufficient relevant information.In recent years, superresolution techniques have been extensively studied and put into practice. CEVA Company firstly combined superresolution techniques with CEVA-MM3101 low-power imaging and vision platform successfully, which used LR sensors to construct HR image, and have been applied to camera-enabled mobile devices. French company SPOT applied superresolution techniques to SPOT-5 satellite which can obtain clearer ground scene images and identify targets more accurately, compared with SPOT-4 which did not adopt superresolution techniques. So superresolution technique has been a hot spot to improve images’ resolution without changing the optical system of the camera.Superresolution techniques have three categories in general, namely, interpolation-based, reconstruction-based, and dictionary-based algorithms. Interpolation-based algorithms are simple but ineffective, in which the unknown image pixel values are calculated from the known pixels using interpolation formula [1–4], such as bilinear, bicubic, and Lanczos interpolations. Reconstruction-based algorithms have process inverse to imaging system solving. These algorithms use prior image knowledge and recover high-frequency information via established imaging models [5–10].Most recently, dictionary-based algorithms have been more widely applied due to the development in compressed sensing [11–15]. This category of algorithms requires a number of external training images, using similarity redundancy structure of natural images to restore missing high-frequency information in input images. Roweis and Saul [16] proposed a nonlinear dimension reduction method, namely, locally linear embedding (LLE), which has been widely applied to image processing. Based on [16], Chang et al. [17] applied LLE technology to superresolution reconstruction. In this method, the single input LR image is divided into blocks with image shift; then, each image block searches for k nearest blocks in all LR training image blocks whereby the optimal weights are obtained; finally, a HR image is constructed from a linear combination of the k HR training blocks corresponding to the k LR training blocks. References [18–20] improved the algorithm in [17] and achieved enhanced reconstruction results. Yang et al. [21] proposed a sparse coding method for dictionary learning. Compared with previous algorithms, their algorithm relies on a larger number of databases, while obtaining missing high-frequency information more effectively. In [22], their sparse coding is improved to strengthen the sparse similarity between LR and HR image blocks. This method is simpler and more effective compared with [21]. He et al. [23] adopted a beta process dictionary learning approach to make the HR and LR dictionaries more accurate and consistent, gaining better results in comparison with [22]. Jeong et al. [24] proposed a superresolution method robust to noise, which involves two phases: the learning phase and the reconstruction phase. In the learning phase, the training images are classified and different dictionaries are constructed according to noise quantity; and, in the reconstruction phase, the input images with various noise quantities adopt appropriate dictionaries to reconstruct HR images.All the above reconstruction methods require a large number of external training images. When external images cannot be accessed, it is feasible to start from the point of similarity redundancy structure of input images. In [25–28], different self-learning methods are proposed. Glasner et al. [25] realized superresolution reconstruction without external images by adopting the gradual magnification scheme. Chen et al. [26] used high-frequency and low-frequency components of input images to train dictionary image blocks. Then, they used the Approximate Nearest Neighbor Search to reconstruct images and recovered the missing high-frequency information. Zhang et al. [27] proposed a self-similarity redundant algorithm, which selects original LR images and corresponding degraded images as the training images and uses LLE algorithm to obtain one HR image. In [28], Ramakanth and Babu proposed single image learning dictionary and adopted the approximate nearest neighbor fields (ANNF) method to find similarities between input LR blocks and LR dictionary. This method reconstructs HR images more effectively and more efficiently compared with other dictionary-based methods.Other studies [25–28] have shown that natural images have self-similarity information on different scales and high-frequency information in small-scale images can be obtained from large-scale images. So this paper proposes a multiple-frame superresolution algorithm based on self-learning, and this algorithm requires no external training image. First, self-similarity information of multiple input images is used to create larger-scale training images through a step by step process. Then, an over-complete dictionary is studied by sparse representations. Finally, the reconstructed images are further improved by exerting the global constraint. Compared with [25–27], the proposed algorithm possesses several unique features as follows.(1)
Larger-scale images are created from input images. Both input images and larger-scale images are involved in the training set. Sparse representation is adopted to learn the mapping relation of the dictionary.
(2)
One larger-scale image is reconstructed from one input image; apply this reconstruction to alln input images. Through gradual magnification, n initial estimates are obtained, each corresponding to one of the input images, with a scale closely approximating that of the target HR image. Then, the NLM method is employed to fuse all estimates into one target HR image.
(3)
In order to restore more high-frequency parts, degraded images of input images are used as LR training set and high-frequency parts of input images are used as HR training set.Our theoretical analysis demonstrates that the proposed algorithm is feasible, and the simulation results show that the proposed algorithm performs better than general dictionary-based superresolution algorithms. In real scene experiments carried out on a camera system, the proposed algorithm obtains clearer real HR images.This paper is organized as follows. Section2 gives a discussion on image sparse models. Section 3 presents the framework of our proposed algorithm as well as its implementation, followed by experimental results in Section 4. Finally, Section 5 gives a conclusion of this paper.


## 2. Image Sparse Model

Superresolution reconstruction is a reverse evaluation problem. Dictionary-based superresolution reconstruction trains the corresponding LR dictionary and HR dictionary by mapping the relationship between LR images and HR images. Then, the relationship between LR dictionary and HR dictionary is solved, and a HR image is reconstructed with recovery of the missing high-frequency information. Imaging model between LR image and HR image is as shown in1: when the real scene is captured through the optical system, we obtain a group of LR image scenes due to the impact of optical blur and downsampling. Consider
(1)
Y
=
S
H
X
.Y is a LR image captured by the camera. X is the real scene image. H is the optical blur matrix. S is the downsampling matrix.Superresolution reconstruction is a highly ill-condition problem. Given one LR input imageY, many HR images X are suitable for formula 1. With the development of compressed sensing, dictionary-based superresolution methods have been applied more widely in image restoration areas. We can divide image into blocks y of the same dimension, every image block is represented by both sparse representation “a” and sparse matrix D, such as y
=
D
×
a, where D is a dictionary matrix and “a” is sparse coefficients. The basic process of dictionary-based reconstruction is shown in Figure 1.Figure 1
Simple diagram of general dictionary-based reconstruction algorithm.First, we require lots of training images and train the corresponding LR dictionary and HR dictionary by learning LR training images and HR training images. Second, the optimal sparse coefficients “a” of LR input image block are solved according to the equation min
⁡
⁡
(
y
-
D
l
×
a
). Then, we get the corresponding HR image block x via solving the formula D
l
×
a and fuse all HR image blocks into one HR image X.


## 3. The Proposed Method

When one LR image is given to reconstruct one HR image, the main problem of dictionary-based methods is how to find lots of HR-LR training images and utilize self-similarity information of external images. But it will cause the reconstructed HR image distortion, given the absence of the input images’ frequency information in the selected HR-LR training set. In this paper, we propose a multiple superresolution reconstruction method based on self-learning dictionary. We collect multiple images of the same scene as both input images and training image set, and larger-scale images, which are constructed from last training set, are also involved in the training set constantly. After that, we get the initial reconstruction images whose scale closely approximates target HR image. Then, we use NLM method to fuse the initial reconstruction images into one HR image. Lastly, IBP idea is adopted to satisfy the global reconstruction constraint. Here, four major parts are detailed in the following. In Section3.1, we mainly discuss how to use input images to gradually magnify image and create a large number of different larger-scale images as training set. In Section 3.2, sparse representation is adopted to jointly learn redundant dictionary. In Section 3.3, we employ NLM method to fuse initial reconstruction images into one HR image and introduce the importance of global reconstruction constraint. In Section 3.4, the working process of the proposed algorithm is introduced. In Section 3.5, we discuss how to simplify the operation.


### 3.1. Self-Learning Method

As we have said above, dictionary-based reconstruction algorithms need lots of training images to learn the overcomplete dictionary for constructing HR image. Literatures [25–28] point out that input image has enough self-similarity information and the high-frequency information of small-scale image can be obtained from large-scale image. They can use multiframe input images to reconstruct images accurately. Therefore, when external images cannot be accessed, this paper proposes a self-learning pyramid method which constructs larger-scale images gradually to join in training set. And it is feasible. We can fully learn all the redundant information from different scale images. First, assuming that the target HR image resolution is P times that of the input LR image resolution, we will enlarge input images resolution p
i times step by step, as the following formula:
(2)
p
1
×
p
2
×
⋯
×
p
n
×
s
=
P
,
s
<
p
i
,
i
=
1
,
…
,
n
,
p
1
×
p
2
×
⋯
×
p
n
≤
P
.Assuming that the input images areN
×
N dimension, there are two cases for the principle of p as follows:(a)
p
1
=
p
2
=
⋯
=
p
n
=
f
/
g; g
n
×
w
=
N (w
<
g and f, g all are integers.);
(b)
p
1
=
f
1
/
g
1
,
…
,
p
n
=
f
n
/
g
n; f
i
/
g
j
>
1 (f
i, g
j all are integers, s
>
p
i, and i
=
1,2
,
…
,
n).We discuss this paper based on condition (a).Then, the degraded imagesY
1
∧ are solved via the formula Y
1
↓
p
↑
p (↓
p is p times downsampling; ↑
p is p times upsampling), which together constitute the HR-LR training set Q
1 with high-frequency part of input images Y
1, as the formula 3. (The degraded images Y
1
∧ are LR training set. High-frequency part (Y
1
-
Y
1
∧) of input images Y
1 is HR training set.) This is the difference between this paper and [25–27], and our method can recover the edge high-frequency better. We train Q
1 to get dictionary D
1 and construct the corresponding larger-scale HR images Y
2 whose resolution is p times that of the input images. Then, degraded images Y
2
∧ are solved, which together constitute new HR-LR training set Q
2 with high-frequency part of Y
2 and Q
1 again. It is as shown in 3 that we reconstruct final HR images whose resolution is closest to target HR images, followed by n repeated learning. Process of training images is as shown in Figure 2. We enlarge every input image step by step just like pyramid. Consider
(3)
Q
1
=
Y
1
∧
∪
Y
1
-
Y
1
∧
,
Q
2
=
Y
2
∧
∪
Y
2
-
Y
2
∧
∪
Q
1
,
⋮
Q
m
=
Y
m
∧
∪
Y
m
-
Y
m
∧
∪
Q
m
-
1
.Figure 2
Process of training images.


### 3.2. Learning Dictionary

In the previous section, we have discussed the self-learning pyramid method. Chen et al. [26] and Zhang et al. [27] use image blocks manifold to learn dictionary. They solve the optimal weight between input image block and several nearest local training blocks. But it cannot effectively utilize global image information. Yang et al. [22] point out that sparse representation can utilize global image information better and construct universal dictionary with a better reconstructed image. So we propose jointly learning dictionary method based on sparse representation in this section. As shown in Section 3.1 that Q
m is the training set, we use images Y
m of Q
m as input images of next reconstruction. x and y are HR training blocks and LR training blocks respectively, which are represented by atomic linear combination of matrix D:  x
=
D
h
×
α; y
=
D
l
×
β (α and β are sparse coefficients. D
h and D
l are HR dictionary and LR dictionary, resp.). In this section, the premise of learning dictionary is that sparse coefficients α and β are the same. It is as shown in 4. Consider
(4)
y
=
D
l
×
α
,
x
=
D
h
×
α
.By solving the sparse coefficientsα of LR dictionary D
l, we can get HR image blocks according to x
=
D
h
×
α. This is the basic idea of training dictionary. Before reconstructing HR image, we must learn dictionaries D
l and D
h which must satisfy constraints of 4 firstly. So we can convert 4 into the following optimization problem, as shown in 5. Consider
(5)
min
⁡
α
0
,
s
.
t
.
y
-
D
l
×
α
2
2
≤
ε
,
min
⁡
α
0
,
s
.
t
.
y
-
D
h
×
α
2
2
≤
ε
.As long as the sparse coefficientsα are sparse enough, 5 is solved by solving L
1 norm problem, as shown in 6 and 7. Consider
(6)
argmin
1
2
x
-
D
h
×
α
2
2
+
μ
α
1
,
(7)
argmin
1
2
y
-
D
l
×
α
2
2
+
μ
α
1
.In order to get HR dictionary and LR dictionary of the same coefficientα, 6 and 7 can be converted into 8. Consider
(8)
argmin
1
2
P
-
D
×
α
2
2
+
μ
α
1
.In8,
(9)
P
=
x
N
y
M
,
D
=
D
h
N
D
l
M
,
where N is dimension of HR image blocks and M is dimension of LR image blocks. In order to solve 8, we set the initial value of matrix D and 8 can be converted into 10. Consider
(10)
min
⁡
1
2
P
-
D
×
α
T
P
-
D
×
α
+
μ
α
.Then, the transformation of10 is as shown in 11. Consider
(11)
min
⁡
1
2
Q
′
α
Q
+
p
′
α
+
μ
α
,
where Q
=
D
T
D and p
=
D
T
y. After obtaining α, we can solve 12 to get the value of matrix D. Consider
(12)
argmin
P
-
D
×
α
2
2
.
Therefore, dictionaries D
l and D
h are obtained. Then, we enlarge every input image of Y
m one by one, as shown in Figure 3. First, every input image is enlarged to A
l
m by interpolation and the upscale factor is p. Second, A
l
m is divided into blocks y of the same size. According to dictionaries D
l and D
h, we solve 7, 8, and 10 to get α. Then, HR image blocks are obtained according to the formula x
=
D
h
×
α. When all of the HR image blocks are gained, they are merged into one HR image A
h
m according to the fixed order. A
h
m
+
A
l
m is the larger-scale reconstructed image and also is one of reconstruction images Y
m
+
1, as shown in Figure 3. Lastly, we obtain the reconstruction images set Y
m
+
1 by using dictionary to construct every image in Y
m. Then, image sets Y
m
+
1 and Q
m
+
1 will become the input images and training images set of next reconstruction enlargement, respectively. Q
m
+
1 is as shown in 13. Consider
(13)
Q
m
+
1
=
Y
m
+
1
∧
∪
Y
m
+
1
-
Y
m
+
1
∧
∪
Q
m
.Figure 3
Process of constructing dictionary and reconstructing image (the upscale factor isp).


### 3.3. Reconstruction Fusion

In Sections3.1 and 3.2, we use self-learning pyramid method to construct training images and get reconstruction set Y
n
+
1. But the scale of reconstruction images is p
n times that of the original input images’ scale which may not be the same scale with target HR image. So we use NLM idea to fuse all images of Y
n
+
1 into one HR image which is the same scale with target HR image. Meanwhile, it can effectively eliminate fusion noise.Nonlocal mean (NLM) filter is a very effective denoising filter, which is based on an assumption that each pixel is a repetition of surrounding pixels. It can be used to obtain the target pixel value with a weighted sum of the surrounding pixels. Now, NLM is widely used in the field of superresolution reconstruction. Just as [9], when we use NLM idea to reconstruct multiframe images, the smaller the magnification is, the smaller the reconstruction error is. So when image dimension of Y
n
+
1 is closest to target HR image, using NLM idea to process Y
n
+
1 is feasible. In this section, we adopt 14 to fuse images of Y
n
+
1 into target HR image X
0. Consider
(14)
X
=
argmin
∑
(
k
,
l
)
∈
ω
∑
t
=
1
ρ
∑
(
i
,
j
)
∈
N
k
,
l
ω
k
,
l
,
i
,
j
,
t
×
D
k
,
l
E
k
,
l
H
X
-
E
k
,
l
L
y
t
,
where ω
(
k
,
l
,
i
,
j
,
t
) is the weight coefficient, usually calculated by the following formula. Consider
(15)
ω
k
,
l
,
i
,
j
,
t
=
exp
⁡
-
P
k
,
l
y
t
-
P
i
,
j
y
t
2
2
σ
2
.The(
k
,
l
) is a pixel of the target HR image. y
t represents one of the image sets, Y
n
+
1. (
i
,
j
) is a pixel of image y
t. P
k
,
l
y
t represents extracting image block from image y
t whose center is (
k
,
l
). After amplification and denoising of reconstruction image, we get the prime estimation image X
0. Finally, we must add global reconstruction constraint to meet Y
=
S
H
X, as shown in 16. Consider
(16)
X
=
argmin
y
-
S
H
X
2
2
+
μ
X
-
X
0
2
2
.Turn into the following formula17. Consider
(17)
X
=
X
0
+
S
H
T
y
-
S
H
X
+
μ
X
-
X
0
.We can solve17 to obtain the optimal estimated HR image X via IBP idea.


### 3.4. The Proposed Algorithm

The process of the proposed algorithm is as shown in the following writing based on the above sections. Suppose that we have four LR images of the same scene.Objective: Reconstruct HR Image from Multiple LR Images
(1)
Initialization: input four LR imagesY
1 and solve the corresponding degrade images by formula Y
1
↓
p
↑
p; construct the first training image set Q
1; set magnification p and iterations n.
(2)
Form
=
1
:
n, consider the following.
(a)
HR image and LR image of the HR-LR training setQ
m are divided into image blocks x and y, respectively. Training dictionaries D
h and D
l are solved according to section  (b) and the following formula. Consider
(18)
argmin
1
2
x
-
D
h
×
α
2
2
+
μ
α
1
,
argmin
1
2
y
-
D
l
×
α
2
2
+
μ
α
1
.
(b)
TakeY
m as new input LR images to be reconstructed. And solve new HR images Y
m
+
1 from dictionaries D
h and D
l. Then, we get four reconstruction images Y
m
+
1, in which resolution is p
m times of the original input images.
(c)
Put high-frequency ofY
m
+
1 and corresponding degrade images into the HR training set and LR training set, respectively. So we get new training image set according to section  (a) and 3. Take Y
m
+
1 as new LR images to be reconstructed. Consider
(19)
Y
m
∧
=
Y
m
↓
p
↑
p
,
Q
m
+
1
=
Y
m
+
1
∧
∪
Y
m
+
1
-
Y
m
+
1
∧
∪
Q
m
.
(3)
End
(4)
Adopt NLM idea to fuse images ofY
n
+
1 into one the same scale HR image X
0 with target image.
(5)
Add global reconstruction constraint to meetY
=
S
H
X. Use IBP idea and gradient descent algorithm to solve 16 and obtain the optimal reconstruction HR image X. Consider
(20)
X
=
X
0
+
S
H
T
y
-
S
H
X
+
μ
X
-
X
0
.
(6)
Output the final HR imageX.


### 3.5. The Discussion of Reconstruction Speed

When we apply the NLM method to adjust reconstruction image in Section3.3, it will spend much operation time by solving 14 directly. So, in this section, we will accelerate reconstruction speed by simplifying section  (c) in this section. Set derivation of the 14 to be equal to zero. We can get 21. Consider
(21)
X
=
∑
(
k
,
l
)
∈
ω
∑
t
=
1
ρ
∑
(
i
,
j
)
∈
N
k
,
l
ω
k
,
l
,
i
,
j
,
t
∑
k
,
l
∈
ω
∑
t
=
1
ρ
∑
i
,
j
∈
N
k
,
l
ω
k
,
l
,
i
,
j
,
t
×
E
k
,
l
H
T
D
k
,
l
T
D
k
,
l
E
k
,
l
H
-
1
×
∑
k
,
l
∈
ω
∑
t
=
1
ρ
∑
i
,
j
∈
N
k
,
l
ω
k
,
l
,
i
,
j
,
t
×
E
k
,
l
H
T
D
k
,
l
T
E
k
,
l
H
y
t
.
E
k
,
l
L represents extracting one pixel from the HR image. Equation 21 becomes 22. Consider
(22)
X
=
∑
t
=
1
ρ
∑
i
,
j
∈
N
k
,
l
ω
k
,
l
,
i
,
j
,
t
y
t
i
,
j
∑
t
=
1
ρ
∑
i
,
j
∈
N
k
,
l
ω
k
,
l
,
i
,
j
,
t
.As described above, it will reduce operation time and avoid complicated matrix operations. Meanwhile, this algorithm can be parallelized to deal with such steps as 2(a), 2(b), and(
4
) of Section 3.4. Therefore, we can take advantage of GPU to further improve the reconstruction speed. As shown in Section 4.2, we use Gforce 780 M GPU to process the captured images in camera system experimentation.Some blocks of the input LR image are smooth with few details, as shown in Figure4 (the black box). These blocks contain little high-frequency information and have no effect for reconstruction HR image. So this section proposes interpolation algorithm to enlarge these blocks, and the algorithm can run faster. Smooth image block is defined by the following formula:
(23)
A
=
1
p
2
z
¯
-
z
i
L
2
,
where z
i
L is the pixel value of image block and z
¯ is the average value of block pixel. We set a fixed value A
0. If A
<
A
0, we use interpolation method to reconstruct the block.Figure 4
The input LR images. (a) is one of several Lena images. (b) is one of several building images. (c) is one of several flower images.
(a)
(b)
(c)


## 3.1. Self-Learning Method

As we have said above, dictionary-based reconstruction algorithms need lots of training images to learn the overcomplete dictionary for constructing HR image. Literatures [25–28] point out that input image has enough self-similarity information and the high-frequency information of small-scale image can be obtained from large-scale image. They can use multiframe input images to reconstruct images accurately. Therefore, when external images cannot be accessed, this paper proposes a self-learning pyramid method which constructs larger-scale images gradually to join in training set. And it is feasible. We can fully learn all the redundant information from different scale images. First, assuming that the target HR image resolution is P times that of the input LR image resolution, we will enlarge input images resolution p
i times step by step, as the following formula:
(2)
p
1
×
p
2
×
⋯
×
p
n
×
s
=
P
,
s
<
p
i
,
i
=
1
,
…
,
n
,
p
1
×
p
2
×
⋯
×
p
n
≤
P
.Assuming that the input images areN
×
N dimension, there are two cases for the principle of p as follows:(a)
p
1
=
p
2
=
⋯
=
p
n
=
f
/
g; g
n
×
w
=
N (w
<
g and f, g all are integers.);
(b)
p
1
=
f
1
/
g
1
,
…
,
p
n
=
f
n
/
g
n; f
i
/
g
j
>
1 (f
i, g
j all are integers, s
>
p
i, and i
=
1,2
,
…
,
n).We discuss this paper based on condition (a).Then, the degraded imagesY
1
∧ are solved via the formula Y
1
↓
p
↑
p (↓
p is p times downsampling; ↑
p is p times upsampling), which together constitute the HR-LR training set Q
1 with high-frequency part of input images Y
1, as the formula 3. (The degraded images Y
1
∧ are LR training set. High-frequency part (Y
1
-
Y
1
∧) of input images Y
1 is HR training set.) This is the difference between this paper and [25–27], and our method can recover the edge high-frequency better. We train Q
1 to get dictionary D
1 and construct the corresponding larger-scale HR images Y
2 whose resolution is p times that of the input images. Then, degraded images Y
2
∧ are solved, which together constitute new HR-LR training set Q
2 with high-frequency part of Y
2 and Q
1 again. It is as shown in 3 that we reconstruct final HR images whose resolution is closest to target HR images, followed by n repeated learning. Process of training images is as shown in Figure 2. We enlarge every input image step by step just like pyramid. Consider
(3)
Q
1
=
Y
1
∧
∪
Y
1
-
Y
1
∧
,
Q
2
=
Y
2
∧
∪
Y
2
-
Y
2
∧
∪
Q
1
,
⋮
Q
m
=
Y
m
∧
∪
Y
m
-
Y
m
∧
∪
Q
m
-
1
.Figure 2
Process of training images.


## 3.2. Learning Dictionary

In the previous section, we have discussed the self-learning pyramid method. Chen et al. [26] and Zhang et al. [27] use image blocks manifold to learn dictionary. They solve the optimal weight between input image block and several nearest local training blocks. But it cannot effectively utilize global image information. Yang et al. [22] point out that sparse representation can utilize global image information better and construct universal dictionary with a better reconstructed image. So we propose jointly learning dictionary method based on sparse representation in this section. As shown in Section 3.1 that Q
m is the training set, we use images Y
m of Q
m as input images of next reconstruction. x and y are HR training blocks and LR training blocks respectively, which are represented by atomic linear combination of matrix D:  x
=
D
h
×
α; y
=
D
l
×
β (α and β are sparse coefficients. D
h and D
l are HR dictionary and LR dictionary, resp.). In this section, the premise of learning dictionary is that sparse coefficients α and β are the same. It is as shown in 4. Consider
(4)
y
=
D
l
×
α
,
x
=
D
h
×
α
.By solving the sparse coefficientsα of LR dictionary D
l, we can get HR image blocks according to x
=
D
h
×
α. This is the basic idea of training dictionary. Before reconstructing HR image, we must learn dictionaries D
l and D
h which must satisfy constraints of 4 firstly. So we can convert 4 into the following optimization problem, as shown in 5. Consider
(5)
min
⁡
α
0
,
s
.
t
.
y
-
D
l
×
α
2
2
≤
ε
,
min
⁡
α
0
,
s
.
t
.
y
-
D
h
×
α
2
2
≤
ε
.As long as the sparse coefficientsα are sparse enough, 5 is solved by solving L
1 norm problem, as shown in 6 and 7. Consider
(6)
argmin
1
2
x
-
D
h
×
α
2
2
+
μ
α
1
,
(7)
argmin
1
2
y
-
D
l
×
α
2
2
+
μ
α
1
.In order to get HR dictionary and LR dictionary of the same coefficientα, 6 and 7 can be converted into 8. Consider
(8)
argmin
1
2
P
-
D
×
α
2
2
+
μ
α
1
.In8,
(9)
P
=
x
N
y
M
,
D
=
D
h
N
D
l
M
,
where N is dimension of HR image blocks and M is dimension of LR image blocks. In order to solve 8, we set the initial value of matrix D and 8 can be converted into 10. Consider
(10)
min
⁡
1
2
P
-
D
×
α
T
P
-
D
×
α
+
μ
α
.Then, the transformation of10 is as shown in 11. Consider
(11)
min
⁡
1
2
Q
′
α
Q
+
p
′
α
+
μ
α
,
where Q
=
D
T
D and p
=
D
T
y. After obtaining α, we can solve 12 to get the value of matrix D. Consider
(12)
argmin
P
-
D
×
α
2
2
.
Therefore, dictionaries D
l and D
h are obtained. Then, we enlarge every input image of Y
m one by one, as shown in Figure 3. First, every input image is enlarged to A
l
m by interpolation and the upscale factor is p. Second, A
l
m is divided into blocks y of the same size. According to dictionaries D
l and D
h, we solve 7, 8, and 10 to get α. Then, HR image blocks are obtained according to the formula x
=
D
h
×
α. When all of the HR image blocks are gained, they are merged into one HR image A
h
m according to the fixed order. A
h
m
+
A
l
m is the larger-scale reconstructed image and also is one of reconstruction images Y
m
+
1, as shown in Figure 3. Lastly, we obtain the reconstruction images set Y
m
+
1 by using dictionary to construct every image in Y
m. Then, image sets Y
m
+
1 and Q
m
+
1 will become the input images and training images set of next reconstruction enlargement, respectively. Q
m
+
1 is as shown in 13. Consider
(13)
Q
m
+
1
=
Y
m
+
1
∧
∪
Y
m
+
1
-
Y
m
+
1
∧
∪
Q
m
.Figure 3
Process of constructing dictionary and reconstructing image (the upscale factor isp).


## 3.3. Reconstruction Fusion

In Sections3.1 and 3.2, we use self-learning pyramid method to construct training images and get reconstruction set Y
n
+
1. But the scale of reconstruction images is p
n times that of the original input images’ scale which may not be the same scale with target HR image. So we use NLM idea to fuse all images of Y
n
+
1 into one HR image which is the same scale with target HR image. Meanwhile, it can effectively eliminate fusion noise.Nonlocal mean (NLM) filter is a very effective denoising filter, which is based on an assumption that each pixel is a repetition of surrounding pixels. It can be used to obtain the target pixel value with a weighted sum of the surrounding pixels. Now, NLM is widely used in the field of superresolution reconstruction. Just as [9], when we use NLM idea to reconstruct multiframe images, the smaller the magnification is, the smaller the reconstruction error is. So when image dimension of Y
n
+
1 is closest to target HR image, using NLM idea to process Y
n
+
1 is feasible. In this section, we adopt 14 to fuse images of Y
n
+
1 into target HR image X
0. Consider
(14)
X
=
argmin
∑
(
k
,
l
)
∈
ω
∑
t
=
1
ρ
∑
(
i
,
j
)
∈
N
k
,
l
ω
k
,
l
,
i
,
j
,
t
×
D
k
,
l
E
k
,
l
H
X
-
E
k
,
l
L
y
t
,
where ω
(
k
,
l
,
i
,
j
,
t
) is the weight coefficient, usually calculated by the following formula. Consider
(15)
ω
k
,
l
,
i
,
j
,
t
=
exp
⁡
-
P
k
,
l
y
t
-
P
i
,
j
y
t
2
2
σ
2
.The(
k
,
l
) is a pixel of the target HR image. y
t represents one of the image sets, Y
n
+
1. (
i
,
j
) is a pixel of image y
t. P
k
,
l
y
t represents extracting image block from image y
t whose center is (
k
,
l
). After amplification and denoising of reconstruction image, we get the prime estimation image X
0. Finally, we must add global reconstruction constraint to meet Y
=
S
H
X, as shown in 16. Consider
(16)
X
=
argmin
y
-
S
H
X
2
2
+
μ
X
-
X
0
2
2
.Turn into the following formula17. Consider
(17)
X
=
X
0
+
S
H
T
y
-
S
H
X
+
μ
X
-
X
0
.We can solve17 to obtain the optimal estimated HR image X via IBP idea.


## 3.4. The Proposed Algorithm

The process of the proposed algorithm is as shown in the following writing based on the above sections. Suppose that we have four LR images of the same scene.Objective: Reconstruct HR Image from Multiple LR Images
(1)
Initialization: input four LR imagesY
1 and solve the corresponding degrade images by formula Y
1
↓
p
↑
p; construct the first training image set Q
1; set magnification p and iterations n.
(2)
Form
=
1
:
n, consider the following.
(a)
HR image and LR image of the HR-LR training setQ
m are divided into image blocks x and y, respectively. Training dictionaries D
h and D
l are solved according to section  (b) and the following formula. Consider
(18)
argmin
1
2
x
-
D
h
×
α
2
2
+
μ
α
1
,
argmin
1
2
y
-
D
l
×
α
2
2
+
μ
α
1
.
(b)
TakeY
m as new input LR images to be reconstructed. And solve new HR images Y
m
+
1 from dictionaries D
h and D
l. Then, we get four reconstruction images Y
m
+
1, in which resolution is p
m times of the original input images.
(c)
Put high-frequency ofY
m
+
1 and corresponding degrade images into the HR training set and LR training set, respectively. So we get new training image set according to section  (a) and 3. Take Y
m
+
1 as new LR images to be reconstructed. Consider
(19)
Y
m
∧
=
Y
m
↓
p
↑
p
,
Q
m
+
1
=
Y
m
+
1
∧
∪
Y
m
+
1
-
Y
m
+
1
∧
∪
Q
m
.
(3)
End
(4)
Adopt NLM idea to fuse images ofY
n
+
1 into one the same scale HR image X
0 with target image.
(5)
Add global reconstruction constraint to meetY
=
S
H
X. Use IBP idea and gradient descent algorithm to solve 16 and obtain the optimal reconstruction HR image X. Consider
(20)
X
=
X
0
+
S
H
T
y
-
S
H
X
+
μ
X
-
X
0
.
(6)
Output the final HR imageX.


## 3.5. The Discussion of Reconstruction Speed

When we apply the NLM method to adjust reconstruction image in Section3.3, it will spend much operation time by solving 14 directly. So, in this section, we will accelerate reconstruction speed by simplifying section  (c) in this section. Set derivation of the 14 to be equal to zero. We can get 21. Consider
(21)
X
=
∑
(
k
,
l
)
∈
ω
∑
t
=
1
ρ
∑
(
i
,
j
)
∈
N
k
,
l
ω
k
,
l
,
i
,
j
,
t
∑
k
,
l
∈
ω
∑
t
=
1
ρ
∑
i
,
j
∈
N
k
,
l
ω
k
,
l
,
i
,
j
,
t
×
E
k
,
l
H
T
D
k
,
l
T
D
k
,
l
E
k
,
l
H
-
1
×
∑
k
,
l
∈
ω
∑
t
=
1
ρ
∑
i
,
j
∈
N
k
,
l
ω
k
,
l
,
i
,
j
,
t
×
E
k
,
l
H
T
D
k
,
l
T
E
k
,
l
H
y
t
.
E
k
,
l
L represents extracting one pixel from the HR image. Equation 21 becomes 22. Consider
(22)
X
=
∑
t
=
1
ρ
∑
i
,
j
∈
N
k
,
l
ω
k
,
l
,
i
,
j
,
t
y
t
i
,
j
∑
t
=
1
ρ
∑
i
,
j
∈
N
k
,
l
ω
k
,
l
,
i
,
j
,
t
.As described above, it will reduce operation time and avoid complicated matrix operations. Meanwhile, this algorithm can be parallelized to deal with such steps as 2(a), 2(b), and(
4
) of Section 3.4. Therefore, we can take advantage of GPU to further improve the reconstruction speed. As shown in Section 4.2, we use Gforce 780 M GPU to process the captured images in camera system experimentation.Some blocks of the input LR image are smooth with few details, as shown in Figure4 (the black box). These blocks contain little high-frequency information and have no effect for reconstruction HR image. So this section proposes interpolation algorithm to enlarge these blocks, and the algorithm can run faster. Smooth image block is defined by the following formula:
(23)
A
=
1
p
2
z
¯
-
z
i
L
2
,
where z
i
L is the pixel value of image block and z
¯ is the average value of block pixel. We set a fixed value A
0. If A
<
A
0, we use interpolation method to reconstruct the block.Figure 4
The input LR images. (a) is one of several Lena images. (b) is one of several building images. (c) is one of several flower images.
(a)
(b)
(c)


## 4. Experimental Results and Analysis

### 4.1. Simulation Experimentation

In this section, we perform 2x magnification experiments on three test images from Figure4 to validate the effectiveness of the proposed method. These images cover a lot of contents including Lena, flower, and building, as shown in Figure 4. The compared group includes five other methods: the interpolation-based method [4], the NLM method based on reconstruction [8], Glassner’s method [25], Yang’s method [23], and Zhang’s method [27]. For example, the input LR Lena images are simulated from a 256 × 256 normative HR image revolved, translated randomly, and downsampled by a factor of two. The image block size is set to 5 × 5 with overlap of 4 pixels between adjacent blocks. And upscale factor P is 2. The gradual enlargement p is set to 1.25, n is 3, and s is 128
/
125. Although the proposed method has a certain similarity in the idea of dictionary-based reconstruction [23], it takes the following unique features.(1)
This paper proposes self-learning pyramid method to construct larger-scale image step by step. And it does not need to search lots of external HR-LR images as a training set.
(2)
The reconstructed result depends on the selected training images. It will cause to the reconstructed HR image distortion, given the absence of the input images’ frequency information in the selected HR-LR training set. This method puts different scale reconstructed image into the training set, and it can provide much real similarity information to ensure the authenticity of the reconstruction HR image.
(3)
Because the reconstruction results have fusion noise and may not be the same scale with target HR image in Section3.3, we use NLM idea to meet both cases and merge them into one HR image, which contains more target image information. Lastly, we use IBP idea to meet global reconstruction constraint.Figures5, 6, and 7 are the reconstruction results of the testing images Lena, flower, and building.Figure 5
Comparison of reconstructed HR images of Lena. (a) Original HR image. (b) Interpolation-based method results [4]. (c) NLM results [8]. (d) Glassner’s method [25]. (e) Yang’s method [23]. (f) Zhang’s method [27]. (g) The proposed method results.
(a)
(b)
(c)
(d)
(e)
(f)
(g)Figure 6
Comparison of reconstructed HR images of building. (a) Original HR image. (b) Interpolation-based method results [4]. (c) NLM results [8]. (d) Glassner’s method [25]. (e) Yang’s method [23]. (f) Zhang’s method [27]. (g) The proposed method results.
(a)
(b)
(c)
(d)
(e)
(f)
(g)Figure 7
Comparison of reconstructed HR images of flower. (a) Original HR image. (b) Interpolation-based method results [4]. (c) NLM results [8]. (d) Glassner’s method [25]. (e) Yang’s method [23]. (f) Zhang’s method [27]. (g) The proposed method results.
(a)
(b)
(c)
(d)
(e)
(f)
(g)We prove the superiority of the proposed algorithm from the visual quality and objective quality of the reconstructed HR image. And peak signal to noise ratio (PSNR), structural similarity (SSIM), mean squared error (MSE), and mean absolute error (MAE) are employed to evaluate the objective quality of the reconstructed HR image. Definitions are as follows.Mean squared error is as follows:(24)
MSE
=
1
M
N
∑
i
=
1
M
∑
j
=
1
N
g
i
,
j
-
f
i
,
j
2
.Peak signal to noise ratio is as follows:(25)
PSNR
=
10
log
⁡
10
⁡
255
2
MSE
.Mean absolute error is as follows:(26)
MAE
=
1
M
N
∑
i
=
1
M
∑
j
=
1
N
g
i
,
j
-
g
-
.Theg
(
i
,
j
) and f
(
i
,
j
) are pixels of the reconstructed HR image and the original HR image, respectively. The M and N are dimensions of reconstructed HR image. The g
- is the mean value of all pixels of reconstructed HR image. When PSNR is larger and MAE is smaller, the quality of the reconstructed image is better. SSIM is to show the reconstructed quality by comparing image intensity, image contrast, and structural similarity based on the visual system. The larger the SSIM is, the clearer the image is. Values of PSNR, SSIM, and MAE are shown in Table 1. The first, second, and third rows for each test image indicate PSNR, MAE, and SSIM values, respectively. Table 1 suggests that the proposed algorithm can achieve better reconstructed result and recover more missing high-frequency information and clearer edge details in comparison with other five methods.Table 1
The results of PSNR, MAE, and SSIM. For every image, there are three rows. The first, second, and third rows for each test image indicate PSNR, MAE, and SSIM values.
Interpolation [4]
NLM method [8]
Glassner’s method [25]
Yang’s method [23]
Zhang’s method [27]
The proposed method
Lena
27.3993
27.4644
30.6150
31.0123
31.5623
32.6016
6.0320
5.8437
4.2831
3.9561
3.8601
3.7735
0.8631
0.8643
0.9051
0.9168
0.9250
0.9538
Building
31.2571
31.2962
37.1000
37.0549
36.9531
37.6990
4.1998
4.0993
2.3300
2.3254
2.5365
2.0510
0.8746
0.8753
0.9537
0.9524
0.9124
0.9653
Flower
26.3318
25.7424
30.5731
31.2439
31.3826
32.1054
8.0847
8.3127
5.240
4.6710
4.6425
4.4371
0.8176
0.7931
0.9031
0.9280
0.9315
0.9501
Average
28.3294
28.1676
32.76
33.1037
33.2993
34.1354
6.1055
6.0852
3.9510
3.6508
3.6797
3.4205
0.8517
0.8442
0.9206
0.9324
0.9229
0.9564To further assess the visual quality obtained by different methods, we compare reconstructed image details of different methods with original HR image, as shown in Figures5, 6, and 7. When we contrast eye and the brim of a hat in Figure 5, the result of interpolation-based method is vaguer and NLM method’s reconstructed result appears distortion and mosaic. On the contrary, the proposed method does not cause distortion and vague compared with interpolation-based and NLM cases. Moreover, reconstructed details of our method are more delicate than other algorithms. In Figures 6 and 7, the proposed method can also produce better reconstructed details, eaves and flower border, both accurately and visually in comparison with other superresolution algorithms.


### 4.2. Camera System Experimentation

In this section, we conduct research on real scene reconstruction to prove the superiority of the proposed algorithm. It is the camera system assembled by ourselves, which consists of a microdisplacement lens, PI turntable, and Gforce 780 M GPU. By controlling the PI turntable, we can continuously receive multiple real images, as shown in Figure8(a) (one of multiple real images). The images are taken to be reconstructed by our method, and result is as shown in Figure 8(f). Figures 8(b)–8(e) are the reconstruction results of the NLM method based on reconstruction [9], Glassner’s method [25], Yang’s method, and Zhang’s method [27]. Signal noise ratio (SNR), average gradient (AG), information entropy (IE), and standard deviation (SD), which do not require reference HR image, are employed to evaluate the objective quality of the reconstructed real HR image in real scene reconstruction of camera system. Definitions are as follows. Consider
(27)
SNR
=
M
ST
D
max
⁡
.Figure 8
Real scene reconstruction. (a) One of the captured images. (b) NLM results [8]. (c) Glassner’s method [25]. (d) Yang’s method [23]. (e) Zhang’s method [27]. (f) The proposed method results.
(a)
(b)
(c)
(d)
(e)
(f)M represents image mean value. The ST
D
max
⁡ represents the max value of local standard deviation. Consider
(28)
IE
=
∑
i
=
0
L
-
1
P
i
lo
g
2
P
i
.L is the total image gray level, and P
i represents the ratio of pixels in gray value i to pixels in all image gray values. Consider
(29)
AG
=
∑
i
=
1
M
-
1
∑
j
=
1
N
-
1
∂
g
x
i
,
y
j
∂
x
i
2
+
∂
g
x
i
,
y
j
y
j
2
×
1
2
×
M
-
1
N
-
1
-
1
,
SD
=
∑
i
=
1
M
∑
j
=
1
N
g
x
i
,
y
j
-
g
¯
2
M
×
N
.The average gradient represents clarity. Information entropy can indicate image information abundance. Standard deviation represents discrete situation of image gray level. When SNR, SD, IE, and AG are larger, the reconstructed image is clearer and contains richer information. Therefore, the numeric results of proposed method are better than those of other algorithms as shown in Table2. Visually, in comparison of contour outline of the reconstructed image detail, the reconstructed image in Figure 8(f) is significantly clearer than the other reconstructed image. Consequently, the proposed method can work out well in the camera system experimentation and be capable of reproducing plausible details and sharp edges with minimal artifacts.Table 2
The results of SNR, AG (average gradient), IE (information entropy), and SD (standard deviation).
Real scene
NLM method [8]
Glassner’s method [25]
Yang’s method [23]
Zhang’s method [27]
The proposed method
SNR
5.3115
6.7257
6.6978
6.7899
6.8049
7.1254
AG
1.1059
1.1895
1.7399
1.7296
1.7327
1.8138
IE
6.3191
6.3151
6.3536
6.3411
6.3609
6.4141
SD
26.2563
26.2807
26.2999
26.3164
26.3043
26.3733


## 4.1. Simulation Experimentation

In this section, we perform 2x magnification experiments on three test images from Figure4 to validate the effectiveness of the proposed method. These images cover a lot of contents including Lena, flower, and building, as shown in Figure 4. The compared group includes five other methods: the interpolation-based method [4], the NLM method based on reconstruction [8], Glassner’s method [25], Yang’s method [23], and Zhang’s method [27]. For example, the input LR Lena images are simulated from a 256 × 256 normative HR image revolved, translated randomly, and downsampled by a factor of two. The image block size is set to 5 × 5 with overlap of 4 pixels between adjacent blocks. And upscale factor P is 2. The gradual enlargement p is set to 1.25, n is 3, and s is 128
/
125. Although the proposed method has a certain similarity in the idea of dictionary-based reconstruction [23], it takes the following unique features.(1)
This paper proposes self-learning pyramid method to construct larger-scale image step by step. And it does not need to search lots of external HR-LR images as a training set.
(2)
The reconstructed result depends on the selected training images. It will cause to the reconstructed HR image distortion, given the absence of the input images’ frequency information in the selected HR-LR training set. This method puts different scale reconstructed image into the training set, and it can provide much real similarity information to ensure the authenticity of the reconstruction HR image.
(3)
Because the reconstruction results have fusion noise and may not be the same scale with target HR image in Section3.3, we use NLM idea to meet both cases and merge them into one HR image, which contains more target image information. Lastly, we use IBP idea to meet global reconstruction constraint.Figures5, 6, and 7 are the reconstruction results of the testing images Lena, flower, and building.Figure 5
Comparison of reconstructed HR images of Lena. (a) Original HR image. (b) Interpolation-based method results [4]. (c) NLM results [8]. (d) Glassner’s method [25]. (e) Yang’s method [23]. (f) Zhang’s method [27]. (g) The proposed method results.
(a)
(b)
(c)
(d)
(e)
(f)
(g)Figure 6
Comparison of reconstructed HR images of building. (a) Original HR image. (b) Interpolation-based method results [4]. (c) NLM results [8]. (d) Glassner’s method [25]. (e) Yang’s method [23]. (f) Zhang’s method [27]. (g) The proposed method results.
(a)
(b)
(c)
(d)
(e)
(f)
(g)Figure 7
Comparison of reconstructed HR images of flower. (a) Original HR image. (b) Interpolation-based method results [4]. (c) NLM results [8]. (d) Glassner’s method [25]. (e) Yang’s method [23]. (f) Zhang’s method [27]. (g) The proposed method results.
(a)
(b)
(c)
(d)
(e)
(f)
(g)We prove the superiority of the proposed algorithm from the visual quality and objective quality of the reconstructed HR image. And peak signal to noise ratio (PSNR), structural similarity (SSIM), mean squared error (MSE), and mean absolute error (MAE) are employed to evaluate the objective quality of the reconstructed HR image. Definitions are as follows.Mean squared error is as follows:(24)
MSE
=
1
M
N
∑
i
=
1
M
∑
j
=
1
N
g
i
,
j
-
f
i
,
j
2
.Peak signal to noise ratio is as follows:(25)
PSNR
=
10
log
⁡
10
⁡
255
2
MSE
.Mean absolute error is as follows:(26)
MAE
=
1
M
N
∑
i
=
1
M
∑
j
=
1
N
g
i
,
j
-
g
-
.Theg
(
i
,
j
) and f
(
i
,
j
) are pixels of the reconstructed HR image and the original HR image, respectively. The M and N are dimensions of reconstructed HR image. The g
- is the mean value of all pixels of reconstructed HR image. When PSNR is larger and MAE is smaller, the quality of the reconstructed image is better. SSIM is to show the reconstructed quality by comparing image intensity, image contrast, and structural similarity based on the visual system. The larger the SSIM is, the clearer the image is. Values of PSNR, SSIM, and MAE are shown in Table 1. The first, second, and third rows for each test image indicate PSNR, MAE, and SSIM values, respectively. Table 1 suggests that the proposed algorithm can achieve better reconstructed result and recover more missing high-frequency information and clearer edge details in comparison with other five methods.Table 1
The results of PSNR, MAE, and SSIM. For every image, there are three rows. The first, second, and third rows for each test image indicate PSNR, MAE, and SSIM values.
Interpolation [4]
NLM method [8]
Glassner’s method [25]
Yang’s method [23]
Zhang’s method [27]
The proposed method
Lena
27.3993
27.4644
30.6150
31.0123
31.5623
32.6016
6.0320
5.8437
4.2831
3.9561
3.8601
3.7735
0.8631
0.8643
0.9051
0.9168
0.9250
0.9538
Building
31.2571
31.2962
37.1000
37.0549
36.9531
37.6990
4.1998
4.0993
2.3300
2.3254
2.5365
2.0510
0.8746
0.8753
0.9537
0.9524
0.9124
0.9653
Flower
26.3318
25.7424
30.5731
31.2439
31.3826
32.1054
8.0847
8.3127
5.240
4.6710
4.6425
4.4371
0.8176
0.7931
0.9031
0.9280
0.9315
0.9501
Average
28.3294
28.1676
32.76
33.1037
33.2993
34.1354
6.1055
6.0852
3.9510
3.6508
3.6797
3.4205
0.8517
0.8442
0.9206
0.9324
0.9229
0.9564To further assess the visual quality obtained by different methods, we compare reconstructed image details of different methods with original HR image, as shown in Figures5, 6, and 7. When we contrast eye and the brim of a hat in Figure 5, the result of interpolation-based method is vaguer and NLM method’s reconstructed result appears distortion and mosaic. On the contrary, the proposed method does not cause distortion and vague compared with interpolation-based and NLM cases. Moreover, reconstructed details of our method are more delicate than other algorithms. In Figures 6 and 7, the proposed method can also produce better reconstructed details, eaves and flower border, both accurately and visually in comparison with other superresolution algorithms.


## 4.2. Camera System Experimentation

In this section, we conduct research on real scene reconstruction to prove the superiority of the proposed algorithm. It is the camera system assembled by ourselves, which consists of a microdisplacement lens, PI turntable, and Gforce 780 M GPU. By controlling the PI turntable, we can continuously receive multiple real images, as shown in Figure8(a) (one of multiple real images). The images are taken to be reconstructed by our method, and result is as shown in Figure 8(f). Figures 8(b)–8(e) are the reconstruction results of the NLM method based on reconstruction [9], Glassner’s method [25], Yang’s method, and Zhang’s method [27]. Signal noise ratio (SNR), average gradient (AG), information entropy (IE), and standard deviation (SD), which do not require reference HR image, are employed to evaluate the objective quality of the reconstructed real HR image in real scene reconstruction of camera system. Definitions are as follows. Consider
(27)
SNR
=
M
ST
D
max
⁡
.Figure 8
Real scene reconstruction. (a) One of the captured images. (b) NLM results [8]. (c) Glassner’s method [25]. (d) Yang’s method [23]. (e) Zhang’s method [27]. (f) The proposed method results.
(a)
(b)
(c)
(d)
(e)
(f)M represents image mean value. The ST
D
max
⁡ represents the max value of local standard deviation. Consider
(28)
IE
=
∑
i
=
0
L
-
1
P
i
lo
g
2
P
i
.L is the total image gray level, and P
i represents the ratio of pixels in gray value i to pixels in all image gray values. Consider
(29)
AG
=
∑
i
=
1
M
-
1
∑
j
=
1
N
-
1
∂
g
x
i
,
y
j
∂
x
i
2
+
∂
g
x
i
,
y
j
y
j
2
×
1
2
×
M
-
1
N
-
1
-
1
,
SD
=
∑
i
=
1
M
∑
j
=
1
N
g
x
i
,
y
j
-
g
¯
2
M
×
N
.The average gradient represents clarity. Information entropy can indicate image information abundance. Standard deviation represents discrete situation of image gray level. When SNR, SD, IE, and AG are larger, the reconstructed image is clearer and contains richer information. Therefore, the numeric results of proposed method are better than those of other algorithms as shown in Table2. Visually, in comparison of contour outline of the reconstructed image detail, the reconstructed image in Figure 8(f) is significantly clearer than the other reconstructed image. Consequently, the proposed method can work out well in the camera system experimentation and be capable of reproducing plausible details and sharp edges with minimal artifacts.Table 2
The results of SNR, AG (average gradient), IE (information entropy), and SD (standard deviation).
Real scene
NLM method [8]
Glassner’s method [25]
Yang’s method [23]
Zhang’s method [27]
The proposed method
SNR
5.3115
6.7257
6.6978
6.7899
6.8049
7.1254
AG
1.1059
1.1895
1.7399
1.7296
1.7327
1.8138
IE
6.3191
6.3151
6.3536
6.3411
6.3609
6.4141
SD
26.2563
26.2807
26.2999
26.3164
26.3043
26.3733


## 5. Conclusion

It is known that, in superresolution reconstruction, distortion of reconstructed images occurs when frequency information of target images is not contained in training images. To solve this problem, this paper proposes a multiple superresolution reconstruction algorithm based on self-learning dictionary. First, images on a series of scales created from multiple input LR images are used as training images. Then, a learned overcomplete dictionary provides sufficient real image information that ensures the authenticity of the reconstructed HR image. In both simulation and real experiments, the proposed algorithm achieves better results than that achieved by other algorithms, recovering missing high-frequency information and edge details more effectively. Finally, several ideas towards higher computation speed are suggested.



---

*Source: 181864-2015-08-27.xml*