# Single Image Super-Resolution viaL

Authors: Zhang Liu; Qi Huang; Jian Li; Qi Wang
Journal: Mathematical Problems in Engineering
 (2014)

---

## Abstract

We propose a single image super-resolution method based on aL
0 smoothing approach. We consider a low-resolution image as two parts: one
is the smooth image generated by the L
0 smoothing method and the other is
the error image between the low-resolution image and the smoothing image. We
get an intermediate high-resolution image via a classical interpolation and then
generate a high-resolution smoothing image with sharp edges by the L
0 smoothing
method. For the error image, a learning-based super-resolution approach, keeping
image details well, is employed to obtain a high-resolution error image. The
resulting high-resolution image is the sum of the high-resolution smoothing image
and the high-resolution error image. Experimental results show the effectiveness
of the proposed method.


---

## Body

## 1. Introduction

Image super-resolution is an active field recently in image processing. It tends to obtain the high-resolution image via one or multiple low-resolution images. If we get the high-resolution image only by one low-resolution input, we call itsingle image super-resolution. The other case is calledclassical image super-resolution. Obviously, single image super-resolution is more useful and challenging for the practical applications, since only one low-resolution image is available sometimes due to the limitation of hardware or other reasons. In particular, we address single image super-resolution in the paper. Image super-resolution has many applications, for example, medical imaging magnetic resonance imaging (MRI), synthetic aperture radar (SAR), and high definition television. How to develop an effective super-resolution model and algorithm is very important.There are many image super-resolution methods up to now. They can be classified into three categories: interpolation-based methods, reconstruction-based methods, and learning-based methods. Interpolation-based methods tend to interpolate the unknown points via the known neighbor points. The two most famous interpolation methods are nearest-neighbor interpolation and bicubic interpolation. In addition, more interpolation methods can be found in [1–3]. Interpolation-based methods are very popular and fast, but they have to face some drawbacks, for example, jaggy effect or blur effect on image edges. Many reconstruction methods [4–10] have been proposed. They obtain the resulting super-resolution images via some reconstruction ideas. In [9], Shan et al. propose a fast image/video super-resolution approach by minimizing an energy function. This method can use fast Fourier transformation (FFT) algorithm to accelerate the speed. It upsamples a low-resolution image and then applies a fast reconstruction method to the upsampled image to get the resulting high-resolution output. In [10], Chambolle and Pock propose a first-order primal-dual algorithm for convex models and apply this algorithm to single image super-resolution. In addition, authors also give the convergence analysis of corresponding algorithm. Learning-based methods are a very popular tool to address image super-resolution problems [11–24]. Excellent results can be obtained via learning-based methods. However, before implementing the method, we have to generate two training data sets. One data set is formed by a mass of low-resolution patches; the other is formed by corresponding high-resolution patches. After getting the relation between the two training data sets, we apply it to a given low-resolution image to get high-resolution image. Obviously, this approach depends on the selection of training data sets and leads to expensive computation. In [19, 20], Yang et al. propose a sparse signal representation method for single image super-resolution. They search a sparse representation for each patches of the low-resolution image and compute the coefficients and then apply the computed coefficients to generate the high-resolution image. Based on the work of [19, 20], Zeyde et al. [22] simplify the overall process and reduce the computation complexity of algorithm. In [23], Timofte et al. propose a fast image super-resolution method which makes no compromise on image quality. It proposes an anchored neighborhood regression and utilizes global collaborative coding for speedup. In [24], Dong et al. learn a mapping between low-resolution and high-resolution images and then consider the mapping as a deep convolutional neural network (CNN). The CNN takes the low-resolution image as input and outputs the resulting high-resolution image.In addition, some different methods also have been proposed, for example, frequency technique [25], pixel classification method [26], and others [10, 27–29].In this paper, we divide a low-resolution image into two parts: one part is smoothing image with sharp edges which is generated byL
0 smoothing method; the other part is error image which is the difference between the low-resolution image and the smoothing image. We get the high-resolution smoothing image by applying L
0 smoothing method to an intermediate high-resolution image. The high-resolution smoothing image contains very important features of super-resolution images: sharp edges. For the error image, we upsample it using an approach of keeping image details to get high-resolution error image. The final resulting super-resolution image is the sum of high-resolution smoothing image and high-resolution error image.The organization of this paper is as follows. In Section2, we introduce an important image smoothing method which is utilized in our work. In addition, we give the proposed method which uses the smoothing method and one learning-based method. The experimental results are shown in Section 3. Finally, we give the conclusions in Section 4.


## 2. The Proposed Method and Related Work

### 2.1.L
0 Image Smoothing

In [30], Xu et al. proposed a novel image smoothing method via L
0 gradient minimization which is used to control the number of nonzero gradients. By controlling the number of nonzero gradients, they establish a L
0 model to smooth the high-frequency details of image, preserving prominent image structures.1D Signal Smoothing. For 1D case, C
(
f
) counts the number of neighbor pixels f
p and f
p
+
1,
(1)
C
(
f
)
=
#
{
p
∣
f
p
-
f
p
+
1
≠
0
}
,
where f
p is the smoothing result and # represents the counting operator and outputs the number of p satisfying |
f
p
-
f
p
+
1
|
≠
0. Note that it can give one value to C
(
f
) to control the smoothness of image. Although C
(
f
) is not functional, the specific objective function can be given as follows:
(2)
min
⁡
f
∑
p
f
p
-
g
p
2
s
.
t
.
,
C
(
f
)
=
k
,
where k
∈
N is the given value and g
p is the known discrete signal at location p. This optimization model is very powerful to abstract the image structure. It flattens image details effectively and preserves sharp image edges obviously. The property of preserving sharp edges is the most important motivation of this paper to deal with image super-resolution.Actually,k can be set from zero to thousands according to practical case. A general regularity form is employed to balance the structure flattening term and the fidelity term; it is shown as follows:
(3)
min
⁡
f
∑
p
f
p
-
g
p
2
+
λ
·
C
(
f
)
,
where λ is the regularity parameter which can control the significance of C
(
f
). The parameter also can be viewed as a smoothing factor: a larger λ will get fewer image edges. In particular, the number of nonzero gradients k is monotone with respect to 1
/
λ.2D Image Smoothing. For 2D image case, I is denoted as the input image and S is denoted as the resulting smoothing image. The gradient of image at pixel p is represented by ∇
S
p
=
(
∂
x
S
p
,
∂
y
S
p
)
T. Similar to (1), it can get the number of nonzero gradients in the following formula:
(4)
C
(
S
)
=
#
{
p
∣
∂
x
S
p
+
∂
y
S
p
≠
0
}
.Similar to  (3), we get the image smoothing model according to (4),
(5)
min
⁡
S
∑
p
S
p
-
I
p
2
+
λ
·
C
(
S
)
,
where λ is the balance parameter similar to (3). Actually, the term C
(
S
) smoothes the image details and (
S
p
-
I
p
)
2 keeps the main image structure similarly. The corresponding solver of (5) will be given in the following.Solving. For the optimization problem (5), it is a nondeterministic polynomial-time (NP) hard problem to find the global optimal solution. Motivated by Wang et al. [32], a splitting scheme is proposed to solve the optimization problem. Two auxiliary variables h
p and v
p are introduced to substitute ∂
x
S
p and ∂
y
S
p. Equation (5) can be rewritten as
(6)
min
⁡
S
,
h
,
v
∑
p
S
p
-
I
p
2
+
λ
·
C
(
h
,
v
)
+
β
∂
x
S
p
-
h
2
+
∂
y
S
p
-
v
2
,
where β is an adaptive parameter to control the distance between (
h
,
v
) and their gradients (
∂
x
S
p
,
∂
y
S
p
). For solving one variable, it should fix the other variable obtained from the previous iteration. This process can be depicted as two subproblems.S-Subproblem. We get S by solving the following minimization problem:
(7)
min
⁡
S
∑
p
S
p
-
I
p
2
+
β
∂
x
S
p
-
h
2
+
∂
y
S
p
-
v
2
.The problem has global minimum solution due to quadratic function. It may be speed up by following fast Fourier transform (FFT) (see details in [32]). Consider
(8)
S
=
F
-
1
×
I
+
β
F
∂
x
*
F
h
+
F
∂
y
*
F
v
F
1
+
β
F
∂
x
*
F
∂
x
+
F
∂
y
*
F
∂
y
,
where F is FFT operator, F
(
1
) is the Fourier transform of delta function, and F
* is the complex conjugate. Note that S is computed very fast due to the FFT operator which has complexity O
(
N
log
⁡
(
N
)
).(
h
,
v
)
-Subproblem. For (
h
,
v
)-subproblem, the corresponding minimization problem is as follows:
(9)
min
⁡
h
,
v
∑
p
β
∂
x
S
p
-
h
2
+
∂
y
S
p
-
v
2
+
λ
·
C
(
h
,
v
)
,
where C
(
h
,
v
)
=
∑
p
‍
H
(
|
h
p
|
+
|
v
p
|
), H
(
|
h
p
|
+
|
v
p
|
)
=
1 if |
h
p
|
+
|
v
p
|
≠
0, and H
(
|
h
p
|
+
|
v
p
|
)
=
0 otherwise. Thus (9) can be rewritten as
(10)
∑
p
min
⁡
h
p
,
v
p
β
∂
x
S
p
-
h
2
+
∂
y
S
p
-
v
2
+
λ
·
H
h
p
+
v
p
.In [30], it has been proved that, for each pixel p, E
p
=
β
(
(
∂
x
S
p
-
h
)
2
+
(
∂
y
S
p
-
v
)
2
)
+
λ
·
H
(
|
h
p
|
+
|
v
p
|
) reaches its minimum under the following condition:
(11)
(
h
p
,
v
p
)
=
(
0,0
)
,
∂
x
S
p
2
+
∂
y
S
p
2
≤
λ
β
,
(
∂
x
S
p
,
∂
y
S
p
)
,
otherwise
.The resulting algorithm for image smoothing is as shown in Algorithm1.Algorithm 1:Image smoothing via L
0 gradient minimization.
Input: Image I, smoothing weight λ, parameter β
0
,
β
max
⁡, rate κ
(1)  Initialization:S
←
I
,
β
←
β
0
,
i
←
0.
(2)While  β
<
β
max
⁡
(3)  Solvingh
p
(
i
)
,
v
p
(
i
) via (10) when fixing S
(
i
)
(4)  SolvingS
(
i
+
1
) via (8) when fixing h
p
(
i
)
,
v
p
(
i
)
(5)  Update:β
←
κ
β
(6)i
←
i
+
1
(7)end
Output: Smoothing image SStep (5) is to updateβ adaptively via a parameter κ for each iteration. The resulting S is the smoothing image which flattens image details but preserves sharp edges.


### 2.2. The Combined Method for Image Super-Resolution

Sharp edges are very important features to super-resolution images. In our work, we utilize the property of preserving sharp edges ofL
0 image smoothing to enhance the quality of super-resolution image. We divide one low-resolution image into two parts: (1) smoothing image obtained by L
0 smoothing method and (2) error image obtained by the difference of the low-resolution image and the smoothing image. After this process, we take two steps to realize the proposed method. First, we upsample the low-resolution image to get intermediate high-resolution image by a fast and classical interpolation method and then apply L
0 smoothing method to it to get high-resolution smoothing image. Note that the high-resolution smoothing image has sharp edges. Second, the error image will be upsampled to high-resolution error image via one learning-based super-resolution method of keeping image details well.In Figure1, we show the proposed strategy for image super-resolution. This strategy combines two methods of different image applications for image super-resolution. In particular, “LR” represents the low-resolution image. “L
0” is the low-resolution smoothing image via L
0 smoothing method. “E” is the error image between “LR” and “L
0.” Actually, “L
0” is just to get the low-resolution error image E. “H
0” is the upsampled image by one classical interpolation method (here we use “bicubic” interpolation method as “Method 1”). “H” is the high-resolution smoothing image by the same L
0 smoothing method. “S” is the high-resolution error image by one learning-based method which can preserve image details well (in our work, we use one learning-based method [31] as “Method 2”).Figure 1
Illustration of the proposed method.We present Figure2 to specify the rationality of the proposed work. From Figure 2, the resulting high-resolution image (Figure 2(a)) is the sum of high-resolution smoothing image (Figure 2(b)) and high-resolution error image (Figure 2(c)). Note that the high-resolution smoothing image has sharp image edges and the high-resolution error image contains obvious image details. It demonstrates that our proposed work can combine the advantages of L
0 image smoothing (preserving sharp edges) and the detail-preserved method [31] (keeping obvious image details).(a) High-resolution image; (b) high-resolution smoothing image byL
0 smoothing method; (c) high-resolution error image by the learning-based method [31] (for better vision, adding 0.3 to the error image).
(a)
(b)
(c)


## 2.1.L
0 Image Smoothing

In [30], Xu et al. proposed a novel image smoothing method via L
0 gradient minimization which is used to control the number of nonzero gradients. By controlling the number of nonzero gradients, they establish a L
0 model to smooth the high-frequency details of image, preserving prominent image structures.1D Signal Smoothing. For 1D case, C
(
f
) counts the number of neighbor pixels f
p and f
p
+
1,
(1)
C
(
f
)
=
#
{
p
∣
f
p
-
f
p
+
1
≠
0
}
,
where f
p is the smoothing result and # represents the counting operator and outputs the number of p satisfying |
f
p
-
f
p
+
1
|
≠
0. Note that it can give one value to C
(
f
) to control the smoothness of image. Although C
(
f
) is not functional, the specific objective function can be given as follows:
(2)
min
⁡
f
∑
p
f
p
-
g
p
2
s
.
t
.
,
C
(
f
)
=
k
,
where k
∈
N is the given value and g
p is the known discrete signal at location p. This optimization model is very powerful to abstract the image structure. It flattens image details effectively and preserves sharp image edges obviously. The property of preserving sharp edges is the most important motivation of this paper to deal with image super-resolution.Actually,k can be set from zero to thousands according to practical case. A general regularity form is employed to balance the structure flattening term and the fidelity term; it is shown as follows:
(3)
min
⁡
f
∑
p
f
p
-
g
p
2
+
λ
·
C
(
f
)
,
where λ is the regularity parameter which can control the significance of C
(
f
). The parameter also can be viewed as a smoothing factor: a larger λ will get fewer image edges. In particular, the number of nonzero gradients k is monotone with respect to 1
/
λ.2D Image Smoothing. For 2D image case, I is denoted as the input image and S is denoted as the resulting smoothing image. The gradient of image at pixel p is represented by ∇
S
p
=
(
∂
x
S
p
,
∂
y
S
p
)
T. Similar to (1), it can get the number of nonzero gradients in the following formula:
(4)
C
(
S
)
=
#
{
p
∣
∂
x
S
p
+
∂
y
S
p
≠
0
}
.Similar to  (3), we get the image smoothing model according to (4),
(5)
min
⁡
S
∑
p
S
p
-
I
p
2
+
λ
·
C
(
S
)
,
where λ is the balance parameter similar to (3). Actually, the term C
(
S
) smoothes the image details and (
S
p
-
I
p
)
2 keeps the main image structure similarly. The corresponding solver of (5) will be given in the following.Solving. For the optimization problem (5), it is a nondeterministic polynomial-time (NP) hard problem to find the global optimal solution. Motivated by Wang et al. [32], a splitting scheme is proposed to solve the optimization problem. Two auxiliary variables h
p and v
p are introduced to substitute ∂
x
S
p and ∂
y
S
p. Equation (5) can be rewritten as
(6)
min
⁡
S
,
h
,
v
∑
p
S
p
-
I
p
2
+
λ
·
C
(
h
,
v
)
+
β
∂
x
S
p
-
h
2
+
∂
y
S
p
-
v
2
,
where β is an adaptive parameter to control the distance between (
h
,
v
) and their gradients (
∂
x
S
p
,
∂
y
S
p
). For solving one variable, it should fix the other variable obtained from the previous iteration. This process can be depicted as two subproblems.S-Subproblem. We get S by solving the following minimization problem:
(7)
min
⁡
S
∑
p
S
p
-
I
p
2
+
β
∂
x
S
p
-
h
2
+
∂
y
S
p
-
v
2
.The problem has global minimum solution due to quadratic function. It may be speed up by following fast Fourier transform (FFT) (see details in [32]). Consider
(8)
S
=
F
-
1
×
I
+
β
F
∂
x
*
F
h
+
F
∂
y
*
F
v
F
1
+
β
F
∂
x
*
F
∂
x
+
F
∂
y
*
F
∂
y
,
where F is FFT operator, F
(
1
) is the Fourier transform of delta function, and F
* is the complex conjugate. Note that S is computed very fast due to the FFT operator which has complexity O
(
N
log
⁡
(
N
)
).(
h
,
v
)
-Subproblem. For (
h
,
v
)-subproblem, the corresponding minimization problem is as follows:
(9)
min
⁡
h
,
v
∑
p
β
∂
x
S
p
-
h
2
+
∂
y
S
p
-
v
2
+
λ
·
C
(
h
,
v
)
,
where C
(
h
,
v
)
=
∑
p
‍
H
(
|
h
p
|
+
|
v
p
|
), H
(
|
h
p
|
+
|
v
p
|
)
=
1 if |
h
p
|
+
|
v
p
|
≠
0, and H
(
|
h
p
|
+
|
v
p
|
)
=
0 otherwise. Thus (9) can be rewritten as
(10)
∑
p
min
⁡
h
p
,
v
p
β
∂
x
S
p
-
h
2
+
∂
y
S
p
-
v
2
+
λ
·
H
h
p
+
v
p
.In [30], it has been proved that, for each pixel p, E
p
=
β
(
(
∂
x
S
p
-
h
)
2
+
(
∂
y
S
p
-
v
)
2
)
+
λ
·
H
(
|
h
p
|
+
|
v
p
|
) reaches its minimum under the following condition:
(11)
(
h
p
,
v
p
)
=
(
0,0
)
,
∂
x
S
p
2
+
∂
y
S
p
2
≤
λ
β
,
(
∂
x
S
p
,
∂
y
S
p
)
,
otherwise
.The resulting algorithm for image smoothing is as shown in Algorithm1.Algorithm 1:Image smoothing via L
0 gradient minimization.
Input: Image I, smoothing weight λ, parameter β
0
,
β
max
⁡, rate κ
(1)  Initialization:S
←
I
,
β
←
β
0
,
i
←
0.
(2)While  β
<
β
max
⁡
(3)  Solvingh
p
(
i
)
,
v
p
(
i
) via (10) when fixing S
(
i
)
(4)  SolvingS
(
i
+
1
) via (8) when fixing h
p
(
i
)
,
v
p
(
i
)
(5)  Update:β
←
κ
β
(6)i
←
i
+
1
(7)end
Output: Smoothing image SStep (5) is to updateβ adaptively via a parameter κ for each iteration. The resulting S is the smoothing image which flattens image details but preserves sharp edges.


## 2.2. The Combined Method for Image Super-Resolution

Sharp edges are very important features to super-resolution images. In our work, we utilize the property of preserving sharp edges ofL
0 image smoothing to enhance the quality of super-resolution image. We divide one low-resolution image into two parts: (1) smoothing image obtained by L
0 smoothing method and (2) error image obtained by the difference of the low-resolution image and the smoothing image. After this process, we take two steps to realize the proposed method. First, we upsample the low-resolution image to get intermediate high-resolution image by a fast and classical interpolation method and then apply L
0 smoothing method to it to get high-resolution smoothing image. Note that the high-resolution smoothing image has sharp edges. Second, the error image will be upsampled to high-resolution error image via one learning-based super-resolution method of keeping image details well.In Figure1, we show the proposed strategy for image super-resolution. This strategy combines two methods of different image applications for image super-resolution. In particular, “LR” represents the low-resolution image. “L
0” is the low-resolution smoothing image via L
0 smoothing method. “E” is the error image between “LR” and “L
0.” Actually, “L
0” is just to get the low-resolution error image E. “H
0” is the upsampled image by one classical interpolation method (here we use “bicubic” interpolation method as “Method 1”). “H” is the high-resolution smoothing image by the same L
0 smoothing method. “S” is the high-resolution error image by one learning-based method which can preserve image details well (in our work, we use one learning-based method [31] as “Method 2”).Figure 1
Illustration of the proposed method.We present Figure2 to specify the rationality of the proposed work. From Figure 2, the resulting high-resolution image (Figure 2(a)) is the sum of high-resolution smoothing image (Figure 2(b)) and high-resolution error image (Figure 2(c)). Note that the high-resolution smoothing image has sharp image edges and the high-resolution error image contains obvious image details. It demonstrates that our proposed work can combine the advantages of L
0 image smoothing (preserving sharp edges) and the detail-preserved method [31] (keeping obvious image details).(a) High-resolution image; (b) high-resolution smoothing image byL
0 smoothing method; (c) high-resolution error image by the learning-based method [31] (for better vision, adding 0.3 to the error image).
(a)
(b)
(c)


## 3. Results

In this section, we compare the proposed method with some competitive image super-resolution methods, for example, 01’TIP [1], 06’TIP [3], and 07’TIP [8]. The experimental computer is a laptop with 3.25 Gb RAM and Intel Core i3-2370M CPU: @2.40 GHz. All experiments are implemented on MATLAB (R2010a). For the corresponding parameters, we set κ
=
2, λ
=
0.005, β
max
⁡
=
1
0
5, and β
0
=
2
λ. In addition, the images with different types and upscaling factors are employed to test the effectiveness of the compared methods.In Figure3, we compare our method with two classical interpolation methods (nearest-neighbor interpolation and bicubic interpolation). Different images, for example, butterfly, and flower, are employed for experiments. From the figure, it is easy to know that the resulting super-resolution images by the proposed method keep sharper image edges than nearest-neighbor and bicubic interpolations. In particular, the resulting super-resolution images by nearest-neighbor interpolation show jaggy effect, and the super-resolution images by bicubic interpolation show blur effect.Figure 3
(a) Low-resolution images; (b) high-resolution images by nearest-neighbor interpolation; (c) high-resolution images by bicubic interpolation; (d) high-resolution images by the proposed method. The upscaling factors of first to third row are 3; the last row is 2.In Figure4, a competitive kernel regression method [8] is employed to compare with the proposed method. We know that the resulting images by our method preserve sharp edges better than other methods. The kernel regression method [8] and bicubic interpolation all show blur effect on the edges.Figure 4
(a) Low-resolution images; (b) high-resolution images by bicubic interpolation; (c) high-resolution error image by the learning-based method [31]; (d) high-resolution images by the proposed method. The upscaling factors are 4.From Figure5, more competitive super-resolution methods are compared with our method. We employ two images with different types in the experiments: one type is natural image and the other type is comic image. From the figure, bicubic interpolation, 01’TIP [1], 06’TIP [3], and 07’TIP [8] all show blur effect, especially on image edges. The proposed method performs best, not only on image edges but also on nonedge regions.Figure 5
(a) Low-resolution images; (b) high-resolution images by bicubic interpolation; (c) high-resolution images by 01’TIP [1]; (d) high-resolution images by 06’TIP [3]; (e) high-resolution images by [8]; (f) high-resolution images by the proposed method. The upscaling factors are 2.


## 4. Conclusions

In this paper, a novel image super-resolution method was proposed based on an image smoothing strategy. We appliedL
0 image smoothing method to a given low-resolution image to get its low-resolution smoothing image which preserved sharp edges. Besides, a low-resolution error image was obtained by the difference between the low-resolution image and the low-resolution smoothing image. For the low-resolution error image and the given low-resolution image, we took two steps to generate the resulting super-resolution image. First, we upsampled the low-resolution image by bicubic interpolation to get intermediate high-resolution image and then applied the L
0 smoothing method to the intermediate high-resolution image to generate the high-resolution smoothing image. Second, a learning-based super-resolution method was utilized to upsample the low-resolution error image to get high-resolution error image. The learning-based method could preserve image details well. The final super-resolution image was the sum of the high-resolution smoothing image and the high-resolution error image. In experimental section, we employed low-resolution images of different types to test our method. Results demonstrated that the proposed method performed better than some competitive image super-resolution methods, especially preserving sharp image edges.



---

*Source: 974509-2014-12-29.xml*