# Multiframe Superresolution of Vehicle License Plates Based on Distribution Estimation Approach

Authors: Renchao Jin; Shengrong Zhao; Xiangyang Xu; Enmin Song
Journal: Journal of Control Science and Engineering
 (2016)

---

## Abstract

Low-resolution (LR) license plate images or videos are often captured in the practical applications. In this paper, a distribution estimation based superresolution (SR) algorithm is proposed to reconstruct the license plate image. Different from the previous work, here, the high-resolution (HR) image is estimated via the obtained posterior probability distribution by using the variational Bayesian framework. To regularize the estimated HR image, a feature-specific prior model is proposed by considering the most significant characteristic of license plate images; that is, the target has high contrast with the background. In order to assure the success of the SR reconstruction, the models representing smoothness constraints on images are also used to regularize the estimated HR image with the proposed feature-specific prior model. We show by way of experiments, under challenging blur with size 7 × 7 and zero-mean Gaussian white noise with variances 0.2 and 0.5, respectively, that the proposed method could achieve the peak signal-to-noise ratio (PSNR) of 22.69 dB and the structural similarity (SSIM) of 0.9022 under the noise with variance 0.2 and the PSNR of 19.89 dB and the SSIM of 0.8582 even under the noise with variance 0.5, which are 1.84 dB and 0.04 improvements in comparison with other methods.


---

## Body

## 1. Introduction

Nowadays, the intelligent transport system (ITS) is increasingly used to address traffic problems. The ITS can apply the advanced information technology, data communication transmission technology, electronic sensor technology, control technology, and computer technology to the whole transportation management system effectively and efficiently. The vehicle license plate character recognition (VLPCR) system is one of the most important parts of the ITS and is widely used in traffic monitor and control. However, if the license plate image is captured at low resolution, the license plate cannot be readable; hence, the ITS could not work well. There are many reasons leading to the degradation of required license plate images, such as downsampling, blurring, warping, and noising. Thus, the problem addressed in this paper is using multiframe superresolution (SR) technique [1–3] to reconstruct license plate images with better quality.The objective of multiframe SR is to fuse a sequence of low-resolution (LR) images representing the same scene in a single high-resolution (HR) image. Such kind of SR technique can be classified into three classes: (i) frequency domain approaches [4, 5], (ii) interpolation approaches [6, 7], and (iii) regularization approaches [8–10]. Among these approaches, the regularization approach is studied widely. Due to the ill-posedness of SR reconstruction problems, the basic idea of regularization approach is to incorporate the prior knowledge of the unknown HR image into the reconstruction process. The regularization approach includes deterministic and stochastic regularization approaches. The former uses prior models as regularization terms, and the latter uses prior models to establish prior probability distributions.The popularly used prior models are the Tikhonov model [11], the total variation (TV) type model [12], and the Markov random field (MRF) model [13]. The Tikhonov model is based on the L2 norm. It might increase the punishment for the noise; however, it may blur the edges. The famous TV type model penalizes the total amount of changes in the image by using the L1 norm to measure the gradient. However, the TV type model could not remove the heavy noise completely, which may lead artifacts to be produced in the estimated HR images. In order to make use of advantages of the L1 and L2 norms, Suresh et al. proposed a discontinuity-adaptive Markov random field (DAMRF) prior model to reconstruct license plate images [14]. In [15], a generalized DAMRF prior model has been used to make the license plate more legible. In [16], the authors proposed a bimodal prior model for the text image and combined it with the Huber prior model. In these methods, the estimation of motion parameters and the reconstruction of HR image are separated and conducted independently, which is a well-known suboptimal solution. And only the translation motion was considered in [14, 15], which is not suitable to many realities of situation. In [15], the authors proposed a method to estimate the regularization parameter; however, they did not consider estimating the noise variance, which is also important for reconstructing the HR image.In this paper, a new method based on the variational Bayesian inference (VBI) estimator is proposed to perform the SR of vehicle license plate, in which the HR image, the motion parameter, and hyperparameters are estimated jointly. The VBI estimator is a distribution estimation algorithm, which could solve the nonlinear, high-dimensional problem effectively [17].The VBI framework used in this paper is similar to the ones used in [12, 18]. The main difference between the proposed method and these similar methods is the image modeling. An accurate and comprehensive image model is very useful to improve the quality of the reconstructed images. The image models proposed in [12, 18] have been proved to be efficient; however, they did not consider a significant feature of the license plate image; that is, its gray-level distribution satisfied the bimodal distribution.The most significant characteristic of license plate is that the target has high contrast with the background to make it readable. Thus, for the gray image of license plate, the target pixels tend to cluster around one center, while the background pixels tend to cluster around another one. From this point of view, we propose a feature-specific model for the license plate image by considering that its gray-level distribution has two peaks. In this paper, this feature-specific prior information is introduced explicitly as constraints into the SR reconstruction of the vehicle license plate. Moreover, in order to assure the reconstruction success, a smoothing prior is combined with the feature-specific prior model to regularize the estimated license plate image. During the reconstruction process, the target pixels and the background pixels tend to cluster to different centers. Thus, the gradient information will be estimated as exactly as possible. Certainly, the smoothing prior is useful to divide the target and background. Experimental results demonstrate that combining the feature-specific and smoothing prior models could reduce artifacts effectively.The paper is organized as follows. The mathematical model for image degradation and VBI estimator are provided in Section2. In Section 3, the prior probability distributions for the HR image, motion parameters, and hyperparameters are presented, and the corresponding optimization procedure is described in Section 4 in detail. In Section 5, experimental results are illustrated. The conclusion then follows.


## 2. Bayesian Framework

### 2.1. Degradation Model

Before attempting to solve the SR problem, it is necessary to know the process of generating the LR images. We assume that a set of LR observations are obtained from their corresponding single HR image. The size of arbitrary LR image and the HR image isN
1
×
N
2 and r
N
1
×
r
N
2, respectively, r is a downsampling factor in the horizontal and vertical direction, and let N
=
N
1
×
N
2. Usually, it assumes that the LR observations are generated from the HR image through a sequence of operations that includes (i) geometrical warps, (ii) blur, (iii) downsampling, and (iv) an additive zero-mean white Gaussian noise. Such SR degradation model for the LR image y
k derived from HR image X is given by(1)
y
k
=
D
k
H
k
M
s
k
X
+
n
k
=
B
s
k
X
+
n
k
,
1
≤
k
≤
L
,where L
>
1 is the total number of the LR observations, X
∈
R
r
2
N
×
1 and y
k
∈
R
N
×
1 are the vectorized version of the HR and LR images, respectively, D
k
∈
R
N
×
r
2
N is a downsampling operator, H
k
∈
R
r
2
N
×
r
2
N is a blurring operator, M
(
s
k
)
∈
R
r
2
N
×
r
2
N is a warp operator that represents subpixel shift between the LR image y
k and the reference frame, and n
k
∈
R
N
×
1 is a noise vector.In this paper, we assume that the downsampling matrixD
k and blurring matrix H
k remain the same between the LR images and are known. The warp matrix M
k
(
s
k
) represents the motion that occurs during the image acquisition. It is considered that, in this paper, the motion parameters are due only to global motion, and the motion model contains global translation and rotation; that is, s
k
=
θ
k
,
d
x
k
,
d
y
k, where θ
k is the rotation angle and d
x
k and d
y
k are the horizontal and vertical translations of the kth HR image with respect to the reference frame. The noise model is assumed as the white Gaussian noise during acquiring LR observations.Supposing that the noisen
k (k
=
1,2
,
…
,
L) is white Gaussian noise and n
k
~
N
(
0
,
β
k
-
1
), we can get(2)
p
n
k
∝
β
k
N
/
2
exp
⁡
-
β
k
2
n
k
2
2
.Then, the following equation is obtained by using (1):(3)
p
y
k
∣
X
,
s
k
,
β
k
∝
β
k
N
/
2
exp
⁡
-
β
k
2
y
k
-
B
s
k
X
2
2
.Since the noise among the LR images is mutually independent, we can obtain(4)
p
y
∣
X
,
s
k
,
β
k
=
∏
k
=
1
L
p
y
k
∣
X
,
s
k
,
β
k
,where y
=
{
y
k
∣
k
=
1,2
,
…
,
L
}.


### 2.2. The VBI Estimator

By using the VBI estimator, the variables including the HR image, motion parameters, and hyperparameters are estimated through their corresponding posterior probability distributions. Usually, the mean value of the obtained posterior probability distribution is used as the estimation of the corresponding variable.The following conditional distribution is obtained by using the Bayes rule:(5)
p
X
,
s
k
,
β
k
,
α
∣
y
=
p
X
∣
α
p
α
∏
k
=
1
L
p
y
k
∣
X
,
β
k
,
s
k
p
s
k
p
β
k
p
y
,where p
(
y
k
∣
X
,
s
k
,
β
k
) represents the conditional distribution of the LR image y
k, p
(
s
k
) and p
(
β
k
) are prior distributions of s
k and β
k, respectively, p
(
X
∣
α
) is the prior distribution for the unknown HR image X, and α is the hyperparameter of the prior distribution p
(
X
∣
α
). For convenience, we denote Θ
=
{
X
,
{
s
k
}
,
{
β
k
}
,
α
}.By using the VBI estimator,p
(
Θ
∣
y
) is approximated by a tractable distribution q
(
Θ
). This approximating distribution is found by minimizing the Kullback-Leibler (KL) divergence, which can measure the difference between the two distributions p
(
Θ
∣
y
) and q
(
Θ
). The KL divergence is defined as follows:(6)
C
K
L
q
Θ
∥
p
Θ
∣
y
=
∫
q
Θ
log
⁡
q
Θ
p
Θ
∣
y
d
Θ
.SinceX, s
k, β
k, and α are mutually independent, q
(
Θ
)
=
q
(
X
)
q
(
α
)
∏
k
=
1
L
q
(
s
k
)
q
(
β
k
). q
(
X
) and q
(
s
k
) are posterior probability distributions of X and s
k, respectively. q
(
β
k
) and q
(
α
) are posterior probability distributions of the hyperparameters β
k and α, respectively.


## 2.1. Degradation Model

Before attempting to solve the SR problem, it is necessary to know the process of generating the LR images. We assume that a set of LR observations are obtained from their corresponding single HR image. The size of arbitrary LR image and the HR image isN
1
×
N
2 and r
N
1
×
r
N
2, respectively, r is a downsampling factor in the horizontal and vertical direction, and let N
=
N
1
×
N
2. Usually, it assumes that the LR observations are generated from the HR image through a sequence of operations that includes (i) geometrical warps, (ii) blur, (iii) downsampling, and (iv) an additive zero-mean white Gaussian noise. Such SR degradation model for the LR image y
k derived from HR image X is given by(1)
y
k
=
D
k
H
k
M
s
k
X
+
n
k
=
B
s
k
X
+
n
k
,
1
≤
k
≤
L
,where L
>
1 is the total number of the LR observations, X
∈
R
r
2
N
×
1 and y
k
∈
R
N
×
1 are the vectorized version of the HR and LR images, respectively, D
k
∈
R
N
×
r
2
N is a downsampling operator, H
k
∈
R
r
2
N
×
r
2
N is a blurring operator, M
(
s
k
)
∈
R
r
2
N
×
r
2
N is a warp operator that represents subpixel shift between the LR image y
k and the reference frame, and n
k
∈
R
N
×
1 is a noise vector.In this paper, we assume that the downsampling matrixD
k and blurring matrix H
k remain the same between the LR images and are known. The warp matrix M
k
(
s
k
) represents the motion that occurs during the image acquisition. It is considered that, in this paper, the motion parameters are due only to global motion, and the motion model contains global translation and rotation; that is, s
k
=
θ
k
,
d
x
k
,
d
y
k, where θ
k is the rotation angle and d
x
k and d
y
k are the horizontal and vertical translations of the kth HR image with respect to the reference frame. The noise model is assumed as the white Gaussian noise during acquiring LR observations.Supposing that the noisen
k (k
=
1,2
,
…
,
L) is white Gaussian noise and n
k
~
N
(
0
,
β
k
-
1
), we can get(2)
p
n
k
∝
β
k
N
/
2
exp
⁡
-
β
k
2
n
k
2
2
.Then, the following equation is obtained by using (1):(3)
p
y
k
∣
X
,
s
k
,
β
k
∝
β
k
N
/
2
exp
⁡
-
β
k
2
y
k
-
B
s
k
X
2
2
.Since the noise among the LR images is mutually independent, we can obtain(4)
p
y
∣
X
,
s
k
,
β
k
=
∏
k
=
1
L
p
y
k
∣
X
,
s
k
,
β
k
,where y
=
{
y
k
∣
k
=
1,2
,
…
,
L
}.


## 2.2. The VBI Estimator

By using the VBI estimator, the variables including the HR image, motion parameters, and hyperparameters are estimated through their corresponding posterior probability distributions. Usually, the mean value of the obtained posterior probability distribution is used as the estimation of the corresponding variable.The following conditional distribution is obtained by using the Bayes rule:(5)
p
X
,
s
k
,
β
k
,
α
∣
y
=
p
X
∣
α
p
α
∏
k
=
1
L
p
y
k
∣
X
,
β
k
,
s
k
p
s
k
p
β
k
p
y
,where p
(
y
k
∣
X
,
s
k
,
β
k
) represents the conditional distribution of the LR image y
k, p
(
s
k
) and p
(
β
k
) are prior distributions of s
k and β
k, respectively, p
(
X
∣
α
) is the prior distribution for the unknown HR image X, and α is the hyperparameter of the prior distribution p
(
X
∣
α
). For convenience, we denote Θ
=
{
X
,
{
s
k
}
,
{
β
k
}
,
α
}.By using the VBI estimator,p
(
Θ
∣
y
) is approximated by a tractable distribution q
(
Θ
). This approximating distribution is found by minimizing the Kullback-Leibler (KL) divergence, which can measure the difference between the two distributions p
(
Θ
∣
y
) and q
(
Θ
). The KL divergence is defined as follows:(6)
C
K
L
q
Θ
∥
p
Θ
∣
y
=
∫
q
Θ
log
⁡
q
Θ
p
Θ
∣
y
d
Θ
.SinceX, s
k, β
k, and α are mutually independent, q
(
Θ
)
=
q
(
X
)
q
(
α
)
∏
k
=
1
L
q
(
s
k
)
q
(
β
k
). q
(
X
) and q
(
s
k
) are posterior probability distributions of X and s
k, respectively. q
(
β
k
) and q
(
α
) are posterior probability distributions of the hyperparameters β
k and α, respectively.


## 3. The Prior Probability Distributions

### 3.1. Image Prior Probability Distribution

In the Bayesian method, the prior information of the original HR image represented by the prior model plays an important role. A significant property of license plate is that the target of license plate has high contrast with the background (see Figure1). The pixels in the license plate images can be divided into two classes. Thus, the original HR license plate image can be represented as(7)
X
=
X
t
a
r
∪
X
b
a
c
,where X
t
a
r represents the set of pixels belonging to the target regions (i.e., the white regions in Figure 1) and X
b
a
c represents the set of pixels belonging to the background regions (i.e., the black regions in Figure 1).Figure 1
An example of the license plate image.Therefore, we assume that, for the license plate images, the target pixels tend to cluster around one center, while the background pixels tend to cluster around another center. In order to make use of this characteristic, we propose the following model to regularize the estimated HR license plate image:(8)
R
X
=
X
-
μ
2
2
,where μ is a vector with size r
2
N
×
1. In the following, we will demonstrate how to obtain the vector μ.In (8), the elements of the vector μ are the mean values of X
t
a
r and X
b
a
c, respectively. That is to say, if X
i
∈
X
t
a
r (or X
b
a
c), thus, μ
i
=
μ
t
a
r (or μ
b
a
c), where μ
t
a
r and μ
b
a
c are the mean values of X
t
a
r and X
b
a
c, respectively. Then, μ
t
a
r and μ
b
a
c can be obtained as follows:(9)
μ
t
a
r
=
∑
X
i
N
t
a
r
,
X
i
∈
X
t
a
r
,
μ
b
a
c
=
∑
X
j
N
b
a
c
,
X
j
∈
X
b
a
c
,where N
t
a
r and N
b
a
c represent the total number of pixels in the target regions and background regions, respectively.In order to obtainμ, first, the following expressions are defined:(10)
μ
1
=
W
μ
t
,
μ
2
=
e
-
W
μ
b
,where μ
t and μ
b are vectors with size r
2
N
×
1, μ
t
(
i
)
=
μ
t
a
r and μ
b
(
i
)
=
μ
b
a
c, for i
=
1,2
,
…
,
r
2
N. W is a diagonal matrix with size r
2
N
×
r
2
N, whose elements are 0s and 1s, and e is a matrix with size r
2
N
×
r
2
N, whose elements are 1s.Then, for obtaining the matrixW, we use OSTU to partition the estimated HR image into target regions and background regions. After the partition, the element values of W corresponding to the pixels in the target regions are set as 1s; the others are set as 0s. We can obtain(11)
W
i
,
i
=
1
,
i
f
X
i
∈
X
t
a
r
,
0
,
o
t
h
e
r
s
,
(12)
e
-
W
i
,
i
=
1
,
i
f
X
i
∈
X
b
a
c
,
0
,
o
t
h
e
r
s
.Thus, (13)
μ
1
i
=
W
μ
t
i
=
μ
t
a
r
,
i
f
X
i
∈
X
t
a
r
,
0
,
o
t
h
e
r
s
,
μ
2
i
=
e
-
W
μ
b
i
=
μ
b
a
c
,
i
f
X
i
∈
X
b
a
c
,
0
,
o
t
h
e
r
s
.Finally, the nonzero elements in μ
1 and μ
2 are extracted and positioned in μ at the corresponding location as desired.Based on the prior model (8), the following prior probability distribution for the estimated HR license plate image is proposed:(14)
p
X
∣
α
1
∝
α
1
r
2
N
/
2
exp
⁡
-
α
1
2
X
-
μ
2
2
,where α
1 is the hyperparameter of this prior distribution.In order to assure the success of the reconstruction, we have adopted the total variation (TV) prior model and the simultaneous autoregressive (SAR) prior model [18] to regularize the estimated HR image. The TV and the SAR prior models are defined as follows:(15)
R
T
V
X
=
∑
i
∇
i
h
X
2
+
∇
i
v
X
2
,where ∇
i
h
X and ∇
i
v
X represent the horizontal and vertical gradient components of the ith element of X, respectively. Consider(16)
R
S
A
R
X
=
c
X
2
2
,where c denotes the Laplacian operator.The TV model’s and the SAR model’s corresponding prior distributions are defined as(17)
p
X
∣
α
2
∝
α
2
r
2
N
/
2
exp
⁡
-
α
2
2
∑
i
∇
i
h
X
2
+
∇
i
v
X
2
,
(18)
p
X
∣
α
3
∝
α
3
r
2
N
/
2
exp
⁡
-
α
3
2
c
X
2
2
.


### 3.2. Motion Prior Probability Distribution

The motion parameters are modeled as stochastic variables following Gaussian distributions, similar to [12, 18]:(19)
p
s
k
=
∏
k
=
1
L
N
s
k
∣
μ
s
k
0
,
σ
s
k
0
,where μ
s
k
0 is the a priori mean vector and σ
s
k
0 is the a priori covariance matrix. These two parameters can incorporate prior knowledge about the motion parameters into the estimation process. Setting μ
s
k
0 and σ
s
k
0 equal to zero represents the fact that no such knowledge is available, which makes only the observed LR images responsible for the estimation process.In this work, the parametersμ
s
k
0 are obtained by using the Lucas-Kanade method [19], and the inverse covariance matrices (
σ
s
k
0
)
-
1 are set equal to zero matrices. And they will be used as the initial values in the following SR method.


### 3.3. Hyperparameter Prior Probability Distribution

The prior information about the hyperparameters is usually expressed using the conjugate prior distribution which is calculated conveniently. Moreover, the corresponding posterior distribution has the same functional form with the prior distribution and hence the analytic solution can be obtained. It is well known that the inverse Gamma distributions are the conjugate priors for the variance of the Gaussian distribution whose mean value is known. Thus, we assume that the hyperparameters obey Gamma distributions; that is,(20)
p
t
=
G
a
m
m
a
t
∣
a
t
0
,
b
t
0
=
b
t
0
a
t
0
Γ
a
t
0
t
a
t
0
-
1
exp
⁡
-
b
t
0
t
,
t
∈
α
1
,
α
2
,
α
3
,
β
kwith the shape parameter a
t
0 and rate parameter b
t
0.These hyperparameters can incorporate prior knowledge about the variances of the HR image and noise among the observed LR images into the estimation process. In the following SR method,a
t
0 and b
t
0 will be used as the initial values. a
t
0 is set equal to 1 and b
t
0 is set equal to 0, which corresponds to utilizing flat prior distributions for the hyperparameters; in this case, only the observed LR images are responsible for the estimation process.


## 3.1. Image Prior Probability Distribution

In the Bayesian method, the prior information of the original HR image represented by the prior model plays an important role. A significant property of license plate is that the target of license plate has high contrast with the background (see Figure1). The pixels in the license plate images can be divided into two classes. Thus, the original HR license plate image can be represented as(7)
X
=
X
t
a
r
∪
X
b
a
c
,where X
t
a
r represents the set of pixels belonging to the target regions (i.e., the white regions in Figure 1) and X
b
a
c represents the set of pixels belonging to the background regions (i.e., the black regions in Figure 1).Figure 1
An example of the license plate image.Therefore, we assume that, for the license plate images, the target pixels tend to cluster around one center, while the background pixels tend to cluster around another center. In order to make use of this characteristic, we propose the following model to regularize the estimated HR license plate image:(8)
R
X
=
X
-
μ
2
2
,where μ is a vector with size r
2
N
×
1. In the following, we will demonstrate how to obtain the vector μ.In (8), the elements of the vector μ are the mean values of X
t
a
r and X
b
a
c, respectively. That is to say, if X
i
∈
X
t
a
r (or X
b
a
c), thus, μ
i
=
μ
t
a
r (or μ
b
a
c), where μ
t
a
r and μ
b
a
c are the mean values of X
t
a
r and X
b
a
c, respectively. Then, μ
t
a
r and μ
b
a
c can be obtained as follows:(9)
μ
t
a
r
=
∑
X
i
N
t
a
r
,
X
i
∈
X
t
a
r
,
μ
b
a
c
=
∑
X
j
N
b
a
c
,
X
j
∈
X
b
a
c
,where N
t
a
r and N
b
a
c represent the total number of pixels in the target regions and background regions, respectively.In order to obtainμ, first, the following expressions are defined:(10)
μ
1
=
W
μ
t
,
μ
2
=
e
-
W
μ
b
,where μ
t and μ
b are vectors with size r
2
N
×
1, μ
t
(
i
)
=
μ
t
a
r and μ
b
(
i
)
=
μ
b
a
c, for i
=
1,2
,
…
,
r
2
N. W is a diagonal matrix with size r
2
N
×
r
2
N, whose elements are 0s and 1s, and e is a matrix with size r
2
N
×
r
2
N, whose elements are 1s.Then, for obtaining the matrixW, we use OSTU to partition the estimated HR image into target regions and background regions. After the partition, the element values of W corresponding to the pixels in the target regions are set as 1s; the others are set as 0s. We can obtain(11)
W
i
,
i
=
1
,
i
f
X
i
∈
X
t
a
r
,
0
,
o
t
h
e
r
s
,
(12)
e
-
W
i
,
i
=
1
,
i
f
X
i
∈
X
b
a
c
,
0
,
o
t
h
e
r
s
.Thus, (13)
μ
1
i
=
W
μ
t
i
=
μ
t
a
r
,
i
f
X
i
∈
X
t
a
r
,
0
,
o
t
h
e
r
s
,
μ
2
i
=
e
-
W
μ
b
i
=
μ
b
a
c
,
i
f
X
i
∈
X
b
a
c
,
0
,
o
t
h
e
r
s
.Finally, the nonzero elements in μ
1 and μ
2 are extracted and positioned in μ at the corresponding location as desired.Based on the prior model (8), the following prior probability distribution for the estimated HR license plate image is proposed:(14)
p
X
∣
α
1
∝
α
1
r
2
N
/
2
exp
⁡
-
α
1
2
X
-
μ
2
2
,where α
1 is the hyperparameter of this prior distribution.In order to assure the success of the reconstruction, we have adopted the total variation (TV) prior model and the simultaneous autoregressive (SAR) prior model [18] to regularize the estimated HR image. The TV and the SAR prior models are defined as follows:(15)
R
T
V
X
=
∑
i
∇
i
h
X
2
+
∇
i
v
X
2
,where ∇
i
h
X and ∇
i
v
X represent the horizontal and vertical gradient components of the ith element of X, respectively. Consider(16)
R
S
A
R
X
=
c
X
2
2
,where c denotes the Laplacian operator.The TV model’s and the SAR model’s corresponding prior distributions are defined as(17)
p
X
∣
α
2
∝
α
2
r
2
N
/
2
exp
⁡
-
α
2
2
∑
i
∇
i
h
X
2
+
∇
i
v
X
2
,
(18)
p
X
∣
α
3
∝
α
3
r
2
N
/
2
exp
⁡
-
α
3
2
c
X
2
2
.


## 3.2. Motion Prior Probability Distribution

The motion parameters are modeled as stochastic variables following Gaussian distributions, similar to [12, 18]:(19)
p
s
k
=
∏
k
=
1
L
N
s
k
∣
μ
s
k
0
,
σ
s
k
0
,where μ
s
k
0 is the a priori mean vector and σ
s
k
0 is the a priori covariance matrix. These two parameters can incorporate prior knowledge about the motion parameters into the estimation process. Setting μ
s
k
0 and σ
s
k
0 equal to zero represents the fact that no such knowledge is available, which makes only the observed LR images responsible for the estimation process.In this work, the parametersμ
s
k
0 are obtained by using the Lucas-Kanade method [19], and the inverse covariance matrices (
σ
s
k
0
)
-
1 are set equal to zero matrices. And they will be used as the initial values in the following SR method.


## 3.3. Hyperparameter Prior Probability Distribution

The prior information about the hyperparameters is usually expressed using the conjugate prior distribution which is calculated conveniently. Moreover, the corresponding posterior distribution has the same functional form with the prior distribution and hence the analytic solution can be obtained. It is well known that the inverse Gamma distributions are the conjugate priors for the variance of the Gaussian distribution whose mean value is known. Thus, we assume that the hyperparameters obey Gamma distributions; that is,(20)
p
t
=
G
a
m
m
a
t
∣
a
t
0
,
b
t
0
=
b
t
0
a
t
0
Γ
a
t
0
t
a
t
0
-
1
exp
⁡
-
b
t
0
t
,
t
∈
α
1
,
α
2
,
α
3
,
β
kwith the shape parameter a
t
0 and rate parameter b
t
0.These hyperparameters can incorporate prior knowledge about the variances of the HR image and noise among the observed LR images into the estimation process. In the following SR method,a
t
0 and b
t
0 will be used as the initial values. a
t
0 is set equal to 1 and b
t
0 is set equal to 0, which corresponds to utilizing flat prior distributions for the hyperparameters; in this case, only the observed LR images are responsible for the estimation process.


## 4. Optimization

In this method, three prior models (i.e., the proposed prior model, the TV prior model, and the SAR prior model) are used to regularize the estimated HR image; however, establishing a prior probability distribution that includes these prior models is difficult. Here, the following linear combination of three KL divergences is used to combine the proposed prior model, the TV prior model, and the SAR prior model:(21)
E
q
Ω
,
y
=
∑
i
=
1
3
λ
i
C
K
L
q
Θ
i
∥
p
Θ
i
∣
y
,where Ω
=
{
X
,
s
k
,
β
k
,
α
1
,
α
2
,
α
3
}, Θ
1
=
X
,
s
k
,
β
k
,
α
1, Θ
2
=
X
,
s
k
,
β
k
,
α
2, Θ
3
=
X
,
s
k
,
β
k
,
α
3, Θ
1, Θ
2, and Θ
3 denote the different sets of all the variables corresponding to the prior probability distributions based on the proposed prior model, the TV prior model, and SAR prior model, respectively, λ
i
≥
0 for i
=
1,2
,
3, and λ
2
+
λ
3
=
1.Then,p
(
Θ
∣
y
) is approximated by minimizing the following function:(22)
q
Ω
=
a
r
g
m
i
n
q
Θ
E
q
Ω
,
y
.Due to the half-quadratic form of TV model, (20) is difficult to be solved. In this work, this difficulty is overcome by resorting to the majorization-minimization (MM) approach [12]. Thus, a lower bound of p
(
X
∣
α
2
) is found by using the MM approach:(23)
p
X
∣
α
2
≥
α
2
r
2
N
/
2
exp
⁡
-
∑
i
=
1
r
2
N
α
2
∇
i
h
X
2
+
∇
i
v
X
2
+
z
i
2
z
i
=
M
X
,
α
2
,
z
,where z
=
{
z
i
,
i
=
1,2
,
…
,
r
2
N
} and the auxiliary variables z
i need to be calculated by using the following formula:(24)
z
i
=
∇
i
h
X
2
+
∇
i
v
X
2
q
X
.Then, posterior probability distributions,q
(
X
), q
(
s
k
), q
(
β
k
), q
(
α
1
), q
(
α
2
), and q
(
α
3
), can be obtained. The solving process is described in the Appendix in detail. Consequently, the following explicit expressions are obtained to calculate the HR image, the motion parameters, and hyperparameters.The formula for calculating the HR image is given by(25)
X
=
∑
k
β
k
B
s
k
T
B
s
k
+
∑
k
∑
i
∑
j
ξ
k
i
j
O
k
i
s
k
T
O
k
j
s
k
+
λ
1
α
1
I
+
λ
2
α
2
∇
h
T
Z
∇
h
+
∇
v
T
Z
∇
v
+
λ
3
α
3
c
T
c
-
1
∑
k
β
k
B
s
k
T
y
k
+
λ
1
α
1
T
μ
.The formula for calculating the motion parameters is given by(26)
s
k
=
δ
k
-
1
+
β
k
Γ
k
+
Ψ
k
-
1
δ
k
-
1
s
-
k
+
β
k
Γ
k
s
k
+
Ψ
k
s
k
+
Q
k
-
Φ
k
.In (25) and (26), I is an identity matrix with size r
2
N
×
r
2
N, and the formulas for ξ
k
i
j, O
k
i, Z, Γ
k, Ψ
k, Φ
k, and Q
k are given in the Appendix.The hyperparameters can be calculated by using the following formulas:(27)
β
k
=
N
/
2
+
a
β
k
0
1
/
2
y
k
-
B
s
k
X
2
2
q
X
,
q
s
k
+
b
β
k
0
,
(28)
α
1
=
r
2
N
/
2
+
a
α
1
0
1
/
2
X
-
μ
2
2
q
X
+
b
α
1
0
,
(29)
α
2
=
r
2
N
+
a
α
2
0
2
∑
i
z
i
+
b
α
2
0
,
(30)
α
3
=
r
2
N
/
2
+
a
α
3
0
1
/
2
c
X
2
2
q
X
+
b
α
3
0
.The optimization procedure can be concluded as shown in Algorithm1.Algorithm 1:Our iterative SR algorithm.
Input: The LR observations y
k; The initial HR image X
0; The auxiliary variables z
0, W
0, μ
t
a
r
0,
μ
b
a
c
0, µ
0; The initial parameters, μ
s
k
0, (
σ
s
k
0
)
-
1, a
t
0 and b
t
0, t
∈
α
1
,
α
2
,
α
3
,
β
k.
n
=
0;
repeat
ComputeX
n by using (25);
ComputeW
n by using (11);
Computeμ
t
a
r
n and μ
b
a
c
n by using (9);
Computeµ
n by using (13);
Computez
n by using (24);
Computes
k
n by using (26);
Computeβ
k
n by using (27);
Computeα
1
n, α
2
n, and α
3
n by using (28)–(30);
n
=
n
+
1;
until convergence criterion is met
Output:  X
=
X
n.


## 5. Experimental Results

We have conducted several experiments to evaluate the effectiveness of the proposed SR method for the license plate images. In the experiments, five observations were generated from each of the original images through translation, rotation, blurring, and downsampling. For translation and rotation, the following parameters are used for each image:s
1
=
0
°
,
0,0
T, s
2
=
3
°
,
0,0.5
T, s
3
=
-
3
°
,
0.5,0
T, s
4
=
5
°
,
1,0
T, and s
5
=
-
5
°
,
0,1
T. For the blurring, a uniform PSF is used. The downsampling factor is r
=
2. The additive white Gaussian noise with σ
=
0.2 and σ
=
0.5 is added to the LR observations.In our experiments,X
n
+
1
-
X
n
2
2
/
X
n
2
2
<
1
0
-
5 (X
n
+
1 and X
n are the estimated images at the (
n
+
1
)th and nth iterations, resp.) is used as the convergence criterion. X
0 is the interpolated image of the first LR image y
1, and z
i
0
=
∇
i
h
X
0
2
+
∇
i
v
X
0
2. The motion parameters μ
s
k
0 are obtained using the Lucas-Kanade method. The inverse covariance matrices (
σ
s
k
0
)
-
1 are set equal to zero matrices. And a
t
0
=
1, b
t
0
=
0, and t
∈
α
1
,
α
2
,
α
3
,
β
k. Thus, we can obtain β
k
0
=
N
/
y
k
-
B
(
s
k
)
X
0
2
2, α
1
0
=
r
2
N
/
X
0
-
μ
0
2
2, α
3
0
=
r
2
N
/
2
∑
i
z
i
0, and α
4
0
=
r
2
N
/
c
X
0
2
2. Due to λ
2
+
λ
3
=
1, we just present the values of λ
1 and λ
2. In the simulation experiments, when the noise variance is set with σ
=
0.2, the values of λ
1 and λ
2 are set with 5 and 0.1, respectively. The values of λ
1 and λ
2 are set with 0.1 and 0.01, respectively, under σ
=
0.5. In the experiments on real data, the settings of λ
1
=
10 and λ
2
=
0.2 are used.Our method is compared with the bicubic interpolation method, the TV-SAR method, and the l1-SAR method. The MATLAB code provided in [18] was used for the testing. For the test images, the performance of reconstruction methods is evaluated by measuring the improvement in peak signal-to-noise ratio (PSNR) and structure similarity (SSIM) index.


### 5.1. Simulation Experiments

In this subsection, we would like to show the experimental results by using the pictures presented in Figure2 as test images. Experimental results are used to illustrate the effectiveness of the proposed models compared with the bicubic interpolation method, the TV-SAR method, and the l1-SAR method. Figures 3, 5, 7, and 9 show the reconstructed images obtained by using different SR methods. Results obtained by applying different approaches to LR images generated from Figures 2(a) and 2(c) are presented in Figures 3 and 7, respectively. These LR images are corrupted with average blur and white Gaussian noise of σ
=
0.2. Results obtained by applying different approaches to LR images generated from Figures 2(b) and 2(d) corrupted with average blur and white Gaussian noise of σ
=
0.5 are presented in Figures 5 and 9, respectively. From these figures, we could see that the HR images reconstructed by the bicubic interpolation are blurred. From these figures, we could see that there exist obvious artifacts in these estimated HR images obtained by using the TV-SAR method and the l1-SAR method. In China, the Chinese character is an important component part of the license plate. The corresponding results are shown in Figures 7 and 9. As shown in Figure 7, the Chinese character obtained by our proposed method is polluted less severely, compared with the results obtained by other methods. Moreover, Chinese character obtained by our proposed method in Figure 9 is more obvious.Figure 2
Four test images.
(a)
(b)
(c)
(d)Figure 3
The reconstructed results for Figure2(a) under white Gaussian noise with σ
=
0.2.
(a)
Bicubic
(b)
TV-SAR
(c)
l1-SAR
(d)
Ours
(e)
HRIn order to make the visual contrast effect more obvious, the corresponding binary results are presented in Figures4, 6, 8, and 10. The binarization step is also usually included in the vehicle license plate recognition. Although the reconstructed results obtained by our proposed method have some vague senses, the corresponding binary results are better than other results obtained by other methods. The binary results obtained by our proposed method are closer to the binary results of the original HR image. For example, there exist less miscellaneous points in the binary results presented in Figures 6 and 10.Figure 4
Binary results corresponding to Figure3.
(a)
Bicubic
(b)
TV-SAR
(c)
l1-SAR
(d)
Ours
(e)
HRFigure 5
The reconstructed results for Figure2(b) under white Gaussian noise with σ
=
0.5.
(a)
Bicubic
(b)
TV-SAR
(c)
l1-SAR
(d)
Ours
(e)
HRFigure 6
Binary results corresponding to Figure5.
(a)
Bicubic
(b)
TV-SAR
(c)
l1-SAR
(d)
Ours
(e)
HRFigure 7
The reconstructed results for Figure2(c) under white Gaussian noise with σ
=
0.2.
(a)
Bicubic
(b)
TV-SAR
(c)
l1-SAR
(d)
Ours
(e)
HRFigure 8
Binary results corresponding to Figure7.
(a)
Bicubic
(b)
TV-SAR
(c)
l1-SAR
(d)
Ours
(e)
HRFigure 9
The reconstructed results for Figure2(d) under white Gaussian noise with σ
=
0.5.
(a)
Bicubic
(b)
TV-SAR
(c)
l1-SAR
(d)
Ours
(e)
HRFigure 10
Binary results corresponding to Figure9.
(a)
Bicubic
(b)
TV-SAR
(c)
l1-SAR
(d)
Ours
(e)
HRThe PSNR and SSIM values of each SR reconstruction method are presented in Tables1 and 2. From these two tables, we see that our proposed method could produce the reconstructed HR image with the highest PSNR and SSIM values. We take the HR image presented in Figure 2(a) for example. The PSNR values achieved by different methods corresponding to σ
=
0.2 and σ
=
0.5, respectively, are shown in the following: the bicubic method (PSNR (dB): 16.32, 15.15), the l1-SAR method (PSNR (dB): 20.23, 16.36), the TV-SAR method (PSNR (dB): 20.23, 16.36), and our proposed method (PSNR (dB): 20.76, 18.14). From these tables, it is noted that, in the case of σ
=
0.2, the PSNR value of our proposed method (20.76 dB) has outperformed that of the bicubic interpolation method more than 4.0 dB. And in this case, the PSNR value of our proposed method is slightly better than those of the l1-SAR method and the TV-SAR method. Even under a stronger noise, in the case of σ
=
0.5, the PSNR value (18.14 dB) of our proposed method is at least 1.5 dB larger than those of the bicubic interpolation method, the TV-SAR method, and the l1-SAR method.Table 1
Comparisons of PSNR (dB) and SSIM with average blur and white Gaussian noise ofσ
=
0.2.
Image
Standard
Method
Bicubic
l1-SAR
TV-SAR
Ours
Figure2(a)
PSNR
16.32
20.23
20.23
20.76
SSIM
0.8229
0.9484
0.9484
0.9522
Figure2(b)
PSNR
17.98
22.13
22.13
22.69
SSIM
0.8266
0.8931
0.8932
0.9022
Figure2(c)
PSNR
15.49
19.37
19.35
19.85
SSIM
0.9010
0.9710
0.9709
0.9714
Figure2(d)
PSNR
15.26
18.38
18.38
18.56
SSIM
0.7221
0.8956
0.8956
0.8980Table 2
Comparisons of PSNR (dB) and SSIM with average blur and white Gaussian noise ofσ
=
0.5.
Image
Standard
Method
Bicubic
l1-SAR
TV-SAR
Ours
Figure2(a)
PSNR
15.15
16.36
16.36
18.14
SSIM
0.7766
0.8907
0.8906
0.9185
Figure2(b)
PSNR
16.86
18.06
18.04
19.89
SSIM
0.7694
0.8125
0.8122
0.8582
Figure2(c)
PSNR
14.77
16.75
16.75
17.80
SSIM
0.8860
0.9486
0.9486
0.9560
Figure2(d)
PSNR
14.48
15.56
15.56
16.47
SSIM
0.6863
0.8161
0.8162
0.8379


### 5.2. Discussion

In this paper, the proposed method and the comparison methods are all based on the variational Bayesian framework for fair. The computational complexity of such kind of method has been analyzed in [12]. The majority of computations are performed for estimating the HR image and the motion parameters. The HR image is calculated by using the conjugate gradient method [12], and the motion parameters are calculated by inverting a 3 × 3 matrix for each observed LR image. Note that the matrix multiplications can be performed very efficiently by implementing the corresponding operators rather than storing full matrices.A comparison of the computation time is listed in Table3, under the average blur with size 7 × 7 and the zero-mean Gaussian white noise with σ
=
0.2. Table 3 shows that the proposed method uses the least time among the iterative methods. In the proposed method, the characteristic of gray-level distribution is introduced as constraints with the TV-SAR model into the reconstruction. During the reconstruction process, the target pixels and the background pixels tend to cluster to different centers, which is beneficial to reduce the computation time.Table 3
Comparison of the time complexities (in seconds).
Image
Method
l1-SAR
TV-SAR
Ours
Figure2(a)
686.04
711.05
606.39
Figure2(b)
863.50
768.99
649.92
Figure2(c)
2190.03
3288.36
2493.18
Figure2(d)
2506.06
2491.23
2213.24In Section5.1, the experimental results under challenging zero-mean Gaussian white noise with variances 0.2 and 0.5, respectively, are presented. In the practical application, sometimes, small noise may exist. In order to verify the performance of the proposed Bayesian framework in the presence of various intensity Gaussian white noise, several simulated experiments are conducted by using Figure 2(b) as a test image, as shown in Figure 11. It can be seen that the PSNR value increases as the noise becomes smaller.Figure 11
The PSNR values obtained by the proposed method under different noise variances.In the experiments under different noise variances, the parametersλ
1 and λ
2 are adjusted by the criteria of obtaining optimal results. By taking the average blur with size 7 × 7 and zero-mean noise with σ
=
0.1 as an example, the relationship between the parameters λ
1 and λ
2 and PSNR is shown in Figure 12. From Figure 12, we see that when λ
1 nears 4 and λ
2 nears 0.1, the highest PSNR value can be obtained. Otherwise, the PSNR value decreases if the stable λ
1 is kept. It is noted that when λ
1 nears 7 and λ
2 nears 0.9, PSNR value decreases by a relatively sharp stage. Their relationship is quantitatively clear.Figure 12
Relationship between the parametersλ
1 and λ
2 and PSNR.
(a)
(b)
X
-
Z view
(c)
Y
-
Z view


### 5.3. Experiments on Real Data

In this subsection, we test our proposed method on several vehicle plate image sequences. A commercial digital continuous shooting mode was used to capture the vehicle image sequences. In this experiment, we use four sequences as examples, and each sequence has ten images. Figure13 shows one of the ten LR images obtained with a camera far off, and some plate image sequences with very poor quality under different conditions are used, as shown in Figure 16. In this experiment, the license plate is selected as the region of interest.Figure 13
One of the LR plate images in the case of long distance. (a) Vehicle sequence I and (b) vehicle sequence II.
(a)
(b)Figures14, 15, 17, and 18 show a comparison between bicubic interpolation method, the l1-SAR method, the TV-SAR method, and our proposed method. In Figure 14(a), the number “5” could be misinterpreted as “S.” And there exist obvious artifacts in the reconstructed images obtained by l1-SAR and TV-SAR, respectively. In Figure 15, the results obtained by the bicubic interpolation method are quite blur, which is hard to read. In Figures 15(b) and 15(c), the number “6” and letter “P” are unreadable. However, the Chinese characteristic is still not readable in Figures 14 and 15, and the letter “A” might be confusing. In Figures 17(b) and 17(c), the letter “C” could be misinterpreted as “0.” In Figure 18, the reconstructed result obtained by the proposed method has a higher degree of visual contrast between the target and background; thus, it could be readable easily. In the proposed method, the characteristic of gray-level distribution is introduced as constraints into the reconstruction. During the reconstruction process, the target pixels and the background pixels tend to cluster to different centers, which is beneficial to use the gradient information to preserve the image edges and suppress noise. Meanwhile, the smoothing prior is useful to divide the target and background. Thus, by using the prior information of the license plate image more fully, the proposed method obtains better reconstructed results.Figure 14
The reconstructed results for the first LR license plates.
(a)
Bicubic
(b)
TV-SAR
(c)
l1-SAR
(d)
OursFigure 15
The reconstructed results for the second LR license plates.
(a)
Bicubic
(b)
TV-SAR
(c)
l1-SAR
(d)
OursFigure 16
One of the LR plate images in the case of (a) rain and fog weather and (b) black vehicle exhaust.
(a)
(b)Figure 17
The reconstructed results for the LR license plates obtained under the rain and fog weather.
(a)
Bicubic
(b)
TV-SAR
(c)
l1-SAR
(d)
OursFigure 18
The reconstructed results for the LR license plates disturbed by black vehicle exhaust.
(a)
Bicubic
(b)
TV-SAR
(c)
l1-SAR
(d)
OursThese experimental results show that the proposed method in this paper achieves the best visual effect. Also, the practical utility and potential of our proposed method in enhancing LR frames captured from real traffic could be demonstrated by using the experiments described previously.


## 5.1. Simulation Experiments

In this subsection, we would like to show the experimental results by using the pictures presented in Figure2 as test images. Experimental results are used to illustrate the effectiveness of the proposed models compared with the bicubic interpolation method, the TV-SAR method, and the l1-SAR method. Figures 3, 5, 7, and 9 show the reconstructed images obtained by using different SR methods. Results obtained by applying different approaches to LR images generated from Figures 2(a) and 2(c) are presented in Figures 3 and 7, respectively. These LR images are corrupted with average blur and white Gaussian noise of σ
=
0.2. Results obtained by applying different approaches to LR images generated from Figures 2(b) and 2(d) corrupted with average blur and white Gaussian noise of σ
=
0.5 are presented in Figures 5 and 9, respectively. From these figures, we could see that the HR images reconstructed by the bicubic interpolation are blurred. From these figures, we could see that there exist obvious artifacts in these estimated HR images obtained by using the TV-SAR method and the l1-SAR method. In China, the Chinese character is an important component part of the license plate. The corresponding results are shown in Figures 7 and 9. As shown in Figure 7, the Chinese character obtained by our proposed method is polluted less severely, compared with the results obtained by other methods. Moreover, Chinese character obtained by our proposed method in Figure 9 is more obvious.Figure 2
Four test images.
(a)
(b)
(c)
(d)Figure 3
The reconstructed results for Figure2(a) under white Gaussian noise with σ
=
0.2.
(a)
Bicubic
(b)
TV-SAR
(c)
l1-SAR
(d)
Ours
(e)
HRIn order to make the visual contrast effect more obvious, the corresponding binary results are presented in Figures4, 6, 8, and 10. The binarization step is also usually included in the vehicle license plate recognition. Although the reconstructed results obtained by our proposed method have some vague senses, the corresponding binary results are better than other results obtained by other methods. The binary results obtained by our proposed method are closer to the binary results of the original HR image. For example, there exist less miscellaneous points in the binary results presented in Figures 6 and 10.Figure 4
Binary results corresponding to Figure3.
(a)
Bicubic
(b)
TV-SAR
(c)
l1-SAR
(d)
Ours
(e)
HRFigure 5
The reconstructed results for Figure2(b) under white Gaussian noise with σ
=
0.5.
(a)
Bicubic
(b)
TV-SAR
(c)
l1-SAR
(d)
Ours
(e)
HRFigure 6
Binary results corresponding to Figure5.
(a)
Bicubic
(b)
TV-SAR
(c)
l1-SAR
(d)
Ours
(e)
HRFigure 7
The reconstructed results for Figure2(c) under white Gaussian noise with σ
=
0.2.
(a)
Bicubic
(b)
TV-SAR
(c)
l1-SAR
(d)
Ours
(e)
HRFigure 8
Binary results corresponding to Figure7.
(a)
Bicubic
(b)
TV-SAR
(c)
l1-SAR
(d)
Ours
(e)
HRFigure 9
The reconstructed results for Figure2(d) under white Gaussian noise with σ
=
0.5.
(a)
Bicubic
(b)
TV-SAR
(c)
l1-SAR
(d)
Ours
(e)
HRFigure 10
Binary results corresponding to Figure9.
(a)
Bicubic
(b)
TV-SAR
(c)
l1-SAR
(d)
Ours
(e)
HRThe PSNR and SSIM values of each SR reconstruction method are presented in Tables1 and 2. From these two tables, we see that our proposed method could produce the reconstructed HR image with the highest PSNR and SSIM values. We take the HR image presented in Figure 2(a) for example. The PSNR values achieved by different methods corresponding to σ
=
0.2 and σ
=
0.5, respectively, are shown in the following: the bicubic method (PSNR (dB): 16.32, 15.15), the l1-SAR method (PSNR (dB): 20.23, 16.36), the TV-SAR method (PSNR (dB): 20.23, 16.36), and our proposed method (PSNR (dB): 20.76, 18.14). From these tables, it is noted that, in the case of σ
=
0.2, the PSNR value of our proposed method (20.76 dB) has outperformed that of the bicubic interpolation method more than 4.0 dB. And in this case, the PSNR value of our proposed method is slightly better than those of the l1-SAR method and the TV-SAR method. Even under a stronger noise, in the case of σ
=
0.5, the PSNR value (18.14 dB) of our proposed method is at least 1.5 dB larger than those of the bicubic interpolation method, the TV-SAR method, and the l1-SAR method.Table 1
Comparisons of PSNR (dB) and SSIM with average blur and white Gaussian noise ofσ
=
0.2.
Image
Standard
Method
Bicubic
l1-SAR
TV-SAR
Ours
Figure2(a)
PSNR
16.32
20.23
20.23
20.76
SSIM
0.8229
0.9484
0.9484
0.9522
Figure2(b)
PSNR
17.98
22.13
22.13
22.69
SSIM
0.8266
0.8931
0.8932
0.9022
Figure2(c)
PSNR
15.49
19.37
19.35
19.85
SSIM
0.9010
0.9710
0.9709
0.9714
Figure2(d)
PSNR
15.26
18.38
18.38
18.56
SSIM
0.7221
0.8956
0.8956
0.8980Table 2
Comparisons of PSNR (dB) and SSIM with average blur and white Gaussian noise ofσ
=
0.5.
Image
Standard
Method
Bicubic
l1-SAR
TV-SAR
Ours
Figure2(a)
PSNR
15.15
16.36
16.36
18.14
SSIM
0.7766
0.8907
0.8906
0.9185
Figure2(b)
PSNR
16.86
18.06
18.04
19.89
SSIM
0.7694
0.8125
0.8122
0.8582
Figure2(c)
PSNR
14.77
16.75
16.75
17.80
SSIM
0.8860
0.9486
0.9486
0.9560
Figure2(d)
PSNR
14.48
15.56
15.56
16.47
SSIM
0.6863
0.8161
0.8162
0.8379


## 5.2. Discussion

In this paper, the proposed method and the comparison methods are all based on the variational Bayesian framework for fair. The computational complexity of such kind of method has been analyzed in [12]. The majority of computations are performed for estimating the HR image and the motion parameters. The HR image is calculated by using the conjugate gradient method [12], and the motion parameters are calculated by inverting a 3 × 3 matrix for each observed LR image. Note that the matrix multiplications can be performed very efficiently by implementing the corresponding operators rather than storing full matrices.A comparison of the computation time is listed in Table3, under the average blur with size 7 × 7 and the zero-mean Gaussian white noise with σ
=
0.2. Table 3 shows that the proposed method uses the least time among the iterative methods. In the proposed method, the characteristic of gray-level distribution is introduced as constraints with the TV-SAR model into the reconstruction. During the reconstruction process, the target pixels and the background pixels tend to cluster to different centers, which is beneficial to reduce the computation time.Table 3
Comparison of the time complexities (in seconds).
Image
Method
l1-SAR
TV-SAR
Ours
Figure2(a)
686.04
711.05
606.39
Figure2(b)
863.50
768.99
649.92
Figure2(c)
2190.03
3288.36
2493.18
Figure2(d)
2506.06
2491.23
2213.24In Section5.1, the experimental results under challenging zero-mean Gaussian white noise with variances 0.2 and 0.5, respectively, are presented. In the practical application, sometimes, small noise may exist. In order to verify the performance of the proposed Bayesian framework in the presence of various intensity Gaussian white noise, several simulated experiments are conducted by using Figure 2(b) as a test image, as shown in Figure 11. It can be seen that the PSNR value increases as the noise becomes smaller.Figure 11
The PSNR values obtained by the proposed method under different noise variances.In the experiments under different noise variances, the parametersλ
1 and λ
2 are adjusted by the criteria of obtaining optimal results. By taking the average blur with size 7 × 7 and zero-mean noise with σ
=
0.1 as an example, the relationship between the parameters λ
1 and λ
2 and PSNR is shown in Figure 12. From Figure 12, we see that when λ
1 nears 4 and λ
2 nears 0.1, the highest PSNR value can be obtained. Otherwise, the PSNR value decreases if the stable λ
1 is kept. It is noted that when λ
1 nears 7 and λ
2 nears 0.9, PSNR value decreases by a relatively sharp stage. Their relationship is quantitatively clear.Figure 12
Relationship between the parametersλ
1 and λ
2 and PSNR.
(a)
(b)
X
-
Z view
(c)
Y
-
Z view


## 5.3. Experiments on Real Data

In this subsection, we test our proposed method on several vehicle plate image sequences. A commercial digital continuous shooting mode was used to capture the vehicle image sequences. In this experiment, we use four sequences as examples, and each sequence has ten images. Figure13 shows one of the ten LR images obtained with a camera far off, and some plate image sequences with very poor quality under different conditions are used, as shown in Figure 16. In this experiment, the license plate is selected as the region of interest.Figure 13
One of the LR plate images in the case of long distance. (a) Vehicle sequence I and (b) vehicle sequence II.
(a)
(b)Figures14, 15, 17, and 18 show a comparison between bicubic interpolation method, the l1-SAR method, the TV-SAR method, and our proposed method. In Figure 14(a), the number “5” could be misinterpreted as “S.” And there exist obvious artifacts in the reconstructed images obtained by l1-SAR and TV-SAR, respectively. In Figure 15, the results obtained by the bicubic interpolation method are quite blur, which is hard to read. In Figures 15(b) and 15(c), the number “6” and letter “P” are unreadable. However, the Chinese characteristic is still not readable in Figures 14 and 15, and the letter “A” might be confusing. In Figures 17(b) and 17(c), the letter “C” could be misinterpreted as “0.” In Figure 18, the reconstructed result obtained by the proposed method has a higher degree of visual contrast between the target and background; thus, it could be readable easily. In the proposed method, the characteristic of gray-level distribution is introduced as constraints into the reconstruction. During the reconstruction process, the target pixels and the background pixels tend to cluster to different centers, which is beneficial to use the gradient information to preserve the image edges and suppress noise. Meanwhile, the smoothing prior is useful to divide the target and background. Thus, by using the prior information of the license plate image more fully, the proposed method obtains better reconstructed results.Figure 14
The reconstructed results for the first LR license plates.
(a)
Bicubic
(b)
TV-SAR
(c)
l1-SAR
(d)
OursFigure 15
The reconstructed results for the second LR license plates.
(a)
Bicubic
(b)
TV-SAR
(c)
l1-SAR
(d)
OursFigure 16
One of the LR plate images in the case of (a) rain and fog weather and (b) black vehicle exhaust.
(a)
(b)Figure 17
The reconstructed results for the LR license plates obtained under the rain and fog weather.
(a)
Bicubic
(b)
TV-SAR
(c)
l1-SAR
(d)
OursFigure 18
The reconstructed results for the LR license plates disturbed by black vehicle exhaust.
(a)
Bicubic
(b)
TV-SAR
(c)
l1-SAR
(d)
OursThese experimental results show that the proposed method in this paper achieves the best visual effect. Also, the practical utility and potential of our proposed method in enhancing LR frames captured from real traffic could be demonstrated by using the experiments described previously.


## 6. Conclusion

Current existing reconstruction methods for license plate images use the image models based on gradient information. However, they do not perform well, especially in the heavy noise. This will result in poor recognition results. In this paper, we proposed a new SR reconstruction method to reconstruct the license plate images. Given the significant characteristic of license plate, a feature-specific prior model has been proposed in this study and combined with the TV-SAR prior model. The target and the background can be divided as far as possible during the reconstruction process, which is beneficial to use the gradient information to preserve the image edges and suppress noise. In this paper, the HR image, the motion parameters, and the hyperparameters are estimated jointly by using the variational Bayesian inference estimator, and hence an unsupervised SR method for reconstructing license plate images is established.In the future work, we will focus our study on estimating the parametersλ
1, λ
2, and λ
3 and finding the relationship between these parameters.



---

*Source: 4194309-2016-04-19.xml*