# Regularized Multiframe Super-Resolution Image Reconstruction Using Linear and Nonlinear Filters

Authors: Mahmoud M. Khattab; Akram M. Zeki; Ali A. Alwan; Belgacem Bouallegue; Safaa S. Matter; Abdelmoty M. Ahmed
Journal: Journal of Electrical and Computer Engineering
 (2021)

---

## Abstract

The primary goal of the multiframe super-resolution image reconstruction is to produce an image with a higher resolution by integrating information extracted from a set of corresponding images with low resolution, which is used in various fields. However, super-resolution image reconstruction approaches are typically affected by annoying restorative artifacts, including blurring, noise, and staircasing effect. Accordingly, it is always difficult to balance between smoothness and edge preservation. In this paper, we intend to enhance the efficiency of multiframe super-resolution image reconstruction in order to optimize both analysis and human interpretation processes by improving the pictorial information and enhancing the automatic machine perception. As a result, we propose new approaches that firstly rely on estimating the initial high-resolution image through preprocessing of the reference low-resolution image based on median, mean, Lucy-Richardson, and Wiener filters. This preprocessing stage is used to overcome the degradation present in the reference low-resolution image, which is a suitable kernel for producing the initial high-resolution image to be used in the reconstruction phase of the final image. Then,L2 norm is employed for the data-fidelity term to minimize the residual among the predicted high-resolution image and the observed low-resolution images. Finally, bilateral total variation prior model is utilized to restrict the minimization function to a stable state of the generated HR image. The experimental results of the synthetic data indicate that the proposed approaches have enhanced efficiency visually and quantitatively compared to other existing approaches.


---

## Body

## 1. Introduction

Over the last two decades, the world has experienced an enormous advancement in software and hardware technologies. Industrial sectors have made the best use of modern technology to generate electronic devices such as computer systems, cellular mobile phones, personal digital assistant (PDA), and innumerable devices at inexpensive costs [1]. Moreover, the manufacturing methods of camera sensors have been highly developed to generate high-quality digital cameras. Additional information is provided from high-resolution (HR) images to have a greater visual perspective. In various imaging applications, HR images are used widely, including but not limited to video surveillance [2], medical imaging [3], forensic imaging [4], and remote sensing [5]. They are still in urgent need for HR image which frequently exceeds the abilities of the HR digital cameras [6, 7]. However, the existing imagery system produces low-resolution (LR) images which must be improved in order to obtain HR images.In many real-life imaging systems, there are a variety of possible factors for reducing the image resolution due to various physical constraints, inadequate photo detectors, a lower rate of spatial sampling, and an inefficient method for capturing images [8–11]. In the past years, super-resolution (SR) image reconstruction approaches have become a powerful and cost-effective solution to increase the quality of the recorded LR images, meeting the growing market demand for HR images [12–14].SR image reconstruction increases the image quality retrieved from a series of corresponding LR images. Sensitive information is fused to rebuild the HR image by passing LR images through three processes, namely, image registration, image interpolation, and image restoration. However, the SR image is limited to three major problems: the inaccurate subpixel registration among a sequence of LR images, the ill-conditional nature of the degrading matrix, and the inadequate amount of LR images [15]. The SR issue for multiframe SR image reconstruction was first presented by Tsai and Huang based on the concept of frequency domain introduced by [16]. Since then, various approaches for overcoming the SR problem have been proposed in the literature [8–11, 17, 18]. The main aim of the regularization method is to resolve the SR issue due to its ill-conditional nature. Recent works have been conducted aiming at resolving the SR issue by exploiting the regularization framework [8–11, 19–26]. The idea of reconstruction of the SR image relies on the regularization process that generates the HR image through attempting to minimize the main objective function, where it incorporates both the fidelity of data and the regularization terms. According to the fidelity of the data term that calculates the convergence of both the measured HR image and the obtained LR images, while the main problem is resolved and the stable solution is achieved using the regularization term, the regularized SR image reconstructing employs a maximum a posteriori (MAP) method to match a posteriori distribution across an HR image as the data-fidelity term, whereas the regularization term may be used as an image prior [10, 27].The process of SR image reconstruction is usually an inverse problem that is difficult to overcome without using any prior knowledge [8, 9, 13, 28]. As a result, a variety of regularization-based SR methods were proposed in the previous works, which integrated the prior information to estimate the unknown HR image. Tikhonov prior-based SR approach proposed using the L2 norm, where it can easily eliminate noises from the image and blur the image boundaries [29]. Farsiu et al. proposed the bilateral total variation (BTV) prior to penalize gradient magnitudes, calculated according to the L1 norm [10, 27, 30]. BTV is used to preserve edges well but artifacts are produced in the smoothed areas [17].In this paper, we intend to enhance the efficiency of multiframe super-resolution image reconstruction in order to optimize both analysis and human interpretation processes by improving the pictorial information and enhancing the automatic machine perception. The regularized framework is used to provide additional information as well as an attempt to overcome the inverse problem that is not appropriate for the image reconstruction process. Therefore, the main contributions are as follows.An efficient initialization approach is developed based on estimating the initial HR image through the preprocessing stage on the reference LR image. This approach uses linear and nonlinear filters to propose two initialization methods. The proposed initialization approaches are as follows:(i)
Proposing a new method inspired by applying the median, mean, and Wiener filters at the reference LR image to remove the noise from the image, smooth the image by reducing the amount of intensity variation between neighboring pixels, and eliminate the blurring from the image, respectively.(ii)
Developing another initialization approach based on the use of median, Lucy-Richardson, and Wiener filters at the reference LR image. It is significantly cleaner and more vivid due to the use of a median filter that is able to remove the noise from the image, Lucy-Richardson filter can smooth the image by reducing the amount of intensity variation between neighboring pixels, and Wiener filter is able to eliminate the blurring from the image.The rest of this paper is organized as follows: Section2 explains the SR problem formulation for image observation or degradation model and the Bayesian framework is shown in Section 3. The image SR reconstruction prior model is illustrated in detail in Section 4. The proposed multiframe SR approaches are presented and discussed in Section 5, while Section 6 demonstrates the experimental results. Lastly, the conclusion is outlined in Section 7.


## 2. Image Observation Model

A suitable image model, which can simulate the physical process of the image degradation for SR reconstruction, has to be established. There are some degrading influences in the practical sampling process including atmospheric instability, the motion of objects, blur effect, and downsampling equipment [12]. Consequently, the choice of a suitable degradation model is the primary key to reconstruct the SR image as shown in Figure 1 which is the core kernel of the image reconstruction process [31].Figure 1
Sketch of the degradation model.In this paper, let us assume that the procedure of image acquisition consists of warping, blurring, downsampling, and noise degradations. Initially, let us assume thatX presents the HR image and Yk is a set of LR images and M is its number. Then, Fk represents the movement matrix which is used to construct the movement degrading process of the kth LR image, Hk is the blurring matrix which represents the point spread function (PSF) for the sensor of the camera, and Dk describes the matrix of downsampling that passes through both the movement and the blur matrices of the kth image. Lastly, the additive system noise is indicated by nk. Accordingly, the image degradation model which is used to simulate the whole process can be rewritten as in the following equation [8, 9, 11, 13]:(1)Yk=DkHkFkX+nk,1≤k≤M.In general, we presume that the generated LR images are produced in similar situations, and the model in the following equation can be derived:(2)Yk=DHFkX+nk,1≤k≤M,where D and H describe the same motion and blur matrices, respectively. The image degradation model is demonstrated in Figure 1 [11, 32].


## 3. Bayesian Framework of Super Resolution

The main purpose of the Bayesian framework is to measure the probability of the uncertainty problem based on the available knowledge or information by integrating various priors into a mathematical model which is more powerful than statistical inference. Therefore, the SR technology for estimating the unknown HR image from a sequence of LR observed images is the uncertainty inference problem. In other words, the observed LR images are evidence of HR image deduction, and the regularization that represents the image prior is limited by the reconstructed solution of SR [17].In the Bayesian framework, SR reconstruction is equivalent to the estimation of the HR image with given LR images. The maximum a posteriori (MAP) method is a popular one for the Bayesian framework because it integrates prior restrictions of the image and achieves results by optimizing the posterior probability cost function. Besides, it is known for its versatility in the preservation of edges and the estimation of joint parameters. In general, it is possible to use Bayesian estimation when there is a need to estimate the posteriori probability distribution for unknown parameters rather than using the specified parameters [33]. MAP estimator is extensively utilized as described in the following equation:(3)X^MAP=argmaxX∏k=1MPYk|X⋅PX,where PYk|X explains the conditional probability of the LR image (Yk) by looking for the HR image (X) and P(X) indicates the prior probability of the HR image.In equation (2), the image degrading model describes the process of producing LR images and also inversely estimates the corresponding unknown HR image. The fundamental structure of multiframe SR includes a data-fidelity term and also a regularization term. M estimator is implemented for the data-fidelity term to minimize the residual among the predicted HR image and the observed LR images. The regularization term is designed to restrict the minimization function to a stable state of the generated HR image [33]. Therefore, the classical multiframe SR framework can be described as follows:(4)X^=argminX∑k=1NρDHFkX−Yk+λRX,where the data-fidelity term is represented by ρ(·), R(·) describes the regularization term, and the regularization parameter is defined by λ.


## 4. Image Reconstruction Model

As discussed earlier, the regularization-based SR framework in spatial domain primarily includes data-fidelity and regularization terms [8]. This section explains the multiframe SR image reconstruction models that need to reproduce the HR image. A brief review of Lp norm is discussed as the data-fidelity term and also the BTV model is described as the regularization term. Farsiu et al. [30] proposed that the Lp error norm is used as the data-fidelity term, and the estimated HR image is solved when p=1 by(5)FX=argminX∑k=1NDHFkX−Ykpp.Farsiu et al. [30] proposed the suitable model called BTV, which integrates a bilateral filter and the total variation (TV) regularization. BTV regularization takes into account the use of a large number of neighbors to measure the gradient at a given pixel. In addition, BTV tends to preserve sharp edges in images with little artifacts as well as being computationally inexpensive and easily implemented. The main formula of BTV prior is defined as follows:(6)RBTVX=∑l=−PP∑m=0Pαm+lX−SxlSymX1,where Sxl shifts X in the horizontal direction by l pixels, Sym shifts X in the vertical direction by m pixels, and l+m≥0. Furthermore, α represents a scaling weight in the range of 0 < α <1 and the control parameter is P.Due to the fact that SR is an ill-posed problem, an unlimited number of solutions are possible and the problem can be solved in unknown situations. In addition, the little amounts of noise in the measurements lead to large perturbations in the final solution which is an unstable solution. Consequently, the methods of using regularization in SR image reconstruction are a very useful way to reach a stable solution to eliminate artifacts from the final solution and enhance the convergence rate. There are a large number of regularization methods and we would like to use one of them to obtain the HR images with sharp edges and ease of implementation. The major benefit of using the regularization term in image reconstruction stage is to compensate for the lost data with some prior knowledge that is used as a penalty function for the minimization objective function [27]. The cost function of the BTV regularization term [30] in equation (6) is presented as follows:(7)X^=argminX∑k=1NDHFkX−Yk1+λ∑l=−PP∑m=0Pαm+lX−SxlSymX1.In this paper, the steepest descent approach is applied to equation (7) to find a closed solution, where the steepest descent approach is used for fast convergence to the closely true HR image and estimating the HR image with low number of iterations. The measurement iteration using the steepest descent approach is shown below:(8)X^n+1=X^n−β∑k=1NFkTHTDTsignDHFkX^n−Yk+λ∑l=−PP∑m=0Pαm+lI−Sy−mSx−lsignX^n−SxlSymX^n,where β is a scalar that regulates the step size in the gradient direction, λ is the regularization parameter, and Sx−l and Sy−m describe the transposing of matrices Sxl and Sym, respectively.


## 5. Proposed Multiframe SR Approaches

As mentioned earlier, the HR imageX^ is measured by a series of LR images. Nevertheless, the inverse SR problem is really ill-posed in the case of noisy and blurring effects. In order to resolve this issue by the use of a Bayesian framework, we consider inserting prior information into the HR image X. Due to the fact that the HR image X is known to have a white Gaussian noise, the estimated vector Yk is considered to be Gaussian. The process of generating the HR image X^ through the Bayes rule means that the minimization problem is solved using the MAP framework in equation (5).The traditional SR approaches use interpolation methods on the reference LR image such as bicubic or bilinear methods to estimate the initial HR image. However, the reference LR image has a poor quality leading to weakness in estimating the initial HR image. In the event that the initial HR image is properly estimated, the minimization function may go to a steady status very quickly. As a result, the final HR image can be recovered with a high quality by attempting to treat the reference LR image. The initial HR image has a significant impact on the reconstruction process which helps to reduce the impact of noise and artifacts, improve the quality of the output solution, and speed up the rate of convergence to converge towards the true estimate.Numerous strategies of the multiframe SR image reconstruction have been proposed to improve the quality of the estimated HR image with various success degrees. The traditional regularization terms, including Tikhonov and TV, cannot differentiate edges from noises. Thus, despite the elimination of noise, the texture information is also suppressed, which reduces the effectiveness of these techniques. Therefore, BTV is the widely used prior function. The BTV approach is effective in retrieving sharp-edged images well because it includes the larger neighborhood in computing the gradient at a specific pixel, but it fails on images with smooth surfaces which suffer from the staircase effect. Also, BTV still suffers from noise and blur effect.In this paper, we propose the new multiframe image SR approaches that firstly rely on estimating the initial HR image through preprocessing of the reference LR image with different filters. This preprocessing stage is used to overcome the degradation present in the reference LR image, which is a suitable kernel for producing the initial HR image to be used in the reconstruction phase of the final image. Then, theL2 norm is employed for the data-fidelity term to minimize the residual among the predicted HR image and the observed LR images. Finally, the BTV prior model is utilized to restrict the minimization function to a stable state of the generated HR image.The two proposed initialization methods are shown in Figures2 and 3. The first proposed initialization method is inspired by applying the median, mean, and Wiener filters at the reference LR image to produce the composed LR image as shown in Figure 2. Meanwhile, the second proposed initialization method is based on the use of median, Lucy-Richardson, and Wiener filters at the reference LR image to generate the composed LR image as shown in Figure 3. Compared with these four filters, the median filter is used to remove the noise from the image. The mean filter is used to smooth the image by reducing the amount of intensity variation between neighbouring pixels. Wiener and Lucy-Richardson filters are used to eliminate the blurring from the image. In the end, regardless of whether the first or the second proposed method is selected to generate the composed LR image, the initial HR image is estimated through the bicubic interpolation method form the composed LR image. Then, the L2 norm is employed for the data-fidelity term to minimize the residual among the predicted HR image and the observed LR images. Finally, BTV prior model is utilized to restrict the minimization function to a stable state of the generated HR image.Figure 2
The block diagram of the first proposed initialization method.Figure 3
The block diagram of the second proposed initialization method.The main aim of the proposed approaches is to preserve the important image characteristics as much as possible including borders and corners, preserve the sharp image edges, and overcome artifacts. In addition, these proposed approaches increase the high-frequency components, eliminate degradations in the image capturing systems, and achieve a good balance for edge protection and noise reduction. The process flow of the proposed approaches comprises the following steps:Step (1): Generate a series of LR images from the original HR image by using the above observation model in equation (3), so that it is used as an input to the proposed approach.Step (2): Assign one of its LR images as the LR reference image to be used for initializing the HR image.Step (3): Estimate the motion parameters by calculating both of the rotated angle and the vertical and horizontal shift among the LR reference image and the next LR image to enhance the warp matrix Fk.Step (4): Apply the first or second proposed initialization method on the reference LR image in Figure 2 or Figure 3, respectively, to produce the composed image which is used as an input to the image interpolation stage.Step (5): Apply the bicubic interpolation on the composed image by using the magnification factor to measure the initial HR image (X0) which is used as an input to the image reconstruction stage.Step (6): Input both of the initial estimation image (X0) and the LR images into an SR image reconstruction model that depends on the L2 norm and BTV prior model according to equation (8) beside using the motion parameters to reconstruct the final HR image (Xf).The schematic diagram of the whole procedure flow for SR image reconstruction approach based on the BTV prior model is shown in Figure4.Figure 4
The schematic diagram of the proposed multiframe SR approach.


## 6. Experimental Results and Discussion

This section presents and discusses the actual performance evaluation of the approach proposed in comparison with several well-known state-of-the-art approaches in the literature. The performance analysis of the proposed approaches is evaluated by many experimental results. Section6.1 defines the experimental environment and also describes the parameter settings. The performance analysis and criteria are illustrated in Section 6.2. In Section 6.3, the simulated experimental results are performed from four common SR approaches compared to our proposed SR approaches.


### 6.1. Experimental Setup

In this subsection, different simulated results are presented to estimate the efficiency of the proposed SR approaches. The proposed SR approaches are evaluated on a large benchmark database on the Internet [15] consisting of photographs of different scenes. We select eight images from a benchmark database of various styles to generate our simulated LR sequences as elaborated in Figure 5. Figures 5(a)–5(f) illustrate the original grayscale images for Acen, Cartap, Foreman, Text, and Brain with the size of 120 × 120 and License Plate image with the size of 250 × 100. Moreover, the original RGB images for Natural Scene and Parrot with the size of 120 × 120 are shown in Figures 5(g)–5(h). All experiments are performed directly on grayscale images and RGB channels for color images. These eight original HR images are very different from each other. Accordingly, the main reasons for choosing these images are their content, their availability for comparisons, and them being realistic, and these images have been frequently used by most recent works related to [3, 34–36]. Additionally, they are very different from each other in the following characteristics: complex structure, amount of irregular lines, texture features, gray level histogram, contrast, and smoothness.Figure 5
A sequence of benchmark images are used in experimental tests as an original image. (a) Acen. (b) Cartap. (c) Foreman. (d) Text. (e) Brain. (f) License Plate. (g) Natural Scene. (h) Parrot.
(a)(b)(c)(d)(e)(f)(g)(h)In practical terms, the performance of SR approaches cannot be measured arithmetically, because the HR images of the ground truth are still unavailable. In all experiments, the proposed approaches are initially examined on synthetic data, because the ground-truth images are available for synthetic LR images. Throughout all experiments, the same data and parameter settings are used to compare the proposed SR approaches to the four following state-of-the-art SR approaches:(1)
An SR method with the Lorentzian error norm in both the data-fidelity and regularization terms is proposed by Patanavijit and Jitapunkul [37] (denoted as “LOR”).(2)
The robustL1 and L2 norms with BTV prior are proposed by Farsiu et al. [30] (denoted as “L1-BTV” and “L2-BTV”).(3)
The adaptive norm with BEP regularization is proposed by Zeng and Yang [27] (denoted as “BEP”).Table1 summarizes the parameter settings to guarantee a fair comparison of all experiments. The parameter settings are determined as follows:(1)
20 LR images are produced from a single HR image and simulate a rigid motion of the displacement of each image. Therefore, uniform distribution of random values for translations and rotations displaces the HR image.(2)
The random translation values vary between −2 and 2 pixels, while the values of the rotation angles shift randomly from −0.5° to 0.5°.(3)
The displaced HR images are blurred with the width of isotropic Gaussian PSFσ = 0.4 and subsampled with magnification factor r = 2.(4)
In the simulated LR sequence, an additive white Gaussian noise is provided. The standard deviation of an additive white Gaussian noise is assigned to 0.025.(5)
The maximum number of iterations is 50 for Lorentzian, BTV, and BEP priors.(6)
The regularization parameterλ for Lorentzian, BTV, and BEP priors is chosen as 0.0025.(7)
Setup image prior for the Lorentzian prior: the window size is set toh = 1.8 and the scale factor to α = 1.(8)
Setup image prior for the BTV prior: the window size is set top=2 and the scale factor to α = 0.6.(9)
Setup image prior for the BEP prior: the window size is set toq = 1, the scale factor to α = 1, and the threshold parameter to c = 0.2.Table 1
The parameter settings of the synthetic dataset in the experiments.
ParameterSetting valueNumber of LR images generated20 LR imagesLower and upper bound for translationsRandomly between −2 and 2 pixelsLower and upper bound for rotation anglesRandomly from −0.5  to 0.5Width of isotropic Gaussian PSFσ = 0.4Magnification factorr = 2Standard deviation of an additive white Gaussian noise0.025Maximum number of iterations50Setup image prior for Lorentzian prior:Regularization weightλ = 0.0025Window sizeh = 1.8Scale factorα = 1Setup image prior for BTV prior:Regularization weightλ = 0.0025Window sizep=2Scale factorα = 0.6Setup image prior for BEP prior:Regularization weightλ = 0.0025Window sizeq = 1Scale factorα = 1Threshold parameterc = 0.2All applied experiments are coded in MATLAB R2020a and run on a computer system with Intel(R) Core (TM) i7-3612QM CPU @ 2.10 GHz, 8 GB memory, 512 MB Radeon graphics card, 1 TB hard disk drive, and 64-bit Windows 7 Home Premium operating system. Table2 summarizes the system setup requirements to implement the comprehensive experiment.Table 2
The system setup requirements.
System requirementsHardwareProcessorIntel(R) Core (TM) i7-3612QM CPUClock speed2.10 GHzRAM capacity8 GBGraphics card capacity512 MBHard disk drive1 TBSoftwareOperating system64-Bit Windows 7 Home PremiumIDEMATLABVersionR2020a


### 6.2. Performance Analysis

To perform the processes of quantitative analysis, evaluation, and comparison for the reconstruction performance, we generate synthetic data and calculate the peak-signal-to-noise ratio (PSNR) indicator and the structural similarity (SSIM) indicator of our estimated HR image with regard to a ground-truth image [8, 26, 31, 38, 39]. Therefore, the PSNR measurement is calculated from the mean square error (MSE), where the MSE reflects the average error among the reconstructed SR image and the original HR image. By giving a reconstructed SR m × n image X^i,j and its original HR image Xi,j, MSE and PSNR are described as(9)MSE=1mn∑i=0m−1∑j=0n−1Xi,j−X^i,j2,(10)PSNR=20log10LMSE.The SSIM indicator is used to assess the symmetry among the recreated SR image and the original HR image. Luminance, contrast, and structural differences between two images are also considered in the SSIM indicator. The SSIM indicator is represented as(11)SSIMX,X^=2μXμX^+C12σXX^+C2μX2+μX^2+C1σX2+σX^2+C2,where μX and μX^ are the means. σX and σX^ are the standard deviations of the original HR images and the reconstructed SR images, respectively. σXX^ is the covariance of X and X^. C1 and C2 present the fixed values to guarantee a stable SSIM computation. The constants C1 = (k1L)2 and C2 = (k2L)2, where k1 and k2 have tiny-fixed values, while L is the pixel dynamic set (255 for 8-bit grayscale images). In particular, a greater value of PSNR shows a good image quality for the restored image. The SSIM value falls within the range (0, 1) so that the similarity is large when its value is closer to 1. Then, the effect of image reconstruction is better.


### 6.3. Experimental Results on Synthetic Data

This research paper uses two types of evaluation criteria, namely, subjective (qualitative or visual analysis) and objective (quantitative analysis) evaluations. On one hand, subjective evaluation is considered the most intuitive and clear approach. Therefore, this approach needs from people to know a few essential image features in advance before starting to analyze the reconstructed HR images. Consequently, the reconstruction results should be evaluated by the human eye to identify the best reconstructed HR images. On the other hand, objective evaluation is considered an approach intended to evaluate the results by calculating the correlation coefficient between the images. Normally, PSNR and SSIM criteria are used to evaluate the effect of SR image reconstruction. In general, if these two values are large, the reconstruction effect becomes better.This subsection presents and discusses the experimental results of the proposed SR approaches on the simulated data to show the SR reconstruction improvement result effectively. The first proposed method is denoted as MMW-BTV, while the second proposed method is denoted as MLRW-BTV. To guarantee a fair comparison, the image formation model is typically executed in equation (2) for all algorithms. After that, we employ the four selected multiframe SR approaches and the proposed SR approaches to rebuild the SR image from the degraded LR images as shown in Figures 6–13.Figure 6
Visual comparison for the generated HR images of the “Acen” image with various SR approaches. (a) Original HR image. (b) Single LR image. (c) LOR. (d)L1-BTV. (e) BEP. (f) L2-BTV. (g) MMW-BTV. (h) MLRW-BTV.
(a)(b)(c)(d)(e)(f)(g)(h)Figure 7
Visual comparison for the generated HR images of the “Cartap” image with various SR approaches. (a) Original HR image. (b) Single LR image. (c) LOR. (d)L1-BTV. (e) BEP. (f) L2-BTV. (g) MMW-BTV. (h) MLRW-BTV.
(a)(b)(c)(d)(e)(f)(g)(h)Figure 8
Visual comparison for the generated HR images of the “Foreman” image with various SR approaches. (a) Original HR image. (b) Single LR image. (c) LOR. (d)L1-BTV. (e) BEP. (f) L2-BTV. (g) MMW-BTV. (h) MLRW-BTV.
(a)(b)(c)(d)(e)(f)(g)(h)Figure 9
Visual comparison for the generated HR images of the “Text” image with various SR approaches. (a) Original HR image. (b) Single LR image. (c) LOR. (d)L1-BTV. (e) BEP. (f) L2-BTV. (g) MMW-BTV. (h) MLRW-BTV.
(a)(b)(c)(d)(e)(f)(g)(h)Figure 10
Visual comparison for the generated HR images of the “Brain” image with various SR approaches. (a) Original HR image. (b) Single LR image. (c) LOR. (d)L1-BTV. (e) BEP. (f) L2-BTV. (g) MMW-BTV. (h) MLRW-BTV.
(a)(b)(c)(d)(e)(f)(g)(h)Figure 11
Visual comparison for the generated HR images of the “License Plate” image with various SR approaches. (a) Original HR image. (b) Single LR image. (c) LOR. (d)L1-BTV. (e) BEP. (f) L2-BTV. (g) MMW-BTV. (h) MLRW-BTV.
(a)(b)(c)(d)(e)(f)(g)(h)Figure 12
Visual comparison for the generated HR images of the “Natural Scene” image with various SR approaches. (a) Original HR image. (b) Single LR image. (c) LOR. (d)L1-BTV. (e) BEP. (f) L2-BTV. (g) MMW-BTV. (h) MLRW-BTV.
(a)(b)(c)(d)(e)(f)(g)(h)Figure 13
Visual comparison for the generated HR images of the “Parrot” image with various SR approaches. (a) Original HR image. (b) Single LR image. (c) LOR. (d)L1-BTV. (e) BEP. (f) L2-BTV. (g) MMW-BTV. (h) MLRW-BTV.
(a)(b)(c)(d)(e)(f)(g)(h)Comparisons are provided to assess the performance of the proposed SR approaches through both visual and quantitative comparisons. Visual quality evaluations of the restored SR images for “Acen,” “Cartap,” “Foreman,” “Text,” “Brain,” “License Plate,” “Natural Scene,” and “Parrot” from Figure5 using the proposed SR approaches are illustrated in Figures 6–13, respectively, in comparison with several current state-of-the-art approaches. Figures 6–13 display the reconstruction results generated for each experiment. The explanations of these figures are described as follows: (a) shows the original HR image, (b) displays a single measured LR image, and ((c)–(f)) demonstrate the restored HR images with LOR, L1-BTV, BEP, and L2-BTV approaches, respectively. Here, (g) and (h) indicate the SR image of the two proposed approaches. Also, parts of these reconstructed images are enlarged for a perfect visualization by displaying them in the downmost right corner of every image.


### 6.4. Qualitative and Visual Analysis

The visual experiments in Figures6–13 represent the reconstruction result for HR images. A set of conclusions can be drawn based on Figures 6–13. Firstly, LOR and L1-BTV are inadequate to eliminate the noise, remove the blurring, and have a zigzagging effect in the reconstructed images as shown in Figures 6–13(c) and 13(d). For example, in the Foreman image as shown in Figures 8(c) and 8(d), a lot of noise remains in the background area as well as a lot of zigzagging at the edges. Secondly, L2-BTV is robust to suppress the white Gaussian noise and preserve the image edges as shown in Figures 6–13(f). This method achieves a result that is the most visible and close to the image quality of the reconstructed HR image in the proposed MMW-BTV approach. For example, in the Foreman image, as shown in Figure 8(f), the image is clear and has better reconstruction; however, there is a little staircase effect at the edges. Finally, the BEP method can adaptively choose error norm to suppress the Gaussian noise but some noise still exists due to its simple initial estimation as shown in Figures 6–13(e). For example, in the Foreman image as shown in Figure 8(e), some noises remain in the background area as well as some zigzagging at the edges.As a result, the proposed MMW-BTV and MLRW-BTV approaches obviously produce the restored HR image, which offers the best visual effects, as opposed to other SR methods (LOR,L1-BTV, BEP, and L2-BTV) and they better prevent the staircase effect. In comparison, image information such as sharp edges is retained, while the noise is suppressed. This improvement is due to the proposed MMW-BTV and MLRW-BTV approaches relying on estimating the initial HR image through preprocessing of the reference LR image based on the median, mean, Lucy-Richardson, and Wiener filters, because the reference LR image has a poor quality and contains some distortions such as blur and noise which lead to weakness in estimating the initial HR image. This preprocessing stage is used to overcome the degradation present in the reference LR image, which is a suitable kernel for producing the initial HR image to be used in the reconstruction phase of the final image. Moreover, median, mean, Wiener, and Lucy-Richardson filters are able to remove the noise from the image, smooth the image by reducing the amount of intensity variation between neighboring pixels, eliminate the blurring from the image, and smooth the image by reducing the amount of intensity variation between neighboring pixels, respectively. Therefore, the proposed MMW-BTV and MLRW-BTV are, finally, considerably clearer, more realistic, and closer to the image of the ground reality.


### 6.5. Quantitative Analysis

Objective quality measurement is an alternative of a tedious and time-consuming subjective assessment. In literature, there are a plenty of metrics that model or not the human visual system (HVS). Most of them are not very popular due to their complexity, difficult calibration، or lack of freely available implementation. This is why metrics like PSNR and SSIM are widely used to compare algorithms [3, 10, 11, 13, 14, 28, 32, 39–42].A computational analysis is required to support the visual appearance. For this reason, the quantitative quality estimation of the generated HR images is performed by the PSNR indicator and the SSIM criterion. The PSNR index is used to assess the generated HR image quality, while the SSIM index is a supplementary measurement that indicates that the image quality is dependent on the recognized features of the HVS. Tables3 and 4 show the computed values of the PSNR and SSIM metrics, respectively, for the eight examined images in Figure 5. PSNR and SSIM have the best value in bold numbers in each row.Table 3
A numerical comparison of PSNR for various SR approaches
ImageLORL1-BTVBEPL2-BTVMMW-BTVMLRW-BTVAcen26.647826.979227.634731.471331.683131.6887Cartap24.061224.943924.724529.400229.594529.5671Foreman31.270330.040430.991533.801633.918133.8741Text25.500525.689525.372728.368128.454428.3840Brain28.607328.451729.193530.542830.594530.6008License Plate29.224930.008630.476832.481132.774632.7314Natural Scene26.506226.349726.322728.114328.116928.1477Parrot30.268330.504130.528834.007934.191634.1212Average PSNR27.7608127.8708928.1556531.0234131.1659631.13938The bold numbers indicate the highest values of the proposed methods.Table 4
A numerical comparison of SSIM for various SR approaches.
ImageLORL1-BTVBEPL2-BTVMMW-BTVMLRW-BTVAcen0.810200.826410.850830.968820.968880.96913Cartap0.884170.857840.905700.979880.982200.98219Foreman0.885330.801840.855530.935800.937350.93650Text0.744710.778990.809220.963170.965930.96530Brain0.897430.866880.895330.933070.933470.93342License Plate0.814350.842670.867510.955100.956500.95640Natural Scene0.885670.876460.877950.949850.949870.94991Parrot0.916950.922550.926690.976740.977460.97731Average SSIM0.8548510.8467050.8735950.9578040.9589580.95877The bold numbers indicate the highest values of the proposed methods.From quantitative quality estimation, Table3 presents the proposed approaches with the best PSNR values that confirm their effectiveness against blurring and noise reduction. Furthermore, the values of SSIM in Table 4 demonstrate the high image quality in terms of image preservation information. Based on the data in the above tables (Tables 3 and 4), the following conclusions can be drawn:(1)
In terms of PSNR and SSIM, the proposed MMW-BTV and MLRW-BTV approaches are better than other SR methods for image reconstruction. They allow the reconstructed images to be more similar to the original images with good deblurring and denoising performance.(2)
In comparison to the BEP method, SSIM measures for the proposed MMW-BTV approach are higher by 0.03814–0.15671. For example, the SSIM value of the proposed MMW-BTV approach (0.96593) is 0.15671 higher than that of the BEP method (0.80922) for the image of Text.(3)
The PSNR and SSIM values of the proposed MLRW-BTV approach are higher than those of the BEP method by 1.4073–4.8426 dB and 0.03809–0.15608, respectively. For example, PSNR of the proposed MLRW-BTV approach (30.6008 dB) is 1.4073 dB higher than that of the BEP method (29.1935 dB) for the image of Brain, and the SSIM difference becomes 0.08097 for the image of Foreman (0.9365 versus 0.85553).(4)
The proposed MMW-BTV approach has SSIM values that are 0.00002–0.00276 higher than theL2-BTV method. For the Foreman image, SSIM of the proposed MMW-BTV approach (0.93735) is 0.0016 higher than that of the L2-BTV method (0.93580).(5)
The proposed MLRW-BTV approach has PSNR and SSIM values that are 0.0159–0.2503 dB and 0.0001–0.0023, respectively, higher than those of theL2-BTV method. For the Text image, PSNR of the proposed MLRW-BTV approach (28.384 dB) is 0.0159 dB higher than that of the L2-BTV method (28.3681 dB). Meanwhile, the SSIM difference becomes 0.0021 (0.9653 versus 0.96317).As a result, the proposed MMW-BTV and MLRW-BTV approaches always have better results as compared to other methods (LOR,L1-BTV, BEP, and L2-BTV) which guarantee their efficiency, because they estimate the initial HR image well through preprocessing of the reference LR image in order to remove the noise, smooth the image, and eliminate the blurring.


### 6.6. Computational Complexity Analysis

In this subsection, we conduct a detailed evaluation of the computational complexity of our proposed SR approaches (MMW-BTV and MLRW-BTV) and the compared SR approaches (LOR,L1-BTV, BEP, and L2-BTV) in terms of run-time. The run-time evaluation is performed using the parameter settings and system setup requirements previously explained in Tables 2 and 3, respectively. The run-time complexity of our proposed SR approaches and the compared SR approaches is shown in Table 5. The proposed SR approaches are less computationally complex in comparison with the state-of-the-art SR approaches such as L1-BTV and BEP. The BEP approach has the highest computational complexity, while the LOR and L2-BTV approaches have lower computational complexity than our proposed SR approaches. However, the quality of our super-resolved images is much better than theirs in terms of both visual evaluation and PSNR/SSIM measurements as shown in Tables 3 and 4.Table 5
CPU times (in seconds) of SR approaches when the magnification factor is 2.
ImageLORL1-BTVBEPL2-BTVMMW-BTVMLRW-BTVAcen4.06847.815712.29294.27445.33525.4912Cartap3.81889.578512.13693.40084.94525.1324Foreman2.54288.54898.78293.71284.14964.3992Text4.46009.422510.88894.22764.25886.8796Brain3.338410.155712.35533.74404.83604.9296License Plate8.389714.882520.29578.03419.01695.7564Natural Scene13.592125.178623.946211.294512.074511.9341Parrot12.894925.709016.86379.937310.826510.8109Average time6.638113.911414.69536.07826.93036.9167


## 6.1. Experimental Setup

In this subsection, different simulated results are presented to estimate the efficiency of the proposed SR approaches. The proposed SR approaches are evaluated on a large benchmark database on the Internet [15] consisting of photographs of different scenes. We select eight images from a benchmark database of various styles to generate our simulated LR sequences as elaborated in Figure 5. Figures 5(a)–5(f) illustrate the original grayscale images for Acen, Cartap, Foreman, Text, and Brain with the size of 120 × 120 and License Plate image with the size of 250 × 100. Moreover, the original RGB images for Natural Scene and Parrot with the size of 120 × 120 are shown in Figures 5(g)–5(h). All experiments are performed directly on grayscale images and RGB channels for color images. These eight original HR images are very different from each other. Accordingly, the main reasons for choosing these images are their content, their availability for comparisons, and them being realistic, and these images have been frequently used by most recent works related to [3, 34–36]. Additionally, they are very different from each other in the following characteristics: complex structure, amount of irregular lines, texture features, gray level histogram, contrast, and smoothness.Figure 5
A sequence of benchmark images are used in experimental tests as an original image. (a) Acen. (b) Cartap. (c) Foreman. (d) Text. (e) Brain. (f) License Plate. (g) Natural Scene. (h) Parrot.
(a)(b)(c)(d)(e)(f)(g)(h)In practical terms, the performance of SR approaches cannot be measured arithmetically, because the HR images of the ground truth are still unavailable. In all experiments, the proposed approaches are initially examined on synthetic data, because the ground-truth images are available for synthetic LR images. Throughout all experiments, the same data and parameter settings are used to compare the proposed SR approaches to the four following state-of-the-art SR approaches:(1)
An SR method with the Lorentzian error norm in both the data-fidelity and regularization terms is proposed by Patanavijit and Jitapunkul [37] (denoted as “LOR”).(2)
The robustL1 and L2 norms with BTV prior are proposed by Farsiu et al. [30] (denoted as “L1-BTV” and “L2-BTV”).(3)
The adaptive norm with BEP regularization is proposed by Zeng and Yang [27] (denoted as “BEP”).Table1 summarizes the parameter settings to guarantee a fair comparison of all experiments. The parameter settings are determined as follows:(1)
20 LR images are produced from a single HR image and simulate a rigid motion of the displacement of each image. Therefore, uniform distribution of random values for translations and rotations displaces the HR image.(2)
The random translation values vary between −2 and 2 pixels, while the values of the rotation angles shift randomly from −0.5° to 0.5°.(3)
The displaced HR images are blurred with the width of isotropic Gaussian PSFσ = 0.4 and subsampled with magnification factor r = 2.(4)
In the simulated LR sequence, an additive white Gaussian noise is provided. The standard deviation of an additive white Gaussian noise is assigned to 0.025.(5)
The maximum number of iterations is 50 for Lorentzian, BTV, and BEP priors.(6)
The regularization parameterλ for Lorentzian, BTV, and BEP priors is chosen as 0.0025.(7)
Setup image prior for the Lorentzian prior: the window size is set toh = 1.8 and the scale factor to α = 1.(8)
Setup image prior for the BTV prior: the window size is set top=2 and the scale factor to α = 0.6.(9)
Setup image prior for the BEP prior: the window size is set toq = 1, the scale factor to α = 1, and the threshold parameter to c = 0.2.Table 1
The parameter settings of the synthetic dataset in the experiments.
ParameterSetting valueNumber of LR images generated20 LR imagesLower and upper bound for translationsRandomly between −2 and 2 pixelsLower and upper bound for rotation anglesRandomly from −0.5  to 0.5Width of isotropic Gaussian PSFσ = 0.4Magnification factorr = 2Standard deviation of an additive white Gaussian noise0.025Maximum number of iterations50Setup image prior for Lorentzian prior:Regularization weightλ = 0.0025Window sizeh = 1.8Scale factorα = 1Setup image prior for BTV prior:Regularization weightλ = 0.0025Window sizep=2Scale factorα = 0.6Setup image prior for BEP prior:Regularization weightλ = 0.0025Window sizeq = 1Scale factorα = 1Threshold parameterc = 0.2All applied experiments are coded in MATLAB R2020a and run on a computer system with Intel(R) Core (TM) i7-3612QM CPU @ 2.10 GHz, 8 GB memory, 512 MB Radeon graphics card, 1 TB hard disk drive, and 64-bit Windows 7 Home Premium operating system. Table2 summarizes the system setup requirements to implement the comprehensive experiment.Table 2
The system setup requirements.
System requirementsHardwareProcessorIntel(R) Core (TM) i7-3612QM CPUClock speed2.10 GHzRAM capacity8 GBGraphics card capacity512 MBHard disk drive1 TBSoftwareOperating system64-Bit Windows 7 Home PremiumIDEMATLABVersionR2020a


## 6.2. Performance Analysis

To perform the processes of quantitative analysis, evaluation, and comparison for the reconstruction performance, we generate synthetic data and calculate the peak-signal-to-noise ratio (PSNR) indicator and the structural similarity (SSIM) indicator of our estimated HR image with regard to a ground-truth image [8, 26, 31, 38, 39]. Therefore, the PSNR measurement is calculated from the mean square error (MSE), where the MSE reflects the average error among the reconstructed SR image and the original HR image. By giving a reconstructed SR m × n image X^i,j and its original HR image Xi,j, MSE and PSNR are described as(9)MSE=1mn∑i=0m−1∑j=0n−1Xi,j−X^i,j2,(10)PSNR=20log10LMSE.The SSIM indicator is used to assess the symmetry among the recreated SR image and the original HR image. Luminance, contrast, and structural differences between two images are also considered in the SSIM indicator. The SSIM indicator is represented as(11)SSIMX,X^=2μXμX^+C12σXX^+C2μX2+μX^2+C1σX2+σX^2+C2,where μX and μX^ are the means. σX and σX^ are the standard deviations of the original HR images and the reconstructed SR images, respectively. σXX^ is the covariance of X and X^. C1 and C2 present the fixed values to guarantee a stable SSIM computation. The constants C1 = (k1L)2 and C2 = (k2L)2, where k1 and k2 have tiny-fixed values, while L is the pixel dynamic set (255 for 8-bit grayscale images). In particular, a greater value of PSNR shows a good image quality for the restored image. The SSIM value falls within the range (0, 1) so that the similarity is large when its value is closer to 1. Then, the effect of image reconstruction is better.


## 6.3. Experimental Results on Synthetic Data

This research paper uses two types of evaluation criteria, namely, subjective (qualitative or visual analysis) and objective (quantitative analysis) evaluations. On one hand, subjective evaluation is considered the most intuitive and clear approach. Therefore, this approach needs from people to know a few essential image features in advance before starting to analyze the reconstructed HR images. Consequently, the reconstruction results should be evaluated by the human eye to identify the best reconstructed HR images. On the other hand, objective evaluation is considered an approach intended to evaluate the results by calculating the correlation coefficient between the images. Normally, PSNR and SSIM criteria are used to evaluate the effect of SR image reconstruction. In general, if these two values are large, the reconstruction effect becomes better.This subsection presents and discusses the experimental results of the proposed SR approaches on the simulated data to show the SR reconstruction improvement result effectively. The first proposed method is denoted as MMW-BTV, while the second proposed method is denoted as MLRW-BTV. To guarantee a fair comparison, the image formation model is typically executed in equation (2) for all algorithms. After that, we employ the four selected multiframe SR approaches and the proposed SR approaches to rebuild the SR image from the degraded LR images as shown in Figures 6–13.Figure 6
Visual comparison for the generated HR images of the “Acen” image with various SR approaches. (a) Original HR image. (b) Single LR image. (c) LOR. (d)L1-BTV. (e) BEP. (f) L2-BTV. (g) MMW-BTV. (h) MLRW-BTV.
(a)(b)(c)(d)(e)(f)(g)(h)Figure 7
Visual comparison for the generated HR images of the “Cartap” image with various SR approaches. (a) Original HR image. (b) Single LR image. (c) LOR. (d)L1-BTV. (e) BEP. (f) L2-BTV. (g) MMW-BTV. (h) MLRW-BTV.
(a)(b)(c)(d)(e)(f)(g)(h)Figure 8
Visual comparison for the generated HR images of the “Foreman” image with various SR approaches. (a) Original HR image. (b) Single LR image. (c) LOR. (d)L1-BTV. (e) BEP. (f) L2-BTV. (g) MMW-BTV. (h) MLRW-BTV.
(a)(b)(c)(d)(e)(f)(g)(h)Figure 9
Visual comparison for the generated HR images of the “Text” image with various SR approaches. (a) Original HR image. (b) Single LR image. (c) LOR. (d)L1-BTV. (e) BEP. (f) L2-BTV. (g) MMW-BTV. (h) MLRW-BTV.
(a)(b)(c)(d)(e)(f)(g)(h)Figure 10
Visual comparison for the generated HR images of the “Brain” image with various SR approaches. (a) Original HR image. (b) Single LR image. (c) LOR. (d)L1-BTV. (e) BEP. (f) L2-BTV. (g) MMW-BTV. (h) MLRW-BTV.
(a)(b)(c)(d)(e)(f)(g)(h)Figure 11
Visual comparison for the generated HR images of the “License Plate” image with various SR approaches. (a) Original HR image. (b) Single LR image. (c) LOR. (d)L1-BTV. (e) BEP. (f) L2-BTV. (g) MMW-BTV. (h) MLRW-BTV.
(a)(b)(c)(d)(e)(f)(g)(h)Figure 12
Visual comparison for the generated HR images of the “Natural Scene” image with various SR approaches. (a) Original HR image. (b) Single LR image. (c) LOR. (d)L1-BTV. (e) BEP. (f) L2-BTV. (g) MMW-BTV. (h) MLRW-BTV.
(a)(b)(c)(d)(e)(f)(g)(h)Figure 13
Visual comparison for the generated HR images of the “Parrot” image with various SR approaches. (a) Original HR image. (b) Single LR image. (c) LOR. (d)L1-BTV. (e) BEP. (f) L2-BTV. (g) MMW-BTV. (h) MLRW-BTV.
(a)(b)(c)(d)(e)(f)(g)(h)Comparisons are provided to assess the performance of the proposed SR approaches through both visual and quantitative comparisons. Visual quality evaluations of the restored SR images for “Acen,” “Cartap,” “Foreman,” “Text,” “Brain,” “License Plate,” “Natural Scene,” and “Parrot” from Figure5 using the proposed SR approaches are illustrated in Figures 6–13, respectively, in comparison with several current state-of-the-art approaches. Figures 6–13 display the reconstruction results generated for each experiment. The explanations of these figures are described as follows: (a) shows the original HR image, (b) displays a single measured LR image, and ((c)–(f)) demonstrate the restored HR images with LOR, L1-BTV, BEP, and L2-BTV approaches, respectively. Here, (g) and (h) indicate the SR image of the two proposed approaches. Also, parts of these reconstructed images are enlarged for a perfect visualization by displaying them in the downmost right corner of every image.


## 6.4. Qualitative and Visual Analysis

The visual experiments in Figures6–13 represent the reconstruction result for HR images. A set of conclusions can be drawn based on Figures 6–13. Firstly, LOR and L1-BTV are inadequate to eliminate the noise, remove the blurring, and have a zigzagging effect in the reconstructed images as shown in Figures 6–13(c) and 13(d). For example, in the Foreman image as shown in Figures 8(c) and 8(d), a lot of noise remains in the background area as well as a lot of zigzagging at the edges. Secondly, L2-BTV is robust to suppress the white Gaussian noise and preserve the image edges as shown in Figures 6–13(f). This method achieves a result that is the most visible and close to the image quality of the reconstructed HR image in the proposed MMW-BTV approach. For example, in the Foreman image, as shown in Figure 8(f), the image is clear and has better reconstruction; however, there is a little staircase effect at the edges. Finally, the BEP method can adaptively choose error norm to suppress the Gaussian noise but some noise still exists due to its simple initial estimation as shown in Figures 6–13(e). For example, in the Foreman image as shown in Figure 8(e), some noises remain in the background area as well as some zigzagging at the edges.As a result, the proposed MMW-BTV and MLRW-BTV approaches obviously produce the restored HR image, which offers the best visual effects, as opposed to other SR methods (LOR,L1-BTV, BEP, and L2-BTV) and they better prevent the staircase effect. In comparison, image information such as sharp edges is retained, while the noise is suppressed. This improvement is due to the proposed MMW-BTV and MLRW-BTV approaches relying on estimating the initial HR image through preprocessing of the reference LR image based on the median, mean, Lucy-Richardson, and Wiener filters, because the reference LR image has a poor quality and contains some distortions such as blur and noise which lead to weakness in estimating the initial HR image. This preprocessing stage is used to overcome the degradation present in the reference LR image, which is a suitable kernel for producing the initial HR image to be used in the reconstruction phase of the final image. Moreover, median, mean, Wiener, and Lucy-Richardson filters are able to remove the noise from the image, smooth the image by reducing the amount of intensity variation between neighboring pixels, eliminate the blurring from the image, and smooth the image by reducing the amount of intensity variation between neighboring pixels, respectively. Therefore, the proposed MMW-BTV and MLRW-BTV are, finally, considerably clearer, more realistic, and closer to the image of the ground reality.


## 6.5. Quantitative Analysis

Objective quality measurement is an alternative of a tedious and time-consuming subjective assessment. In literature, there are a plenty of metrics that model or not the human visual system (HVS). Most of them are not very popular due to their complexity, difficult calibration، or lack of freely available implementation. This is why metrics like PSNR and SSIM are widely used to compare algorithms [3, 10, 11, 13, 14, 28, 32, 39–42].A computational analysis is required to support the visual appearance. For this reason, the quantitative quality estimation of the generated HR images is performed by the PSNR indicator and the SSIM criterion. The PSNR index is used to assess the generated HR image quality, while the SSIM index is a supplementary measurement that indicates that the image quality is dependent on the recognized features of the HVS. Tables3 and 4 show the computed values of the PSNR and SSIM metrics, respectively, for the eight examined images in Figure 5. PSNR and SSIM have the best value in bold numbers in each row.Table 3
A numerical comparison of PSNR for various SR approaches
ImageLORL1-BTVBEPL2-BTVMMW-BTVMLRW-BTVAcen26.647826.979227.634731.471331.683131.6887Cartap24.061224.943924.724529.400229.594529.5671Foreman31.270330.040430.991533.801633.918133.8741Text25.500525.689525.372728.368128.454428.3840Brain28.607328.451729.193530.542830.594530.6008License Plate29.224930.008630.476832.481132.774632.7314Natural Scene26.506226.349726.322728.114328.116928.1477Parrot30.268330.504130.528834.007934.191634.1212Average PSNR27.7608127.8708928.1556531.0234131.1659631.13938The bold numbers indicate the highest values of the proposed methods.Table 4
A numerical comparison of SSIM for various SR approaches.
ImageLORL1-BTVBEPL2-BTVMMW-BTVMLRW-BTVAcen0.810200.826410.850830.968820.968880.96913Cartap0.884170.857840.905700.979880.982200.98219Foreman0.885330.801840.855530.935800.937350.93650Text0.744710.778990.809220.963170.965930.96530Brain0.897430.866880.895330.933070.933470.93342License Plate0.814350.842670.867510.955100.956500.95640Natural Scene0.885670.876460.877950.949850.949870.94991Parrot0.916950.922550.926690.976740.977460.97731Average SSIM0.8548510.8467050.8735950.9578040.9589580.95877The bold numbers indicate the highest values of the proposed methods.From quantitative quality estimation, Table3 presents the proposed approaches with the best PSNR values that confirm their effectiveness against blurring and noise reduction. Furthermore, the values of SSIM in Table 4 demonstrate the high image quality in terms of image preservation information. Based on the data in the above tables (Tables 3 and 4), the following conclusions can be drawn:(1)
In terms of PSNR and SSIM, the proposed MMW-BTV and MLRW-BTV approaches are better than other SR methods for image reconstruction. They allow the reconstructed images to be more similar to the original images with good deblurring and denoising performance.(2)
In comparison to the BEP method, SSIM measures for the proposed MMW-BTV approach are higher by 0.03814–0.15671. For example, the SSIM value of the proposed MMW-BTV approach (0.96593) is 0.15671 higher than that of the BEP method (0.80922) for the image of Text.(3)
The PSNR and SSIM values of the proposed MLRW-BTV approach are higher than those of the BEP method by 1.4073–4.8426 dB and 0.03809–0.15608, respectively. For example, PSNR of the proposed MLRW-BTV approach (30.6008 dB) is 1.4073 dB higher than that of the BEP method (29.1935 dB) for the image of Brain, and the SSIM difference becomes 0.08097 for the image of Foreman (0.9365 versus 0.85553).(4)
The proposed MMW-BTV approach has SSIM values that are 0.00002–0.00276 higher than theL2-BTV method. For the Foreman image, SSIM of the proposed MMW-BTV approach (0.93735) is 0.0016 higher than that of the L2-BTV method (0.93580).(5)
The proposed MLRW-BTV approach has PSNR and SSIM values that are 0.0159–0.2503 dB and 0.0001–0.0023, respectively, higher than those of theL2-BTV method. For the Text image, PSNR of the proposed MLRW-BTV approach (28.384 dB) is 0.0159 dB higher than that of the L2-BTV method (28.3681 dB). Meanwhile, the SSIM difference becomes 0.0021 (0.9653 versus 0.96317).As a result, the proposed MMW-BTV and MLRW-BTV approaches always have better results as compared to other methods (LOR,L1-BTV, BEP, and L2-BTV) which guarantee their efficiency, because they estimate the initial HR image well through preprocessing of the reference LR image in order to remove the noise, smooth the image, and eliminate the blurring.


## 6.6. Computational Complexity Analysis

In this subsection, we conduct a detailed evaluation of the computational complexity of our proposed SR approaches (MMW-BTV and MLRW-BTV) and the compared SR approaches (LOR,L1-BTV, BEP, and L2-BTV) in terms of run-time. The run-time evaluation is performed using the parameter settings and system setup requirements previously explained in Tables 2 and 3, respectively. The run-time complexity of our proposed SR approaches and the compared SR approaches is shown in Table 5. The proposed SR approaches are less computationally complex in comparison with the state-of-the-art SR approaches such as L1-BTV and BEP. The BEP approach has the highest computational complexity, while the LOR and L2-BTV approaches have lower computational complexity than our proposed SR approaches. However, the quality of our super-resolved images is much better than theirs in terms of both visual evaluation and PSNR/SSIM measurements as shown in Tables 3 and 4.Table 5
CPU times (in seconds) of SR approaches when the magnification factor is 2.
ImageLORL1-BTVBEPL2-BTVMMW-BTVMLRW-BTVAcen4.06847.815712.29294.27445.33525.4912Cartap3.81889.578512.13693.40084.94525.1324Foreman2.54288.54898.78293.71284.14964.3992Text4.46009.422510.88894.22764.25886.8796Brain3.338410.155712.35533.74404.83604.9296License Plate8.389714.882520.29578.03419.01695.7564Natural Scene13.592125.178623.946211.294512.074511.9341Parrot12.894925.709016.86379.937310.826510.8109Average time6.638113.911414.69536.07826.93036.9167


## 7. Conclusion

In this paper, we propose new multiframe image SR approaches that firstly rely on estimating the initial HR image through preprocessing of the reference LR image based on median, mean, Lucy-Richardson, and Wiener filters. Then, theL2 norm is employed for the data-fidelity term and BTV prior model is utilized to the regularization term. The proposed multiframe SR approaches are used for increasing the components of high frequency and preventing image degradation produced by the imaging systems. Also, they ensure a good balance among the preservation of edge and the suppression of noise and adaptively enhance the SR reconstruction process. The efficiency of the proposed SR approaches is measured for synthetic images and they are comparable to several related state-of-the-art approaches. Extensive experiments have been conducted and the experimental results illustrate that our proposed approaches increase the visual efficiency of state-of-the-art algorithms. Furthermore, we illustrate that the proposed approaches always generate the best PSNR and SSIM values quantitatively. However, the high computational cost of the proposed approaches remains a challenging point.



---

*Source: 8309910-2021-12-18.xml*