# Image Superresolution Based on Locally Adaptive Mixed-Norm

Authors: Osama A. Omer; Toshihisa Tanaka
Journal: Journal of Electrical and Computer Engineering
 (2010)

---

## Abstract

In a typical superresolution algorithm, fusion error modeling, including registration error and additive noise, has a great influence on the performance of the super-resolution algorithms. In this letter, we show that the quality of the reconstructed high-resolution image can be increased by exploiting proper model for the fusion error. To properly model the fusion error, we propose to minimize a cost function that consists of locally and adaptively weightedL1- and L2-norms considering the error model. Binary weights are used so as to adaptively select L1- or L2-norm, based on the local errors. Simulation results demonstrate that proposed algorithm can overcome disadvantages of using either L1- or L2-norm.


---

## Body

## 1. Introduction

Super-resolution (SR) is an approach to obtain high-resolution (HR) image(s) from a set of low-resolution (LR) images. In recent SR algorithms [1, 2] the SR results depend on model of the fusion error, including the registration error and contaminating noise. As a cost function, the L2- or L1-norm is used to fuse LR images [1, 2]. The choice of norm depends on a model of fusion error: whether the error is Gaussian (L2-norm [1]) or Laplacian (L1-norm [2]).UtilizingL2-norm to solve the super-resolution problem [1] can yield optimal solution when the fusion error has a Gaussian distribution. Although the additive noise in the image generation process can be modeled by Gaussian distribution in many applications [1], the fusion error consisting of a combination of the additive noise and the registration error may be characterized by other distributions. In [2], it has been shown that registration error is proper to be characterized by Laplacian distribution more than Gaussian. However, the assumption of Laplacian distribution for regions that do not suffer from registration error is not always proper assumption.For these reasons, we propose a mixed-norm-based SR algorithm by minimizing a cost function consisting ofL1- and L2-norm with adaptive mixing parameters. Since the registration error is space variant, the mixing parameters are chosen based on local error. In addition, the regularization parameter is adaptively estimated.


## 2. Problem Description

Assume thatK LR images of the same scene in Lexicographical order denoted by Yk (1≤k≤K), each containing M2 pixels, are observed, and they are generated from the HR image denoted by X, containing L2 pixels, where L≥M. The observation of K LR images is modeled by the following degradation process [1, 2]:(1)Yk=DHFkX+Vk,
where Fk, H, and D are matrices that represent the motion, the blurring, and the downsampling operators, respectively, X is the unknown HR image,  and Vk is a vector with the same size as X representing the additive noise for the kth image. The size of Fk, H, and D is L2×L2, L2×L2, and M2×L2, respectively. Throughout the letter, we assume that D and H are known and the additive noise has Gaussian distribution.Due to the fact that motion operator is not known,Fk should be estimated from the successive LR frames. This motion estimation is not always accurate; then the so-estimated motion operator can be described as F̂k=Fk+ΔFk, where ΔFk is the registration error matrix. Therefore, utilizing (1), the simulated kth LR image can be described as(2)Ŷk=DHFkX+DHΔFkX+Vk=DHFkX+V̂k,
where V̂k is the combination of additive noise and registration error, which represents the noise to be suppressed in the SR process.


## 3. SR Based on Mixed-Norm

As the Gaussian and Laplacian distributions are the two major candidates for modeling of this mixed noise, we propose to use adaptive mixedL1- and L2-norms where L1- and L2-norms are locally and adaptively weighted, based on the local error.


### 3.1. Proposed Cost Function

DefineEk=DHFkX-Yk. Let ζ1k and ζ2k be diagonal matrices of size M2×M2 with entries 0 or 1 such that ζ1k+ζ2k=I. The proposed algorithm is based on the minimization of(3)J(X,λ(X))=∑k=1K[w∥Ek∥1,ζ1k+∥Ek∥2,ζ2k2]+λ(X)R(X)=∑k=1K∑i=1M2{wζ1k(i,i)|Ek(i)|+ζ2k(i,i)|Ek(i)|2}+λ(X)R(X).
Matrices ζ1k and ζ2k represent the mixing parameters that select between L1- and L2-norms, R(X) is the regularization function, λ(X) is the regularization parameter, and w is a scalar that is used to adjust the convergence rate and the regularization parameter for regions that dominated by L1-norm. When ζ1k=I, (3) reverts to the L1-norm of the error, whereas when ζ1k=0, (3) reverts to the L2-norm of the error. The adaptive adjustment of ζ1 thereby provides an algorithm that adaptively changes between the L1-norm, L2-norm or an intermediate performance between that of L1 and L2-norms.


### 3.2. Choice ofζ1k and ζ2k

The choice ofζ1k and ζ2k is a critical issue in the proposed algorithm. Because of the efficiency of erf to provide an automatic switching parameter, as proved experimentally in [3], we propose to adopt erf in the mixing matrix. Therefore, the (i,i)th element of the mixing parameter, ζ1k, for frame k can be described as(4)ζ1k(i,i)=erf{|E0k(i)|σ̂E0k},
where E0 is the error at the initial estimate of the HR image, and σ̂E0 is the standard deviation for of the error at frame k. ζ1k can be used as a binary version of (4) by thresholding the erf value by certain value threshold (θ), so that ζ1k(i,i)=1 if the erf value is greater than that threshold and equals zero otherwise. With this choice of the mixing parameters, ζ2k will be very small (zero in case of using binary values) at locations that have high error which make the algorithm jump over these locations by updating the HR image with the use of the sign of these errors. Both of ζ1k and ζ2k can be adapted based on the partially estimated HR image after each iteration. In the rest of this letter, we will use ζ1k as binary values.


## 3.1. Proposed Cost Function

DefineEk=DHFkX-Yk. Let ζ1k and ζ2k be diagonal matrices of size M2×M2 with entries 0 or 1 such that ζ1k+ζ2k=I. The proposed algorithm is based on the minimization of(3)J(X,λ(X))=∑k=1K[w∥Ek∥1,ζ1k+∥Ek∥2,ζ2k2]+λ(X)R(X)=∑k=1K∑i=1M2{wζ1k(i,i)|Ek(i)|+ζ2k(i,i)|Ek(i)|2}+λ(X)R(X).
Matrices ζ1k and ζ2k represent the mixing parameters that select between L1- and L2-norms, R(X) is the regularization function, λ(X) is the regularization parameter, and w is a scalar that is used to adjust the convergence rate and the regularization parameter for regions that dominated by L1-norm. When ζ1k=I, (3) reverts to the L1-norm of the error, whereas when ζ1k=0, (3) reverts to the L2-norm of the error. The adaptive adjustment of ζ1 thereby provides an algorithm that adaptively changes between the L1-norm, L2-norm or an intermediate performance between that of L1 and L2-norms.


## 3.2. Choice ofζ1k and ζ2k

The choice ofζ1k and ζ2k is a critical issue in the proposed algorithm. Because of the efficiency of erf to provide an automatic switching parameter, as proved experimentally in [3], we propose to adopt erf in the mixing matrix. Therefore, the (i,i)th element of the mixing parameter, ζ1k, for frame k can be described as(4)ζ1k(i,i)=erf{|E0k(i)|σ̂E0k},
where E0 is the error at the initial estimate of the HR image, and σ̂E0 is the standard deviation for of the error at frame k. ζ1k can be used as a binary version of (4) by thresholding the erf value by certain value threshold (θ), so that ζ1k(i,i)=1 if the erf value is greater than that threshold and equals zero otherwise. With this choice of the mixing parameters, ζ2k will be very small (zero in case of using binary values) at locations that have high error which make the algorithm jump over these locations by updating the HR image with the use of the sign of these errors. Both of ζ1k and ζ2k can be adapted based on the partially estimated HR image after each iteration. In the rest of this letter, we will use ζ1k as binary values.


## 4. Choice ofλ(X)

The regularization parameter,λ, adjusts the relative importance between the data fidelity and the smoothness of the solution. Using a relative high value for λ may lead to very smooth HR image which deviates from the optimal solution of the inverse problem. Also, using small value for λ may lead to noisy HR image which also deviate from the optimal solution. Therefore, λ is required to be adaptively estimated. Following an approach similar to [4], the regularization parameter is estimated as a function of the HR image as(5)λ(X)=f(J(X,λ(X))),
where f(·) is a monotonically increasing function. Using λ(X) as a simple linear function of J(X,λ(X)) as(6)λ(X)=ξ[J(X,λ(X))],=ξ[∑k=1K[w∥Ek∥1,ζ1k+∥Ek∥2,ζ2k2]+λ(X)R(X)],
then,(7)λ(X)=∑k=1K[w∥Ek∥1,ζ1k+∥Ek∥2,ζ2k2]1/ξ-R(X),ξ is chosen so as to preserve the convexity of the cost function, J(X,λ(X)). The sufficient condition for the convexity of J(X,λ(X)) is given as follows.Proposition 1.
The cost function,J(X,λ(X)), is convex under condition
(8)1ξ≥R(X).
The proof of this proposition is given in the appendix.


## 5. Simulation Results and Discussion

### 5.1. Experiment Setup

To demonstrate the efficiency of the proposed algorithm, the simulation results of Car sequence [5] is included in this section. This sequence consists of 64 LR images and contains a moving object while the camera is stable.In the experiments, motion operators are estimated from the LR images by using gradient-based registration technique based on affine motion model [6] and the blurring operator is assumed to be Gaussian with kernel size equals 5×5 with variance equals 2. The steepest descent optimization is used for all the compared algorithms with maximum number of iterations equals 100, step size equals 0.1 (2 in case of L1-norm-based algorithm), and w is set to 20. For more fair comparison, the bilateral total variation (BTV) [2] is used as a regularization function for all the compared algorithms, that is,(9)R(X)=∑l=-PP∑m=-PPα|l|+|m|∥X-SxlSymX∥1,
where Sym is the shifting operator in direction y with m pixels, P is a factor that determines the smoothness support size and α is the bilateral parameter, 0<α≤1. P and α are set to 2 and 0.7, respectively, and θ is set to 0.7. For all the experiments, 1/ξ in (6) is set to 2∥Z∥1, where Z is the bilinearly interpolated reference frame.


### 5.2. Results

The results of the Car sequence are shown in Figure1. The 1st LR image and the corresponding mixing parameter, ζ11, are shown in Figures 1(a) and 1(b), respectively. Two regions of interest are shown by red rectangles in Figures 1(a) and 1(b). Zoom of the regions of interest of the resulting HR image using L2-norm-based algorithm [1] is shown in Figures 1(c) and 2(a). From these figures, we can see that using L2-norm can be good selection in some regions which is dominated by Gaussian noise as in the smooth regions and the Car's plate in Figure 1(c). However, in regions that dominated by Laplacian noise (due to high registration error), using L2-norm can result in deformation in the HR image as shown in Figure 2(a). On contrary, zoomed regions of interest of the resulting HR image using L1-norm-based algorithm [2] are shown in Figures 1(d) and 2(b). From these figures, we can see that using L1-norm is robust against the registration error as shown in Figure 1(d). However, in some regions which are dominated by Gaussian noise such as the smooth region and the plate in Figure 2(b), L1-norm fails to reconstruct the optimal solution. Moreover, small objects can disappear from the reconstructed HR image in some regions as shown in Figure 1(d). On the other hand, using adaptive mixed-norm can overcome the drawbacks of both norms as shown in Figures 1(e) and 1(f), for Case I and Case II, where Case I corresponds to the mixed-norm with fixed mixing matrix, based on erf of the initial guess of the HR image. Case II corresponds to the mixed-norm with mixing matrix updated every iteration. And the results of these two cases in the other zoomed parts are shown in Figures 2(c) and 2(d), respectively. Indeed updating the mixing matrix at every iteration gives better results than using fixed mixing matrix.Car sequence: (a) the 1st  LR image, (b) mixing parameterζ11, reconstructed HR image using, (c) L2-norm-based [1], (d) L1-norm-based [2], and (e) and (f) proposed algorithm for Case I and Case II, respectively.
(a)(b)(c)(d)(e)(f)Car sequence: reconstructed HR image using, (a)L2-norm-based [1], (b) L1-norm-based [2], and (c) and (d) proposed algorithm for Case I and Case II, respectively.
(a)(b)(c)(d)Finally, a plot of the normalized cost function,J(X,λ(X)), for different image sequences is shown in Figure 3 to justify proposition 1. It is worth to mention that the complexity of the proposed algorithm is in order O(L2).Figure 3
Convergence of the cost function for Car sequence.


## 5.1. Experiment Setup

To demonstrate the efficiency of the proposed algorithm, the simulation results of Car sequence [5] is included in this section. This sequence consists of 64 LR images and contains a moving object while the camera is stable.In the experiments, motion operators are estimated from the LR images by using gradient-based registration technique based on affine motion model [6] and the blurring operator is assumed to be Gaussian with kernel size equals 5×5 with variance equals 2. The steepest descent optimization is used for all the compared algorithms with maximum number of iterations equals 100, step size equals 0.1 (2 in case of L1-norm-based algorithm), and w is set to 20. For more fair comparison, the bilateral total variation (BTV) [2] is used as a regularization function for all the compared algorithms, that is,(9)R(X)=∑l=-PP∑m=-PPα|l|+|m|∥X-SxlSymX∥1,
where Sym is the shifting operator in direction y with m pixels, P is a factor that determines the smoothness support size and α is the bilateral parameter, 0<α≤1. P and α are set to 2 and 0.7, respectively, and θ is set to 0.7. For all the experiments, 1/ξ in (6) is set to 2∥Z∥1, where Z is the bilinearly interpolated reference frame.


## 5.2. Results

The results of the Car sequence are shown in Figure1. The 1st LR image and the corresponding mixing parameter, ζ11, are shown in Figures 1(a) and 1(b), respectively. Two regions of interest are shown by red rectangles in Figures 1(a) and 1(b). Zoom of the regions of interest of the resulting HR image using L2-norm-based algorithm [1] is shown in Figures 1(c) and 2(a). From these figures, we can see that using L2-norm can be good selection in some regions which is dominated by Gaussian noise as in the smooth regions and the Car's plate in Figure 1(c). However, in regions that dominated by Laplacian noise (due to high registration error), using L2-norm can result in deformation in the HR image as shown in Figure 2(a). On contrary, zoomed regions of interest of the resulting HR image using L1-norm-based algorithm [2] are shown in Figures 1(d) and 2(b). From these figures, we can see that using L1-norm is robust against the registration error as shown in Figure 1(d). However, in some regions which are dominated by Gaussian noise such as the smooth region and the plate in Figure 2(b), L1-norm fails to reconstruct the optimal solution. Moreover, small objects can disappear from the reconstructed HR image in some regions as shown in Figure 1(d). On the other hand, using adaptive mixed-norm can overcome the drawbacks of both norms as shown in Figures 1(e) and 1(f), for Case I and Case II, where Case I corresponds to the mixed-norm with fixed mixing matrix, based on erf of the initial guess of the HR image. Case II corresponds to the mixed-norm with mixing matrix updated every iteration. And the results of these two cases in the other zoomed parts are shown in Figures 2(c) and 2(d), respectively. Indeed updating the mixing matrix at every iteration gives better results than using fixed mixing matrix.Car sequence: (a) the 1st  LR image, (b) mixing parameterζ11, reconstructed HR image using, (c) L2-norm-based [1], (d) L1-norm-based [2], and (e) and (f) proposed algorithm for Case I and Case II, respectively.
(a)(b)(c)(d)(e)(f)Car sequence: reconstructed HR image using, (a)L2-norm-based [1], (b) L1-norm-based [2], and (c) and (d) proposed algorithm for Case I and Case II, respectively.
(a)(b)(c)(d)Finally, a plot of the normalized cost function,J(X,λ(X)), for different image sequences is shown in Figure 3 to justify proposition 1. It is worth to mention that the complexity of the proposed algorithm is in order O(L2).Figure 3
Convergence of the cost function for Car sequence.


## 6. Conclusion

An algorithm for image SR is presented based on adaptive mixed-norms. The proposed algorithm adaptively and locally weightsL1- and L2-norms based on local error. The proposed algorithm is suitable for suppressing both registration error and additive noise in the SR process.



---

*Source: 435194-2009-12-30.xml*