# Locally Regularized Collaborative Representation and an Adaptive Low-Rank Constraint for Single Image Superresolution

Authors: Rui Gao; Deqiang Cheng; Liangliang Chen; Qiqi Kou
Journal: Mobile Information Systems
 (2022)

---

## Abstract

Learning-based superresolution reconstruction is an efficient image processing technique that has become a popular topic in recent years. Since superresolution is an ill-conditioned problem, appropriate image priors or examples are the key factors for recovering high-quality images with rich details. However, in some current advanced superresolution methods, the mappings established using dictionaries of low- and high-resolution examples cannot effectively reflect the relationship between the low- and high-resolution spaces. Therefore, we introduce a local structure prior to the collaborative representation to constrain the projection matrix; this structure can better represent the nonlinear mapping between the low- and the high-resolution feature spaces. Then, based on the redundancy of similar image patches, a shape-adaptive low-rank constraint is utilized to explore the images’ nonlocal self-similarity, and the local and nonlocal priors complement each other to enhance the recovered image quality. Finally, an iterative optimization algorithm is adopted to solve our proposed superresolution model. Numerous experiments were performed to verify the proposed method, and the results demonstrate its superiority to some state-of-the-art methods both quantitatively and qualitatively.


---

## Body

## 1. Introduction

With the rapid development of artificial intelligence, video surveillance, biomedicine, and other fields, the requirements for high-resolution (HR) images and videos are increasingly stringent. The limitations of low-cost imaging equipment and poor imaging environments usually result in unpleasant HR images. Without reducing these inherent limitations, single image superresolution (SISR) methods can recover missing details and produce a satisfying HR image from an observed low-resolution (LR) image. The image degradation process can be expressed mathematically as follows:(1)Y=DHX+v,where D refers to a downsampling operator, H represents a blur matrix, X and Y, respectively, denote the original HR image and the degraded LR image, and v is additive Gaussian noise. Generally, existing SISR methods can be divided into interpolation-based approaches [1–3], reconstruction-based approaches [4–7], and learning-based approaches [8–17].Learning-or example-based SR approaches have been widely studied due to their outstanding advantages in terms of image reconstruction quality and speed. Their crucial idea is the discovery of the mapping relationship between LR and HR exemplar image patch pairs. According to the establishment of a mapping relationship, learning-based methods generally include neighbor-based approaches [14, 15, 18, 19], sparse representation-based approaches [8, 20–25], example regression-based approaches [9, 10, 12, 13], and deep learning-based approaches [11, 16, 17, 26–32].Freeman et al. [18] first introduced the example learning idea to the SISR model and established the mapping relationship using Markov networks. Chang et al. [19] proposed a neighbor embedding (NE) algorithm based on locally linear embedding (LLE) to produce HR patches using reconstruction weights and by linearly combining the HR neighbors. Xu et al. [15] proposed a new NE-based self-learning SR approach that explored a propagation strategy from both vertical and horizontal directions to better perform patch matching. From investigating compressive sensing to relying on the signal sparsity theory, many sparse representation-based SR methods have emerged. Sparse representation-based approaches are also known as dictionary-based approaches, as dictionary learning is crucial for this process [33]. Based on the assumption that each LR image patch shares the same sparse coefficient as its HR counterpart, Yang et al. obtained a plausible HR image using sparse coefficients and two jointly trained dictionaries [8]. Dehkordi et al. [21] proposed an SR method that operates under a self-learning framework using sparse representation; this method iteratively obtains high-frequency details through the updated difference between the degraded version of the SR output and LR input, which can further optimize the SR accuracy. To improve the reconstruction speed and decrease the computational burden without sacrificing the SR quality, Timofte et al. [9] utilized the NE method and a sparsely trained dictionary to develop an anchor neighbor regression (ANR) algorithm, which is a representative method of example regression-based approaches. Then, they further improved the ANR algorithm to acquire the adjusted ANR (A+) algorithm [10], which learns the linear regressors using the full training materials. Zhang et al. [12] proposed an effective collaborative representation cascade (CRC) framework that extracts features from the intermediate recovered resulting image; their methods can gradually enhance the LR input while also improving the HR image quality. Recently, deep learning-based methods have been greatly developed and have achieved perfect performance. Dong et al. [17] applied a convolutional neural network (CNN) to address the SR reconstruction problem by constructing three convolution layers that reveal the end-to-end mapping between low- and high-resolution images. To improve the training stability, Tian et al. [30] proposed a coarse-to-fine CNN-based SR method. By cascading different types of modular blocks, this method combines LR and HR features and proposes a feature fusion strategy using heterogeneous convolutions that can significantly improve the reconstruction efficiency under the premise of ensuring SR quality. Anwar and Barnes [31] presented a densely residual Laplacian network (DRLN) for SR reconstruction. This network adopts the residual structural design of the cascading residual and models the key features with Laplacian attention to improve the network’s learning ability. Although deep learning-based SR methods can produce impressive SR results, they generally have a strong dependence on device configuration and training data.From the image degradation process shown in (1), we know that SISR reconstruction is based on an observed LR image, eliminating the influence of factors such as blurring, downsampling and noise, and thereby restoring an HR image. Like other image restoration (IR) problems such as image dehazing [34], image deblurring [35], and image deraining [36, 37], this problem is an ill-posed inverse process. Therefore, high-frequency details lost in the degradation process usually need to be recovered by different image priors, and the appropriate image priors have great significance in terms of the quality of the reconstructed image. Commonly used priors are gradient priors [5, 6, 38–40], sparsity priors [41, 42], and nonlocal priors. In fact, different regions of natural images usually contain similar local content, and nonlocal self-similarity (NLSS) priors for images have received extensive attention. Representative NLSS priors include nonlocal mean (NLM)-based priors [4], nonlocal total variation (NLTV)-based priors [7], and nonlocal low-rank (NLR) priors [43–47]. Matrices containing nonlocal similar patches tend to be low in rank and NLR priors are a promising type of prior information in many SR reconstruction methods.The combination of multiple complementary priors is more competitive in terms of improving the SR quality when compared to an individual single prior. Furthermore, integrating the advantages of learning- and reconstruction-based approaches can further enhance the SR performance [6, 7, 35, 43, 45, 47]. For example, Ren et al. enhanced the NLTV prior with the decay kernel (DK) in addition to stable group similarity reliability (SGSR) schemes, used a CNN to extract multi-directional features from external samples as a local prior, and then combined two priors to construct a new SISR model that could provide fine details while preserving edges [7]. Dong et al. built a novel adaptive sparse domain selection (ASDS) scheme by integrating local autoregressive (AR) and nonlocal self-similarity (NLSS); this method could efficiently perform image deblurring and SR reconstruction [35]. Merging low rank with local rank regularization, Gong et al. proposed a novel multilayer strategy and reconstruction model [43].Motivated by the above work, in this paper, we design a locally regularized collaborative representation SR method with an adaptive low-rank constraint. The A+ algorithm adopts the collaborative representation to determine neighborhoods from training samples and uses them to precompute the projection matrices offline; the matrices are then directly multiplied by the LR input patches to generate the corresponding HR outputs. Therefore, the reconstruction complexity is reduced, while the reconstruction quality is guaranteed. However, the A+ algorithm usually assumes that the LR and HR feature spaces are structurally similar. In reality, the two spaces are not exactly similar, and the obtained projection matrix from the LR space cannot appropriately describe the HR space structure. In view of this information, as the locality prior knowledge is more important than the sparsity prior, the locality prior is introduced into the process of exploring the nonlinear mapping relationship between the LR and HR spaces, and the resultant projection matrix more accurately represents their mapping relationship. Then, to further reveal the underlying structures of similar patches and make the SR results more stable, we implemented the shape-adaptive low-rank (SLR) constraint to enhance the recovered image performance. Thus, this SR model can effectively combine the local priors learned from the external databases with the image’s internal nonlocal priors, fully exploiting the images’ prior knowledge. Specific experimental comparisons show that the proposed approach is better and more stable than some state-of-the-art approaches. This paper’s primary contributions are as follows:(1)
We consider the importance of the locality in terms of revealing the nonlinear mapping, and we introduce the local structure prior to constraining the reconstruction weights and constructing a locality-constraint collaborative representation model. The resultant projection matrix can better reflect the relationship between the LR image patches and the HR anchor points; this strategy is beneficial in terms of gaining high-frequency prior information from the external databases.(2)
The shape-adaptive grouping method is applied to search for similar patches, capture the image’s intrinsic structure, and remove dissimilar pixels in each similar image patch. Therefore, enforcing the SLR constraint between similar patches can efficiently separate noise and preserve edges, resulting in better SR results.The rest of the paper is organized as follows: Section2 summarizes related work on collaborative representation for SISR and NLR constraints. Section 3 introduces, describes, and analyzes the proposed method. Section 4 discusses the experiments that verify the proposed method’s effectiveness, and conclusions are drawn in Section 5.


## 2. Related Work

### 2.1. Collaborative Representation for SISR

The A+ [10] algorithm is an SISR method based on collaborative representation (also known as ridge regression) that has achieved considerable attention from researchers due to its efficiency in terms of training and reconstruction stages as well as its excellent reconstruction performance. During the training phase, starting from LR sample patches YL=pL1,pL2,…,pLNp, the A+ algorithm builds an LR dictionary DL=dL1,dL2,…,dLM using the same strategy described by Zeyde et al. [20], where Np and M represent the size of the samples and dictionary, respectively, and Np≫M. Each dictionary atom dLj is also called an anchor point.The A+ algorithm relies on two important premises. The first is that each LR image patchyi can be approximately linearly represented by the K nearest neighbors NLj=pLj,1,pLj,2,…,pLj,K of its nearest dictionary atom dLj. Here, NLj is searched for from among the full training samples YL. This process is unlike that for NLj in the ANR algorithm, which is found in LR dictionary DL. Then, the l2-norm least square regression problem for the representation coefficients can be expressed as follows:(2)wi=argminwiyi−NLjwi22+λwi22,where wi denotes the coefficients of yi in the neighbor and λ is the balance parameter. The closed solution to (2) is written as follows:(3)wi=NLjTNLj+λI−1NLjTyi.The other assumption is that the LR and HR feature space have similar local structures; in other words, they may share the same representation coefficients. Therefore, the corresponding HR image patchxi can be estimated using xi=NHjwi, where NHj denotes the corresponding HR neighbor. For higher computation efficiency, the projection matrix Pj can be computed by Pj=NHjNLjTNLj+λI−1NLjT during the learning stage; thus, the HR image patch can be calculated using xi=Pjyi.


### 2.2. Nonlocal Low-Rank Constraint

There are many similar local patterns in different locations throughout a natural image. As a result, natural images may exhibit the characteristics of the low-rank structure. Fully exploring the low-rank characteristics of data is beneficial for robust subspace recovery [48–50]. Therefore, in reconstruction-based SR problems, the NLR constraint is successful in terms of exploiting image NLSS, and it has received extensive attention.In the application of NLR, an image is usually first divided into overlapped patches. Then, thel most similar patches of each input image patch xi are searched for in the search window to form matrix Xi. Obviously, matrix Xi contains similar content, so it may satisfy the low-rank property. However, in practice, some noise may corrupt matrix Xi, resulting in the destruction of the low-rank property. Therefore, the low-rank matrix X˜i close to Xi can be recovered via the following equation:(4)X˜i=argminXiXi−X˜i22+γRankX˜i.After each low-rank matrixX˜i is obtained, the underlying low-rank structure information in the image can be revealed. In reconstruction-based SR methods, the reconstruction model can be expressed as follows:(5)X=argminXY−DHX22+βΦNLR,where(6)ΦNLR=∑iRiX−X˜i22+γRankX˜i,γ and β are balance parameters, and Ri represents the matrix extracting the similar patches for each exemplar image patch xi.


## 2.1. Collaborative Representation for SISR

The A+ [10] algorithm is an SISR method based on collaborative representation (also known as ridge regression) that has achieved considerable attention from researchers due to its efficiency in terms of training and reconstruction stages as well as its excellent reconstruction performance. During the training phase, starting from LR sample patches YL=pL1,pL2,…,pLNp, the A+ algorithm builds an LR dictionary DL=dL1,dL2,…,dLM using the same strategy described by Zeyde et al. [20], where Np and M represent the size of the samples and dictionary, respectively, and Np≫M. Each dictionary atom dLj is also called an anchor point.The A+ algorithm relies on two important premises. The first is that each LR image patchyi can be approximately linearly represented by the K nearest neighbors NLj=pLj,1,pLj,2,…,pLj,K of its nearest dictionary atom dLj. Here, NLj is searched for from among the full training samples YL. This process is unlike that for NLj in the ANR algorithm, which is found in LR dictionary DL. Then, the l2-norm least square regression problem for the representation coefficients can be expressed as follows:(2)wi=argminwiyi−NLjwi22+λwi22,where wi denotes the coefficients of yi in the neighbor and λ is the balance parameter. The closed solution to (2) is written as follows:(3)wi=NLjTNLj+λI−1NLjTyi.The other assumption is that the LR and HR feature space have similar local structures; in other words, they may share the same representation coefficients. Therefore, the corresponding HR image patchxi can be estimated using xi=NHjwi, where NHj denotes the corresponding HR neighbor. For higher computation efficiency, the projection matrix Pj can be computed by Pj=NHjNLjTNLj+λI−1NLjT during the learning stage; thus, the HR image patch can be calculated using xi=Pjyi.


## 2.2. Nonlocal Low-Rank Constraint

There are many similar local patterns in different locations throughout a natural image. As a result, natural images may exhibit the characteristics of the low-rank structure. Fully exploring the low-rank characteristics of data is beneficial for robust subspace recovery [48–50]. Therefore, in reconstruction-based SR problems, the NLR constraint is successful in terms of exploiting image NLSS, and it has received extensive attention.In the application of NLR, an image is usually first divided into overlapped patches. Then, thel most similar patches of each input image patch xi are searched for in the search window to form matrix Xi. Obviously, matrix Xi contains similar content, so it may satisfy the low-rank property. However, in practice, some noise may corrupt matrix Xi, resulting in the destruction of the low-rank property. Therefore, the low-rank matrix X˜i close to Xi can be recovered via the following equation:(4)X˜i=argminXiXi−X˜i22+γRankX˜i.After each low-rank matrixX˜i is obtained, the underlying low-rank structure information in the image can be revealed. In reconstruction-based SR methods, the reconstruction model can be expressed as follows:(5)X=argminXY−DHX22+βΦNLR,where(6)ΦNLR=∑iRiX−X˜i22+γRankX˜i,γ and β are balance parameters, and Ri represents the matrix extracting the similar patches for each exemplar image patch xi.


## 3. Proposed Method

In this section, we introduce our method, which combines locally regularized collaborative representation (LCR) with an SLR constraint to effectively realize the combination of the learning- and reconstruction-based methods.


### 3.1. Locally Regularized Collaborative Representation

The A+ algorithm is an extension of the ANR algorithm, and it is an efficient and high-quality SR method. As discussed in Section2.2, we notice that the calculation of the representation coefficients only considers the relationship between LR image patch yi and the dictionary atoms, not the relationship between the dictionary atoms and the samples in the neighbor NLj=pLj,1,pLj,2,…,pLj,K. That is, yi treats each sample in neighbor NLj equally, which leads to inaccuracy in the representation coefficients. Thus, the obtained projection matrix cannot better reflect the nonlinear mapping between the LR space and the HR space, and dissimilar patches may be assigned larger weights, degrading the reconstruction quality. To address this issue, we introduce a local structure prior to the objective function (2) of the representation weights, which can adaptively assign reconstruction weights to each input image patch yi according to the similarity between the anchor atom and each sample in neighbor NLj, thereby optimizing the projection matrix. The details of the presented method are as follows.During the dictionary learning stages, unlike the A+ algorithm, we measure the distance between the dictionary atoms and the atoms or samples using mutual coherence. The specific steps are as follows: we extract the LR features from the LR training images. Then, an LR feature set is obtained via the principal component analysis (PCA) dimension reduction algorithm, and the LR dictionaryDL is then trained using the following expression:(7)minDL,AYLf−DLAF+α∑i∑j,i≠jdLiTdLj,s.t.∀m,dLm2=1,∀n,an0≤p,where ∑i∑j,i≠jdLiTdLj denotes the mutual coherence between the dictionary atoms, A represents the coefficient matrix of the LR feature set YLf, α is the trade-off between the reconstruction error and mutual coherence, and p represents the sparsity level. The optimization problem (7) can be solved through MI-KSVD, which can be found in [51]. After acquiring the learned LR dictionary, we construct the corresponding HR dictionary DH.It has been proved that locality priors contribute to exploring nonlinear relationships in data [52, 53]. Therefore, we consider constructing a locality-constrained collaborative representation model for reconstruction coefficients. For each LR image patch yi, (2) can be rewritten as follows:(8)wi=argminwiyi−NLjwi22+λsi⊗wi22,s.t.1Twi=1,where ⊗ represents element-wise multiplication, and λ is a balance parameter. si is a K-dimensional vector, and its k-th element si,k=1/corrdLi,pLkσ is used to measure the relationship between anchor atom dLi and k-th neighbor sample pLk, where k=CKpLi. CKpLi is the index set of the K nearest neighbors of LR dictionary atom dLi. Further, σ is used to control the decay speed of the locality adapter. Note that k-th element si,k in vector si is inversely proportional to the correlation corrdLi,pLk; this relationship means that the higher the correlation between the dictionary atoms and the neighbor samples, the larger the assigned weights. Therefore, the purpose of adaptively assigning weights is achieved. Then, we can solve (8) and obtain its closed-form solution as follows:(9)wi=NLjTNLj+λdiagsi,kTdiagsi,k−1NLjTyi.Therefore, the projection matrix can be counted using the following equation:(10)Pj=NHjNLjTNLj+λdiagsi,kTdiagsi,k−1NLjT.The resultant projection matrix can accurately reveal the nonlinear mapping between the LR and HR feature spaces. Once the projection matrix has been calculated, the desired HR image patch can be generated viaxi=Pjyi. Moreover, the projection matrix is still computed during the training phase to improve the reconstruction quality without sacrificing speed. Algorithm 1 describes the projection matrix learning process.Algorithm 1: The projection matrix learning process.
Step 1: Input LR and HR Training imagesIHi,ILi, scale factor t, and dictionary size M; construct a series of image patches pHi,pLi, extract the corresponding features, and train the LR and HR dictionary of size M using MI-KSVD [51].Step 2: For each dictionary atomdLm: m=1 to M, search all training samples to obtain its K nearest LR neighbors NLj based on the correlation vector and the corresponding HR neighbors NHj.Step 3: Compute the projection matrixPm with a local constraint using (10) and store it for reconstruction.


### 3.2. Shape-Adaptive Low-Rank Constraint

Natural images usually contain plentiful redundant structures, indicating that the images satisfy the low-rank characteristic. Exploring the images’ NLR characteristics helps reveal their underlying internal structures, and NLR constraints have been widely used in SR reconstruction. In addition, when combined, local and nonlocal priors can complement each other and improve the reconstruction performance. Therefore, to further upgrade the LCR algorithm, we apply an NLR constraint to the proposed LCR model.General NLR-based SR methods exploit the fixed-shape grouping to find similar patches and construct low-rank matrixes. However, the fixed-shape grouping sometimes fails to accurately find similar patches for the target patch. Figure1 provides a simplistic example that illustrates the shortcomings of the fixed-shape NLR model. The black square represents the search window, the red square in the middle is the target patch, and the green squares are the similar patches that are searched for. We observed that the similar patches found using fixed-shape grouping strategy contain different counterparts from the target patch, i.e., the purple ellipse regions. Vectorizing the target and its similar patches, in addition to using the low-rank characteristic on the resultant matrix, results in poor performance.Figure 1
Graphical illustration of the low-rank model with fixed-shape grouping.To remedy the above problem, we incorporate the SLR constraint in the recovered SR image; this strategy can accurately search for similar patches, efficiently separate the image’s noise, and preserve its edges. Specifically, for each target patchx1i in the reconstructed SR image X0, the steer kernel [54] is first utilized to extract the local structure around its central pixel. Then, we compare the obtained kernel value with the preset threshold and select pixels with a kernel value greater than the threshold as the homogeneous pixels. In general NLR-based methods, similar patches (i.e., squares in Figure 1) are found using the l2 distance (i.e., x1i−xqi2 where xqi is the q-th adjacent patch of x1i), and then the target patch x1i and all similar patches are combined. Unlike general NLR-based methods, in the SLR model, after identifying all the homogeneous pixels in x1i, we fetch the homogeneous areas of different patches using Tix1i−Tixqi2, where Ti denotes the matrix used to extract the homogeneous pixels from the i-th target patch x1i, that is, the shape-adaptive grouping method. As shown in Figure 2, the solid red triangle represents the homogeneous pixel area of the target patch (i.e., the adaptive patch shape), the dashed red line delineates the part that must be excluded from the original fixed-shape patch, and the solid green triangles denote the shape-adaptive similar patches. Finally, we lexicographically stack all of the shape-adaptive similar patches together to form low-rank matrix X˜i, which has a stronger low-rank characteristic. In addition, it is not difficult to find that each similar patch in Figure 1 is larger than each similar patch in Figure 2, indicating that the dimension of the low-rank matrix X˜i that was constructed using SLR is also smaller and that the computation efficiency is higher. Low-rank matrix X˜i can be recovered via the following equation:(11)X˜i=argminX˜i∑iTiRiX0−X˜i22+γRankTiX˜i.Figure 2
Graphical illustration of the low-rank model with shape-adaptive grouping.After obtaining each low-rank matrixX˜i, the underlying low-rank structure of the HR image X0 may be revealed. We can obtain the entire HR image via the following minimization problem:(12)X=argminXY−DHX22+β∑iTiRiX−X˜i22.


### 3.3. The Optimization of the LCR_SLR Model

Refining the image derived from the LCR model, we enforce the SLR constraint on it and propose the LCR_SLR method. According to the previous analysis, the proposed method primarily includes two steps. The first step involves learning the reconstruction weights corresponding to the anchor neighbors of each LR image patch. The second step involves upgrading the resultant image using the SLR constraint. In the first step, we obtain the weights via (9), and the projection matrix can be calculated offline. Once all of the HR image patches are obtained, they are integrated together to form the required HR image X0. In the second step, we impose the SLR constraint on the initial reconstruction result and uncover its underlying NLSS using (11). However, (11) is NP-hard since it is a rank minimization problem. We used the weighted nuclear norm [51] to replace the matrix rank. (11) can be expressed as follows:(13)X˜i=argminX˜i∑iTiRiX0−X˜i22+γTiX˜iwi,∗,where TiX˜iwi,∗=∑lwliσli, σli is the l-th singular value of TiRiX0, and wi refers to the nonnegative weight vector formed by wli. When gi=TiX˜i, (13) can be rewritten as follows:(14)gi=argminX˜i∑iTiRiX0−gi22+γgiwi,∗.Using the method in [55], the r+1 th iteration can be described as follows:(15)gir+1=UΣ−γdiagwi+VT,where UΣVT is the singular value decomposition (SVD) of TiRiX0, χ+=maxχ,0 and wi is the weight vector.After solving for allTiX˜i, the entire HR image can be recovered via optimization problem (12). Then, the closed solution of X can be written as follows:(16)X=DHTDH+β∑iTiRiTTiRi−1⋅DHTY+β∑iTiRiTTiX˜i.In practice, to reduce the computational burden of (16), results can be produced by the conjugate gradient (CG) method. Algorithm 2 describes our proposed LCR_SLR method in detail.Algorithm 2: The SR method via LCR_SLR.
Step 1: Input LR test imageY, learned LR dictionary DL, HR dictionary DH, scale factor t, regularization parameter η, β and γ, and the nearest neighbor number M.Step 2: For each LR image patchyLii=1 to s, search on DL to find the nearest neighbor and its corresponding index, obtain the HR counterpart of yLi via xHi=PjyLi, and combine all HR image patches to form a whole HR image X0.Step 3: For each target patchxi in X0, utilize steer kernel regression to estimate matrix Ti, perform shape-adaptive grouping on X0 and calculate each low-rank matrix TiX˜i via (15).Step 4: Obtain the final reconstructed HR imageX^=XH via (16).


## 3.1. Locally Regularized Collaborative Representation

The A+ algorithm is an extension of the ANR algorithm, and it is an efficient and high-quality SR method. As discussed in Section2.2, we notice that the calculation of the representation coefficients only considers the relationship between LR image patch yi and the dictionary atoms, not the relationship between the dictionary atoms and the samples in the neighbor NLj=pLj,1,pLj,2,…,pLj,K. That is, yi treats each sample in neighbor NLj equally, which leads to inaccuracy in the representation coefficients. Thus, the obtained projection matrix cannot better reflect the nonlinear mapping between the LR space and the HR space, and dissimilar patches may be assigned larger weights, degrading the reconstruction quality. To address this issue, we introduce a local structure prior to the objective function (2) of the representation weights, which can adaptively assign reconstruction weights to each input image patch yi according to the similarity between the anchor atom and each sample in neighbor NLj, thereby optimizing the projection matrix. The details of the presented method are as follows.During the dictionary learning stages, unlike the A+ algorithm, we measure the distance between the dictionary atoms and the atoms or samples using mutual coherence. The specific steps are as follows: we extract the LR features from the LR training images. Then, an LR feature set is obtained via the principal component analysis (PCA) dimension reduction algorithm, and the LR dictionaryDL is then trained using the following expression:(7)minDL,AYLf−DLAF+α∑i∑j,i≠jdLiTdLj,s.t.∀m,dLm2=1,∀n,an0≤p,where ∑i∑j,i≠jdLiTdLj denotes the mutual coherence between the dictionary atoms, A represents the coefficient matrix of the LR feature set YLf, α is the trade-off between the reconstruction error and mutual coherence, and p represents the sparsity level. The optimization problem (7) can be solved through MI-KSVD, which can be found in [51]. After acquiring the learned LR dictionary, we construct the corresponding HR dictionary DH.It has been proved that locality priors contribute to exploring nonlinear relationships in data [52, 53]. Therefore, we consider constructing a locality-constrained collaborative representation model for reconstruction coefficients. For each LR image patch yi, (2) can be rewritten as follows:(8)wi=argminwiyi−NLjwi22+λsi⊗wi22,s.t.1Twi=1,where ⊗ represents element-wise multiplication, and λ is a balance parameter. si is a K-dimensional vector, and its k-th element si,k=1/corrdLi,pLkσ is used to measure the relationship between anchor atom dLi and k-th neighbor sample pLk, where k=CKpLi. CKpLi is the index set of the K nearest neighbors of LR dictionary atom dLi. Further, σ is used to control the decay speed of the locality adapter. Note that k-th element si,k in vector si is inversely proportional to the correlation corrdLi,pLk; this relationship means that the higher the correlation between the dictionary atoms and the neighbor samples, the larger the assigned weights. Therefore, the purpose of adaptively assigning weights is achieved. Then, we can solve (8) and obtain its closed-form solution as follows:(9)wi=NLjTNLj+λdiagsi,kTdiagsi,k−1NLjTyi.Therefore, the projection matrix can be counted using the following equation:(10)Pj=NHjNLjTNLj+λdiagsi,kTdiagsi,k−1NLjT.The resultant projection matrix can accurately reveal the nonlinear mapping between the LR and HR feature spaces. Once the projection matrix has been calculated, the desired HR image patch can be generated viaxi=Pjyi. Moreover, the projection matrix is still computed during the training phase to improve the reconstruction quality without sacrificing speed. Algorithm 1 describes the projection matrix learning process.Algorithm 1: The projection matrix learning process.
Step 1: Input LR and HR Training imagesIHi,ILi, scale factor t, and dictionary size M; construct a series of image patches pHi,pLi, extract the corresponding features, and train the LR and HR dictionary of size M using MI-KSVD [51].Step 2: For each dictionary atomdLm: m=1 to M, search all training samples to obtain its K nearest LR neighbors NLj based on the correlation vector and the corresponding HR neighbors NHj.Step 3: Compute the projection matrixPm with a local constraint using (10) and store it for reconstruction.


## 3.2. Shape-Adaptive Low-Rank Constraint

Natural images usually contain plentiful redundant structures, indicating that the images satisfy the low-rank characteristic. Exploring the images’ NLR characteristics helps reveal their underlying internal structures, and NLR constraints have been widely used in SR reconstruction. In addition, when combined, local and nonlocal priors can complement each other and improve the reconstruction performance. Therefore, to further upgrade the LCR algorithm, we apply an NLR constraint to the proposed LCR model.General NLR-based SR methods exploit the fixed-shape grouping to find similar patches and construct low-rank matrixes. However, the fixed-shape grouping sometimes fails to accurately find similar patches for the target patch. Figure1 provides a simplistic example that illustrates the shortcomings of the fixed-shape NLR model. The black square represents the search window, the red square in the middle is the target patch, and the green squares are the similar patches that are searched for. We observed that the similar patches found using fixed-shape grouping strategy contain different counterparts from the target patch, i.e., the purple ellipse regions. Vectorizing the target and its similar patches, in addition to using the low-rank characteristic on the resultant matrix, results in poor performance.Figure 1
Graphical illustration of the low-rank model with fixed-shape grouping.To remedy the above problem, we incorporate the SLR constraint in the recovered SR image; this strategy can accurately search for similar patches, efficiently separate the image’s noise, and preserve its edges. Specifically, for each target patchx1i in the reconstructed SR image X0, the steer kernel [54] is first utilized to extract the local structure around its central pixel. Then, we compare the obtained kernel value with the preset threshold and select pixels with a kernel value greater than the threshold as the homogeneous pixels. In general NLR-based methods, similar patches (i.e., squares in Figure 1) are found using the l2 distance (i.e., x1i−xqi2 where xqi is the q-th adjacent patch of x1i), and then the target patch x1i and all similar patches are combined. Unlike general NLR-based methods, in the SLR model, after identifying all the homogeneous pixels in x1i, we fetch the homogeneous areas of different patches using Tix1i−Tixqi2, where Ti denotes the matrix used to extract the homogeneous pixels from the i-th target patch x1i, that is, the shape-adaptive grouping method. As shown in Figure 2, the solid red triangle represents the homogeneous pixel area of the target patch (i.e., the adaptive patch shape), the dashed red line delineates the part that must be excluded from the original fixed-shape patch, and the solid green triangles denote the shape-adaptive similar patches. Finally, we lexicographically stack all of the shape-adaptive similar patches together to form low-rank matrix X˜i, which has a stronger low-rank characteristic. In addition, it is not difficult to find that each similar patch in Figure 1 is larger than each similar patch in Figure 2, indicating that the dimension of the low-rank matrix X˜i that was constructed using SLR is also smaller and that the computation efficiency is higher. Low-rank matrix X˜i can be recovered via the following equation:(11)X˜i=argminX˜i∑iTiRiX0−X˜i22+γRankTiX˜i.Figure 2
Graphical illustration of the low-rank model with shape-adaptive grouping.After obtaining each low-rank matrixX˜i, the underlying low-rank structure of the HR image X0 may be revealed. We can obtain the entire HR image via the following minimization problem:(12)X=argminXY−DHX22+β∑iTiRiX−X˜i22.


## 3.3. The Optimization of the LCR_SLR Model

Refining the image derived from the LCR model, we enforce the SLR constraint on it and propose the LCR_SLR method. According to the previous analysis, the proposed method primarily includes two steps. The first step involves learning the reconstruction weights corresponding to the anchor neighbors of each LR image patch. The second step involves upgrading the resultant image using the SLR constraint. In the first step, we obtain the weights via (9), and the projection matrix can be calculated offline. Once all of the HR image patches are obtained, they are integrated together to form the required HR image X0. In the second step, we impose the SLR constraint on the initial reconstruction result and uncover its underlying NLSS using (11). However, (11) is NP-hard since it is a rank minimization problem. We used the weighted nuclear norm [51] to replace the matrix rank. (11) can be expressed as follows:(13)X˜i=argminX˜i∑iTiRiX0−X˜i22+γTiX˜iwi,∗,where TiX˜iwi,∗=∑lwliσli, σli is the l-th singular value of TiRiX0, and wi refers to the nonnegative weight vector formed by wli. When gi=TiX˜i, (13) can be rewritten as follows:(14)gi=argminX˜i∑iTiRiX0−gi22+γgiwi,∗.Using the method in [55], the r+1 th iteration can be described as follows:(15)gir+1=UΣ−γdiagwi+VT,where UΣVT is the singular value decomposition (SVD) of TiRiX0, χ+=maxχ,0 and wi is the weight vector.After solving for allTiX˜i, the entire HR image can be recovered via optimization problem (12). Then, the closed solution of X can be written as follows:(16)X=DHTDH+β∑iTiRiTTiRi−1⋅DHTY+β∑iTiRiTTiX˜i.In practice, to reduce the computational burden of (16), results can be produced by the conjugate gradient (CG) method. Algorithm 2 describes our proposed LCR_SLR method in detail.Algorithm 2: The SR method via LCR_SLR.
Step 1: Input LR test imageY, learned LR dictionary DL, HR dictionary DH, scale factor t, regularization parameter η, β and γ, and the nearest neighbor number M.Step 2: For each LR image patchyLii=1 to s, search on DL to find the nearest neighbor and its corresponding index, obtain the HR counterpart of yLi via xHi=PjyLi, and combine all HR image patches to form a whole HR image X0.Step 3: For each target patchxi in X0, utilize steer kernel regression to estimate matrix Ti, perform shape-adaptive grouping on X0 and calculate each low-rank matrix TiX˜i via (15).Step 4: Obtain the final reconstructed HR imageX^=XH via (16).


## 4. Experimental Results and Analysis

In this section, we perform experiments to analyze the validity of our method. The experiments were all implemented on a computer configured with an Intel(R) Core (TM) i7-6500U processor in the MATLAB R2017b programming environment.


### 4.1. Experimental Settings

To verify the performance of our proposed LCR_SLR model, we chose several representative methods as competitors, including SCSR [8], A+ [10], ASCSR [23], LPGSSR [46], CRC [12], and SRCNN [17]. The test images in the experiments originate from four commonly used image datasets: Set 5, Set 14, BSD100, and Urban100. These datasets are composed of 5, 14, 100, and 100 images, respectively. For all images, we only recovered the Y channel (YCbCr color space) utilizing the proposed method, while both Cb and Cr channels use bicubic interpolation.In all experiments, we obtained simulated LR images using a7×7 Gaussian blurring filter with a standard deviation of 1.6 and downsampling factors with different scales. Moreover, when verifying our method’s robustness, additive Gaussian noise with a standard deviation of σ=5 was added to the simulated LR images. In our LCR_SLR model, we set η=0.001, β=0.007, and γ=2.5. The patch size was 5×5, the group size was 20, and the search window size was 25. Under noisy conditions, since the dependability of the fidelity term in (13) decreases, we increased β to 0.175.


### 4.2. Experiments on Noiseless Images

Table1 lists the average values of the peak signal-to-noise ratio (PSNR) and structural similarity (SSIM) generated by various methods when there was no noise and the scale factors were ×3 and ×4. Among all of the compared methods, the SCSR method’s performance was the lowest, and the LPGSSR method demonstrated the most excellent performance. As the CRC method constructs a multilayer mapping model for learning the LR and HR feature pairs, the CRC method is superior to the A+ algorithm. When the scale factor was ×4, the average PSNR and SIIM obtained by the SRCNN on the BSD100 dataset were higher than those of the CRC method. The SRCNN is a deep learning-based SR method, and due to the excellent performance of deep learning in SR reconstruction, it has developed rapidly in recent years. Both the ASCSR and LPGSSR are SR methods that include a low-rank constraint. On the four datasets, the reconstruction results for the LPGSSR are better than those for the ASCSR. As listed in Table 1, our proposed method is more competitive than others since it achieved the highest PSNR and SSIM values on the four datasets.Table 1
Average PSNR and SSIM results (×3 and ×4) on noiseless datasets σn=0.
SetsSet5Set14BSD100Urban100Scale×3×4×3×4×3×4×3×4SCSR30.31—27.62—27.23—23.67—0.8624—0.7756—0.7483—0.7227—A+32.2830.0628.8127.2427.9426.6024.5823.290.90210.85840.80770.74740.76950.70630.73240.6628ASCSR32.0529.8828.6627.0327.8026.3824.3523.160.89630.84570.80240.74280.76710.70080.72810.6597LPGSSR32.7730.4229.1427.4428.1726.6725.0123.450.91040.87090.81320.75270.78020.70790.75250.6772CRC32.4830.2128.9827.3328.1626.6424.7423.420.90540.86250.81180.74910.77800.70720.74700.6751SRCNN32.4230.1428.9227.2828.1326.7024.7223.310.90530.85880.80920.74770.77670.70910.73720.6621Proposed33.4330.8429.4627.6928.3826.8625.4823.660.91780.88050.82080.76190.78770.71460.76780.6834Aiming to visually evaluate the proposed method, Figures3 and 4 provide a visual detail comparison of an image, Butterfly, from Set 5 and an image, ppt3, from Set 14, respectively, with a magnification set to ×3. Figure 5 shows the visual results for the img087 image from Urban100 at ×4 magnification. The sharpness of Figures 3(e)–3(g) is similar; they are clearer than those in Figures 3(b) and 3(d). The same case also appears in Figure 5. The edges of the letters in Figure 4(e) are slightly clearer than those in Figures 4(f) and 4(g). Obviously, compared with the other methods, the results generated by our method have clearer edges and richer structural information.Figure 3
Visual SR results on butterfly produced by different methods (scaling factors = 3): (a) original, (b) SCSR [8], (c) A+ [10], (d) ASCSR [23], (e) LPGSSR [46], (f) CRC [12], (g) SRCNN [17], and (h) proposed method.
(a)(b)(c)(d)(e)(f)(g)(h)Figure 4
Visual SR results on PPT3 produced by different methods (scaling factors = 3): (a) original, (b) SCSR [8], (c) A+ [10], (d) ASCSR [23], (e) LPGSSR [46], (f) CRC [12], (g) SRCNN [17], and (h) proposed method.
(a)(b)(c)(d)(e)(f)(g)(h)Figure 5
Visual SR results on Img087 produced by different methods (scaling factors = 4): (a) original, (b) Bicubic, (c) A+ [10], (d) ASCSR [23], (e) LPGSSR [46], (f) CRC [12], (g) SRCNN [17], and (h) proposed method.
(a)(b)(c)(d)(e)(f)(g)(h)


### 4.3. Experiments on Noisy Images

Compared methods such as A+ [10], CRC [12], and SRCNN [17] cannot effectively handle noise. Therefore, before using these SR methods, we first apply the BM3D [56] denoising algorithm to the LR images. Table 2 shows the average PSNR and SSIM values for the above four datasets. Clearly, the LPGSSR method exceeds the compared methods. On all four datasets, the performance of the CRC method is superior to that of the SRCNN method. In general, our proposed method still surpasses all other methods, indicating that our method is robust to noise.Table 2
Average PSNR and SSIM results (×3 and ×4) on noiseless datasets (σn=5).
SetsSet5Set14BSD100Urban100Scale×3×4×3×4×3×4×3×4SCSR29.21—26.77—26.30—23.18—0.8126—0.7149—0.6762—0.6611—A+30.3328.9427.6126.4726.7525.8023.9122.980.84690.81840.74590.70520.69330.65800.69010.6348ASCSR30.1328.7227.3726.2326.5625.5823.6922.740.84010.81290.73880.70090.68720.64920.68530.6307LPGSSR30.7829.3127.8326.5627.0325.9124.3223.200.85760.83270.75430.71280.72150.66210.70210.6442CRC30.6629.2327.7726.5926.9025.8824.2023.130.85260.82910.75120.71040.71480.66440.69790.6411SRCNN30.4129.0227.6826.5226.8325.8424.1323.090.84760.82030.74470.70700.69810.65920.69220.6368Proposed31.1629.6628.1626.7927.2826.0424.6723.450.87170.84410.76960.72600.72830.67880.72900.6637Figures6 and 7 illustrate the visual effect on image 145086 from BSD100 and another image, Baboon, from Set 14 with a scale factor of ×3. Obviously, the SR results of the A+ [10], CRC [12], and SRCNN [17] have lost some details, and the edges in their resultant images are less clear. Due to the addition of the low-rank constraint, ASCSR and LGPSSR can repress noise to a certain extent. Additionally, LGPSSR is more efficient than ASCSR, but their recovered SR images contain some artifacts. Our method can produce reconstructed images with rich details and clear edges while suppressing noise, which shows that our method has stronger robustness.Figure 6
Visual SR results on 145086 produced by different methods (scaling factors = 3, σn=5): (a) original, (b) SCSR [8], (c) A+ [10], (d) ASCSR [23], (e) LPGSSR [46], (f) CRC [12], (g) SRCNN [17], and (h) proposed method.
(a)(b)(c)(d)(e)(f)(g)(h)Figure 7
Visual SR results on Baboon produced by different methods (scaling factors = 3, σn=5): (a) original, (b) SCSR [8], (c) A+ [10], (d) ASCSR [23], (e) LPGSSR [46], (f) CRC [12], (g) SRCNN [17], and (h) proposed method.
(a)(b)(c)(d)(e)(f)(g)(h)


### 4.4. Discussion of Parameters

This section discusses the parameters of the proposed model. For simplicity, we merely discuss all of the results under noiseless conditions where the scale factor is×3, and the blur kernel is the same as that in Section 4.1.


#### 4.4.1. Influence of Trade-Off Parametersη and β

Parameterη controls the local structure prior weight of the objective function, and parameter β indicates the importance of the nonlocal low-rank constraint. With changes to parameters η and β, Figures 8 and 9 display the average PSNR and SSIM for all of the images contained in Set 5 that were generated by our method. Figure 8 shows that the SR performance is improved as η increases. Specifically, the SR performance is better when η is in the range 10−6 to 10−4, and the curves become stable after η=10−4. Therefore, we set η=0.001 in the experiments. In Figure 9, the SR results are the best when setting β to 0.007. Therefore, in our experiments, we selected β=0.007.Figure 8
Influence of the parameterη in the proposed LCR_SLR method, (a) PSNR, (b) SSIM.
(a)(b)Figure 9
Influence of the parameterβ in the proposed LCR_SLR method, (a) PSNR, (b) SSIM.
(a)(b)


#### 4.4.2. Influence of the Patch Magnitude and Group Magnitude

Tables3 and 4 list the influence of the patch magnitude and group magnitude on the SR results, respectively; they list the average PSNR and SSIM values for all of the images in Set 5 for different patch sizes and group sizes. Clearly, the SR performance is the best when the patch size and the group size are 7 × 7 and 25, respectively. Both large patch or large group sizes may incur computational burdens, so we selected the patch size as 5 × 5 and the group size as 20 in our experiments.Table 3
Influence of the patch size in the proposed LCR_SLR method.
Patch size3 × 35 × 57 × 79 × 9PSNR32.4233.4333.4633.44SSIM0.90770.91780.91830.9180Time (s)199.4290.8410.6482.4Table 4
Influence of the group size in the proposed LCR_SLR method.
Group size10152025PSNR32.8033.2733.4333.45SSIM0.91260.91630.91780.9181Time (s)205.5229.3290.8404.1


### 4.5. Discussion of the Performance for Real-World Images

To evaluate the performance of the proposed method on real-world noisy images, we performed experiments using the SUN database [57]. As shown in Figure 10, we selected two noisy images, Heliport and Building Facade. The regions of interest (ROI) marked by green boxes in both figures were used for reconstruction to verify the practical effectiveness of our proposed approach. Both ROIs with a size of 128 × 128 were used to generate HR images. Figures 11 and 12 show the SR visual comparison results produced by our method and other different SR methods at a magnification factor of 3. From the two figures, the following conclusions can be drawn: First, the HR images generated by Bicubic are the blurriest. Second, ASCSR and LPGSSR outperform other compared methods in terms of the reconstruction effect, but ASCSR produces images with artifacts that are not as sharp as those obtained by LPGSSR. Finally, the recovered HR images using the proposed method have the cleanest edges and richest details. The experimental results show that our proposed method is more robust against noise while preserving image edges. Furthermore, it can be effectively applied to real-world noisy images, as it outperforms other advanced SR algorithms. In addition, the SR model we designed realizes the complementary advantages of local and nonlocal image priors, so we firmly believe that the proposed SR model can also achieve better performances for image inpainting, image deblurring, and image denoising tasks.Figure 10
From left to right, real-world images: (a) heliport (b) building facade, and their corresponding ROI areas for testing.
(a)(b)Figure 11
Visual SR results of the ROIs in imageHeliport produced by different methods (scaling factor s = 3): (a) bicubic, (b) SCSR [8], (c) A+ [10], (d) ASCSR [23], (e) LPGSSR [46], (f) CRC [12], (g) SRCNN [17], and (h) proposed method.
(a)(b)(c)(d)(e)(f)(g)(h)Figure 12
Visual SR results of the ROIs in imagebuilding facade produced by different methods (scaling factor s = 3): (a) bicubic, (b) SCSR [8], (c) A+ [10], (d) ASCSR [23], (e) LPGSSR [46], (f) CRC [12], (g) SRCNN [17], and (h) proposed method.
(a)(b)(c)(d)(e)(f)(g)(h)


## 4.1. Experimental Settings

To verify the performance of our proposed LCR_SLR model, we chose several representative methods as competitors, including SCSR [8], A+ [10], ASCSR [23], LPGSSR [46], CRC [12], and SRCNN [17]. The test images in the experiments originate from four commonly used image datasets: Set 5, Set 14, BSD100, and Urban100. These datasets are composed of 5, 14, 100, and 100 images, respectively. For all images, we only recovered the Y channel (YCbCr color space) utilizing the proposed method, while both Cb and Cr channels use bicubic interpolation.In all experiments, we obtained simulated LR images using a7×7 Gaussian blurring filter with a standard deviation of 1.6 and downsampling factors with different scales. Moreover, when verifying our method’s robustness, additive Gaussian noise with a standard deviation of σ=5 was added to the simulated LR images. In our LCR_SLR model, we set η=0.001, β=0.007, and γ=2.5. The patch size was 5×5, the group size was 20, and the search window size was 25. Under noisy conditions, since the dependability of the fidelity term in (13) decreases, we increased β to 0.175.


## 4.2. Experiments on Noiseless Images

Table1 lists the average values of the peak signal-to-noise ratio (PSNR) and structural similarity (SSIM) generated by various methods when there was no noise and the scale factors were ×3 and ×4. Among all of the compared methods, the SCSR method’s performance was the lowest, and the LPGSSR method demonstrated the most excellent performance. As the CRC method constructs a multilayer mapping model for learning the LR and HR feature pairs, the CRC method is superior to the A+ algorithm. When the scale factor was ×4, the average PSNR and SIIM obtained by the SRCNN on the BSD100 dataset were higher than those of the CRC method. The SRCNN is a deep learning-based SR method, and due to the excellent performance of deep learning in SR reconstruction, it has developed rapidly in recent years. Both the ASCSR and LPGSSR are SR methods that include a low-rank constraint. On the four datasets, the reconstruction results for the LPGSSR are better than those for the ASCSR. As listed in Table 1, our proposed method is more competitive than others since it achieved the highest PSNR and SSIM values on the four datasets.Table 1
Average PSNR and SSIM results (×3 and ×4) on noiseless datasets σn=0.
SetsSet5Set14BSD100Urban100Scale×3×4×3×4×3×4×3×4SCSR30.31—27.62—27.23—23.67—0.8624—0.7756—0.7483—0.7227—A+32.2830.0628.8127.2427.9426.6024.5823.290.90210.85840.80770.74740.76950.70630.73240.6628ASCSR32.0529.8828.6627.0327.8026.3824.3523.160.89630.84570.80240.74280.76710.70080.72810.6597LPGSSR32.7730.4229.1427.4428.1726.6725.0123.450.91040.87090.81320.75270.78020.70790.75250.6772CRC32.4830.2128.9827.3328.1626.6424.7423.420.90540.86250.81180.74910.77800.70720.74700.6751SRCNN32.4230.1428.9227.2828.1326.7024.7223.310.90530.85880.80920.74770.77670.70910.73720.6621Proposed33.4330.8429.4627.6928.3826.8625.4823.660.91780.88050.82080.76190.78770.71460.76780.6834Aiming to visually evaluate the proposed method, Figures3 and 4 provide a visual detail comparison of an image, Butterfly, from Set 5 and an image, ppt3, from Set 14, respectively, with a magnification set to ×3. Figure 5 shows the visual results for the img087 image from Urban100 at ×4 magnification. The sharpness of Figures 3(e)–3(g) is similar; they are clearer than those in Figures 3(b) and 3(d). The same case also appears in Figure 5. The edges of the letters in Figure 4(e) are slightly clearer than those in Figures 4(f) and 4(g). Obviously, compared with the other methods, the results generated by our method have clearer edges and richer structural information.Figure 3
Visual SR results on butterfly produced by different methods (scaling factors = 3): (a) original, (b) SCSR [8], (c) A+ [10], (d) ASCSR [23], (e) LPGSSR [46], (f) CRC [12], (g) SRCNN [17], and (h) proposed method.
(a)(b)(c)(d)(e)(f)(g)(h)Figure 4
Visual SR results on PPT3 produced by different methods (scaling factors = 3): (a) original, (b) SCSR [8], (c) A+ [10], (d) ASCSR [23], (e) LPGSSR [46], (f) CRC [12], (g) SRCNN [17], and (h) proposed method.
(a)(b)(c)(d)(e)(f)(g)(h)Figure 5
Visual SR results on Img087 produced by different methods (scaling factors = 4): (a) original, (b) Bicubic, (c) A+ [10], (d) ASCSR [23], (e) LPGSSR [46], (f) CRC [12], (g) SRCNN [17], and (h) proposed method.
(a)(b)(c)(d)(e)(f)(g)(h)


## 4.3. Experiments on Noisy Images

Compared methods such as A+ [10], CRC [12], and SRCNN [17] cannot effectively handle noise. Therefore, before using these SR methods, we first apply the BM3D [56] denoising algorithm to the LR images. Table 2 shows the average PSNR and SSIM values for the above four datasets. Clearly, the LPGSSR method exceeds the compared methods. On all four datasets, the performance of the CRC method is superior to that of the SRCNN method. In general, our proposed method still surpasses all other methods, indicating that our method is robust to noise.Table 2
Average PSNR and SSIM results (×3 and ×4) on noiseless datasets (σn=5).
SetsSet5Set14BSD100Urban100Scale×3×4×3×4×3×4×3×4SCSR29.21—26.77—26.30—23.18—0.8126—0.7149—0.6762—0.6611—A+30.3328.9427.6126.4726.7525.8023.9122.980.84690.81840.74590.70520.69330.65800.69010.6348ASCSR30.1328.7227.3726.2326.5625.5823.6922.740.84010.81290.73880.70090.68720.64920.68530.6307LPGSSR30.7829.3127.8326.5627.0325.9124.3223.200.85760.83270.75430.71280.72150.66210.70210.6442CRC30.6629.2327.7726.5926.9025.8824.2023.130.85260.82910.75120.71040.71480.66440.69790.6411SRCNN30.4129.0227.6826.5226.8325.8424.1323.090.84760.82030.74470.70700.69810.65920.69220.6368Proposed31.1629.6628.1626.7927.2826.0424.6723.450.87170.84410.76960.72600.72830.67880.72900.6637Figures6 and 7 illustrate the visual effect on image 145086 from BSD100 and another image, Baboon, from Set 14 with a scale factor of ×3. Obviously, the SR results of the A+ [10], CRC [12], and SRCNN [17] have lost some details, and the edges in their resultant images are less clear. Due to the addition of the low-rank constraint, ASCSR and LGPSSR can repress noise to a certain extent. Additionally, LGPSSR is more efficient than ASCSR, but their recovered SR images contain some artifacts. Our method can produce reconstructed images with rich details and clear edges while suppressing noise, which shows that our method has stronger robustness.Figure 6
Visual SR results on 145086 produced by different methods (scaling factors = 3, σn=5): (a) original, (b) SCSR [8], (c) A+ [10], (d) ASCSR [23], (e) LPGSSR [46], (f) CRC [12], (g) SRCNN [17], and (h) proposed method.
(a)(b)(c)(d)(e)(f)(g)(h)Figure 7
Visual SR results on Baboon produced by different methods (scaling factors = 3, σn=5): (a) original, (b) SCSR [8], (c) A+ [10], (d) ASCSR [23], (e) LPGSSR [46], (f) CRC [12], (g) SRCNN [17], and (h) proposed method.
(a)(b)(c)(d)(e)(f)(g)(h)


## 4.4. Discussion of Parameters

This section discusses the parameters of the proposed model. For simplicity, we merely discuss all of the results under noiseless conditions where the scale factor is×3, and the blur kernel is the same as that in Section 4.1.


### 4.4.1. Influence of Trade-Off Parametersη and β

Parameterη controls the local structure prior weight of the objective function, and parameter β indicates the importance of the nonlocal low-rank constraint. With changes to parameters η and β, Figures 8 and 9 display the average PSNR and SSIM for all of the images contained in Set 5 that were generated by our method. Figure 8 shows that the SR performance is improved as η increases. Specifically, the SR performance is better when η is in the range 10−6 to 10−4, and the curves become stable after η=10−4. Therefore, we set η=0.001 in the experiments. In Figure 9, the SR results are the best when setting β to 0.007. Therefore, in our experiments, we selected β=0.007.Figure 8
Influence of the parameterη in the proposed LCR_SLR method, (a) PSNR, (b) SSIM.
(a)(b)Figure 9
Influence of the parameterβ in the proposed LCR_SLR method, (a) PSNR, (b) SSIM.
(a)(b)


### 4.4.2. Influence of the Patch Magnitude and Group Magnitude

Tables3 and 4 list the influence of the patch magnitude and group magnitude on the SR results, respectively; they list the average PSNR and SSIM values for all of the images in Set 5 for different patch sizes and group sizes. Clearly, the SR performance is the best when the patch size and the group size are 7 × 7 and 25, respectively. Both large patch or large group sizes may incur computational burdens, so we selected the patch size as 5 × 5 and the group size as 20 in our experiments.Table 3
Influence of the patch size in the proposed LCR_SLR method.
Patch size3 × 35 × 57 × 79 × 9PSNR32.4233.4333.4633.44SSIM0.90770.91780.91830.9180Time (s)199.4290.8410.6482.4Table 4
Influence of the group size in the proposed LCR_SLR method.
Group size10152025PSNR32.8033.2733.4333.45SSIM0.91260.91630.91780.9181Time (s)205.5229.3290.8404.1


## 4.4.1. Influence of Trade-Off Parametersη and β

Parameterη controls the local structure prior weight of the objective function, and parameter β indicates the importance of the nonlocal low-rank constraint. With changes to parameters η and β, Figures 8 and 9 display the average PSNR and SSIM for all of the images contained in Set 5 that were generated by our method. Figure 8 shows that the SR performance is improved as η increases. Specifically, the SR performance is better when η is in the range 10−6 to 10−4, and the curves become stable after η=10−4. Therefore, we set η=0.001 in the experiments. In Figure 9, the SR results are the best when setting β to 0.007. Therefore, in our experiments, we selected β=0.007.Figure 8
Influence of the parameterη in the proposed LCR_SLR method, (a) PSNR, (b) SSIM.
(a)(b)Figure 9
Influence of the parameterβ in the proposed LCR_SLR method, (a) PSNR, (b) SSIM.
(a)(b)


## 4.4.2. Influence of the Patch Magnitude and Group Magnitude

Tables3 and 4 list the influence of the patch magnitude and group magnitude on the SR results, respectively; they list the average PSNR and SSIM values for all of the images in Set 5 for different patch sizes and group sizes. Clearly, the SR performance is the best when the patch size and the group size are 7 × 7 and 25, respectively. Both large patch or large group sizes may incur computational burdens, so we selected the patch size as 5 × 5 and the group size as 20 in our experiments.Table 3
Influence of the patch size in the proposed LCR_SLR method.
Patch size3 × 35 × 57 × 79 × 9PSNR32.4233.4333.4633.44SSIM0.90770.91780.91830.9180Time (s)199.4290.8410.6482.4Table 4
Influence of the group size in the proposed LCR_SLR method.
Group size10152025PSNR32.8033.2733.4333.45SSIM0.91260.91630.91780.9181Time (s)205.5229.3290.8404.1


## 4.5. Discussion of the Performance for Real-World Images

To evaluate the performance of the proposed method on real-world noisy images, we performed experiments using the SUN database [57]. As shown in Figure 10, we selected two noisy images, Heliport and Building Facade. The regions of interest (ROI) marked by green boxes in both figures were used for reconstruction to verify the practical effectiveness of our proposed approach. Both ROIs with a size of 128 × 128 were used to generate HR images. Figures 11 and 12 show the SR visual comparison results produced by our method and other different SR methods at a magnification factor of 3. From the two figures, the following conclusions can be drawn: First, the HR images generated by Bicubic are the blurriest. Second, ASCSR and LPGSSR outperform other compared methods in terms of the reconstruction effect, but ASCSR produces images with artifacts that are not as sharp as those obtained by LPGSSR. Finally, the recovered HR images using the proposed method have the cleanest edges and richest details. The experimental results show that our proposed method is more robust against noise while preserving image edges. Furthermore, it can be effectively applied to real-world noisy images, as it outperforms other advanced SR algorithms. In addition, the SR model we designed realizes the complementary advantages of local and nonlocal image priors, so we firmly believe that the proposed SR model can also achieve better performances for image inpainting, image deblurring, and image denoising tasks.Figure 10
From left to right, real-world images: (a) heliport (b) building facade, and their corresponding ROI areas for testing.
(a)(b)Figure 11
Visual SR results of the ROIs in imageHeliport produced by different methods (scaling factor s = 3): (a) bicubic, (b) SCSR [8], (c) A+ [10], (d) ASCSR [23], (e) LPGSSR [46], (f) CRC [12], (g) SRCNN [17], and (h) proposed method.
(a)(b)(c)(d)(e)(f)(g)(h)Figure 12
Visual SR results of the ROIs in imagebuilding facade produced by different methods (scaling factor s = 3): (a) bicubic, (b) SCSR [8], (c) A+ [10], (d) ASCSR [23], (e) LPGSSR [46], (f) CRC [12], (g) SRCNN [17], and (h) proposed method.
(a)(b)(c)(d)(e)(f)(g)(h)


## 5. Conclusion

In this paper, we propose an SR approach that combines locally regularized collaborative representation with a shape-adaptive low-rank constraint. As the locality prior plays an important role in exploring the nonlinear data relationship, we introduce the locality-constraint parameter when calculating the projection matrix. This parameter fully uses the relationship between the dictionary atoms and the neighbor samples; the resultant projection matrix can better reflect the nonlinear mapping between the LR space and the HR space. Then, to further reveal the images’ internal NLSS structure, we apply the SLR constraint to the generated HR image to efficiently separate sparse noise, preserve image edges, and make the generated reconstruction images more robust. The extensive experimental results illustrate that our method achieves remarkable SR performance, surpassing some other advanced SR methods.The proposed method combines the local and nonlocal priors, which complement each other. Additionally, the reconstruction image quality is excellent, but some shortcomings remain. Searching for similar patches and optimization problems are usually complex issues, so we will try to study more effective search strategies in the future to further improve the reconstruction efficiency. Meanwhile, due to the robustness of our proposed framework to noisy images, we will also explore extending it to other IR work in complex underground mine coal environments, including image deblurring, and image denoising.



---

*Source: 8565910-2022-07-12.xml*