# An Efficient New Robust PCA Method for Joint Image Alignment and Reconstruction via theL2,1 Norms and Affine Transformation

Authors: Habte Tadesse Likassa; Yu Xia; Butte Gotu
Journal: Scientific Programming
 (2022)

---

## Abstract

In this study, an effective robust PCA is developed for joint image alignment and recovery viaL2,1 norms and affine transformations. To alleviate the potential impacts of outliers, heavy sparse noises, occlusions, and illuminations, the L2,1 norms along with affine transformations are taken into consideration. The determination of the parameters involved and the updating affine transformations is arranged in the form of a constrained convex optimization problem. To reduce the computation load, we also further decompose the error as sparse error and Gaussian noise; additionally, the alternating direction method of multipliers (ADMM) is considered to develop a new set of recursive equations to update the optimization parameters and the affine transformations iterative. The convergence of the derived updating equation is explained as well. Conducted simulations illustrate that the new method is superior to the baseline works in terms of precision on some public databases.


---

## Body

## 1. Introduction

Robust methods are the fundamental methods for image alignment and reconstruction [1, 2], video surveillance, text, video, and bioinformatics. Such approach is important to get the true underlying objects in highly correlated and complex high-dimensional images in which the approaches are applied in various applications in the areas of signal processing, images, texts, videos, and bioinformatics [3–8]. Moreover, this problem faces some severe challenges due to various annoying effects. It is because of this that developing an efficient robust PCA for joint image alignment and reconstruction methods, which can tackle the above adverse effects. Thus, developing new approaches that can better robust and replace instead of PCA [9] and RPCA [10] has been required to addressed; however, many of these existing approaches have a serious problem in reducing the rank and tackle the potential impact of outliers, occlusions, and illuminations. Nevertheless, none of these methods yield better performance and have no guarantees under broad conditions.A number of effective robust methods [11, 12] have been addressed for rank minimization in solving an image alignment and reconstruction problems. For example, De et al. [13] addressed a parameterized component analysis method to find the low-rank component and tackle the problem of misalignment via reducing the mean square error. Moreover, it is nonconvex and therefore lacks a polynomial-time method to solve the problem. Moreover, Reference [14] proposed an effective alignment approach for video separation. Moreover, Ma and Aybat [15] addressed a comprehensive review on methods for solving and variants of robust PCA. Moreover, Reference [16] addressed robust principal component analysis (RPCA) for a partially observed data matrix to eliminate outliers from the highly corrupted big data. Ebadi et al. [17] considered an efficient method for the tasks of batch image aligning and reconstruction. Chen et al. [18] considered a nonconvex with a penalized quadratic low-rank and sparse decomposition (NQLSD) method to reduce the potential impacts of outliers and sparse errors. However, it was unable to reconstruct and align images when the outliers and noises follow complex statistical distributions other than the mixture of Gaussian distributions. Kim et al. [19] addressed the L1 along with the gradient, but it lacks robustness with outliers and heavy sparse noises. Peng et al. [20] proposed an efficient robust image alignment to reduce the potential impact for sparse from the low-rank decomposition (RASL) method to recover the low-rank component. In addition, it only considered the potential impact of heavy sparse noises and outliers from a single subspace. Wang et al. [21] addressed RPCA with the L2,1 norm to alleviate the potential impact of outliers and heavy sparse noises; then, we further separate the objects as the fast separation model, where the recovering is the clean part of the data along with the scaled factor matrix. Oh et al. [22] mentioned a new method via the partial sum of singular value thresholding (PSVT) from which an incorporation of PSVT replaces the nuclear norms in the RPCA method [23, 24] to search the true underlying structure. It lacks the boosting performance in dealing about occlusion and illuminations when there are lots of data samples. Bouwmans et al. [25] addressed a new algorithm for decomposing highly correlated and complex high-dimensional images via L1 norm. There are several works conducted in the areas of signal processing. For instance, Reference [26–28] proposed novel semisupervised feature selection method from a new perspective; however, it seeks considering affine transformation and L21 norms to better prune out the potential impacts of annoying effects. The studies conducted by [29, 30], which is an effective constrained relaxation method. But, it cannot deal with outliers lying outside the main subspaces. To mitigate this problem, Reference [31] addressed an efficient method (IA-RPCA) by combining geometric transformations with the L1 and nuclear norm denoted L∗. But, it cannot well deal with miscellaneous effects in signal processing. To solve this problem, Zheng et al. [32] proposed a IGO-RIA to reduce the potential impacts in signal processing.In this work, an effective robust PCA is proposed to align and reconstruct high-dimensional images via affine transformation andL2,1 norms. In this new method, we need to minimize the ranks so that the highly correlated and complex high-dimensional images are jointly aligned and reconstructed. To develop a robust method in against the potential impacts of outliers and heavy sparse noises, the new method integrates the affine transformations, the L2,1 norms, and with decomposing large errors into the sparse and Gaussian noises, so the highly corrupted images can be rectified to render more faithful image reconstruction and representation. Additionally, the L2,1 norm is further considered to remove the correlated samples and reduce the errors across the images, enabling the new method to be more strong in dealing with occlusions and large variations in the images so that the rank minimization is carried out. The determination of the parameters involved, and the variables corresponding to the Gaussian noise and sparse errors with affine transformations with the L2,1 norms are arranged as a constrained convex optimization problem. To reduce the computational complexity and make the problem more stable, first we decompose the errors as a sparse errors and Gaussian noises; following this, the ADMM method is utilized to derive a new sets of recursive equations of updating the optimization parameters and the affine transformations iterative in a round-robin manner. The convergence issue of the model is experimentally tested and verified. The simulation result shows that superior to the baselines on various public databases.The contribution of this work is structured as follows:(1)
TheL2,1 norm is added with affine transformations, further decomposing the errors as the sparse errors and the Gaussian noise, then the true underlying objects to fix the distortion or misalignment problem in the batch of corrupted images.(2)
Additionally, theL2,1 norm is newly added with affine transformation instead of the L1, to get a more trustful new model.(3)
To mitigate the large errors in high-dimensional images, which will impact the accuracy of image alignment and reconstruction with rank minimization, theL2,1 norm is utilized to alleviate the potential impact of outliers and the sparse errors. The L2,1 norm is more robust as it further decomposes the errors as the sparse errors and Gaussian noises. This norm, when combined with the affine transformations, plus decompose the errors into the different types, can further enhance the performance. To minimize the modeling error of L1 norms, the L2,1 norm penalization approach is considered to attain with better recovering and aligning performances based on various datasets.(4)
TheL2,1 norm of G is taken to estimate the Gaussian noise in real applications so as to minimize the potential impact of outliers, occlusions and illuminations, and heavy sparse errors. Unlike to the others work [2, 33–39], the new method tries to decompose the aggregated errors as the Gaussian noise and sparse error, which make the new method more novel.(5)
The new set of equations, which are derived in more detailed including affine transformation using an ADMM method, is used to improve the robustness and solve the set of new optimization problems. The convergence of the developed iterative equations is addressed.The flow of this paper is structured as follows. Section2 indicates the related overview of the related works. Section 3 also explains problem formulation. Sections 4 and 5 develop updating equations to solve the proposed problem and analyze its convergence characteristics, respectively. In Section 6, experimental results are conducted to justify the proposed method. Section 7 illustrates some concluding remarks to summarize the paper.


## 2. Related Works

A number of works has been conducted for image recovery and alignments. For example, Vedaldi et al. [52] proposed a model to remove the errors via the log-determinant cost function. Lia et al. [53] addressed the issue of the spatial dependency between images through low-order polynomials. However, the performance is too weak. To tackle this dilemma, Reference [54, 55] addressed RPCA methods, which decomposed the corrupted data into two basic components. Lu et al. [56, 57] addressed a novel algorithm to tackle the misalignment dilemma, which is designed to find the low-rank component from the illuminated data. Oh et al. [58] introduced a new dynamic imaging method, which also utilizes rank minimization. To improve the performance, Reference [58, 59] addressed a randomized method for finding the low-rank part after decomposing via rank minimization. Podosinnikova et al. [60] developed a robust PCA to minimize the reconstruction error. Additionally, there are several methods [33, 33–35, 61, 62]; however, the performance in minimizing the ranks needs to be taken into account.Shahid et al. [63] incorporated the spectral graph regularization with the robust PCA. Shaker et al. [64] proposed an online sequential framework to find the clean part via pruning out the sparse corruptions. Hu et al. [65] introduced an approximate of the low-rank assumption for the matrix via a low-rank regularization to solve the face image denoising problems. Wright et al. [66] proposed a RPCA for image decomposition as low-rank and sparse errors; however, it lacks scalability. Kang et al. [67] addressed a robust method via nonconvex rank approximation. Zhang et al. [68] and Rahmani et al. [69] addressed a robust subspace recovery to tackle the influence of annoying effects. There are several methods for image recovery. However, its complexity is jeopardized when the outliers and sparse noises are heavy in the data. Additionally, Reference [70, 71] both addressed robust subspace learning and RPCA with rank minimization to tackle the potential impact of occlusions, illuminations, outliers, and heavy sparse noises. Shang et al. [72] proposed a novel algorithm for rank minimization by using double nuclear norm with nuclear norm hybrid penalties to alleviate the adverse dilemmas. However, decomposing the error further is very important for better performance. Additionally, there are several low-rank subspace methods to decompose the images [7, 73–75].


## 3. Problem Formulation

We assume that we haven images, Ii0∈ℜw×h, where w and h indicate the weight and height of highly correlated and complex high-dimensional images, respectively. In many situations, these images are occluded due to annoying effects. We can stack these images into a matrix: M=vecI10∣vecI20…∣vecIn0∈ℜm×n, where vec⋅ indicates the vector stacking operator.The method estimates the true underlying objects denoted byA and the error part E, and the others are considered as G. Thereby, we can further decompose M into the true underlying objects and the sparse noise part given by E of large matrix containing an additive noise part G [76, 77]: M=A+E+G, where the subspaces are not be independent from each other or the data are contaminated by large noises; then, the low-rank component lies in a union of subspaces, in which A∈ℜm×n is a clean low-rank dictionary and C∈ℜn×n is a coefficient matrix being used to represent M, and E∈ℜm×n indicates a sparse error matrix incurred by outliers or corruptions. The data matrix originally M with the low-rank component A and errors are denoted by E and the other a Gaussian noise matrix G; finally, the overall are given by Moτ=A+E+G.In reality,Ii0 is generally not well aligned, entailing the decomposition is incorrect. To tackle this dilemma, inspired by Reference [4, 20, 33–35, 78, 79], we apply affine transformations τi to the potentially misaligned input images Ii0 to get a collection of transformed images Ii=Ii0oτi, where the operator o indicates the transformation. We can then stack these aligned images into a matrix and obtain Moτ=vecI1vecI2…|vecIn∈ℜm×n. The aligned images can be treated as samples taken from a union of low-dimensional subspaces, which, if well align, should exhibit a low-rank subspace structure as the rank of the transformed images is as small as possible, up to some outliers and heavy sparse errors. The updating of the variables corresponding to the constraints Moτ=A+E+G is intractable due to its nonlinearity. To solve this dilemma, we assume that the change produced by these affine transformation of τ is small and an initial affine transformation of τ is known. We can then linearize Moτ by using the first-order Taylor approximation as Moτ+Δτ≈Moτ+∑i=1nJiΔτviviT, where Moτ∈ℜm×n is the transformed image, Δτ∈ℜp×n with p being the number of variables, Ji=∂vecIioτi/∂τi∈ℜm×p denotes the Jacobian of the ith image with respect to τi, and vi is the standard basis for ℜn. In this way, we obtain approximate transformations to recover the clean part from the underlying subspaces.Thus, to develop a stable method for image alignment with rank minimization viaL2,1 for image representation applications, the stable formulation on A+E+G decomposition assumes that the datum matrix M is denoted as M=A+E+G where G a noise term, which are independently identically distributed on each entry matrix. To align and recovery the low-rank component, we indicate so as to solve the following optimization problem. To make the new method more strong in dealing with heavy sparse noises and others, the L2,1 norm, which combines the advantages of the L1 and L2 norms, is used and characterize the sparsity and the low-rank subspace property. It can also tackle the sparse errors in data points, which are highly correlated across all data points in the images. The overall problem is formulated as a convex optimization constrained problem given by(1)minA,E,ΔτA∗+λ1E2,1+λ2G2,1s.tMoτ+∑i=1nJiΔτviviT=A+E+G,where A∗=∑i=1minm,nσiA represents the nuclear norm of A, in which σiA indicates the singular values of A, λ1 and λ2 are the regularization parameters, and E2,1=maxx≠0Ex1/x2 represents the L2,1 norm of E, and M2,1=∑i=1m∑j=1nMij2.TheA in Reference (1) indicates the nuclear norm lying in the low-dimensional subspaces. The second term C∗ constrains the representation to be low rank, and the third and fourth terms denoted by Q indicate regularizes the coefficient matrix and the sparse error E to be sparse, respectively. They are employed to have more faithful subspaces and capture the global structure of data. For easy reference, see Table 1.Table 1
Comparison of the proposed approach with other related algorithms in terms of the objective function and constraints.
MethodsObjective functionConstraints[40]minA,CM−ACF2ATA=I[13, 23, 33, 35, 41–43]minA,EA∗+λE1M=A+E[44]minA,EA∗+λE1+γtrAΦATM=A+E[18, 35, 45–48]minA,EA∗+λE1Moτ+∑i=1nJiΔτviviT=A+E[7, 34, 49–51]minC,E,AA∗+λE2,1M=AC+E,C=QOursminA,E,C,Q,ΔτA∗+λ1E2,1+λ2G2,1Moτ+∑i=1nJiΔτviviT=A+E+G


## 4. Proposed Method

To solve (1), augmented Lagrangian function is considered as given by(2)LA,E,Δτ,Z1=A∗+λ1E2,1+λ2G2,1+Z1,B−A−E−G,from which Z1∈ℜm×n, μ1, X,Y=TraceXTY, and ⋅F2 denotes the Lagrangian multipliers, penalty parameters, and Frobenius norm, respectively, and B=Moτ+∑i=1nJiΔτviviT. By using augmented Lagrange multiplier with adaptive penalty [80], Reference (2) can be rewritten as(3)LA,E,Δτ,Z1=A∗+λ1E2,1+λ2G2,1+μ1B−A−E−G+Z1μ1F2.Solving (3) directly is computationally not advisable; thereby, we need to update the parameters iteratively alternatively via ADMM [80, 81].To update the parameterA first, the parameters E and Δτ remain constant; therefore, Ak+1 can be determined by(4)Ak+1=argminALA,Ek,G,Δτk,where k indicates the iteration index. Again, by removing all irrelevant terms of A, Reference (4) can be simplified as(5)Ak+1=argminAA∗+μ12Bk−A−Ek−Gk+Z1kμ1kF2.Now, problem (5) is completely separable in which we can solve as convex constrained program. Then, we employed the shrinkage operator from which we can attain the following:(6)UΣV=svdBk+1μ1kZ1k−Ek−GkA=UςΣVT,where ς denotes the singular value decomposition operator. Secondly, to update E, we keep A, Q, C, and Δτ as constants, so Ek+1 can be obtained by(7)Ek+1=argminELAk+1,E,G,Δτk.Again, by ignoring all irrelevant terms ofE, Reference (10) can be simplified as(8)Ek+1=argminEλ1E2,1+μ1k2Bk−Ak−E−G+Z1kμ1kF2.By using lemma [82], the update of the ith column of Ek+1, Eik+1, is given by(9)Eik+1=Vik2−λ2k/μ1kVik2Vik,ifVik2≥λ2kμ1k,0,otherwise,where Vk=Bk−Ak−Gk+Z1k/μ1k.Thirdly, to updateG, we keep A, E, and Δτ as constants, so Gk+1 can be determined by(10)Gk+1=argminGLAk+1,Ek+1,G,Δτk,Again, by ignoring all irrelevant terms of E, Reference (10) can be simplified as(11)Gk+1=argminGμ1k2Bk−Ak−Ek−G−Z1kμ1kF2+λ2G2,1.By using lemma [82], the update of the ith column of Ek+1, Eik+1, is given by(12)Gik+1=Vik2−λ2k/μ1kVik2Vik,ifVik2≥λ2kμ1k,0,otherwise,where Vk=Bk−Ak−Ek+Z1k/μ1k.Lastly, to get an update ofΔτ, we keep A and E as constants and Δτk+1 can be obtained by(13)Δτk+1=argminΔτLAk+1,Ek+1,Δτ.By ignoring all irrelevant terms ofΔτ, we can obtain(14)Δτk+1=argminΔτμ12Bk−Ak+1−Ek+1−Gk+1−Z1kμ1kF2.Solving (15) with the threshold operators [18, 20], we can get an update of Δτk+1 as(15)Δτk+1=∑i=1nJi+Ak+1+Ek+1+Gk+1−Moτ−Z1kμ1kviviT,where Ji+ denotes Moore–Penrose pseudoinverses of Ji [83].Finally, following the same steps as the above, the Lagrangian multipliersZ1 are updated by(16)Z1k+1=Z1k+μ1k+1Bk−Ak+1−Ek+1−Gk+1.Likewise, the regularization parametersμ1 are updated, respectively, by(17)μ1k+1=minμmax,ρμ1k,where ρ is considered to be fixed and μmax is a tunable parameter that is used for an adjustment to fasten the convergence of the proposed method.The overall updating equations of our method can be briefly given as follows. We first updateA, E, G, and Δτ by (5), (8), (10)and (14) sequentially. Following this, the Lagrangian multipliers, Z1, and the penalty regularization parameter, μ1, are updated by (17-18).Then, all the updates of the equation are given in the form an iterative in a round-robin manner until we attain the convergences.


## 5. Convergence Analysis

The following two different theorems are developed in this work.Theorem 1.
Ifμ1k is nondecreasing, then the sequences Ak, Ek, Gk, Δτk, and Z1k generated by ADMM converge to a Karush–Kuhn–Tucker (KKT) point of , and (14).The proofs of both theorems can be readily extended from Reference [34, 77, 84, 85].Theorem 1 implies that the variablesAk, Ek, Gk, Δτk, and Z1k are guaranteed to converge to the global optimum with an appropriate choice of the Lagrange multipliers Z1k and a sufficiently large penalty parameters μ1k.


## 6. Experimental Results and Discussion

We need to examine the effectiveness of the new method for joint image alignment and reconstruction on some public databases. To attain the goal of the research, five different databases mainly the MINST database [86], the Wild database [87], the Al Gore talking video [20], CMU Multi-Pie dummy [88], and complicated window taken from objects [20].The baselines are RASL [20], TRPCA-SVD [56], NQLSD [18], and TRPCA-t [57]. Thereby, invoked by RASL and NQLSD, which are the most popular methods, we additionally considered the L2,1 norm. Moreover, our method is supported with more novel ideas to tackle the potential impacts of outliers and heavy sparse noises. All the conducted simulations are used to further illustrate the performances of the proposed method over the criterion of nuclear norms, L1 norm penalization, and the new tensor nuclear norms. To further verify the effectiveness of the new algorithm for image recovery, the final ranks are taken as a comparison criterion based on some public databases. Both the visual effect and the final rank evaluation criterion’s are made based on the five different databases include the MINST database, the Wild database, and the Al Gore talking video, the dummy, and complicated windows. Apparently, our proposed method gives the best results in all databases compared with the RASL, TRPCA-SVD, NQLSD, and TRPCA-t.


### 6.1. Experimental Convergence Performance

In this subsection, some experimental analysis of the proposed method based on four sets of images. The first set of image is based on the synthetic data, the second is based on the handwritten images, the third one is based on the Natural Faces labeled in the Wild database [87], and the fourth is based on the video face images, and the convergence characteristic is assessed by comparing the relative square error (RSE) vs. the iteration number, where RSE is defined as(18)C^−CFCF,in which C^ is the recovered image and C is the original one. Also, we set λ1=1/minm,n as suggested in Reference [89–92]. The resulting convergence curves are as shown in Figure 1, from which we can see that the RSEs of the proposed algorithm for all images decrease with the iteration number and then reach at a constant after a few iterations. This fact justifies the convergence of the proposed algorithm.Figure 1
Convergence curves of the proposed algorithm on: (a) synthetic data; (b) handwritten digits images; (c) natural face images; (d) video face images.
(a)(b)(c)(d)


### 6.2. Comparison with Baseline Approach

In this subsection, I made a comparison with the proposed approach, which adds in the affine transformations with theL2,1 norm, with some recently reported works in terms of the quantitative measures of the final ranks based on five different databases. Four different baselines approaches, including RASL [20], TRPCA-SVD [56], NQLSD [18], and TRPCA-t [57], and the proposed algorithm are conducted for comparison. Moreover, there are several works conducted in the areas of signal processing. For instance, Reference [26–28] proposed a supervised approach to prune out the annoying effects; however, they lack performance. Then, the performance results from these baselines are obtained based on publicly available codes.


#### 6.2.1. Handwritten Digits

In this experiment, 30 number of digits are taken from the MINST dataset [86] with sizes 29×29.We compare the final rank being considered as the evaluation criteria of the proposed approach with the aforementioned four baselines, as given in Table2, from which we can realize that NQLSD has better results than RASL, as NQLSD employs the local linear approximation with a quadratic penalty approach to tackle the potential setback of outliers and sparse noises in the images. Moreover, the TRPCA has enhanced the recovery over the other approaches, as it considers tensor to tensor products instead of singular value decomposition to exactly recover the low ranks. However, still it lacks robustness as the L2,1 norm is not taken into consideration. We can also observe from Table 2 that the suggested approach provides the best achievement. This is because it incorporates a set of affine transformations and employs the L2,1 norms, the approach also splits the annoying effects as the sparse error and Gaussian noise, which further reduce the final ranks and which simultaneously enhance the aligning and recovering the hand written digits, both of which enable the new algorithm to be more strong in dealing with all the annoying effects.Table 2
Rank minimization evaluation criterion of the different image separation algorithms on five different real bases.
MethodHandwritten digitsDummy face imagesNatural face imagesVideo face imagesWindow imagesRASL [20]5860181168TRPCA-SVD [56]5345121616NQLSD [18]7842310TRPCA-t [57]627112Ours21111As an illustration, some visual images of the recovered handwritten digits based on the baselines algorithms are given in Figure2, from where we can cogently observe that the proposed approach recovers the handwritten images better as compared with the other four baselines. As shown in Figure 2(e), the recovered handwritten images provide clearer visual quality by properly removing the annoying dilemmas mainly outliers and heavy sparse noises. This is in agreement with the results in Table 2; then, we can further justify that the proposed algorithm is more resilient to occlusions and heavy sparse noises.Figure 2
Handwritten digits images recovery: (a) RASL; (b) TRPCA-SVD; (c) NQLSD; (d) TRPCA; (e) ours.
(a)(b)(c)(d)(e)


#### 6.2.2. Dummy Face Image Alignment

In this subsection, we consider 30 from 100 images with the size of49×49 CMU Multi-Pie [88].We can compare the newly developed approach with the state of the arts as given in Table2, from which we can cogently see that the NQLSD is better align than the RALS, TRPCA-SVD, as it considers the convex penalization with some quadratic penalty and local linear approximation. Moreover, the TRPCA is considered as the second best performance over the other three methods, this considers tensor to tensor product with a new nuclear norm to exactly recover the handwritten images. Finally, the newly developed algorithm better performs in rank minimization during the face alignments than all aforementioned baselines. This is newly incorporated the L2,1 norms, the affine transformations are also considered to tackle the misalignment problems, and the annoying effects are also further split into the sparse errors and the Gaussian noises, which ultimately reduce the final ranks.Again, for the visualization, the pictures of the true globe are with uncontrolled misalignments, under different illuminations. Figure3 shows the recovered dummy face image based on our algorithm and the other four other baseline methods. We can observe from Figure 3(e) that the proposed approach gives more clear visual quality. This is briefly the effectiveness of our method pruning out all the annoying effects.Figure 3
Dummy face image recovery: (a) RASL; (b) TRPCA-SVD; (c) NQLSD; (d) TRPCA; (e) ours.
(a)(b)(c)(d)(e)


#### 6.2.3. Natural Face Images

Following this, we conduct simulations on a very challenging natural faces from the natural faces labeled in the Wild database [87]. In this experiment, 21 natural face images with size 80×60 are considered.The comparison of the proposed method compared with the baselines is made on the natural face image recovers. Thereby, some performance results are given in Table2, from which we can recognize that NQLSD outperforms RASL, as it considers the convex penalization with quadratic penalty along with the local linear approximations to minimize the final ranks based on the distorted natural face images. TRPCA provides the second best performance, as it can cope with outliers and heavy sparse noises via considering the tensor to tensor products so as to exactly recover the low-rank components over the other three baselines. We can also search that the proposed algorithm providing the smallest final ranks compared with the other baselines, as it incorporates the affine transformations with the L2,1 norms and further decomposing the annoying effects as the sparse error and the Gaussian noise is done so as to circumvent the outliers and sparse noises in the natural face images.Again, as an illustration, some recovered natural face images based on the proposed method and the aforementioned baselines are given in Figure4. The recovered images by the proposed algorithm are shown in Figure 4(e), from which we can observe that the visual quality of the proposed method is better than all of the baselines. This is in line with the numerical results in Table 2.Figure 4
Natural face image recovery: (a) RASL; (b) TRPCA-SVD; (c) NQLSD; (d) TRPCA; (e) ours.
(a)(b)(c)(d)(e)


#### 6.2.4. Video Face Images

Finally, we conduct an experiment on more complicated face images in videos taken from the Al Gore talking [20]. In this video, 21 different video face images with size 232×312 are considered.The comparison of the final rank using proposed method and the other four baselines is given in Table2, from which we can see that the TRPCA yields the second best performance. This is because it considers tensor to tensor product with the new nuclear norm instead of using the singular value decomposition so as to minimize the tensor tubal ranks, which corresponds to the low-rank components. We can also notice from Table 2 that the proposed method still outperforms all baselines, as it includes affine transformations and utilizes the L2,1 norms to render a more robust recovery against outliers and heavy sparse noises. In addition to this, a new incorporation of the L2,1 norms’ affine transformation along with splitting the annoying effects into the sparse error and the Gaussian noises enhanced the performance of the proposed method as compared with the aforementioned baselines, which makes the new approach more robust.As an illustration, some recovered face images based on the above algorithms are provided in Figure5(e), and thus, we can cogently see that the proposed method again produces recovering images with better visual quality as compared with the other four baselines. These results are once again in agreement with the numerical results in Table 2.Figure 5
Video sequences of face image recovery: (a) RASL; (b) TRPCA-SVD; (c) NQLSD; (d) TRPCA; (e) ours.
(a)(b)(c)(d)(e)


#### 6.2.5. Results of Controlled Window Images

In this subsection, we also considered some practical experimental simulation via using sets of transformation mainly affine transformations assisted by theL2,1 norms, and our proposed method is also applied on more challenging controlled data, which is taken from the Reference [20].The comparison of the proposed method and all other four baselines is given in Table2, from which the performance of the TRPCA based on the final ranks is better as compared with the other three baselines (RASL, NQLSD, and TRPCA-SVD). This is because the TRPCA is used for further exact recovery of complicated windows since the tensor to tensor product is taken into consideration. Finally, the proposed approach is better than all the aforementioned baselines based on its final ranks, as it is an aggregated by combining the affine transformations, the L2,1 norms, and further decomposing the annoying effects as the sparse errors and the Gaussian noises. This has boosted the performance of the newly developed approach, which makes it more robust with these annoying effects.In this subsection, we further demonstrate the effectiveness of the new algorithm in aligning and recovering the complicated images. Figure6 reveals 4 images of the windows, which are taken from different corners of the camera viewpoints, and with illuminations influenced with some trees. Then, we employed the proposed method, with planar homography via an affine transformation and the L2,1 norms to correctly align the images in the frame of the canonicals. We can also see from Figure 6(e), as our proposed algorithm tackles the potential errors by removing the trees from the windows entailing as the alignment is made correctly. This example points out suggests that the proposed algorithm is very useful in recovering the most challenging images. Then, this result is also more resembled with the final ranks obtained in Table 2. This makes the newly proposed method more robust with more complicated windows with various annoying effects.Figure 6
Complicated window image recovery: (a) RASL; (b) TRPCA-SVD; (c) NQLSD; (d) TRPCA; (e) ours.
(a)(b)(c)(d)(e)


## 6.1. Experimental Convergence Performance

In this subsection, some experimental analysis of the proposed method based on four sets of images. The first set of image is based on the synthetic data, the second is based on the handwritten images, the third one is based on the Natural Faces labeled in the Wild database [87], and the fourth is based on the video face images, and the convergence characteristic is assessed by comparing the relative square error (RSE) vs. the iteration number, where RSE is defined as(18)C^−CFCF,in which C^ is the recovered image and C is the original one. Also, we set λ1=1/minm,n as suggested in Reference [89–92]. The resulting convergence curves are as shown in Figure 1, from which we can see that the RSEs of the proposed algorithm for all images decrease with the iteration number and then reach at a constant after a few iterations. This fact justifies the convergence of the proposed algorithm.Figure 1
Convergence curves of the proposed algorithm on: (a) synthetic data; (b) handwritten digits images; (c) natural face images; (d) video face images.
(a)(b)(c)(d)


## 6.2. Comparison with Baseline Approach

In this subsection, I made a comparison with the proposed approach, which adds in the affine transformations with theL2,1 norm, with some recently reported works in terms of the quantitative measures of the final ranks based on five different databases. Four different baselines approaches, including RASL [20], TRPCA-SVD [56], NQLSD [18], and TRPCA-t [57], and the proposed algorithm are conducted for comparison. Moreover, there are several works conducted in the areas of signal processing. For instance, Reference [26–28] proposed a supervised approach to prune out the annoying effects; however, they lack performance. Then, the performance results from these baselines are obtained based on publicly available codes.


### 6.2.1. Handwritten Digits

In this experiment, 30 number of digits are taken from the MINST dataset [86] with sizes 29×29.We compare the final rank being considered as the evaluation criteria of the proposed approach with the aforementioned four baselines, as given in Table2, from which we can realize that NQLSD has better results than RASL, as NQLSD employs the local linear approximation with a quadratic penalty approach to tackle the potential setback of outliers and sparse noises in the images. Moreover, the TRPCA has enhanced the recovery over the other approaches, as it considers tensor to tensor products instead of singular value decomposition to exactly recover the low ranks. However, still it lacks robustness as the L2,1 norm is not taken into consideration. We can also observe from Table 2 that the suggested approach provides the best achievement. This is because it incorporates a set of affine transformations and employs the L2,1 norms, the approach also splits the annoying effects as the sparse error and Gaussian noise, which further reduce the final ranks and which simultaneously enhance the aligning and recovering the hand written digits, both of which enable the new algorithm to be more strong in dealing with all the annoying effects.Table 2
Rank minimization evaluation criterion of the different image separation algorithms on five different real bases.
MethodHandwritten digitsDummy face imagesNatural face imagesVideo face imagesWindow imagesRASL [20]5860181168TRPCA-SVD [56]5345121616NQLSD [18]7842310TRPCA-t [57]627112Ours21111As an illustration, some visual images of the recovered handwritten digits based on the baselines algorithms are given in Figure2, from where we can cogently observe that the proposed approach recovers the handwritten images better as compared with the other four baselines. As shown in Figure 2(e), the recovered handwritten images provide clearer visual quality by properly removing the annoying dilemmas mainly outliers and heavy sparse noises. This is in agreement with the results in Table 2; then, we can further justify that the proposed algorithm is more resilient to occlusions and heavy sparse noises.Figure 2
Handwritten digits images recovery: (a) RASL; (b) TRPCA-SVD; (c) NQLSD; (d) TRPCA; (e) ours.
(a)(b)(c)(d)(e)


### 6.2.2. Dummy Face Image Alignment

In this subsection, we consider 30 from 100 images with the size of49×49 CMU Multi-Pie [88].We can compare the newly developed approach with the state of the arts as given in Table2, from which we can cogently see that the NQLSD is better align than the RALS, TRPCA-SVD, as it considers the convex penalization with some quadratic penalty and local linear approximation. Moreover, the TRPCA is considered as the second best performance over the other three methods, this considers tensor to tensor product with a new nuclear norm to exactly recover the handwritten images. Finally, the newly developed algorithm better performs in rank minimization during the face alignments than all aforementioned baselines. This is newly incorporated the L2,1 norms, the affine transformations are also considered to tackle the misalignment problems, and the annoying effects are also further split into the sparse errors and the Gaussian noises, which ultimately reduce the final ranks.Again, for the visualization, the pictures of the true globe are with uncontrolled misalignments, under different illuminations. Figure3 shows the recovered dummy face image based on our algorithm and the other four other baseline methods. We can observe from Figure 3(e) that the proposed approach gives more clear visual quality. This is briefly the effectiveness of our method pruning out all the annoying effects.Figure 3
Dummy face image recovery: (a) RASL; (b) TRPCA-SVD; (c) NQLSD; (d) TRPCA; (e) ours.
(a)(b)(c)(d)(e)


### 6.2.3. Natural Face Images

Following this, we conduct simulations on a very challenging natural faces from the natural faces labeled in the Wild database [87]. In this experiment, 21 natural face images with size 80×60 are considered.The comparison of the proposed method compared with the baselines is made on the natural face image recovers. Thereby, some performance results are given in Table2, from which we can recognize that NQLSD outperforms RASL, as it considers the convex penalization with quadratic penalty along with the local linear approximations to minimize the final ranks based on the distorted natural face images. TRPCA provides the second best performance, as it can cope with outliers and heavy sparse noises via considering the tensor to tensor products so as to exactly recover the low-rank components over the other three baselines. We can also search that the proposed algorithm providing the smallest final ranks compared with the other baselines, as it incorporates the affine transformations with the L2,1 norms and further decomposing the annoying effects as the sparse error and the Gaussian noise is done so as to circumvent the outliers and sparse noises in the natural face images.Again, as an illustration, some recovered natural face images based on the proposed method and the aforementioned baselines are given in Figure4. The recovered images by the proposed algorithm are shown in Figure 4(e), from which we can observe that the visual quality of the proposed method is better than all of the baselines. This is in line with the numerical results in Table 2.Figure 4
Natural face image recovery: (a) RASL; (b) TRPCA-SVD; (c) NQLSD; (d) TRPCA; (e) ours.
(a)(b)(c)(d)(e)


### 6.2.4. Video Face Images

Finally, we conduct an experiment on more complicated face images in videos taken from the Al Gore talking [20]. In this video, 21 different video face images with size 232×312 are considered.The comparison of the final rank using proposed method and the other four baselines is given in Table2, from which we can see that the TRPCA yields the second best performance. This is because it considers tensor to tensor product with the new nuclear norm instead of using the singular value decomposition so as to minimize the tensor tubal ranks, which corresponds to the low-rank components. We can also notice from Table 2 that the proposed method still outperforms all baselines, as it includes affine transformations and utilizes the L2,1 norms to render a more robust recovery against outliers and heavy sparse noises. In addition to this, a new incorporation of the L2,1 norms’ affine transformation along with splitting the annoying effects into the sparse error and the Gaussian noises enhanced the performance of the proposed method as compared with the aforementioned baselines, which makes the new approach more robust.As an illustration, some recovered face images based on the above algorithms are provided in Figure5(e), and thus, we can cogently see that the proposed method again produces recovering images with better visual quality as compared with the other four baselines. These results are once again in agreement with the numerical results in Table 2.Figure 5
Video sequences of face image recovery: (a) RASL; (b) TRPCA-SVD; (c) NQLSD; (d) TRPCA; (e) ours.
(a)(b)(c)(d)(e)


### 6.2.5. Results of Controlled Window Images

In this subsection, we also considered some practical experimental simulation via using sets of transformation mainly affine transformations assisted by theL2,1 norms, and our proposed method is also applied on more challenging controlled data, which is taken from the Reference [20].The comparison of the proposed method and all other four baselines is given in Table2, from which the performance of the TRPCA based on the final ranks is better as compared with the other three baselines (RASL, NQLSD, and TRPCA-SVD). This is because the TRPCA is used for further exact recovery of complicated windows since the tensor to tensor product is taken into consideration. Finally, the proposed approach is better than all the aforementioned baselines based on its final ranks, as it is an aggregated by combining the affine transformations, the L2,1 norms, and further decomposing the annoying effects as the sparse errors and the Gaussian noises. This has boosted the performance of the newly developed approach, which makes it more robust with these annoying effects.In this subsection, we further demonstrate the effectiveness of the new algorithm in aligning and recovering the complicated images. Figure6 reveals 4 images of the windows, which are taken from different corners of the camera viewpoints, and with illuminations influenced with some trees. Then, we employed the proposed method, with planar homography via an affine transformation and the L2,1 norms to correctly align the images in the frame of the canonicals. We can also see from Figure 6(e), as our proposed algorithm tackles the potential errors by removing the trees from the windows entailing as the alignment is made correctly. This example points out suggests that the proposed algorithm is very useful in recovering the most challenging images. Then, this result is also more resembled with the final ranks obtained in Table 2. This makes the newly proposed method more robust with more complicated windows with various annoying effects.Figure 6
Complicated window image recovery: (a) RASL; (b) TRPCA-SVD; (c) NQLSD; (d) TRPCA; (e) ours.
(a)(b)(c)(d)(e)


## 6.2.1. Handwritten Digits

In this experiment, 30 number of digits are taken from the MINST dataset [86] with sizes 29×29.We compare the final rank being considered as the evaluation criteria of the proposed approach with the aforementioned four baselines, as given in Table2, from which we can realize that NQLSD has better results than RASL, as NQLSD employs the local linear approximation with a quadratic penalty approach to tackle the potential setback of outliers and sparse noises in the images. Moreover, the TRPCA has enhanced the recovery over the other approaches, as it considers tensor to tensor products instead of singular value decomposition to exactly recover the low ranks. However, still it lacks robustness as the L2,1 norm is not taken into consideration. We can also observe from Table 2 that the suggested approach provides the best achievement. This is because it incorporates a set of affine transformations and employs the L2,1 norms, the approach also splits the annoying effects as the sparse error and Gaussian noise, which further reduce the final ranks and which simultaneously enhance the aligning and recovering the hand written digits, both of which enable the new algorithm to be more strong in dealing with all the annoying effects.Table 2
Rank minimization evaluation criterion of the different image separation algorithms on five different real bases.
MethodHandwritten digitsDummy face imagesNatural face imagesVideo face imagesWindow imagesRASL [20]5860181168TRPCA-SVD [56]5345121616NQLSD [18]7842310TRPCA-t [57]627112Ours21111As an illustration, some visual images of the recovered handwritten digits based on the baselines algorithms are given in Figure2, from where we can cogently observe that the proposed approach recovers the handwritten images better as compared with the other four baselines. As shown in Figure 2(e), the recovered handwritten images provide clearer visual quality by properly removing the annoying dilemmas mainly outliers and heavy sparse noises. This is in agreement with the results in Table 2; then, we can further justify that the proposed algorithm is more resilient to occlusions and heavy sparse noises.Figure 2
Handwritten digits images recovery: (a) RASL; (b) TRPCA-SVD; (c) NQLSD; (d) TRPCA; (e) ours.
(a)(b)(c)(d)(e)


## 6.2.2. Dummy Face Image Alignment

In this subsection, we consider 30 from 100 images with the size of49×49 CMU Multi-Pie [88].We can compare the newly developed approach with the state of the arts as given in Table2, from which we can cogently see that the NQLSD is better align than the RALS, TRPCA-SVD, as it considers the convex penalization with some quadratic penalty and local linear approximation. Moreover, the TRPCA is considered as the second best performance over the other three methods, this considers tensor to tensor product with a new nuclear norm to exactly recover the handwritten images. Finally, the newly developed algorithm better performs in rank minimization during the face alignments than all aforementioned baselines. This is newly incorporated the L2,1 norms, the affine transformations are also considered to tackle the misalignment problems, and the annoying effects are also further split into the sparse errors and the Gaussian noises, which ultimately reduce the final ranks.Again, for the visualization, the pictures of the true globe are with uncontrolled misalignments, under different illuminations. Figure3 shows the recovered dummy face image based on our algorithm and the other four other baseline methods. We can observe from Figure 3(e) that the proposed approach gives more clear visual quality. This is briefly the effectiveness of our method pruning out all the annoying effects.Figure 3
Dummy face image recovery: (a) RASL; (b) TRPCA-SVD; (c) NQLSD; (d) TRPCA; (e) ours.
(a)(b)(c)(d)(e)


## 6.2.3. Natural Face Images

Following this, we conduct simulations on a very challenging natural faces from the natural faces labeled in the Wild database [87]. In this experiment, 21 natural face images with size 80×60 are considered.The comparison of the proposed method compared with the baselines is made on the natural face image recovers. Thereby, some performance results are given in Table2, from which we can recognize that NQLSD outperforms RASL, as it considers the convex penalization with quadratic penalty along with the local linear approximations to minimize the final ranks based on the distorted natural face images. TRPCA provides the second best performance, as it can cope with outliers and heavy sparse noises via considering the tensor to tensor products so as to exactly recover the low-rank components over the other three baselines. We can also search that the proposed algorithm providing the smallest final ranks compared with the other baselines, as it incorporates the affine transformations with the L2,1 norms and further decomposing the annoying effects as the sparse error and the Gaussian noise is done so as to circumvent the outliers and sparse noises in the natural face images.Again, as an illustration, some recovered natural face images based on the proposed method and the aforementioned baselines are given in Figure4. The recovered images by the proposed algorithm are shown in Figure 4(e), from which we can observe that the visual quality of the proposed method is better than all of the baselines. This is in line with the numerical results in Table 2.Figure 4
Natural face image recovery: (a) RASL; (b) TRPCA-SVD; (c) NQLSD; (d) TRPCA; (e) ours.
(a)(b)(c)(d)(e)


## 6.2.4. Video Face Images

Finally, we conduct an experiment on more complicated face images in videos taken from the Al Gore talking [20]. In this video, 21 different video face images with size 232×312 are considered.The comparison of the final rank using proposed method and the other four baselines is given in Table2, from which we can see that the TRPCA yields the second best performance. This is because it considers tensor to tensor product with the new nuclear norm instead of using the singular value decomposition so as to minimize the tensor tubal ranks, which corresponds to the low-rank components. We can also notice from Table 2 that the proposed method still outperforms all baselines, as it includes affine transformations and utilizes the L2,1 norms to render a more robust recovery against outliers and heavy sparse noises. In addition to this, a new incorporation of the L2,1 norms’ affine transformation along with splitting the annoying effects into the sparse error and the Gaussian noises enhanced the performance of the proposed method as compared with the aforementioned baselines, which makes the new approach more robust.As an illustration, some recovered face images based on the above algorithms are provided in Figure5(e), and thus, we can cogently see that the proposed method again produces recovering images with better visual quality as compared with the other four baselines. These results are once again in agreement with the numerical results in Table 2.Figure 5
Video sequences of face image recovery: (a) RASL; (b) TRPCA-SVD; (c) NQLSD; (d) TRPCA; (e) ours.
(a)(b)(c)(d)(e)


## 6.2.5. Results of Controlled Window Images

In this subsection, we also considered some practical experimental simulation via using sets of transformation mainly affine transformations assisted by theL2,1 norms, and our proposed method is also applied on more challenging controlled data, which is taken from the Reference [20].The comparison of the proposed method and all other four baselines is given in Table2, from which the performance of the TRPCA based on the final ranks is better as compared with the other three baselines (RASL, NQLSD, and TRPCA-SVD). This is because the TRPCA is used for further exact recovery of complicated windows since the tensor to tensor product is taken into consideration. Finally, the proposed approach is better than all the aforementioned baselines based on its final ranks, as it is an aggregated by combining the affine transformations, the L2,1 norms, and further decomposing the annoying effects as the sparse errors and the Gaussian noises. This has boosted the performance of the newly developed approach, which makes it more robust with these annoying effects.In this subsection, we further demonstrate the effectiveness of the new algorithm in aligning and recovering the complicated images. Figure6 reveals 4 images of the windows, which are taken from different corners of the camera viewpoints, and with illuminations influenced with some trees. Then, we employed the proposed method, with planar homography via an affine transformation and the L2,1 norms to correctly align the images in the frame of the canonicals. We can also see from Figure 6(e), as our proposed algorithm tackles the potential errors by removing the trees from the windows entailing as the alignment is made correctly. This example points out suggests that the proposed algorithm is very useful in recovering the most challenging images. Then, this result is also more resembled with the final ranks obtained in Table 2. This makes the newly proposed method more robust with more complicated windows with various annoying effects.Figure 6
Complicated window image recovery: (a) RASL; (b) TRPCA-SVD; (c) NQLSD; (d) TRPCA; (e) ours.
(a)(b)(c)(d)(e)


## 7. Computational Complexity

The time complexity of the proposed method as compared to the state-of-the-art works is described in this section. On a very standard desktop computer, the computational load of the baselines mainly RASL [20], TRPCA-SVD [56], NQLSD [18], and TRPCA-t [57] along with the proposed method is given in Table 3, from which we note that the proposed method has small number of running time due to the proposed method as the number of parameters is small and the novel ideas also better align the images, which make the proposed method fast. Moreover, the new method handles batches of images within a few minutes as compared with the baselines, which makes the new method to be guaranteed faster time complexity compared with the baselines, as shown in Table 3.Table 3
Running time of the proposed method compared with the baselines in minutes
MethodHandwritten digitsDummy face imagesRASL [20]4.354.23TRPCA-SVD [56]3.002.95NQLSD [18]2.402.30TRPCA-t [57]2.502.12Ours2.101.90


## 8. Conclusions

In this paper, an efficient new model is proposed via affine transformation andL2,1 norms developed to minimize the ranks, align, and recovery the images. The searching of affine transformations with the L2,1 norms incorporated in the latest work, and the optimizing parameters in the formulation are considered as a convex problem. As such, the potential impacts of outliers, heavy sparse noises, occlusions, and illumination are removed from the original datasets. The convex constrained optimization problem is solved via ADMM technique and all updating parameters are solved in a round-robin manner. Thereby, the convergence of these new updating equations is scrutinized as well. We also validated the convergence of the proposed method on all data considered; then, the curves are converging. Conducted simulations show that the new method outperforms to the state of the art of the works on five public datasets.



---

*Source: 5682492-2022-08-02.xml*