# New Robust Part-Based Model with Affine Transformations for Facial Landmark Localization and Detection in Big Data

Authors: Chentao Zhang; Habte Tadesse Likassa; Peidong Liang; Jielong Guo
Journal: Modelling and Simulation in Engineering
 (2021)

---

## Abstract

In this paper, we developed a new robust part-based model for facial landmark localization and detection via affine transformation. In contrast to the existing works, the new algorithm incorporates affine transformations with the robust regression to tackle the potential effects of outliers and heavy sparse noises, occlusions and illuminations. As such, the distorted or misaligned objects can be rectified by affine transformations and the patterns of occlusions and outliers can be explicitly separated from the true underlying objects in big data. Moreover, the search of the optimal parameters and affine transformations is cast as a constrained optimization programming. To mitigate the computations, a new set of equations is derived to update the parameters involved and the affine transformations iteratively in a round-robin manner. Our way to update the parameters compared to the state of the art of the works is relatively better, as we employ a fast alternating direction method for multiplier (ADMM) algorithm that solves the parameters separately. Simulations show that the proposed method outperforms the state-of-the-art works on facial landmark localization and detection on the COFW, HELEN, and LFPW datasets.


---

## Body

## 1. Introduction

Localization of detailed facial features arises in a variety of applications including occlusion coherence for prediction in high-dimensional images [1], landmark localization [2–4], head pose estimation [5, 6], face image alignment [7], low-rank estimation [8–10], and object detection [11–14], etc. However, high-dimensional data are easily impacted by outliers and occlusions. It is thus of importance to develop a deep robust part-based models combining the robust regression algorithm with affine transformation for separating the landmark localization from the occlusions as in [15] and detection [16]. Thus, developing a new algorithm that is resilient with adverse effects while processing big data particularly for face detection and landmark localization is highly indispensable.A number of algorithms have been suggested for robust facial landmark localization and object detection [17–24]. For instance, [18, 25, 26] addressed these problems using a regression approach. However, these approaches are mostly an approximate and, additionally, they are no longer pose-invariant. To improve the performance of the algorithm and overcome the aforementioned setback, deformable part models (DPMs) based on high-dimensional images [27–30] were taken into account. However, they can not deal with a huge amount of outliers and occlusions. Zhang et al. [31] proposed a tasks-constrained deep convolutional network (TCDCN) through multitask learning which directly conducted the simulations based on high-dimensional images. However, it ignores the potential annoying effects and the correlation between images which jeopardize the robustness of the algorithm. Plus, the issue of the misalignment problem is also not tackled. Moreover, to tackle the misalignment problem, [8, 9, 32–34] addressed several algorithms via affine transformation and the L2,1 norms. To circumvent this dilemma, Martinez et al. [35] considered a robust facial landmark detection (RFLD) based on high-dimensional image data via L2,1 norm regularization against poor initializations. This approach, however, may lead to a suboptimal solution by performing a dimension reduction and a regression process. Zhang et al. [36] addressed joint face detection alignment using multitask cascaded convolutional networks and [37] feature selection with multiview data; however, affine transformation is not taken into consideration. While [38] suggested a fast and accurate facial landmark detection network through using an augmented (EMTCNN) instead of [36], yet the performance is not promising. To boost the performance of the algorithms, [38–40] proposed a novel deep network method for detecting facial region and landmarks to learn from low-dimensional image representations; however, the proposed techniques seek affine transformation to deal with occlusions and illuminations. To tackle this dilemma, [8, 32, 33] proposed new methods assisted via affine transformation in high-dimensional images to estimate the optimal parameters corresponding to the low rank recovery.To deal with outliers and occlusions, Tzimiropoulos and Pantic [41] proposed an active appearance model via a fast simultaneous inverse composition (Fast-SIC) based on image datasets for landmark localization by considering the shape and appearance of images to enhance the face fitting accuracy. However, its localization outputs are distorted due to the presence of outliers and occlusions that is also why the performance of the method is not promising. To solve this problem, [1, 42] proposed hierarchical part models (HPMs) relying on high-dimensional images to explicitly model partial occlusions and detect facial points and occlusions simultaneously based on high-dimensional images using the landmark localization error. In addition to this, [43, 44] proposed a latent multiview self-representations for clustering via the tensor nuclear norm, while [44] addressed the robust sparse low-rank embedding for image dimension reduction. But, in real scenarios, the outliers and occlusion patterns can be very diverse and almost unpredictable. Nada et al. [45] considered a new annotated unconstrained face detector (UF-DD) to enhance the performance; however, it lacks robustness to large-scale variations and is required to perform an exhaustive search of the optimal transformation to obtain satisfactory performance.


## 2. Characteristics of the State-of-the-Art Methods

In this section, we illustrate the characteristics of the five state-of-the-art works such as TCDCN [31], RFLD [35], Fast-SIC [41], HPM [42], and UF-DD [45] with the proposed approach, which further adds affine transformations and multiple subspaces. The merits of the proposed method are quite different from the baselines in terms of its novelity and updating the parameters. For instance, TCDCN [31] addressed the idea of facial land mark location to deal with occlusion and pose variation via deep multitask learning, but it yet requires to incorporate affine transformation to explore deep multitask learning in dense landmark detection and high-dimensional image recovery. This method considers a few regularization parameters with about six different labels that fail to penalize the complexity of the weights. To tackle the dilemmas of gross errors and variation in images, RFLD [35] proposed a novel regression method that substitutes the commonly used least squares regression developed by [31] via the use of the L2,1 norm, but it lacks performance when there are a large number of landmarks. Moreover, there are seven numbers of parameters taken into account, which increases the computational complexity of the method. To alleviate the problems in facial deformable models for the unconstrained images, Fast-SIC was developed by [41], where six different parameters including the vector appearance, localization, and warping parameters involved in problem formulation, which makes the method difficult to optimize the parameters. To solve this setback, [42] proposed HPM for facial landmark localization and occlusion estimation to boost the performance, with five parameters and a number of network hierarchical structure of layers which is increasing the computational complexity of the algorithm. To tackle the setback of outliers and heavy sparse noises, occulusions and illuminations, [45] proposed a novel UF-DD method for face detection where five numbers of parameters are involved; however, the impacts of large outliers and heavy sparse noises are not alleviated. As a new annotated unconstrained face detector is addressed in [45], the time complexity is relatively better over RFLD [35], Fast-SIC [41], and HPM [42]. In the latest work, there are about seven number of parameters involved including affine transformation. As affine transformation is newly added in the new model, it shares the advantage of pruning out the potential impacts of occlusions and illuminations, outliers, and heavy sparse noises in landmark localization’s and facial detection from which we note that the time complexity of the proposed method is highly reduced and the performance is enhanced. Another advantage of the new part-based model over the state-of-the-art works is it considers multiple subspace parameters to constrain each parameters and shares the advantages of positive semidefinite. However, the new technique seeks to establish the spatial dependency between different images that can be captured incorporating the spatial weight matrix, as such the performance of the new part-based model be boosted.In this paper, we present a new robust regression algorithm for facial landmark localization and face detection. Invoked by the novel idea of affine transformations in [8], and to be more resilient to occlusions and outliers, the new algorithm incorporates affine transformations with the robust regression based on the hierarchical part-based models [1, 42]. As such, the distorted or misaligned images can be rectified by affine transformations and the patterns of occlusions and outliers can be explicitly separated. The proposed method via affine transformation contributes in clearly separating the facial landmark localization and occlusion estimation. The search of the optimal parameters and the affine transformations is cast as a constrained optimization programming. To mitigate the computational overhead, a new set of equations are derived to update the parameters involved and the affine transformations iteratively using ADMM approach in a round-robin manner. Conducted simulations show that the proposed method outperforms the state-of-the-art works on face detection and landmark localization on some common on the COFW, HELEN, and LFPW datasets. The major contributions of this work include the following:
(1)
The affine transformations are incorporated in the robust regression based on the part-based models to take advantages of both schemes in the learning process(2)
The affine transformations are aggregated with the low-rank-sparse representation, where the low-rank component lies in a union of subspaces instead of one single subspace. These transformations can fix the distortion or misalignment in a batch of corrupted images to render more faithful image decomposition, thereby being more robust against heavy sparse errors and outliers(3)
The ADMM method is employed to solve the new convex optimization problem, and a set of updating equations is derived to iteratively update the optimization variables and the affine transformations(4)
New set of updating equations is established to iteratively solve the constrained optimization problem(5)
We conduct experiments on several benchmark datasets, and the experimental results demonstrate the effectiveness of our new method


## 3. Problem Formulation

Givenn images Ii0∈Rw×h,i=1,⋯,n, all of which contain the same object and are linearly correlated, where w and h denote the weight and height of the images, respectively. Based on the part-based models [1], we approximate A=vecI10⋯vecIn0∈Rl×n by a shape parameter, where vec· is the vector stacking operator [46] and l is the number of landmarks. However, in practice, the images Ii0 are usually contaminated by occlusions and outliers, so the images M∈Rl×n can be represented as M=AB+E, where M∈Rl×n is a corrupted high-dimensional images, E∈Rl×n is a term accounting for the occlusions and outliers, and B∈Rn×n is an appearance parameter.To solve the misalignment problem incurred by the outliers and occlusions inM, the original images need to be linearized, which can be done by affine transformations. Applying a set of affine transformations, denoted by τ1, ...,τn to M [33, 47], then, we then have Mτo, where Mτo denotes the transformed data. Assume that the differences between the consecutive affine transformations are small, so we can approximate Mτo by Mτo+∑i=1nJiΔτviviT, Δτ∈Rq×n, in which q denotes the number of parameters, Ji=∂vecIiτi/∂τi∈Rl×p denotes the Jacobian of the ith training images with respect to τi, and vi denotes the standard basis for Rn. Thus, consequently, the main objective is to minimize the localization error via incorporating a novel idea of affine transformation for better performance, so the overall problem can thus be posted as the following constrained optimization problem
(1)minA,L,β^,E,B,Δτγ2UF2+A∗+B∗+λ1G1+λ2E1,(2)s.tQ=AB+E,β^=AB;1T,B=G,G±0,where U=WR−Lβ^, Q=Mτo+∑i=1nJiΔτviviT, W∈Rdr×dr is a weight matrix, L∈Rdr×l+1 is the regression matrix mapping Mτo∈Rl×n to R∈Rdr×n, β^∈Rl+1×n is a regression coefficient that controls the appearance variation, 1nT∈Rn denotes as a vector of all ones, E∈Rl×n, λ1, γ and λ2 are the regularization parameters, A∗=∑i=1minl,nσiA with σiA being the singular values of A, AF2=TraceATA, and E1=max1≤j≤n∑i=1lEij.


### 3.1. Proposed Approach

To solve a constrained optimization problem (2), we consider an ADMM given by
(3)minL,β^,A,E,B,Δτγ2UF2+A∗+B∗+λ1G1+λ2E1+Y1,C+μ12CF2+Y2,β^−AB;1T+μ22β^−AB;1TF2+Y3,B−G+μ32B−GF2,where Y1∈Rl×n, Y2∈Rl+1×n, and Y3∈Rn×n are the Lagrangian multipliers, C=Q−AB−E, μ1, μ2, and μ3 are the penalty parameters, and X,Y=TraceXTY. This can be solved by using ADMM. The major necessary condition of ADMM is the convexity issue; we give sufficient conditions under which the algorithm asymptotically reaches the standard first-order necessary conditions for local optimality.Then, based on a linearized alternating direction method [48], the augmented Lagrangian multiplier in (3) can be reexpressed as
(4)minL,β^,A,E,B,G,Δτγ2UF2+A∗+B∗+λ1G1+λ2E1+μ12C+Y1μ1F2+μ22V+Y2μ2F2+μ32B−G+Y3μ3F2,whereV=β^−AB;1T. Directly solving (4) is computationally expensive, so to follow, we consider a set of equations to iteratively update the parameters in (4). So typically, we can choose to minimize the Lagrangian function to update all involved parameters involved alternatively.Firstly, the updates ofL and β^ are determined, respectively, by
(5)Lk+1=argminLLLk,β∧k,Ak,Ek,Bk,Gk,Δτkβ∧k+1=argminβ^LLk+1,β∧k,Ak,Ek,Bk,Gk,Δτk,where k is the index of iterations, and by ignoring all irrelevant terms and applying the ordinary least square regression procedures on L and β^, finally, the updates of L and β^ are given by
(6)Lk+1=β∧β∧T+γIdm+1−1Rβ∧Tβ∧k+1=NL+μ2Id−1NR−Z,where N=γLTWTW and Z=Y2+μ2AB;1T and Idm+1 is an identity matrix.Secondly, to update the shape parameterA, we fix E, L, β^, G, B, and Δτ remain as constants, and Ak+1 can be determined by
(7)Ak+1=argminALLk+1,β∧k+1,Ak,Ek,Bk,Gk,Δτk.Using the soft shrinkage operator, we can get the update ofA as
(8)Ak+1=Ω1/κAAk−PAkκA,where Ω is the singular value threshold, PAk=μ1+μ2ATB−μ1Q−E−Y1−μ2β^+Y2BT, κA=μ1+μ2ηA/2, ηA>σBTB is the proximal parameter, in which σBTB denotes the spectral radius of BTB.Thirdly, to updateE, we fix β^, A, L, G, B, and Δτ, and Ek+1 can be determined by
(9)Ek+1=argminELLk+1,β∧k,Ak,Ek,Bk,Gk,Δτk.By invoking a linearized alternating direction,E is updated by
(10)Ek+1=Γλ1/μ1Q−AkB+Y1μ1,where Γ is the soft shrinkage operator [49, 50].Next, to get the optimal update ofB, we again keep all A, E, L, β^, G, and Δτ as constants, then the updating paramter B can be determined by
(11)Bk+1=argminBLLk+1,β∧k+1,Ak+1,Ek+1,Bk,Gk,Δτk.Then, by employing a singular value threshold operators, the update ofB is given by
(12)Bk+1=Ω1κBBk−PBkκB,where κB=μ1+μ2+μ3ηB/2,ηB>σAAT and PBk=Akμ1+μ2Ak+1−μ1Q−E−Y1−μ2β^+Y2+μ3Bk−Gk+Y3.To find the optimal update ofG, we fix A, E, L, β^, B, and Δτ as constants, then Gk+1 can be obtained by
(13)Gk+1=argminGLLk+1,β∧k+1,Ak+1,Ek+1,Bk+1,Gk,Δτk.Similarly, employing the soft threshold operator and an augmented Lagrangian multiplier to subproblem (14), then finally, G is updated by
(14)Gk+1=Γλ2/μ3Bk+1+Y3μ3.Invoked with an affine transformations, we obtained an additional optimal parameter. Therefore, we need to derive an additional update forΔτ.To do so, we updateΔτ, we fix A, E, L, β^, B, and G, and Δτk+1 can be obtained by
(15)Δτk+1=argminΔτLLk+1,β∧k+1,Ak+1,Ek+1,Bk+1,Gk+1,Δτk.Along the same line, the affine transformationsΔτ can be updated by
(16)Δτk+1=∑i=1nχi+Ak+1Bk+1+Ek+1−Mτo−Y1μ1viviT,where χi+ is a More-Penrose-pseudoinverse of χi.Similarly, the updates ofY1, Y2, and Y3 are given, respectively, by
(17)Y1k+1=Y1k+μ1k+1Q−Ak+1Bk+1−Ek+1Y2k+1=Y2k+μ2k+1β^−Ak+1Bk+1;1k+1Y3k+1=Y3k+μ3k+1Bk+1−Gk+1.The updates of the regularization parametersμ1, μ2, and μ3 are given by
(18)μ1k+1=minμmax,ρμ1kμ2k+1=minμmax,ρμ2kandμ3k+1=minμmax,ρμ3k,where ρ is a properly chosen constant and μmax are tunable parameters adjusting the convergence of the proposed algorithm. We sequentially update L, β^, A, E, B, G, and Δτ independently by keeping all of the other parameters unchanged. First, the regression matrix L and β^ are updated by (6). Next, we can update A, E, B, G, and the optimal affine transformation parameters, Δτ, by (8), (10), (12), (14), and (16), respectively. Finally, the Lagrangian multipliers Y1, Y2, and Y3 are updated by (17) and the regularization parameters μ1, μ2, and μ3 are updated by (18). These new updating equations proceed in a round-robin manner until convergence. Since a monotonically decreasing and bounded below sequence will convergence, the above algorithm is ensured to converge. It is noteworthy that based on the augmented Lagrangian multiplier, ordinary least squares procedures, and soft-threshold operators, all the updating parameters are achieved. The overall summary of the proposed method compared with other related works is summarized in Table 1.Table 1
Comparison of the proposed approach with other related algorithms in terms of the objective function and constraints.
MethodsObjective functionConstraints[51–55]minA,EA∗+λE1M=A+E[56–60]minA,EA∗+λE1Moτ+∑i=1nJiΔτviviT=A+E[61–64]minB,E,GB∗+λE2,1M=AB+E,B=GOursminA,L,β^E,B,∆τγ/2WR−Lβ^F2+A∗+B∗+⋋1G1+⋋2E1Mτo+∑i=1nJi∆τviviT=AB+E,β^=AB;1T,B=G,G≽0


## 3.1. Proposed Approach

To solve a constrained optimization problem (2), we consider an ADMM given by
(3)minL,β^,A,E,B,Δτγ2UF2+A∗+B∗+λ1G1+λ2E1+Y1,C+μ12CF2+Y2,β^−AB;1T+μ22β^−AB;1TF2+Y3,B−G+μ32B−GF2,where Y1∈Rl×n, Y2∈Rl+1×n, and Y3∈Rn×n are the Lagrangian multipliers, C=Q−AB−E, μ1, μ2, and μ3 are the penalty parameters, and X,Y=TraceXTY. This can be solved by using ADMM. The major necessary condition of ADMM is the convexity issue; we give sufficient conditions under which the algorithm asymptotically reaches the standard first-order necessary conditions for local optimality.Then, based on a linearized alternating direction method [48], the augmented Lagrangian multiplier in (3) can be reexpressed as
(4)minL,β^,A,E,B,G,Δτγ2UF2+A∗+B∗+λ1G1+λ2E1+μ12C+Y1μ1F2+μ22V+Y2μ2F2+μ32B−G+Y3μ3F2,whereV=β^−AB;1T. Directly solving (4) is computationally expensive, so to follow, we consider a set of equations to iteratively update the parameters in (4). So typically, we can choose to minimize the Lagrangian function to update all involved parameters involved alternatively.Firstly, the updates ofL and β^ are determined, respectively, by
(5)Lk+1=argminLLLk,β∧k,Ak,Ek,Bk,Gk,Δτkβ∧k+1=argminβ^LLk+1,β∧k,Ak,Ek,Bk,Gk,Δτk,where k is the index of iterations, and by ignoring all irrelevant terms and applying the ordinary least square regression procedures on L and β^, finally, the updates of L and β^ are given by
(6)Lk+1=β∧β∧T+γIdm+1−1Rβ∧Tβ∧k+1=NL+μ2Id−1NR−Z,where N=γLTWTW and Z=Y2+μ2AB;1T and Idm+1 is an identity matrix.Secondly, to update the shape parameterA, we fix E, L, β^, G, B, and Δτ remain as constants, and Ak+1 can be determined by
(7)Ak+1=argminALLk+1,β∧k+1,Ak,Ek,Bk,Gk,Δτk.Using the soft shrinkage operator, we can get the update ofA as
(8)Ak+1=Ω1/κAAk−PAkκA,where Ω is the singular value threshold, PAk=μ1+μ2ATB−μ1Q−E−Y1−μ2β^+Y2BT, κA=μ1+μ2ηA/2, ηA>σBTB is the proximal parameter, in which σBTB denotes the spectral radius of BTB.Thirdly, to updateE, we fix β^, A, L, G, B, and Δτ, and Ek+1 can be determined by
(9)Ek+1=argminELLk+1,β∧k,Ak,Ek,Bk,Gk,Δτk.By invoking a linearized alternating direction,E is updated by
(10)Ek+1=Γλ1/μ1Q−AkB+Y1μ1,where Γ is the soft shrinkage operator [49, 50].Next, to get the optimal update ofB, we again keep all A, E, L, β^, G, and Δτ as constants, then the updating paramter B can be determined by
(11)Bk+1=argminBLLk+1,β∧k+1,Ak+1,Ek+1,Bk,Gk,Δτk.Then, by employing a singular value threshold operators, the update ofB is given by
(12)Bk+1=Ω1κBBk−PBkκB,where κB=μ1+μ2+μ3ηB/2,ηB>σAAT and PBk=Akμ1+μ2Ak+1−μ1Q−E−Y1−μ2β^+Y2+μ3Bk−Gk+Y3.To find the optimal update ofG, we fix A, E, L, β^, B, and Δτ as constants, then Gk+1 can be obtained by
(13)Gk+1=argminGLLk+1,β∧k+1,Ak+1,Ek+1,Bk+1,Gk,Δτk.Similarly, employing the soft threshold operator and an augmented Lagrangian multiplier to subproblem (14), then finally, G is updated by
(14)Gk+1=Γλ2/μ3Bk+1+Y3μ3.Invoked with an affine transformations, we obtained an additional optimal parameter. Therefore, we need to derive an additional update forΔτ.To do so, we updateΔτ, we fix A, E, L, β^, B, and G, and Δτk+1 can be obtained by
(15)Δτk+1=argminΔτLLk+1,β∧k+1,Ak+1,Ek+1,Bk+1,Gk+1,Δτk.Along the same line, the affine transformationsΔτ can be updated by
(16)Δτk+1=∑i=1nχi+Ak+1Bk+1+Ek+1−Mτo−Y1μ1viviT,where χi+ is a More-Penrose-pseudoinverse of χi.Similarly, the updates ofY1, Y2, and Y3 are given, respectively, by
(17)Y1k+1=Y1k+μ1k+1Q−Ak+1Bk+1−Ek+1Y2k+1=Y2k+μ2k+1β^−Ak+1Bk+1;1k+1Y3k+1=Y3k+μ3k+1Bk+1−Gk+1.The updates of the regularization parametersμ1, μ2, and μ3 are given by
(18)μ1k+1=minμmax,ρμ1kμ2k+1=minμmax,ρμ2kandμ3k+1=minμmax,ρμ3k,where ρ is a properly chosen constant and μmax are tunable parameters adjusting the convergence of the proposed algorithm. We sequentially update L, β^, A, E, B, G, and Δτ independently by keeping all of the other parameters unchanged. First, the regression matrix L and β^ are updated by (6). Next, we can update A, E, B, G, and the optimal affine transformation parameters, Δτ, by (8), (10), (12), (14), and (16), respectively. Finally, the Lagrangian multipliers Y1, Y2, and Y3 are updated by (17) and the regularization parameters μ1, μ2, and μ3 are updated by (18). These new updating equations proceed in a round-robin manner until convergence. Since a monotonically decreasing and bounded below sequence will convergence, the above algorithm is ensured to converge. It is noteworthy that based on the augmented Lagrangian multiplier, ordinary least squares procedures, and soft-threshold operators, all the updating parameters are achieved. The overall summary of the proposed method compared with other related works is summarized in Table 1.Table 1
Comparison of the proposed approach with other related algorithms in terms of the objective function and constraints.
MethodsObjective functionConstraints[51–55]minA,EA∗+λE1M=A+E[56–60]minA,EA∗+λE1Moτ+∑i=1nJiΔτviviT=A+E[61–64]minB,E,GB∗+λE2,1M=AB+E,B=GOursminA,L,β^E,B,∆τγ/2WR−Lβ^F2+A∗+B∗+⋋1G1+⋋2E1Mτo+∑i=1nJi∆τviviT=AB+E,β^=AB;1T,B=G,G≽0


## 4. Experimental Results

In this section, some simulations are conducted to assess the effectiveness of the proposed algorithm. Three datasets are considered in the simulations, including the Labeled Face Parts in the Wild (LFPW) [65], the HELEN68 [21], and the more challenging Caltech Occluded Faces in the Wild (COFW) [66] datasets. First, we consider the novel ideas of affine transformation that tackle the adverse effects of outliers, heavy sparse noises, occlusions, and illuminations. After an intensive mathematical derivation, we experimentally supported the effectiveness of the proposed method through some numerical simulations.


### 4.1. Comparison with the State-of-the-Art Methods

In this subsection, we compare the proposed approach, which adds affine transformations with some recently reported works in terms of precision-recall curves, landmark localization error, and time complexity on the aforementioned three databases. In this work, five different state-of-the-art methods, including TCDCN [31], RFLD [35], Fast-SIC [41], HPM [42], UF-DD [45], and the proposed algorithm are conducted based on the performance evaluation and time complexity. The results from these baselines are to reimplement of the publicly available codes.


### 4.2. Face Detection

First, we assess the proposed approach for face detection based on the LFPW and COFW databases in terms of the precision-recall curve. The comparisons of the proposed approach with UF-DD [45] and HPM [42] are shown in Figure 1, from which we can see that UF-DD outperforms HPM, as it downsampled the images with high resolution using an unconstrained face detector to tackle the impact of large variations. As noted in Figure 1, the HPM without occlusion is relatively better compared to the HPM with occlusion, Multiresolution HPM with rotation outperforming to the HPM with occlusion and multiresolution HPM without rotation is relatively outperforming to all other versions of the HPM in face detecting. We can also notice from Figure 1 that the proposed algorithm is superior than all HPM versions and UF-DD by achieving better recall rate on both datasets. The superiority of the new approach is due to an incorporation of an affine transformation and the multiple subspaces and constraining the parameters taking the advantages of positive semidefinite, which makes it to be more robust with a various adverse effects of outliers, illuminations, occlusions, and heavy sparse noises. This is because the new algorithm combines affine transformations with the part-based model so that it is more robust against the impact of occlusions and outliers. To justify the effectiveness of the proposed method, comparison of the localization error is taken into account, from which we note that the proposed method is outperforming to five different baselines. The performance of the proposed method is superior to all the baselines. This is due to affine transformation; first, it aligns the misaligned images and it prunes out the potential impact of annoying effects which boosts the performance of the proposed method.Figure 1
Precision-recall curves of the proposed method and baselines methods on the (a) COFW dataset and (b) LFPW dataset.
(a)(b)


### 4.3. Facial Landmark Localization

In this subsection, we compare the proposed algorithm with some state-of-the-art for the facial landmark localization on the HELEN68 and COFW datasets. The comparison with five baselines, TCDCN [31], RFLD [35], Fast-SIC [41], HPM [42], and UF-DD [45] in terms of the root mean square error is as shown in Table 2, from which we can see that TCDCN has the largest error compared with the other baselines, as it ignores the multicollinearity during multitasks to localize the objects. Also, RFLD [35] and Fast-SIC [41] can not provide satisfactory landmark localization accuracy, as their localization outputs are distorted and the patterns of occlusions can not be explicitly separated. We can also notice that HPM outperforms [31, 35, 41], as it utilizes a hierarchical-based structures to explicitly model the occlusions to attain accurate landmark localization. UF-DD outperforms [31, 35, 41], since UF-DD further downsampled the images with high resolution using a face detector. We can also find that the proposed method outperforms all baselines on both datasets. Thus, the results of Table 2 justify the effectiveness of the proposed approach based on the localization error for the facial landmark localization as compared with the main-state-of-the art works. This is because it aggregates robust regression using affine transformations with part-based models to accurately localize facial landmarks even under outliers and occlusions.Table 2
Comparison of localization error for facial landmark localization.
MethodsHELEN68COFWTCDCN [31]0.09100.0892RFLD [35]0.07990.0687Fast-SIC [41]0.04220.0382HPM [42]0.01380.0178UF-DD [45]0.01220.0142Ours0.00670.0063As an illustration, we also show three images for facial landmark localization based on the aforementioned algorithms as shown in Figure2, from which we can see that TCDCN, RFLD, and Fast-SIC provide poor localization. This is because the approaches fail to clearly separate the landmark localization from the occluded ones as TCDCN [31] considers combined multitask learning and ignores multicollinearity while RFLD [35] considers a large number of concatenating histogram of oriented gradient (HOG) descriptors with a multiple initializations which influences the separation, which still makes the separation challenging, when the images are distorted by the presence of occlusions as shown in Figures 2(a)–2(c). Moreover, it is also observed that the Fast-SIC [41] is relatively better compared to TCDCN [31] and RFLD [35] in separating landmark localization from the annoying effects such as occlusion, illuminations, outliers, and heavy sparse noises, at it considers the project out optimization framework. UF-DD [45] which incorporates the new annotated unconstrained face detection and downsampled strategy the images with high resolution provides better localization than HPM [31, 35, 41, 42], as it better separates the landmark localization (blue) and occluded landmarks (red) explicitly as depicted in Figure 2(e). The proposed approach works better under the impact of outliers and occlusion and explicitly separates the landmark localization’s from the occlusions as shown in Figure 2(f). This once again justifies that effectiveness of the combination of the affine transformations and part-based models, which makes the proposed method to be more robust with outliers and images which are highly influenced by occlusions.Figure 2
Performance evaluation of the proposed method via facial landmark localization on the HELEN68 dataset: (a) TCDCN, (b) RFLD, (c) Fast-SIC, (d) HPM, (e) UF-DD, and (f) ours. The red points indicate the key points which are predicted as being occluded.
(a)(b)(c)(d)(e)(f)In summary, the proposed method is relatively better in landmark localization and facial detection; this is due to affine transformation considered detects the potential impact of outliers, heavy sparse noises, occlusions and illuminations. However, the major drawback of the proposed method is it needs to account the spatial dependency between images through incorporating a spatial weight matrix in the mathematical formulation.


## 4.1. Comparison with the State-of-the-Art Methods

In this subsection, we compare the proposed approach, which adds affine transformations with some recently reported works in terms of precision-recall curves, landmark localization error, and time complexity on the aforementioned three databases. In this work, five different state-of-the-art methods, including TCDCN [31], RFLD [35], Fast-SIC [41], HPM [42], UF-DD [45], and the proposed algorithm are conducted based on the performance evaluation and time complexity. The results from these baselines are to reimplement of the publicly available codes.


## 4.2. Face Detection

First, we assess the proposed approach for face detection based on the LFPW and COFW databases in terms of the precision-recall curve. The comparisons of the proposed approach with UF-DD [45] and HPM [42] are shown in Figure 1, from which we can see that UF-DD outperforms HPM, as it downsampled the images with high resolution using an unconstrained face detector to tackle the impact of large variations. As noted in Figure 1, the HPM without occlusion is relatively better compared to the HPM with occlusion, Multiresolution HPM with rotation outperforming to the HPM with occlusion and multiresolution HPM without rotation is relatively outperforming to all other versions of the HPM in face detecting. We can also notice from Figure 1 that the proposed algorithm is superior than all HPM versions and UF-DD by achieving better recall rate on both datasets. The superiority of the new approach is due to an incorporation of an affine transformation and the multiple subspaces and constraining the parameters taking the advantages of positive semidefinite, which makes it to be more robust with a various adverse effects of outliers, illuminations, occlusions, and heavy sparse noises. This is because the new algorithm combines affine transformations with the part-based model so that it is more robust against the impact of occlusions and outliers. To justify the effectiveness of the proposed method, comparison of the localization error is taken into account, from which we note that the proposed method is outperforming to five different baselines. The performance of the proposed method is superior to all the baselines. This is due to affine transformation; first, it aligns the misaligned images and it prunes out the potential impact of annoying effects which boosts the performance of the proposed method.Figure 1
Precision-recall curves of the proposed method and baselines methods on the (a) COFW dataset and (b) LFPW dataset.
(a)(b)


## 4.3. Facial Landmark Localization

In this subsection, we compare the proposed algorithm with some state-of-the-art for the facial landmark localization on the HELEN68 and COFW datasets. The comparison with five baselines, TCDCN [31], RFLD [35], Fast-SIC [41], HPM [42], and UF-DD [45] in terms of the root mean square error is as shown in Table 2, from which we can see that TCDCN has the largest error compared with the other baselines, as it ignores the multicollinearity during multitasks to localize the objects. Also, RFLD [35] and Fast-SIC [41] can not provide satisfactory landmark localization accuracy, as their localization outputs are distorted and the patterns of occlusions can not be explicitly separated. We can also notice that HPM outperforms [31, 35, 41], as it utilizes a hierarchical-based structures to explicitly model the occlusions to attain accurate landmark localization. UF-DD outperforms [31, 35, 41], since UF-DD further downsampled the images with high resolution using a face detector. We can also find that the proposed method outperforms all baselines on both datasets. Thus, the results of Table 2 justify the effectiveness of the proposed approach based on the localization error for the facial landmark localization as compared with the main-state-of-the art works. This is because it aggregates robust regression using affine transformations with part-based models to accurately localize facial landmarks even under outliers and occlusions.Table 2
Comparison of localization error for facial landmark localization.
MethodsHELEN68COFWTCDCN [31]0.09100.0892RFLD [35]0.07990.0687Fast-SIC [41]0.04220.0382HPM [42]0.01380.0178UF-DD [45]0.01220.0142Ours0.00670.0063As an illustration, we also show three images for facial landmark localization based on the aforementioned algorithms as shown in Figure2, from which we can see that TCDCN, RFLD, and Fast-SIC provide poor localization. This is because the approaches fail to clearly separate the landmark localization from the occluded ones as TCDCN [31] considers combined multitask learning and ignores multicollinearity while RFLD [35] considers a large number of concatenating histogram of oriented gradient (HOG) descriptors with a multiple initializations which influences the separation, which still makes the separation challenging, when the images are distorted by the presence of occlusions as shown in Figures 2(a)–2(c). Moreover, it is also observed that the Fast-SIC [41] is relatively better compared to TCDCN [31] and RFLD [35] in separating landmark localization from the annoying effects such as occlusion, illuminations, outliers, and heavy sparse noises, at it considers the project out optimization framework. UF-DD [45] which incorporates the new annotated unconstrained face detection and downsampled strategy the images with high resolution provides better localization than HPM [31, 35, 41, 42], as it better separates the landmark localization (blue) and occluded landmarks (red) explicitly as depicted in Figure 2(e). The proposed approach works better under the impact of outliers and occlusion and explicitly separates the landmark localization’s from the occlusions as shown in Figure 2(f). This once again justifies that effectiveness of the combination of the affine transformations and part-based models, which makes the proposed method to be more robust with outliers and images which are highly influenced by occlusions.Figure 2
Performance evaluation of the proposed method via facial landmark localization on the HELEN68 dataset: (a) TCDCN, (b) RFLD, (c) Fast-SIC, (d) HPM, (e) UF-DD, and (f) ours. The red points indicate the key points which are predicted as being occluded.
(a)(b)(c)(d)(e)(f)In summary, the proposed method is relatively better in landmark localization and facial detection; this is due to affine transformation considered detects the potential impact of outliers, heavy sparse noises, occlusions and illuminations. However, the major drawback of the proposed method is it needs to account the spatial dependency between images through incorporating a spatial weight matrix in the mathematical formulation.


## 5. Computational Complexity

The time complexity of the proposed method as compared to the state-of-the-art works is described in this section. On a very standard desktop computer, the computational load of the baselines mainly TCDCN [31], RFLD [35], Fast-SIC [41], HPM [42], and UF-DD [45] along with the proposed method is given in Table 3, from which we note that the proposed method has small number of running time due to in the proposed method there are less number of parameters involved in updating the parameters. Additionally, our algorithm can handle batches of over one hundred images in a few minutes on a standard PC as the number of parameters involved is small as compared to the state-of-the-art works. The new algorithm is guaranteed faster time complexity compared to the state-of-the-art algorithms as shown in Table 3. This is due to affine transformation removing the extreme values, and various regularization parameters are also involved to make the situation of the proposed model stable and gross errors including outliers, occlusions, and illuminations.Table 3
Comparison of time complexity in minutes.
MethodsTime complexityTCDCN [31]4.34RFLD [35]4.56Fast-SIC [41]4.78HPM [42]5.67UF-DD [45]4.20Ours3.56


## 6. Conclusions

In this paper, we developed a new model for facial landmark localization and detection, comprising an affine transformation and ADMM methods. The new algorithm combines the efficacious affine transformation-assisted robust regression with the part-based models to enjoy the advantages of both schemes. The problem is formulated as a constrained optimization programming, and a set of equations are established to iteratively update the parameters and the affine transformations. The search of the affine transformations and the optimization variables is formulated as a constrained convex optimization problem. The ADMM approach is then employed, and a new set of equations is established to iteratively update the optimization variables and the affine transformations. Simulations justify the effectiveness of the new algorithm on some common on the COFW, HELEN, and LFPW datasets compared to the state-of-the-art works. To minimize the time complexity, the ADMM approach is further considered in a round-robin manner. In this work, the affine transformation taken into account is to correct the alignments of the individual images, but not consider the issue of spatial dependency between images. This can be captured as one of our future work through incorporating the spatial weight matrix between images. Another future work is to excel this new part-based model in high-dimensional tensor datasets.



---

*Source: 9995074-2021-11-03.xml*