System and method for pose-invariant face alignment

A computing system includes a processing system with at least one processing unit. The processing system is configured to execute a face alignment method upon receiving image data with a facial image. The processing system is configured to apply a neural network to the facial image. The neural network is configured to provide a final estimate of parameter data for the facial image based on the image data and an initial estimate of the parameter data. The neural network includes at least one visualization layer, which is configured to generate a feature map based on a current estimate of the parameter data. The parameter data includes head pose data and face shape data.

FIELD OF THE INVENTION

This disclosure relates to systems and methods for face alignment.

BACKGROUND

In general, face alignment technologies, which are implemented with cascades of Convolutional Neural Networks (CNNs), experience at least the following drawbacks: lack of end-to-end training, hand-crafted feature extraction, and slow training speed. For example, without end-to-end training, the CNNs cannot be optimized jointly, thereby leading to a sub-optimal solution. In addition, these type of face alignment technologies often implement simple hand-crafted feature extraction methods, which do not take into account various facial factors, such as pose, expression, etc. Moreover, these cascades of CNNs typically have shallow frameworks, which are unable to extract deeper features by building upon the extracted features of early-stage CNNs. Furthermore, training for these CNNs is usually time-consuming because each of the CNNs is trained independently and sequentially and also because hand-crafted feature extraction is required between two consecutive CNNs.

SUMMARY

The following is a summary of certain embodiments described in detail below. The described aspects are presented merely to provide the reader with a brief summary of these certain embodiments and the description of these aspects is not intended to limit the scope of this disclosure. Indeed, this disclosure may encompass a variety of aspects that may not be explicitly set forth below.

In an example embodiment, a computing system includes a processing system with at least one processing unit. The processing system is configured to execute a face alignment method upon receiving image data with a facial image. The processing system is configured to apply a neural network to the facial image. The neural network is configured to provide a final estimate of parameter data for the facial image based on the image data and an initial estimate of the parameter data. The neural network includes at least one visualization layer, which is configured to generate a feature map based on a current estimate of the parameter data. The parameter data includes head pose data and face shape data.

In an example embodiment, a computer-implemented method includes receiving image data with a facial image. The computer-implemented method includes implementing a neural network to provide a final estimate of parameter data for the facial image based on the image data and an initial estimate of the parameter data. The neural network includes at least one visualization layer, which is configured to generate a feature map based on a current estimate of the parameter data. The parameter data includes head pose data and face shape data.

In an example embodiment, non-transitory computer-readable media comprises at least computer-readable data that, when executed by a processing system with at least one processing unit, performs a method that includes receiving image data with a facial image. The method includes implementing a neural network to provide a final estimate of parameter data for the facial image based on the image data and an initial estimate of the parameter data. The neural network includes at least one visualization layer, which is configured to generate a feature map based on a current estimate of the parameter data. The parameter data includes head pose data and face shape data.

These and other features, aspects, and advantages of the present invention are further clarified by the following detailed description of certain exemplary embodiments in view of the accompanying drawings throughout which like characters represent like parts.

DETAILED DESCRIPTION

The embodiments described above, which have been shown and described by way of example, and many of their advantages will be understood by the foregoing description, and it will be apparent that various changes can be made in the form, construction, and arrangement of the components without departing from the disclosed subject matter or without sacrificing one or more of its advantages. Indeed, the described forms of these embodiments are merely explanatory. These embodiments are susceptible to various modifications and alternative forms, and the following claims are intended to encompass and include such changes and not be limited to the particular forms disclosed, but rather to cover all modifications, equivalents, and alternatives falling with the spirit and scope of this disclosure.

FIG. 1is a block diagram of a computer system100configured to implement pose-invariant face alignment. In this regard, the computer system100includes various software and hardware components. For example, the computer system100includes at least a memory system110, a face detection module120, a pose-invariant face alignment module130, a processing system140, a communication system150, and other functional modules160. In an example embodiment, the computer system100is configured to implement and execute a pose-invariant face alignment method, as disclosed herein and as provided by the pose-invariant face alignment module130. In addition, in an example embodiment, the computer system100is also configured to implement and execute face detection, as disclosed herein and as provided by the face detection module120, prior to implementing and executing the pose-invariant face alignment method.

In an example embodiment, the memory system110includes various data, including training data and other data associated with the pose-invariant face alignment module130. In an example embodiment, the memory system110is a computer or electronic storage system, which is configured to store and provide access to various data to enable at least the operations and functionality, as disclosed herein. In an example embodiment, the memory system110comprises a single device or a plurality of devices. In an example embodiment, the memory system110can include electrical, electronic, magnetic, optical, semiconductor, electromagnetic, or any suitable technology. For instance, in an example embodiment, the memory system110can include random access memory (RAM), read only memory (ROM), flash memory, a disk drive, a memory card, an optical storage device, a magnetic storage device, a memory module, any suitable type of memory device, or any combination thereof. In an example embodiment, with respect to the computer system100, the memory system110is local, remote, or a combination thereof (e.g., partly local and partly remote). In an example embodiment, the memory system110can include at least a cloud-based storage system (e.g. cloud-based database system), which is remote from the other components of the computer system100.

In an example embodiment, the face detection module120includes hardware, software, or a combination thereof. In an example embodiment, the face detection module120is at least configured to receive an image, identify a facial image within the image, and provide image data220relating to the facial image. In an example embodiment, the processing system140includes at least a central processing unit (CPU), a graphics processing unit (GPU), a Field-Programmable Gate Array (FPGA), an Application-Specific Integrated Circuit (ASIC), a System-on-a-chip system (SOC), a programmable logic device (PLD), any suitable computing technology, or any combination thereof.

In an example embodiment, the communication system150includes suitable communications technology that enables any suitable combination of components of the computer system100to communicate with each other. In an example embodiment, the communication system150includes wired-based technology, wireless-based technology, and/or a combination thereof. In an example embodiment, the communication system150includes a wired network, a wireless network, or a combination thereof. In an example embodiment, the communication system150includes any suitable type of computer network and/or architecture. In an example embodiment, the communication system150includes a connection to the Internet.

In an example embodiment, the other functional modules160include hardware, software, or a combination thereof. For instance, the other functional modules28include logic circuitry, an operating system, I/O devices (e.g., a display, etc.), other computer technology, or any combination thereof. More specifically, in an example embodiment, the other functional modules28enable the pose-invariant face alignment module130to operate and function, as disclosed herein. In an example embodiment, the other functional modules160include a camera and/or optical system. In this regard, the camera and/or optical system is configured to provide an image to the face detection module120and/or the processing system140such that image data220is provided to the pose-invariant face alignment module130. Also, in an example embodiment, the other functional modules160includes a facial analysis module, such as a face recognition module, an expression estimation module, a 3D face reconstruction module, any suitable facial analysis module, or any combination thereof. In this regard, the facial analysis module is configured to perform facial analysis in accordance with output, such as a final estimation of parameter data relating to the facial image, from the CNN200.

FIG. 2illustrates a pose-invariant face alignment module130according to an example embodiment. In an example embodiment, the pose-invariant face alignment module130includes a single CNN200. In an example embodiment, this CNN200is configured to receive at least image data220and a set of parameters230as input. Upon receiving the image data220(i.e., a single facial image with an arbitrary head pose) from the face detection module120and upon obtaining a set of parameters230, the pose-invariant face alignment module130is configured to estimate the 2D landmarks with their visibility labels by fitting a 3D face model. In this regard, the pose-invariant face alignment module130includes a single CNN200with end-to-end training for model fitting.

FIG. 3illustrates an example architecture of the CNN200according to an example embodiment. As shown, the CNN200includes a plurality of connected visualization blocks210. For instance, as a non-limiting example, the CNN200includes at least six visualization blocks210. In this regard, the CNN200includes any suitable number of visualization blocks210that provides the desired results. In an example embodiment, the inputs include image data220and an initial estimation of at least one parameter, e.g. parameter P0, and the output is the final estimation290of the set of parameters. Compared to a related system with a cascade of CNN200s, due to the joint optimization of all visualization blocks210with backpropagation of the loss functions, the pose-invariant face alignment module130has a CNN200with an architecture, which is able to converge in substantially fewer epochs during training.

In an example embodiment, the system100includes a 3D Morphable Model (3DMM). In an example embodiment, the memory system110(e.g., training data), the pose-invariant face alignment module130, or a combination thereof includes the 3DMM. In an example embodiment, the 3DMM represents the 3D shape of a face. More specifically, 3DMM represents a 3D face Spas a linear combination of mean shape S0, identity bases SIand expression bases SEvia the following equation:
Sp=S0+ΣkNIpkISkI+ΣkNEpkESkE[Equation 1]

In an example embodiment, the pose-invariant face alignment module130uses a vector p=[pI, pE] for the 3D shape parameters, where pI=[p0I, . . . , pNII] are the identity parameters and pE=[p0E, . . . , pNEE] are the expression parameters. In an example embodiment, the pose-invariant face alignment module130uses a Basel 3D face model, which has 199 bases, as identity bases and the face warehouse model with 29 bases as the expression bases. In this case, each 3D face shape comprises a set of Q 3D vertexes:

In an example embodiment, the 2D face shapes are the projection of 3D shapes. In an example embodiment, the weak perspective projection model is used with 6 degrees of freedoms, i.e., one for scale, three for rotation angles, and two for translations, which projects the 3D face shape Sponto 2D images to obtain the 2D shape U as expressed by the following equation:

In this case, U collects a set of N 2D landmarks, M is the camera projection matrix, with misuse of notation P={M, p}, and the N-dim vector b includes 3D vertex indexes which are semantically corresponding to 2D landmarks. In an example embodiment, m1=[m1m2m3] and m2=[m5m6m7] denote the first two rows of the scaled rotation component, while m4and m8are the translations.

Equation 3 establishes the relationship, or equivalency, between 2D landmarks U and P, i.e., 3D shape parameters p and the camera projection matrix M. Given that almost all the training images for face alignment have only 2D labels, i.e., U, the processing system140perform a data augmentation step to compute their corresponding P. Given image data220, the pose-invariant face alignment module130is configured to estimate the parameter P, based on which the 2D landmarks and their visibilities can be derived.

FIG. 4illustrates a conceptual diagram of a visualization block210according to an example embodiment. As shown inFIG. 4, the visualization block210includes a visualization layer240, which reconstructs the 3D face shape from the estimated parameters inside the CNN200and synthesizes a 2D image via the surface normal vectors808/810of visible vertexes. In an example embodiment, the visualization layer240visualizes the alignment result of the previous visualization block210and utilizes it for the current visualization block210. In an example embodiment, the visualization layer240is derived from the surface normals808/810of the underlying 3D face model806and encodes the relative pose between a face and a camera (FIG. 8). Also, in an example embodiment, the visualization layer240is differentiable, which allows the gradient to be computed analytically, thereby enabling end-to-end training of the CNN200. Furthermore, as shown inFIG. 4, the visualization layer240utilizes a mask600/700to differentiate between pixels in the middle and contour parts of a facial image and to also make the pixel value of the visualized images similar across different poses. Moreover, as shown inFIG. 4, the final estimate290of the parameters of the facial image, as provided by the last visualization block210of the CNN200, can be provided to a facial analysis module to obtain facial landmark detection results300.

FIG. 5illustrates a visualization block210of the CNN200according to an example embodiment. As aforementioned, each visualization block210includes a visualization layer240that is based on the latest parameter estimation. In this regard, each visualization layer240serves as a bridge between consecutive visualization blocks210. In an example embodiment, each visualization layer240generates a feature map250based on the current estimated, or input, parameter P. Each convolutional layer260is followed by a batch normalization (BN) layer and a rectified linear units (ReLU) layer. Each convolutional layer260extracts deeper features based on input features provided by the previous visualization block210and the visualization layer240. Between the two fully connected layers270, the first convolutional layer260is followed by a ReLU layer and a dropout layer, while the second convolutional layer260simultaneously estimates the update of M and p, ΔP. In an example embodiment, the outputs of the visualization block210include output data280A and a new estimation (or current estimation)280B of the parameters230, for example, when adding ΔP to the input P. InFIG. 5, the output data280A includes deeper features and the image data220. In an alternative example embodiment, the output data280A includes deeper features. In another alternative example embodiment, the output data280A includes the image data220. In an example embodiment, as shown inFIG. 5, basically the top part of the visualization block210focuses on learning deeper features, while the bottom part utilizes such features to estimate the parameters230in a ResNet-like structure. During a backward pass of the training phase, the visualization block210backpropagates the loss through both of its inputs to adjust the convolutional layers260and fully connected layers270in the previous visualization blocks210. This operation allows the visualization block210to extract better features that are suitable for the next visualization block210and improve the overall parameter estimation.

In an example embodiment, the CNN200is configured to employ at least two types of loss functions. In this case, for example, the first type of loss function is a Euclidean loss between the estimation and the target of the parameter update, with each parameter weighted separately as expressed by following equation:
EPi=(ΔPi−ΔPi)TW(ΔPi−ΔPi)T[Equation 6]

where EPiis the loss, ΔPiis the estimation and ΔPiis the target (or ground truth) at the i-th visualization block210. In this equation, the diagonal matrix W contains the weights. For each element of the shape parameter p, its weight is the inverse of the standard deviation that was obtained from the data used in 3DMM training. To compensate the relative scale among the parameters of M, the processing system140computes the ratio r between the average of scaled rotation parameters and average of translation parameters in the training data. In this regard, the weights of the scaled rotation parameters of M are set to

1r
and the weights of the translation of M are set to 1. In addition, the second type of loss function is the Euclidean loss on the resultant 2D landmarks as expressed by the following equation:
ESi=∥f(Pi−ΔPi)−Ū∥2[Equation 7]

where Ū is the ground truth 2D landmarks, and Piis the input parameter to the i-th block, i.e., the output of the i−1-th block. In this regard, f (⋅) computes 2D landmark locations using the currently updated parameters via Equation 3. In an example embodiment, for backpropagation of this loss function to the parameter ΔP, the chain rule is used to compute the gradient, as expressed by the following equation:

In an example embodiment, for the first three visualization blocks210of the CNN200, the Euclidean loss on the parameter updates (Equation 6) is used, while the Euclidean loss on 2D landmarks (Equation 7) is applied to the last three blocks of the CNN200. The first three blocks estimate parameters to align 3D shape to the face image roughly and the last three blocks leverage the good initialization to estimate the parameters and the 2D landmark locations more precisely.

In an example embodiment, the visualization layer240is based on surface normals of the 3D face that provide surface orientations in local neighborhoods. In an example embodiment, the processing system140uses the z coordinate of surface normals of each vertex transformed with the pose. In this regard, the z coordinate is an indicator of a “frontability” of a vertex, i.e., the amount that the surface normal is pointing towards a camera800. This quantity is used to assign an intensity value at its projected 2D location to construct visualization data242(e.g., a visualization image). In an example embodiment, the frontability measure g, a Q-dim vector, can be computed via the following equation:

where × is the cross product, and ∥⋅∥ denotes the L2norm. The 3×Q matrix N0is the surface normal vectors of a 3D face shape. To avoid the high computational cost of computing the surface normals after each shape update, the processing system140approximates N0as the surface normals of the mean 3D face.

In an example embodiment, both the face shape and head pose are still continuously updated across various visualization blocks210, and are used to determine the projected 2D location. Hence, this approximation would only slightly affect the intensity value. To transform the surface normal based on the head pose, the processing system140applies the estimation of the scaled rotation matrix (m1and m2) to the surface normals computed from the mean face. The value is then truncated with the lower bound of 0, as shown in Equation 9. The pixel intensity of a visualized image V(u,v) is computed as the weighted average of the frontability measures within a local neighbourhood as expressed by the following equation:

where D (u, v) is the set of indexes of vertexes whose 2D projected locations are within the local neighborhood of the pixel (u, v). (xqt,yqt) is the 2D projected location of q-th 3D vertex. The weight w is the distance metric between the pixel (u, v) and the projected location (xqt,yqt),

In addition, a is a Q-dim mask vector with positive values for vertexes in the middle area of the face and negative values for vertexes around the contour area of the face as expressed by the following equation:

where (xn; yn; zn) is the vertex coordinate of the nose tip.

Also, in this equation, a(q) is pre-computed and normalized for zero-mean and unit standard deviation. In an example embodiment, the processing system140uses the mask600to discriminate between the central and boundary areas of the face, as well as to increase similarity across visualization of different faces.

In an example embodiment, to allow backpropagation of the loss functions through the visualization layer240, the processing system140computes the derivative of V with respect to the elements of the parameters M and p. In this regard, the processing system140computes the partial derivatives,

∂g∂mk,∂w⁡(u,v,xit,yit)∂mk,and⁢⁢∂w⁡(u,v,xit,yit)∂pj.
In an example embodiment, the processing system140then computes the derivatives of

∂V∂mk⁢⁢and⁢⁢∂V∂pj
based on Equation 10.

FIG. 6illustrates two views of a visualization of an exemplary mask600according to an example embodiment. Specifically,FIG. 6includes a frontal view602of the mask600and a side view604(or profile view) of the mask600. In this case, the mask600is expressed, for instance, by at least Equation 12. As shown inFIG. 6, the mask600, as expressed by a(q), has positive values in the middle area and negative values in the contour area, as indicated by the scale606.

FIG. 7illustrates another example of a mask700according to an alternative example embodiment. Specifically,FIG. 7includes a frontal view702of the mask700and a side view704(or profile view) of the mask700. In this example, the mask700has five positive areas, which include the two eye areas, the nose tip area, and the two lip corner areas, as indicated by the scale706. Also, in this example, the values are normalized to zero-mean and unit standard deviation. In this regard, the mask700makes the pixel value of visualized images to be similar for faces with different poses and discriminates between the middle-area and contour-areas of the face. The mask700ofFIG. 7is more complex and conveys more information about the informative facial areas compared to that provided by the mask600ofFIG. 6.

FIG. 8illustrates a position of a camera800relative to an image plane802with a plurality of pixels. In addition,FIG. 8shows a pixel axis804that extends along an image pixel of the image plane802together with a visualization of a human facial image of the image plane802as a 3D object806.FIG. 8also includes surface normal vectors with negative z coordinates, as pointed out at arrow808, and a surface normal vector with a positive z coordinate and smaller depth, as pointed out by arrow810. In this regard, visualizing the human face as a 3D object806at an arbitrary view angle requires the estimation of the visibility of each 3D vertex. To avoid the computationally expensive visibility test via rendering, the processing system140is configured to implement at least two strategies for approximation. As one strategy, for example, the processing system140is configured to prune the vertexes whose frontability measures g equal to 0, i.e., the vertexes pointing against the camera800. Secondly, if multiple vertexes project to a same image pixel via the pixel axis804, the processing system140is configured to keep only the one with the smallest depth values, as indicated, for example, by arrow810inFIG. 8.

FIG. 9Ais an example of image data220according to an example embodiment. As shown inFIG. 9A, in this example, the image data220includes at least a large face pose with an open-mouth expression (e.g., a smile).FIG. 9Bis an example of an initialization of a visualization layer240according to an example embodiment.FIG. 9Cis an example of visualization data242of a visualization layer240associated with a first visualization block210of a CNN200according to an example embodiment.FIG. 9Dis an example of visualization data242of a visualization layer240associated with a second visualization block210of a CNN200according to an example embodiment.FIG. 9Eis an example of visualization data242of a visualization layer240associated with a third visualization block210of a CNN200according to an example embodiment.FIG. 9Fis an example of visualization data242of a visualization layer240associated with a fourth visualization block210of a CNN200according to an example embodiment.FIG. 9Gis an example of visualization data242of a visualization layer240associated with a fifth visualization block210of a CNN200according to an example embodiment.FIG. 9His an example of visualization data242of a visualization layer240associated with a sixth visualization block210of a CNN200according to an example embodiment of this disclosure. As progressively shown inFIGS. 9C-9H, the pose-invariant face alignment module130is able to recover the expression and the head pose of the facial image of the image data220, as shown inFIG. 9A. In an example embodiment, the pose-invariant face alignment module130is able to provide these results at least by extracting deeper features and employing the backpropagation of loss functions.

FIG. 10Ais an example of image data220according to an example embodiment of this disclosure. As shown inFIG. 10A, in this example, the image data220includes at least a large face pose with a relatively neutral expression.FIG. 10Bis an example of an initialization of a visualization layer240according to an example embodiment of this disclosure.FIG. 10Cis an example of visualization data242of a visualization layer240associated with a first visualization block210of a CNN200according to an example embodiment of this disclosure.FIG. 10Dis an example of visualization data242of a visualization layer240associated with a second visualization block210of a CNN200according to an example embodiment of this disclosure.FIG. 10Eis an example of visualization data242of a visualization layer240associated with a third visualization block210of a CNN200according to an example embodiment of this disclosure.FIG. 10Fis an example of visualization data242of a visualization layer240associated with a fourth visualization block210of a CNN200according to an example embodiment.FIG. 10Gis an example of visualization data242of a visualization layer240associated with a fifth visualization block210of a CNN200according to an example embodiment.FIG. 10His an example of visualization data242of a visualization layer240associated with a sixth visualization block210of a CNN200according to an example embodiment. As progressively shown inFIGS. 10C-10H, the pose-invariant face alignment module130is able to recover the expression and the head pose of the facial image of the image data220, as shown inFIG. 10A. In an example embodiment, the pose-invariant face alignment module130is able to provide these results at least by extracting deeper features and employing the backpropagation of loss functions.

FIG. 11Ais an example of image data220according to an example embodiment of this disclosure. As shown inFIG. 11A, in this example, the image data220includes at least a large face pose with a relatively neutral expression. Also, the image data220ofFIG. 11Aincludes a side of a face that is different than a side of a face that is included in the image data220ofFIG. 10A.FIG. 11Bis an example of an initialization of a visualization layer240according to an example embodiment.FIG. 11Cis an example of visualization data242of a visualization layer240associated with a first visualization block210of a CNN200according to an example embodiment.FIG. 11Dis an example of visualization data242of a visualization layer240associated with a second visualization block210of a CNN200according to an example embodiment.FIG. 11Eis an example of visualization data242of a visualization layer240associated with a third visualization block210of a CNN200according to an example embodiment.FIG. 11Fis an example of visualization data242of a visualization layer240associated with a fourth visualization block210of a CNN200according to an example embodiment.FIG. 11Gis an example of visualization data242of a visualization layer240associated with a fifth visualization block210of a CNN200according to an example embodiment.FIG. 11His an example of visualization data242of a visualization layer240associated with a sixth visualization block210of a CNN200according to an example embodiment. As progressively shown inFIGS. 11C-11H, the pose-invariant face alignment module130is able to recover the expression and the head pose of the facial image of the image data220, as shown inFIG. 11A. In an example embodiment, the pose-invariant face alignment module130is able to provide these results at least by extracting deeper features and employing the backpropagation of loss functions.

As described above, the system100includes a number of advantageous features. For example, the system100is configured to implement a large-pose face alignment method with end-to-end training via a single CNN200. In addition, the CNN200includes at least one differentiable visualization layer240, which is integrated into the neural network, i.e. the CNN200, and enables joint optimization by backpropagating the error from at least one later visualization block210to at least one earlier visualization block210. In addition, the system100is configured such that each visualization block210is enabled to extract deeper features by utilizing the extracted features from previous visualization blocks210without the need to extract hand-crafted features. Also, the pose-invariant alignment method converges faster during the training phase compared to that provided by a related system involving a cascade of CNNs. In this regard, for example, one of the main advantages of end-to-end training of a single CNN200is the reduced training time. In addition, the CNN200includes at least one visualization layer240, which is differentiable and encodes the face geometry details via surface normals. Moreover, the pose-invariant face alignment module130is enabled to guide the CNN200to focus on the face area that incorporates both the pose and expression information. Furthermore, the CNN200can be configured to achieve greater levels of precision and accuracy by simply increasing the number of visualization blocks210in its architecture.

That is, the above description is intended to be illustrative, and not restrictive, and provided in the context of a particular application and its requirements. Those skilled in the art can appreciate from the foregoing description that the present invention may be implemented in a variety of forms, and that the various embodiments may be implemented alone or in combination. Therefore, while the embodiments of the present invention have been described in connection with particular examples thereof, the general principles defined herein may be applied to other embodiments and applications without departing from the spirit and scope of the described embodiments, and the true scope of the embodiments and/or methods of the present invention are not limited to the embodiments shown and described, since various modifications will become apparent to the skilled practitioner upon a study of the drawings, specification, and following claims. For example, components and functionality may be separated or combined differently than in the manner of the various described embodiments, and may be described using different terminology. These and other variations, modifications, additions, and improvements may fall within the scope of the disclosure as defined in the claims that follow.