Patent Description:
Facial recognition technology is considered a convenient and competitive bio-recognition technology that may verify a target without contact with the target, dissimilar to other recognition technologies, for example, fingerprint and iris recognition, that require a user to conduct a certain motion or an action. Such facial recognition technology has been widely used in various application fields, for example, security systems, mobile authentication, and multimedia searches due to convenience and effectiveness of the facial recognition technology.

Known from the art is the image processing apparatus disclosed in <CIT>, which includes a projecting unit that projects a registered face image containing at least part of a face onto a surface of a three-dimensional model having a shape in which at least part of the three-dimensional model in one direction on the surface onto which an image is projected is bent to a front side, so that a horizontal direction of the face contained in the registered face image substantially coincides with the one direction, a transforming unit that transforms the three-dimensional model on the basis of an orientation of a face contained in a target image, a generating unit that generates a two-dimensional image by projecting the registered face image projected on the surface of the transformed three-dimensional model, onto a plane, and an identifying unit that identifies the face contained in the target image, by comparing the generated two-dimensional image against the target image.

Also known in the art are the method and system disclosed in <CIT>, for generating 3D images of faces from 2D images, for generating 2D images of the faces at different image conditions from the 3D images, and for recognizing a 2D image of a target face based on the generated 2D images is provided. The recognition system provides a 3D model of a face that includes a 3D image of a standard face under a standard image condition and parameters indicating variations of an individual face from the standard face. To generate the 3D image of a face, the recognition system inputs a 2D image of the face under a standard image condition. The recognition system then calculates parameters that map the points of the 2D image to the corresponding points of a 2D image of the standard face. The recognition system uses these parameters with the 3D model to generate 3D images of the face at different image conditions.

According to a first aspect of the invention, a facial recognition method according to accompanying claim <NUM> is provided. According to a second aspect of the invention, a facial recognition apparatus according to accompanying claim <NUM> is provided. Preferred embodiments are covered by the dependent claims.

Hereinafter, some example embodiments will be described in detail with reference to the accompanying drawings. Regarding the reference numerals assigned to the elements in the drawings, it should be noted that the same elements will be designated by the same reference numerals, wherever possible, even though they are shown in different drawings. Also, in the description of example embodiments, detailed description of well-known related structures or functions will be omitted when it is deemed that such description will cause ambiguous interpretation of the present disclosure.

It should be understood, however, that there is no intent to limit this disclosure to example embodiments disclosed. On the contrary, example embodiments are to cover all modifications, equivalents, and alternatives falling within the scope of the example embodiments. Like numbers refer to like elements throughout the description of the figures.

In addition, terms such as first, second, A, B, (a), (b), and the like may be used herein to describe components.

Unless specifically stated otherwise, or as is apparent from the discussion, terms such as "processing" or "computing" or "calculating" or "determining" or "displaying" or the like, refer to the action and processes of a computer system, or similar electronic computing device, that manipulates and transforms data represented as physical, electronic quantities within the computer system's registers and memories into other data similarly represented as physical quantities within the computer system memories or registers or other such information storage, transmission or display devices.

In the drawings, the thicknesses of layers and regions are exaggerated for clarity.

<FIG> is a diagram illustrating an overall operation of a facial recognition system <NUM> according to at least one example embodiment. The facial recognition system <NUM> may recognize a face of a user from a two-dimensional (2D) input image used for facial recognition. The facial recognition system <NUM> may extract and identify the face of the user appearing in the 2D input image by analyzing the 2D input image. The facial recognition system <NUM> may be used in various application fields, for example, security and surveillance systems, mobile authentication, and multimedia data searches.

The facial recognition system <NUM> may register a three-dimensional (3D) facial model of the user and perform the facial recognition using the registered 3D facial model. The 3D facial model may be a deformable 3D model that may be deformed depending on a facial pose or a facial expression of the user appearing in the 2D input image. For example, when a facial pose appearing in the 2D input image faces a left side, the facial recognition system <NUM> may rotate the registered 3D facial model to face the left side. In addition, the facial recognition system <NUM> may adjust a facial expression of the 3D facial model based on a facial expression of the user appearing in the 2D input image. For example, the facial recognition system <NUM> may analyze the facial expression of the user based on a facial feature point detected from the 2D input image, and adjust a shape of an eye, a lip, and a nose of the 3D facial model to allow the adjusted shape to correspond to the analyzed facial expression.

The facial recognition system <NUM> may generate a 2D projection image from the registered 3D facial model, and perform the facial recognition by comparing the 2D projection image to the 2D input image. The facial recognition may be performed in real time using 2D images. The 2D projection image refers to a 2D image obtained by projecting the 3D facial model to a plane. For example, the 2D projection image may be a 2D image obtained by projecting the 3D facial model matched to the 2D input image at a viewpoint identical or similar to a viewpoint in the 2D input image and thus, a facial pose appearing in the 2D projection image may be identical or similar to a facial pose of the user appearing in the 2D input image. The facial recognition may be performed by matching the prestored 3D facial model to the facial pose appearing in the 2D input image, and comparing the 2D projection image to the 2D input image. Although the facial pose of the user appearing in the 2D input image does not face a front side, an improved recognition rate may be achieved in response to a change in the pose by matching the 3D facial model to the facial pose appearing in the 2D input image and performing the facial recognition.

Hereinafter, operations of the facial recognition system <NUM> will be described in detail. Facial recognition performed by the facial recognition system <NUM> may include a process <NUM> of registering a 3D facial model of a user and a process <NUM> of recognizing a face of the user from a 2D input image using the registered 3D facial model.

Referring to <FIG>, in operation <NUM> of process <NUM>, the facial recognition system <NUM> obtains a plurality of 2D face images of the user used for face registration. The 2D face images may include images of the face of the user captured from various viewpoints. For example, the facial recognition system <NUM> may obtain 2D face images captured through a camera from a front side and a lateral side of the face of the user. A 2D face image may refer to an image including a face region of the user, but may not necessarily include an entire region of the face of the user. In operation <NUM> of process <NUM>, the facial recognition system <NUM> detects facial feature points, for example, landmarks, from the 2D face images. For example, the facial recognition system <NUM> may detect feature points including eyebrows, eyes, a nose, lips, a chin, hair, ears, and/or a facial contour from the 2D face images of the user.

In operation <NUM> of process <NUM>, the facial recognition system <NUM> individualizes a 3D model by applying, to a predetermined and/or selected 3D standard model, the feature points extracted from the 2D face images used for the face registration. For example, the 3D standard model may be a deformable 3D shape model generated based on 3D face training data. The 3D standard model may include a 3D shape and a 3D texture, and parameters expressing the 3D shape. The facial recognition system <NUM> may generate a 3D facial model on the face of the user by matching feature points of the 3D standard model to the feature points extracted from the 2D face images. The generated 3D facial model may be registered and stored as a 3D facial model of the user appearing in the 2D face images.

Alternatively, the facial recognition system <NUM> may generate the 3D facial model of the user using the 2D face images used for the face registration, and motion data of the 2D face images and the 3D standard model. The facial recognition system <NUM> may obtain direction data of the 2D face images through a motion sensor along with the 2D face images, and generate 3D data on the face of the user based on the direction data and matching information of the 2D face images. The 3D data on the face of the user may be a set of 3D points configuring a shape of the face of the user. The facial recognition system <NUM> may generate the 3D facial model of the user by matching the 3D data on the face of the user to the 3D standard model. The generated 3D facial model may be stored and registered as a 3D facial model of the user appearing in the 2D face images.

In process <NUM>, the facial recognition system <NUM> obtains a 2D input image including a face region of the user through a camera. Although the facial recognition system <NUM> may perform facial recognition using a single 2D input image, example embodiments may not be limited thereto. In operation <NUM> of process <NUM>, the facial recognition system <NUM> adjusts the prestored 3D facial model of the user based on the facial pose or expression appearing in the 2D input image. The facial recognition system <NUM> may adjust a pose of the 3D facial model to match the facial pose appearing in the 2D input image, and adjust an expression of the 3D facial model to match the facial expression appearing in the 2D input image.

The facial recognition system <NUM> generates a 2D projection image from the 3D facial model matched to the 2D input image used for the facial recognition. In operation <NUM> of process <NUM>, the facial recognition system <NUM> performs the facial recognition by comparing the 2D input image to the 2D projection image and outputs a result of the facial recognition. For example, the facial recognition system <NUM> may determine a degree of similarity between a face region in the 2D input image and a face region in the 2D projection image, and output the result of the facial recognition as "facial recognition successful" in a case of the degree of similarity satisfying a predetermined and/or desired condition, and output "facial recognition failed" in other cases.

The facial recognition system <NUM> may include any one of a 3D facial model generating apparatus (e.g., a 3D facial model generating apparatus <NUM> of <FIG>, a 3D facial model generating apparatus <NUM> of <FIG>), and a facial recognition apparatus (e.g., a facial recognition apparatus <NUM> of <FIG>). The process <NUM> of registering the 3D facial model of the user may be performed by the 3D facial model generating apparatus <NUM> or the 3D facial model generating apparatus <NUM>. The process <NUM> of recognizing the face of the user from the 2D input image may be performed by the facial recognition apparatus <NUM>.

<FIG> is a diagram illustrating a configuration of the 3D facial model generating apparatus <NUM> according to at least one example embodiment. The 3D facial model generating apparatus <NUM> may generate a 3D facial model of a face of a user from a plurality of 2D face images used for face registration. The 3D facial model generating apparatus <NUM> may generate a 3D shape model and a 3D texture model as the 3D facial model, and register the generated 3D shape model and the generated 3D texture model as the 3D facial model of the user. Referring to <FIG>, the 3D facial model generating apparatus <NUM> includes an image acquirer <NUM>, a feature point detector <NUM>, a 3D facial model generator <NUM>, and a 3D facial model registerer <NUM>. The image acquirer <NUM>, the feature point detector <NUM>, the 3D facial model generator <NUM> and the 3D facial model registerer <NUM> may be implemented using hardware components and/or hardware components executing software components as is described below.

In the event where at least one of the image acquirer <NUM>, the feature point detector <NUM>, the 3D facial model generator <NUM> and the 3D facial model registerer <NUM> is a hardware component executing software, the hardware component is configured as a special purpose machine to execute the software, stored in a memory (non-transitory computer-readable medium) <NUM>, to perform the functions of the at least one of the image acquirer <NUM>, the feature point detector <NUM>, the 3D facial model generator <NUM> and the 3D facial model registerer <NUM>.

While the memory <NUM> is illustrated outside of the 3D facial model generating apparatus <NUM>, the memory <NUM> may be included in the 3D facial model generating apparatus <NUM>.

The image acquirer <NUM> obtains the 2D face images of the user for the face registration. The 2D face images may include a face region of the user including various facial poses. For example, the image acquirer <NUM> obtains the 2D face images captured through a camera from a plurality of viewpoints such as a front image or a profile image. Information on an overall 2D shape of the face of the user and texture information of the face of the user may be extracted from the front image, and detailed information on a shape of the face of the user may be extracted from the profile image. For example, information on a 3D shape of the face of the user may be determined by the 3D facial model generating apparatus <NUM> by comparing a face region of the user in the front image to a face region of the user in the profile image. According to an example embodiment, the image acquirer <NUM> may capture the 2D face images through a camera to register a 3D facial model, and the image acquirer <NUM> may store the 2D face images captured through the camera in the memory <NUM>.

The feature point detector <NUM> detects a face region from a 2D face image and facial feature points or landmarks in the detected face region. For example, the feature point detector <NUM> may detect feature points positioned on contours of eyebrows, eyes, a nose, lips, and/or a chin from the 2D face images. According to an example embodiment, the feature point detector <NUM> may detect the facial feature points from the 2D face images using an active shape model (ASM), an active appearance model (AAM), or a supervised descent method (SDM).

The 3D facial model generator <NUM> generates the 3D facial model on the face of the user based on the feature points detected from the 2D face images. A deformable 3D shape model and a deformable 3D texture model on the face of the user may be generated as the 3D facial model. The 3D facial model generator <NUM> includes a 3D shape model generator <NUM> and a 3D texture model generator <NUM>.

The 3D shape model generator <NUM> generates the 3D shape model of the face of the user using the 2D face images captured from different viewpoints. The 3D shape model refers to a 3D model having a shape without a texture. The 3D shape model generator <NUM> generates the 3D shape model based on the facial feature points detected from the 2D face images. The 3D shape model generator <NUM> determines a parameter to map the feature points detected from the 2D face images to feature points of a 3D standard model, and generates the 3D shape model by applying the determined parameter to the 3D standard model. For example, the 3D shape model generator <NUM> may generate the 3D shape model of the face of the user by matching feature points of eyebrows, eyes, a nose, lips, and/or a chin detected from the 2D face images to the feature points of the 3D standard model.

Generating a 3D shape model using 2D face images captured from different viewpoints may enable generation of a more detailed 3D shape model. In a case of generating a 3D shape model only using a front image obtained by capturing a face of a user from a front side, determining a 3D shape such as a height of a nose and a shape of cheekbones in the 3D shape model may not be easy. However, in a case of generating a 3D shape model using a plurality of 2D face images captured from different viewpoints, a more detailed 3D shape model may be generated because information on, for example, a height of a nose and a shape of cheekbones, may be additionally considered.

The 3D texture model generator <NUM> generates the 3D texture model based on texture information extracted from at least one of the 2D face images and the 3D shape model. For example, the 3D texture model generator <NUM> may generate a 3D texture model by mapping a texture extracted from a front image to the 3D shape model. The 3D texture model refers to a model having both a shape and a texture of a 3D model. The 3D texture model may have a higher level of detail than the 3D shape model, and include vertexes of the 3D shape model. The 3D shape model and the 3D texture model may have a fixed shape of a 3D model, and a deformable pose and expression. The 3D shape model and the 3D texture model may have an identical or similar pose and expression by an identical parameter.

The 3D facial model registerer <NUM> registers and stores the 3D shape model and the 3D texture model as a 3D facial model of the user. For example, when a user of a 2D face image obtained by the image acquirer <NUM> is "A," the 3D facial model registerer <NUM> may register a 3D shape model and a 3D texture model generated with respect to A as a 3D facial model of A and the memory <NUM> may store the 3D shape model and the 3D texture model of A.

<FIG> is a diagram illustrating a configuration of a facial recognition apparatus <NUM> according to at least one example embodiment. The facial recognition apparatus <NUM> may perform facial recognition for a user appearing in a 2D input image used for the facial recognition using a registered 3D facial model. The facial recognition apparatus <NUM> may generate a 2D projection image by rotating the 3D facial model to allow the 3D facial model to have a facial pose identical or similar to a facial pose of the user appearing in the 2D input image. The facial recognition apparatus <NUM> may perform the facial recognition by comparing the 2D projection image to the 2D input image. The facial recognition apparatus <NUM> may provide a facial recognition method robust against a change in a pose of the user by matching the registered 3D facial model to the facial pose appearing in the 2D input image and performing the facial recognition. Referring to <FIG>, the facial recognition apparatus <NUM> includes an image acquirer <NUM>, a 3D facial model processor <NUM>, and a face recognizer <NUM>. The 3D facial model processor <NUM> includes a face region detector <NUM> and a feature point detector <NUM>.

The image acquirer <NUM>, the 3D facial model processor <NUM> (including the face region detector <NUM> and the feature point detector <NUM>), and the face recognizer <NUM> may be implemented using hardware components and/or hardware components executing software components as is described below.

In the event where at least one of the image acquirer <NUM>, the 3D facial model processor <NUM> (including the face region detector <NUM> and the feature point detector <NUM>), and the face recognizer <NUM> is a hardware component executing software, the hardware component is configured as a special purpose machine to execute the software, stored in a memory (non-transitory computer-readable medium) <NUM>, to perform the functions of the at least one of the image acquirer <NUM>, the 3D facial model processor <NUM> (including the face region detector <NUM> and the feature point detector <NUM>), and the face recognizer <NUM>.

While the memory <NUM> is illustrated as part of the facial recognition apparatus <NUM>, the memory <NUM> may be separate from the facial recognition apparatus <NUM>.

The image acquirer <NUM> obtains a 2D input image for recognizing a face including a face region of a user. The image acquirer <NUM> obtains a 2D input image for recognizing or authenticating the user through a camera or the like. Although the facial recognition apparatus <NUM> may perform facial recognition on the user using a single 2D input image, example embodiments are not limited thereto.

The face region detector <NUM> detects the face region of the user from the 2D input image. The face region detector <NUM> identifies the face region from the 2D input image using information on a brightness distribution, a movement of an object, a color distribution, an eye location, and the like of the 2D input image, and extracts location information of the face region. For example, the face region detector <NUM> detects the face region from the 2D input image using a Haar-based cascade Adaboost classifier which is generally used in related technical fields.

The feature point detector <NUM> detects a facial feature point from the face region of the 2D face image. For example, the feature point detector <NUM> detects, from the face region, feature points including eyebrows, eyes, a nose, lips, a chin, hair, ears, and/or a facial contour. According to an example embodiment, the feature point detector <NUM> detects the facial feature point from the 2D input image using an ASM, an AAM, or an SDM.

The 3D facial model processor <NUM> adjusts a prestored 3D facial model based on the detected feature point. The 3D facial model processor <NUM> matches the 3D facial model to the 2D input image based on the detected feature point. Based on a result of the matching, the 3D facial model may be transformed to be matched to a facial pose and expression appearing in the 2D input image. The 3D facial model processor <NUM> adjusts a pose and an expression of the 3D facial model by mapping the feature point detected from the 2D input image to the 3D facial model. The 3D facial model may include a 3D shape model and a 3D texture model. The 3D shape model may be used to be fast matched to the facial pose appearing in the 2D input image, and the 3D texture model may be used to generate a high-resolution 2D projection image.

The 3D facial model processor <NUM> adjusts a pose of the 3D shape model based on the pose appearing in the 2D input image. The 3D facial model processor <NUM> matches the pose of the 3D shape model to the pose appearing in the 2D input image by matching the feature point detected from the 2D input image to feature points of the 3D shape model. The 3D facial model processor <NUM> adjusts a pose parameter and an expression parameter of the 3D shape model based on the feature point detected from the 2D input image.

In addition, the 3D facial model processor <NUM> adjusts the 3D texture model based on parameter information of the 3D shape model. The 3D facial model processor <NUM> applies, to the 3D texture model, the pose parameter and the expression parameter determined in the matching of the 3D shape model to the 2D input image. Based on a result of the applying, the 3D texture model may be adjusted to have a pose and an expression identical or similar to the pose and the expression of the 3D shape model. Subsequent to the adjusting of the 3D texture model, the 3D facial model processor <NUM> may generate the 2D projection image by projecting the adjusted 3D texture model to a plane.

The face recognizer <NUM> performs the facial recognition by comparing the 2D projection image to the 2D input image. The face recognizer <NUM> performs the facial recognition based on a degree of similarity between the face region appearing in the 2D input image and a face region appearing in the 2D projection image. The face recognizer <NUM> determines the degree of similarity between the 2D input image and the 2D projection image, and outputs a result of the facial recognition based on whether the determined degree of similarity satisfies a predetermined and/or desired condition.

The face recognizer <NUM> may use a feature value determining method which is generally used in a field of facial recognition technology to determine the degree of similarity between the 2D input image and the 2D projection image. For example, the face recognizer <NUM> may determine the degree of similarity between the 2D input image and the 2D projection image using a feature extracting filter such as a Gabor filter, a local binary pattern (LBP), a histogram of oriented gradient (HoG), a principal component analysis (PCA), and a linear discriminant analysis (LDA). The Gabor filter refers to a filter to extract a feature from an image using a multifilter having various magnitudes and angles. The LBP refers to a filter to extract a difference between a current pixel and an adjacent pixel as a feature from an image. According to an example embodiment, the face recognizer <NUM> may divide the face region appearing in the 2D input image and the 2D projection image into cells of a predetermined and/or selected size and calculate a histogram associated with the LBP for each cell, for example, a histogram on LBP index values included in a cell. The face recognizer <NUM> determines a vector obtained by linearly connecting the calculated histograms to be a final feature value, and compare a final feature value of the 2D input image to a final feature value of the 2D projection image to determine the degree of similarity between the 2D input image and the 2D projection image.

According to an example embodiment, the facial recognition apparatus <NUM> further includes a display <NUM>. The display <NUM> displays the 2D input image, the 2D projection image, and/or the result of the facial recognition. In a case that the user determines that a face of the user is not properly captured based on the displayed 2D input image, or the display <NUM> displays a final result of the facial recognition to be a failure, the user may re-capture the face and the facial recognition apparatus <NUM> may re-perform facial recognition on a 2D input image generated by the re-capturing.

<FIG> illustrates a process of detecting feature points from 2D face images according to at least one example embodiment. Referring to <FIG>, an image <NUM> is a 2D face image obtained by a 3D facial model generating apparatus by capturing a face of a user from a front side, and an image <NUM> and an image <NUM> are 2D face images obtained by the 3D facial model generating apparatus (e.g., <NUM> and <NUM>) by capturing the face of the user from profile sides. Information on an overall 2D shape of the face of the user and texture information of the face of the user may be extracted by the 3D facial model generating apparatus (e.g., <NUM> and <NUM>) from the image <NUM>. More detailed information on a shape of the face may be extracted from the images <NUM> and <NUM>. For example, a basic model on the face of the user may be set based on the shape of the face of the user extracted from the image <NUM>, and a 3D shape of the basic model may be determined by the 3D facial model generating apparatus (e.g., <NUM> and <NUM>) based on the shape of the face of the user extracted from the images <NUM> and <NUM>.

The feature point detector <NUM> of the 3D facial model generating apparatus <NUM> of <FIG> may detect facial feature points from 2D face images captured from a plurality of viewpoints such as the images <NUM>, <NUM>, and <NUM>. The facial feature points refer to feature points located in contour regions of eyebrows, eyes, a nose, lips, a chin, and the like. The feature point detector <NUM> may detect the facial feature points from the images <NUM>, <NUM>, and <NUM> using an ASM, an AAM, or an SDM which is generally used in related technical fields. Initialization of a pose, a scale, or a location of the ASM, the AAM, or the SDM model may be performed based on a result of face detection.

An image <NUM> is a resulting image from which feature points <NUM> are detected within a face region <NUM> of the image <NUM>. An image <NUM> is a resulting image from which feature points <NUM> are detected within a face region <NUM> of the image <NUM>. Similarly, an image <NUM> is a resulting image from which feature points <NUM> are detected within a face region <NUM> of the image <NUM>.

<FIG> illustrates a process of generating a 3D facial model using a 3D standard model according to at least one example embodiment. Referring to <FIG>, a model <NUM> indicates a 3D standard model. The 3D standard model, which is a deformable 3D shape model generated based on 3D face training data, may be a parametric model indicating an identity of a face of a user through an average shape and parameter.

The 3D standard model may include an average shape and a quantity of a change in a shape as expressed in Equation <NUM>. The quantity of a change in a shape indicates a weighted sum of a shape parameter and a shape vector.

In Equation <NUM>, "S" denotes elements configuring a 3D shape of a 3D standard model, and "S<NUM>" denotes elements associated with an average shape of the 3D standard model. "Si" denotes shape elements corresponding to an index factor "i," and "pi" denotes a shape parameter to be applied to the shape elements corresponding to the index factor i.

S may include coordinates of 3D points as expressed in Equation <NUM>.

In Equation <NUM>, "S" denotes a variable indicating an index of 3D points, for example, x, y, and z, and "T" denotes "transpose.

The 3D shape model generator <NUM> of the 3D facial model generating apparatus <NUM> of <FIG> may individualize a 3D standard model based on 2D face images captured from a plurality of viewpoints to register a face of a user. The 3D shape model generator <NUM> may determine a parameter to match feature points included in the 3D standard model to facial feature points detected from the 2D face images, and generate a 3D shape model on the face of the user by applying the determined parameter to the 3D standard model.

Referring to <FIG>, a model <NUM> and a model <NUM> are 3D shape models of the face of the user generated from the model <NUM>, which is the 3D standard model. The model <NUM> indicates a 3D shape model viewed from a front side, and the model <NUM> indicates a 3D shape model viewed from a profile side. A 3D shape model may have shape information without texture information, and be used to be matched to a 2D input image at a high speed in a process of user authentication.

The 3D texture model generator <NUM> of <FIG> may generate a 3D texture model by mapping a texture extracted from at least one of the 2D face images to a surface of the 3D shape model. For example, the mapping of the texture to the surface of the 3D shape model may indicate adding depth information obtained from the 3D shape model to the texture information extracted from a 2D face image captured from a front side.

A model <NUM> and a model <NUM> are 3D texture models generated based on the 3D shape model. The model <NUM> indicates a 3D texture model viewed from a front side, and the model <NUM> indicates a 3D texture model viewed from a diagonal direction. A 3D texture model may be a model including both shape information and texture information and used to generate a 2D projection image in the process of user authentication.

The 3D shape model and the 3D texture model are a 3D model in which a face shape indicating a unique characteristic of the user is fixed and a pose or an expression is deformable. The 3D texture model may have a higher level of detail and include a greater number of vertexes compared to the 3D shape model. Vertexes included in the 3D shape model may be a subset of the vertexes included in the 3D texture model. The 3D shape model and the 3D texture model may indicate an identical or similar pose and expression by an identical parameter.

<FIG> illustrates a process of adjusting a 3D facial model based on a feature point detected from a 2D input image according to at least one example embodiment. Referring to <FIG>, an image <NUM> is a 2D input image input to a facial recognition apparatus for facial recognition or user authentication, and indicates a face pose image captured through a camera.

The face region detector <NUM> of the facial recognition apparatus <NUM> of <FIG> may detect a face region from the 2D input image, and the feature point detector <NUM> of <FIG> may detect feature points located on a contour of eyes, eyebrows, a nose, lips, or a chin in the detected face region. For example, the feature point detector <NUM> may detect a facial feature point from the 2D input image using an ASM, an AAM, or an SDM.

An image <NUM> is a resulting image obtained by detecting a face region <NUM> from the image <NUM> by the face region detector <NUM> and detecting feature points <NUM> within the face region <NUM> by the feature point detector <NUM>.

The 3D facial model processor <NUM> may match a preregistered and stored 3D shape model to the 2D input image. The 3D facial model processor <NUM> may adjust a parameter of the 3D shape model based on the facial feature point detected from the 2D input image to adjust a pose and an expression. A model <NUM> is a preregistered and stored 3D shape model of a face of a user, and a model <NUM> is a 3D shape model in which a pose and an expression are adjusted based on the feature points <NUM> detected from the image <NUM>. The 3D facial model processor <NUM> may adjust a pose of the prestored 3D shape model to be identical or similar to a facial pose appearing in the 2D input image. The face of the user takes a laterally rotated pose in the image <NUM>, which is the 2D input image, and the 3D shape model in which the pose is adjusted by the 3D facial model processor <NUM> takes a laterally rotated pose identical or similar to the pose of the user in the image <NUM>.

<FIG> illustrates a process of performing facial recognition by comparing a 2D input image to a 2D projection image according to at least one example embodiment. The 3D facial model processor <NUM> of the facial recognition apparatus <NUM> of <FIG> may adjust a pose parameter and an expression parameter of a 3D shape model based on a facial feature point detected from a 2D input image used for facial recognition. The 3D facial model processor <NUM> may apply the adjusted pose parameter and the adjusted expression parameter of the 3D shape model to a 3D texture model to adjust a pose and an expression of the 3D texture model to be identical or similar to the pose and the expression of the 3D shape model. Subsequently, the 3D facial model processor <NUM> may generate a 2D projection image by projecting the 3D texture model to an image plane. The face recognizer <NUM> may perform the facial recognition based on a degree of similarity between the 2D input image and the 2D projection image, and output a result of the facial recognition.

Referring to <FIG>, an image <NUM> is a 2D input image used for facial recognition. An image <NUM> is a reference image to be compared to the image <NUM>, which is the 2D input image, for the face recognizer <NUM> to perform the facial recognition. A region <NUM> included in the image <NUM> indicates a region in which a 2D projection image generated from a 3D texture model is reflected. For example, the image <NUM> may be a face region obtained by projecting, to an image plane, the texture model to which a texture is mapped to the 3D shape model <NUM> of <FIG>. The face recognizer <NUM> may perform the facial recognition by comparing a face region of a user appearing in the 2D input image to the face region appearing in the 2D projection image. Alternatively, the face recognizer <NUM> may perform the facial recognition by comparing the image <NUM>, which is the 2D input image, to an overall region of the image <NUM>, which is a resulting image obtained by reflecting the 2D projection image in the 2D input image.

<FIG> is a flowchart illustrating a 3D facial model generating method according to at least one example embodiment.

Referring to <FIG>, in operation <NUM>, a 3D facial model generating apparatus obtains 2D face images of a user captured through a camera from different viewpoints. The 2D face images may be used to register a face of the user. For example, the 2D face images may include images including various facial poses such as a front and a profile image.

In operation <NUM>, the 3D facial model generating apparatus detects facial feature points from the 2D face images. For example, the 3D facial model generating apparatus may detect facial feature points located on a contour of eyebrows, eyes, a nose, lips, a chin, and the like from the 2D face images using an ASM, an AAM, or an SDM which is generally known in related technical fields.

In operation <NUM>, the 3D facial model generating apparatus generates a 3D shape model based on the detected feature points. The 3D facial model generating apparatus may generate the 3D shape model by matching the feature points of the eyebrows, the eyes, the nose, the lips, the chin, and the like detected from the 2D face images to feature points of a 3D standard model. The 3D facial model generating apparatus determines a parameter to map the feature points detected from the 2D face images to the feature points of the 3D standard model, and generate the 3D shape model by applying the determined parameter to the 3D standard model.

In operation <NUM>, the 3D facial model generating apparatus generates a 3D texture model based on the 3D shape model and texture information extracted from a 2D face image. The 3D facial model generating apparatus may generate the 3D texture model on the face of the user by mapping a texture extracted from at least one 2D face image to the 3D shape model. The 3D texture model to which a parameter of the 3D shape model is applied may have a pose and an expression identical or similar to the 3D shape model.

In operation <NUM>, the 3D facial model generating apparatus registers and stores the 3D shape model and the 3D texture model as a 3D facial model of the user. The stored 3D shape model and the 3D texture model may be used to authenticate the user appearing in the 2D input image in a process of user authentication.

<FIG> is a flowchart illustrating a facial recognition method according to at least one example embodiment.

Referring to <FIG>, in operation <NUM>, a facial recognition apparatus detects a facial feature point from a 2D input image used for facial recognition. The facial recognition apparatus detects a face region from the 2D input image, and detects a facial feature points located on a contour of eyes, eyebrows, a nose, a chin, lips, and the like in the detected face region. For example, the facial recognition apparatus may detect the face region from the 2D input image using a Haar-based cascade Adaboost classifier, and detect the facial feature points within the face region using an ASM, an AAM, or an SDM.

In operation <NUM>, the facial recognition apparatus adjusts a preregistered 3D facial model of a user based on the feature point detected from the 2D input image. The facial recognition apparatus may match the preregistered 3D facial model to the 2D input image based on the feature point detected from the 2D input image. The facial recognition apparatus may transform the 3D facial model to match a facial pose and expression of the 3D facial model to a facial pose and expression appearing in the 2D input image.

The 3D facial model may include a 3D shape model and a 3D texture model. The facial recognition apparatus may adjust a pose of the 3D shape model based on the feature point detected from the 2D input image, and adjust the 3D texture model based on parameter information of the 3D shape model in which the pose is adjusted. The facial recognition apparatus may adjust a pose parameter and an expression parameter of the 3D shape model based on the feature point detected from the 2D input image, and may apply the adjusted parameters of the 3D shape model to the 3D texture model. Based on a result of the applying of the parameters, the 3D texture model may be adjusted to have a pose and an expression identical or similar to the pose and the expression of the 3D shape model.

In operation <NUM>, the facial recognition apparatus generates a 2D projection image from the 3D texture model. The facial recognition apparatus generates the 2D projection image by projecting, to a plane, the 3D texture model adjusted based on the 3D shape model in operation <NUM>. A facial pose appearing in the 2D projection image may be identical or similar to the facial pose appearing in the 2D input image. For example, when the facial pose of the user appearing in the 2D input image is a pose facing a profile side, the 2D projection image generated through operations <NUM> through <NUM> may have a facial pose of the 3D texture model facing a profile side identical or similar to the 2D input image.

In operation <NUM>, the facial recognition apparatus performs facial recognition by comparing the 2D input image to the 2D projection image. The facial recognition apparatus performs the facial recognition based on a degree of similarity between a face region appearing in the 2D input image and a face region appearing in the 2D projection image. The fade recognition apparatus determines the degree of similarity between the 2D input image and the 2D projection image, and outputs a result of the facial recognition based on whether the determined degree of similarity satisfies a predetermined and/or desired condition. For example, in a case that the degree of similarity between the 2D input image and the 2D projection image satisfies the predetermined and/or desired condition, the facial recognition apparatus may output a result of "facial recognition successful", and "facial recognition failed" in other cases.

<FIG> is a diagram illustrating another example of a configuration of a 3D facial model generating apparatus <NUM> according to at least one example embodiment. The 3D facial model generating apparatus <NUM> may generate a 3D facial model of a face of a user from a plurality of 2D face images used for face registration. The 3D facial model generating apparatus <NUM> may generate the 3D facial model of the user using the 2D face images captured from different directions, motion data on the 2D face images, and a 3D standard model. Referring to <FIG>, the 3D facial model generating apparatus <NUM> includes an image acquirer <NUM>, a motion sensing unit <NUM>, a 3D facial model generator <NUM>, and a 3D facial model registerer <NUM>.

The image acquirer <NUM>, the motion sensing unit <NUM>, the 3D facial model generator <NUM>, and the 3D facial model registerer <NUM> may be implemented using hardware components and/or hardware components executing software components as is described below.

In the event where at least one of the image acquirer <NUM>, the motion sensing unit <NUM>, the 3D facial model generator <NUM>, and the 3D facial model registerer <NUM> is a hardware component executing software, the hardware component is configured as a special purpose machine to execute the software, stored in a memory (non-transitory computer-readable medium) <NUM>, to perform the functions of the at least one of the image acquirer <NUM>, the motion sensing unit <NUM>, the 3D facial model generator <NUM>, and the 3D facial model registerer <NUM>.

The image acquirer <NUM> obtains 2D face images captured from different viewpoints used for face registration. The image acquirer <NUM> obtains the 2D face images in which a face of a user is captured through a camera from different directions. For example, the image acquirer <NUM> may obtain the 2D face images captured from different viewpoints, for example, a front image and a profile image.

The motion sensing unit <NUM> obtains direction data of the 2D face images. The motion sensing unit <NUM> determines the direction data of the 2D face images using motion data sensed through various sensors. The direction data of the 2D face images may include information on a direction from which each 2D face image is captured. For example, the motion sensing unit <NUM> may determine direction data of each 2D face image using an inertial measurement unit (IMU) such as an accelerometer, a gyroscope, and/or a magnetometer.

For example, the user may capture the face of the user by rotating a camera in different directions, and obtain the 2D face images captured from various viewpoints as a result of the capturing. During capture of the 2D face images, the motion sensing unit <NUM> may calculate motion data including, for example, a change in a speed, a direction, a roll, a pitch, and a yaw of the camera capturing the 2D face images, based on sensing information output from the IMU, and determine the direction data on directions from which the 2D face images are captured.

The 3D facial model generator <NUM> generates a 3D facial model of the user appearing in the 2D face images. The 3D facial model generator <NUM> detects facial feature points or landmarks from the 2D face images. For example, the 3D facial model generator <NUM> may detect feature points located on a contour of eyebrows, eyes, a nose, lips, a chin, and the like from the 2D face images. The 3D facial model generator <NUM> determines information on matching points among the 2D face images based on the facial feature points detected from the 2D face images.

The 3D facial model generator <NUM> generates 3D data of the face of the user based on information on the facial feature points detected from the 2D face images, the information on the matching points, and the direction data of the 2D face images. For example, the 3D facial model generator <NUM> may generate the 3D data of the face of the user using an existing stereo matching method. The 3D data of the face of the user may be a set of 3D points configuring a shape or a surface of the face of the user.

The 3D facial model generator <NUM> transforms a deformable 3D standard model to a 3D facial model of the user using the 3D data on the face of the user. The 3D facial model generator <NUM> transforms the 3D standard model to the 3D facial model of the user by matching the 3D standard model to the 3D data on the face of the user. The 3D facial model generator <NUM> transforms the 3D standard model to the 3D facial model of the user by matching feature points of the 3D data to feature points of the 3D standard model. The 3D facial model of the user may include a 3D shape model associated with a shape of the face of the user and/or a 3D texture model including texture information.

The 3D facial model registerer <NUM> registers and stores the 3D facial model of the user generated by the 3D facial model generator <NUM>. The stored 3D facial model of the user may be used to recognize the face of the user and a shape of the 3D facial model may be transformed in a process of facial recognition.

<FIG> is a flowchart illustrating another 3D facial model generating method according to at least one example embodiment.

Referring to <FIG>, in operation <NUM>, a 3D facial model generating apparatus obtains a plurality of 2D face images used for face registration and direction data of the 2D face images. The 3D facial model generating apparatus obtains the 2D face images of a user captured through a camera from different viewpoints. The 3D facial model generating apparatus obtains the 2D face images in which a face of the user is captured from different directions, for example, a front image and a profile image.

The 3D facial model generating apparatus obtains the direction data of the 2D face images using motion data sensed by a motion sensor. For example, the 3D facial model generating apparatus may obtain direction data of each 2D face image using motion data sensed by an IMU including an accelerometer, a gyroscope, and/or a magnetometer. The direction data of the 2D face images may include information on a direction from which each 2D face image is captured.

In operation <NUM>, the 3D facial model generating apparatus determines information on matching points among the 2D face images. The 3D facial model generating apparatus detects facial feature points from the 2D face images, and detects the matching points based on the detected feature points.

In operation <NUM>, the 3D facial model generating apparatus generates 3D data on the face of the user. For example, the 3D data of the face of the user may be a set of 3D points configuring a shape or a surface of the face of the user, and include a plurality of vertexes. The 3D facial model generating apparatus generates the 3D data of the face of the user based on information of the facial feature points detected from the 2D face images, the information on the matching points, and the direction data of the 2D face images. The 3D facial model generating apparatus may generate the 3D data of the face of the user using an existing stereo matching method.

In operation <NUM>, the 3D facial model generating apparatus transforms a 3D standard model to a 3D facial model of the user using the 3D data generated in operation <NUM>. The 3D facial model generating apparatus transforms the 3D standard model to the 3D facial model of the user by matching the 3D standard model to the 3D data on the face of the user. The 3D facial model generating apparatus generates the 3D facial model of the user by matching feature points of the 3D standard model to feature points of the 3D data. The 3D facial model generating apparatus generates a 3D shape model and/or a 3D texture model as the 3D facial model of the user. The generated 3D facial model of the user may be stored and registered, and be used to recognize the face of the user.

The units and/or modules described herein may be implemented using hardware components and/or hardware components executing software components. For example, the hardware components may include microphones, amplifiers, band-pass filters, audio to digital convertors, and processing devices. A processing device may be implemented using one or more hardware device configured to carry out and/or execute program code by performing arithmetical, logical, and input/output operations. The processing device(s) may include a processor, a controller and an arithmetic logic unit, a digital signal processor, a microcomputer, a field programmable array, a programmable logic unit, a microprocessor or any other device capable of responding to and executing instructions in a defined manner. The processing device may run an operating system (OS) and one or more software applications that run on the OS. The processing device also may access, store, manipulate, process, and create data in response to execution of the software. For purpose of simplicity, the description of a processing device is used as singular; however, one skilled in the art will appreciated that a processing device may include multiple processing elements and multiple types of processing elements. For example, a processing device may include multiple processors or a processor and a controller. In addition, different processing configurations are possible, such a parallel processors.

Claim 1:
A facial recognition method, comprising:
detecting (<NUM>) facial feature points from a two-dimensional (2D) input image of a user to be recognized;
adjusting (<NUM>) a registered, and stored three-dimensional (3D) facial model of the user, the 3D facial model comprising a 3D shape model and a 3D texture model, the adjusting comprising:
adjusting a pose parameter and an expression parameter of the 3D shape model based on the detected facial feature points; and,
applying the adjusted parameters of the 3D shape model to the 3D texture model;
generating (<NUM>) a 2D projection image from the adjusted 3D facial model, comprising projecting the 3D facial model matched to the 2D input image at a viewpoint identical or similar to a viewpoint in the 2D input image, a facial pose appearing in the 2D projection image being identical or similar to a facial pose of the user appearing in the 2D input image; and
performing (<NUM>) facial recognition by comparing the 2D input image and the 2D projection image.