Patent Description:
Facial recognition (i.e., face recognition) generally refers to a method of identifying a user by comparing a live capture or digital image data against one or more stored records for the user. Facial recognition is becoming a popular way to control user access to devices, locations, and services.

<CIT> discloses a system for detecting a genuine user which comprises a retriever, a coplanarity determiner, a constructor and a detector, wherein the retriever receives an image sequence of a subject including at least a first and a second image.

<CIT> pertains to distinguishing live faces from flat surfaces and suggests that multiple images including a face presented by a user are accessed, wherein one or more determinations are made based on the multiple images, such as a determination of whether the face included in the multiple images is a <NUM>-dimensional structure or a flat surface and/or a determination of whether motion is present in one or more face components.

<CIT> discloses a method of analyzing an image of a user to determine whether the image is authentic, wherein a first image of a user's face is received with a camera and four or more two-dimensional feature points can be located that do not lie on the same two-dimensional plane, and wherein additional images of the user's face are received and the at least four two-dimensional feature points can be located on each additional image with the image processor.

For example, facial recognition systems are widely used for biometric authentication in our daily lives. Such systems, however, are highly vulnerable to various types of spoofing attacks.

One embodiment provides a method for face liveness detection as defined in the appended claims. Another embodiment provides an electronic device for face liveness detection as defined in the appended claims.

These and other features, aspects and advantages of the one or more embodiments will become understood with reference to the following description, appended claims and accompanying figures.

The following description is made for the purpose of illustrating the general principles of one or more embodiments and is not meant to limit the inventive concepts claimed herein. Further, particular features described herein can be used in combination with other described features in each of the various possible combinations and permutations. Unless otherwise specifically defined herein, all terms are to be given their broadest possible interpretation including meanings implied from the specification as well as meanings understood by those skilled in the art and/or as defined in dictionaries, treatises, etc..

In this specification, the term "presentation attack" is used to generally refer to a method of spoofing (i.e., a spoofing attack) involving presenting a two-dimensional (2D) image of a face of a user to an image sensor (e.g., a camera) utilized by a facial recognition system. The 2D image may be presented to the image sensor using a printed medium (e.g., a printed photograph) or a display device (e.g., a mobile device, such as a smartphone, a tablet, etc.). For example, an attacker may obtain a portrait picture of a user, print/display the picture on a planar surface (i.e., a printed medium or a display device), and present the planar surface to the image sensor.

In this specification, the term "face liveness detection" is used to generally refer to a process of determining whether an object presented to an image sensor (e.g., a camera) utilized by a facial recognition system is a real three-dimensional (3D) face of a user (e.g., the user is standing in front of the image sensor) or a 2D facial image of the user (e.g., the 2D facial image is presented to the image sensor using a printed medium or a display device as part of a presentation attack).

One or more embodiments relate generally to facial recognition, and in particular, a method and system for detecting a presentation attack via a facial recognition system. One embodiment provides a method for face liveness detection. The method comprises receiving a first image comprising a face of a user, determining one or more two-dimensional (2D) facial landmark points based on the first image, and determining a three-dimensional (3D) pose of the face in the first image based on the one or more determined 2D facial landmark points detected and one or more corresponding 3D facial landmark points in a 3D face model for the user. The method further comprises determining a homography mapping between the one or more determined 2D facial landmark points and one or more corresponding 3D facial landmark points that are perspectively projected based on the 3D pose, and determining liveness of the face in the first image based on the homography mapping.

Conventional methods for face liveness detection utilize multiple images of a user with different 3D pose variations to determine whether a face presented to an image sensor of a facial recognition system is a real 3D face or a 2D facial image printed/ displayed on a planar surface, such as a printed medium or a display device. Such conventional methods involve determining scene geometry by tracking key points (e.g., 2D facial landmark points) in multiple images. Such conventional methods affect usability of facial recognition systems as they require more user actions (e.g., multiple images of a user are required).

One embodiment provides a facial recognition system configured to detect a presentation attack by determining whether an object (e.g., a face of an actual, live user, a printed photograph, etc.) presented to an image sensor utilized by the system has a 2D planar surface (e.g., the object is a 2D printed/displayed image presented to the camera using a printed medium or a display device) or a live 3D surface (e.g., the object is an actual 3D face of a user positioned within proximity of the image sensor). In one embodiment, the system is configured to perform at least one of the following: (<NUM>) 3D shape construction from multiple views, (<NUM>) image sensor calibration using a planar calibration object, (<NUM>) determining 2D facial landmark points, (<NUM>) 3D pose estimation, and (<NUM>) determining a homography mapping that maps facial landmark points on one plane to another. The system is configured to detect spoofing attacks such as presentation attacks using a known 3D facial shape of a user and a calibrated image sensor to determine scene and image sensor geometry. In one embodiment, the system is configured to perform face liveness detection based on a single image captured by the image sensor and one or more camera intrinsic parameters for the image sensor. Unlike conventional methods, the system does not require capture of multiple facial images with 3D pose variations to detect a presentation attack, thereby reducing turnaround time.

<FIG> illustrates an example computing architecture <NUM> for implementing a facial recognition system <NUM>, in one or more embodiments. The computing architecture <NUM> comprises an electronic device <NUM> including computation resources, such as one or more processors <NUM> and one or more storage units <NUM>. One or more applications may execute/operate on the electronic device <NUM> utilizing the computation resources of the electronic device <NUM>.

Examples of an electronic device <NUM> include, but are not limited to, a desktop computer, a mobile electronic device (e.g., a tablet, a smart phone, a laptop, etc.), or a consumer product such as a smart television, a smart car, or any other product utilizing facial recognition for authentication.

In one embodiment, the electronic device <NUM> comprises an image sensor <NUM> integrated in or coupled to the electronic device <NUM>, such as a camera. One or more applications on the electronic device <NUM> may utilize the image sensor <NUM> to capture an image of an object presented to the image sensor <NUM>.

In one embodiment, the applications on the electronic device <NUM> include, but are not limited to, a facial recognition system <NUM> configured to perform at least one of the following: (<NUM>) receive an image of an object, and in one embodiment the object is presented to the image sensor <NUM>, (<NUM>) perform facial verification to verify that the image received and stored records for a registered user capture the same user, and (<NUM>) perform face liveness detection to determine whether the object is a real 3D face of a user <NUM> or a 2D facial image printed/displayed <NUM> on a planar surface, such as a printed medium (e.g., a printed photograph) or a display device (e.g., a mobile device, such as a smartphone, a tablet, etc.). In one embodiment, the facial recognition system <NUM> further configured to receive a first image comprising a portion corresponding to a face of a user, obtain two-dimensional (2D) image information related to the face of the user based on the first image, obtain three-dimensional (3D) shape information of the face of the user based on a 3D face model for the user, providing a mapping between the 2D image information and the 3D shape information(e.g., a mapping between one or more determined 2D facial landmark points included the 2D image information and one or more corresponding perspectively projected 3D facial landmark points included the 3D shape information), and identify liveness of the face in the first image based on the mapping.

In one embodiment, the applications on the electronic device <NUM> may further include one or more software mobile applications <NUM> loaded onto or downloaded to the electronic device <NUM>, such as a mobile banking application. A software mobile application <NUM> on the electronic device <NUM> may exchange data with the facial recognition system <NUM>. For example, a mobile banking application may invoke the facial recognition system <NUM> to verify an identity of a user when the user logins to the mobile banking application.

In one embodiment, the electronic device <NUM> may further include one or more additional sensors, such as a microphone, a GPS, or a depth sensor. A sensor of the electronic device <NUM> may be utilized to capture content and/or sensor-based contextual information. For example, the facial recognition system <NUM> and/or a software mobile application <NUM> may utilize the one or more additional sensors of the electronic device <NUM> to capture content and/or sensor-based contextual information, such as a microphone for audio data (e.g., voice recordings), a GPS for location data (e.g., location coordinates), or a depth sensor for a shape of an object presented to the image sensor <NUM>.

In one embodiment, the electronic device <NUM> comprises one or more input/output (I/O) units <NUM> integrated in or coupled to the electronic device <NUM>, such as a keyboard, a keypad, a touch interface, or a display screen.

In one embodiment, the electronic device <NUM> is configured to exchange data with one or more remote servers <NUM> or remote electronic devices over a connection (e.g., a wireless connection such as a WiFi connection or a cellular data connection, a wired connection, or a combination of the two). For example, a remote server <NUM> may be an online platform for hosting one or more online services (e.g., an online banking service) and/or distributing one or more software mobile applications <NUM>.

In one embodiment, the computing architecture <NUM> is a centralized computing architecture. In another embodiment, the computing architecture <NUM> is a distributed computing architecture.

<FIG> illustrates the facial recognition system <NUM> in detail, in one or more embodiments. In one embodiment, the facial recognition system <NUM> utilizes the image sensor <NUM> to capture an image of an object presented to the image sensor <NUM>. The image sensor <NUM> may be invoked to capture an image of an object by the facial recognition system <NUM> and/or a software mobile application <NUM> on the electronic device <NUM>.

In one embodiment, the facial recognition system <NUM> has at least two different operating phases (i.e., modes) - a registration (i.e., setup) phase and a recognition phase. As described in detail later herein, the registration phase involves performing at least one of the following: (<NUM>) calibration of the image sensor <NUM>, and (<NUM>) registration of a user(i.e., a new user or first user) with the facial recognition system <NUM> using multiple images of the user at different 3D poses. In one embodiment, the facial recognition system <NUM> comprises a user registration system <NUM> configured to perform the registration phase.

In one embodiment, the registration phase is performed once for each user.

In one embodiment, the registration phase may take place offline (i.e., not on the electronic device <NUM>). For example, in one embodiment, the registration phase may take place utilizing a remote server <NUM> or a remote electronic device.

As described in detail later herein, the recognition phase involves performing at least one of the following: (<NUM>) facial verification based on a single image captured by the image sensor <NUM>, and (<NUM>) face liveness detection based on the single image. By requiring only a single image for face liveness detection, the facial recognition system <NUM> removes the need for an expensive sensor to capture a 3D shape of an object presented to the image sensor <NUM> during the recognition phase.

In one embodiment, the facial recognition system <NUM> comprises a face liveness detection system <NUM> configured to perform the recognition phase.

In one embodiment, the recognition phase is executed each time the image sensor <NUM> captures an image of an object presented to the image sensor <NUM> for facial recognition.

In one embodiment, the recognition phase may take place online (i.e., on the electronic device <NUM>).

<FIG> illustrates one or more components of a user registration system <NUM>, in one or more embodiments. In this specification, the term "camera intrinsic parameter" generally refers to a parameter for an image sensor (e.g., a camera), wherein the parameter is associated with an image formation process. Examples of different camera intrinsic parameters include, but are not limited to, focal length, center of projection, radial distortion, etc. In this specification, the term "camera intrinsic matrix" generally refers to a matrix representing one or more camera intrinsic parameters for an image sensor.

In one embodiment, the user registration system <NUM> comprises a calibration unit <NUM>. In the registration phase, the calibration unit <NUM> is configured to: (<NUM>) perform a calibration of the image sensor <NUM> of the electronic device <NUM>, and (<NUM>) based on the calibration, generate camera intrinsic information <NUM> comprising one or more camera intrinsic parameters for the image sensor <NUM>. In one embodiment, the calibration performed by the calibration unit <NUM> involves utilizing the image sensor <NUM> to capture multiple images of a planar calibration object (e.g., a checker board, etc.) at different 3D poses. The camera intrinsic information <NUM> may be stored on the one or more storage units <NUM> of the electronic device <NUM> as stored records for the image sensor <NUM>.

In one embodiment, the image sensor <NUM> needs only be calibrated once if the image sensor <NUM> is a fixed focal length camera.

In one embodiment, the user registration system <NUM> comprises a multi-view capture unit <NUM> configured to receive a request to register a user with the facial recognition system <NUM>, wherein the request includes multiple facial images <NUM> of the user at different 3D poses. In one embodiment, the multiple facial images <NUM> may be captured using the image sensor <NUM> or a different image sensor (e.g., a camera on a remote electronic device). The multiple facial images <NUM> may be stored on the one or more storage units <NUM> of the electronic device <NUM> as stored records for the registered user.

In one embodiment, the user registration system <NUM> comprises a 3D shape reconstruction unit <NUM> configured to generate a 3D face model (i.e., structure, shape) <NUM> of a face of a user using motion techniques. Specifically, the 3D shape reconstruction unit <NUM> is configured to: (<NUM>) receive camera intrinsic information <NUM> for the image sensor <NUM> from the calibration unit <NUM>, (<NUM>) receive multiple facial images <NUM> of the user at different 3D poses from the multi-view capture unit <NUM>, (<NUM>) determine/identify and track 2D facial landmark points in the multiple facial images <NUM>, and (<NUM>) generate a corresponding 3D face model <NUM> for the user by recovering/reconstructing a 3D shape of the face of the user based on the determined 2D facial landmark points and the camera intrinsic information <NUM>. The corresponding 3D face model <NUM> may be stored on the one or more storage units <NUM> of the electronic device <NUM> as stored records for the registered user. The corresponding 3D face model <NUM> may include one or more labeled 3D facial landmark points that are reconstructed from the determined 2D facial landmark points in the multiple facial images <NUM>. As described in detail later herein, the corresponding 3D face model <NUM> generated during the registration phase may be utilized during the recognition phase.

In one embodiment, a 3D face model <NUM> for a user may be formed using sensor-based contextual information captured using one or more depth sensors of the electronic device <NUM> or a remote electronic device.

<FIG> illustrates one or more components of a face liveness detection system <NUM>, in one or more embodiments. In this specification, term "query image" generally refers to a single image of an object presented to the image sensor <NUM> during the recognition phase, wherein the query image is captured/formed by the image sensor <NUM> and forwarded to the facial recognition system <NUM> for facial recognition.

In one embodiment, the face liveness detection system <NUM> comprises a single capture unit <NUM> configured to receive a request for facial recognition of an object presented to the image sensor <NUM>. In one embodiment, the request is received from at least one of a software mobile application <NUM> on the electronic device <NUM>, a remote server <NUM>, or a remote electronic device. The request includes a query image <NUM> of the object, wherein the query image <NUM> is captured by the image sensor <NUM>.

In one embodiment, the face liveness detection system <NUM> comprises a landmark determination unit <NUM> configured to: (<NUM>) receive a query image <NUM> of an object from the single capture unit <NUM>, and (<NUM>) determine/identify 2D facial landmark points <NUM> in the query image <NUM>. In one embodiment, the determined 2D facial landmark points <NUM> are represented as coordinates relative to the query image <NUM> ("image coordinates").

In one embodiment, the face liveness detection system <NUM> comprises a facial verification unit <NUM> configured to: (<NUM>) receive a query image <NUM> of an object from the single capture unit <NUM>, (<NUM>) perform facial verification based on the query image <NUM>, and (<NUM>) generate a verification status <NUM> indicative of a result of the facial verification. In one embodiment, the facial verification unit <NUM> performs facial verification by comparing the query image <NUM> against stored records for a registered user (e.g., facial images captured during the registration phase and stored on the one or more storage units <NUM>) to determine if the query image <NUM> and the stored records capture the same user. In one embodiment, the verification status <NUM> is one of the following: (<NUM>) a positive verification status indicating that a user captured in the query image <NUM> is verified (i.e., the user is a registered user), or (<NUM>) a negative verification status indicating that the user captured in the query image <NUM> is not verified (i.e., the user is not a registered user).

In one embodiment, the face liveness detection system <NUM> comprises a 3D pose estimation unit <NUM> configured to determine an estimated 3D pose of a face of a user captured in a query image <NUM>.

In one embodiment, the face liveness detection system <NUM> comprises a first control unit <NUM> configured to control whether face liveness detection should be performed or bypassed based on a result of facial verification performed on a query image <NUM>. Specifically, in response to receiving a negative verification status from the facial verification unit <NUM>, the first control unit <NUM> bypasses face liveness detection and generates a failed verification report <NUM> indicating that a user captured in the query image <NUM> is not verified (i.e., not a registered user). In response to receiving a positive verification status <NUM> from the facial verification unit <NUM>, the first control unit <NUM> proceeds with face liveness detection by invoking the 3D pose estimation unit <NUM> to determine an estimated 3D pose of a face of a user captured in the query image <NUM>.

In one embodiment, the face liveness detection system <NUM> comprises a 3D pose and homography optimization unit <NUM> configured to receive, as inputs, each of the following: (<NUM>) a 3D face model <NUM> corresponding to a user captured in a query image <NUM> (e.g., a 3D face model <NUM> generated during the registration phase and stored on the one or more storage units <NUM>), (<NUM>) camera intrinsic information <NUM> for the image sensor <NUM> (e.g., camera intrinsic information <NUM> generated during the registration phase and stored on the one or more storage units <NUM>), (<NUM>) determined 2D facial landmark points <NUM> in the query image <NUM> (e.g., determined 2D facial landmark points <NUM> from the landmark determination unit <NUM>), and (<NUM>) an estimated 3D pose of a face of a user captured in the query image <NUM> (e.g., an estimated 3D pose from the 3D pose estimation unit <NUM>).

In one embodiment, each registered user has a corresponding 3D face model <NUM> of his/her face captured and stored during the registration phase. If the face liveness detection system <NUM> generates, in response to receiving a query image <NUM>, a positive verification status indicating that a user captured in the query image <NUM> is verified (i.e., the user is a registered user), the face liveness detection system <NUM> retrieves a 3D face model <NUM> corresponding to the user (e.g., from the one or more storage units <NUM>).

The 3D pose and homography optimization unit <NUM> is further configured to: (<NUM>) determine correspondences between the determined 2D facial landmark points <NUM> in the query image <NUM> and labeled 3D facial landmark points in the 3D face model <NUM>, (<NUM>) generate a first image of 2D facial landmark points by multiplying the determined 2D facial landmark points <NUM> with an inverse of a camera intrinsic matrix included in the camera intrinsic information <NUM> to transform the determined 2D facial landmark points <NUM> in image coordinates to coordinates relative to a homogenous camera ("homogenous camera coordinates"), (<NUM>) generate a second image of 3D facial landmark points by perspective projection of corresponding 3D facial landmark points based on the estimated 3D pose, and (<NUM>) jointly optimize via iterative refinement the estimated 3D pose and a homography mapping between the first image and the second image to reduce or minimize a distance between 2D facial landmark points in the first image and corresponding perspectively projected 3D landmarks points in the second image. In one embodiment, the 3D pose and homography optimization unit <NUM> is further configured to determine the homography mapping by decomposing the homography mapping based on one or more camera intrinsic parameters for the image sensor <NUM>.

In one embodiment, the face liveness detection system <NUM> further comprises: (<NUM>) a distance unit <NUM> configured to determine a distance between the homography mapping and an identity matrix representing an identity transform, and (<NUM>) a comparison unit <NUM> configured to compare the distance between the homography mapping and the identity matrix against a pre-determined threshold. In one embodiment, if the distance between the homography mapping and the identity matrix exceeds the pre-determined threshold, the comparison unit <NUM> determines that the object presented to the image sensor <NUM> has a planar 2D surface, and generates a failed face liveness detection report <NUM> indicating that the object is a 2D printed/ displayed facial image <NUM> that may have been presented to the image sensor <NUM> as part of a presentation attack (i.e., the object is a spoof). If the distance between the homography mapping and the identity matrix does not exceed the pre-determined threshold, the comparison unit <NUM> determines that the object has a live 3D surface, and generates a successful face liveness detection report <NUM> indicating that the object is a real 3D face (i.e., the object is live). In one embodiment, the homography mapping is the same as the identify matrix if the object is a real 3D face that is presented to the image sensor <NUM> at a particular scale and a particular distance. Therefore, the face liveness detection system <NUM> is configured to detect whether the image sensor <NUM> forms the query image <NUM> from an actual 3D object or a 2D image of the object.

In one embodiment, the face liveness detection system <NUM> is configured to perform face liveness detection based on a sequence of images captured utilizing the image sensor <NUM>. The face liveness detection system <NUM> aggregates results and makes a final determination as to whether an object presented to the image sensor <NUM> is live is based on the aggregated results.

<FIG> illustrates a face liveness detection process performed by the face liveness detection system <NUM>, in one or more embodiments. Assume an image sensor C" used to capture a real 3D face of a user. The face of the user has a 3D shape (i.e., model, structure) S and a 3D pose [RIT] relative to the image sensor C'. In one embodiment, the 3D shape S of the face of the user is known to the facial recognition system <NUM>. For example, in one embodiment, one or more components of the facial recognition system <NUM> (e.g., the calibration unit <NUM>, the multi-view capture unit <NUM>, and the 3D shape reconstruction unit <NUM>) are configured to determine/acquire the 3D shape S of the face of the user when the user registers with the facial recognition system <NUM> during the registration phase.

The image sensor C' forms a 2D image I' of the face of the user via multiple transformations including: (<NUM>) a perspective projection of the shape S onto a perspective projection plane P', and (<NUM>) an affine transformation K' by multiplying the perspective projection by a camera intrinsic matrix K' for the image sensor C'. An attacker may produce a 2D photograph J of the face of the user to use during a presentation attack by printing/displaying the 2D image I' onto a printed medium or a display device, such that the 2D image I' undergoes another affine transformation A.

Assume an image sensor C is used by a facial recognition system to capture an object presented to the facial recognition system for facial recognition (e.g., the image sensor <NUM> of the facial recognition system <NUM>). Assume an attacker presents the 2D photograph J to the image sensor C as part of a presentation attack. The face of the user in the 2D photograph J has a 3D pose [rlt] relative to the image sensor C. The image sensor C forms a 2D image I of the 2D photograph J via multiple transformations including: (<NUM>) a perspective projection of the 2D photograph J onto a perspective projection plane P, and (<NUM>) an affine transformation K by multiplying the perspective projection by a camera intrinsic matrix K for the image sensor C. In one embodiment, one or more components of the facial recognition system <NUM> (e.g., the calibration unit <NUM>) are configured to determine the camera intrinsic matrix K for the image sensor C by performing a calibration of the image sensor C during the registration phase.

In one embodiment, the 2D image I may be represented in accordance with equation Math <FIG> provided below:
<MAT>
wherein.

In one embodiment, one or more components of the facial recognition system <NUM> (e.g., the 3D pose and homography optimization <NUM>) is configured to estimate the homography mapping H and the 3D pose [R|T] of the face of the user relative to the image sensor C' using a non-linear optimization approach. Specifically, the facial recognition system <NUM> first initializes the homography mapping H to an identity matrix representing an identity transform and the 3D pose [R|T] to a frontal 3D pose. The facial recognition system <NUM> next determines correspondences between detected 2D facial landmark points in the 2D image I and labeled 3D facial landmark points in the 3D shape S. Using the correspondences, the facial recognition system <NUM> then estimates the homography mapping H and the 3D pose [R|T] by minimizing an error E in accordance with equation Math <FIG> provided below:
<MAT>
wherein.

In one embodiment, one or more components of the facial recognition system <NUM> (e.g., the distance unit <NUM> and the comparison unit <NUM>) is configured to detect a presentation attack based on the estimated homography mapping H, the estimated 3D pose [R|T], and the minimized error E. Specifically, the facial recognition system <NUM> is configured to detect a presentation attack based on: (<NUM>) a deviation of an estimated homography mapping H from the identity matrix, and (<NUM>) the minimized error E. In one embodiment, a neural network is trained for face liveness detection based on the minimized error E and a deviation of each element of the estimated homography mapping H from the identity matrix.

If there is no spoofing attempt (e.g., no presentation attack) and an object presented to the image sensor C is a real 3D face of the user instead of the 2D photograph J (e.g., the user is standing in front of the image sensor C), the homography mapping H is the same as the identity matrix.

In one embodiment, the facial recognition system <NUM> may be implemented as an object recognition system configured to recognize and perform liveness detection of different 3D objects (e.g., cars, animals, individuals, etc.).

<FIG> is a flowchart of an example process <NUM> for performing face liveness detection to detect a presentation attack, in one or more embodiments. Process block <NUM> includes receiving a first image (e.g., query image <NUM>) comprising a face of a user (e.g., as captured by an image sensor <NUM>). In one embodiment, the process block <NUM> includes receiving a first image comprising a portion corresponding to a face of a user. Process block <NUM> includes determining one or more 2D facial landmark points based the first image. In one embodiment, the process block <NUM> includes obtaining two-dimensional (2D) image information related to the face of the user based on the first image Process block <NUM> includes determining a 3D pose of the face in the first image based on the one or more determined 2D facial landmark points and one or more corresponding 3D facial landmark points in a 3D face model for the user. In one embodiment, the process block <NUM> includes obtaining three-dimensional (3D) shape information of the face of the user based on a 3D face model for the user. Process block <NUM> includes determining a homography mapping between the one or more determined 2D facial landmark points and one or more corresponding 3D facial landmark points that are perspectively projected based on the 3D pose. In one embodiment, the process block <NUM> includes providing a mapping between the 2D image information and the 3D shape information(e.g., a mapping between one or more determined 2D facial landmark points included the 2D image information and one or more corresponding perspectively projected 3D facial landmark points included the 3D shape information). Process block <NUM> includes determining liveness of the face in the first image based on the homography mapping (i.e., whether the face in the first image is a real 3D face). In one embodiment, the process block <NUM> includes identifing liveness of the face in the first image based on the mapping.

In one embodiment, process blocks <NUM>-<NUM> may be performed by one or more components of the facial recognition system <NUM>, such as the face liveness detection system <NUM>.

<FIG> is a high-level block diagram showing an information processing system comprising a computer system <NUM> useful for implementing the disclosed embodiments. Each system <NUM>, <NUM>, <NUM> may be incorporated in a display device or a server device. The computer system <NUM> includes one or more processors <NUM>, and can further include an electronic display device <NUM> (for displaying video, graphics, text, and other data), a main memory <NUM> (e.g., random access memory (RAM)), storage device <NUM> (e.g., hard disk drive), removable storage device <NUM> (e.g., removable storage drive, removable memory module, a magnetic tape drive, optical disk drive, computer readable medium having stored therein computer software and/or data), user interface device <NUM> (e.g., keyboard, touch screen, keypad, pointing device), and a communication interface <NUM> (e.g., modem, a network interface (such as an Ethernet card), a communications port, or a PCMCIA slot and card). The communication interface <NUM> allows software and data to be transferred between the computer system and external devices. The system <NUM> further includes a communications infrastructure <NUM> (e.g., a communications bus, cross-over bar, or network) to which the aforementioned devices/modules <NUM> through <NUM> are connected.

Information transferred via communications interface <NUM> may be in the form of signals such as electronic, electromagnetic, optical, or other signals capable of being received by communications interface <NUM>, via a communication link that carries signals and may be implemented using wire or cable, fiber optics, a phone line, a cellular phone link, an radio frequency (RF) link, and/or other communication channels. Computer program instructions representing the block diagram and/or flowcharts herein may be loaded onto a computer, programmable data processing apparatus, or processing devices to cause a series of operations performed thereon to generate a computer implemented process. In one embodiment, processing instructions for process <NUM> (<FIG>) may be stored as program instructions on the memory <NUM>, storage device <NUM>, and/or the removable storage device <NUM> for execution by the processor <NUM>.

Embodiments have been described with reference to flowchart illustrations and/or block diagrams of methods, apparatus (systems) and computer program products. Each block of such illustrations/diagrams, or combinations thereof, can be implemented by computer program instructions. The computer program instructions when provided to a processor produce a machine, such that the instructions, which execute via the processor create means for implementing the functions/operations specified in the flowchart and/or block diagram. Each block in the flowchart /block diagrams may represent a hardware and/or software module or logic. In alternative implementations, the functions noted in the blocks may occur out of the order noted in the figures, concurrently, etc..

The terms "computer program medium," "computer usable medium," "computer readable medium", and "computer program product," are used to generally refer to media such as main memory, secondary memory, removable storage drive, a hard disk installed in hard disk drive, and signals. These computer program products are means for providing software to the computer system. The computer readable medium allows the computer system to read data, instructions, messages or message packets, and other computer readable information from the computer readable medium. The computer readable medium, for example, may include non-volatile memory, such as a floppy disk, ROM, flash memory, disk drive memory, a CD-ROM, and other permanent storage. It is useful, for example, for transporting information, such as data and computer instructions, between computer systems. Computer program instructions may be stored in a computer readable medium that can direct a computer, other programmable data processing apparatus, or other devices to function in a particular manner, such that the instructions stored in the computer readable medium produce an article of manufacture including instructions which implement the function/act specified in the flowchart and/or block diagram block or blocks.

As will be appreciated by one skilled in the art, aspects of the embodiments may be embodied as a system, method or computer program product. Accordingly, aspects of the embodiments may take the form of an entirely hardware embodiment, an entirely software embodiment (including firmware, resident software, micro-code, etc.) or an embodiment combining software and hardware aspects that may all generally be referred to herein as a "circuit," "module" or "system. " Furthermore, aspects of the embodiments may take the form of a computer program product embodied in one or more computer readable medium(s) having computer readable program code embodied thereon.

In one embodiment, a non-transitory computer readable storage medium including instructions to perform a method for face liveness detection, the method comprising: receiving a first image comprising a portion corresponding to a face of a user; determining one or more two-dimensional (2D) facial landmark points based on the first image; determining a three-dimensional (3D) pose of the face in the first image based on the one or more determined 2D facial landmark points and one or more corresponding 3D facial landmark points in a 3D face model for the user; determining a homography mapping between the one or more determined 2D facial landmark points and one or more corresponding 3D facial landmark points that are perspectively projected based on the 3D pose; and determining liveness of the face in the first image based on the homography mapping.

Computer program code for carrying out operations for aspects of one or more embodiments may be written in any combination of one or more programming languages, including an object oriented programming language such as Java, Smalltalk, C++ or the like and conventional procedural programming languages, such as the "C" programming language or similar programming languages.

Aspects of one or more embodiments are described above with reference to flowchart illustrations and/or block diagrams of methods, apparatus (systems) and computer program products. These computer program instructions may be provided to a special purpose computer, or other programmable data processing apparatus to produce a machine, such that the instructions, which execute via the processor of the computer or other programmable data processing apparatus, create means for implementing the functions/acts specified in the flowchart and/or block diagram block or blocks.

References in the claims to an element in the singular is not intended to mean "one and only" unless explicitly so stated, but rather "one or more. " All structural and functional equivalents to the elements of the above-described exemplary embodiment that are currently known or later come to be known to those of ordinary skill in the art are intended to be encompassed by the present claims.

Claim 1:
A method for face liveness detection, comprising:
receiving (<NUM>) a first image comprising a portion corresponding to a face of a user;
detecting (<NUM>) two-dimensional, 2D, image information related to the face of the user based on the first image, wherein the 2D image information includes one or more 2D facial landmark points determined based on the first image;
obtaining (<NUM>) three-dimensional, 3D, shape information of the face of the user based on a 3D face model for the user, wherein the 3D shape information includes one or more corresponding 3D facial landmark points that are projected based on a 3D pose in the 3D face model for the user, wherein the 3D pose of the face in the first image is determined based on the one or more determined 2D facial landmark points and the one or more 3D facial landmark points in the 3D face model for the user;
performing (<NUM>) a homography mapping by determining correspondences between the one or more 2D facial landmark points and the one or more corresponding 3D facial landmark points that are perspectively projected based on the 3D pose; and
identifying (<NUM>) liveness of the face in the first image based on the homography mapping by determining a deviation of the homography mapping.