Method and apparatus to recognize user

Disclosed is a user recognition apparatus and method, the apparatus configured to determine a validity of an iris area for each frame, to perform an iris recognition when the iris area is valid, and to extract a facial feature for a fusion recognition when the iris area is invalid.

CROSS-REFERENCE TO RELATED APPLICATION(S)

BACKGROUND

The following description relates to technology for recognizing a user.

2. Description of Related Art

With developments in various mobile devices such as a smartphone and wearable devices, the importance of secured authentication has increased. In biometrics authentication technology, a user may be authenticated based on a biological feature, such as, for example, a DNA fingerprint, a fingerprint, an iris, voice, a face, and a blood vessel. The biometrics authentication technology may use a unique biometric property of each user. The biometric property is robust against forgery or falsification, does not cause any inconvenience in carrying the authentication code, and change very little over a lifetime.

SUMMARY

In one general aspect, there is provided a method to recognize a user including extracting facial feature data corresponding to a portion of a face from a frame image, in response to quality information of an iris area extracted from the frame image being invalid, and performing an iris recognition based on the iris area, in response to the quality information being valid.

The performing of the iris recognition may include determining availability of facial feature data extracted from a previous frame image, in response to the iris area not matching registered iris information, and performing a user recognition based on the iris area and the facial feature data, in response to facial feature data extracted from the previous frame image being available.

The performing of the iris recognition may include waiting for a subsequent frame image is to be acquired, in response to facial feature data extracted from the previous frame image not being available.

The performing of the user recognition may include determining that an object included in the current frame image matches a registered user, in response to an iris score being greater than a first threshold score set for a registered iris of the registered user, the iris score being calculated based on the iris area, and a facial feature score being greater than a second threshold score set for a registered facial feature of the registered user, the facial feature score calculated based on the facial feature data.

The performing of the user recognition may include determining that an object included in the current frame image matches a registered user in response to a combined score being greater than a combined threshold score, the combined score being calculated based on the iris area and the facial feature data.

The determining that the object matches the registered user may include calculating the combined score by applying an iris weight to an iris score based on the iris area and applying a feature weight to a facial feature score based on the facial feature data.

The performing of the iris recognition may include recognizing the face as a face of a registered user corresponding to registered iris information, in response to the iris area matching the registered iris information.

The performing of the iris recognition may include performing a user recognition based on the iris area and cumulative facial feature data up to facial feature data of a frame previous to the frame, in response to the iris area not matching registered iris information.

The extracting of the facial feature data may include accumulating facial feature data of different parts, and storing the accumulated facial feature data.

The extracting of the facial feature data may include extracting, from the frame image, facial feature data of a part different from a part of which facial feature data is extracted from a previous frame.

The extracting of the facial feature data may include determining a target part from the frame image based on an extraction order; and extracting the facial feature data of the target part.

The extracting of the facial feature data may include selecting a facial area corresponding to a landmark set from a plurality of facial areas; and extracting facial feature data of a part corresponding to the selected facial area.

The extracting of the facial feature data may include selecting a facial area corresponding to an area having highest quality information, in response to an overall face area or portions of the overall face area being identified from the frame image; and extracting facial feature data of a part corresponding to the selected facial area.

The method may include extracting the facial feature data from a subsequent frame and performing the iris recognition, in response to the quality information of the iris area extracted from the frame image being less than a minimum threshold.

The extracting of the facial feature data may include extracting the facial feature data, in response to the quality information of the iris area extracted from the frame image being greater than or equal to a minimum threshold and less than a valid threshold.

The may include terminating the user recognition in response to a frame image matching a registered user being absent, in response to a number of frames are processed.

The method may include calculating the quality information of the iris area based on any one or any combination of an exposure amount of the iris area and a shake amount of the iris area; and determining whether the quality information of the iris area is valid.

The method may include projecting an infrared light to an object and acquiring the frame image based on an infrared light reflected from the object.

In another general aspect, there is provided an apparatus to recognize a user including an image acquirer configured to acquire frame images, and a processor configured to extract facial feature data corresponding to a portion of a face from a frame image among the plurality of frame images, in response to quality information of an iris area extracted from the frame image being invalid, and to perform an iris recognition based on the iris area, in response to the quality information being valid.

DETAILED DESCRIPTION

Various alterations and modifications may be made to the examples. Here, the examples are not construed as limited to the disclosure and should be understood to include all changes, equivalents, and replacements within the idea and the technical scope of the disclosure.

The terminology used herein is for the purpose of describing particular examples only, and is not to be used to limit the disclosure. As used herein, the terms “a,” “an,” and “the” are intended to include the plural forms as well, unless the context clearly indicates otherwise.

FIG. 1illustrates an example of a user recognition.

A user recognition apparatus110recognizes an object190based on various features representing the object190.

In an example, the user recognition apparatus110acquires an input image including the object190by capturing an external area120of the user recognition apparatus110. The input image may include a plurality of frame images and in an example, the input image is an infrared intensity image representing an intensity to which an infrared light projected to the object190is reflected. The user recognition apparatus110projects an infrared light to the object190and acquires a current frame image based on an infrared light reflected from the object190.

When the object190is a person, the user recognition apparatus110extracts a feature associated with an iris of the person and a feature associated with a face of the person from the input image. The user recognition apparatus110recognizes the object190in the input image by integrally using the feature associated with the iris and the feature associated with the face. The user recognition apparatus110extracts an area130of an eye and performs iris recognition.

Although the present disclosure describes the object190as a person, embodiments are not limited thereto. Depending on examples, the object190may be an animal and features associated with a head of an animal and an iris of the animal may also be integrally used.

To perform the user recognition by integrally using information on a face and information on an iris, the user recognition apparatus110efficiently schedules a facial recognition task and an iris recognition task. The user recognition apparatus110selects an appropriate recognition scheme based on a status of an input frame image, thereby performing the recognition with increased speed and accuracy.

In an example, the user recognition apparatus110is applied to, for example, an intelligent agent, a mobile phone, a cellular phone, a smartphone, a personal computer (PC), a laptop, a notebook, a subnotebook, a netbook, or an ultra-mobile PC (UMPC), a tablet personal computer (tablet), a phablet, a mobile internet device (MID), a personal digital assistant (PDA), an enterprise digital assistant (EDA), a digital camera, a digital video camera, a portable game console, an MP3 player, a portable/personal multimedia player (PMP), a handheld e-book, an ultra mobile personal computer (UMPC), a portable lab-top PC, a global positioning system (GPS) navigation, a personal navigation device or portable navigation device (PND), a handheld game console, an e-book, and devices such as a television (TV), a high definition television (HDTV), an optical disc player, a DVD player, a Blue-ray player, a setup box, robot cleaners, a home appliance, a smart appliance, content players, communication systems, image processing systems, graphics processing systems, various other Internet of Things (IoT) devices that are controlled through a network, or any other device capable of wireless communication or network communication consistent with that disclosed herein.

The user recognition apparatus110may also be implemented as a wearable device, which is worn on a body of a user. In one example, a wearable device may be self-mountable on the body of the user, such as, for example, a ring, a watch, a pair of glasses, glasses-type device, a bracelet, an ankle bracket, a belt, a band, an anklet, a belt necklace, an earring, a headband, a helmet, a device embedded in the cloths, or as an eye glass display (EGD), which includes one-eyed glass or two-eyed glasses. The user recognition apparatus110may be applied to, for example, perform an iris-based user recognition in the smartphone and a mobile device, perform iris-based payment and banking in the smartphone and the mobile device, and perform a user recognition in a smart door, the TV, and a smart home.

FIGS. 2 through 4illustrate examples of a user recognition method. The operations inFIGS. 2-4may be performed in the sequence and manner as shown, although the order of some operations may be changed or some of the operations omitted without departing from the spirit and scope of the illustrative examples described. Many of the operations shown inFIGS. 2-4may be performed in parallel or concurrently. One or more blocks ofFIGS. 2-4, and combinations of the blocks, can be implemented by special purpose hardware-based computer that perform the specified functions, or combinations of special purpose hardware and computer instructions. In addition to the description ofFIGS. 2-4below, the descriptions ofFIG. 1is also applicable toFIGS. 2-4, and are incorporated herein by reference. Thus, the above description may not be repeated here.

A user recognition apparatus acquires consecutive frame images as input images including an object. In an example, the user recognition apparatus determines whether to extract facial feature data of a face or to perform an iris recognition for each of the frame images while acquiring the frame images.

Referring toFIG. 2, when a current frame image is loaded, the user recognition apparatus extracts facial feature data corresponding to a face or performs the iris recognition based on whether quality information of the current frame image is valid.

Hereinafter, feature data may indicate abstracted information of a feature of an object, such as, for example, a feature of a face of a person, a feature of each part of the face, and a feature of an iris of the person. The quality information may be information indicating a quality of a whole image or quality of a portion of the whole image and include, for example, an exposure amount of image, a shake amount of image, a degree of blur in an image, and an image quality. In an example, the shake amount of image is calculated based on, for example, the degree of blur in an image.

In operation210, the user recognition apparatus extracts facial feature data corresponding to at least a portion of a face from a current frame image and stores the extracted facial feature data when quality information of an iris area extracted from the current frame image is invalid. When the quality is invalid, iris recognition may be difficult or the accuracy of recognition may be reduced. Thus, the user recognition apparatus extracts the facial feature data of the face to be used for a subsequent frame, instead of performing the iris recognition.

In operation220, the user recognition apparatus performs an iris recognition based on the iris area when the quality information is valid. When the quality information is valid, an accuracy of iris recognition may be guaranteed, and thus, the user recognition apparatus performs the iris recognition. The iris recognition may be, for example, an operation of identifying an input iris matching a registered iris or verifying whether an input iris matches a registered iris.

Hereinafter, a verification may be an operation of determining input data is true or false, and an identification may be an operation of determining a label indicated by input data among a plurality of labels.

The user recognition apparatus determines that the recognition is successful or additionally performs a fusion recognition based on a result of the iris recognition. The fusion recognition may be a recognition performed based on a plurality of attributes of an object such as, for example, an operation of identifying an object matching a registered user or verifying whether an object matches a registered user based on an iris and iris feature data of a corresponding object represented in an input image.

FIG. 3illustrates an example of a user recognition. In operation301, the user recognition apparatus acquires a frame image. A user recognition apparatus consecutively receives a plurality of frame images and acquires a current frame image. The user recognition apparatus acquires an ithframe image in operation301, and performs a processing on the ithframe image in operations302,303,304,311,312,321,322,331,332, and333. Also, the user recognition apparatus performs a processing on an (i+1)thimage simultaneously while performing the processing on the ithframe image in operations302,303,304,311,312,321,322,331,332, and333. “ ” being an integer greater than or equal to 1. The ithframe image is also referred to as the current frame image.

In operation302, the user recognition apparatus detects an iris area. The user recognition apparatus extracts the iris area from the current frame image.

In operation303, the user recognition apparatus checks quality information. The user recognition apparatus calculates the quality information based on at least one of, for example, an exposure amount of the iris area or a shake amount of the iris area.

In operation304, the user recognition apparatus determines whether the quality information of the iris area is valid. Additional description of the determination of whether the quality information is valid will also be described with reference toFIG. 4.

In operation210, the user recognition apparatus performs facial recognition when the quality information is not sufficiently valid. In operation311, the user recognition apparatus extracts a facial feature. The user recognition apparatus extracts facial feature data corresponding to at least a portion of the face from the current frame image. In operation312, the user recognition apparatus stores the facial feature. The user recognition apparatus stores the extracted facial feature data while a number of frames are processed. In an example, the number of frames that are processed is a frame length set for a recognition to be performed. When a corresponding frame length ends, the user recognition apparatus determines the recognition as failed.

The user recognition apparatus compares the facial feature data extracted from the current frame image to registered facial feature data corresponding to a registered user, calculates a facial feature score, and stores the facial feature score. In an example, the facial feature score is also used for a subsequent fusion recognition.

In operation332, the user recognition apparatus waits for a subsequent frame after the face recognition is performed in operation210. The user recognition apparatus waits until a subsequent frame image is acquired.

In operation220, the user recognition apparatus performs an iris recognition when the quality information is valid.

In operation321, the user recognition apparatus performs an iris recognition. The user recognition apparatus calculates an iris score of the iris area.

In operation322, the user recognition apparatus determines whether the iris score is greater than or equal to an iris threshold score. In an example, the user recognition apparatus determines that the iris area matches registered iris information when the iris score is greater than or equal to an iris threshold set for a registered iris of the registered user. The registered iris information may be information associated with the registered iris of the registered user and include, for example, iris feature data and an image of the registered iris.

The user recognition apparatus performs the user recognition in operation331when the iris score is less than the iris threshold score. In operation331, the user recognition apparatus determines whether a face extraction result of the previous frame is present and a fusion recognition satisfies a recognition reference. The recognition reference may be set individually for the iris area and the facial feature data or set based on the iris area and the facial feature data. In operation333, when the iris area matches the registered iris information, the user recognition apparatus recognizes the face as a face of a registered user corresponding to registered iris information. In an example, the registered iris information is information on an iris previously registered in a database included in the user recognition apparatus. In an example, a previous frame includes a frame immediately previous to a current frame and a frame previous to the immediately previous frame. For example, a previous frame of an ithframe may be at least one of a first frame through an (i−1)thframe, i being an integer greater than or equal to 1.

When the iris area does not match the registered iris information, in operation331, the user recognition apparatus determines whether facial feature data extracted from a previous frame is present. When facial feature data extracted from a previous frame is present, the user recognition apparatus performs the user recognition based on the iris area and the facial feature data extracted from the previous frame. A example of determining whether the fusion recognition satisfies the recognition reference will be described below.

To perform the user recognition, the user recognition apparatus calculates the iris score based on the iris area and calculates the facial feature score based on facial feature data stored for the previous frame. The user recognition apparatus calculates a similarity between an input iris area and the registered iris, for example, using a Euclidean distance as the iris score, and calculates a similarity between input facial feature data and registered facial feature data as the facial feature score. When the iris score is greater than a first threshold score set for the registered iris of the registered user, and when the facial feature score is greater than a second threshold score set for the registered facial feature of the registered user, the user recognition apparatus determines that the object included in the current frame image matches the registered user. In operation333, in an example, when the iris score and the face feature score exceed individually set thresholds, the user recognition apparatus determines that the recognition is successful.

In an example, the user recognition apparatus calculates the facial feature score using the facial feature data stored for up to the previous frame without a need to extract facial feature data of a face from the current frame image. Thus, the user recognition apparatus improves the efficiency of performing user recognition.

In an example, the user recognition apparatus calculates a combined score based on the iris area and the facial feature data. In an example, the user recognition apparatus applies an iris weight to the iris score based on the iris area and applies a feature weight to the facial feature score based on the facial feature data, thereby calculating the combined score. The user recognition apparatus determines that the object included in the current frame image matches the registered user when the combined score is greater than a combined threshold score. In operation333, When the combined score obtained by integrally calculating the iris area and the facial feature data is greater than the combined threshold score, the user recognition apparatus determines that the user recognition is successful.

In an example, when facial feature data extracted from the previous frame is absent, in operation332, the user recognition apparatus waits until a subsequent frame image is acquired. In an example, when the fusion recognition does not satisfy the recognition reference, the user recognition apparatus waits for the subsequent frame image in operation332.

The iris threshold score may be a threshold set to determine whether the iris recognition is successful. The first threshold score, the second threshold score, and the combined threshold score may be thresholds set to determine whether the fusion recognition is successful. Thus, values of the iris threshold score, the first threshold score, the second threshold score, and the combined threshold score may be set based on a design.

FIG. 4is a diagram illustrating an example of a procedure of determining whether quality information is valid in operation304ofFIG. 3.

In operation410, the user recognition apparatus determines whether quality information is greater than or equal to minimum threshold. When quality information of an iris area extracted from a current frame image is less than a minimum threshold, in332, the user recognition apparatus waits for a subsequent frame instead of extracting facial feature data and performing an iris recognition.

In operation420, the user recognition apparatus determines whether the quality information is greater than or equal to a valid threshold. The valid threshold is set as, for example, a value higher than the minimum threshold. When the quality information of an iris area extracted from the current frame image is greater than or equal to the minimum threshold and less than the valid threshold, the user recognition apparatus extracts facial feature data in operation311.

When the quality information of the iris area is greater than or equal to the valid threshold, the user recognition apparatus performs the iris recognition based on the iris area in operation220.

The minimum threshold and the valid threshold may be thresholds indicating whether quality information of an iris area is valid and thus, determined based on a design.

FIGS. 5 through 7illustrate examples of performing an iris recognition, a face recognition, and a fusion recognition.

Referring toFIG. 5, a user recognition apparatus performs a frame loading510, a quality check520, an iris recognition530, a face recognition540, and a fusion recognition550for each frame.

A user recognition apparatus loads an ithframe image511, for example, a current frame image. When the loading511of the ithframe image is completed, the user recognition apparatus performs a quality information check520on an iris area of the ithframe image. When it is determined that the quality information is valid, the user recognition apparatus performs iris recognition531. When the iris recognition531is successful, the user recognition apparatus determines that a user recognition is successful. In an example, the user recognition apparatus loads an (i+1)thframe image512while performing the quality information check521and the iris recognition531.

An example in which quality information of an ithframe is invalid is described with reference toFIG. 6. When a loading611of an ithframe is completed, the user recognition apparatus performs a determination621on quality information of an ithframe image. When it is determined that the quality information of the ithframe image is invalid, the user recognition apparatus performs a face recognition641. The user recognition apparatus extracts facial feature data corresponding to a facial feature of a face and stores the extracted facial feature data. When a loading612of an (i+1)thframe image is completed, and when a result of a determination622indicates that quality information of the (i+1)thframe image is valid, the user recognition apparatus performs an iris recognition632. When the iris recognition632fails, the user recognition apparatus performs a fusion recognition652. As discussed above, in an example, the fusion recognition652is a recognition based on the iris recognition632performed on the (i+1)thframe image and the facial feature data extracted through the face recognition641performed on the ithframe image.

Referring toFIG. 7, a user recognition apparatus performs a user recognition for each frame until a recognition is successful.

The user recognition apparatus performs an extraction741of facial feature data of a face in an ithframe image when quality information of the ithframe image is invalid. An iris recognition and a fusion recognition752may fail even when quality information of an (i+1)thframe image is valid. Thereafter, when quality information of an (i+2)thframe image is invalid, the user recognition apparatus performs an extraction743of facial feature data of a face in the (i+2)thframe image. When an iris area does not match registered iris information, the user recognition apparatus performs a user recognition in operation759based on the iris area and cumulative facial feature data up to facial feature data of a frame previous to a current frame.

FIG. 8illustrates an example of selecting a part to be a target of a face recognition.

When facial feature data is extracted with respect to an overall face or a combination of different parts of a face, a user recognition apparatus accumulates the extracted facial feature data and stores the accumulated facial feature data. Referring toFIG. 8, the user recognition apparatus performs an extraction841of facial feature data of an eye part891in an ithframe image and stores the facial feature data. The user recognition apparatus performs an extraction842of facial feature data of an nose part892in an (i+1)thframe image and stores the facial feature data. The user recognition apparatus performs an extraction843of facial feature data of a mouth part893in an (i+2)thframe image and stores the facial feature data. In an example, the user recognition apparatus extracts facial feature data associated with a combination of parts as well as a single part. For example, the user recognition apparatus extracts facial feature data of parts of an eye and a nose and stores the facial feature data.

As described above, the user recognition apparatus extracts, from a current frame image, facial feature data of a part different from a part of which facial feature data is extracted from a previous frame image. In an example, the user recognition apparatus repetitively uses at least one piece of facial feature data extracted from upto the previous frame to calculate a face feature score or a combined score.

The user recognition apparatus determines a target part for extracting facial feature data from the current frame image based on an extraction order. The user recognition apparatus extracts facial feature data of the determined target part. The extraction order may be an order set for each part of a face and may vary based on a design.

In an example, the user recognition apparatus uses a plurality of landmarks, for example, at least three landmarks. A set of landmarks to be applied to an overall face or portions of the face may be defined. The user recognition apparatus generates facial feature data and a facial image normalized based on the set of landmarks. The user recognition apparatus selects a facial area corresponding to the defined set of landmarks from a plurality of facial areas, and extracts facial feature data of a part corresponding to the selected facial area.

The user recognition apparatus preferentially selects facial feature data having a relatively high quality from facial feature data of the overall face or the portions of the face extracted in the previous frame. The landmark is a part having a feature representing a face, for example, an eye, a nose, a mouth, and an ear.

The user recognition apparatus selects facial feature data of the overall face or portions of the face corresponding to an area having the highest quality information when extracting the facial feature data using a plurality of landmarks in the current frame image. When an area of the overall face or a plurality of areas corresponding to the portions of the face is identified, the user recognition apparatus selects a facial area corresponding to the highest quality information from a plurality of facial areas. The user recognition apparatus calculates quality information of the overall face or the portions of the face from the plurality of landmarks and selects an area having the highest quality information. The user recognition apparatus extracts facial feature data of a part corresponding to the selected facial area.

The user recognition apparatus terminates an operation of the user recognition in response to a frame image determined as an image matching a registered user being absent after a number of frames are processed. The user recognition apparatus performs a validity check849on quality information when a loading of an Nthframe is completed, N being an integer greater than or equal to 1. When the quality information is invalid, the user recognition apparatus terminates an operation of recognition. Also, even when the quality information associated with an Nthframe image is valid, the user recognition apparatus terminates the operation of recognition in response to a failure of the iris recognition.

FIG. 9illustrates an example of a user recognition apparatus. A user recognition apparatus900includes an image acquirer910and a processor920.

The image acquirer910acquires a plurality of frame images.

The processor920extracts face feature data corresponding to at least a portion of a face from a current frame image in response to quality information of an iris area extracted from the current frame image among the plurality of frame images being invalid, and performs an iris recognition based on the iris area in response to the quality information being valid.

Operations of the image acquirer910and the process920are not limited to the aforementioned example, and the image acquirer910and the process920may also perform operations in combination with the operations described with reference toFIGS. 1 through 8. In addition to the description of the image acquirer910and the process920, the descriptions ofFIG. 1-8is also applicable toFIG. 9, and are incorporated herein by reference.