Patent Description:
Since authentication through biometric information is convenient and easy to access, biometrics authentication has been introduced in various fields. Typically, biometric authentication includes the matching process of verifying whether a user attempting authentication has the authority to access predetermined information, and the anti-spoofing (ASP) process of determining whether biometric information is forged/spoofed.

Spoofing is performed by mimicking, falsifying, or duplicating biometric information of a user to attempt authentication. Thus, if spoofing is determined using only images, it is difficult to improve the accuracy of spoofing detection due to insufficient information. Therefore, the use of additional information in addition to the images may help to improve the accuracy. In this case, the accuracy may be improved through information of various dimensions when the additional information is used. However, if the accuracy of the additional information is low, the fusion between the images and the additional information may rather decrease the detection performance.

Document <CIT> (<NUM>-<NUM>-<NUM>) discloses a method for biometric spoof detection by fusing two probabilities computed on two biometric samples.

In a general aspect, a biometric information spoofing detection method according to claim <NUM> is provided.

The biometric information may include one of a fingerprint, an iris, and a face of the user.

In a general aspect, an apparatus that detects whether biometric information is spoofed, the apparatus includes the features of claim <NUM>.

Throughout the specification, when an element, such as a layer, region, or substrate is described as being "on," "connected to," or "coupled to" another element, it may be directly "on," "connected to," or "coupled to" the other element, or there may be one or more other elements intervening therebetween.

Unless otherwise defined, all terms, including technical and scientific terms, used herein have the same meaning as commonly understood by one of ordinary skill in the art.

Hereinafter, examples will be described in detail with reference to the accompanying drawings. When describing the examples with reference to the accompanying drawings, like reference numerals refer to like components and a repeated description related thereto will be omitted.

In what follows, the use of <<may>> is explicitly not meant to indicate something as being optional in relation to the technical features of the independent claims.

Similarly, all the features associated with instances of <<may>> are prima facie not considered optional when comprised in the dependent claims, unless explicitly indicated.

<FIG> illustrates an example method of detecting whether biometric information is spoofed, in accordance with one or more embodiments. Referring to <FIG>, an apparatus that detects whether biometric information is spoofed (hereinafter, referred to as the "detection apparatus") may detect whether biometric information is spoofed through the process of operations <NUM> to <NUM>. In an example, it is assumed that the operations described below are performed after the matching process of verifying whether a user attempting user authentication with biometric information in the biometrics authentication process has the authority to access a system, that is, is an enrolled user of the system. However, examples are not limited thereto.

In operation <NUM>, the detection apparatus obtains, from a sensor, first feature information including a static feature related to biometric information of a user and a dynamic feature obtained based on images related to the biometric information. The sensor may include, as non-limiting examples, any one or any combination of an ultrasonic fingerprint sensor, a depth sensor, and an image sensor. However, examples are not limited thereto. Among the sensors, any one sensor or two or more sensors may be used. The first feature information is information that may be obtained directly from the sensor, and may also be referred to as, for example, a "hand-crafted feature (HCF)". Hereinafter, the term "first feature information" and the term "HCF" may be used interchangeably. The static feature, the dynamic feature, and the process of obtaining the first feature information by the detection apparatus will be described in more detail with reference to <FIG>.

In operation <NUM>, the detection apparatus may detect whether the biometric information is spoofed based on a first score calculated based on the first feature information obtained in operation <NUM>. Here, "spoofed" biometric information refers to fake or false biometric information other than live biometric information, and may be construed as including, as examples, duplication, forgery, and falsification of biometric information.

In operation <NUM>, for example, the detection apparatus calculates the first score based on the first feature information, and determine whether the first score falls within a first threshold range for an early decision of whether the biometric information is spoofed. The first score may correspond to, for example, a similarity score calculated based on a result of comparing the first feature information to authentication information (for example, feature information of a fingerprint image) included in an enrollment database. However, examples are not limited thereto. The first score may be calculated by, for example, a trained classifier or deep neural network. However, examples are not limited thereto. In this example, the first threshold range may be construed as a criterion for clearly determining whether the first score falls within a range in which biometric information is determined as live information, or falls within a range in which biometric information is determined as spoof information. The first threshold range may be determined based on, for example, a first threshold corresponding to the maximum probability that the biometric information is determined as spoof information based on the first score and a second threshold corresponding to the minimum probability that the biometric information is determined as live information based on the first score in the probability distribution of the first score.

In an example, in response to the determination that the first score falls within the first threshold range, the detection apparatus detects whether the biometric information is spoofed based on the first score. The detection apparatus detects whether the biometric information is spoofed by classifying the biometric information corresponding to the first score as spoof information or live information using a neural network such as the trained classifier. However, examples are not limited thereto. In response to the determination that the first score falls within the first threshold range, operation <NUM> of fusing the first score with a second score, and operation <NUM> of detecting whether the biometric information is spoofed based on a fused score is not performed, and a determination that the biometric information is spoofed may be early detected based on the first score.

Accordingly, the process of detecting whether the biometric information is spoofed based on the first score may correspond to the "early decision process" <NUM> which will be described later with reference to <FIG>. The detection apparatus may quickly detect whether the biometric information is spoofed based on a relatively small amount of information such as the first score, through a small-sized neural network. An example of detecting whether the biometric information is spoofed based on the first score by the detection apparatus will be described in more detail with reference to <FIG>. Additionally, the first threshold range will be described in more detail with reference to <FIG>.

In operation <NUM>, the detection apparatus fuses the first score with a second score if a determination cannot be made that the biometric information is spoofed based on the first score in operation <NUM>. In operation <NUM>, the detection apparatus extracts second feature information from images. Since the second feature information is extracted from the images, the second feature information may also be referred to as an "image feature". The detection apparatus calculates the second score based on the second feature information. The detection apparatus determines whether to fuse the first score with the second score calculated based on the second feature information extracted from the images, and fuses the first score with the second score in response to determining to fuse the first score with the second score. In an example, the detection apparatus may determine whether to fuse the first score with the second score based on whether the first score falls within a second threshold range in which the confidence thereof is accepted. An example of fusing the first score with the second score by the detection apparatus will be described in more detail with reference to <FIG>.

In operation <NUM>, the detection apparatus detects whether the biometric information is spoofed based on a fused score obtained in operation <NUM>. The detection apparatus may detect whether the biometric information is spoofed through, for example, a multi-stage decision logic shown in <FIG>. The process of detecting whether the biometric information is spoofed based on the fused score by the detection apparatus will be described in more detail through the "score fusion process" shown in <FIG>.

The detection apparatus may, for example, output a result of detecting whether the biometric information is spoofed in operation <NUM>, through an output device (for example, <NUM> in <FIG>) such as a display and/or a speaker. Alternatively, the detection apparatus may match a result of detecting whether the biometric information is spoofed in operation <NUM> to the biometric information, and output a matching result to the outside of the detection apparatus.

For example, the detection apparatus may improve the anti-spoofing efficiency and accuracy in a mobile device having limited resources through the multi-stage decision logic which will be described with reference to <FIG>.

Hereinafter, the configuration of the detection apparatus will be described with reference to <FIG>.

<FIG> illustrates an example of an apparatus that detects whether biometric information is spoofed.

Referring to <FIG>, a detection apparatus <NUM> may improve the speed of detecting whether biometric information is spoofed, and may improve the accuracy of determining whether the biometric information is spoofed when a user requests authentication using the biometric information, and may perform, for example, the process <NUM> of generating features, and the process <NUM> of determining whether the biometric information is spoofed through multi stages. The process <NUM> of generating features may be construed as including extraction of the features. Additionally, the process <NUM> of determining whether the biometric information is spoofed may be construed as including detection of whether the biometric information is spoofed.

In the process <NUM>, the detection apparatus <NUM> may extract, for example, a static feature related to biometric information of a user and a dynamic feature based on images related to the biometric information. The static feature may include physical measures that are obtainable directly from a sensor <NUM>, such as, for example, oxygen saturation, impedance, face depth information, electrical resistance, temperature (body temperature), heart rate, and the like. However, examples are not limited thereto. The static feature may also be referred to as a "physical feature" in the sense that it includes a physical measure sensed by the sensor <NUM>. A different physical feature may be sensed by each sensor <NUM>. In an example, when the sensor <NUM> is an ultrasonic fingerprint sensor, the sensor <NUM> may obtain fingerprint image data and impedance information that is a physical feature of a fingerprint. Since impedance varies according to the characteristics of a medium, the impedance information varies according to the material of a forgery fingerprint. Thus, the impedance information may be utilized to determine a forgery fingerprint. Alternatively, when the sensor <NUM> is an image sensor, a difference between a plurality of pieces of image data obtained through the sensor <NUM> may be utilized as a feature to determine a forgery fingerprint. The image data obtained in the manner described above may be utilized as static features or dynamic features.

In an example, the sensor <NUM> may include a facial recognition sensor or an iris scan sensor, but is not limited thereto.

In an example, the sensor <NUM> may directly obtain images and output a dynamic feature calculated based on a difference between image features extracted respectively from the images. Alternatively, the sensor <NUM> may obtain images related to biometric information through photographing or capturing. In this example, the images obtained through the sensor <NUM> may be, for example, fingerprint images, eye (iris) images, and/or face images. The images obtained through the sensor <NUM> may be full images or partial images.

When the sensor <NUM> obtains the images, the detection apparatus <NUM> may respectively extract the image features from the images, and generate the dynamic feature based on the difference between the image features. In this example, the detection apparatus <NUM> may calculate the dynamic feature from the images by using various feature detection techniques used for image processing, not through a separate neural network. Since the dynamic feature obtained in the manner described above does not have to be extracted again through a separate neural network, the time for feature extraction may be reduced. The static feature and dynamic feature extracted in the process <NUM> may correspond to the first feature information described above, that is, an HCF <NUM>.

Additionally, in the process <NUM>, the detection apparatus <NUM> may obtain images <NUM>. The images <NUM> may be, for example, the images obtained through detection by the sensor <NUM>, or images that are separately obtained.

When the HCF <NUM> is generated in the process <NUM>, the detection apparatus <NUM> may detect whether the biometric information is spoofed by utilizing the HCF <NUM> in the process <NUM>. In this example, the detection apparatus <NUM> may determine whether the biometric information is spoofed through multiple stages (for example, two stages) by utilizing the HCF <NUM> in the process <NUM>.

In the first stage, the detection apparatus <NUM> may quickly determine whether the biometric information is spoofed with respect to the HCF <NUM> using small feature information and a small-sized network. In an example, the detection apparatus <NUM> may calculate an HCF score <NUM> based on the HCF <NUM> through an HCF deep neural network (DNN) <NUM> and detect whether the biometric information is spoofed based on the HCF score <NUM>, in operation <NUM>. The DNN may include a plurality of layers. For example, the deep neural network <NUM> may include an input layer to which input data is applied, an output layer for outputting a result derived through prediction based on training and the input data, and a plurality of hidden layers for performing a neural network operation between the input layer and the output layer.

Technological automation of pattern recognition or analyses, for example, has been implemented through processor implemented neural network models, as specialized computational architectures, that after substantial training may provide computationally intuitive mappings between input patterns and output patterns or pattern recognitions of input patterns. The trained capability of generating such mappings or performing such pattern recognitions may be referred to as a learning capability of the neural network. Such trained capabilities may also enable the specialized computational architecture to classify such an input pattern, or portion of the input pattern, as a member that belongs to one or more predetermined groups. Further, because of the specialized training, such specially trained neural network may thereby have a generalization capability of generating a relatively accurate or reliable output with respect to an input pattern that the neural network may not have been trained for, for example.

In such an example, the DNN <NUM> may be one or more of a fully connected network, a convolution neural network, a recurrent neural network, and the like, or may include different or overlapping neural network portions respectively with such full, convolutional, or recurrent connections, according to an algorithm used to process information. The DNN <NUM> may be configured to perform, as non-limiting examples, object classification, object recognition, voice recognition, and image recognition by mutually mapping input data and output data in a nonlinear relationship based on deep learning. Such deep learning is indicative of processor implemented machine learning schemes for solving issues, such as issues related to automated image or speech recognition from a data set, as non-limiting examples. Herein, it is noted that use of the term 'may' with respect to an example or embodiment, e.g., as to what an example or embodiment may include or implement, means that at least one example or embodiment exists where such a feature is included or implemented while all examples and embodiments are not limited thereto.

The detection apparatus <NUM> may determine whether the biometric information is live information or spoof information based on the HCF score <NUM>. The HCF score <NUM> corresponds to the first score described above. Thus, the term "HCF score" and the term "first score" may be used interchangeably. The HCF <NUM> has a relatively small amount of information compared to the images <NUM>. Therefore, the detection apparatus <NUM> may detect whether the biometric information is spoofed only when whether the biometric information is live or spoof information is clearly determined based on the HCF score <NUM> in the first stage. As such, the process of immediately detecting whether the biometric information is spoofed based on the HCF score <NUM> may be referred to as the "early decision" process.

In an example, if it is not possible to clearly determine whether the biometric information is spoofed based on the HCF score <NUM> in the first stage, the detection apparatus <NUM> may defer a determination on whether the biometric information is spoofed, and transmit the HCF score <NUM> to the following process of detecting whether the biometric information is spoofed by fusing scores. The HCF data <NUM> corresponding to an HCF score <NUM>, for which whether the biometric information is spoofed is not determined in the early decision process may be utilized in the following process of detecting whether the biometric information is spoofed based on the images <NUM>. The HCF <NUM> contains information of a different dimension from the images <NUM> and thus, may have a complementary relationship with the images <NUM>. Accordingly, the detection apparatus <NUM> may improve the speed of detecting whether the biometric information is spoofed and the accuracy of spoofing detection through the mutual complementation between the HCF <NUM> and the images <NUM>. In this example, the images <NUM> may be, for example, multiple pieces of image data obtained by the image sensor in the process <NUM>, or separate test images. The test images may be, as an example, fingerprint images. The process <NUM> of detecting whether the biometric information is spoofed through the mutual complementation between the HCF <NUM> and the images <NUM>, more specifically, between the HCF score <NUM> and an image score <NUM>, may be referred to as the "multi-modality score fusion" or "score fusion" process.

In the process <NUM>, the detection apparatus <NUM> may extract image features from the images <NUM> by implementation of, as an example, an image DNN <NUM>, and calculate the image score <NUM> based on the image features. The image features may correspond to the second feature information described above. The image score <NUM> corresponds to the second score described above. Thus, the term "image score" and the term "second score" may be used interchangeably. The detection apparatus <NUM> may calculate a fused score by fusing the HCF score <NUM> with the image score <NUM>, in operation <NUM>. In this example, to improve the performance, the detection apparatus <NUM> may exclude an HCF score <NUM> the confidence of which is lower than a predetermined criterion from among HCF scores <NUM> received from the early decision process, and thereby perform score fusion using an HCF score <NUM> the confidence of which is higher than the predetermined criterion. In this example, the predetermined criterion for determining the confidence of the HCF score <NUM> may be, for example, a second threshold range. In operation <NUM>, the detection apparatus <NUM> may detect whether the biometric information is live or spoofed based on the fused score calculated through fusion <NUM>. A multi-stage decision logic in the process <NUM> will be described in more detail with reference to <FIG>.

<FIG> illustrates an example of obtaining first feature information. Referring to <FIG>, the configuration of first feature information is illustrated. The first feature information corresponds to the HCF <NUM> of <FIG> and <FIG>.

A detection apparatus may obtain physical features <NUM> and images <NUM> from a sensor (for example, the sensor <NUM> of <FIG>). In an example, the sensor may include a physical sensor that senses the physical features <NUM> related to biometric information of a user, and a biometric sensor (as non-limiting examples, an ultrasonic fingerprint sensor, an iris scanner, or a facial recognition sensor) that captures the images <NUM>. The images <NUM> may be full images (for example, full fingerprint images, full facial images, or full iris images) or partial images (for example, partial fingerprint images, partial facial images, or partial iris images). In an example, the images <NUM> may correspond to the images <NUM> of <FIG>.

The detection apparatus may obtain a static feature <NUM> from the physical features <NUM>. The static feature <NUM> may be a physical feature <NUM> obtainable by the sensor, such as oxygen saturation, impedance, temperature, and the like. Further, the detection apparatus may obtain a dynamic feature <NUM> based on a difference or variation between image features <NUM> extracted from the images <NUM>.

The detection apparatus may generate the HCF <NUM> by combining the static feature <NUM> and the dynamic feature <NUM>. The static feature <NUM> and the dynamic feature <NUM> may serve as elements that complement each other. In other words, combining the static feature <NUM> and the dynamic feature <NUM> may improve the accuracy of the first feature information (for example, the HCF <NUM>).

For example, the detection apparatus may generate the HCF <NUM> by making the dimensions of the static feature <NUM> the same as the dimensions of the dynamic feature <NUM> and then combining the static feature <NUM> and the dynamic feature <NUM>. Here, "combining" may correspond to, for example, concatenating or calculating a weighted sum.

In an example, the detection apparatus may generate the HCF <NUM> by concatenating or adding up an embedding vector corresponding to the static feature <NUM> and an embedding vector corresponding to the dynamic feature <NUM>. In an example, if the embedding vector corresponding to the static feature <NUM> is <NUM>-dimensional and the embedding vector corresponding to the dynamic feature <NUM> is <NUM>-dimensional, the HCF <NUM> may be <NUM>-dimensional information.

As described above with reference to <FIG>, the HCF <NUM> may be utilized for an early decision detection determination. A neural network, (for example, the HCF DNN <NUM> of <FIG>) that determines whether biometric information is spoofed based on the HCF <NUM>, may only have to process a relatively small amount of information when compared to a neural network (for example, the image DNN <NUM> of <FIG>) that determines whether biometric information is spoofed based on the images <NUM>, and thus, may need less time to determine whether biometric information is spoofed. Additionally, the HCF <NUM> is generated by combining the static feature <NUM> and the dynamic feature <NUM>. Since the static feature <NUM> corresponds to information that is obtainable directly from a sensor, an additional time for feature extraction may not be needed. The feature extraction schemes used to extract the dynamic feature <NUM> may only have to process a relatively small amount of information when compared to the neural network (for example, the image DNN <NUM> of <FIG>) that determines whether biometric information is spoofed based on the images <NUM>.

An example of the process of obtaining the dynamic feature <NUM> by the detection apparatus is as follows.

The detection apparatus may extract the image features <NUM> respectively from the images <NUM>. The detection apparatus may extract the image features <NUM> using feature extraction schemes used for image processing. Here, unlike the image features extracted by the image DNN <NUM> of <FIG>, the image features <NUM> may be extracted in a short time using the following various feature extraction schemes, rather than a deep neural network. The image features <NUM> may correspond to hand-craft image features shown in <FIG>. The feature extraction schemes may include, for example, a scheme of extracting a feature vector by normalizing a signal-to-noise ratio (SNR), a scheme of extracting a local binary pattern (LBP) feature, a scheme of extracting a skew, a scheme of measuring an image quality, a scheme of extracting a histogram of oriented gradient (HoG) feature, and a scheme of extracting a blob representing a set of image pixels concatenated to each other.

The detection apparatus may obtain the images <NUM> from the sensor. The biometric sensor may generate the images <NUM> by capturing biometric information of the user with a time difference therebetween. At this time, the images <NUM> may be, for example, an image (for example, a first image) obtained in the matching process of verifying whether the user attempting authentication has the authority to access, and an image (for example, an Nth image) obtained in the anti-spoofing (ASP) process of detecting whether biometric information is spoofed with a time difference thereafter. In this case, N may be <NUM>.

Alternatively, the biometric sensor may generate the images <NUM> using an image generated by capturing the biometric information of the user and an image obtained by converting the image through image processing. For example, the biometric sensor may generate a second image by performing preprocessing (for example, noise removal or sharpness enhancement) on the first image obtained in the matching process of verifying whether the user attempting authentication has the authority to access predefined or predetermined information.

<FIG> illustrates an example of detecting whether biometric information is spoofed by a first score, in accordance with one or more embodiments.

Referring to <FIG>, the process of detecting whether biometric information is spoofed by a classifier <NUM> based on the HCF <NUM> obtained from the sensor <NUM> is illustrated.

A detection apparatus may detect whether biometric information is spoofed by applying the HCF <NUM>, for example, to the classifier <NUM>, wherein the HCF <NUM> is obtained based on the physical features <NUM> of biometric information (for example, a fingerprint) being a detection target and the image features <NUM>. In this example, the image features <NUM> may be, for example, image features extracted from a single image, or dynamic features extracted based on differences between images.

In an example, the classifier <NUM> may calculate a first score based on the HCF <NUM> and determine whether the first score falls within a first threshold range. In response to the determination that the first score falls within the first threshold range, the classifier <NUM> may classify the biometric information corresponding to the first score as live information or spoof information. In this case, the classifier <NUM> may determine whether the biometric information is spoofed by performing a binary decision between Live or Spoof based on the first score. The first score may be calculated by a regression neural network instead of the classifier <NUM>. When the first score calculated based on the HCF <NUM> is out of the first threshold range, the detection apparatus may transmit the first score to the multi-modality score fusion process for more specific determination. The first threshold range will be described in detail with reference to <FIG>.

<FIG> illustrates an example of a first threshold range, in accordance with one or more embodiments.

Referring to <FIG>, a graph <NUM> showing a probability distribution of a first score corresponding to spoof information and a graph <NUM> showing a probability distribution of a first score corresponding to live information are illustrated.

In an example, it may be assumed that first feature information may be represented as in forms <NUM>, <NUM>, and <NUM>, and "<NUM>" in the form <NUM> corresponds to spoof information, and "<NUM>" in the form <NUM> corresponds to live information. If it is possible to clearly determine whether the first score corresponding to the biometric information falls within the range in which biometric information is determined as live information or falls within the range in which biometric information is determined as spoof information, as in the forms <NUM> and <NUM>, the detection apparatus may immediately detect whether the biometric information is spoofed based on the first score.

However, if it is difficult to clearly determine whether the information is "<NUM>" or "<NUM>", as in the form <NUM>, in other words, if it is impossible to clearly determine whether the first score corresponding to the information falls within the range in which biometric information is determined as live information or falls within the range in which biometric information is determined as spoof information, the detection apparatus may not immediately determine whether the biometric information is spoofed based on the first score.

In an example, the first threshold range may correspond to a probability range in which it is possible to clearly determine whether biometric information is spoofed based on the first score. The first threshold range is for distinguishing a first score for which a determination is not made whether biometric information is spoofed ("Not Decided") as in the form <NUM>. If a determination that the biometric information is spoofed is not immediately made based on the first score, the detection apparatus may transmit the first score to the following multi-modality score fusion process.

In <FIG>, a graph <NUM> shows a probability distribution of a first score for which the biometric information is determined as spoof information, and a graph <NUM> shows a probability distribution of a first score for which the biometric information is determined as live information. In this example, a reject threshold <NUM> in the graph <NUM> may correspond to a maximum value (Max(ScoreSpoof)) of the probability that the first score is determined to fall within the range in which biometric information is clearly determined as spoof information. In addition, an accept threshold <NUM> in the graph <NUM> may correspond to a minimum value (Min(ScoreLive)) of the probability that the first score is determined to fall within the range in which biometric information is clearly determined as live information.

The first threshold range may be determined based on the reject threshold <NUM> corresponding to the maximum probability (Max(ScoreSpoof)) that the first score is determined to fall within the range in which biometric information is determined as spoof information and the accept threshold <NUM> corresponding to the minimum probability (Min(ScoreLive)) that the first score is determined to fall within the range in which biometric information is determined as live information. The first threshold range may correspond to a section <NUM> that is greater than the reject threshold <NUM> in the graph <NUM> and less than the accept threshold <NUM> in the graph <NUM>. In the graph <NUM>, if the first score falls within a section <NUM> that is less than or equal to the reject threshold <NUM>, the first score may be determined to fall within the range in which biometric information is clearly determined as spoof information. Further, in the graph <NUM>, if the first score falls within a section <NUM> that is greater than or equal to the accept threshold <NUM>, the first score may be determined to fall within the range in which biometric information is clearly determined as live information.

Accordingly, if the first score falls within the section <NUM> and the section <NUM>, the detection apparatus may determine that the first score falls within the first threshold range. Unlikely, if the first score falls within the section <NUM>, the detection apparatus may determine that the first score does not fall within the first threshold range.

<FIG> illustrates an example of fusing a first score with a second score, in accordance with one or more embodiments.

Referring to <FIG>, the score fusion process is shown.

A detection apparatus may calculate a fused final score <NUM> by fusing the HCF score <NUM> calculated by the HCF DNN <NUM> based on the HCF <NUM> with the image score <NUM> calculated by the image DNN <NUM> based on the images <NUM>.

As described above, since the HCF <NUM> contains information of a different dimension from the images <NUM>, when a determination is made that biometric information is spoofed using the HCF score <NUM> and the image score <NUM> together, the mutual complementation therebetween may improve the speed and accuracy of determining whether biometric information is spoofed.

When the detection apparatus fuses or combines the scores in operation <NUM>, if less accurate scores are fused, the accuracy of the fused final score <NUM> may be low. Therefore, the detection apparatus may compare HCF scores <NUM> with a predetermined criterion, thereby classifying the HCF scores <NUM> into a score <NUM> the confidence of which is lower than a predetermined criterion (hereinafter, the low confidence score <NUM>) and a score <NUM> the confidence of which is higher than the predetermined criterion (hereinafter, the high confidence score <NUM>). In this example, the predetermined criterion for determining the confidence of an HCF score <NUM> may be, for example, a second threshold range in which the confidence of the first score is accepted.

In operation <NUM>, the detection apparatus may exclude the low confidence score <NUM>, and fuse or combine the high confidence score <NUM> with the image score <NUM>. Through this, the detection apparatus may prevent performance degradation that may occur due to score fusion.

In this example, the predetermined criterion for determining whether the HCF score <NUM> has a high confidence may be a second threshold range. In an example, the detection apparatus may determine whether to fuse an HCF score <NUM> with the image Score <NUM>, for example, based on whether the HCF score <NUM> falls within the second threshold range in which the confidence thereof is accepted. The detection apparatus may fuse the HCF score <NUM> that falls within the second threshold range (for example, the high confidence score <NUM>) with the image score <NUM>, in operation <NUM>. Here, the second threshold range may be determined, for example, based on a third threshold corresponding to a point at which a false acceptance rate (FAR) and a false rejection rate (FRR) match in the probability distribution of the first score. The second threshold range will be described in detail with reference to <FIG>.

In an example, if HCF scores <NUM> are all out of the second threshold range, the detection apparatus may not perform fusion with the image score <NUM> and may detect whether the biometric information is spoofed based on the image score <NUM> only.

In an example, operation <NUM> may be performed by the fusion between an embedding vector generated before the HCF score <NUM> is calculated by the HCF DNN <NUM> and an embedding vector generated before the image score <NUM> is calculated by the image DNN <NUM>. In this example, the fusion of the embedding vectors may be performed by a separate neural network that performs fusion, other than the HCF DNN <NUM> and the Image DNN <NUM>. In the case of fusing the embedding vectors, the embedding vectors and information regarding whether the embedding vectors have high confidence or low confidence may be transmitted together to the separate neural network.

<FIG> illustrate examples of a second threshold range.

Referring to <FIG>, graph <NUM> illustrates respective graphs of distributions of scores for which biometric information is determined as spoof information, and distributions of scores for which biometric information is determined as live information. In graph <NUM>, a graph <NUM> illustrating a distribution of a first score for which biometric information is determined as spoof information, and a graph <NUM> illustrating a distribution of a first score for which biometric information is determined as live information, are illustrated.

The graph <NUM> may correspond to the probability distribution of the first score for which the biometric information is determined as spoof information, and the graph <NUM> may correspond to the probability distribution of the first score for which the biometric information is determined as live information. In this example, in a section <NUM> of the graph <NUM>, the first score may be determined to fall within the range in which biometric information is determined as spoof information with high confidence. Further, in a section <NUM> of the graph <NUM>, the first score may be determined to fall within the range in which biometric information is determined as live information with high confidence. The sections <NUM> and <NUM> may be sections in which the confidence is accepted, that is, sections that fall within a second threshold range.

On the other hand, in the middle section <NUM>, there may occur frequent false acceptance (FA) that falsely accepts biometric information as live information although the first score falls within the section in which biometric information is determined as spoof information, and frequent false rejection (FR) that falsely rejects biometric information as spoof information although the first score falls within the section in which biometric information is determined as live information. The section <NUM> may be a section that does not fall within the second threshold range, that is, a section that is out of the second threshold range. In this example, FA and FR may occur at a point at which the FAR and the FRR match in the graph <NUM> of <FIG>, that is, at a point of an equal error rate (EER), for example, <NUM>. The FAR and the FRR may be in inverse proportion. The second threshold range may be determined based on a point <NUM> at which the FAR is <NUM> in the graph <NUM> and a point <NUM> at which the FRR is <NUM> in the graph <NUM>. The second threshold range may be determined based on, for example, a third threshold (for example, <NUM>) corresponding to the point at which the FAR and the FRR match.

<FIG> illustrates an example of a structure of a multi-stage decision logic that detects whether biometric information is spoofed, in accordance with one or more embodiments.

Referring to <FIG>, the cascade score fusion process performed by a detection apparatus <NUM> is illustrated.

In response to the completion of feature extraction, the detection apparatus <NUM> may determine whether biometric information is spoofed through two stages, an early decision stage <NUM>, and a score fusion stage <NUM>.

In the early decision stage <NUM>, the detection apparatus <NUM> may calculate the HCF score <NUM> by applying the HCF <NUM> to the HCF DNN <NUM>, and determine a range within which the HCF score <NUM> falls, among live information <NUM>, not decided information <NUM>, and spoof information <NUM>. The HCF DNN <NUM> may be a neural network that calculates the HCF score <NUM> based on the HCF <NUM>, and classifies the HCF score <NUM> as the live information <NUM>, the not decided information <NUM>, or the spoof information <NUM>. The HCF DNN <NUM> may be, for example, a lightened network trained to perform early decision described above, or may be the classifier <NUM> described above.

In the early decision stage <NUM>, the detection apparatus <NUM> detects whether the biometric information is spoofed using the lightened network <NUM>. Thus, the amount of information and/or the capacity of the network <NUM> is limited when compared to the image DNN <NUM> used for score fusion <NUM>. Therefore, the detection apparatus <NUM> determines whether the biometric information is spoofed if a determination that the biometric information is spoofed is clearly ascertained through the HCF DNN <NUM> in the early decision stage <NUM>, and if not, defers a determination on whether the biometric information is spoofed to be determined in the score fusion stage <NUM>.

In the score fusion stage <NUM>, the detection apparatus may finally determine whether the biometric information is spoofed by fusing or combining the HCF score <NUM>, calculated based on the HCF <NUM>, and the image score <NUM> in operation <NUM>. In the score fusion stage <NUM>, the detection apparatus may fuse only a high confidence HCF score <NUM> with the image score <NUM>, as described above with reference to <FIG>.

<FIG> illustrates an example method of detecting whether biometric information is spoofed, in accordance with one or more embodiments. The operations in <FIG> may be performed in the sequence and manner as shown, although the order of some operations may be changed or some of the operations omitted without departing from the scope of the illustrative examples described. Many of the operations shown in <FIG> may be performed in parallel or concurrently. One or more blocks of <FIG>, and combinations of the blocks, can be implemented by special purpose hardware-based computer that perform the specified functions, or combinations of special purpose hardware and computer instructions. In addition to the description of <FIG> below, the descriptions of <FIG> are also applicable to <FIG>, and are incorporated herein by reference. Thus, the above description may not be repeated here.

Referring to <FIG>, the process of detecting whether biometric information is spoofed by a detection apparatus through operations <NUM> to <NUM> is illustrated.

In operation <NUM>, the detection apparatus may receive information sensed by a sensor. In an example, the detection apparatus may receive a physical value, such as, but not limited to, oxygen saturation, impedance, face depth information, electrical resistance, temperature (body temperature), heart rate, and/or similar image information.

In operation <NUM>, the detection apparatus may extract an HCF from the received information.

In operation <NUM>, the detection apparatus may calculate an HCF Score from the extracted HCF through an HCF DNN.

In operation <NUM>, the detection apparatus may perform an early decision process based on the calculated HCF Score. In operation <NUM>, the detection apparatus may detect whether biometric information is spoofed by determining whether the biometric information is live information or spoof information based on the HCF score. The detection apparatus may transmit information for which whether the biometric information is live information or spoof information is not determined in the early decision operation <NUM>, to the following process, that is, the fusion decision operation <NUM>.

In operation <NUM>, the detection apparatus may determine whether the received information has a score the confidence of which is higher than a preset criterion. If the information has a score the confidence of which is higher than the preset criterion, the detection apparatus may transmit the high confidence score to fusion operation <NUM>. In this example, a low confidence score may be excluded from the fusion operation.

Additionally, the detection apparatus may load an image in operation <NUM>. In an example, operations <NUM> and <NUM> may be performed concurrently or at a predetermined time interval. In operation <NUM>, the detection apparatus may calculate an image score through an image DNN.

In operation <NUM>, the detection apparatus may calculate one final score by fusing the high confidence score with the image score.

In operation <NUM>, the detection apparatus may determine whether the biometric information is live information or spoof information based on the calculated final score. In operation <NUM>, the detection apparatus may detect whether the biometric information is spoofed.

<FIG> illustrates an example of applying an apparatus to detect whether biometric information is spoofed to a terminal.

Referring to <FIG>, a terminal that performs the early decision process and the score fusion process described above is illustrated. The terminal may include the detection apparatus described above, or may include the function of the detection apparatus.

The terminal may obtain an image <NUM> from a matcher that performs the matching process of verifying whether a user attempting user authentication with biometric information (for example, a fingerprint) has the authority to access a system, that is, is an enrolled user of the system. Additionally, after the matching process, the terminal may obtain an image <NUM> from ASP that determines whether the biometric information is live information or spoof information. The image <NUM> and the image <NUM> may have a predetermined time difference therebetween.

Additionally, the terminal may calculate a first score by applying an HCF <NUM> obtained from the sensor to an HCF DNN <NUM>. In this example, the terminal may transmit, to the HCF DNN <NUM>, dynamic features extracted from the image <NUM> and the image <NUM> through filtering by a filter <NUM>, or may transmit, to an image DNN <NUM>, second feature information extracted from the image <NUM> and the image <NUM>.

In an example, in response to the determination that a first score falls within a first threshold range, the terminal may early determine whether the biometric information is spoofed based on the first score by turning on a switch <NUM>, in operation <NUM>. In this example, thresholds (for example, a first threshold and a second threshold) (Th) <NUM> for determining the first threshold range may be adjusted through feedback according to the result of the final decision <NUM>, which will be described later, or may be adjusted in a user-customized manner. When the thresholds <NUM> are adjusted, a determination whether to perform the early decision process <NUM> may be activated by turning on/off the switch <NUM> based on the thresholds <NUM>.

When the early decision process <NUM> is not performed, the detection apparatus may calculate a second score based on the second feature information transmitted to the image DNN <NUM>. In this example, the terminal may fuse the first score with the second score by turning on the switch <NUM>, in operation <NUM>. The terminal may perform a final decision through the fusion of the first score and the second score, in operation <NUM>.

In an example, it is possible to determine whether to activate the spoofing detection function by turning on/off the switch <NUM> to perform the early decision process <NUM> and the score fusion process <NUM> through a control variable. In this example, the terminal may refer to the result of final decision <NUM> to activate the spoofing detection function in early decision <NUM>.

Further, in an example, optimal thresholds for the first threshold range and/or the second threshold range may be derived for each terminal through adaptive learning, and the function to perform the early decision process <NUM> and the score fusion process <NUM> may be activated if the early decision process <NUM> and the score fusion process <NUM> help to improve the performance.

<FIG> illustrates an example of an apparatus that detects whether biometric information is spoofed. Referring to <FIG>, an apparatus <NUM> that detects whether biometric information is spoofed (hereinafter, the "detection apparatus") may include a sensor <NUM>, a communication interface <NUM>, a processor <NUM>, an output device <NUM>, and a memory <NUM>. The sensor <NUM>, the communication interface <NUM>, the processor <NUM>, the output device <NUM>, and the memory <NUM> may be connected to each other through a communication bus <NUM>.

The sensor <NUM> senses biometric information of a user. The sensor <NUM> may include, as non-limiting examples, any one or any combination of an ultrasonic fingerprint sensor, a depth sensor, an image sensor, an iris scanner, or a facial recognition sensor. However, examples are not limited thereto. The biometric information may include, as non-limiting examples, any one of a fingerprint, an iris, and a face of the user. However, examples are not limited thereto.

The communication interface <NUM> receives, from the sensor <NUM>, a static feature related to the biometric information of the user and images related to the biometric information. In an example, the communication interface <NUM> may output the biometric information and/or the images related to the biometric information received from the sensor <NUM> to the outside of the detection apparatus <NUM>. Additionally, the communication interface <NUM> may transmit information regarding whether the biometric information is spoofed, detected by the processor <NUM>, together with the biometric information to another device, or output the same to the outside of the detection apparatus <NUM>. In this example, the biometric information and whether the biometric information is spoofed may be matched with each other.

The processor <NUM> obtains first feature information including the static feature received through the communication interface <NUM> and a dynamic feature extracted based on the images. The processor <NUM> detects whether the biometric information is spoofed based on a first score calculated based on the first feature information. Further, the processor <NUM> fuses the first score and a second score calculated based on second feature information extracted from the images, based on a result of determining whether the biometric information is spoofed based on the first score. The processor <NUM> detects whether the biometric information is spoofed based on a fused score.

However, the operation of the processor <NUM> is not limited thereto. Alternatively, the processor <NUM> may perform the above operation together with at least one of the operations described above with reference to <FIG>.

The processor <NUM> may be a neural network or detection apparatus implemented by hardware including a circuit having a physical structure to perform desired operations. In an example, the desired operations may include instructions or codes included in a program. In an example, the hardware-implemented detection apparatus may include a microprocessor, a central processing unit (CPU), a graphics processing unit (GPU), a processor core, a multi-core processor, a multiprocessor, an application-specific integrated circuit (ASIC), a field-programmable gate array (FPGA), and a neural processing unit (NPU).

The processor <NUM> may execute the program and control the detection apparatus <NUM>. Program codes to be executed by the processor <NUM> may be stored in the memory <NUM>.

The apparatuses, units, modules, devices, and other components described herein, are implemented by hardware components. Examples of hardware components that may be used to perform the operations described in this application where appropriate include controllers, sensors, generators, drivers, memories, comparators, arithmetic logic units, adders, subtractors, multipliers, dividers, integrators, and any other electronic components configured to perform the operations described in this application. In other examples, one or more of the hardware components that perform the operations described in this application are implemented by computing hardware, for example, by one or more processors or computers. A processor or computer may be implemented by one or more processing elements, such as an array of logic gates, a controller and an arithmetic logic unit, a digital signal processor, a microcomputer, a programmable logic controller, a field-programmable gate array, a programmable logic array, a microprocessor, or any other device or combination of devices that is configured to respond to and execute instructions in a defined manner to achieve a desired result. In one example, a processor or computer includes, or is connected to, one or more memories storing instructions or software that are executed by the processor or computer. Hardware components implemented by a processor or computer may execute instructions or software, such as an operating system (OS) and one or more software applications that run on the OS, to perform the operations described in this application. The hardware components may also access, manipulate, process, create, and store data in response to execution of the instructions or software. For simplicity, the singular term "processor" or "computer" may be used in the description of the examples described in this application, but in other examples multiple processors or computers may be used, or a processor or computer may include multiple processing elements, or multiple types of processing elements, or both. For example, a single hardware component or two or more hardware components may be implemented by a single processor, or two or more processors, or a processor and a controller. One or more hardware components may be implemented by one or more processors, or a processor and a controller, and one or more other hardware components may be implemented by one or more other processors, or another processor and another controller. One or more processors, or a processor and a controller, may implement a single hardware component, or two or more hardware components. A hardware component may have any one or more of different processing configurations, examples of which include a single processor, independent processors, parallel processors, single-instruction single-data (SISD) multiprocessing, single-instruction multiple-data (SIMD) multiprocessing, multiple-instruction single-data (MISD) multiprocessing, and multiple-instruction multiple-data (MIMD) multiprocessing.

The output device <NUM> may output whether the biometric information is spoofed, detected by the processor <NUM>. The output device <NUM> may include, for example, a display, an alarm, a speaker, or other various types of output devices for informing the user of whether the biometric information is spoofed.

The memory <NUM> may store the biometric information of the user sensed by the sensor <NUM>, that is, the static feature related to the biometric information of the user and the images obtained by capturing the biometric information. Further, the memory <NUM> may store the first feature information and/or the second feature information obtained by the processor <NUM>. The memory <NUM> may store the first score, the second score, and the fused score calculated by the processor <NUM>. In addition, the memory <NUM> may store the biometric information and whether the biometric information is spoofed, detected by the processor <NUM>, by matching them to each other.

The memory <NUM> may store a variety of information generated in the processing process of the processor <NUM> described above. In addition, the memory <NUM> may store a variety of data and programs. The memory <NUM> may include a volatile memory or a non-volatile memory. The memory <NUM> may include a large-capacity storage medium such as a hard disk to store a variety of data.

In another example, the instructions or software includes higher-level code that is executed by the one or more processors or computers using an interpreter.

Claim 1:
A biometric information spoofing detection method, the method comprising:
receiving, from a sensor (<NUM>), first feature information (<NUM>) including a static feature (<NUM>) associated with biometric information of a user and a dynamic feature (<NUM>) obtained based on images (<NUM>, <NUM>) associated with the biometric information, as well as the images (<NUM>, <NUM>), wherein the static feature (<NUM>) includes a physical measure obtainable directly from the sensor (<NUM>) and the dynamic feature (<NUM>) is calculated based on a difference between image features extracted respectively from the images, the method further comprising
calculating a first score (<NUM>) by applying the first feature information (<NUM>) to a first neural network (<NUM><NUM>, <NUM>),
extracting second feature information from the images (<NUM>, <NUM>) and calculating a second score (<NUM>) based on the extracted second feature information by implementing a second neural network (<NUM>, <NUM>) ;
determining whether the first score (<NUM>) is in a first threshold range for an early decision that the biometric information is spoofed;
in response to determining that the first score (<NUM>) falls outside of the first threshold range:
determining whether to fuse the first score (<NUM>) with the second score (<NUM>);
in response to determining to fuse the first score (<NUM>) with the second score (<NUM>);
determining a final score by fusing the first score (<NUM>) with the second score (<NUM>)and
determining that the biometric information is spoofed based on the final score.