Patent Description:
Over time, updates of the firmware and the machine-learned model defined by the firmware may be developed for increased security and quality, such as better spoof rejection, better facial recognition, and so on. However, different versions of the machine-learned model can produce embeddings that are not backward compatible with older versions of the machine-learned model that execute within prior versions of the firmware. In addition, gradual changes to the user's face may occur over time, such as beard growth, aging, new glasses, etc. These changes can cause the success rate of the face authentication to decrease over time because the embeddings generated during successive face-authentication attempts may drift away from the stored embeddings that were created at enrollment. Because of this embedding incompatibility, either triggered from an update to the machine-learned model or due to embedding-drift, the user is typically forced to re-enroll for face authentication as if it were the first time enrolling with face authentication on the user device. Forcing the user to re-enroll to compensate for embedding drift or otherwise incompatible embeddings is inefficient and significantly diminishes the user experience. <CIT> describes a data processing method and apparatus. <CIT>describes techniques for implementing face-based authentication with situational adaptivity. <CIT>describes methods and systems for face detection and recognition in images captured by a camera on a device.

This document describes techniques and systems, according to the independent claims <NUM>, <NUM>, <NUM> and <NUM>, that enable face authentication embedding migration and drift-compensation. A user device is updated to host both a current version of firmware including a current face-authentication model and an updated version of the firmware including an updated version of the face-authentication model. During an authentication attempt, the user device loads both versions of the face-authentication model for execution in the same secure session. Embeddings are generated side-by-side by the two different face-authentication model versions using the same input images during authentication. The current face-authentication model is used to generate embeddings for validation against a stored set of enrolled embeddings for face authentication, while the updated face-authentication model is used to generate new embeddings for migration to an updated set of embeddings. The embeddings generated by the updated face-authentication model are stored in secure storage until a complete set has been collected. In response to obtaining a complete set of new embeddings, the user device may switch to the updated face-authentication model and uses the new set of embeddings for subsequent authentication attempts. Further, the original set of enrolled embeddings are deleted. A migration may not be necessary for every face-authentication model update, but may be necessary for updates that do not have compatible embeddings.

This summary is provided to introduce simplified concepts concerning face authentication embedding migration and drift-compensation, which is further described below in the Detailed Description and Drawings. This summary is not intended to identify essential features of the claimed subject matter, nor is it intended for use in determining the scope of the claimed subject matter.

The details of one or more aspects of face authentication embedding migration and drift-compensation are described in this document with reference to the following drawings. The same numbers are used throughout the drawings to reference like features and components:.

This document describes techniques and systems that enable face authentication embedding migration and drift-compensation. Updated versions of firmware including an updated face-authentication model used for face authentication may not be backwards compatible with older versions of the firmware, which dramatically reduces code complexity and code size of the updated versions of the firmware. Consequently, each time a user updates their computing device, the user may be forced to re-enroll for face authentication by repeating a setup process to create a new user profile using the updated firmware. To avoid this inefficient process, techniques and systems are described to load two versions of firmware in the same secure session, executing a first, current firmware for authentication and a second, updated firmware for embedding migration. After the user is authenticated using the current firmware, the user device creates a new user profile by executing the updated firmware as a background task, over a period of time and without requiring the user to re-enroll. The user device may then automatically switch to executing the updated firmware in place of the old version of the firmware for authentication when the new user profile is sufficiently complete. Thus, the embeddings used for face authentication can be updated seamlessly and without user supervision, over a period of time (e.g., one week) by migrating the embeddings to a new, updated profile while maintaining the entire secure workflow and without leaking any image or embedding information outside of the secure pipeline. Further, because the updated embeddings are generated after the user is authenticated, the migration of the updated embeddings does not provide user visible latency. In this way, the techniques described herein provide a more efficient process of updating a user-profile during an authenticating process, avoiding the need for a separate re-enrollment procedure. Aspects can thus provide an improved process of updating a user-profile, e.g., for a new firmware or for drift-compensation. Furthermore, aspects may provide a more accurate user profile, by allowing embeddings to be captured over a period of time in varying conditions.

In aspects, a method of migrating face authentication profile data in a user device performed by the user device is described. The method includes updating the user device to include both a current version of firmware and an updated version of the firmware. The method also includes, subsequent to the updating, receiving an indication of a face-authentication attempt and image data associated with a user's face. In addition, the method includes loading both the current version of the firmware and the updated version of the firmware in a same secure session, and executing the current version of the firmware to generate comparison data using the captured image data. Further, the method includes authenticating the user to unlock the user device based on the comparison data substantially matching a first profile, the first profile based on a set of enrolled embeddings that were previously generated using the current version of the firmware on previously captured image data of the user's face. Also, the method includes, responsive to authenticating the user to unlock the user device, generating one or more new embeddings by executing the updated version of the firmware and processing the captured image data. Additionally, the method includes storing the one or more new embeddings at a separate location from the set of enrolled embeddings, the one or more new embeddings stored as part of a second profile for the user.

In aspects, a method of migrating face authentication profile data for drift-compensation in a user device is described. The method includes receiving, by a hypervisor of the user device, a profile identity and image data associated with a face-authentication attempt of a user's face. In addition, the method includes specifying, by the hypervisor, the profile identity and an optional migration profile. Also, the method includes generating, by a processor unit, a new embedding using the image data. Further, the method includes authenticating the user to unlock the user device based on the new embedding substantially matching an enrolled embedding from a set of enrolled embeddings corresponding to the profile identity. The method also includes determining that the new embedding differs from the enrolled embedding by an amount that is within a range bounded by first and second threshold differences, the first threshold difference representing a maximum difference that is acceptable to be considered a match, the second threshold difference being less than the first threshold difference and representing a minimum difference that is acceptable to be considered drift. Additionally, the method includes, based on the determining, storing the new embedding as a drifted embedding in a migration region corresponding to the migration profile.

These are but a few examples of how the described techniques and devices may be used to enable face authentication embedding migration and drift-compensation. Other examples and implementations are described throughout this document. The document now turns to an example operating environment, after which example devices, methods, and systems are described.

<FIG> illustrates an example environment <NUM> in which techniques enabling face authentication embedding migration and drift-compensation can be implemented. The example environment <NUM> includes a user device <NUM> (e.g., electronic device), which includes, or is associated with, a near-infrared (NIR) camera system <NUM>, a secure storage unit <NUM> (e.g., security chip), an authentication system <NUM>, and a hypervisor <NUM>. As further described below, the authentication system <NUM> is configured to implement a facial recognition module <NUM>. For security, image data captured by the NIR camera system <NUM> are passed to the authentication system <NUM> via the hypervisor <NUM>. Near infrared (NIR) light generally refers to light within a range of wavelengths between approximately <NUM> and <NUM>,<NUM>, and is invisible to the human eye. The facial recognition module <NUM> is also configured to use a neural network (e.g., convolutional neural network) or other type of machine-learned model, trained using machine learning techniques, to generate one or more embeddings from the captured image data.

If the image data is captured during an enrollment process of face authentication, the facial recognition module <NUM> stores the embeddings generated by the machine-learned model in the secure storage unit <NUM>. If, however, the user device <NUM> is in a locked state <NUM>-<NUM> and a user <NUM> is attempting to unlock the user device <NUM> subsequent to enrollment, the facial recognition module <NUM> compares the embedding to a set of embeddings stored in the secure storage unit <NUM>. Based on the comparison, the machine-learned model of the facial recognition module <NUM> either rejects the authentication or authenticates the user <NUM> to unlock the user device <NUM> to an unlocked state <NUM>-<NUM>. If the authentication is rejected, the user device <NUM> remains in the locked state <NUM>-<NUM>. If the user <NUM> is authenticated, the user device <NUM> transitions to the unlocked state <NUM>-<NUM>, as represented in <FIG> by an unlock icon <NUM>. In this way, the user <NUM> essentially uses the user device <NUM> to take a "selfie" to unlock the user device <NUM>, without being required to provide any additional input.

Throughout this disclosure examples are described where a computing system (e.g., the user device <NUM>, a client device, a server device, a computer, or other type of computing system) may analyze information (e.g., radar, inertial, and facial-recognition sensor data) associated with a user, such as the just-mentioned facial features. The computing system, however, can be configured to only use the information after the computing system receives explicit permission from the user of the computing system to use the data. For example, in situations where the user device <NUM> analyzes sensor data for facial features to authenticate the user <NUM>, the sensor data is contained in a secure pipeline, which includes a secure camera pipeline, the hypervisor <NUM>, and an image processor (described below), and cannot be removed from this secure pipeline. Likewise, the embeddings cannot leave this secure pipeline but are stored in the secure storage unit <NUM> and cannot be retrieved for use outside of the secure storage unit <NUM>. The individual users may have constant control over what programs can or cannot do with sensor data. In addition, information collected may be pre-treated in one or more ways before it is transferred, stored, or otherwise used, so that personally-identifiable information is removed. For example, before the user device <NUM> shares sensor data with another device (e.g., to train a model executing at another device), the user device <NUM> may pre-treat the sensor data to ensure that any user-identifying information or device-identifying information embedded in the data is removed. Thus, the user may have control over whether information is collected about the user and the user's device, and how such information, if collected, may be used by the computing device and/or a remote computing system.

In more detail, consider <FIG>, which illustrates an example implementation <NUM> of the user device <NUM> that can implement face authentication embedding migration and drift-compensation. The user device <NUM> of <FIG> is illustrated with a variety of example devices, including a smartphone <NUM>-<NUM>, a tablet <NUM>-<NUM>, a laptop <NUM>-<NUM>, a desktop computer <NUM>-<NUM>, a computing watch <NUM>-<NUM>, computing spectacles <NUM>-<NUM>, a gaming system <NUM>-<NUM>, a home-automation and control system <NUM>-<NUM>, and a microwave <NUM>-<NUM>. The user device <NUM> can also include other devices, such as televisions, entertainment systems, audio systems, automobiles, drones, track pads, drawing pads, netbooks, e-readers, home security systems, and other home appliances. Note that the user device <NUM> can be wearable, non-wearable but mobile, or relatively immobile (e.g., desktops and appliances).

The user device <NUM> also includes one or more computer processors <NUM> and one or more computer-readable media <NUM>, which includes memory media and storage media. Applications and/or an operating system <NUM> implemented as computer-readable instructions on the computer-readable media <NUM> can be executed by the computer processors <NUM> to provide some or all of the functionalities described herein. For example, the computer-readable media <NUM> can include the facial recognition module <NUM> along with multiple versions of firmware, such as a current version of firmware <NUM> and an updated version of firmware <NUM>, managed by a hypervisor <NUM> of the operating system <NUM> executing on the user device <NUM>. Each version of firmware includes a machine-learned model trained to generate embeddings from captured image data. The facial recognition module <NUM> can call on the different machine-learned models depending on which version of firmware is loaded by the hypervisor <NUM>. The computer-readable media <NUM> includes the secure storage unit <NUM>, which is not accessible by processes or applications in the user space.

The secure storage unit <NUM> is configured to store security data (e.g., user credentials) used for privacy controls, such as controls to unlock the user device <NUM> (including face authentication data, password/passcode information, fingerprint data, and so on). Although this security data can be used to authenticate the user <NUM> to unlock the user device <NUM> using face authentication, password/passcode authentication, fingerprint authentication, and so on, personal information about the user <NUM> cannot be obtained by the security data. Specifically, the user <NUM> cannot be identified by the security data. Rather, the security data is used to simply determine whether data received from a user attempting to unlock the phone matches stored profile data representing a user that set up the security on the user device <NUM>. In an example, the embeddings generated from captured images of the user's face are numerical vector representations of facial features of the user <NUM>. These embeddings are simply used for comparison to new embeddings, generated from images captured during a face-authentication attempt, to locate a match.

The user device <NUM> may also include a network interface <NUM>. The user device <NUM> can use the network interface <NUM> for communicating data over wired, wireless, or optical networks. By way of example and not limitation, the network interface <NUM> may communicate data over a local-area-network (LAN), a wireless local-area-network (WLAN), a personal-area-network (PAN), a wide-area-network (WAN), an intranet, the Internet, a peer-to-peer network, point-to-point network, or a mesh network.

Various implementations of the authentication system <NUM> can include a System-on-Chip (SoC), one or more Integrated Circuits (ICs), a processor with embedded processor instructions or configured to access processor instructions stored in memory, hardware with embedded firmware, a printed circuit board with various hardware components, or any combination thereof. As described in further detail below, the authentication system <NUM> can, in a secure mode, compare authentication data received from the user <NUM> to the security data stored in the secure storage unit <NUM> for authenticating the user <NUM> to unlock the user device <NUM>. In some aspects, the authentication system <NUM> generates the authentication data using image data obtained from the NIR camera system <NUM> and provides the authentication data to the secure storage unit <NUM> to enable the secure storage unit <NUM> to compare the authentication data to the stored security data and determine if there is a match.

The NIR camera system <NUM> is implemented to capture NIR image data usable to generate a three-dimensional depth map of an object, such as a user's face. Further details are described with respect to <FIG> below.

The user device <NUM> can also include one or more sensors <NUM>. The one or more sensors <NUM> can include any of a variety of sensors, such as an audio sensor (e.g., a microphone), a touch-input sensor (e.g., a touchscreen), an image-capture device (e.g., a camera or video-camera), proximity sensors (e.g., capacitive sensors), or ambient light sensors (e.g., a photodetector). In at least some implementations, the user device <NUM> can include a radar system (not shown) to detect a proximity of the user <NUM> to the user device <NUM>, and based on that proximity, initiate one or more components and/or functions, such as firing up the NIR camera system <NUM> and the authentication system <NUM> to initiate an attempt for face authentication.

The user device <NUM> can also include a display device <NUM>. The display device <NUM> can include any suitable display device, such as a touchscreen, a liquid crystal display (LCD), thin film transistor (TFT) LCD, an in-place switching (IPS) LCD, a capacitive touchscreen display, an organic light emitting diode (OLED) display, an active-matrix organic light-emitting diode (AMOLED) display, super AMOLED display, and so forth.

<FIG> illustrates an example implementation <NUM> of the authentication system <NUM> and NIR camera system <NUM> of <FIG> and <FIG> in further detail. The authentication system <NUM> includes, or communicates with, the facial recognition module <NUM>, a processor unit <NUM>, and a secure storage unit <NUM>.

The processor unit <NUM> is configured with the functionalities of both an image processing unit (IPU) and a tensor processing unit (TPU). Specifically, the processor unit <NUM> is configured for image processing of digital image input as well as for neural network machine learning. Essentially, the processor unit <NUM> is a combination of an IPU and an artificial-intelligence accelerator application-specific integrated circuit (ASIC) (e.g., the TPU). The processor unit <NUM> specifically is configured to generate embeddings for face authentication by using images of a person's face captured in the near-infrared spectrum as input to a machine learned model.

The NIR camera system <NUM> includes a dot projector <NUM>, a flood illuminator <NUM>, and one or more NIR camera sensors <NUM>. The flood illuminator <NUM> illuminates the subject with NIR light and the NIR camera sensors <NUM> each capture an image of the subject based on the NIR light. The dot projector <NUM> projects thousands of NIR dots onto the subject and the NIR camera sensors <NUM> each capture an image of the resulting dot pattern. The processor unit <NUM> of the authentication system <NUM> reads the image and the dot pattern and generates a three-dimensional facial map. When multiple (e.g., two) NIR cameras are used, the processor unit <NUM> calculates a difference between matching points on the different captured images, which provides a depth for respective pixels and enhances the accuracy of the three-dimensional facial map.

These and other capabilities and configurations, as well as ways in which entities of <FIG> act and interact, are set forth in greater detail below. These entities may be further divided, combined, and so on. The environment <NUM> of <FIG>, the implementations <NUM> and <NUM> of <FIG> and <FIG>, and the detailed illustrations of <FIG> illustrate some of many possible environments and devices capable of employing the described techniques.

<FIG> illustrate example implementations <NUM>, <NUM>, and <NUM> of face authentication embedding migration relative to a firmware update. Firstly, the user updates the user device <NUM> (from <FIG> and <FIG>), such as via over-the-air (OTA) transmission, to a system image that includes both the current version of firmware <NUM> (from <FIG>), such as firmware-<NUM><NUM>, and the updated version of the firmware <NUM> (from <FIG>), such as firmware-<NUM><NUM>. During a face-authentication attempt, the hypervisor <NUM> enters a secure session and loads the firmware-<NUM><NUM> into the authentication system <NUM>, which attempts to authenticate the user <NUM>. The firmware-<NUM><NUM> was used during enrollment to create a set of enrolled embeddings <NUM> (e.g., <NUM>-<NUM>, <NUM>-<NUM>, <NUM>-<NUM>,. <NUM>-n) corresponding to a first profile, such as profile-<NUM><NUM> (also referred to as an enrollment profile). For example, the facial recognition module <NUM> sends a message <NUM>, including a profile identity (ID), to the hypervisor <NUM> of the operating system <NUM> of the user device <NUM>. The message <NUM> may also include images <NUM> (e.g., image data) captured by the NIR camera system <NUM>. The profile ID may be cached by the facial recognition module <NUM>, which manages user profiles on the user device <NUM>. The facial recognition module <NUM> may associate the profile ID with an embedding version (e.g., version of embeddings generated based on a particular version of firmware or drift in biometrics of the user's face).

The hypervisor <NUM> passes the images <NUM> to the processor unit <NUM>, which applies the firmware-<NUM><NUM> to the images <NUM> to generate one or more embeddings <NUM>. In at least one aspect, the processor unit <NUM> directly trains its output to be a compact <NUM>-dimension embedding.

Embeddings <NUM> created during face-authentication attempts, such as this, are used for comparison to the enrolled embeddings <NUM>, and are referred to herein as comparison embeddings <NUM>. Because embeddings are vectors, vector distance can be used to calculate the similarity between two vectors. The comparison embeddings <NUM> are passed to the secure storage unit <NUM>, which attempts to locate a match, or at least a substantial match that matches to within a threshold value of vector distance. The secure storage unit <NUM> returns an authentication status <NUM> (e.g., match or no match) back to the hypervisor <NUM> via the processor unit <NUM>, and the hypervisor <NUM> passes the authentication status <NUM> to the user space for unlocking the user device <NUM>. In the illustrated example, the secure storage unit <NUM> finds a matching enrolled embedding <NUM>-<NUM> that matches the comparison embedding <NUM>. In addition, the secure storage unit <NUM> associates a last-authenticated user ID with a user ID of the profile-<NUM><NUM> that was just authenticated.

At some point after updating the user device <NUM> to the system image that includes both the current and updated versions of the firmware, the authentication system <NUM> sends a request to the secure storage unit <NUM> to create a new profile, such as profile-<NUM><NUM>. This new profile is also referred to as a migration profile and corresponds to a new profile ID. In the illustrated example, the profile-<NUM><NUM> has been created but it is currently empty (e.g., new embeddings <NUM> have not yet been generated or stored for this new profile-<NUM><NUM>). In addition, the secure storage unit <NUM> associates the profile-<NUM><NUM> with the last-authenticated user ID in order to correlate the profile-<NUM><NUM> with the same user as the profile-<NUM><NUM>.

Continuing with <FIG>, after authenticating the user <NUM> and during the same secure session, the hypervisor <NUM> triggers the firmware-<NUM><NUM> to run an enrollment update with the same image data (e.g., images <NUM>) that was collected during the authentication attempt. For example, the hypervisor <NUM> can transmit a signal or message to the processor unit <NUM> to use the firmware-<NUM><NUM> to generate one or more new embeddings <NUM> based on the images <NUM>. Because of differences in the updated firmware-<NUM><NUM>, some of these new embeddings <NUM> may not be compatible with the current firmware-<NUM><NUM>. The new embeddings <NUM> are stored in a separate location in the secure storage unit <NUM> from the set of enrolled embeddings <NUM>. For example, in <FIG>, the enrolled embedding <NUM>-<NUM> was identified as a match to the comparison embedding <NUM> produced by the firmware-<NUM><NUM> using the images <NUM>. Then, when the firmware-<NUM><NUM> is run on the images <NUM>, the resulting new embedding is stored as embedding <NUM>-<NUM> in the profile-<NUM><NUM>. This new embedding <NUM>-<NUM> may be referred to as a migration embedding. The secure storage unit <NUM> returns a status <NUM> to the hypervisor <NUM> indicating that the new embedding <NUM>-<NUM> has been enrolled in the profile-<NUM><NUM>. The status <NUM> also indicates whether the new enrollments are full, such that the profile-<NUM><NUM> is ready (e.g., sufficiently complete) for use in authenticating the user <NUM>. With only a small number (e.g., <NUM> to <NUM>) of new embeddings, the profile-<NUM><NUM> may not be usable to authenticate the user <NUM> with a high confidence level. Therefore, more of the new embeddings <NUM> are likely required in order to fill in the profile-<NUM><NUM>.

After the migration embedding is stored in the profile-<NUM><NUM>, the secure storage unit <NUM> resets the last-authenticated user ID. Then, the secure session ends, e.g., the hypervisor <NUM> exits the secure session.

To collect additional new embeddings <NUM>, this process repeats over a period of time, such as throughout a week, or for a quantity of times that satisfies a time threshold (e.g., fifty times), in response to a series of successive face-authentication attempts. During these subsequent successful face-authentication attempts, the user <NUM> is authenticated against the set of enrolled embeddings <NUM> corresponding to the profile-<NUM><NUM>, and additional new embeddings (e.g., embeddings <NUM>-<NUM>, <NUM>-<NUM>,. <NUM>-n) are generated using the updated firmware, firmware-<NUM><NUM>. However, a new embedding is not necessarily stored as part of the profile-<NUM><NUM> during each and every face-authentication attempt. For instance, if the secure storage unit <NUM> determines that the new embedding is substantially the same as another new embedding already stored in the migration region, then the new embedding is determined to be duplicative and is not saved.

When a sufficient amount or number of new embeddings <NUM> are generated and stored in the profile-<NUM><NUM>, then the secure storage unit <NUM> returns the status <NUM> indicating that the profile-<NUM><NUM> is sufficiently complete (substantially all (e.g., ninety-five percent) embeddings for profile-<NUM><NUM> are filled). Also, the secure storage unit <NUM> determines whether the profile-<NUM><NUM> could have been used to authenticate the user during this most-recent authentication attempt. If not, then this determination is repeated during a next face-authentication attempt after generating an additional new embedding <NUM> that may replace a previously stored embedding <NUM> to improve the profile-<NUM><NUM>.

If, however, the user <NUM> could have been authenticated against the profile-<NUM><NUM>, then, continuing with <FIG>, the profile-<NUM><NUM> is saved as the enrollment profile to be used in subsequent face-authentication attempts. In addition, the authentication system <NUM> clears the profile-<NUM><NUM> by deleting the set of enrolled embeddings <NUM>. During subsequent successful face-authentication attempts, the authentication system <NUM> no longer uses the firmware-<NUM><NUM> and instead only relies on the firmware-<NUM><NUM> to authenticate the user <NUM>. In some aspects, the firmware-<NUM><NUM> may be deleted.

The original enrollment process attempts to fill out the embedding space of a profile by capturing embeddings from multiple angles. The migration embeddings (e.g., new embeddings <NUM>), however, may not be captured from each angle because the authentication system <NUM> does not prompt the user <NUM> to capture images <NUM> from different poses. This does not mean that the migration cannot cover as large of an embedding space as the original enrollment. Rather, in contrast to the original enrollment, which uses a short window of time (e.g., one minute, <NUM> seconds) to capture multiple different angles of the user's face, the migration enrollment is not limited to such a short time frame but can capture embeddings over the course of several days. Ambient lighting conditions may vary from one authentication attempt to another on different days or different hours of the day. Because embeddings are captured in varying conditions, they are likely to cover a much larger embedding space than the original enrollment and also be much more indicative of the user's actual behavior.

Over the course of the migration (e.g., generating and storing new embeddings according to the updated firmware-<NUM><NUM> in the background during multiple different face-authentication attempts that use the current firmware-<NUM><NUM>), the authentication system <NUM> monitors the new embeddings to avoid duplicate data. For example, to avoid unnecessary duplication of the embeddings, the authentication system <NUM> determines which embeddings to store based on a distance between different embedding values in comparison to a distance threshold. This provides embeddings that are sufficiently different from one another to enable the user <NUM> to unlock the user device <NUM> in a variety of, or regardless of, ambient lighting conditions or the orientation of the user device <NUM> relative to the user's face.

These techniques provide a simple migration scheme for migrating various profiles across various combinations of firmware. Also, the firmware can remain relatively fixed in size across updates. Further, these techniques require no additional latency cost on the critical path for validation of the user profile ID.

<FIG> and <FIG> illustrate example implementations <NUM> and <NUM> of face authentication embedding migration for drift-compensation to correct biometric drift. As mentioned above, the unlock success rate may decrease over time due to gradual changes to the user's face that occur over time (e.g., beard growth, aging, new glasses, etc.) because the embeddings generated during face-authentication attempts may drift away from the stored embeddings that were created at enrollment. The example implementations <NUM> and <NUM>, described with respect to <FIG> and <FIG>, illustrate a way to seamlessly generate a new, drifted profile based on gradual changes to the user's face, without requiring the user <NUM> to re-enroll in a setup process for face authentication.

In the example implementation <NUM>, the hypervisor <NUM> uses the firmware-<NUM><NUM> in the processor unit <NUM> to authenticate the user <NUM> in a manner similar to that described with respect to <FIG>. The hypervisor <NUM> receives, from the facial recognition module <NUM>, the message <NUM>, which includes the profile ID (e.g., profile-<NUM><NUM>) for the face-authentication attempt and an optional migration profile ID (e.g., profile-<NUM><NUM>). The hypervisor <NUM> triggers the processor unit <NUM> to apply the firmware-<NUM><NUM> to the images <NUM> to generate comparison embeddings <NUM>. The comparison embeddings <NUM> are passed to the secure storage unit <NUM> to locate a matching enrolled embedding <NUM> corresponding to the profile-<NUM><NUM>. The processor unit <NUM> returns an authentication status <NUM> to the hypervisor <NUM> via the processor unit <NUM> to indicate authentication success or failure. The hypervisor <NUM> then returns the authentication status <NUM> to the user space. If the authentication is a success, the facial recognition module <NUM> can trigger a new profile (e.g., profile-<NUM><NUM>) to open for migration by tying the profiles (e.g., profile-<NUM><NUM> and profile-<NUM><NUM>) together with the same user ID.

If the comparison embedding <NUM> passes validation but is greater than a threshold distance from the matching enrolled embedding <NUM>, then the comparison embedding <NUM> is stored in the migration profile as a drifted embedding <NUM>. In the illustrated example, the comparison embedding <NUM> substantially matches the enrolled embedding <NUM>-<NUM> sufficient for authentication. The "match," however, may not be identical. Although the comparison embedding <NUM> is sufficiently similar to the enrolled embedding <NUM>-<NUM> to be considered a match for authentication, it is also sufficiently different than the enrolled embedding <NUM>-<NUM> to be considered "drifted" from the biometrics of the user at the time of enrollment. A range of distance values may be used to determine if the comparison embedding <NUM> is drifted, such as by using a range bounded by a lower threshold distance and a higher threshold distance. In aspects, the higher threshold distance may indicate a maximum difference that is acceptable for the embeddings to be considered a match. The lower threshold distance may indicate a minimum difference in the embedding values that is acceptable for the comparison embedding <NUM> to be considered not only a match but also drifted from the enrollment embedding. If the comparison embedding <NUM> falls within this range (e.g., between the lower threshold distance and the higher threshold distance), then it is migrated to the new, migration profile and assigned a new version number. The threshold distance may also be referred to as a threshold difference due to its value representing a difference between two vectors (e.g., embeddings). In the illustrated example, the comparison embedding <NUM> is a drifted version of the enrolled embedding <NUM>-<NUM>, version <NUM>, and is thus stored in the migration profile-<NUM><NUM> as embedding <NUM>-<NUM>, version <NUM>.

Continuing with implementation <NUM> in <FIG>, additional drifted embeddings, such as drifted embeddings <NUM>-<NUM> and <NUM>-<NUM>, are collected over time and stored in the migration region corresponding to the profile-<NUM><NUM>. In this way, the profile-<NUM><NUM> is slowly built over a period of time, such as one week, two weeks, one month, etc., to account for gradual changes to the user's face. After a number of the drifted embeddings <NUM> are stored in the profile-<NUM><NUM>, the user device <NUM> may trigger a merge of the embedding regions. For example, the profile-<NUM><NUM> may be merged with the profile-<NUM><NUM> based on a matching user ID that corresponds to both of the profiles. In the merge, if the profile-<NUM><NUM> includes any empty embedding spaces, the empty embedding spaces can be filled with corresponding enrolled embeddings by merging corresponding embeddings from the profile-<NUM><NUM>. For example, the enrolled embedding <NUM>-n, version <NUM> is stored in a corresponding location in the profile-<NUM><NUM>. The profile-<NUM><NUM> is removed by deleting the enrolled embeddings <NUM>. Further, the profile-<NUM><NUM> is then used for user-authentication in successive face-authentication attempts.

One challenge of embedding migration is that a poor migration from a firmware update risks increasing false recognition rate. To avoid increasing the false recognition rate of the face authentication, an authentication token can be used to initiate an embedding migration rather than, or in addition to, identification of biometric drift in the comparison embeddings <NUM>. Returning to <FIG>, in an example implementation, the message <NUM> received by the hypervisor <NUM>, from the facial recognition module <NUM>, includes an enroll migration command along with the profile ID of the enrolled embeddings (e.g., profile-<NUM><NUM>), the profile ID of the migration region (e.g., profile-<NUM><NUM>), and an authentication token. The processor unit <NUM> generates an embedding to be enrolled based on the firmware-<NUM><NUM> and the images <NUM>. The processor unit <NUM> sends information <NUM> to the secure storage unit <NUM>. The information <NUM> includes the profile ID of the enrolled embeddings, the profile ID of the migration region, the authentication token, and an embedding version. The secure storage unit <NUM> compares a user ID of the authentication token to a user ID of the enrolled embeddings <NUM> and then locks the migration profile (e.g., profile-<NUM><NUM>) to that user ID.

The secure storage unit <NUM> selects an embedding to be replaced. In the illustrated example, the secure storage unit <NUM> selects the enrolled embedding <NUM>-<NUM> for replacement. Then, the secure storage unit <NUM> stores the new embedding (e.g., the comparison embedding <NUM>) in the migration region as embedding <NUM>-<NUM>, according to a new embedding version (e.g., version <NUM>). Additional new embeddings <NUM> are added over time, during a series of subsequent face-authentication attempts. As above, based on a number of new embeddings <NUM> stored in the migration region, the user device <NUM> can trigger a merging of the embedding regions. As a result, the specified profile-<NUM><NUM> is merged with the profile-<NUM><NUM>, which corresponds to a user ID that matches the user ID of the profile-<NUM><NUM>. Then, the profile-<NUM><NUM> is erased by deleting the set of enrolled embeddings <NUM>.

<FIG> depicts an example method <NUM> for migrating face authentication profile data, such as embeddings. The method <NUM> can be performed by the user device <NUM>, which uses the authentication system <NUM> (including the processor unit <NUM>) to perform a face-authentication attempt using image data, obtained by the NIR camera system <NUM>, and a set of embeddings stored in the secure storage unit <NUM>.

The method <NUM> is shown as a set of blocks that specify operations performed but are not necessarily limited to the order or combinations shown for performing the operations by the respective blocks. Further, any of one or more of the operations may be repeated, combined, reorganized, or linked to provide a wide array of additional and/or alternate methods. In portions of the following discussion, reference may be made to the example operating environment <NUM> of <FIG> or to entities or processes as detailed in <FIG>, reference to which is made for example only. The techniques are not limited to performance by one entity or multiple entities operating on one device.

At <NUM>, a user device is updated to include both a first version of firmware and a second version of the firmware. The second version of the firmware (e.g., the firmware-<NUM><NUM>) is an updated version of the first version (e.g., the firmware-<NUM><NUM>). In aspects, the user device <NUM> is updated to a system image that includes both the first and second versions of the firmware, such that both versions of the firmware can be loaded during the same secure session.

At <NUM>, a secure session is initiated. For example, the hypervisor <NUM> enters a secure session for passing information and data between the NIR camera system <NUM> and the authentication system <NUM>.

At <NUM>, an image of a user is captured during a face-authentication attempt. In an example, the facial recognition module <NUM> initiates the NIR camera system <NUM> to operate in a secure mode and capture one or more images <NUM> of the user's face using the dot projector <NUM>, the flood illuminator <NUM>, and the NIR camera(s) <NUM>. Resulting image data (e.g., the images <NUM>) is passed to the hypervisor <NUM> of the operating system <NUM> of the user device <NUM> for face authentication and the hypervisor <NUM> passes the image data to the processor unit <NUM>.

At <NUM>, the first version of the firmware is loaded. For example, the hypervisor <NUM> triggers the processor unit <NUM> to load the first version of the firmware. The hypervisor <NUM> can specify the first version of the firmware based on a profile ID received from the facial recognition module <NUM>.

At <NUM>, the first version of the firmware is used to generate comparison data. For example, the hypervisor <NUM> uses the first version (e.g., the firmware-<NUM><NUM>) of the firmware that is loaded in the processor unit <NUM> to generate the comparison data, such as a comparison embedding <NUM>, from the images <NUM>. This comparison embedding <NUM> is usable for comparison to a set of enrolled embeddings <NUM> that are stored in association with the profile ID.

At <NUM>, the secure storage unit compares the comparison data to a set of enrolled embeddings corresponding to a first profile. For example, the secure storage unit <NUM>, operating a secure mode, receives the comparison embedding <NUM> and the profile ID and uses the profile ID to identify the set of enrolled embeddings <NUM>. Then, the secure storage unit <NUM> attempts to locate one or more embeddings in the set of enrolled embeddings <NUM> that substantially match the comparison embedding <NUM>.

At <NUM>, the secure storage unit determines if the comparison data (e.g., comparison embedding <NUM>) substantially matches one or more of the enrolled embeddings. The secure storage unit <NUM> can determine that an embedding from the set of enrolled embeddings <NUM> substantially matches the comparison embedding <NUM> by a measurable amount (e.g., vector distance) that is within a threshold distance. If there is no match, then at <NUM>, the secure storage unit returns a fail status to the hypervisor <NUM>. To maintain security, information that leaves the secure storage unit may be limited to an authentication status, which indicates a match (e.g., "True") or no match (e.g., "false"), referring to whether the comparison embedding <NUM> substantially matches an enrolled embedding <NUM> in the enrollment profile (e.g., profile-<NUM><NUM>).

If, however, the secure storage unit identifies a matching embedding in the set of enrolled embeddings that substantially matches the comparison data, then the process proceeds to <NUM> in <FIG>. At <NUM>, based on a match being identified, the processor unit loads the second version of the firmware. For instance, after passing the authentication status <NUM> to the hypervisor <NUM> to indicate validation of the image data, the processor unit <NUM> loads the firmware-<NUM><NUM> to initiate a migration workflow.

At <NUM>, the processor unit uses the second version of the firmware to generate new embeddings from the images. Because the image data was authenticated using the first version of the firmware, the same image data is now used by the second version of the firmware for migration purposes. Accordingly, the processor unit <NUM> generates a new embedding using the firmware-<NUM><NUM> on the image data (e.g., images <NUM>).

At <NUM>, the secure storage unit stores the new embeddings as part of a second profile. For example, the secure storage unit <NUM> stores the new embedding in a migration region, which is a separate location from the enrollment region (e.g., the set of enrolled embeddings), and associates the new embedding with a migration profile ID, such as profile-<NUM><NUM>. The new embedding can now be considered a migration embedding that is associated with the migration profile.

At <NUM>, the secure storage unit determines if the second profile is sufficiently complete to be usable to authenticate the user with a high level of confidence. In aspects, the secure storage unit <NUM> can check the number of new embeddings stored in the migration region associated with the profile-<NUM><NUM> to determine a level of completeness of the profile-<NUM><NUM>. Additionally, each time the process is repeated, the secure storage unit <NUM> can determine if the user could have been reliably authenticated using the new embeddings <NUM> in the profile-<NUM><NUM>. If the second profile is not sufficiently complete (e.g., "NO"), then at <NUM> the secure storage unit returns a status, to the hypervisor via the processor unit, indicating that the second profile is not ready to be used for face authentication.

At <NUM>, the hypervisor exits the secure session. During a subsequent face-authentication attempt, the method returns to <NUM> of <FIG> to repeat the operations of <NUM> to <NUM>. If the secure storage unit determines that the second profile is sufficiently complete, such that it is ready to be used to reliably authenticate the user (e.g., "YES"), then the method proceeds to <NUM>.

At <NUM>, the hypervisor exits the secure session. At <NUM>, the set of enrolled embeddings is erased. Deleting the set of enrolled embeddings frees up limited storage space in the secure storage unit <NUM> and prevents the secure storage unit <NUM> from using outdated embeddings.

At <NUM>, the kernel stops using the first version of the firmware and only uses the second version of the firmware on succeeding face-authentication attempts, until the user device is updated again. For example, the facial recognition module <NUM> manages the profile IDs and specifies the profile-<NUM><NUM> for subsequent face-authentication attempts, treating the profile-<NUM><NUM> and its corresponding migration embeddings now as the enrollment profile and enrollment embeddings.

Generally, any of the components, modules, methods, and operations described herein can be implemented using software, firmware, hardware (e.g., fixed logic circuitry), manual processing, or any combination thereof. Some operations of the example methods may be described in the general context of executable instructions stored on computer-readable storage memory that is local and/or remote to a computer processing system, and implementations can include software applications, programs, functions, and the like. Alternatively or in addition, any of the functionality described herein can be performed, at least in part, by one or more hardware logic components, such as, and without limitation, Field-programmable Gate Arrays (FPGAs), Application-specific Integrated Circuits (ASICs), Application-specific Standard Products (ASSPs), System-on-a-chip systems (SoCs), Complex Programmable Logic Devices (CPLDs), and the like.

<FIG> illustrates various components of an example computing system <NUM> that can be implemented as any type of client, server, and/or electronic device as described with reference to the previous <FIG> to implement face authentication embedding migration and drift-compensation.

The computing system <NUM> includes communication devices <NUM> that enable wired and/or wireless communication of device data <NUM> (e.g., radar data, authentication data, reference data, received data, data that is being received, data scheduled for broadcast, and data packets of the data). The device data <NUM> or other device content can include configuration settings of the device, media content stored on the device, and/or information associated with a user of the device (e.g., an identity of a person within a radar field or customized air gesture data). Media content stored on the computing system <NUM> can include any type of radar, biometric, audio, video, and/or image data. The computing system <NUM> includes one or more data inputs <NUM> via which any type of data, media content, and/or inputs can be received, such as human utterances, interactions with a radar field, touch inputs, user-selectable inputs or interactions (explicit or implicit), messages, music, television media content, recorded video content, and any other type of audio, video, and/or image data received from any content and/or data source.

The computing system <NUM> also includes communication interfaces <NUM>, which can be implemented as any one or more of a serial and/or a parallel interface, a wireless interface, any type of network interface, a modem, and as any other type of communication interface. The communication interfaces <NUM> provide a connection and/or communication links between the computing system <NUM> and a communication network by which other electronic, computing, and communication devices communicate data with the computing system <NUM>.

The computing system <NUM> includes one or more processors <NUM> (e.g., any of microprocessors, controllers, or other controllers) that can process various computer-executable instructions to control the operation of the computing system <NUM> and to enable techniques for, or in which can be implemented, face authentication embedding migration and drift-compensation. Alternatively or additionally, the computing system <NUM> can be implemented with any one or combination of hardware, firmware, or fixed logic circuitry that is implemented in connection with processing and control circuits, which are generally identified at <NUM>. Although not shown, the computing system <NUM> can include a system bus or data transfer system that couples the various components within the device. A system bus can include any one or combination of different bus structures, such as a memory bus or memory controller, a peripheral bus, a universal serial bus, and/or a processor or local bus that utilizes any of a variety of bus architectures.

The computing system <NUM> also includes computer-readable media <NUM>, such as one or more memory devices that enable persistent and/or non-transitory data storage (in contrast to mere signal transmission), examples of which include random access memory (RAM), non-volatile memory (e.g., any one or more of a read-only memory (ROM), flash memory, EPROM, EEPROM, etc.), and a disk storage device. A disk storage device may be implemented as any type of magnetic or optical storage device, such as a hard disk drive, a recordable and/or rewriteable compact disc (CD), any type of a digital versatile disc (DVD), and the like. The computing system <NUM> can also include a mass storage media device (storage media) <NUM>.

The computer-readable media <NUM> provides data storage mechanisms to store the device data <NUM>, as well as various device applications <NUM> and any other types of information and/or data related to operational aspects of the computing system <NUM>. For example, an operating system <NUM> can be maintained as a computer application with the computer-readable media <NUM> and executed on the processors <NUM>. The device applications <NUM> may include a device manager, such as any form of a control application, software application, signal-processing and control modules, code that is native to a particular device, an abstraction module, an air gesture recognition module, and other modules. The device applications <NUM> may also include system components, engines, modules, or managers to implement face authentication embeddings migration. The computing system <NUM> may also include, or have access to, one or more machine learning systems.

Claim 1:
A method of migrating face authentication profile data in a user device, the method comprising:
updating (<NUM>) the user device to include both a current version of firmware and an updated version of the firmware;
subsequent to the updating, receiving an indication of a face-authentication attempt and image data associated with a user's face;
loading both the current version of the firmware and the updated version of the firmware in a same secure session;
executing (<NUM>) the current version of the firmware to generate comparison data using the image data;
authenticating the user to unlock the user device based on the comparison data substantially matching a first profile, the first profile based on a set of enrolled embeddings that were previously generated using the current version of the firmware on previously-captured image data of the user's face;
responsive to authenticating the user to unlock the user device, generating (<NUM>) one or more new embeddings by executing the updated version of the firmware using the image data associated with the user's face;
storing (<NUM>) the one or more new embeddings at a separate location from the set of enrolled embeddings, the one or more new embeddings stored as part of a second profile for the user; and
generating additional new embeddings by repeatedly executing the updated version of the firmware over a series of subsequent successful face-authentication attempts until a sufficient amount of the new embeddings are generated and stored to enable the second profile to be usable to authenticate the user.