Patent Description:
<CIT> discloses a computing device which obtains a source image depicting a facial region having one or more makeup effects. The computing device performs facial alignment and defines a plurality of source regions having the one or more makeup effects, the source regions corresponding to facial features in the source image. The computing device extracts attributes of the one or more makeup effects for each source region and identifies a closest matching feature template for each source region based on the attributes. The computing device obtains a digital image of a facial region of a user. The computing device performs facial alignment and identifies a plurality of target regions corresponding to the plurality of source regions. The computing device applies a matching feature template of a corresponding source region to each of the target regions.

The matter for protection is set out in the appended claims.

The present disclosure is directed to techniques for identifying a digital representation of a face of a user in an image and applying digital makeup enhancements to the digital representation of the face based on previously saved enhancement settings for that particular user, which may be stored, e.g., locally on a computing device after previously receiving consent from the user. Thus, the computing device may, with user consent, store customized digital makeup enhancements for various different users of the device, and may also potentially store different groups of customized enhancements for each individual user. The device may then be configured to identify a digital representation of a face within an image (e.g., within a real-time image or video, within a previously saved image included in a photo album), and to apply the previously saved digital makeup enhancements to the digital representation of the face of the user in order to generate a modified image, where the modified image indicates corresponding makeup enhancements to the digital representation of the face.

In one example, a method according to claim <NUM> is provided.

In further examples, a computing device according to claim <NUM> and a computer-readable storage device according to claim <NUM> are provided.

<FIG> is a conceptual diagram illustrating an example computing device <NUM> that is configured to apply digital makeup enhancement data to facial data of a recognized digital representation of a face <NUM> of a user that is included in one or more digital images, such as a digital image <NUM>, in accordance with one or more aspects of the present disclosure. Examples of computing device <NUM> may include, but are not limited to, a mobile phone, a tablet computer, a personal digital assistant (PDA), a laptop computer, a portable gaming device, a portable media player, a wearable computing device (e.g., a watch, a wrist-mounted computing device, a head-mounted computing device), a television platform, or other type of computing device. As will be described in further detail below, computing device <NUM> may be or include one or more processors. Computing device <NUM> includes a display device <NUM> and one or more camera devices <NUM>, and computing device may be configured to execute one or more applications <NUM>.

As shown in <FIG>, display device <NUM> (e.g., a presence-sensitive display device) is configured to output one or more images for display, such as image <NUM> that includes a digital representation of a face <NUM> of a user. Face <NUM> may comprise a graphical or digital representation of the real-life face of the user. Throughout this disclosure, for ease of explanation, any such digital representation of a face, such as face <NUM>, may be referred to herein simply as a "face. " In various instances, the user may wish to use computing device <NUM> to apply one or more digital filters to alter the appearance of face <NUM> displayed in image <NUM>. For instance, computing device <NUM> may execute software to apply one or more digital makeup enhancements (e.g., virtual makeup) to face <NUM>. Certain existing applications, such as face-editing tools, include one or more filters that may be used to apply this type of virtual makeup to face <NUM>.

For example, if the user wishes to use camera device <NUM> to take a photo of the user (e.g., a selfie), the user may utilize a traditional face-editing tool to select a particular filter to obtain a real-time preview of the filter as it is applied to face <NUM> at image capture time. In this example, the selected filter may include a combination of one or more digital makeup enhancements (e.g., digital enhancements for the eyes and cheeks). However, this traditional filter will typically be applied to all faces that are included in image <NUM>. Although only one such face <NUM> is shown in the example of <FIG>, image <NUM> may include any number of one or more faces. The selected filter will often be applied to all of the detected faces in image <NUM>. Thus, when camera device <NUM> is activated to take a snapshot, the selected filter will be applied to all of the faces in the captured digital photo. However, this result does not allow customization of digital makeup application on a face-by-face basis.

The user may also use traditional face-editing tools to modify images within previously captured digital photos. In these cases, the user may select a particular image that is to be edited, as well as a particular face included in the image. The user may then select the digital makeup enhancements that are to be applied to the face, and may even adjust the strengths or other related settings of the digital makeup enhancements that are applied. However, the user must repeat these actions for each individual face in the image for which digital makeup is applied in order to selectively apply different or customized digital enhancements on a face-by-face basis.

With these types of traditional face-editing tools, therefore, it is apparent that users may not have the flexibility of allowing a computing device to automatically apply different digital makeup enhancements on different faces during image capturing time (e.g., when taking photos or selfies with camera device <NUM>), and users may need to manually repeat various steps to apply customized digital makeup enhancements to different faces that are included within previously captured images. However, by instead utilizing one or more techniques of the present disclosure, computing device <NUM> may use a digital makeup enhancement module <NUM> to create, save, and later automatically apply (e.g., for a live, real-time preview of) different digital makeup enhancements for different recognized user faces, including three-dimensional makeup enhancements. Digital makeup enhancement module <NUM> may comprise a stand-alone application or may be integrated with one or more of applications <NUM> (e.g., with a camera application and/or a photo album application).

For example, one or more techniques of the present disclosure utilize a more simplified approach for enabling computing device <NUM> to apply digital makeup enhancements (e.g., two-dimensional or three-dimensional enhancements) to faces that are included in digital images. As will be described in further detail below, digital makeup enhancement module <NUM> may first perform various setup operations to create and store digital makeup enhancement data for each individual enrolled user, where the data may be stored based on user consent. For example, during execution of digital makeup enhancement module <NUM>, a user may select the enrolling user's face included in an image (e.g., a real-time image output at display device <NUM>, a previously stored image that is included in digital images <NUM> stored either locally or remotely in the cloud). The facial data (e.g., feature data) for this selected face may, with a user's knowledge and consent, be stored in reference facial data <NUM>, locally on computing device <NUM> or, in some alternate examples, in the cloud (which is another way of referring to one or more external servers that are communicatively coupled to computing device <NUM> via one or more networks).

The user may select one or more digital makeup enhancements that are to be applied to the face, and the user may, in various cases, view a live, real-time preview of these enhancements as applied to the face. These enhancements are stored within digital makeup enhancement data <NUM>, either locally or in the cloud. This setup may, in various cases, be performed once for each individual face that is to be customized with digital makeup enhancements. In some cases, as will be described in more detail below, a user may even store multiple different enhancement settings that are associated with the same enrolled user (e.g., first makeup enhancement data associated with personal use, second makeup enhancement data associated with business use, etc.). The storage of user data such as facial data, image data, and/or digital makeup enhancement data associated with a user may only occur, in various examples, in response to computing device <NUM> (e.g., via UI module <NUM>) receiving an affirmative consent from the user. UI module <NUM> may, for example, provide an explicit request to the user regarding the storage and subsequent use of such data, and computing device <NUM> may proceed to store the data either locally or in the cloud only upon receiving (e.g., via module <NUM>) an affirmative response and consent from the user.

After computing device <NUM> saves digital makeup enhancement data <NUM>, either locally on computing device <NUM> (e.g., digital makeup enhancement data <NUM>) or remotely in the cloud, computing device <NUM> may subsequently use or execute digital makeup enhancement module <NUM> (e.g., while using camera devices <NUM> or while accessing stored images in a photo album) to apply, in response to a user request, customized digital makeup enhancements stored in the digital makeup enhancement data to one or more recognized faces that have different associated enhancement data. For example, while using camera devices <NUM>, computing device <NUM> may output, for display (e.g., at display device <NUM>), one or more real-time images captured by camera devices <NUM>. Digital makeup enhancement module <NUM> may then apply digital makeup enhancement data <NUM> to the facial data of the digital representation of the face of the user to generate the one or more modified digital images, where the one or more modified digital images comprise one or more modified real-time images. Computing device <NUM> may then output, for display (e.g., at display device <NUM>), the one or more modified real-time images to provide a live preview of the at least one corresponding digital makeup enhancement to the digital representation of the face of the user.

As a result, computing device <NUM> may store customized digital makeup enhancements for various different enrolled users of computing device <NUM>, and may also potentially store different groups of customized enhancements for each individual user. In the example of <FIG>, digital makeup enhancement module <NUM> has applied data for four different digital makeup enhancements to the digital representation of face <NUM>*;: a first digital makeup enhancement <NUM> to one cheek of the user, a second digital makeup enhancement <NUM> to another cheek of the user, a third digital makeup enhancement <NUM> underneath one eye of the user, and a fourth digital makeup enhancement <NUM> underneath another eye of the user.

In some examples, users of the same computing device or of different computing devices may share saved enhancement settings, which may be associated with one or more of these users. This makeup sharing feature may be performed by, e.g., applications <NUM> and/or digital makeup enhancement module <NUM> via one or more sharing operations. Using such a feature, any user of computing device <NUM><NUM> may share the user's digital makeup enhancement data <NUM> for use on any other device (e.g., another device external to but communicatively coupled to computing device <NUM>, such as via one or more wired and/or wireless networks). As a result, in these cases, digital makeup enhancement data <NUM> of a user of computing device <NUM> may be shared, stored, and used on another external device and applied to any live or stored image that includes this user's face. In addition, any user of computing device <NUM> may also request receipt of digital makeup enhancement data that is stored on another device and associated with another user, such that this data may also be stored on computing device <NUM> within digital makeup enhancement data <NUM> and used in applying corresponding makeup enhancements to any live or stored image that includes the face of this other user.

For example, if a "user A" has stored makeup enhancement settings for "user A" in digital makeup enhancement data <NUM>, "user A" may use one or more sharing operations to share these customized makeup enhancement settings with a friend, "user B," who uses another device, so that "user B" may use this other device to apply these settings for any images (e.g., live or stored images) that may include one or more images of "user A. " To do so, "user B" may use the other device to send, to computing device <NUM>, a share request for the customized makeup enhancement settings of "user A" that are stored in digital makeup enhancement data <NUM>. Upon receiving this request, computing device <NUM> (e.g., using UI module <NUM>) may receive from "user A" a consent or approval to share these settings with "user B. " Computing device <NUM> may then send to the device of "user B" (e.g., via one or more wireless or wired connections) the digital makeup enhancement data <NUM> for "user A. " In some cases, computing device <NUM> may send this data upon receiving approval from "user A," whether or not computing device <NUM> receives an explicit request from the device of "user B.

When the device of "user B" receives the makeup settings for "user A," this device may store and associate these settings with "user A" (e.g., based on a unique user identifier for "user A"). At this point, the device of "user B" may apply the makeup enhancement settings for "user A" to any image (e.g., a live image, a stored photo-album image) that includes "user A" to obtain images with customized makeup that is applied to the face of "user A," based on the specific digital makeup enhancement data that was previously selected and saved by "user A" on computing device <NUM>. In some cases, if "user B" likes or wants to use these particular settings on the face of "user B," "user B" may apply and store these same settings on the device in association with the face of "user B," such that these customized settings are also applied to the face of "user B" that may be included in any image (e.g., a live image, a stored photo-album image).

Similarly, "user A" of computing device <NUM> may send a share request to the device of "user B" to obtain a copy of the customized makeup settings of "user B" that are stored on the device of "user B. " Upon "user B" approving the request, the device of "user B" may send these customized settings to computing device <NUM>. Computing device <NUM> may store and associate these settings with "user B" (e.g., based on a unique user identifier for "user B"). Digital makeup enhancement module <NUM> may apply the makeup enhancement settings for "user B" to any image (e.g., a live image captured by camera devices <NUM>, a stored photo-album image) that includes "user B" to obtain images with customized makeup that is applied to the face of "user B," based on the specific digital makeup enhancement data that was previously selected and saved by "user B. " In addition, if "user A" likes or wants to use these particular settings, "user A" may apply and store these same settings in digital makeup enhancement data <NUM> in association with the face of "user A," such that these customized settings are also applied to the face of "user A" that may be included in any image.

In certain cases, various users may use or be associated with the same device. For instance, "user A" and "user B" may both be users of the same computing device <NUM>. In these cases, "user A" and "user B" may each include respective customized makeup data in digital makeup enhancement data <NUM> that is saved on computing device <NUM>. "User A" and/or "user B" may share their respective data with each other and/or with any other user for applying respective makeup enhancements in any images that include the face of "user A" or "user B. " In addition, in certain cases, a first user may allow another user, based on received input, to generate or customize the makeup settings for the first user, such that these settings are then applied to any image of the first user. For instance, "user A" may allow or enable "user B" via a sharing operation to create or modify, based on input from "user B", any customized makeup settings in digital makeup enhancement data <NUM> for "user A. " In this case, these generated makeup settings for "user A" may be applied to any image (e.g., live image, stored image) that includes the face of "user A. " In such fashion, any given user is enabled to share that user's customized makeup enhancement setting data with one or more other users (e.g., friends), which may be applied to the face of that given user when included in an image, or which may in some cases be applied to a face of any friend of the given user.

Further to the descriptions above. a user may be provided with controls allowing the user to make an election as to both if and when systems, programs, or features described herein may enable collection and/storage of user information (e.g., information about a user's social network, social actions, facial and/or image data, digital makeup enhancement data. user preference data, user location data), and/or if and when systems, programs, or features described herein may enable transmission of content or communications between devices In addition. certain data may be treated in one or more ways before it is stored or used, so that identifiable information is removed. For example. a user's identity may be treated so that no identifiable information can be determined for the user. or a user's geographic location may be generalized where location information is obtained (such as to a city, ZIP code, or state level), so that a particular location of a user cannot be determined. Thus, the user may have control over what information is collected about the user. how that information is stored and/or used, and what infomtation is provided to the user.

As shown in <FIG>. computing device <NUM> includes display device <NUM>. Display device <NUM> may function as an input device and/or an output device for computing device <NUM>. Display device <NUM> may be implemented using various technologies. For instance. display device <NUM> may function as an input device using a presence-sensitive input device, such as a resistive touchscreen, a surface acoustic wave touchscreen, a capacitive touchscreen, a projective capacitance touchscreen, a presence-sensitive screen, an acoustic pulse recognition touchscreen, a presence-sensitive screen that detects motion via radar technology, or another presence-sensitive technology. Display device <NUM> may function as an output device using any of one or more display devices, such as a liquid crystal display (LCD), dot matrix display, light emitting diode (LED) display, organic light-emitting diode (OLED) display, e-ink, or similar monochrome or color display capable of outputting visible information to a user of computing device <NUM>.

As one example, display device <NUM> of computing device <NUM> may include a presence-sensitive screen that may receive tactile input from a user of computing device <NUM>. Display device <NUM> may receive indications of the tactile input by detecting one or more gestures from a user of computing device <NUM> (e.g., the user touching or pointing to one or more locations of display device <NUM> with a finger or a stylus pen). The presence-sensitive screen of display device <NUM> may also present output to a user. Display device <NUM> may present the output as or in a graphical user interface, which may be associated with functionality provided by computing device <NUM>. For example, display device <NUM> may present output associated with various user interfaces of applications <NUM> executing at computing device <NUM>. A user may interact with a respective user interface of each of applications <NUM> to cause computing device <NUM> to perform operations relating to corresponding application functionality.

In some examples, computing device <NUM> may include one or more communication units (such as shown in <FIG>). These communication units may send data to and/or receive data from one or more other computing devices. In some examples, the communication units support wireless and/or wired communication, and they may send and/or receive data using any variety of communication protocols.

Computing device <NUM> may also include user interface ("UI") module <NUM> and digital makeup enhancement module <NUM>. UI module <NUM>, applications <NUM>, and digital makeup enhancement module <NUM> may perform operations described herein using any combination of software, hardware, and/or firm ware residing in and/or executing at computing device <NUM>. Computing device <NUM> may execute modules <NUM>, <NUM> and applications <NUM> using one or more processors. Computing device <NUM> may, in some cases, execute modules <NUM>, <NUM> and applications <NUM> as one or more virtual machines executing on underlying hardware. Modules <NUM>, <NUM> and applications <NUM> may be implemented in various ways. For example, any of modules <NUM>, <NUM> and/or applications <NUM> may be implemented as a downloadable or pre-installed application or "app. " In some examples, one of more of these may execute as a service of an operating system or computing platform.

Applications <NUM> of computing device <NUM> may perform various functions or access one or more services for computing device <NUM>. An e-mail application, a camera application, a calendar application, a messaging application, a social media application, a travel application, a game application, a stock application, and a weather application are all examples of applications <NUM>. UI module <NUM> may cause display device <NUM> to present a graphical user interface to a user. For example, the graphical user interface may include graphical elements (e.g., indications) displayed at various locations of display device <NUM>.

UI module <NUM> may, in some cases, act as an intermediary between various components of computing device <NUM> to make determinations based on input detected by display device <NUM> and to generate output presented by display device <NUM>. For instance, UI module <NUM> may receive information from display device <NUM> related to input detected by display device <NUM> and transmit the input information to one or more of applications <NUM>. UI module <NUM> may also receive notification information and/or content from digital makeup enhancement module <NUM> and/or a camera application (e.g., one of applications <NUM>). Such a camera application may control or otherwise interact with or manage camera devices <NUM>. UI module <NUM> may provide content and/or other information associated with digital makeup enhancement module <NUM> or the camera application (e.g., digital image <NUM>) to display device <NUM> for output to a user.

Computing device <NUM> further includes one or more camera devices <NUM>. Camera devices <NUM> are configured to capture one or more images during execution of a camera application (e.g., one of applications <NUM>), such as one or more still and/or moving images. In cases in which computing device <NUM> comprises a mobile computing device, camera devices <NUM> may include one or more camera devices on a front-side and/or on a rear-side of computing device <NUM>. Examples of camera devices <NUM> are further described below in reference to <FIG>.

According to one or more examples, and as further described below, computing device <NUM> may execute digital makeup enhancement module <NUM> to implement various techniques of the present disclosure. In some cases, digital makeup enhancement module <NUM> may comprise a standalone module that interacts with UI module <NUM> and/or applications <NUM>. In other cases, digital makeup enhancement module <NUM> may be part of or integrated with UI module <NUM> or one or applications <NUM>, such as a camera application.

Digital makeup enhancement module <NUM> may output, for display, one or more digital images (e.g., one or more still or moving images), which include a digital representation of face <NUM> of a user. As indicated earlier, face <NUM> may comprise a digital representation of a real-life face of the user, and thus, throughout this disclosure and for ease of explanation, any such digital representation of a face, such as face <NUM>, may be referred to herein simply as a "face. " In some cases, computing device <NUM> may use camera devices <NUM> to capture these images in real time. In other cases, computing device <NUM> may retrieve the digital images from a stored digital photo album, such as from digital images <NUM> shown in <FIG>. One or more of digital images <NUM> may optionally be stored locally on computing device <NUM> or may be stored remotely in the cloud.

Digital makeup enhancement module <NUM> may receive, based on a facial recognition process (e.g., a process performed locally on computing device <NUM> or externally in the cloud), an indication of a match between facial data associated with face <NUM> of the user and reference facial data (e.g., reference facial data <NUM>) associated with a face of an enrolled user, where the reference facial data is associated with digital makeup enhancement data previously stored for the enrolled user. This enrolled user is one who was previously enrolled on computing device <NUM>, where computing device <NUM> has previously obtained digital makeup enhancement data associated with the reference facial data of this enrolled user, such that the digital makeup enhancement data has been previously stored either locally on computing device <NUM> (e.g., in digital makeup enhancement data <NUM>) or remotely in the cloud. For instance, the user in the example of <FIG> may have previously enrolled his/her face with computing device <NUM>. The reference facial data associated with the face of this enrolled user may be stored in reference facial data <NUM> and may be associated with the stored digital makeup enhancement data. In some optional cases, reference facial data <NUM> may be stored locally on computing device <NUM>. In other cases, reference facial data <NUM> may be stored external from computing device <NUM> in the cloud.

As noted above, in some cases, computing device <NUM> (e.g., using one or more of applications <NUM> and/or digital makeup enhancement module <NUM>) may perform one or more facial recognition processes locally on computing device <NUM>. In these cases, computing device <NUM> may perform the facial recognition process on one or more digital images and retrieve reference facial data <NUM> associated with the face of the enrolled user. Computing device <NUM> may then compare the facial data associated with face <NUM> of the user to the reference facial data associated with the face of the enrolled user. Computing device <NUM> may then provide, based on the comparing, the indication of the match between the facial data associated with face <NUM> of the user and the reference facial data associated with the face of the enrolled user.

However, in other examples, one or more of these facial recognition processes may be performed remotely from computing device <NUM> in the cloud. For instance, computing device <NUM> may send, to an external device, infomation associated with the one or more digital images. Computing device <NUM> may then receive from the external device, based on the facial recognition process performed by the external device, the indication of the match between the facial data associated with face <NUM> of the user and the reference facial data associated with the face of the enrolled user.

After receiving the indication of the match between the facial data associated with face <NUM> of the user and reference facial data associated with the face of the enrolled user, digital makeup enhancement module <NUM> may retrieve digital makeup enhancement data that is associated with the reference facial data. This data may be included in digital makeup enhancement data <NUM> shown in <FIG>. In some optional cases, digital makeup enhancement data <NUM> may be stored locally on computing device <NUM>. In other cases, digital makeup enhancement data <NUM> may be stored externally from computing device <NUM> in the cloud.

After retrieval of this data, digital makeup enhancement module <NUM> may apply digital makeup enhancement data <NUM> to the facial data of face <NUM> of the user to generate one or more modified digital images, such as image <NUM> shown in <FIG>, which indicate at least one corresponding digital makeup enhancement to face <NUM> of the user. Digital makeup enhancement module <NUM> may then output modified digital image <NUM> for display to the user. Any of the digital images output by digital makeup enhancement module <NUM> may comprise one or more still or moving images.

In various examples, digital makeup enhancement module <NUM> may use digital makeup enhancement data <NUM> to apply two-dimensional or three-dimensional makeup enhancements to face <NUM>. For example, in some cases, digital makeup enhancement module <NUM> may apply a three-dimensional makeup rendering process with digital makeup enhancement data <NUM> to apply or provide three-dimensional makeup enhancements to face <NUM> when generating modified digital image <NUM>. In these cases, camera devices <NUM> may or may not include a depth sensor. For instance, in examples where camera devices <NUM> do not include a depth sensor, the three-dimensional makeup rendering process may detect one or more facial landmarks in face <NUM> and construct a three-dimensional facial model, including a depth map model of facial information or related features, based on a source (e.g., red-green-blue) image. In various examples, digital makeup enhancement module <NUM> may use the three-dimensional makeup rendering process to apply makeup enhancements via a blending process with respect to the pixel values (e.g., pixel values for color, brightness) of the image(s). Thus, the rendering process may not necessarily make face <NUM> appear flat or remove any particular effects (e.g., depth effects, shadowing effects), but may instead take both the shape of face <NUM> and any effects such as depth or shadowing into account when performing the three-dimensional rendering process and applying the makeup enhancements to face <NUM>. The rendering process use by digital makeup enhancement module <NUM> is therefore capable, in various examples, of applying three-dimensional makeup enhancements or effects to face <NUM> using digital makeup enhancement data <NUM>.

Various different digital makeup enhancements may be saved and subsequently applied to face <NUM>, and the data associated with these enhancements may be stored in digital makeup enhancement data <NUM>. For instance, in the example of <FIG>, four distinct digital makeup enhancements have been applied to face <NUM>: a first digital makeup enhancement <NUM> to one cheek of the user, a second digital makeup enhancement <NUM> to another cheek of the user, a third digital makeup enhancement <NUM> underneath one eye of the user, and a fourth digital makeup enhancement <NUM> underneath another eye of the user. Each of these digital makeup enhancements may be represented or otherwise associated with corresponding data that is included in digital makeup enhancement data <NUM>, which may indicate the type of makeup enhancement and corresponding attributes.

As one example, the first digital makeup enhancement <NUM> to a cheek of the user may comprise a foundation makeup enhancement of a particular color. The data that corresponds to this enhancement, as stored in digital makeup enhancement data <NUM>, may indicate that the enhancement in a foundation makeup enhancement, and may further specify the attributes of this enhancement, such as the color, style, strength, and like (e.g., such as provided in one or more name/value pairs).

The stored digital makeup enhancement data corresponding to first digital makeup enhancement <NUM> may also be associated specifically with the face of the enrolled user (e.g., via a unique identifier of the face of this enrolled user), as well as particular reference facial data included in reference facial data <NUM> for this first digital makeup enhancement <NUM>. For instance, first digital makeup enhancement <NUM> may be associated with reference facial data (e.g., one or more facial features) for one particular cheek of the enrolled user. The reference facial data may indicate one or more specific facial information or features, including facial location or region information (e.g., in one or more name/value pairs) associated with such features.

After receiving an indication that face <NUM> matches the face of the enrolled user, digital makeup enhancement module <NUM> may retrieve first digital makeup enhancement <NUM> for the enrolled user and identify a portion of face <NUM> that is associated with facial data corresponding to the reference facial data (e.g., one particular cheek) of first digital makeup enhancement <NUM>. Digital makeup enhancement module <NUM> may then apply the data for first digital makeup enhancement <NUM> to this facial data of face <NUM> to apply this corresponding digital makeup enhancement <NUM> (e.g., foundation applied to the cheek of user <NUM>).

Thus, the present disclosure is directed to techniques for identifying face <NUM> of a user in an image and applying digital makeup enhancements to face <NUM> based on previously saved digital makeup enhancement data <NUM> (e.g., setting data) for that particular user. Digital makeup enhancement module <NUM> may access customized digital makeup enhancements for various different users of computing device <NUM>, and may also potentially access different groups of customized enhancements for each individual user, as will be described in further detail below. One or more techniques of the present disclosure may utilize a more simplified approach for enabling computing device <NUM> to automatically apply digital makeup enhancements to recognized faces that are included in digital images.

<FIG> is a block diagram illustrating an example computing device <NUM>, in accordance with one or more aspects of the present disclosure. Computing device <NUM> may comprise one example of computing device <NUM> illustrated in <FIG>. <FIG> illustrates only one particular example of computing device <NUM>, and many other examples of computing device <NUM> may be used in other instances and may include a subset of the components included in example computing device <NUM> or may include additional components not shown in <FIG>.

In the example of <FIG>, computing device <NUM> includes presence-sensitive display device <NUM>, one or more processors <NUM>, one or more input components <NUM>, one or more communication units <NUM>, one or more output components <NUM>, one or more camera devices <NUM>, one or more sensors <NUM>, a power source <NUM>, and one or more storage devices <NUM>. Communication channels <NUM> may interconnect each of the components <NUM>, <NUM>, <NUM>, <NUM>, <NUM>, <NUM>, <NUM>, <NUM>, and/or <NUM> for inter-component communications (physically, communicatively, and/or operatively). In some examples, communication channels <NUM> may include a system bus, a network connection, an inter-process communication data structure, or any other method for communicating data between hardware and/or software.

One or more input components <NUM> of computing device <NUM> may receive input, such as input from a user. Examples of input are touch/tactile, presence-sensitive, and audio input. Examples of input components <NUM> include a presence-sensitive screen, touch-sensitive screen, touchscreen, mouse, keyboard, trackpad, voice responsive system, video camera, microphone or any other type of device for detecting input from a human or machine.

One or more output components <NUM> of computing device <NUM> may generate output. Examples of output are haptic, audio, and visual output. Examples of output components <NUM> include a presence-sensitive screen, touch-sensitive screen, touchscreen, sound card, video graphics adapter card, speaker, cathode ray tube (CRT) display, liquid crystal display (LCD), haptic device, or any other type of device for generating output to a human or machine.

One or more communication units <NUM> of computing device <NUM> may communicate with external devices via one or more networks by transmitting and/or receiving network signals on the one or more networks (e.g., one or more wired and/or wireless networks). For example, computing device <NUM> may use communication units <NUM> to transmit and/or receive radio signals on a radio network such as a cellular radio network. Likewise, communication units <NUM> may transmit and/or receive satellite signals on a satellite network such as a global positioning system (GPS) network. Examples of communication units <NUM> include a network interface card (e.g. such as an Ethernet card), an optical transceiver, a radio frequency transceiver, a GPS receiver, or any other type of device that can send and/or receive information. Other examples of communication units <NUM> may include short wave radios, cellular data radios, wireless Ethernet network radios, as well as universal serial bus (USB) controllers.

Presence-sensitive display device <NUM> of computing device <NUM> includes display component <NUM> and presence-sensitive input component <NUM>. In some examples, presence-sensitive display device <NUM> may provide output to a user using haptic, audio, or visual stimuli as described above with reference to output components <NUM>. For example, display component <NUM> may provide display or video output as described with reference to output components <NUM>. Presence-sensitive display device <NUM> may also provide input capabilities such as that described above with reference to input components <NUM>. For example, presence-sensitive input component <NUM> may provide input capabilities as described with reference to input components <NUM>.

Display component <NUM> may be a screen at which information is displayed by presence-sensitive display device <NUM>, and presence-sensitive input component <NUM> may detect an object at and/or near display component <NUM>. As one example range, presence-sensitive input component <NUM> may detect an object, such as a finger or stylus that is within two inches or less of display component <NUM>. Presence-sensitive input component <NUM> may determine a location (e.g., an (x,y) coordinate) of display component <NUM> at which the object was detected. In another example range, presence-sensitive input component <NUM> may detect an object six inches or less from display component <NUM> and other ranges are also possible. Presence-sensitive input component <NUM> may determine the location of display component <NUM> selected by a user's finger using capacitive, inductive, radar-based, and/or optical recognition techniques. In some examples, presence sensitive input component <NUM> also provides output to a user using touch, presence-sensitive, audio, or video stimuli as described with respect to display component <NUM>. Display component <NUM> may be any type of output device that provides visual output, such as described with respect to output components <NUM>.

While illustrated as an internal component of computing device <NUM>, presence-sensitive display device <NUM> may also represent an external component that shares a data path with computing device <NUM> for transmitting and/or receiving input and output. For instance, in one example, presence-sensitive display device <NUM> represents a built-in component of computing device <NUM> located within and physically connected to the external packaging of computing device <NUM> (e.g., a screen on a mobile phone). In another example, presence-sensitive display device <NUM> represents an external component of computing device <NUM> located outside and physically separated from the packaging of computing device <NUM> (e.g., a monitor and/or a projector that shares a wired and/or wireless data path with a tablet computer).

Presence-sensitive display device <NUM> of computing device <NUM> may detect two-dimensional and/or three-dimensional gestures as input from a user of computing device <NUM>. For instance, a sensor of presence-sensitive display device <NUM> (e.g., sensor of presence-sensitive input component <NUM>) may detect a user's movement (e.g., moving a hand, an arm, a pen, a stylus) within a threshold distance of the sensor of presence-sensitive display device <NUM>. Presence-sensitive display device <NUM> may determine a two- or three-dimensional vector representation of the movement and correlate the vector representation to a gesture input (e.g., a hand-wave, a pinch, a clap, a pen stroke) that has multiple dimensions. In other words, presence-sensitive display device <NUM> can detect a multi-dimensional gesture without requiring the user to gesture at or near a screen or surface (e.g., display component <NUM>) at which presence-sensitive display device <NUM> outputs information for display. Instead, presence-sensitive display device <NUM> can detect a multi-dimensional gesture performed at or near a sensor which may or may not be located near the screen or surface at which presence-sensitive display device <NUM> outputs information for display.

One or more storage devices <NUM> within computing device <NUM> may store information for processing during operation of computing device <NUM> (e.g., during execution of one or more of UI module <NUM>, applications <NUM>, operating system <NUM>, or digital makeup enhancement module <NUM>). In some examples, storage devices <NUM> include temporary memory, meaning that a primary purpose of storage devices <NUM> is not long-term storage. Storage devices <NUM> on computing device <NUM> may be configured for short-tenn storage of information as volatile memory and therefore not retain stored contents if powered off. Examples of volatile memories include random access memories (RAM), dynamic random-access memories (DRAM), static random-access memories (SRAM), and other forms of volatile memories known in the art.

Storage devices <NUM>, in some examples, include one or more computer-readable storage media. Storage devices <NUM> may be configured to store larger amounts of information than volatile memory. Storage devices <NUM> may further be configured for long-term storage of information as non-volatile memory space and retain information after power on/off cycles. Examples of non-volatile memories include magnetic hard discs, optical discs, floppy discs, flash memories, or forms of electrically programmable memories (EPROM) or electrically erasable and programmable (EEPROM) memories. Storage devices <NUM> may store program instructions and/or data associated with one or more applications <NUM>, UI module <NUM>, operating system <NUM>, and digital makeup enhancement module <NUM>. As illustrated in <FIG>, storage devices <NUM> may optionally include reference facial data <NUM>, digital images <NUM>, and digital makeup enhancement data <NUM>, when these are stored locally on computing device <NUM>. In other cases, reference facial data <NUM>, digital images <NUM>, and/or digital makeup enhancement data <NUM> may be stored remotely in the cloud. UI module <NUM>, applications <NUM>, digital makeup enhancement module <NUM>, reference facial data <NUM>, digital images <NUM>, and digital makeup enhancement data <NUM> may comprise examples of corresponding UI module <NUM>, applications <NUM>, digital makeup enhancement module <NUM>, reference facial data <NUM>, digital images <NUM>, and digital makeup enhancement data <NUM> shown in <FIG>.

In certain examples, storage devices <NUM>, or one or more of components included in storage devices <NUM>, may be stored on one or more remote computing devices that are external to computing device <NUM> (e.g., on one or more external servers). In some examples, one or more remote computing devices may store and/or execute UI module <NUM>, applications <NUM>, and/or operating system <NUM>. In these examples, the one or more remote computing devices may perform functionality similar to that described herein in reference to processors <NUM>.

As shown in <FIG>, computing device <NUM> may include a power source <NUM>. In some examples, power source <NUM> may be a battery. Power source <NUM> may provide power to one or more components of computing device <NUM>. Non-limiting examples of power source <NUM> may include, but are not necessarily limited to, batteries having zinc-carbon, lead-acid, nickel cadmium (NiCd), nickel metal hydride (NiMH), lithium ion (Li-ion), and/or lithium ion polymer (Li-ion polymer) chemistries. In some examples, power source <NUM> may have a limited capacity (e.g., <NUM>-<NUM> mAh).

Computing device <NUM> also includes one or more sensors <NUM>. In some examples, one or more of sensors <NUM> may be examples of one or more of input components <NUM>. Sensors <NUM> may include one or more inertial measurement units. For example, sensors <NUM> may include one or more gyroscopes, such as one or more high-frequency gyroscopes (e.g., a <NUM> gyroscope). As described herein, sensors <NUM> of computing device <NUM> may be configured to determine a real-time orientation or rotation of computing device <NUM> in three-dimensional space. In some examples, sensors <NUM> may also include one or more accelerometers.

Computing device <NUM> further includes one or more camera devices <NUM>. Camera devices <NUM> may be one example of camera devices <NUM> shown in <FIG>. Camera devices <NUM> are configured to capture one or more images during execution of an application or module (e.g., a camera application, one or more of applications <NUM>, digital makeup enhancement module <NUM>), such as one or more still and/or moving images. Camera devices <NUM> may include one or more camera devices on a front-side and/or on a rear-side of computing device <NUM>.

Camera devices <NUM> may be one or more of any appropriate type of image acquisition or capture device, such as a camera or charge-coupled device. In some examples, camera devices <NUM> may include one or more infrared cameras with a high field-of-view and shallow depth of focus, and may include a backlit infrared camera having a particular field-of-view. In other examples, camera devices <NUM> may be or may further include one or more other types of cameras or image sensors, which may include one or more other infrared cameras, thermographic cameras, thermal imaging cameras, light-sensitive cameras, range sensors, tomography devices, radar devices, red-green-blue (RGB) cameras, or ultrasonic cameras. In some examples, camera devices <NUM> may include any image capture device appropriate for application of computer vision techniques. Depending on the type of camera devices used, the resulting images may include two-dimensional images, three-dimensional volumes, and/or an image sequence. Pixel values typically correspond to light intensity in one or more spectral bands, but might also be related to various physical measures, such as depth, absorption or reflectance of sonic or electromagnetic waves, or nuclear magnetic resonance.

One or more processors <NUM> may implement functionality and/or execute instructions within computing device <NUM>. For example, processors <NUM> on computing device <NUM> may receive and execute instructions stored by storage devices <NUM> that execute the functionality of applications <NUM>, operating system <NUM>, and/or camera application <NUM>. These instructions executed by processors <NUM> may cause computing device <NUM> to store information within storage devices <NUM> during program execution. Processors <NUM> may execute instructions of operating system <NUM> and applications <NUM> to perform one or more operations. That is, operating system <NUM> and applications <NUM> may be operable by processors <NUM> to perform various functions described herein.

In some alternate examples, computing device <NUM> may only comprise or otherwise include processors <NUM>. In these examples, input components <NUM>, presence-sensitive display device <NUM>, communication units <NUM>, output components <NUM>, power source <NUM>, and storage devices <NUM> may be external to, yet communicatively coupled with (e.g., via communication channels <NUM>), computing device <NUM>.

Applications <NUM> may include one or more different various applications. An e-mail application, a camera application, a map or navigation application, calendar application, a messaging application, a social media application, a travel application, a game application, a stock application, and a weather application are all examples of applications <NUM>.

<FIG> are screen diagrams illustrating examples of applying and storing digital makeup enhancement data with respect to reference facial data of a face of an enrolled user, in accordance with one or more aspects of the present disclosure. A computing device, such as computing device <NUM> or computing device <NUM>, may be configured to output digital images included in these screen diagrams for display. For purposes of illustration only, <FIG> are described in reference to computing device <NUM>.

UI module <NUM> may output one or more digital images <NUM> (<FIG>), <NUM> (<FIG>), and <NUM> (<FIG>) for display. In some cases, digital images <NUM>, <NUM>, <NUM> may be images that are captured by a camera application (e.g., one of applications <NUM>) in real time, and these may comprise one or more still or moving images. In other cases, digital images <NUM>, <NUM>, <NUM> may comprise stored images that are included, e.g., in a photo album (e.g., for stored in digital images <NUM>). Using techniques of the present disclosure, digital makeup enhancement module <NUM> may perform an initial setup of selecting and storing preferred digital makeup enhancement settings for one or more enrolled users of computing device <NUM>, which can later be applied to images of a recognized face of those enrolled users. The facial data (e.g., feature data) for any selected faces may be stored in reference facial data <NUM>, either locally on computing device <NUM> or in the cloud. Digital images <NUM>, <NUM> may comprise initial digital images used for enrollment, and digital image <NUM> may comprise a modified initial digital image based on application of selected digital makeup enhancements.

As shown in <FIG>, in one particular example, digital makeup enhancement module <NUM> may initiate the process of makeup selection for an enrolled user upon receipt of a user confirmation, such as a user selection of a graphical object <NUM> that corresponds to a particular makeup or portrait mode. In some cases, user selection of graphical object <NUM> initiates a distinct makeup or portrait mode associated with execution of digital makeup enhancement module <NUM>. In other cases, such as when the functionality of digital makeup enhancement module <NUM> is included within that of other functions or modes (e.g., as an option under or within a face retouching mode), the makeup or portrait functionality may be integrated as such without having its own, separate mode.

After receiving a user selection of a graphical object <NUM>, digital makeup enhancement module <NUM> may output a graphical boundary <NUM> around a detected face in initial digital image <NUM>. A facial detection process may be executed, either locally on computing device <NUM> or remotely in the cloud, to detect a face within digital image <NUM> based on one or more detected facial features. Digital makeup enhancement module <NUM> may select the detected face for further processing based upon a selection (e.g., user selection using input components <NUM> and/or presence-sensitive display device <NUM>) of graphical boundary <NUM>.

Based on user feedback from the enrolled user, digital makeup enhancement module <NUM> may then select one or more digital makeup enhancements that are to be applied to the detected face of the enrolled user. In doing so, digital makeup enhancement module <NUM> may output one or more graphical makeup bars (e.g., graphical makeup bars <NUM>, <NUM>), which each include one or more makeup objects. Digital makeup enhancement module <NUM> may apply one or more digital makeup enhancements to the face of the enrolled user based on user selection of the makeup objects included in these graphical makeup bars. These enhancements are stored within digital makeup enhancement data <NUM>, either locally or in the cloud. This setup may, in various cases, be performed once for each individual face of an enrolled user that is to be customized with digital makeup enhancements. In some cases, as will be described in more detail below, a user may even store multiple different enhancement settings that are associated with the same enrolled user (e.g., first makeup enhancement data associated with personal use, second makeup enhancement data associated with business use).

Upon selection of graphical border <NUM>, digital makeup enhancement module <NUM> may interact with UI module <NUM> to output image <NUM>, which may in some cases include an enlarged version of the face of the enrolled user. At this stage, digital makeup enhancement module <NUM> may provide one or digital makeup enhancement to the face of the enrolled user based upon user selection of graphical makeup bars <NUM> and/or <NUM>.

In non-limiting cases, digital makeup enhancement data is associated with a digital makeup enhancement to the reference facial data of the enrolled user for one or more of an eyebrow, a cheek, an eyelash, a nose, lips, contour, or cheeks of the face of the user. In the particular example of <FIG>, graphical makeup bar <NUM> includes four distinct makeup objects associated with a lip makeup enhancement, a foundation makeup enhancement, a contour makeup enhancement, and an eyelash makeup enhancement (e.g., in left-to-right order in makeup bar <NUM>). In <FIG>, the user has selected the makeup object for the lip makeup enhancement. As a result, graphical makeup bar <NUM> includes six distinct makeup objects associated with different lip makeup enhancement options. These options may be associated with different types, styles, colors, strengths, etc. of lip makeup enhancements. The enrolled user may select any of the makeup objects in graphical makeup bar <NUM> and apply the associated makeup enhancements to corresponding portions of the face of the enrolled user displayed in image <NUM>. For example, the enrolled user may select one of the makeup objects included in graphical makeup bar <NUM> and apply this lip makeup enhancement to one or more portions of the lips of the face shown in <NUM>. These portions of the lips correspond to particular reference facial data of the face of the enrolled user, such as one or more facial features for these portions of the lips.

The data that corresponds to this enhancement, as stored in digital makeup enhancement data <NUM>, may indicate that the enhancement in a lip makeup enhancement, and may further specify the attributes of this enhancement, such as the color, style, strength, and like (e.g., such as provided in one or more name/value pairs). The stored digital makeup enhancement data corresponding to this digital makeup enhancement may also be associated specifically with the face of the enrolled user (e.g., via a unique identifier of the face of this enrolled user), as well as the particular reference facial data associated with the one or more portions of the lips, which may be included in reference facial data <NUM> for this digital makeup enhancement. For instance, the lips digital makeup enhancement may be associated with reference facial data (e.g., one or more facial features) for the lips of the enrolled user. The reference facial data may indicate one or more specific facial information or features, including facial location or region information (e.g., in one or more name/value pairs) associated with such features.

The enrolled user may use the makeup objects included in graphical makeup bar <NUM> to select and apply any number of different digital makeup enhancements to the face of the enrolled user displayed in image <NUM>. When the user selects a particular makeup object from makeup bar <NUM>, different respective options may be displayed in makeup bar <NUM>. For instance, if the user selects a makeup object from makeup bar <NUM> associated with eyelashes or with eyeliner, makeup bar <NUM> may include various makeup objects associated with eyelash or eyeliner enhancements. If the user selects a makeup object from makeup bar <NUM> associated with foundation, makeup bar <NUM> may include various makeup objects associated with foundation enhancements. Each of these digital makeup enhancements may be represented or otherwise associated with corresponding data that is included in digital makeup enhancement data <NUM>, which may indicate the type of makeup enhancement and corresponding attributes. Digital makeup enhancement module <NUM> may generate, based on the selected digital makeup enhancements, the corresponding digital makeup enhancement data for storage in digital makeup enhancement data <NUM>, and may associate this data with the corresponding reference facial data <NUM> to which the makeup enhancement data is applied.

Digital makeup enhancement module <NUM> may output the digital makeup enhancements that have been applied to the face of the enrolled user. For example, <FIG> illustrates an example digital image <NUM> in which multiple example digital makeup enhancements have been applied to the face of the enrolled user of <FIG>. Similar to the enhancements illustrated in <FIG>, four distinct digital makeup enhancements have been applied to the face of the enrolled user: a first digital makeup enhancement <NUM> (e.g., foundation enhancement) to one cheek of the user, a second digital makeup enhancement <NUM> (e.g., foundation enhancement) to another cheek of the user. a third digital makeup enhancement <NUM> (e.g., eyeliner enhancement) underneath one eye of the user, and a fourth digital makeup enhancement <NUM> (e.g., eyeliner enhancement) underneath another eye of the user. Each of these digital makeup enhancements may be represented or otherwise associated with corresponding data that is included in digital makeup enhancement data <NUM>, which may indicate the type of makeup enhancement and corresponding attributes, and which may also be associated with respective reference facial data (e.g., facial data associated with the user's cheeks and/or eyes) that is included in reference facial data <NUM> that is stored for the enrolled user.

In certain cases, an enrolled user may also store multiple different groups of digital makeup settings or profiles in digital makeup enhancement data <NUM>. For example, an enrolled user may choose to create and store different groups of digital makeup settings in digital makeup enhancement data <NUM> corresponding to different types of anticipated or desired uses. For instance, an enrolled user may wish to store a first group of enhancement settings that are associated with business or formal use. This same user, however, may also wish to store a second group of enhancement settings that are associated with personal use (e.g., for use at a party setting). Digital makeup enhancement data <NUM> may include any number of groups of digital makeup enhancement settings, one or more of which may be associated with the same enrolled user.

<FIG> is a screen diagram illustrating different portions of a digital image that correspond to different facial features of a user, in accordance with one or more aspects of the present disclosure. The example image illustrated in <FIG> may be one example of any of the digital images illustrated in <FIG> or <FIG>.

<FIG> illustrates multiple different regions or portions <NUM>, <NUM>, <NUM>, <NUM>, <NUM>, <NUM>, <NUM>, <NUM> of face <NUM> included in the example digital image. Each of these portions correspond to respective facial data of face <NUM>, such as respective facial features of face <NUM>. This facial data may comprise reference facial data for face <NUM> when face <NUM> is one of an enrolled user.

As an example, portion <NUM> illustrated in <FIG> may correspond to facial data for a first eyebrow of face <NUM>. This facial data may comprise or indicate one or more specific facial features (e.g., reference facial features for reference facial data of an enrolled user), including any facial location or region information (e.g., in one or more name/value pairs) on face <NUM> that are associated with such features, such as the first eyebrow of face <NUM>.

Similarly, portion <NUM> may correspond to facial data for a second eyebrow of face <NUM>. Portion <NUM> may correspond to facial data for a first eye of face <NUM>, while portion <NUM> may correspond to facial data for a second eye of face <NUM>. Portion <NUM> may correspond to facial data for a first cheek of face <NUM>, while portion <NUM> may correspond to facial data for a second check of face <NUM>. Portion <NUM> may correspond to facial data for a nose of face <NUM>, and portion <NUM> may correspond to a mouth of face <NUM>. Each of portions <NUM>-<NUM> may therefore correspond to respective facial data (e.g., feature data) with respect to face <NUM>.

When digital makeup enhancement module <NUM> stores digital makeup enhancement data <NUM> and reference facial data <NUM> (e.g., reference facial features) for an enrolled user, such as described above in reference to <FIG>, digital makeup enhancement module <NUM> may identify the portions of the face of the enrolled user (e.g., portions <NUM>-<NUM> shown in <FIG>) to which the digital makeup enhancements are applied. Digital makeup enhancement module <NUM> may store the digital makeup enhancement data (e.g., settings) <NUM> for these enhancements in association with reference facial data <NUM> that corresponds to the portions of the face of the enrolled user to which the enhancements are applied.

At a later point in time, as will be described further in reference to <FIG>, the same user may wish to use computing device <NUM> to apply saved digital makeup enhancement data <NUM> to the face of the user included in a new digital image (e.g., a real-time image captured by camera devices <NUM>, a stored digital image included in digital images <NUM>). Digital makeup enhancement module <NUM> may apply digital makeup enhancement data <NUM> that was previously stored for this enrolled user, and apply it to the facial data of the face of the user within one or more modified digital images. In some examples, while using camera devices <NUM>, computing device <NUM> may output, for display (e.g., at display device <NUM>), one or more real-time images captured by camera devices <NUM>. Digital makeup enhancement module <NUM> may then apply digital makeup enhancement data <NUM> to the facial data of the digital representation of the face of the user to generate the one or more modified digital images, where the one or more modified digital images comprise one or more modified real-time images. Computing device <NUM> may then output, for display (e.g., at display device <NUM>), the one or more modified real-time images to provide a live preview of the at least one corresponding digital makeup enhancement to the digital representation of the face of the user.

In doing so, after the face of the user has been recognized as the enrolled face (e.g., by one or more facial recognition processes that match the facial data associated with the face and the reference facial data associated with the face of the enrolled user), digital makeup enhancement module <NUM> may access the digital makeup enhancement data <NUM> that corresponds to the face of the enrolled user (e.g., via a unique identifier associated with the face of the enrolled user).

This retrieved digital makeup enhancement data <NUM> is associated with corresponding reference facial data <NUM> for the face of the enrolled user. Digital makeup enhancement module <NUM> may then identify portions of the face within the new digital image having facial data (e.g., one or more facial features) that correspond to reference facial data <NUM> (e.g., one or more reference facial features) of the enrolled user. Digital makeup enhancement module <NUM> may then apply the retrieved digital makeup enhancement data (e.g., settings) <NUM> that is associated with this reference facial data <NUM> to the facial data of the face included in the new image, thereby applying corresponding digital makeup enhancements to this face.

<FIG> are screen diagrams illustrating an example of a selective application of stored digital makeup enhancement data to facial data of a recognized face of a user that is included in one or more digital images, in accordance with one or more aspects of the present disclosure. For purposes of illustration only, the examples shown in <FIG> are described in reference to computing device <NUM> shown in <FIG>.

As shown in <FIG>, a digital image includes two faces: face <NUM> of a first user and face <NUM> of a second user. Face <NUM> and face <NUM> may be included in respective first and second portions of the displayed image. In some cases, this digital image may comprise a real-time image that is displayed at presence-sensitive display device <NUM> based on input provided by camera devices <NUM> (e.g., during execution of a camera application included in applications <NUM>). In other cases, the digital image may comprise a stored image included in digital images <NUM> (e.g., an image included in a stored photo album).

Digital makeup enhancement module <NUM> may receive (e.g. based on a facial recognition process executed locally on computing device <NUM> or executed remotely in the cloud) an indication of two faces included in the image, namely face <NUM> of the first user and face <NUM> of the second user. Digital makeup enhancement module <NUM> may output graphical indications of detection of faces <NUM>, <NUM> by outputting graphical borders <NUM> and <NUM>. Graphical border <NUM> includes a border around at least a portion of face <NUM>, and graphical border <NUM> includes a border around at least a portion of face <NUM>. Graphical borders <NUM> and <NUM> may have a distinct pattern or color (e.g., white).

Digital makeup enhancement module <NUM> may further receive, based on the facial recognition process, an indication of a match between facial data associated with face <NUM> and reference facial data associated with the face of an enrolled user (e.g., the enrolled user associated with <FIG>) for which digital makeup enhancement data <NUM> has been previously captured and stored. In response, digital makeup enhancement module <NUM> may change graphical border <NUM> and output instead a graphically emphasized border <NUM> (e.g., a dotted border or a border of a particular color, such as green), which at least partially surrounds face <NUM> of the first user, such as is shown in <FIG>. However, if digital makeup enhancement module <NUM> does not receive an indication of a recognition of face <NUM> with that of an enrolled user, digital makeup enhancement module <NUM> may not output a graphically emphasized border around face <NUM>, but may instead maintain graphical border <NUM> around face <NUM> (e.g., the same graphical border <NUM>). As shown in <FIG>, digital makeup enhancement module <NUM> has only recognized face <NUM> as one of a previously enrolled user on computing device <NUM>, and therefore only outputs graphically emphasized border <NUM> around face <NUM> of this first user.

Upon outputting graphically emphasized border <NUM> around at least a portion of face <NUM>, digital makeup enhancement module <NUM> may receive an indication of a user selection of face <NUM>. For example, this first enrolled user may provide a user selection of face <NUM> (e.g., using input components <NUM> and/or presence-sensitive display <NUM>) to cause digital makeup enhancement module <NUM> to automatically retrieve and apply stored digital makeup enhancement data to the facial data associated with face <NUM>.

Digital makeup enhancement module <NUM> may retrieve previously stored digital makeup enhancement data <NUM> for the first enrolled user (e.g., based on a unique identifier associated with this user), and may further identify the reference facial data <NUM> for the enrolled user that corresponds to the digital makeup enhancement data <NUM> that was previously applied and stored during the enrollment process (e.g., such as shown in <FIG>). Digital makeup enhancement module <NUM> may, in some cases, output, for display, a graphical indication of the retrieved digital makeup enhancement data (e.g., in a displayed menu or field). Digital makeup enhancement module <NUM> may then receive a selection of the graphical indication of the digital makeup enhancement data, such as by the first user, to confirm that the first user wishes computing device <NUM> to automatically apply the saved enhancement data to face <NUM>.

Digital makeup enhancement module <NUM> may then apply the saved digital makeup enhancement data <NUM> to face <NUM>, such as shown in <FIG>. To do so, in various examples, digital makeup enhancement module <NUM> may identify facial data (e.g., one or more facial features), which is associated with one or more portions of face <NUM>, that corresponds to the reference facial data <NUM> (e.g., one or more reference facial features) for the face of the enrolled user associated with digital makeup enhancement data <NUM>. Digital makeup enhancement module <NUM> may then apply the digital makeup enhancement data (e.g., settings) <NUM> to the corresponding facial data for the one or more portions of face <NUM>. For instead, the corresponding facial data may include facial features associated with the eyes and cheeks on face <NUM>.

As shown in the example of <FIG>, four distinct digital makeup enhancements have been applied to face <NUM>, based on the previously saved makeup settings for the enrolled user (e.g., the enrolled user referenced in <FIG>): a first digital makeup enhancement <NUM> (e.g., foundation enhancement) to one cheek of the user, a second digital makeup enhancement <NUM> (e.g., foundation enhancement) to another cheek of the user. a third digital makeup enhancement <NUM> (e.g., eyeliner enhancement) underneath one eye of the user, and a fourth digital makeup enhancement <NUM> (e.g., eyeliner enhancement) underneath another eye of the user. Each of these digital makeup enhancements may be represented or otherwise associated with corresponding data that is included in digital makeup enhancement data <NUM>, which may indicate the type of makeup enhancement and corresponding attributes, and which may also be associated with respective facial data (e.g., facial data associated with the user's cheeks and/or eyes).

In cases in which the user associated with face <NUM> is also a previously enrolled user, having previously stored enhancement data stored in digital makeup enhancement data <NUM>, similar digital makeup enhancements may be similarly applied to face <NUM> of this second user, although this is not illustrated in the example of <FIG>.

In some cases, prior to applying digital makeup enhancements to the face of the user, such as illustrated in <FIG>, the user may use computing device <NUM> to make one or more modifications or updates to the previously stored enhancements for that user before they are applied. In these cases, digital makeup enhancement module <NUM> may retrieve digital makeup enhancement data <NUM> associated with reference facial data <NUM> for the user (e.g., the enrolled user), and may output an indication or representation of this data to the user. For example, digital makeup enhancement module <NUM> may interact with UI module <NUM> to output a graphical menu of various settings for digital makeup enhancement data <NUM> for this user.

Digital makeup enhancement module <NUM> may then receive an indication of at least one modification to digital makeup enhancement data <NUM> based on user selection. For instance, the user may interact with the graphical menu of settings to modify the settings for first digital makeup enhancement <NUM> (e.g., foundation enhancement) that is to be applied to one cheek of the user, and/or to third digital makeup enhancement <NUM> (e.g., eyeliner enhancement) that is to be applied underneath one eye of the user. These modifications may relate to any sort of adjustments, such as adjustments to the type, style, color, strength, etc., of the enhancements. Digital makeup enhancement module <NUM> may then generate, based on the at least one modification, modified digital makeup enhancement data, and apply the modified digital makeup enhancement data to the facial data of the face of the user to generate the modified digital images, similar to that shown in <FIG>.

In certain cases, and as noted earlier, a previously enrolled user may also store multiple different groups of digital makeup settings or profiles in digital makeup enhancement data <NUM>. For example, an enrolled user may choose to create and store different groups of digital makeup settings in digital makeup enhancement data <NUM> corresponding to different types of anticipated or desired uses, such as a first group of enhancement settings that are associated with business or formal use. This same user, however, may also wish to store a second group of enhancement settings that are associated with personal use (e.g., for use at a party setting). Digital makeup enhancement data <NUM> may include any number of groups of digital makeup enhancement settings, one or more of which may be associated with the same enrolled user. In these cases, digital makeup enhancement module <NUM> may use UI module <NUM> to output a graphical menu that includes all of the groups of enhancement settings that have been previously saved for the user, and which are currently available for application. Digital makeup enhancement module <NUM> may then receive a selection of one of these groups of digital makeup settings for retrieval and application to the facial data of the user. For instance, if the user wishes to apply the group of settings associated with personal use for an upcoming party, the user may select the second group of enhancement settings that were previously saved and associated with personal use, and digital makeup enhancement module <NUM> may apply these settings to the facial data of the face of the user for output, similar to that shown in <FIG>.

<FIG> is a flow diagram illustrating example operations of a process <NUM> that is performed by a computing device, such as computing device <NUM> (<FIG>) and/or computing device <NUM> (<FIG>), in accordance with one or more aspects of the present disclosure. For purposes of illustration only, the operations of <FIG> are described with reference to computing device <NUM> shown in <FIG>.

As shown in <FIG>, digital makeup enhancement module <NUM> may output (<NUM>), for display (e.g., using or interacting with UI module <NUM>), one or more digital images that include a digital representation of a face of a user. Digital makeup enhancement module <NUM> may receive (<NUM>), based on a facial recognition process (e.g., a process performed locally on computing device <NUM> by digital makeup enhancement module <NUM>, or a process performed remotely from computing device <NUM> in the cloud), an indication of a match between facial data associated with the digital representation of the face of the user and reference facial data <NUM> associated with a digital representation of a face of an enrolled user, wherein the reference facial data is associated with digital makeup enhancement data previously stored for the enrolled user.

Digital makeup enhancement module <NUM> may retrieve (<NUM>) digital makeup enhancement data <NUM> that is associated with reference facial data <NUM>. Digital makeup enhancement module <NUM> may apply (<NUM>) digital makeup enhancement data <NUM> to the facial data of the digital representation of the face of the user to generate one or more modified digital images that indicate at least one corresponding digital makeup enhancement to the digital representation of the face of the user. Digital makeup enhancement module <NUM> may then output (<NUM>), for display (e.g., using or interacting with UI module <NUM>), the one or more modified digital images. Any of the digital images received or processed by digital makeup enhancement module <NUM> may be stored in digital images <NUM>. Any of the reference facial data <NUM>, digital images <NUM>, and/or digital makeup enhancement data <NUM> may be stored locally on computing device <NUM> or in the cloud.

If implemented in software, the functions may be stored on or transmitted over, as one or more instructions or code, a computer-readable medium and executed by a hardware-based processing unit. In this manner, computer-readable media generally may correspond to (<NUM>) tangible computer-readable storage media, which is non-transitory or (<NUM>) a communication medium such as a signal or carrier wave.

By way of example, and not limitation, such computer-readable storage media can comprise RAM, ROM, EEPROM, CD-ROM or other optical disk storage, magnetic disk storage, or other magnetic storage devices, flash memory, or any other storage medium that can be used to store desired program code in the form of instructions or data structures and that can be accessed by a computer.

Accordingly, the term "processor," as used herein may refer to any of the foregoing structures or any other structure suitable for implementation of the techniques described herein. In addition, in some aspects, the functionality described herein may be provided within dedicated hardware and/or software modules.

Rather, as described above, various units may be combined in a hardware unit or provided by a collection of intraoperative hardware units, including one or more processors as described above, in conjunction with suitable software and/or firmware.

It is to be recognized that, depending on the embodiment, certain acts or events of any of the methods described herein can be performed in a different sequence, may be added, merged, or left out altogether (e.g., not all described acts or events are necessary for the practice of the method). Moreover, in certain embodiments, acts or events may be performed concurrently, e.g., through multi-threaded processing, interrupt processing, or multiple processors, rather than sequentially.

Claim 1:
A computer-implemented method comprising:
outputting (<NUM>), by a computing device and for display, one or more digital images that include a digital representation of a face of a user;
receiving (<NUM>), by the computing device and based on a facial recognition process, an indication of a match between facial data associated with the digital representation of the face of the user and reference facial data associated with a digital representation of an enrolled face of the user, wherein the reference facial data is associated with digital makeup enhancement data previously stored for the user;
retrieving (<NUM>), by the computing device, the digital makeup enhancement data that is associated with the reference facial data;
applying (<NUM>), by the computing device, the digital makeup enhancement data to the facial data of the digital representation of the face of the user to generate one or more modified digital images that indicate at least one corresponding digital makeup enhancement to the digital representation of the face of the user; and
outputting (<NUM>), by the computing device and for display, the one or more modified digital images.