Patent Description:
The present application relates, generally, to interactions with devices having a built-in camera and, more specifically, to hand-over-face input sensing for such interaction.

With the success of electronic devices that include touch based input devices, such as touchscreens, touchpads, trackpads, smartwatches, interactive blackboards and the like, touch interaction has become the dominant method of interacting with such electronic devices. Touch interaction can include single touch or multi-touch interaction, such as tapping, pinching, flicking, etc. on a touch based input device.

However, there are many scenarios where touch interaction with the touch based input device cannot be detected. Also, there are many scenarios where touch interaction with the touch based input device is not desirable or feasible, for example, when a user is driving a car and wishes to interact with the touch based input device of the vehicle, or when the touch based input device is outside the reach of the user. In these scenarios, a user could benefit from alternative interaction mechanisms for interacting with an electronic device that does not involve touch interaction.

Another example scenario relates to the touch interaction with the touch screen displays of mobile phones (e.g., smartphones). A generation is growing up with social media, and one aspect of current social media social media and sharing of self-portrait photos (also known as "selfies") and videos captured using smartphones. Some popular smartphone applications allow a user to select a "filter. " Often, the filter adds augmented reality elements to an image or a video. To select a filter and, thereby, select one or more available augmented reality elements to add to an image or video, the user typically touches the screen and scrolls through various filter choices.

<CIT> describes a method and system for computer vision based control of a device in which a shape detection algorithm is applied on an image to identify in the image a finger positioned over or near a user's lips. A device may be controlled based on the detection of the shape, for example, a change of volume of an audio output of the device may be caused.

<CIT> describes an intelligent terminal human-computer interaction method capable of fusing a human face and a gesture. The intelligent terminal human-computer interaction method performs normalization processing on human face images obtained by an intelligent terminal camera, calculates center coordinates of major characteristic of the human face after the normalization processing is performed on the obtained human face images, performs hand complexion and background modeling according to gesture images collected by the intelligent terminal camera, and describes characters of the gesture images detected in the step three. <CIT> describes that an image processing apparatus comprises a gesture detection part that detects a gesture to the body and detects a part of the body to which the gesture is made, and an image processing part that performs image processing according to the gesture on a part of the body included in an image corresponding to the part of the body to which the gesture is made. <CIT> describes a real-time digital image makeup applying method of virtual reality.

<CIT>, <CIT>, <CIT> and <CIT> relate to methods and systems for augmented reality.

In accordance with an aspect of the present invention, there is provided a method of sensing an interaction with an electronic device comprising a camera. The method includes: receiving image data for one or more images captured by the camera; processing the image data for the one or more images to determine a type for a hand gesture and a location of the hand gesture in the one or more images; processing the image data for the one or more images to determine a plurality of face landmark locations in the image; comparing the location of the hand gesture to each face landmark location of the plurality of face landmark locations; identifying, based on the comparing, a selected face landmark, the selected face landmark having a selected face landmark location, among the plurality of face landmark locations, having a greatest proximity to the location of the hand gesture; identifying, based on the comparing and the type for the hand gesture, an action, wherein the identifying the action is further based on the selected face landmark; and providing an indication of the action, wherein the action comprises adding an augmented reality element to the image and the method further comprises selecting, based on the selected face landmark, the augmented reality element to add to the image. The processing the image comprises adding the augmented reality element to the processed image spatially associated with the selected face landmark.

In accordance with another aspect of the present invention, there is provided an electronic device comprising: a camera adapted to capture one or more images; a display screen; and a processor. The processor is configured to: process image data for the one or more images received from the camera to determine a type for a hand gesture and a location of the hand gesture in the one or more images; process image data for the one or more images received from the camera to determine a plurality of face landmark locations in the image; compare the location of the hand gesture to each face landmark location of the plurality of face landmark locations; identify, based on the comparing, a selected face landmark, the selected face landmark having a selected face landmark location, among the plurality of face landmark locations, having a greatest proximity to the location of the hand gesture; identify, based on the comparing and the type for the hand gesture, an action, wherein the identifying the action is further based on the selected face landmark; and provide an indication of the action, wherein the action comprises adding an augmented reality element to the image and the method further comprises selecting, based on the selected face landmark, the augmented reality element to add to the image. The processor is further configured to add the augmented reality element to the processed image spatially associated with the selected face landmark.

In accordance with another aspect of the present invention, there is provided a non-transitory computer-readable medium storing instructions, wherein execution of the instructions causes a processor of an electronic device comprising a camera to: receive image data for one or more images captured by the camera; process the image data for the one or more images to determine a type for a hand gesture and a location of the hand gesture in the one or more images; process the image data for the one or more images to determine a plurality of face landmark locations in the one or more images; compare the location of the hand gesture to each face landmark location of the plurality of face landmark locations; identify, based on the comparing, a selected face landmark, the selected face landmark having a selected face landmark location, among the plurality of face landmark locations, having a greatest proximity to the location of the hand gesture; identify, based on the comparing and the type for the hand gesture, an action, wherein the identifying the action is further based on the selected face landmark; and provide an indication of the action, wherein the action comprises adding an augmented reality element to the image and the method further comprises selecting, based on the selected face landmark, the augmented reality element to add to the image. The processing the image comprises adding the augmented reality element to the processed image spatially associated with the selected face landmark.

Reference will now be made, by way of example, to the accompanying drawings which show example implementations; and in which:.

Touch input is, currently, the leading interaction mechanism with electronic devices that include a touchscreen display, such as mobile phones, tablets, televisions, vehicle infotainment systems, smartphones, and the like. However, touch is challenging or limited in certain situations, such as when the device is in a certain distance from the user. One situation relates to taking self-portraits from a distance and augmenting face with one or more of many available augmented reality (AR) elements. Specifically, in certain use cases related to the human face, such as performing virtual makeup, adding AR elements to the face, and in photo face editing scenarios, it may be considered that touch interaction is not the best option.

Users often use touch input for interaction with the device, such as for example, navigating through different levels of menus to select a facial element and select an applicable action to apply to the selected facial element. Using touch input for interaction with the device requires that the touch surface of the device be maintained at a reachable distance. This need to maintain a reachable distance makes touch interaction with the device difficult when the device is being maintained at a given distance that is beyond a reachable distance, such as when the user wants to take a selfie or other photos from the given distance. Additionally, interaction with the device is also made difficult by cold weather during which taking off gloves to perform a touch input may be considered to be onerous.

It may also be considered that using touch input does not allow more than one person to interact with the device due to the screen size of the touchscreen display of the device. However, in use cases such as photo taking, multiple users can be present in the photo and each user among the multiple users might want to have individual control over their appearance.

In overview, it is proposed herein to take advantage of human facial structure to allow users to interact with their face and, in particular, interact with different face landmarks for touch interaction with an electronic device that includes a camera. That is, the face is employed as a touch surface, or a touch based input device, for touch interactions with an electronic device that includes a camera.

The present invention relates to an electronic device that includes a camera that allows a user to interact with different face landmarks as an input channel for touch interaction with the electronic device. That is, the face is employed as a touch surface or touch based input device for touch interactions with the electronic device. The camera of the electronic device captures one or more images of a user's touch interaction with different face landmarks, and a processor of the electronic device receives the one or more captured images and processes the one or more captured images to determine a type of touch interaction performed by the user. The processor may also determine an action to be performed by an application running on the electronic device or a hardware component of the electronic device based on the determined type of touch interaction, and transmit a command to the application or hardware component to perform the action.

According to an aspect not encompassed by the wording of the appended claims, there is provided a method of augmenting an image captured by a camera of a device, the device having a display screen. The method includes receiving an image from the camera, receiving an indication of a fingertip location in the image, receiving indications of a plurality of face landmark locations in the image, comparing the fingertip location to each face landmark location of the plurality of face landmark locations, identifying, based on the comparing, a selected face landmark, the selected face landmark having a selected face landmark location, among the plurality of face landmark locations, having a greatest proximity to the fingertip location, processing the image to generate a processed image, the processed image including an additional element spatially associated with the selected face landmark and providing the processed image to the display screen. In some examples, a device is provided having a graphics element for carrying out this method and a computer readable medium is provided for adapting a processor in a device to carry out this method.

According to another aspect not encompassed by the wording of the appended claims, there is provided a method of receiving an instruction. The method includes receiving a plurality of images from a camera element, receiving an indication of a gesture, an indication of a plurality of face landmarks and a location of the gesture in relation to a particular face landmark among the plurality of face landmarks, selecting, based on the indication of the gesture and the location of the gesture, an instruction and providing the instruction to a processor.

Other aspects and features of the present invention will become apparent to those of ordinary skill in the art upon review of the following description of specific implementations of the application in conjunction with the accompanying figures.

<FIG> illustrates, in a front elevation view, an exemplary electronic device. In the example embodiment illustrated in <FIG>, the electronic device is a mobile device <NUM>. Examples of mobile devices <NUM> includes mobile phones, smartphones, tablets, laptop computers, smart television sets. The mobile device <NUM> includes a display screen <NUM> and a front-facing lens <NUM> of a camera <NUM> (see <FIG>) of the mobile device <NUM>.

<FIG> illustrates, schematically, a block diagram of components of the mobile device <NUM> of <FIG>. The mobile device <NUM> includes a processor <NUM> that controls the overall operation of the mobile device <NUM>. The processor <NUM> is coupled to and interacts with various other components of the mobile device <NUM>, including a memory <NUM>, a camera <NUM> and the display screen <NUM>, shown in <FIG>. The processor <NUM> is coupled to and interacts with the various other components via, for example, a bus. Components of the camera <NUM> include a charge-coupled device (CCD) <NUM> and the front-facing lens <NUM>, shown in <FIG>.

<FIG> illustrates an exemplary system <NUM>. The system <NUM> includes a computer vision element <NUM>, an interaction control element <NUM> and a graphics element <NUM>. The system <NUM> communicates with the camera <NUM> and the display screen <NUM>, both of which are shown in <FIG>, as described below. In an embodiment, the system <NUM> is a software system and the computer vision element <NUM>, the interaction control element <NUM> and the graphics element <NUM> are software elements or software modules of the software system <NUM>. The software system <NUM> (and the computer vision element <NUM>, the interaction control element <NUM> and the graphics element <NUM>) include computer-readable instructions that may be stored in the memory <NUM> and the computer-readable instructions may be executed by the processor <NUM>.

The computer vision element <NUM> and the graphics element <NUM> receive image data from the camera <NUM>. The image data is representative of an image captured by the camera <NUM>. The computer vision element <NUM> is illustrated as having two components. The first component is a face landmark detection component <NUM>. The second computer vision component is a fingertip detection component <NUM>. The face landmark detection component <NUM> is software of the computer vision element <NUM> that is configured to: receive image data representative of an image captured by the camera; process the image data using a computer vision method to detect a face landmark in the image, identify the detected face landmark, and generate a location of the identified face landmark in the image; and output a label indicative of the identified face landmark and the location of the identified face landmark in the image. The fingertip detection component <NUM> is software of the computer vision element <NUM> that is configured to receive image data representative of an image captured by the camera; process the image data using a computer vision method to detect a fingertip in the received image data; determine a location of the fingertip in the image; and output a label indicative of the detected fingertip and the location of the detected fingertip in the image. The computer vision methods used to process the image data representative of the captured image include for example, image classification, object detection, object tracking, sematic segmentation, feature detection and matching, and context and scene understanding.

Example steps in a method of operation of the camera <NUM> are illustrated in <FIG>. In a manner typical of mobile device operation, the camera <NUM> captures (step <NUM>) an image (or a sequence of images for a video) through the camera lens <NUM>. The camera <NUM> provides image data representative of the captured image to the processor <NUM>, which executes the computer-readable instructions of the system <NUM>. In particular, the camera <NUM> provides (step <NUM>) the image data to the graphics element <NUM>. In some examples, the graphics element <NUM> is a rendering engine.

Example steps in a method of operation of the graphics element <NUM> are illustrated in <FIG>. In a manner typical of mobile device operation, the graphics element <NUM> receives (step <NUM>) the image data representative of the captured image from the camera <NUM>. The graphics element <NUM> then determines (step <NUM>) whether augmentation is to be added to the image.

Upon determining (step <NUM>), based on information provided by the interaction control element <NUM>, that augmentation is not to be added to the image, the graphics element <NUM> then provides (step <NUM>) the image data representative of the clean (i.e., non-augmented) image to the display screen <NUM> of the mobile device <NUM> for displaying the image thereon in a manner typical of mobile device <NUM> operation.

Upon determining (step <NUM>), based on information provided by the interaction control element <NUM>, that augmentation is to be added to the image, the graphics element <NUM> then receives (step <NUM>), from the interaction control element <NUM>, an indication of a selected face landmark along with an indication of the location, in the image, of the selected face landmark. That is, the graphics element <NUM> receives (step <NUM>), from the interaction control element <NUM>, a location (e.g., pixel coordinates) of the selected face landmark that aligns with the fingertip location. For example, the interaction control element <NUM> may indicate coordinates, in the image, for a specific point on a chin.

The graphics element <NUM> also receives (step <NUM>), from the interaction control element <NUM>, an indication of an additional element. The additional element may be referred to hereinafter as an augmented reality element or an AR element. The additional element may be selected from a catalogue of AR elements that are associated with the selected face landmark. The catalogue of AR elements may be stored, for example, in the memory <NUM> in a manner that is accessible to the interaction control element <NUM> executed on the processor <NUM>. In one example, if the forehead is the selected face landmark, the catalogue of AR elements that are associated with the forehead may comprise many hats, such as: a Stetson; a bowler; a baseball cap; a crown; a tiara; and a hockey helmet.

Based on the received information about the selected face landmark, the graphics element processes (step <NUM>) the image data to produce augmented image data. The result of the processing (step <NUM>) of the image data may be considered to be processed image data or augmented image data. The augmented image data includes the additional element spatially associated with the selected face landmark. In an alternative, wherein the action is an increase in zoom level, the processed image data may be image data that has been subjected to a zoom. Indeed, in another alternative, wherein the action is an increase in audio volume level, the graphics elements <NUM> may not process (step <NUM>) the image data. Instead, the graphics elements <NUM> may signal an audio control element (not shown) with a command to increase audio volume. The audio control element may then appropriately control and audio component.

The graphics element <NUM> may then provide (step <NUM>) the processed image data to the display screen <NUM> of the mobile device <NUM>.

Returning to <FIG>, the camera <NUM> also provides (step <NUM>) the image data to the computer vision element <NUM>. Although the provision (step <NUM>), by the camera element <NUM>, of the image data to the graphics element <NUM> is illustrated, in <FIG>, as preceding the provision (step <NUM>), by the camera <NUM>, of the image data to the computer vision element <NUM>, it should be clear, to a person of ordinary skill in the art that the order may be reversed or the two steps may happen in parallel, that is, nearly simultaneously.

Example steps in a method of operation of the computer vision element <NUM> are illustrated in <FIG>. The computer vision element <NUM> receives (step <NUM>) image data from the camera <NUM>. It may be stated, more precisely, that, in many embodiments, the computer vision element <NUM> receives image data (step <NUM>) from the camera <NUM>. In parallel, the face landmark detection component <NUM> performs (step <NUM>) face landmark detection and the fingertip detection component <NUM> performs (step <NUM>) fingertip detection. The face landmark detection may be performed (step <NUM>) on the image data representative of each captured image using a learning-based object detector, which has been trained to detect and localize face landmarks in the captured images (e.g., provide coordinates of the detected face landmark in the captured image). The learning-based object detector may be implemented using a feature extractor (not shown) which has been trained to detect face landmarks, a classifier (not shown) that classifies the detected face landmarks and a localizer (not shown) that outputs the location, in the captured image, of the detected face landmarks. The feature extractor, the classifier and the localizer may be implemented using distinct, trained neural networks. Alternatively, the object detector may be implemented using a single deep neural network that has been trained for face landmark detection and localization using a training dataset comprising samples of different face landmarks.

The learning-based object detector may be designed using an application development platform. Google® LLC of Mountain View, California has an application development platform called Firebase. Part of the Firebase platform is a machine learning Software Development Kit (SDK) called "ML Kit. " Conveniently, aspects of ML Kit relate specifically to face detection (see firebase. com/docs/ml-kit/detect-faces).

Beyond merely locating face landmarks, tracking of the face landmarks, that is, repetitive face landmark detection, may be considered to assist the provision of real-time interactions.

The fingertip detection may be performed (step <NUM>) using a learning-based object detector that is trained to detect fingertips in images and localize the fingertip (e.g., output the location of the fingertip in the image). The learning-based object detector may be implemented using a feature extractor that has been trained to detect fingertips and a localizer that outputs the location, in the image of the detected fingertips. The feature extractor and the localizer may be implemented using distinct, trained neural networks. Alternatively, the learning-based object detector may be implemented using a deep neural network that has been trained for fingertip detection and localization using a training dataset comprising samples of different fingertips. Beyond merely locating the fingertip, tracking of the fingertip, that is, repetitive fingertip detection, may be considered to assist the provision of real-time interactions.

Upon completion of the performing (step <NUM>) of face landmark detection, the face landmark detection component <NUM> provides (step <NUM>) a location of a variety of face landmarks to the interaction control element <NUM>. The face landmarks may, for example, include: left eye position; right eye position; left cheek position; right cheek position; tip of nose; left mouth position; right mouth position; and bottom mouth position. Locations for each of the face landmarks may be expressed as coordinates in relation to a frame of reference for the captured image.

Upon completion of the performing (step <NUM>) of fingertip detection, the fingertip detection component <NUM> provides (step <NUM>) a location of a user's fingertip to the interaction control element <NUM>. The location of the user's fingertip may be expressed as coordinates in relation to a reference frame of the captured image.

Example steps in a method of operation of the interaction control element <NUM> are illustrated in <FIG>.

The interaction control element <NUM> receives (step <NUM>) the label that includes the fingertip location coordinates from the fingertip detection component <NUM>. The interaction control element <NUM> also receives (step <NUM>) the face landmark locations from the face landmark detection component <NUM>. The interaction control element <NUM> then compares (step <NUM>) the fingertip location to the locations of the various face landmarks. The interaction control element <NUM> determines (step <NUM>) whether a location of a face landmark corresponds with a location of a fingertip. This determination could be based on finding a result for greatest proximity (e.g., minimum distance) between various ones of the facial landmarks and the fingertip location. When there is a correspondence, the interaction control element <NUM> identifies (step <NUM>) that one of the face landmarks has been selected by the user. The interaction control element <NUM> may consider the identified face landmark to be a "selected" face landmark. The selected face landmark location may, for example, be a face landmark location, among the plurality of face landmark locations, determined to have a greatest proximity (e.g., a minimum distance) to the fingertip location. Based on the selected face landmark, the interaction control element <NUM> may select an AR element. The interaction control element <NUM> then provides (step <NUM>), to the graphics element <NUM>, the location of the selected face landmark and the selected AR element.

As discussed hereinbefore in the context of <FIG>, responsive to receiving (step <NUM>) the label associated with location of the selected face landmark, the graphics element processes (step <NUM>) the image data representative of the image to produce augmented image data. The graphics element <NUM> then provides (step <NUM>) the augmented image data to the display screen <NUM> of the mobile device <NUM>. In one example of processing (step <NUM>) the received image data, the graphics element <NUM> processes (step <NUM>) the image data to include the selected AR element superimposed over the selected face landmark.

A first example of use relates to adding facial AR elements when taking selfie photos.

<FIG> illustrates a first person <NUM> and a second person 802F in the act of preparing to take a selfie with the mobile device <NUM>. The first person <NUM> has a finger <NUM> and face <NUM> with a chin <NUM>. The first person <NUM> is illustrated as touching the finger <NUM> to the chin <NUM>. The second person 802F has a finger 806F and face 808F with a pair of eyes 809F. The second person 802F is illustrated as touching the finger 806F to the face 808F near one eye among the pair of eyes 809F.

<FIG> illustrates an augmented image <NUM> as presented on the display screen <NUM> of the mobile device <NUM>. Relative to an original image (not shown) representative of an image captured through the front-facing lens <NUM> of the camera <NUM> of the mobile device <NUM>, the augmented image <NUM> of <FIG> includes additional, user-selected AR elements. The user-selected AR elements correspond to the face landmarks touched, as illustrated in <FIG>. In the case of the first person <NUM>, the augmented image <NUM> of <FIG> includes a beard <NUM> covering the chin <NUM>. In the case of the second person 802F, the augmented image <NUM> of <FIG> includes a pair of glasses 909F covering the pair of eyes 809F. Although <FIG> does not illustrate the first person <NUM> touching his forehead, it may be considered that the appearance, in the augmented image <NUM> of <FIG>, of a hat <NUM> covering the forehead of the first person <NUM> may be attributed to the first person <NUM> having touched his forehead.

Notably, the pair of glasses 909F may be the default glasses selected, by the interaction control element <NUM>, responsive to identifying (step <NUM>) the eyes 809F as the selected face landmark. Optionally, by repeatedly bringing the fingertip to the eye landmark, the user may cycle through a catalogue of glasses associated with the eyes as a face landmark. Furthermore, in some examples, more than one fingertip may be detected by the fingertip detection component, thereby enabling a version of multi-touch on the face surface. The user may increase the distance between an index finger and a thumb, while holding the index finger and the thumb near the eye face landmark. Responsively, the graphics element <NUM> may process (step <NUM>) the image data for the augmented image <NUM> to increase a size of the selected glasses. Conversely, responsive to the user pinching the index finger and thumb together, the graphics element <NUM> may process (step <NUM>) the image data the augmented image <NUM> to decrease a size of the selected glasses.

Rather than cycling through a catalogue of glasses by repeatedly tapping the eye face landmark, the user may draw glasses around the eye face landmark. Responsive to sensing a shape for the glasses that the user has drawn, the graphics element <NUM> may process (step <NUM>) the image data to select glasses that most closely match the sensed shape.

The user may opt to augment the facial images with virtual make-up. Clearly, a user may touch an appropriate face landmark and cycle through shades of eye shadow, blush or lipstick.

<FIG> illustrates an exemplary system <NUM>. The system <NUM> is an alternative to the system <NUM> of <FIG>. The system <NUM> includes a computer vision element <NUM>, the interaction control element <NUM> and the graphics element <NUM>, the latter two elements are similar to those described with reference to <FIG>. The system <NUM> also includes the camera <NUM> and the display screen <NUM>, both of which are shown in <FIG>. The system <NUM> also includes components implemented, in software, such as the computer vision element <NUM>, the interaction control element <NUM> and the graphics element <NUM>. The software may be stored as computer-readable instructions on the memory <NUM> and the computer-readable instructions may be executed by the processor <NUM>.

The computer vision element <NUM> and to the graphics element <NUM> receive image data from the camera <NUM>. The computer vision element <NUM> is illustrated as having a single component. Instead of the face landmark detection component <NUM> and the fingertip detection component <NUM> in the computer vision element <NUM> of <FIG>, the computer vision element <NUM> of <FIG> has a merged face landmark detection and fingertip detection component <NUM>.

In this embodiment, a deep neural network is used for face landmark detection and fingertip detection <NUM>. The deep neural network is trained to detect face landmarks and detect the location of a fingertip relative to the detected face landmarks. This stands in contrast to detecting the location of a fingertip relative to a coordinate system, with the same coordinate system being used when detecting face landmarks.

In comparison to the embodiment represented by <FIG>, the embodiment represented by <FIG> is expected to run faster on the mobile device <NUM> due to using only one trained neural network, rather than two trained neural networks. However, the embodiment represented by <FIG> may be considered to involve more effort, in terms of data collection and annotation, than the embodiment represented by <FIG>.

Aspects of the present invention may be considered to be effective, not only for interaction with mobile devices but, also, for interaction with any device having a front-facing camera in combination with a display screen. Such devices may include tablets, e-readers, desktop computers, laptop computers, smart watches, televisions, interactive advertising displays, photo booths and smart mirrors. It may be considered that touch interactions with a smart watch, in particular, are even more challenging than touch interactions with a mobile device due to the small screen size of the typical smart watch.

Advantageously, and as illustrated in the context of <FIG> and <FIG>, aspects of the present invention may allow for identifying interactions performed by more than one user who are present in the image.

Since the use cases are related to the facial elements (e.g., virtual makeup and facial AR), it may be considered that a face-based input channel would be intuitive and easy to understand for users.

Notably, hand-over-face gestures could be useful in other applications not directly related to the face landmarks in the manner that AR elements are related to the face landmarks. Consider a scenario wherein the mobile device <NUM> is mounted in a car in front of a driver of the car. The mobile device <NUM> may be mounted at such a distance that reaching the mobile device <NUM> by hand is challenging. Furthermore, distracted driving laws may discourage the driver from touching the mobile device <NUM>.

Aspects of the present invention may be extended generalized beyond mere fingertip detection. Indeed, more than one fingertip may be detected, say, thumb and forefinger, thereby allowing for a pinching gesture for use when interacting with an application providing output to the display screen <NUM> of the mobile device <NUM>. For example, if the application providing output to the display screen <NUM> is a mapping application, the application may respond to detection of the pinching by zooming in on a map displayed to the display screen <NUM>. When the location of a fingertip is tracked over time, a gesture may be detected wherein the driver swipes the fingertip across the driver's right cheek. For example, if the application providing output to the display screen <NUM> is a music streaming application, the application may respond to detection of the swipe across the right cheek by advancing to the next song. For another example, if the application providing output to the display screen <NUM> is a messaging application, the application may respond to detection of the swipe across the right cheek by proceeding to provide text to voice output of a next message.

Aspects of the present invention may relate to controlling settings for the front-facing camera <NUM>. Examples camera settings include: zooming level; shutter release; and brightness. It is clearly convenient that a user may adjust camera settings while the user's face is in the frame captured by the camera. Thus, interactions with face landmarks for controlling the front-facing camera <NUM> becomes easier especially when the mobile device <NUM> is in a certain distant from the user.

<FIG> illustrates an exemplary system <NUM>. The system <NUM> includes a computer vision element <NUM>, an interaction control element <NUM> and a graphics element <NUM>. The system <NUM> also includes the camera <NUM> and the display screen <NUM>, both of which are familiar from <FIG>. The system <NUM> includes components implemented in software, such as the computer vision element <NUM>, the interaction control element <NUM> and the graphics element <NUM>. The software may be stored as computer-readable instructions on the memory <NUM> and the computer-readable instructions may be executed by the processor <NUM>.

The camera <NUM> connects to the computer vision element <NUM> and to the graphics element <NUM>. The computer vision element <NUM> is illustrated as having three main components: a face landmark detection component <NUM>; a hand gesture detection component <NUM>; and a hand gesture localization component <NUM>.

In operation, the face landmark detection component <NUM> acts to detect face landmarks, the hand gesture detection component <NUM> acts to detect a hand gesture over the face and the hand gesture localization component <NUM> acts to find coordinates of the hand within received image data. Subsequently, the face landmark detection component <NUM> provides a location of a variety of face landmarks to the interaction control element <NUM>. Additionally, the hand gesture detection component <NUM> provides an indication of a hand gesture to the interaction control element <NUM>. Furthermore, the hand gesture localization component <NUM> provides the coordinates of the hand to the interaction control element <NUM>.

<FIG> illustrates a block diagram of components of an exemplary electronic device <NUM>. Examples of electronic devices include mobile phones, smartphones, tablets, smart televisions, interactive blackboards, vehicle infotainment systems, and the like. The electronic device <NUM> includes a processor <NUM> that controls the overall operation of the electronic device <NUM>. The processor <NUM> is coupled to and interacts with various other components of the electronic device <NUM>, including a memory <NUM>, a camera <NUM> and an output device <NUM>, via, for example, a bus. Components of the camera <NUM> include a CCD <NUM> and a lens <NUM>. The output device <NUM> may be a display screen, a speaker, or a light sensor. The electronic device <NUM> may include one processor <NUM> or multiple processors <NUM>, one memory <NUM> or multiple memories <NUM>, and one output device <NUM> or multiple output devices <NUM>. The memory stores programs, applications, and data of the electronic device <NUM>.

<FIG> illustrates a hand-over-face (HOF) gesture interpretation system <NUM>. The HOF gesture interpretation system <NUM> may be a software program which includes computer-readable instructions that are stored in memory <NUM> of the electronic device <NUM>. The computer-readable instructions of HOF gesture interpretation system <NUM> may be executed by the processor <NUM>. The (HOF) gesture interpretation system <NUM> includes a face landmark detection and localization component <NUM>, a hand gesture detection and localization component <NUM>, and an interaction control component <NUM>. The HOF gesture interpretation system <NUM> is configured to receive image data for one or more images captured by the camera <NUM>, analyze the image data for the one or more images captured by the camera <NUM> using computer vision techniques to: detect face landmarks in the one or more images; determine a location for each of the face landmarks in the one or more images; determine a type of hand gesture in the one or more images; and determine a location of the hand gesture in the one or more images. The HOF gesture interpretation system <NUM> is also configured to determine a command based on the location of the face landmarks, the type of hand gesture, and the location of hand gesture, and output the command.

Operation of the HOF gesture interpretation system <NUM> will now be described. The HOF gesture interpretation system <NUM> receives image data for one or more image captured by the camera <NUM>, which is provided to the face landmark detection and localization component <NUM> and the hand gesture detection and localization component <NUM>. The face landmark detection and localization component <NUM> may be software component (e.g., a software module) of the HOF gesture interpretation system <NUM> that includes computer-readable instructions which are executable by the processor <NUM>. The hand gesture detection and localization component <NUM> may also be a software component (e.g., a software module) of the HOF gesture interpretation system <NUM> that includes computer-readable instructions which are executable by the processor <NUM>. The interaction control component <NUM> may also be a software component (e.g., a software module) of the HOF gesture interpretation system <NUM> that includes computer-readable instructions which are executable by the processor <NUM>.

The face landmark detection and localization component <NUM> is configured to receive image data for one or more images captured by the camera <NUM>, process the image data for the one or more images using computer vision methods to detect one or more face landmarks in the captured image, determine a location of each face landmark detected in the capture image, and output a label indicative of the location of each detected face landmark in the captured image. The hand gesture detection and localization component <NUM> is configured to receive image data for one or more images captured by the camera <NUM>, process the image data for the one or more images using computer vision methods to detect hand gesture in the captured image, determine the type of the detected hand gesture, determine a location for the hand gesture, and output a label indicative of the detected the location of each face landmark in the captured image.

The interaction control component <NUM> is coupled to the face landmark detection and localization component <NUM> to receive the output from the face landmark detection and localization component <NUM>. The interaction control component <NUM> is also coupled to the hand gesture detection and localization component <NUM> to receive the output from the hand gesture detection and localization component <NUM>.

<FIG> illustrates a method performed by the face landmark detection and localization component <NUM> (<FIG>). The method begins with the face landmark detection and localization component <NUM> receiving (step <NUM>) image data for one or more images captured by the camera <NUM>. As mentioned as above, the image data for an image is data that is representative of the image. The method then proceeds to step <NUM> where the face landmark detection and localization component <NUM> performs face landmark detection and localization (step <NUM>) to detect one or more face landmarks in the captured image, determine a location of each face landmark detected in the capture image, and generates a label that includes identifications of different face landmarks detected in the one or more images (e.g., left eye, nose tip, right cheek). Each face landmark identification may specify a type for the face landmark and a location in the one or more images. After generating the label, the face landmark detection and localization component <NUM> provides (step <NUM>) the label to the interaction control component <NUM>. The label provided by the face landmark detection and localization component <NUM> may be metadata indicative of each face landmark identification.

<FIG> illustrates a method performed by the hand gesture detection and localization component <NUM>. The method begins with the hand gesture detection and localization component <NUM> receiving (step <NUM>) the image data for one or more images captured by the camera <NUM> of the electronic device <NUM>. The hand gesture detection and localization component <NUM> then performs (step <NUM>) hand gesture detection and localization using a computer vision method to detect a hand gesture in the captured image, determine a type of the hand gesture, determine a location for the hand gesture, and generate a label indicative of the detected the location of each face landmark in the captured image. The label generated by the hand gesture detection and localization component <NUM> may, for example, include an identification of a hand gesture. The hand gesture identification may include a type for the hand gesture (e.g., pinch, index finger pointing) and a location in a frame of reference of the one or more images of the hand gesture. After generating the label, the hand gesture detection and localization module <NUM> provides the label (step <NUM>) the to the interaction control module <NUM>. The label provided by the hand gesture detection and localization module <NUM> may be metadata indicative the identification of a hand gesture.

<FIG> illustrates an exemplary method performed by the interaction control component <NUM>. The output of the face landmark detection and localization component <NUM> and the outputs of the hand gesture detection and localization component <NUM> (e.g., the labels provided by the face landmark detection and localization component <NUM> and the hand gesture detection and localization component <NUM>) are received (step <NUM>) by the interaction control component <NUM>. The interaction control component <NUM> then compares the location of the hand gesture, received (step <NUM>) from the hand gesture detection and localization component <NUM>, with the locations of different face landmarks, received (step <NUM>) from the face landmark detection and localization component <NUM>, to determine (step <NUM>) where, on the face (e.g., on cheek, forehead, chin), the hand gesture has been performed. That is, the interaction control component <NUM> then determines there is a correspondence between the location for the hand gesture and the location of a selected face landmark among the plurality of face landmarks. The interaction control component <NUM> then determines (step <NUM>) an action that is to be performed by an application or program or hardware component of the electronic device <NUM>. Depending on the target context, the interaction control component <NUM> can determine (step <NUM>) different actions. The determining (step <NUM>) may be based on: a) the location at which the hand gesture has been performed on the face; b) the hand gesture type; and c) the application running on the device. The interaction control component <NUM> then prepares (step <NUM>) a command to be sent to a program or application or hardware component of the electronic device <NUM>, to perform the action and provide feedback to the user. In some aspects, the interaction control component <NUM> sends a command indicative of the action to be performed to a program, application or hardware component of the electronic device <NUM>. The program, application or hardware component performs the action in response to receipt of the command.

The interaction control component <NUM> is configured to determine the location of the hand gesture by comparing the face landmarks with the hand gesture location in the captured images; and determine what action is to be performed by the electronic device <NUM>, based on the type and shape of the hand gesture and the location of the hand gesture. For instance, when the gesture is a pinch on the cheek, the corresponding action may be increasing the zoom level for the camera <NUM> of the electronic device <NUM>. Thus, the interaction control element <NUM> is configured to determine a gesture has been performed on the cheek; and determine that a pinching gesture on the cheek corresponds to zooming in on the image. The interaction control element <NUM> may then provide the output, including an indication of an action, to the graphics element.

Some actions may not be directly related to an image. For example, a combination of a face landmark and a gesture may be associated with audio volume control.

Applications for aspects of the present invention are many and varied. For example, when shopping for cosmetics, aspects may allow a customer to virtually apply a particular shade to a particular location on a representation of the customer's face so that the customer may review the extent to which the shade is suitable for their face. Similarly, aspects my allow a customer to virtually try on a hat, eye glasses, sun glasses and ear rings, among many other possibilities. This aspect may particularly helpful when shopping online.

Aspects of the present invention may allow for self-portrait self-editing. Additionally, a customer of a face painter may consider a preview of how the face paint might look when complete.

Aspects of the present invention may be used for camera control and audio control. Such control may also be extended to be used to control smart speakers and control other smart home devices, such as smart lights, smart blinds and smart thermostats. In these aspects, the interactive control component <NUM> sends a command using a communication interface (not shown) of the electronic device to another electronic device, such as internet of things (IoT) devices (e.g., smartwatches, smart speaker, smart lights, smart blinds and smart thermostats, vehicle infotainment systems) to cause the other electronic device to perform the action.

Voice control may be considered to be one known alternative to employing the face as a touch-based input device for interactions with a device. Conveniently, employing the face as a touch-based input device may be considered to be two-dimensional (or even three-dimensional or four-dimensional if depth and/or facial expression is taken into account). Such multi-dimensional input may, accordingly, be considered to be more efficient than voice input, since voice input may be considered to be linear and sequential. Furthermore, employing the face as a touch-based input device may be considered more natural than voice input and especially efficient for continuous input, like volume control or zooming. Moreover, employing the face as a touch-based input device may be considered to be a more reliable alternative than voice input when the environment includes background noise. Further still, employing the face as a touch-based input device may be considered to be more discreet and less disturbing to others when compared to voice control.

A midair gesture may be considered to be another known alternative to employing the face as a touch-based input device for interactions with a device. In this case, employing the face as a touch-based input device may be considered to have a more reliable implementation due to the additional face structure. Furthermore, employing the face as a touch-based input device may be seen to result in gesture detection that is easier and is associated with a higher accuracy than midair gestures, since facial reference landmark points allow for more precise gesture detection and recognition. Indeed, employing the face as a touch-based input device may be considered to provide a richer interaction than midair gestures. It may further be considered that the addition of face structure provides more natural interaction that available for midair gestures.

The HOF gesture interpretation system <NUM> may be always on or may be triggered for operation as described above. When the HOF gesture interpretation system <NUM>, the HOF gesture interpretation system <NUM> could include continuously receive image data for images captured by the camera and, perform gesture detection only when some movement is detected in the captured images. For example, when the HOF gesture interpretation system <NUM> detects some pre-defined gestures (e.g., waving, flicking) in the captured image data, this indicates the start and end of the gesture control. Alternatively, the HOF gesture interpretation system <NUM> may be triggered for operation by detection of an input on an input device (not shown) of the electronic device <NUM>. For example, the processor <NUM> may detect an input on the input device (not shown) of the electronic device, such as actuation of a physical button or detection of a sound (e.g., clapping, snapping, etc.) or voice command, and active the camera <NUM> and the HOF gesture interpretation system <NUM> for gesture control.

Claim 1:
A method of sensing an interaction with an electronic device comprising a camera, the method comprising:
receiving (<NUM>) image data representative of one or more images captured by the camera;
processing (<NUM>) the image data representative of the one or more images to determine a type for a hand gesture and a location of the hand gesture in the one or more images;
processing (<NUM>) the image data representative of the one or more images to determine a plurality of face landmark locations in the image;
comparing (<NUM>) the location of the hand gesture to each face landmark location of the plurality of face landmark locations;
identifying (<NUM>), based on the comparing, a selected face landmark, the selected face landmark having a selected face landmark location, among the plurality of face landmark locations, having a greatest proximity to the location of the hand gesture;
identifying, based on the comparing and the type for the hand gesture, an action, wherein the identifying the action is further based on the selected face landmark; and
providing an indication of the action, wherein the action comprises adding an augmented reality element to the image and the method further comprises selecting, based on the selected face landmark, the augmented reality element to add to the image, wherein the processing the image comprises adding the augmented reality element spatially associated with the selected face landmark to the processed image.