Patent Description:
User head pose information has many uses and applications in various systems and devices that may interact with a user by providing services, outputting and/or retrieving information, displaying and/or playing back media content, and the like. Head pose information may indicate a position of the user's head relative to a point of reference associated with the electronic system or device. For example, the point of reference may correspond to a camera or image capture device used to capture an image of the user's head and/or face, or otherwise detect a presence of the user on behalf of the electronic system or device. The head pose of the user may affect how the electronic system or device responds to user inputs from a particular user. <CIT> discloses a method of determining a facial pose angle of a human face within an image.

The present embodiments are illustrated by way of example and are not intended to be limited by the figures of the accompanying drawings.

In the following description, numerous specific details are set forth such as examples of specific components, circuits, and processes to provide a thorough understanding of the present disclosure. The term "coupled" as used herein means connected directly to or connected through one or more intervening components or circuits. Also, in the following description and for purposes of explanation, specific nomenclature is set forth to provide a thorough understanding of the aspects of the disclosure. However, it will be apparent to one skilled in the art that these specific details may not be required to practice the example embodiments. In other instances, well-known circuits and devices are shown in block diagram form to avoid obscuring the present disclosure. Some portions of the detailed descriptions which follow are presented in terms of procedures, logic blocks, processing and other symbolic representations of operations on data bits within a computer memory. The interconnection between circuit elements or software blocks may be shown as buses or as single signal lines. Each of the buses may alternatively be a single signal line, and each of the single signal lines may alternatively be buses, and a single line or bus may represent any one or more of myriad physical or logical mechanisms for communication between components.

Unless specifically stated otherwise as apparent from the following discussions, it is appreciated that throughout the present application, discussions utilizing the terms such as "accessing," "receiving," "sending," "using," "selecting," "determining," "normalizing," "multiplying," "averaging," "monitoring," "comparing," "applying," "updating," "measuring," "deriving" or the like, refer to the actions and processes of a computer system, or similar electronic computing device, that manipulates and transforms data represented as physical (electronic) quantities within the computer system's registers and memories into other data similarly represented as physical quantities within the computer system memories or registers or other such information storage, transmission or display devices.

The techniques described herein may be implemented in hardware, software, firmware, or any combination thereof, unless specifically described as being implemented in a specific manner. Any features described as modules or components may also be implemented together in an integrated logic device or separately as discrete but interoperable logic devices. If implemented in software, the techniques may be realized at least in part by a non-transitory computer-readable storage medium comprising instructions that, when executed, performs one or more of the methods described above. The non-transitory computer-readable storage medium may form part of a computer program product, which may include packaging materials.

The non-transitory processor-readable storage medium may comprise random access memory (RAM) such as synchronous dynamic random access memory (SDRAM), read only memory (ROM), non-volatile random access memory (NVRAM), electrically erasable programmable readonly memory (EEPROM), FLASH memory, other known storage media, and the like. The techniques additionally, or alternatively, may be realized at least in part by a processor-readable communication medium that carries or communicates code in the form of instructions or data structures and that can be accessed, read, and/or executed by a computer or other processor.

The various illustrative logical blocks, modules, circuits and instructions described in connection with the embodiments disclosed herein may be executed by one or more processors. The term "processor," as used herein, may refer to any general-purpose processor, conventional processor, controller, microcontroller, and/or state machine capable of executing scripts or instructions of one or more software programs stored in memory. The term "sensing device," as used herein, may refer to any device capable of detecting a user of the device and/or sensing user inputs. Examples of sensing devices may include, but are not limited to, smart speakers, home automation devices, voice command devices, virtual assistants, personal computing devices (e.g., desktop computers, laptop computers, tablets, web browsers, and personal digital assistants (PDAs)), data input devices (e.g., remote controls and mice), data output devices (e.g., display screens and printers), remote terminals, kiosks, video game machines (e.g., video game consoles, portable gaming devices, and the like), communication devices (e.g., cellular phones such as smart phones), media devices (e.g., recorders, editors, and players such as televisions, set-top boxes, music players, digital photo frames, and digital cameras), and the like.

<FIG> shows an example environment <NUM> in which the present embodiments may be implemented. The environment <NUM> includes a user <NUM> and a sensing device <NUM>. The sensing device <NUM> may detect a presence of one or more users (such as user <NUM>) within its field-of-view (FOV) <NUM> and determines a pose or position of each user's head relative to the sensing device <NUM>. In some implementations, the sensing device <NUM> may also detect and respond to user inputs by the user <NUM> based, at least in part, on the user's head pose.

The sensing device <NUM> includes one or more sensors <NUM>, a head-pose determination module <NUM>, and a user interface <NUM>. The sensor <NUM> may be configured to receive user inputs and/or collect data (e.g., images, video, audio recordings, and the like) about the surrounding environment. Example suitable sensors include, but are not limited to, cameras, capacitive sensors, microphones, and the like. In some aspects, one or more of the sensors <NUM> (e.g., a camera) is configured to capture images and/or video of the user <NUM>. For example, the camera may be configured to capture images (e.g., still-frame images and/or video) of a scene, including any objects within the FOV <NUM> of the sensing device <NUM>.

The head-pose determination module <NUM> processes the images captured by the sensors <NUM> to determine a head pose of the user <NUM>. In some embodiments, the head-pose determination module <NUM> estimates the user's head pose based, at least in part, on facial landmark information extracted from the images. The facial landmark information indicates respective locations (e.g., pixel locations or coordinates) of one or more facial features including, but not limited to, the eyes, mouth, and nose. Such facial landmark information may be extracted from any images containing a near-frontal view of a user's face (e.g., where at least the nose can be seen). In some aspects, the head-pose determination module <NUM> may implement one or more neural network models to infer the presence and locations of the facial features in a captured image. The neural network models may be trained to recognize human faces, including characteristic features of each face (e.g., eyes, mouth, nose, etc.), through deep learning.

Deep learning is a particular form of machine learning in which the training phase is performed over multiple layers, generating a more abstract set of rules in each successive layer. Deep learning architectures are often referred to as artificial neural networks due to the way in which information is processed (e.g., similar to a biological nervous system). For example, each layer of the deep learning architecture may be composed of a number of artificial neurons. The neurons may be interconnected across the various layers so that input data (e.g., the raw data) may be passed from one layer to another. More specifically, each layer of neurons may perform a different type of transformation on the input data that will ultimately result in a desired output (e.g., the answer). The interconnected framework of neurons may be referred to as a neural network model. Thus, the neural network models may include a set of rules that can be used to describe a particular object or feature such as, for example, a human face.

In some embodiments, the head-pose determination module <NUM> may determine the pose or position of the user's head based on relative distances between the detected facial features. As the pitch of the user's head increases (e.g., tilts towards the sky), the upper portion of the face becomes compressed while the lower portion of the face becomes more pronounced in a two-dimensional (2D) image. For example, when the user's head is titled back, the pixel distance from the nose to the mouth may be greater than the pixel distance from the nose to the eyes. It is also noted that, as the yaw of the user's head increases (e.g., turns towards the right), the right side of the face becomes compressed while the left side of the face becomes more pronounced in a 2D image. For example, when the user's head is turned to the right, the pixel distance from the nose to one or more features on the left half of the face may be greater than the pixel distance from the nose to similar features (e.g., due to facial symmetry) on the right half of the face. Aspects of the present disclosure recognize that the nose may be used as a point of reference for determining the user's head pose.

The user interface <NUM> may provide an interface through which the user <NUM> can operate, interact with, or otherwise use the sensing device <NUM> or an electronic system (not shown for simplicity) coupled to the sensing device <NUM>. For example, the user interface <NUM> may include one or more speakers, displays, or other media output devices. In some embodiments, the user interface <NUM> may dynamically sense and/or respond to user inputs based, at least in part, on the head pose information generated by the head-pose determination module <NUM>. In some aspects, the user interface <NUM> may respond to user inputs by dynamically updating the display, for example, to navigate a graphical user interface (GUI), display new content, track cursor movement, and the like. In some other aspects, the user interface <NUM> may generate a search query in response to certain user inputs.

To prevent accidental GUI updates and/or unintended searches, the user interface <NUM> may ensure that the user <NUM> is actively engaged with or paying attention to the sensing device <NUM> before processing user inputs from the user <NUM>. In some embodiments, the user interface <NUM> may determine the attentiveness of the user <NUM> based, at least in part, on the head pose information generated by the head-pose determination module <NUM>. For example, the user interface <NUM> may determine that the user <NUM> is likely paying attention (and thus intending to interact with the sensing device <NUM>) if the user <NUM> is facing, looking at, or attending to the sensing device <NUM>. Accordingly, the user interface <NUM> may sense, process, or respond to user inputs only when the user <NUM> is paying attention to the sensing device <NUM>.

As described above, head pose estimation has many uses and applications in computer vision systems and devices. For example, many Internet of Things (IoT) systems implement low-power sensors and/or devices that are also capable of local data processing (e.g., without having to send the data to a server or cloud computing environment). To satisfy the low power requirement, many IoT devices are configured to operate with limited or minimal computational complexity. Further, many head pose estimation techniques are implemented across a wide range of devices with different cameras configurations (e.g., from different manufacturers). To ensure robust performance across myriad devices, aspects of the present disclosure provide a head pose estimation technique that is camera-agnostic.

Among other advantages, the present embodiments provide a robust approach for head pose estimation without the need for camera calibration. In some embodiments, the head-pose determination module <NUM> may estimate a user's head pose without estimating a translation and a rotation vector of the face with respect to the camera. More specifically, in some aspects, the head-pose determination module <NUM> may estimate a user's head pose using only the facial landmark information extracted from an image of the user's face. Accordingly, the present embodiments do not require knowledge of the camera's intrinsic parameters (such as focal length and the principal point), which may vary across sensing devices and/or different instances of the same device. Thus, the present embodiments may be implemented by any camerabased sensing device with no a priori knowledge of the camera's intrinsic parameters.

<FIG> shows a block diagram of a sensing device <NUM>, in accordance with some embodiments. The sensing device <NUM> may be one embodiment of the sensing device <NUM> of <FIG>. The sensing device <NUM> includes a camera <NUM>, a neural network <NUM>, a head-pose estimator <NUM>, and a user interface <NUM>. In the embodiment of <FIG>, the camera <NUM> is depicted as part of the sensing device <NUM>. In other embodiments, the camera <NUM> may be separate from (e.g., coupled to) the sensing device <NUM>.

The camera <NUM> is configured to capture one or more images <NUM> of the environment surrounding the sensing device <NUM>. The camera <NUM> may be one embodiment of one or more of the sensors <NUM> of <FIG>. Thus, the camera <NUM> may be configured to capture images (e.g., still-frame images and/or video) of a scene within a FOV of the sensing device <NUM>. For example, the camera <NUM> may comprise one or more optical sensors (e.g., photodiodes, CMOS image sensor arrays, CCD arrays, and/or any other sensors capable of detecting wavelengths of light in the visible spectrum, the infrared spectrum, and/or the ultraviolet spectrum).

The neural network <NUM> is configured to extract facial landmark data <NUM> from the captured images <NUM>. The neural network <NUM> includes a user detection module <NUM> and a feature extraction module <NUM>. The user detection module <NUM> may detect a presence of one or more users in the captured images <NUM>. For example, the user detection module <NUM> may detect the presence of a user using any known face detection algorithms and/or techniques. In some embodiments, the neural network <NUM> may implement one or more neural network models to generate inferences about human faces in the captured images <NUM>. In some embodiments, the neural network models may be trained to identify one or more facial features (e.g., eyes, nose, mouth, ears, eyebrows, teeth, lips, chin, cheeks, beard, moustache, etc.) from 2D images of a user's face.

The feature extraction module <NUM> may determine one or more feature vectors from the identified user's face. Each of the feature vectors may identify a location (e.g., pixel location) of a respective facial feature in the captured image <NUM>. Feature vectors include a left-eye vector, a right-eye vector, a left-mouth vector, a right-mouth vector, and a nose vector. The left-eye vector describes the location of the user's left eye, the right-eye vector describes the location of the user's right eye, the left-mouth vector describes the location of the left portion of the user's mouth, the right-mouth vector describes the location of the right portion of the user's mouth, and the nose vector describes the location of the user's nose.

The head-pose estimator <NUM> receives the feature vectors (as facial landmark data <NUM>) from the neural network <NUM>. The head-pose estimator <NUM> is configured to generate head pose data <NUM> based on the received feature vectors. In some embodiments, the head-pose estimator <NUM> may estimate the user's head pose based on relative distances between two or more of the feature vectors. For example, in some aspects, the head-pose estimator <NUM> may determine the pose or position of the user's head based on distances from the nose vector to one or more of the remaining feature vectors. The head pose estimator <NUM> may include at least a pitch estimation module <NUM> and a yaw estimation module <NUM>.

The pitch estimation module <NUM> determines a pitch of the user's head based on the relative distances between two or more feature vectors. For example, as the pitch of the user's head increases (e.g., tilts towards the sky), the distance from the nose to the mouth may increase relative to the distance from the nose to the eyes. On the other hand, as the pitch of the user's head decreases (e.g., tilts towards the ground), the distance from the nose to the mouth may decrease relative to the distance from the nose to the eyes. Thus, in some embodiments, the pitch estimation module <NUM> may determine the pitch of the user's head based on the distances from the nose vector to each of a midpoint between the left-eye vector and the right-eye vector and a midpoint between the left-mouth vector and the right-mouth vector.

The yaw estimation module <NUM> determines a yaw of the user's head based on the relative distances between two or more feature vectors. For example, as the yaw of the user's head increases (e.g., turns towards the right), the distance from the nose to a feature on the left side of the face increases relative to the distance from the nose to a similar feature on the right side of the face. On the other hand, as the yaw of the user's head decreases (e.g., turns towards the left), the distance from the nose to a feature on the left side of the face decreases relative to the distance from the nose to a similar feature on the right side of the face. Thus, in some embodiments, the yaw estimation module <NUM> may determine the yaw of the user's head based on the distances form the nose vector to each of a midpoint between the left-eye vector and the left-mouth vector and a midpoint between the right-eye vector and the right-mouth vector.

The user interface <NUM> may control or adjust an operation of the sensing device <NUM> based, at least in part, on the head pose data <NUM>. The user interface <NUM> may be one embodiment of the user interface <NUM> of <FIG>. Thus, the user interface <NUM> may provide an interface through which a user can operate, interact with, or otherwise use the sensing device <NUM> or an electronic system (not shown for simplicity) coupled to the sensing device <NUM>. For example, the user interface <NUM> may include one or more speakers, displays, or other media output devices. In some embodiments, the user interface <NUM> may dynamically sense and/or respond to user inputs based, at least in part, on the head pose data <NUM>.

In some implementations, the sensing device <NUM> may be (part of) a smart speaker. For example, a user may query the smart speaker for information (e.g., recipes, instructions, directions, and the like), to playback media content (e.g., music, videos, audiobooks, and the like), or to control various devices in the user's home or office environment (e.g., lights, thermostats, garage doors, and other home automation devices). Such queries may be triggered by the user's voice, for example, in the form of verbal commands or instructions. To prevent the smart speaker from processing false or accidental voice queries, the smart speaker may be configured to listen and/or respond to the user only when the user is paying attention to the device. In some embodiments, the smart speaker may determine the attentiveness of the user based, at least in part, on the head pose data <NUM>.

In some other implementations, the sensing device <NUM> may be (part of) a smart doorbell. For example, when a visitor approaches a house equipped with a smart doorbell, the smart doorbell records an image and/or video of the visitor and uploads the media to a network or cloud-based service so that the owner of the smart doorbell may view the recorded images and/or video. Many smart doorbells are configured to record the entire scene within it's FOV, which may include one or more passersby. Due to privacy concerns, the smart doorbell may be configured to record only the visitor(s) of the house while obfuscating any faces in the background. In some embodiments, the smart doorbell may determine the faces to be recorded (e.g., the person facing the doorbell) based, at least in part, on the head pose data <NUM>.

Still further, in some implementations, the sensing device may be (part of) a pair of smart glasses. For example, the smart glasses may provide the user (or wearer) with a heads-up display (HUD) and/or augmented reality (AR) which includes information about the scene the user is viewing, which may include person(s) not facing or interacting with the user. The information is generated by sending images and/or video of the scene to a network or cloud-based service. Due to privacy concerns, the smart glasses may be configured to record only the person(s) facing the user and/or within a threshold distance from the user while obfuscating the faces of any person(s) the user may not be interacting with. Moreover, in devices designed to aid the visually-impaired, the smart glasses may be configured to provide the user with a notification that the person(s) facing the user is directly in front, to the left, or to the right of the user. In some embodiments, the smart glasses may determine the faces to be recorded and/or relative locations of the faces based, at least in part, on the head pose data <NUM>.

<FIG> shows a block diagram of a head-pose estimation circuit <NUM>, in accordance with some embodiments. The head-pose estimation circuit <NUM> may be one embodiment of the head-pose estimator <NUM> of <FIG>. The head-pose estimation circuit <NUM> includes a number of midpoint calculators <NUM>(A)-<NUM>(D) (or multiple instances of a single midpoint calculator), a number of distance-to-nose (DTN) calculators <NUM>(A)-<NUM>(D) (or multiple instances of a single DTN calculator), a pitch detector <NUM>, a yaw detector <NUM>, and a distance detector <NUM>.

The head-pose estimation circuit <NUM> is configured to determine a pitch <NUM>, yaw <NUM>, and/or distance <NUM> of a user's head relative to a camera based on facial landmark data <NUM> extracted from an image captured by the camera. The facial landmark data <NUM> may be inferred from the image using one or more neural network models (e.g., as described with respect to <FIG>). In the embodiment of <FIG>, the facial landmark data <NUM> is shown to include a nose vector (n), a left-eye vector (el), a right-eye vector (er), a left-mouth vector (ml), and a right-mouth vector (mr). In other embodiments, the facial landmark data <NUM> may include fewer or more feature vectors including and/or excluding any of those depicted in <FIG>.

Each of the feature vectors n, el, er, ml, and mr indicates a respective point (e.g., pixel or subset of pixels) in the underlying image that is associated with the corresponding facial feature. More specifically, each feature vector may point to a particular location in 2D space. For example, the feature vectors may be described or represented by a set of cartesian coordinates (x, y). Thus, the vector n may point to a location of the user's nose, the vector el may point to a location of the user's left eye, the vector er may point to a location of the user's right eye, the vector ml may point to a location of a left portion (e.g., corner) of the user's mouth, and the vector mr may point to a location of a right portion (e.g., corner) of the user's mouth.

Each of the midpoint calculators <NUM>(A)-<NUM>(D) is configured to receive a different pair of feature vectors (n, el, er, ml, and mr) and calculate a midpoint between them. For example, the first midpoint calculator <NUM>(A) may calculate a midpoint (ce) between the left-eye vector el and the right-eye vector er (e.g., <MAT>), the second midpoint calculator <NUM>(B) may calculate a midpoint (cm) between the left-mouth vector ml and the right-mouth vector mr(e.g., <MAT>), the third midpoint calculator <NUM>(C) may calculate a midpoint (cl) between the left-eye vector el and the left-mouth vector ml (e.g., <MAT>), and the fourth midpoint calculator <NUM>(D) may calculate a midpoint (cr) between the right-eye vector er and the right-mouth vector mr (e.g., <MAT>).

Each of the DTN calculators <NUM>(A)-<NUM>(D) is configured to receive a respective one of the midpoints (ce, cm, cl, and cr) and calculate a distance from the nose vector n to the respective midpoint. For example, the first DTN calculator <NUM>(A) may calculate a distance (de) from the nose vector n to the midpoint ce between the left and right eyes (e.g., de = ∥ce - n∥<NUM>), the second DTN calculator <NUM>(B) may calculate a distance (dm) from the nose vector n to the midpoint cm between the left and right portions of the mouth (e.g., dm = ∥cm - n∥<NUM>), the third DTN calculator <NUM>(C) may calculate a distance (dl) from the nose vector n to the midpoint cl between the left eye and the left portion of the mouth (e.g., dl = ∥cl - n∥<NUM>), the fourth DTN calculator <NUM>(D) may calculate a distance (dr) from the nose vector n to the midpoint cr between the right eye and the right portion of the mouth (e.g., dr = ∥cr - n∥<NUM>).

The pitch detector <NUM> is configured to determine the pitch <NUM> of the user's head based on the distances de and dm. With reference to the example image <NUM> of <FIG>, it can be seen that the distances de and dm have a given ratio when the user is directly facing the camera or sensing device. As the user tilts his or her head backward (e.g., pitch increases), the distance from the nose to the mouth (dm) may increase and/or the distance from the nose to the eyes (de) may decrease. As the user tilts his or her head forward (e.g., pitch decreases), the distance from the nose to the mouth (dm) may decrease and/or the distance from the nose to the eyes (de) may increase. Accordingly, the pitch <NUM> can be expressed as a ratio of the distances de and dm: <MAT>.

The yaw detector <NUM> is configured to determine the yaw <NUM> of the user's head based on the distances dl and dr. With reference to the example image <NUM> of <FIG>, it can be seen that the distances dl and dr have a given ratio when the user is directly facing the camera or sensing device. As the user turns his or her head to the right (e.g., yaw increases), the distance from the nose to the left side of the face (dl) may increase and/or the distance from the nose to the right side of the face (dr) may decrease. As the user turns his or her head to the left (e.g., yaw decreases), the distance from the nose to the left side of the face (dl) may decrease and/or the distance from the nose to the right side of the face (dr) may increase. Accordingly, the yaw <NUM> can be expressed as a ratio of the distances dl and dr: <MAT>.

The distance detector <NUM> is configured to determine the distance <NUM> of the user's head based on the distances de, dm, dl, and dr. With reference to the example image <NUM> of <FIG>, it can be seen that the combined distances de + dm and/or dl + dr are approximately proportional to the distance between the user and the camera. As the user moves further from the camera (e.g., distance increases), the distance from the eyes to the mouth (de + dm) may decrease and/or the distance from the left side of the user's face to the right side of the user's face (dl + dr) may decrease. As the user moves closer to the camera (e.g., distance decreases), the distance between the eyes and the mouth (de + dm) may increase and/or the distance between the left and right sides of the user's face (dl + dr) may increase. However, the pitch of the user's head may affect the overall distance between the eyes and the mouth (de + dm) and the yaw of the user's head may affect the overall distance between the left and right sides of the user's face (dl + dr). Accordingly, the distance <NUM> can be expressed as proportional to a maximum of the combined distances de + dm or dl + dr: <MAT>.

The estimated distance <NUM> represents a relative distance (e.g., proportionality) between the user's head and the camera. For example, the distance value is generally larger when the user is closer to the camera and is generally smaller when the user is further away from the camera. To determine an absolute distance between the user's head and the camera, the estimated distance <NUM> may be multiplied by a scalar quantity associated with a known distance to the camera. In some aspects, the scalar quantity may be predetermined by a manufacturer of the camera and/or sensing device. In some other aspects, the scalar quantity may be estimated by a user of the sensing device. Still further, in some aspects, the scalar quantity may be calibrated during an operation of the sensing device.

For example, the camera may capture an image of the user's head while the user is holding and/or operating (e.g., touching) the sensing device. Aspects of the present disclosure recognize that the user's head will be within a threshold distance (e.g., no more than an arm's length) of the camera when the user is in contact with the sensing device. Thus, in some embodiments, the sensing device may calibrate the scalar quantity using images acquired while the user is operating the sensing device. The scalar quantity may be calculated using any known distance estimation techniques including, but not limited to, estimations based on ear-to-ear measurements, chin-to-hair measurements, and/or the total area of the user's face or head. The scalar quantity may be further updated (e.g., based on a running average) each time the camera captures a new image of the user's head during a subsequent operation of the sensing device. In some embodiments, the scalar quantity may be further used to determine the absolute distance of other objects and/or features in the captured image (e.g., not just the user's head).

The head-pose estimation circuit <NUM> may be configured to generate fewer or more head pose data than what is shown in <FIG> (e.g., pitch <NUM>, yaw <NUM>, and distance <NUM>). For example, in some embodiments, the head-pose estimation circuit <NUM> may further determine a location of the face relative to the camera or sensing device. The location information may be determined based on the location of one or more of the feature vectors (n, el, er, ml, and mr) with respect to the dimensions of the image <NUM>. If the feature vectors are positioned closer toward the left edge than the right edge of the image <NUM>, the user may be located to the right of the camera. If the feature vectors are positioned closer toward the right edge than the left edge of the image <NUM>, the user may be located to the left of the camera. Other head pose data may also be determined using the feature vectors (n, el, er, ml, and mr) without deviating from the scope of this disclosure.

<FIG> shows another block diagram of a sensing device <NUM>, in accordance with some embodiments. The sensing device <NUM> may be one embodiment of the sensing device <NUM> of <FIG> and/or sensing device <NUM> of <FIG>. The sensing device <NUM> includes a sensor interface <NUM>, a processor <NUM>, and a memory <NUM>.

The sensor interface <NUM> may be used to communicate with one or more sensors coupled to the sensing device <NUM>. Example sensors may include, but are not limited to, cameras, capacitive sensors, microphones, and the like. In some implementations, the sensor interface <NUM> may be configured to communicate with a camera of the sensing device <NUM> (e.g., camera <NUM> of <FIG>). For example, the sensor interface <NUM> may transmit signals to, and receive signals from, the camera to capture an image of a scene facing the sensing device <NUM>.

The memory <NUM> may also include a non-transitory computer-readable medium (e.g., one or more nonvolatile memory elements, such as EPROM, EEPROM, Flash memory, a hard drive, etc.) that may store at least the following software (SW) modules:.

Each software module includes instructions that, when executed by the processor <NUM>, cause the sensing device <NUM> to perform the corresponding functions. The non-transitory computer-readable medium of memory <NUM> thus includes instructions for performing all or a portion of the operations described below with respect to <FIG>.

Processor <NUM> may be any suitable one or more processors capable of executing scripts or instructions of one or more software programs stored in the sensing device <NUM>. For example, the processor <NUM> may execute the face detection SW module <NUM> to detect a user in one or more images captured by the camera and extract facial landmark data from the image of the user. The processor <NUM> may also execute the head-pose estimation SW module <NUM> to estimate a pose or position of the user's head relative to the camera based on the facial landmark data. Still further, the processor <NUM> may execute the user interface SW module <NUM> to dynamically control or adjust an operation of the sensing device <NUM>, or an electronic system coupled to the sensing device <NUM>, based at least in part on the head pose data.

In executing the head-pose estimation SW module <NUM>, the processor <NUM> may further execute the pitch estimation sub-module <NUM>, the yaw estimation sub-module <NUM>, the distance estimation sub-module <NUM>, and/or the location estimation sub-module <NUM>. The processor <NUM> may execute the pitch estimation sub-module <NUM> to determine a pitch of the user's head based on the facial landmark data. The processor <NUM> may also execute the yaw estimation sub-module <NUM> to determine a yaw of the user's head based on the facial landmark data. Further, the processor <NUM> may execute the distance estimation sub-module <NUM> to determine a distance of the user's head based on the facial landmark data. Still further, the processor <NUM> may execute the location estimation sub-module <NUM> to determine a location of the user's head based on the facial landmark data.

<FIG> is an illustrative flowchart depicting an example operation <NUM> for determining a head pose of a user by a sensing device, in accordance with some embodiments. With reference for example to <FIG>, the operation <NUM> may be performed by the sensing device <NUM> to determine a pose or position of a user's head relative to the sensing device <NUM>.

The sensing device detects a face in the captured image (<NUM>). The image may include still-frame images and/or videos captured by a camera in, or coupled to, the sensing device. For example, the sensing device may detect the presence of one or more users using any known face detection algorithms and/or techniques. In some embodiments, the neural network <NUM> may implement one or more neural network models to generate inferences about human faces in the captured images <NUM>. In some embodiments, the neural network models may be trained to identify one or more facial features (e.g., eyes, nose, mouth, etc.) from 2D images of a user's face.

The sensing device also identifies a plurality of points in the image corresponding to respective features of the detected face (<NUM>). In some embodiments, the plurality of points may include at least a first point corresponding to a location of the user's nose. For example, the sensing device may determine one or more feature vectors from the identified user's face. Each of the feature vectors may identify a location (e.g., pixel location) of a respective facial feature in the captured image <NUM>. Example feature vectors may include, but are not limited to, a left-eye vector, a right-eye vector, a left-mouth vector, a right-mouth vector, and a nose vector.

The sensing device may then determine a position of the face relative to the sensing device based, at least in part, on a distance between the first point in the image and one or more of the remaining points (<NUM>). In some embodiments, the sensing device may determine the pitch of the user's head based on the distances from the nose vector to each of a midpoint between the left-eye vector and the right-eye vector and a midpoint between the left-mouth vector and the right-mouth vector. In some other embodiments, the sensing device may determine the yaw of the user's head based on the distances from the nose vector to each of a midpoint between the left-eye vector and the left-mouth vector and a midpoint between the right-eye vector and the right-mouth vector.

<FIG> is an illustrative flowchart depicting an example operation <NUM> for determining the pitch of a face in a captured image, in accordance with some embodiments. With reference for example to <FIG>, the operation <NUM> may be performed by the head-pose estimation circuit <NUM>.

The head-pose estimation circuit may first receive facial landmark data (<NUM>). In some embodiments, the facial landmark data may include at least a nose vector (n), a left-eye vector (el), a right-eye vector (er), a left-mouth vector (ml), and a right-mouth vector (mr). Each of the feature vectors n, el, er, ml, and mr indicates a respective point in the underlying image that is associated with the corresponding facial feature.

The head-pose estimation circuit may calculate a first midpoint (ce) between the left and right eyes in the image (<NUM>). For example, as shown in <FIG>, the first midpoint calculator <NUM>(A) may calculate the midpoint ce between the left-eye vector el and the right-eye vector er (e.g., <MAT>).

The head-pose estimation circuit may calculate a second midpoint (cm) between the left and right portions of the mouth in the image (<NUM>). For example, as shown in <FIG>, the second midpoint calculator <NUM>(B) may calculate the midpoint cm between the left-mouth vector ml and the right-mouth vector mr (e.g., <MAT>).

The head-pose estimation circuit may further calculate a distance (de) from the nose in the image to the first midpoint ce (<NUM>). For example, as shown in <FIG>, the first DTN calculator <NUM>(A) may calculate the distance de from the nose vector n to the midpoint ce between the left and right eyes (e.g., de = ∥ce - n∥<NUM>).

The head-pose estimation circuit may also calculate a distance (dm) from the nose in the image to the second midpoint cm (<NUM>). For example, as shown in <FIG>, the second DTN calculator <NUM>(B) may calculate the distance dm from the nose vector n to the midpoint cm between the left and right portions of the mouth (e.g., dm = ∥cm - n∥<NUM>).

The head-pose estimation circuit may then determine the pitch of the face based on a ratio of the distances de and dm (<NUM>). For example, as shown in <FIG>, the pitch detector <NUM> may determine the pitch of the user's head or face by comparing the distance from the nose to the mouth dm with the distance from the nose to the eyes de: <MAT>.

<FIG> is an illustrative flowchart depicting an example operation <NUM> for determining the yaw of a face in a captured image, in accordance with some embodiments. With reference for example to <FIG>, the operation <NUM> may be performed by the head-pose estimation circuit <NUM>.

The head-pose estimation circuit may first receive facial landmark data (<NUM>). In some embodiments, the facial landmark data may include at least a nose vector (n, a left-eye vector (el), a right-eye vector (er), a left-mouth vector (ml), and a right-mouth vector (mr). Each of the feature vectors n, el, er, ml, and mrindicates a respective point in the underlying image that is associated with the corresponding facial feature.

The head-pose estimation circuit may calculate a first midpoint (cl) between the left eye in the image and the left portion of the mouth (<NUM>). For example, as shown in <FIG>, the third midpoint calculator <NUM>(C) may calculate the midpoint cl between the left-eye vector el and the left-mouth vector ml (e.g., <MAT>).

The head-pose estimation circuit may calculate a second midpoint (cr) between the right eye in the image and the right portion of the mouth (<NUM>). For example, as shown in <FIG>, the fourth midpoint calculator <NUM>(D) may calculate the midpoint cr between the right-eye vector er and the right-mouth vector mr (e.g., <MAT>).

The head-pose estimation circuit may further calculate a distance (dl) from the nose in the image to the first midpoint cl (<NUM>). For example, as shown in <FIG>, the third DTN calculator <NUM>(C) may calculate the distance dl from the nose vector n to the midpoint cl between the left eye and the left portion of the mouth (e.g., dl = ∥cl - n∥<NUM>).

The head-pose estimation circuit may also calculate a distance (dr) from the nose in the image to the second midpoint cr (<NUM>). For example, as shown in <FIG>, the fourth DTN calculator <NUM>(D) may calculate the distance dr from the nose vector n to the midpoint cr between the right eye and the right portion of the mouth (e.g., dr = ∥cr - n∥<NUM>).

The head-pose estimation circuit may then determine the yaw of the face based on a ratio of the distances dl and dr (<NUM>). For example, as shown in <FIG>, the yaw detector <NUM> may determine the pitch of the user's head or face by comparing the distance from the nose to the left side of the face dl with the distance from the nose to the right side of the face dr: <MAT>.

<FIG> is an illustrative flowchart depicting an operation <NUM> for determining the distance of a face in a captured image, in accordance with the present invention. With reference for example to <FIG>, the operation <NUM> may be performed by the head-pose estimation circuit <NUM>.

The head-pose estimation circuit first receive facial landmark data (<NUM>). The facial landmark data may include at least a nose vector (n, a left-eye vector (el), a right-eye vector (er), a left-mouth vector (ml), and a right-mouth vector (mr). Each of the feature vectors n, el, er, ml, and mr indicates a respective point in the underlying image that is associated with the corresponding facial feature.

The head-pose estimation circuit calculates a first midpoint (ce) between the left and right eyes in the image (<NUM>). For example, as shown in <FIG>, the first midpoint calculator <NUM>(A) calculates the midpoint ce between the left-eye vector el and the right-eye vector er (e.g., <MAT>).

The head-pose estimation circuit calculates a second midpoint (cm) between the left and right portions of the mouth in the image (<NUM>). For example, as shown in <FIG>, the second midpoint calculator <NUM>(B) calculates the midpoint cm between the left-mouth vector mland the right-mouth vector mr (e.g., <MAT>).

The head-pose estimation circuit calculates a third midpoint (cl) between the left eye in the image and the left portion of the mouth (<NUM>). For example, as shown in <FIG>, the third midpoint calculator <NUM>(C) calculates the midpoint cl between the left-eye vector eland the left-mouth vector ml (e.g., <MAT>).

The head-pose estimation circuit calculates a fourth midpoint (cr) between the right eye in the image and the right portion of the mouth (<NUM>). For example, as shown in <FIG>, the fourth midpoint calculator <NUM>(D) calculates the midpoint cr between the right-eye vector er and the right-mouth vector mr (e.g., <MAT>).

The head-pose estimation circuit further calculates respective distances (de, dm, dl, and dr) from the nose in the image to each of the midpoints ce, cm, cl, and cr (<NUM>). For example, as shown in <FIG>, each of the DTN calculators <NUM>(A)-<NUM>(D) may receive the nose vector n and a respective one of the midpoints ce, cm, cl, and cr, and may further calculate a respective one of the distances de, dm, dl, and dr (e.g., de = ∥ce - n∥<NUM>, dm = ∥cm - n∥<NUM>, dl = ∥cl - n∥<NUM>, dr = ∥cr - n∥<NUM>).

The head-pose estimation circuit then determines the distance of the face based on the greater of the combined distances de + dm or dl + dr (<NUM>). For example, as shown in <FIG>, the distance detector <NUM> may determine the distance of the user's head or face by comparing the distance between the eyes and the mouth (de + dm) with the distance between the left and right sides of the user's face (dl + dr): <MAT>.

Skilled artisans may implement the described functionality in varying ways for each particular application, but such implementation decisions should not be interpreted as causing a departure from the scope of the invention as defined in the claims.

Claim 1:
A method of determining a head pose of a user by a sensing device, comprising:
detecting a face of the user in an image captured by the sensing device;
identifying a plurality of points in the image corresponding to respective features of the detected face, the plurality of points including at least a first point corresponding to a location of a first facial feature, a left eye, a right eye, a left portion of a mouth, and a right portion of the mouth, wherein the first facial feature is a nose and the remaining points correspond to locations of a mouth or eyes; and
determining a position of the face relative to the sensing device based at least in part on a distance,
wherein the determining comprises:
calculating a first midpoint between the location of the left eye and the location of the right eye;
calculating a second midpoint between the location of the left portion of the mouth and the location of the right portion of the mouth;
calculating a third midpoint between the location of the left eye and the location of the left portion of the mouth;
calculating a fourth midpoint between the location of the right eye and the location of the right portion of the mouth;
calculating a first combined distance being a sum of the distances from the first point to each of the first midpoint and the second midpoint;
calculating a second combined distance being a sum of the distances from the first point to each of the third midpoint and the fourth midpoint; and
determining the distance of the face relative to the sensing device as a proportion of the greater of the first combined distance or the second combined distance.