Visibility of Frames

In some examples, an electronic device includes an image sensor and a processor to determine a facial landmark count of a frame captured by the image sensor and adjust, responsive to the facial landmark count, a visibility of the frame.

BACKGROUND

Electronic devices such as notebooks, laptops, desktops, tablets, and smartphones include image sensors that enable the electronic devices to capture images. During a videoconference, an electronic device utilizes an image sensor to enable a user to share audiovisual data with an audience, for instance.

DETAILED DESCRIPTION

As described above, electronic devices include image sensors that enable the electronic devices to share video signals with an audience. In some instances, the user turns or moves away from the image sensor to attend to another task and the image sensor captures background features (e.g., undesirable angles of the user, objects not intended for display, etc.) that distracts the audience and embarrasses the user, reducing an effectiveness of communication between the user and the audience and impacting user productivity.

To reduce distractions for the audience and enhance communication between the user and the audience by preventing unwanted transmissions of background features, an electronic device adjusts a visibility of a video signal based on a facial landmark count within a frame of the video signal. Visibility, as used herein, is an opacity of the frame such that 0% opacity indicates contents of the frame are viewable and 100% opacity indicates contents of the frame are not viewable. The electronic device receives the frame, determines an aspect ratio of the frame to determine a height and a width of the frame, and decomposes the frame to detect facial features. The height and width of the frame is expressible in units of pixels that each have unique coordinates in a two-dimensional plane (e.g., an x-y plane). The facial features include a forehead, eyebrows, eyes, a nose, lips, a hairline, cheekbones, a jawline, or a combination thereof. The electronic device determines the facial landmark count by determining whether facial landmark points of the facial features are within the frame. A facial landmark point, as used herein, is a coordinate associated with a facial feature. The facial landmark count is determinable by comparing the coordinate associated with a facial landmark point to the coordinates of the pixels of the frame. Responsive to a determination that the facial landmark count is less than a threshold, the electronic device adjusts the visibility of the frame. In some examples, the electronic device adjusts the visibility by an amount determined by a multiplier that is based on the facial landmark count such that the amount of the adjustment increases as the facial landmark count decreases. In various examples, the electronic device mutes a microphone as well as adjusts the visibility of the frame. The electronic device causes a display device to display the frame having the adjusted visibility, a network interface to transmit the frame having the adjusted visibility, or a combination thereof.

Utilizing the electronic device that adjusts the visibility of the frame based on the facial landmark count within the frame provides for an enhanced user and audience experience by reducing distractions for the audience. Enhancing the effectiveness of communication between the user and the audience enhances user productivity.

In some examples in accordance with the present description, an electronic device is provided. The electronic device includes an image sensor and a processor. The processor is to determine a facial landmark count of a frame captured by the image sensor and adjust, responsive to the facial landmark count, a visibility of the frame.

In other examples in accordance with the present description, an electronic device is provided. The electronic device includes an image sensor and a processor. The processor is to determine whether a facial landmark count of a frame captured by the image sensor exceeds a threshold. Responsive to a determination that the facial landmark count is less than the threshold, the processor is to adjust a visibility of the frame and cause a display device to display the frame that has the adjusted visibility. Responsive to a determination that the facial landmark count exceeds the threshold, the processor is to cause the display device to display the frame.

In yet other examples in accordance with the present description, a non-transitory machine-readable medium storing machine-readable instructions is provided. The machine-readable instructions, when executed by a processor of an electronic device, cause the processor to determine a facial landmark count of a frame captured by an image sensor, adjust a visibility of the frame by a multiplier that is based on the facial landmark count, and cause a display device to display the frame, a network interface to transmit the frame, or a combination thereof.

Referring now toFIG.1, a flow diagram depicting a method100for an electronic device to control visibility of frames is provided, in accordance with various examples. The method100includes a start point102at which the electronic device starts a process for processing a frame captured by an image sensor. During a receive process104of the method100, the electronic device receives the frame from the image sensor. The electronic device determines an aspect ratio of the frame during a determine process106of the method100. The electronic device decomposes the frame during a decompose process108of the method100. During a detect process110of the method100, the electronic device detects facial features of the decomposed frame. Utilizing the detected facial features, the electronic device determines a facial landmark count during a determine process112of the method100. At a decision point114of the method100, the electronic device determines whether the facial landmark count exceeds a threshold. Responsive to a determination that the facial landmark count does not exceed the threshold (e.g., the facial landmark count is less than the threshold), the electronic device adjusts a visibility of the frame during an adjust process116of the method100. The electronic device causes a display device to display the frame having the adjusted visibility during a display process118. Responsive to a determination that the facial landmark count exceeds the threshold (e.g., the facial landmark count is equivalent to or greater than the threshold), the electronic device causes the display device to display the frame as received during the receive process104.

In some examples, the start point102occurs in response to an application requesting access to an image sensor. The application is a videoconferencing application, for example. As described herein, the terms “application,” “software,” and “firmware” are considered to be interchangeable in the context of the examples provided. “Firmware” is considered to be machine-readable instructions that a processor of the electronic device executes prior to execution of the operating system (OS) of the electronic device, with a small portion that continues after the OS bootloader executes (e.g., a callback procedure). “Application” and “software” are considered broader terms than “firmware,” and refer to machine-readable instructions that execute after the OS bootloader starts, through OS runtime, and until the electronic device shuts down. “Application,” “software,” and “firmware,” as used herein, are referred to as executable code.

During the receive process104, the electronic device intercepts the video signal of the image sensor. In various examples, the electronic device intercepts the video signal in response to an application requesting access to the image sensor, a determination that the video signal includes a user presence, or a combination thereof.

During the determine process106, the electronic device determines a width and a height of the frame received during the receive process104. In some examples, the electronic device determines the aspect ratio as a ratio of the width to the height of the frame. In various examples, the aspect ratio is 1.43:1, 4:3, 9:16, 16:9, 16:10, 18:9, 18.5:9, 21:9, or any other comparable digital aspect ratio utilized by image sensors. In other examples, the electronic device determines the aspect ratio as a ratio of the width in pixels to the height in pixels. For example, the aspect ratio is 640:480, 1280:780, 1920:1080, 2048:1080, 2560:1440, or any other comparable pixel aspect ratio utilized by image sensors. The electronic device determines coordinates of the frame captured by the image sensor based on the width and height, for example.

In various examples, during the decompose process108, the electronic device decomposes the frame utilizing a pre-processing technique. Decomposing, as used herein, reduces objects to edge-like structures. Examples of pre-processing techniques include grayscaling, blurring, sharpening, thresholding, or a combination thereof. Grayscaling, as used herein, converts the frame to gray tonal values. Blurring, as used herein, utilizes a low pass filter to remove noise, or outlier pixels, from the frame. Sharpening, as used herein, enhances a contrast along edges of objects of the frame. Thresholding, as used herein, separates the frame into foreground values of black and background values of white. Post-thresholding, the frame is reduced to edge-like structures denoted by black pixels, for example.

To detect the facial features during the detect process110, the electronic device utilizes a face detection technique to detect facial features of the user in the frame received during the receive process104. In some examples, the electronic device utilizes a face detection technique to determine whether low intensity regions of the decomposed frame include facial features during the detect process110. The low intensity regions are areas of the frame that have similar intensity values such that darkened faces or features are difficult to detect. The low intensity region is a shadowed area captured by the image sensor, for example. The electronic device utilizes the locations of the facial features to determine coordinates for facial landmark points. For example, a first facial feature is an eyebrow. The electronic device determines coordinates that overlap the location of the eyebrow, that are contiguous to the location of the eyebrow, or a combination thereof. The electronic device increments a facial landmark count for each facial landmark point having coordinates within the width and height of the frame determined during the determine process106. Within, as used herein, includes coordinate inside the boundary defined by the width and height of the frame in terms of pixels, as well as coordinates of the boundary.

In various examples, at the decision point114, the electronic device determines whether the facial landmark count exceeds a threshold. Exceeds a threshold, as used herein, includes a value that is equivalent to the threshold as well as values that are greater than the threshold. The threshold is a percentage of a total number of facial landmark points, for example. In some examples, the electronic device determines the total number of facial landmark points by extrapolating from the coordinates of the facial landmark points and an approximate symmetry of facial features. In an example, the total number of facial landmark points is 400, the facial landmark count is 250, and the threshold is 300, or 75% of 400. Thus, the electronic device determines that the facial landmark count of 250 does not exceed the threshold of 300. In some examples, the threshold is adjustable utilizing a graphical user interface (GUI).

While in the examples described above the electronic device performs the determine process106and the decompose process108simultaneously, in other examples, the electronic device performs the determine process106and the decompose process108sequentially. While in the examples described above the electronic device performs the decompose process108, the detect process110, and the determine process112, sequentially, in other examples, the electronic device performs the decompose process108, the detect process110, and the determine process112concurrently. For example, the electronic device utilizes a machine learning technique to decompose the frame, detect the facial features, determine the facial landmark count, or a combination thereof. To decompose the frame, the machine learning technique compares the facial features to multiple templates to determine that the features indicate a face. In various examples, the electronic device utilizes a machine learning technique that implements a convolution neural network (CNN) to determine whether the image includes the face. The CNN is trained with a training set that includes multiple images of multiple users, for example. The multiple images include users having different facial positions. Utilizing the trained CNN, the electronic device identifies facial features of the image, determines facial landmark points of the facial features, and determines the facial landmark count. In some examples, the CNN implements a Visual Geometry Group (VGG) network, a Residual Network (ResNet) network, a SqueezeNet network, an AlexNet network, or a LeNet network. The electronic device utilizes the total number of facial landmark points of the trained CNN to determine the threshold, in some examples. For example, the trained CNN has a total number of facial landmark points (e.g., 68, 98, 468) based on a model utilized. In another example, the machine learning technique utilizes a pipeline that includes a face detector and a facial landmark model to detect the facial features and determine the facial landmark count. The machine learning technique utilizing the pipeline is MediaPipe Face Mesh or a comparable face geometry solution, for example. The machine learning pipeline determines a three-dimensional (3D) surface geometry of a face represented in the frame. The 3D surface geometry utilizes a grid to generate coordinates of facial landmark points and to determine a total number of facial landmark points.

In some examples, during the adjust process116, the electronic device generates a duplicate of the frame received during the receive process104and adjusts the visibility of the duplicate frame. During the display process118, the electronic device causes the display device to display the duplicate frame. In other examples, the electronic device utilizes post-processing techniques to adjust the visibility of the frame received during the receive process104. To adjust the visibility of the frame, the post-processing techniques adjust a brightness, a contrast, a saturation, a hue, a fade, a vignette, or a combination thereof of the frame, for example. During the display process118, the electronic device causes the display device to display the post-processed frame.

Referring now toFIG.2, a frame202utilized by an electronic device for controlling visibility of frames is provided, in accordance with various examples. The frame202includes an area204of an environment200. The environment200includes areas204,206and a user208. The user208has facial features210and facial landmark points212. The facial features210include eyes, eyebrows, a nose, and a mouth, for example. The facial landmark points212include coordinates that outline the facial features210, for example.

In various examples, the frame202outlines a field of view of an image sensor. The field of view of the image sensor captures the area204. The area204includes a partial face of the user208. The electronic device receives the frame202from the image sensor and determines an aspect ratio of the frame202. The electronic device decomposes the frame202to detect the facial features210of the frame202. Utilizing the detected facial features210, the electronic device determines a facial landmark count of the facial landmark points212of the frame202. The electronic device determines whether the facial landmark count exceeds a threshold. Responsive to a determination that the facial landmark count does not exceed the threshold, the electronic device adjusts a visibility of the frame202and causes a display device to display the frame202having the adjusted visibility. Responsive to a determination that the facial landmark count exceeds the threshold, the electronic device causes the display device to display the frame202as received from the image sensor.

Referring now toFIG.3, a schematic diagram depicting an electronic device300for controlling visibility of frames is provided, in accordance with various examples. The electronic device300is a desktop, a laptop, a notebook, a tablet, a smartphone, or any other suitable computing device for receiving and processing frames. The electronic device300includes a processor302, an image sensor304, and a storage device308. The processor302is a microprocessor, a microcomputer, a microcontroller, or another suitable processor or controller for managing operations of the electronic device300. The processor302is a central processing unit (CPU), graphics processing unit (GPU), system on a chip (SoC), image signal processor (ISP), or a field programmable gate array (FPGA), for example. The image sensor304is an internal camera, an external camera, or any other suitable device for capturing an image, recording a video signal, or a combination thereof. The storage device308includes a hard drive, solid state drive (SSD), flash memory, random access memory (RAM), or other suitable memory for storing data or executable code of the electronic device300.

While not explicitly shown, the electronic device300includes network interfaces, video adapters, sound cards, local buses, peripheral devices (e.g., a keyboard, a mouse, a touchpad, a speaker, a microphone, a display device), wireless transceivers, connectors, or a combination thereof. While the image sensor304is shown as an integrated image sensor of the electronic device300, in other examples, the image sensor304couples to any suitable connection for enabling communications between the electronic device300and the image sensor304. The connection may be via a wired connection (e.g., a Universal Serial Bus (USB)) or via a wireless connection (e.g., BLUETOOTH®, Wi-Fi®). In some examples, the display device is an integrated display device of the electronic device300. In other examples, the display device couples to any suitable connection for enabling communications between the electronic device300and the display device. The display device is a liquid crystal display (LCD), a light-emitting diode (LED) display, a plasma display, a quantum dot (QD) display, or any suitable device for displaying data of the electronic device300for viewing. The connection may be via a wired connection (e.g., USB, a High-Definition Multimedia Interface (HDMI) connector, or a Video Graphics Array (VGA) connector, Digital Visual Interface (DVI)) or via a wireless connection (e.g., BLUETOOTH®, Wi-Fi®).

In some examples, the processor302couples to the image sensor304, and the storage device308. The storage device308stores machine-readable instructions which, when executed by the processor302, cause the processor302to perform some or all of the actions attributed herein to the processor302. The machine-readable instructions are the machine-readable instructions310,312.

In various examples, the machine-readable instructions310,312, when executed by the processor302, cause the processor302to adjust a visibility of a frame (e.g., the frame202). The machine-readable instruction310, when executed by the processor302, causes the processor302to determine a facial landmark count. Based on the facial landmark count, the machine-readable instruction312, when executed by the processor302, causes the processor302to adjust the visibility of the frame.

For example, the machine-readable instruction310, when executed by the processor302, causes the processor302to determine a facial landmark count of the frame captured by the image sensor304. Responsive to determining the facial landmark count, the machine-readable instruction312, when executed by the processor302, causes the processor302to adjust the visibility of the frame. The processor302adjusts the visibility incrementally based on the facial landmark count relative to the total number of facial landmark points (e.g., the facial landmark points212), for example.

Executing machine-readable instructions, the processor302implements the method100in some examples. For example, in response to an application requesting access to an image sensor, a machine-readable instruction (not explicitly shown), when executed by the processor302, causes the processor302to intercept a video signal of the image sensor304. Another machine-readable instruction (not explicitly shown), when executed by the processor302, causes the processor302to isolate a frame of the video signal. Executing another machine-readable instruction (not explicitly shown) causes the processor302to determine an aspect ratio of the frame. Executing another machine-readable instruction (not explicitly shown) causes the processor302to decompose the frame. The processor302implements a technique described above with respect toFIG.1to decompose the frame, for example. Another machine-readable instruction (not explicitly shown), when executed by the processor302, causes the processor302to detect facial features (e.g., the facial features210). The processor302implements a technique described above with respect toFIG.1to detect the facial features, for example. The machine-readable instruction310, when executed by the processor302, causes the processor302to determine the facial landmark count by comparing coordinates of facial landmark points (e.g., the facial landmark points212) to coordinates of pixels of the frame captured the image sensor304, as described above with respect toFIG.1, for example. Responsive to a determination that coordinates of a facial landmark point are within the frame captured by the image sensor304, the processor302increments the facial landmark count. The machine-readable instruction312, when executed by the processor302, causes the processor302to determine whether the facial landmark count exceeds a threshold and to adjust, based on a result of the determination, the visibility of the frame, as described above with respect toFIG.1.

In some examples, the processor302adjusts the visibility of the frame by a multiplier that is based on the facial landmark count. For example, responsive to the facial landmark count of 250 and a total number of facial landmark points of 400, the processor302determines the multiplier is 250 divided by 400, or 0.625. The processor302reduces the visibility of the frame by 0.375. In various examples, the processor302adjusts the visibility of the frame by the multiplier rounded to a nearest tenth or hundredth. For example, the processor302adjusts the visibility of the frame by 0.38 or 0.4.

In other examples, responsive to a number of subsequent frames that have a similar multiplier (e.g., multipliers are within a tolerance of each other) and that is less than a delay threshold, the processor302replaces the frame with a static image, a video, or a combination thereof. The number of subsequent frames is based on an elapsed time, for example. The number of subsequent frames is based on a frame rate of the image sensor304, for example. Responsive to a frame rate of 30 frames per second and an elapsed time of 30 seconds, the processor302replaces the frame with a static image, a video, or a combination thereof after a delay threshold of 900 (e.g., 30×30). In various examples, the delay threshold, the number of subsequent frames, the elapsed time, or a combination thereof are adjustable utilizing a GUI.

In various examples, the processor302causes the display device (not explicitly shown) to display the frame having the adjusted visibility, a network interface (not explicitly shown) to transmit the frame having the adjusted visibility, or a combination thereof. In some examples, the processor302causes the display device (not explicitly shown) to display the frame captured by the image sensor304and a network interface to transmit the frame having the adjusted visibility.

Utilizing the electronic device300that adjusts the visibility of the frame based on the facial landmark count within the frame provides for an enhanced user and audience experience by reducing distractions for the audience. Enhancing the effectiveness of communication between the user and the audience enhances user productivity.

Referring now toFIG.4, a flow diagram depicting of a method400for an electronic device (e.g., the electronic device300) to control visibility of frames is provided, in accordance with various examples. The method400includes a start point402during which the electronic device starts processing a frame (e.g., the frame202) captured by an image sensor (e.g., the image sensor304). During a receive process404of the method400, the electronic device receives the frame from the image sensor. The electronic device determines an aspect ratio of the frame during a determine process406of the method400. The electronic device decomposes the frame during a decompose process408of the method400. During a detect process410of the method400, the electronic device detects facial features of the decomposed frame. Utilizing the detected facial features, the electronic device determines whether a facial landmark point is detected at a decision point412of the method400. Responsive to a determination that the facial landmark point is not detected at the decision point412, the electronic device determines whether the frame has been processed at a decision point414of the method400. Responsive to a determination that the frame has not been processed at the decision point414, the electronic device returns to the decision point412to determine whether another facial landmark point is detected.

Responsive to a determination at the decision point412that the facial landmark point is detected, the electronic device determines whether the facial landmark point is within the aspect ratio at a decision point416of the method400. Responsive to a determination that the facial landmark point is not within coordinates of pixels of the frame during the decision point416, the electronic device returns to the decision point412to determine whether another facial landmark point is detected. Responsive to a determination at the decision point416that the facial landmark point is within the coordinates of pixels of the frame, the electronic device increases a facial landmark count during an increase process418. At a decision point420of the method400, the electronic device determines whether the facial landmark count exceeds a threshold. Responsive to a determination at the decision point420that the facial landmark count exceeds the threshold, the electronic device causes a display device to display the frame received during the receive process404. Responsive to a determination at the decision point420that the facial landmark count does not exceed the threshold, the electronic device returns to the decision point414to determine whether the frame is processed. Responsive to a determination at the decision point414that the frame is processed, the electronic device adjusts a visibility of the frame during an adjust process424of the method400. The electronic device causes a display device to display the frame having the adjusted visibility during the display process422. By utilizing the method400, the electronic device reduces an amount of time to process the frame in examples in which the facial landmark count exceeds the threshold before the frame is processed.

Referring now toFIG.5, a schematic diagram depicting an electronic device500for controlling visibility of frames is provided, in accordance with various examples. The electronic device500is the electronic device300, for example. The electronic device500includes a processor502, an image sensor504, a display device506, and a storage device508. The processor502is the processor302, for example. The image sensor504is the image sensor304, for example. The display device506is a liquid crystal display (LCD), a light-emitting diode (LED) display, a plasma display, a quantum dot (QD) display, or any suitable device for displaying data of the electronic device300for viewing. The storage device508is the storage device308, for example.

In some examples, the processor502couples to the image sensor504, the display device506, and the storage device508. The storage device508stores machine-readable instructions which, when executed by the processor502, cause the processor502to perform some or all of the actions attributed herein to the processor502. The machine-readable instructions are the machine-readable instructions510,512,514.

In various examples, the machine-readable instructions510,512,514, when executed by the processor502, cause the processor502to adjust a visibility of a frame (e.g., the frame202). The machine-readable instruction510, when executed by the processor502, causes the processor502to determine whether a facial landmark count exceeds a threshold. Responsive to the facial landmark count being less than the threshold, the machine-readable instruction512, when executed by the processor502, causes the processor502to adjust the visibility of the frame and cause the display of the frame. Responsive to the facial landmark count exceeding the threshold, the machine-readable instruction514, when executed by the processor502, causes the processor502to cause the display of the frame.

For example, the machine-readable instructions510,512,514, when executed by the processor502, cause the processor502to adjust a visibility of a frame (e.g., the frame202). The machine-readable instruction510, when executed by the processor502, causes the processor502to determine whether a facial landmark count of the frame captured by the image sensor504exceeds a threshold. Responsive to a determination that the facial landmark count is less than the threshold, the machine-readable instruction512, when executed by the processor502, causes the processor502to adjust a visibility of the frame and cause the display device506to display the frame that has the adjusted visibility. Responsive to a determination that the facial landmark count exceeds the threshold, the machine-readable instruction514, when executed by the processor502, causes the processor502to cause the display device506to display the frame captured by the image sensor504.

In some examples, responsive to a determination that the facial landmark count is less than the threshold, the processor502causes a network interface (not explicitly shown) to transmit the frame that has the adjusted visibility. Responsive to a determination that the facial landmark count exceeds the threshold, the processor502causes the network interface (not explicitly shown) to transmit the frame captured by the image sensor504.

As described above with respect toFIG.3, in some examples, the processor502adjusts the visibility of the frame by a multiplier that is based on the facial landmark count. As described with respect toFIGS.1and6, the threshold is adjustable utilizing a GUI, in various examples.

Referring now toFIG.6, a graphical user interface (GUI)600of an electronic device (e.g., the electronic device300,500) for controlling visibility of frames is provided, in accordance with various examples. The GUI600includes a first threshold602, a second threshold604, and a third threshold606. The GUI600enables a user of the electronic device to adjust the first threshold602, the second threshold604, the third threshold606, or a combination thereof. For example, the GUI600is a slider that represents a range of fade from zero to 100. The first threshold602is adjusted to a value that defines an upper range (e.g., 66 to 100), the second threshold604is adjusted to a value that defines a first middle range (e.g., 45 to 65) and a second middle range (e.g., 26 to 44), and the third threshold606is adjusted to a value that defines a lower range (e.g., zero to 25). The upper range has a fade represented by a white background, the first middle range has a fade represented by a black-dotted white background, the second middle range has a fade represented by a gray background, and the lower range has a fade represented by a black background, for example. The white background represents no fade (e.g., an opacity of the frame of 100%) and the black background represents a fade to black (e.g., an opacity of the frame of 0%), for example.

In some examples, a processor (e.g., the processor302,502) of the electronic device determines whether a first facial landmark count of a first frame (e.g., the frame202) captured by the image sensor (e.g., the image sensor304,504) exceeds the first threshold602. Responsive to a determination that the first facial landmark count is less than the first threshold602, the processor adjusts a first visibility of the first frame and causes a display device (e.g., the display device506) to display the frame that has the first visibility. In various examples, the processor calculates a percentage by dividing the first facial landmark count by a total number of facial landmark points, compares the percentage to the first middle range, the second middle range, and the lower range to determine a range of the percentage, and adjusts the visibility of the frame by applying the fade associated with the range of the percentage. Responsive to a determination that the first facial landmark count exceeds the first threshold602, the processor causes the display device to display the frame captured by the image sensor.

In various examples, the processor determines whether a second facial landmark count of a second frame captured by the image sensor exceeds the second threshold604. Responsive to a determination that the second facial landmark count is less than the second threshold604, the processor adjusts a second visibility of the second frame and causes the display device to display the second frame that has the second visibility. Responsive to a determination that the second facial landmark count exceeds the second threshold604, the processor determines whether the second facial landmark count exceeds the first threshold602. Responsive to a determination that the second facial landmark count is less than the first threshold602, the processor adjusts the second visibility of the second frame to the first visibility of the first frame and causes the display device to display the second frame that has the second visibility. Responsive to a determination that the second facial landmark count exceeds the first threshold602, the processor causes the display device to display the second frame captured by the image sensor.

In other examples, responsive to a determination that the facial landmark count is less than the first threshold602, the processor reduces the visibility of the frame by 25%. Responsive to a determination that the facial landmark count is less than the second threshold604, the electronic device reduces the visibility of the frame by 50%. Responsive to a determination that the facial landmark count is less than the third threshold606, the electronic device reduces the visibility of the frame by 100%.

Referring now toFIGS.7A and7B, schematic diagrams depicting a display device700of an electronic device (e.g., the electronic device300,500) for controlling visibility of frames702,704are provided, in accordance with various examples.FIG.7Ashows the display device700displaying a frame702that includes a user706.FIG.7Bshows the display device700displaying a frame704of the user706. The frame704is a subsequent frame to the frame702, for example.

In various examples, a processor (e.g., the processor302,502) of the electronic device determines that a facial landmark count of the frame702exceeds a threshold and causes the display device700to display the frame702captured by an image sensor (e.g.,304,504). The visibility of the frame702is 100% or has no visibility adjustments, for example. The processor determines that a facial landmark count of the frame704is less than the threshold and causes the display device700to display the frame704having an adjusted visibility, as represented by the black-dotted white background. The adjusted visibility is based on the facial landmark count as described above with respect toFIG.1, a multiplier as described above with respect toFIG.3, a comparison to ranges established by multiple thresholds as described above with respect toFIG.6, or a combination thereof.

Referring now toFIG.8, a schematic diagram depicting an electronic device800for controlling visibility of frames is provided, in accordance with various examples. The electronic device800is the electronic device300,500, for example. The electronic device800includes a processor802and a non-transitory machine-readable medium804. The processor802is the processor302,502, for example. The non-transitory machine-readable medium804is the storage device308,508, for example. The term “non-transitory” does not encompass transitory propagating signals.

In various examples, the processor802couples to the non-transitory machine-readable medium804. The non-transitory machine-readable medium804stores machine-readable instructions. The machine-readable instructions are the machine-readable instructions806,808,810. The machine-readable instructions806,808,810, when executed by the processor802, cause the processor802to perform some or all of the actions attributed herein to the processor802.

In some examples, when executed by the processor802, the machine-readable instructions806,808,810cause the processor802to adjust a visibility of a frame (e.g., the frame202,702,704). The machine-readable instruction806, when executed by the processor802, causes the processor802to determine a facial landmark count of the frame. The machine-readable instruction808, when executed by the processor802, causes the processor802to adjust a visibility of the frame by a multiplier. The machine-readable instruction810, when executed by the processor802, causes the processor802to cause the frame to be displayed, transmitted, or a combination thereof.

For example, the machine-readable instruction806, when executed by the processor802, causes the processor802to determine a facial landmark count of the frame captured by an image sensor (e.g., the image sensor304,504). The machine-readable instruction808, when executed by the processor802, causes the processor802to adjust a visibility of the frame by a multiplier that is based on the facial landmark count. The machine-readable instruction810, when executed by the processor802, causes the processor802to cause a display device (e.g., the display device506,700) to display the frame, a network interface (not explicitly shown) to transmit the frame, or a combination thereof.

In various examples, the processor802determines whether an audio signal captured by a microphone (not explicitly shown) includes an indicator, and responsive to a determination that the indicator is absent, mutes the microphone. The processor802utilizes a machine learning technique to identify the indicator, for example. The indicator is a topic of discussion, a phrase, or a keyword that indicates a user is engaging in conversation with the audience. The machine learning technique utilizes a speech recognition technique, a speech model, or a combination thereof to identify the topic, the phrase, or the keyword, for example. The speech recognition technique utilizes a Hidden Markov Model (HMM) to recognize patterns in the audio data of the audio signal, in some examples. The speech model accounts for grammar, vocabulary, or a combination thereof, for example.

In some examples, the electronic device determines whether the facial landmark count in relation to the total number of facial landmark points exceeds a multiplier. For example, given the total number of facial landmark points is 400, the facial landmark count is 250, and the multiplier is 0.75, the electronic device determines that a result of 250 divided by 400 (e.g., 0.625) does not exceed the multiplier. In some examples, the processor802mutes the microphone responsive to the multiplier being less than a threshold and unmutes the microphone responsive to the multiplier exceeding the threshold. In various examples, as described above with respect toFIGS.1and6, whether to mute or unmute the microphone is adjustable utilizing a GUI.

In other examples, the processor802adjusts the visibility of the frame by a multiplier that is based on the facial landmark count responsive to a determination that an eye gaze direction of a user in the frame is away from the display device. For example, the processor802utilizes a tracking technique (e.g., an eye tracking technique, a gaze tracking technique, or a combination thereof) to determine the eye gaze direction based on a location of facial landmark points (e.g., the eyes), as determined utilizing the face detection technique described above with respect toFIG.1. Utilizing the face detection technique, the machine learning technique, or a combination thereof described above with respect toFIG.1, the processor802identifies eyes of a user (e.g., the user208,706) in the frame recorded by the image sensor. The processor802determines locations of the facial landmark points. Based on the locations of the facial landmark points relative to a location of an intersection of an optical axis of the image sensor with the user, the processor802determines the eye gaze direction of the user. The optical axis, as used herein, is an imaginary line along which there is a degree of rotational symmetry in the image sensor. In another example, the processor802causes the image sensor to emit an infrared light. The image sensor detects a reflection of the infrared light. The processor802analyzes data of the reflection to determine the eye gaze angle. In various examples, the processor802utilizes a machine learning technique that utilizes two levels of CNNs. The first level CNN detects facial features. The second level CNN refines a number of points around the eyes and mouth to enhance an accuracy of the technique. The machine learning technique utilizes a transform to calculate a pitch, a yaw, and a roll for angles of the face. The machine learning technique identifies the eye landmarks and locate the iris utilizing the eye landmarks. Utilizing a location of the iris, the machine learning technique determines the eye gaze angle. In such examples, while determining an eye gaze angle for a frame of a first image sensor of the multiple image sensors, the electronic device detects facial features and eye landmarks of a frame of a second image sensor of the multiple image sensors.

The method100,400is implemented by machine-readable instructions stored to a storage device (e.g., the storage device308,508, the non-transitory machine-readable medium804) of an electronic device (e.g., the electronic device300,500,800), in various examples. A processor (e.g., the processor302,502,802) of the electronic device executes the machine-readable instructions to perform the method100,400, for example. A process, as used herein, refers to operations performed by execution of machine-readable instructions by the processor. A decision point, as used herein, refers to operations performed by execution of machine-readable instructions by the processor. Unless infeasible, some or all of the blocks (e.g., process, decision point) of the method100,400, may be performed concurrently or in different sequences. For example, the processor performs a block that occurs responsive to a command sequential to the block describing the command. In another example, the processor performs a block that depends upon a state of a component after the state of the component is enabled.

Values for thresholds and settings described in the above examples are determined during a manufacture process, for example. As described above with respect toFIGS.1and6, an executable code provides a GUI to enable a user (e.g., the user208,706) of an electronic device (e.g., the electronic device300,500,800) to adjust the thresholds and settings. The thresholds and settings are stored to a storage device (e.g., the storage device308,508, the non-transitory machine-readable medium804) of the electronic device.

Utilizing the electronic device300,500,800that adjusts the visibility of the frame (e.g., the frame202,702,704) based on the facial landmark count within the frame provides for an enhanced user and audience experience by reducing distractions for the audience. Enhancing the effectiveness of communication between the user and the audience enhances user productivity.

The above description is meant to be illustrative of the principles and various examples of the present description. Numerous variations and modifications become apparent to those skilled in the art once the above description is fully appreciated. It is intended that the following claims be interpreted to embrace all such variations and modifications.

In the figures, certain features and components disclosed herein are shown in exaggerated scale or in somewhat schematic form, and some details of certain elements are not shown in the interest of clarity and conciseness. In some of the figures, in order to improve clarity and conciseness, a component or an aspect of a component is omitted.

In the above description and in the claims, the term “comprising” is used in an open-ended fashion, and thus should be interpreted to mean “including, but not limited to . . . .” Also, the term “couple” or “couples” is intended to be broad enough to encompass both direct and indirect connections. Thus, if a first device couples to a second device, that connection may be through a direct connection or through an indirect connection via other devices, components, and connections. Additionally, the word “or” is used in an inclusive manner. For example, “A or B” means any of the following: “A” alone, “B” alone, or both “A” and “B.”