DEPTH ASSISTED IMAGES REFINEMENT

In some examples in accordance with the present description, an electronic device is provided. The electronic device includes a controller to implement an image segmentation process. The controller is to obtain color information of an image. The controller also is to obtain depth information of the image. The controller also is to determine a depth of a face represented in the color information. The controller also is to segment a foreground of the image from a background of the image according to the color information and the depth information based on the depth of the face.

BACKGROUND

Electronic devices such as desktops, laptops, notebooks, tablets, and smartphones include executable code that enables users to perform video conferencing. During video conferencing sessions, video may be captured by a user's device and transmitted to a viewer's device in substantially real time (e.g., accounting for transmission lag but not having a delay to allow any meaningful amount of processing to be performed). Some video conferencing experiences enable the virtual modification of a user's background, either via blurring of the user's background or replacement of the user's background.

DETAILED DESCRIPTION

As described above, electronic devices such as desktops, laptops, notebooks, tablets, and smartphones include executable code that enables users to perform video conferencing. During video conferencing sessions, video may be captured by a user's device and transmitted to a viewer's device in substantially real time (e.g., accounting for transmission lag but not having a delay to allow any meaningful amount of processing to be performed). Some video conferencing experiences enable the virtual modification of a user's background, either via blurring of the user's background or replacement of the user's background.

Background blur and background replacement each include separation or segmentation of a foreground (e.g., the user) from a background of the video. Such separation may be performed according to an object segmentation process. However, challenges may arise in performing object segmentation for transparent or semi-transparent foreground elements, as well as for small portions of background encircled by the foreground.

To increase a capability for performing object segmentation for transparent or semi-transparent foreground elements and small portions of background encircled by the foreground, segmentation may be performed according to a combination of red-green-blue (RGB) image data and a corresponding depth map, such as may be captured by a depth-capable camera or other sensor(s) capable of determining and providing depth information. In some examples, the depth-capable camera is implemented as a stereo infrared sensor.

An segmentation according to RGB data may be performed, resulting in isolation of the foreground from the background. However, the foreground may be inaccurate, either including elements as foreground that are in fact background and/or omitting elements as background that are in fact foreground.

To mitigate these inaccuracies, facial detection may be performed on the RGB image data to identify a face. A bounding box may be identified around the face and depth sampling of the depth map may be performed at a point or points within the bounding box. A depth cutoff of the depth image may be performed for points having greater than a threshold distance from the sampled point(s) in the depth map. The cutoff depth image and the RGB image data may be concatenated and provided to a convolutional neural network (CNN) for processing. Based on that processing, the CNN provides a segmentation result. In at least some examples, a combined RGB image and depth image approach to image segmentation improves a user experience of a user implementing the image segmentation by, for example, more accurately segmenting a foreground from a background of an image.

In some examples in accordance with the present description, an electronic device is provided. The electronic device includes an image sensor, a depth sensor, and a controller. The controller is to receive, from the image sensor, RGB information of an image. The controller also is to receive, from the depth sensor, depth information of the image. The controller also is to perform facial detection to identify a face in the RGB information. The controller also is to truncate the depth information to exclude information for depth points not within a threshold variance from a depth of the identified face. The controller also is to process the image and the truncated depth information according to a machine learning process to segment a foreground of the image from a background of the image.

In some examples in accordance with the present description, an electronic device is provided. The electronic device includes a controller to implement an image segmentation process. The controller is to obtain color information of an image. The controller also is to obtain depth information of the image. The controller also is to determine a depth of a face represented in the color information. The controller also is to segment a foreground of the image from a background of the image according to the color information and the depth information based on the depth of the face.

In some examples in accordance with the present description, a non-transitory computer-readable medium storing machine-readable instructions is provided. The instructions, when executed by a controller of an electronic device, cause the controller to obtain color information of an image, obtain depth information of the image, determine a depth of a face present in the image, perform a depth cutoff of the depth information for points having greater than a threshold variance from the depth of the face, and process the image according to the cutoff depth information and the color information to obtain a foreground of the image.

FIG.1is a block diagram of an electronic device100in accordance with various examples. The electronic device100may be a laptop computer, a desktop computer, a notebook, a tablet, a server, a smartphone, or any other suitable electronic device having a camera and capable of participating in video conferencing sessions. The electronic device100may include a controller102(e.g., a central processing unit (CPU), a microprocessor, etc.), a storage104(e.g., random access memory (RAM), read-only memory (ROM)), an image sensor106to capture images and video in an environment of the electronic device100, a depth sensor107to capture depth information, a microphone108to capture audio in an environment of the electronic device100, and a network interface110. The network interface110enables the controller102, the image sensor106, the depth sensor107, and/or the microphone108to communicate with other electronic devices external to the electronic device100. For example, the network interface110enables the controller102to transmit signals to and receive signals from another electronic device over the Internet, a local network, etc., such as during a video conferencing session. A network112may couple the controller102, storage104, image sensor106, depth sensor107, microphone108, and network interface110to each other. Storage104may store executable code114(e.g., an operating system (OS)) and executable code116(e.g., an application, such as a video conferencing application that facilitates video conferencing sessions with electronic devices via the network interface110). In examples, the image sensor106may capture and store images and/or video (which is a consecutive series of images, or image frames) to the storage104. In examples, the microphone108may capture and store audio to the storage104. In examples, the storage104includes a buffer or buffers (not shown) to temporarily store image and/or video captured by the image sensor106, the depth sensor107, and/or audio captured by the microphone108prior to transmission via the network interface110or manipulation by the controller102.

In operation, the controller102executes the executable code116to participate in a video conferencing session. As the controller102executes the executable code116, the controller102receives images and/or video captured by the image sensor106, depth information captured by the depth sensor107, and/or audio captured by the microphone108and provides the image, video, depth information, and/or audio data to the network interface110for transmission to another electronic device that is participating in the video conferencing session with the electronic device100.

As described above, a user of the electronic device100may be participating in the video conferencing session and may wish to alter a background of the video conferencing session. To perform such alteration, object segmentation is performed to separate a foreground subject of the video conferencing session from the background of the video conferencing session. However, challenges can arise in separating transparent or semi-transparent foreground elements and small portions of background encircled by foreground elements. To mitigate these challenges, information from the image sensor106and the depth sensor107is combined to perform the object segmentation. For example, the image sensor106provides RGB information and the depth sensor107provides depth information, such as a depth map. The RGB information includes the color representation of the foreground and background. The depth information includes information relating to the distance of the surfaces represented in the RGB information from a viewpoint, such as the depth sensor107, which may be positioned adjacent to the image sensor106and directed in a same direction as the image sensor106. In some examples, the image sensor106and the depth sensor107may be components of a same module or device. In some examples, the depth sensor107includes multiple imagers, such as a left imager and a right imager disposed on respective opposing sides of an RGB camera (e.g., the image sensor106). The depth map is determined according to the below equation 1, for example.

In the above equation 1, Di,jis a depth value at pixel (i, j) of the depth map, d is a distance between the imagers of the depth sensor107, f is an average focal length of the left and right imagers of the depth sensor107, and dvi,jis a disparity value for the pixel at (i, j). Each pixel in the depth map is projected from (i, j) in a 2-dimensional pixel plane (e.g., as captured by the depth sensor107) to three dimensional (x, y, z) space based on the following equations 2 through 4.

In the above equations 2 through 4, Cxand Cyare center pixel positions on the horizontal and vertical axis of the depth map, respectively, and fxand fyare focal lengths of the imagers of the depth sensor107on the horizontal and vertical axes, respectively.

To combine the RGB information and the depth information, the controller102performs facial detection on the RGB information to identify a face of the user. Based on the identified face, the controller102forms or otherwise defines a bounding box that encompasses the identified face in two dimensions. In examples of the RGB information in which multiple faces are identified, a bounding box may be formed around each identified face and the following operations are performed for each identified face/bounding box. The controller102samples the depth map within the bounding box to determine an approximate distance of the user from the viewpoint. In various examples, any number of locations greater than or equal to one within the bounding box may be sampled. In some examples, multiple samples may be averaged to determine an average depth of the user from the viewpoint. In other example, a deepest sampled depth may be determined as the depth of the user.

After determining a depth of the user from the viewpoint, the controller102performs a depth cutoff. The depth cutoff removing points of the depth map that have a depth greater than the depth of the user, or greater than the depth of the user plus a threshold amount (e.g., truncates the depth map). In some examples, the controller102performs the depth cutoff based on clustering. For example, the controller102performs Euclidean distance clustering to perform the depth cutoff. The Euclidean distance clustering implements region growing in 3D space (e.g., the (x, y, z) space), identifying points in the depth map that are connected to the sampled point (e.g., the point in the (i, j) space representing the depth of the user) and removing points in the depth map that are unconnected to the sampled point. A result of the depth cutoff may be considered a depth map segmentation result, such that the depth map approximates a segmentation of the foreground of the depth map from the background of the depth map.

An output of the image sensor106(e.g., RGB information in separate R, G, and B channels) and the depth map segmentation result are concatenated to form a 4-channel input. The controller102may apply a suitable machine learning technique, such as a CNN, to the 4-channel input to determine a segmentation result. In various other examples, other machine learning, artificial intelligence, or machine vision processing techniques or process(es) are applied to the 4-channel input to determine the segmentation result. In examples, the segmentation result is represented as an image mask, such as having black pixels in a region determined to be the background and white pixels in a region determined to be the foreground. The controller102may subsequently apply the mask to the RGB information to perform object segmentation, obtaining a separate representation of the foreground that does not include the background. The controller102may manipulate (blur, replace, recolor, augment, etc.) the RGB information and overlay the masked foreground on the manipulated representation of the RGB information to provide final video data for use in the video conferencing session, such as for transmission by the network interface110.

FIG.2is a diagram of pseudocode200in accordance with various examples. The pseudocode200may be representative of operations to be performed by the controller102to perform object segmentation, as described herein. For example, the pseudocode200may be a plain-language description of operations that may be performed by the controller102based on machine-executable instructions (e.g., the executable code116) stored to the storage104and executed by the controller102in response to receipt of a request from the user to manipulate a background of an image or video for a video conferencing session.

The pseudocode200begins with the controller102obtaining an RGB data frame (RGB_Frame) and a depth data frame (Depth_Frame). In some examples, the controller102obtains the data from cameras, such as the image sensor106and/or depth sensor107. In other examples, the controller102obtains the data from the storage104, such as a buffer or other location to which the image sensor106and/or depth sensor107temporarily stored the data. The controller102performs facial detection according to any suitable process to identify a face in the RGB_Frame and defines a bounding box surrounding the face. For each newly received RGB_Frame and Depth_Frame (e.g., such as multiple sequentially captured and obtained frames that together form video) the controller102executes an operational loop. The loop includes sampling a depth in the bounding box, performing a depth cutoff of the Depth_Frame based on the sampled depth, concatenating the RGB_Frame with the cutoff Depth_Frame, providing the concatenation result to a trained CNN (or other machine learning process), obtaining a segmentation result, and generating a new face bounding box based on the segmentation result, such as via facial detection, as described above. The controller102performs the above actions according to processes or operations substantially as described above with respect toFIG.1.

FIG.3is a pictographical representation300of an object segmentation process in accordance with various examples. The representation300includes images302,304,306, and308. The image302represents the RGB information obtained from the image sensor106(or the RGB_Frame of the pseudocode200). Image304represents a result of object segmentation performed on the image302without the teachings of this disclosure, such as without including the use of depth map information. The image306represents the depth information obtained from the depth sensor107(or the Depth_Frame of the pseudocode200). The image308represents the depth information of the image306after undergoing truncation or depth cutoff, as described here. The image310represents the image302, masked according to the mask determined by the controller102based on processing of both the RGB data of the image302and the depth information of the image308. As shown by a comparison of the image304(determined without use of the teachings of this disclosure) to the image310(determined according to the depth assisted images refinement teachings of this disclosure), the image310is a more accurate representation of the foreground of the image302, thereby resulting in an improved user experience when performing object segmentation for a video conferencing session.

FIG.4is a flow diagram of a method400in accordance with various examples. In some examples, the method400is implemented by the controller102, such as to perform object segmentation as described herein according to a depth assisted image refinement process. The controller102may perform or execute the method400as a result of executing the executable code116, for example. The method400includes obtaining RGB data (402) and obtaining Depth data (404). In a first iteration of the method400(e.g., for first received RGB data and first received Depth data), there may be no bounding box yet defined. As such, the foreground depth map may be the Depth information directly such that a segmentation result is substantially similar to that shown in image304. In a second iteration of the method400, the method400includes sampling the Depth data according to a determined bounding box (406). The method400includes performing a depth cutoff of the sampled Depth data to form a foreground depth map (408). The method400includes concatenating the RGB data with the foreground depth map to form a 4-channel input (410). The method includes processing the 4-channel input according to a machine learning technique, such as a CNN, to determine a segmentation result (412). The method400includes processing the segmentation result to identify a bounding box (414). The bounding box may be useful for performing depth sampling at operation406for subsequently received RGB and Depth data.

Although not shown inFIG.4, in examples the method400also includes manipulating the image, or the background of the image based on an inversion of the segmentation result, masking the image according to the segmentation result to segment the foreground from the background, and overlaying the masked foreground on top of the manipulated background. The resulting combined image, in some examples, is transmitted, such as in a video conferencing session.

The method400is implemented by machine-readable instructions (e.g., the executable code116) stored to a storage device (e.g., the storage104) of an electronic device (e.g., the electronic device100), in various examples. A processor (e.g., the controller102) of the electronic device executes the machine-readable instructions to perform the method400, for example. Unless infeasible, some or all of the method400may be performed concurrently or in different sequences. For example, the processor performs a block that occurs responsive to a command sequential to the block describing the command. In another example, the processor performs a block that depends upon a state of a component after the state of the component is enabled or disabled.

FIGS.5and6are block diagrams of the electronic device100, including the controller102coupled to the storage104, in accordance with various examples. Specifically,FIG.5shows an example of the electronic device100, including the controller coupled to the storage104along with image sensor106and depth sensor107coupled to the controller102. The storage104stores executable instructions that may be executed by the controller102. The storage104includes executable instruction500, which causes the controller102to receive, from the image sensor, RGB information of an image. The storage104includes executable instruction502, which causes the controller102to receive, from the depth sensor, depth information of the image. The storage104includes executable instruction504, which causes the controller102to perform facial detection to identify a face in the RGB information. The storage104includes executable instruction506, which causes the controller102to truncate the depth information to exclude information for depth points not within a threshold variance from a depth of the identified face. The storage104includes executable instruction508, which causes the controller102to process the RGB information of the image and the truncated depth information according to a machine learning process to segment a foreground of the image from a background of the image. Actions performed by the controller102pursuant to executing the executable instructions may be carried out in a manner substantially similar to that described above with respect toFIG.1, or any other figures of this disclosure.

FIG.6shows an example of the electronic device100, including the controller102coupled to the storage104along with image sensor106and depth sensor107coupled to the controller102. The storage104stores executable instructions that may be executed by the controller102. The storage104includes executable instruction600, which causes the controller102to obtain color information of an image. The storage104includes executable instruction602, which causes the controller102to obtain depth information of the image. The storage104includes executable instruction604, which causes the controller102to determine a depth of a face represented in the color information. The storage104includes executable instruction606, which causes the controller102to segment a foreground of the image from a background of the image according to the color information and the depth information based on the depth of the face. Actions performed by the controller102pursuant to executing the executable instructions may be carried out in a manner substantially similar to that described above with respect toFIG.1, or any other figures of this disclosure.

FIG.7is a block diagram of non-transitory, computer-readable media in accordance with various examples. Specifically,FIG.7depicts an example of the electronic device100, including the controller102coupled to the storage104. The storage104includes executable instruction700, which causes the controller102to obtain color information of an image. The storage104includes executable instruction702, which causes the controller102to obtain depth information of the image. The storage104includes executable instruction704, which causes the controller102to determine a depth of a face present in the image. The storage104includes executable instruction706, which causes the controller102to perform a depth cutoff of the depth information for points having greater than a threshold variance from the depth of the face. The storage104includes executable instruction708, which causes the controller102to process the image according to the cutoff depth information and the color information to obtain a foreground of the image. Actions performed by the controller102pursuant to executing the executable instructions may be carried out in a manner substantially similar to that described above with respect toFIG.1, or any other figures of this disclosure.

As described herein, executable code includes an “application,” “software,” and “firmware.” The terms “application,” “software,” and “firmware” are considered to be interchangeable in the context of the examples provided. “Firmware” is considered to be machine-readable instructions that a processor of the electronic device executes prior to execution of the operating system (OS) of the electronic device, with a small portion that continues after the OS bootloader executes (e.g., a callback procedure). “Application” and “software” are considered broader terms than “firmware,” and refer to machine-readable instructions that execute after the OS bootloader starts, through OS runtime, and until the electronic device shuts down.

The above description is meant to be illustrative of the principles and various examples of the present description. Numerous variations and modifications become apparent to those skilled in the art once the above description is fully appreciated. It is intended that the following claims be interpreted to embrace all such variations and modifications.

In the figures, certain features and components disclosed herein are shown in exaggerated scale or in somewhat schematic form, and some details of certain elements are not shown in the interest of clarity and conciseness. In some of the figures, in order to improve clarity and conciseness, a component or an aspect of a component is omitted.

In the above description and in the claims, the term “comprising” is used in an open-ended fashion, and thus should be interpreted to mean “including, but not limited to . . . .” Also, the term “couple” or “couples” is intended to be broad enough to encompass both direct and indirect connections. Thus, if a first device couples to a second device, that connection may be through a direct connection or through an indirect connection via other devices, components, and connections. Additionally, the word “or” is used in an inclusive manner. For example, “A or B” means any of the following: “A” alone, “B” alone, or both “A” and “B.”