Patent ID: 12261895

DESCRIPTION OF EXAMPLE EMBODIMENTS

Overview

Techniques are provided herein for framing video during an online conference session based on distance data. These techniques may be embodied as a method, an apparatus, a system, and instructions in a computer-readable storage media to perform the method.

According to at least one example embodiment, the techniques include obtaining at least a video stream during an online conference session. The video stream, an audio stream received with the video stream, or both the video stream and the audio stream are analyzed and a framing that either focuses on a speaker in the video stream or provides an overview of participants in the video stream, the framing being is composed based on the analyzing. A potential error in the framing is detected based on further analysis of at least one of the video stream, the audio stream, or distance sensor data received with the video stream. If the distance sensor data contradicts the potential error, the framing is maintained, but if the distance sensor data confirms the potential error, a new framing is generated.

EXAMPLE EMBODIMENTS

The techniques presented herein provide distance-based framing during an online conference session. More specifically, the techniques generate framings for a video stream during an online conference session based, at least in part, on distance sensor data, such as radar data. The techniques may utilize the distance sensor data to attempt to confirm potential errors in current framings (e.g., generated with speaker tracking framing techniques) that are detected based on analyzing video data, audio data, and/or the distance sensor data. For example, distance sensor data may indicate that a current framing has a potential error. Alternatively, if analyses of video data and audio data generate different speaker locations (e.g., analyses produce non-matching results), this may be a potential error. By using distance-based data to confirm or deny potential errors, the techniques presented herein can correct erroneous framing decisions that are sometimes made based on erroneous results from video and/or audio analysis techniques. This may reduce unnecessary framing switches and/or improve the accuracy of framings during an online conference session.

In order to describe the systems, apparatus, and/or techniques presented herein, terms such as “left,” “right,” “top,” “bottom,” “front,” “rear,” “side,” “height,” “length,” “width,” “upper,” “lower,” “interior,” “exterior,” “inner,” “outer,” “depth,” and the like as may be used. However, it is to be understood that these terms merely describe points of reference and do not limit the present embodiments to any particular orientation or configuration. For example, the terms “right” and “left” may be used to describe certain embodiments presented herein, but it is to be understood that these terms are not intended to limit the systems, apparatus, and/or techniques presented herein to a single orientation. Instead, the systems, apparatus, and/or techniques presented herein, or portions thereof, may be oriented in any a number of orientations. Thus, even if a certain feature is described herein as being oriented on the “right,” it may be understood that this feature may be oriented on the “left” when a system or apparatus is in a different orientation (or when a technique is executed with the system or apparatus in a different orientation).

Reference is first made toFIG.1, which illustrates an online conference/meeting environment100in which an online conference session may utilize the distance-based framing techniques presented herein. In the environment100, an online conference server102(i.e., meeting server102) communicates, via the Internet110, with a plurality of endpoints120. For simplicity, only two endpoints120are shown inFIG.1; however, in other embodiments, the plurality of endpoints120may include any number of endpoints/computing devices.

The online conference server102includes at least one processor104, a network interface unit106, and a memory108. The processor104is configured to execute instructions stored on memory108and the network interface unit106enables connectivity to the Internet110. The online conference server102also includes a server application160that may reside in memory108and serves conference session support for online conference client applications170(also referred to herein as client applications170, for simplicity) that may be installed on the plurality of endpoints120(e.g., downloaded via the Internet110). Generally, the server application160is configured to direct online conference traffic flows between any online conference client applications170participating in an online conference session. Thus, once an online conference session is initiated, each client application170is operatively connected to the server application160such that any client applications170connected to the session are in communication with each other in an online conference session via the server application160. The session may be established using any suitable protocols now known or developed hereafter.

The server application160may include a server framing module162that is configured to receive and process video captured at any of the endpoints120(e.g., via one or more camera at each endpoint120). For example, the server framing module162may process the video from an endpoint to analyze or generate one or more framings for the online conference session based on distance sensor data gathered at that endpoint. Additionally, the server framing module162may receive and process data from a variety of sensors, including image sensors (e.g., cameras or lenses), audio sensors (e.g., microphones), and distance sensors (e.g., radar sensors). Additionally or alternatively, the distance-based framing techniques presented herein may be executed on one or more of the endpoints120participating in a conference session. Thus, inFIG.1, client framing module172is shown in dashed lines within memory156of each of the plurality of endpoints120. To be clear, client framing module172may work with or instead of server framing module162to execute the techniques presented herein.

Each of the plurality of endpoints120includes a processor152configured to execute instructions stored in a memory156and a network interface unit154that provides connectivity to the Internet110. For example, the processor152may be configured to execute instructions to install the client application170(and potentially client framing module172). Generally, each of the plurality of endpoints120may be any computing device/endpoint compatible to support the online conference client application170. For example, one endpoint120may be a tablet computer, desktop computer, laptop computer, and another endpoint120may be a smartphone, desktop, virtual machine, or any other device, provided that each of the plurality of endpoints includes or is associated with a processor152configured to support the online conference client application170and network interface unit154configured to connect the device to the Internet110, respectively. Additionally or alternatively, one or more of the endpoints may be embodied entirely as one or more software applications running on a computing device, such as in a cloud or data center environment. Thus, an endpoint may be a physical device or a software process.

Additionally, although each module described herein is shown stored in memory, such as memory108, each module described herein may be implemented on hardware, or a combination of hardware and software. For example, each module may include and/or initiate execution of an application specific integrated circuit (ASIC), a Field Programmable Gate Array (FPGA), a circuit, a digital logic circuit, an analog circuit, a combination of discrete circuits, gates, or any other type of hardware, or combination thereof. Accordingly, as used herein, execution of a module by a processor can also refer to logic based-processing by the module that is initiated directly or indirectly by the processor to complete a process or obtain a result. Additionally or alternatively, each module can include memory hardware, such as at least a portion of a memory, for example, that includes instructions executable with a processor to implement one or more of the features of the module. When any one of the modules includes instructions stored in memory and executable with the processor, the module may or may not include a processor. In some examples, each module may include only memory storing instructions executable with the processor to implement the features of the corresponding module without the module including any other hardware.

Moreover, memory108and/or memory156may also be configured to store any video data, audio data, video processing algorithms, audio processing algorithms, thresholds, or other such data related to distance-based framing during an online conference session. Generally, memory108and/or memory156may include read only memory (ROM), random access memory (RAM), magnetic disk storage media devices, optical storage media devices, flash memory devices, electrical, optical or other physical/tangible (e.g., non-transitory) memory storage devices. Thus, in general, the memory108and/or memory156may be or include one or more tangible (non-transitory) computer readable storage media (e.g., a memory device) encoded with software comprising computer executable instructions. For example, memory108and/or memory156may store instructions that may be executed by processor104or processor152, respectively, for performing the distance-based framing techniques described below with reference to the figures. In other words, memory108and/or memory156may include instructions, that when executed by one or more processors, cause the one or more processors to carry out the operations described below in connection with the figures.

Reference is now made toFIG.2, which shows an example embodiment of a video conference endpoint200that may execute at least a portion of the techniques presented herein. The video conference endpoint200includes a display screen204, one or more sensors210A-E, and a distance sensor212. The display screen204may include an electronic whiteboard or other such collaborative display. Additionally, the display screen204may be configured to detect writing or drawing on the display screen204and/or may include embedded sensors configured to detect writing or drawing on the display screen204. For example, the display screen204may be a touch screen display that detects and records touches. Additionally or alternatively, image analysis may be used to analyze where information is placed on the display screen204. However, in other embodiments, an apparatus configured to execute at least a portion of the techniques presented herein need not include a display screen204and could, for example, be a self-contained apparatus that is attachable to or mountable near a display screen.

Meanwhile, the sensors210A-E may be any suitable sensors, including microphones, microphone arrays (or portions thereof), image sensors, etc., that are configured to collectively gather audio and video from a space or environment in which the video conference endpoint200is disposed. As one example, sensor210A may be one or more image sensors (e.g., one or more cameras) while sensors210B,210C,210D, and210E may be microphones that collectively form a microphone array. However, this is only one example and other embodiments may include any arrangement of cameras/image sensors and microphones/audio sensors, or other such equipment, to gather audio and video in the environment. This distance sensor212may be any sensor that can detect people or objects in the environment of endpoint200and generate distance-based data indicative of motion and/or a location of people or objects in the environment. However, the sensors210A-E and the distance sensor212depicted inFIG.2are not intended to be limiting and, the endpoint200(or any other apparatus configured to execute at least a portion of the techniques presented herein) may also include any other type of sensor now known or developed hereafter (e.g., in combination with cameras, microphones, and a distance sensor).

In one embodiment, the distance sensor212is a radar sensor; however, in other embodiments, the distance sensor212could include one or more of a radar sensor, an ultrasound sensor, an infrared sensor, or any other sensor configured to detect locations and/or movements of people and/or objects. When the distance sensor212is a radar sensor, the radar sensor may operate at 10 hertz (Hz) to provide five detections per second. Each detection may provide data relating to an angle in a horizontal plane and distance from the distance sensor. These data points can then be plotted onto an angular mapping of an endpoint's environment to track movement and location of participants in the environment, as is described in further detail below in connection withFIG.7. In at least some embodiments, the detections only detect moving people or objects.

In the depicted embodiment, the image sensor210A is centered above the display screen204. Positioning an image sensor centered above the display screen204may most effectively capture images of users who are interacting with the endpoint200and encourage eye contact between users at different endpoints, at least because content is most frequently displayed or added (e.g., written or drawn) at a top-center portion of the display screen204. Meanwhile, microphones210B-E and distance sensor212may be disposed below or around the display screen204to provide wide coverage and allow for efficient speaker tracking and movement detection, respectively. However, as mentioned, this is only an example, and in other embodiments, sensors210A-E and the distance sensor212could be included in any location on endpoint200. Alternatively,210A-E and the distance sensor212could be included in a housing that is positionable above or below a display screen (e.g., the “endpoint” could be a camera/sensor unit).

Now referring toFIG.3, a block diagram300of video conference endpoint200, configured in accordance with an example embodiment, is shown. Reference is also made toFIGS.1and2for purposes of the description ofFIG.3. In this particular embodiment, the six sensors included in video conference endpoint200(sensors210A-E and distance sensor212) are each operatively coupled to a controller310. The controller310includes a processor320, memory322, an analysis module330, and various output ports340. Additionally, the controller310is also operatively coupled to the display screen204(which may also be or include sensors), a speaker342, and a network interface unit350that enables network communication.

In one example, the functions of the processor320and the analysis module330may be implemented by fixed or programmable digital logic gates (e.g., in a programmable gate array). Alternatively, the functions of the controller310may be implemented by software stored in a memory322(e.g., client framing module172ofFIG.1) that, when executed, causes the processor320to perform the operations described herein for the controller310. Put another way, processor320may be equivalent to processor152ofFIG.1and memory322may be equivalent to memory156ofFIG.1. Likewise, network interface unit350may be equivalent to network interface unit154ofFIG.1. Thus, any description of processor152, memory156, and network interface unit154included above may apply to processor320, memory322, and network interface unit350, respectively. For example, memory322may store instructions that may be executed by processor320for performing tasks associated with analyzing sensor input and adjusting framings for video content in the manner described herein.

In some embodiments, the controller310may include processing pipelines configured to receive high resolution video signals from one or more image sensors included in sensors210A-E (e.g., from one or more image sensors210A) and convert these signals into a video stream to be output to another video conferencing endpoint. The pipelines may also include croppers and scalers that can digitally process video signals provided by the image sensors included in sensors210A-E (e.g., from one or more image sensors210A). Additionally or alternatively, the controller310may include decoders or coders to facilitate the transmission and receipt of video streams (including participant streams and data streams) during a video conference session. The analysis module330, based on instructions from the processor320(e.g., when executing client framing module172ofFIG.1), performs analysis of distance-based data from distance sensor212(i.e., distance sensor data) to detect where in an environment people are sitting, moving, standing, and to otherwise detect activity on or proximate display screen204of a video conference endpoint200. Additionally, the analysis module330may determine who is speaking to find a presenter/speaker. Still further, the analysis module330, based on instructions from the processor320(e.g., when executing client framing module172ofFIG.1), may compose framings for a video stream and determine when new framings are needed and/or when a current framing has a potential error, as is described in further detail below.

Reference is now made toFIG.4, with reference toFIG.1-3, for a high-level description of a method400for distance-based framing during an online conference session. For simplicity, method400is described herein as being performed by an endpoint or components thereof, such as via execution of the software instructions included in client framing module172by endpoint120, endpoint200or components thereof. However, this is just an example and in other embodiments, the techniques presented herein, or at least portions thereof, may be executed at any computing device participating in an online conference session, including an online conference server102serving conference session support.

In method400, an endpoint initially obtains, at410, at least a video stream during an online conference session. For example, endpoint200may obtain video data from one or more image sensors and processor320and/or analysis module330may generate a video stream from this video data. At420, the endpoint analyzes the video stream, an audio stream also obtained at that endpoint (e.g., when processor320and/or analysis module330generate an audio stream from audio data captured by one or more microphones), or both the video stream and the audio stream. This analysis may identify a “speaker” in the video stream.

However, as used herein, the term “speaker” does not necessarily require that the speaker be continuously talking. Instead, a “speaker” may be the participant who most recently spoke, the participant that is currently speaking, and/or the participant that is the most active participant. An active participant may be identified based on which participant is interacting with a display, such as by pointing to a display, writing on a display, or by touching a display. Regardless, a “speaker” may be identified from audio and/or video data in any manner now known or developed hereafter. For example, a speaker may be identified in video data using facial recognition techniques, head detection techniques, and/or eye detection techniques. Additionally or alternatively, a speaker may be identified with audio captured by multiple microphones and triangulation techniques or any other audio location techniques.

At430, the endpoint may compose a framing that either focuses on a speaker in the video stream or provides an overview of participants in the video stream. For example, to focus on a speaker, the framing may span a certain distance above, below, and beside a face of the speaker that was detected with facial recognition techniques applied to video data. In some instances, the framing can be a digital framing composed by a processor (e.g., with croppings, digital zoom, etc.), but in other instances, the framing can be generated by changing the pan, tilt, zoom, or focus of an image sensor (e.g., a lens of a camera). Still further, a combination of these techniques might be utilized. Meanwhile, an overview of participants may be provided by locating participants and generating a framing in that captures as many and/or as much of the participants as possible. For example, a camera's widest-angle lens may be used to capture a wide area with as many participants as possible and/or a camera may be panned, tilted, and/or zoomed (mechanically or digitally) to capture as many participants as possible. However, generally, the framing or framings composed at430can be composed in any manner for selecting at least a portion of a video stream now known or developed hereafter.

At440, the endpoint detects a potential error in the framing. The endpoint detects this potential error based on a further analysis of any one of or any combination of the video stream, the audio stream, and distance sensor data received with the video stream (and/or with the audio stream). A potential error will indicate that the framing is potentially incorrect. The endpoint may detect a potential error when there is a mismatch between different types of data and/or when analysis of a specific type of data indicates an error or small amount of certainty. For example, if facial detection, head detection, and/or eye detection is performed on video data and is/are failing and/or indicating a low degree of confidence/certainty, this may be considered a potential error. Likewise, if speaker location techniques performed on audio data are failing and/or indicating a low degree of confidence/certainty, this may be considered a potential error. The techniques may also detect a potential error when it determines that an overview framing is not capturing all participants and/or capturing too much space around all of the participants. Still further, if distance sensor data indicates that there is a high degree of movement in the endpoint's environment and/or that participants have moved too close to the endpoint, this may also be considered a potential error.

Any combination of these evaluations could also be used to detect a potential error, which might allow lower thresholds to be used to analyze specific data. For example, if the endpoint detects a moderate amount of movement based on analysis of distance sensor data, but also determines that facial detection has a limited amount of confidence, this combination of data may be considered a potential error. Still further, if different types of data provide different information (i.e., mismatch), this may be considered a potential error. For example, if facial detection indicates a speaker is in one location and audio analysis indicates a speaker is in a different location, this may be considered a potential error.

However, the techniques presented herein do not automatically alter the framing when a potential error is detected. Instead, at450, the endpoint attempts to confirm the potential error with distance sensor data. More specifically, at450, the endpoint may analyze distance sensor data to determine how much movement is occurring in the environment of the endpoint, whether the speaker is moving, and/or where the speaker is located with respect to the endpoint. If one or more of these analyses confirms the potential error, the endpoint will, at460, generate a new framing. If, instead, these analyses do not confirm the potential error-that is, if one or more of these analyses contradicts the potential error-then the endpoint will, at470, maintain its current framing. As an example, if a potential error is detected because facial recognition has failed, the endpoint may analyze the distance sensor data to determine whether the speaker has moved. If the speaker has not moved, this may indicate the speaker has turned his head and contradicts the error. Audio data might also be analyzed to confirm the speaker is still speaking in the same location and further support the analysis of the distance sensor data. If, instead, facial detection has failed and the distance sensor data indicates that the speaker has moved and/or that there is a high amount of motion in the environment, this may confirm the potential error and a new framing may be generated. The new framing may provide an overview, or a new overview if the initial framing was an overview, of the environment to ensure that a moving speaker, or many moving participants are captured in the video stream.

By comparison, in online conference systems that utilize conventional framing algorithms, potential errors may lead to an automatic change in framing. For example, if facial detection fails or provides a low indication of confidence, conventional framing algorithms may automatically switch the framing to an overview of the endpoint's environment. Unfortunately, this may switch the framing unnecessarily and/or too often. For example, if a speaker looks away from the endpoint while continuing to talk, switching the framing away from the speaker may be distracting and unnecessary. Advantageously, confirming potential errors at450with at least distance sensor data may avoid unnecessarily reframing the video stream. This may reduce the amount of processing operations during an online conference session and also improve the meeting experience (by reducing or eliminating continuous reframing). Moreover, confirming potential errors may ensure that the video stream is reframed when necessary. In at least some embodiments, the sensitivity of potential errors can be adjusted, e.g., dynamically and/or via user inputs, to strike a balance between minimizing reframings and identifying a maximum number of errors.

Reference is now made toFIG.5.FIG.5illustrates a flowchart of a method500executed by the distance-based framing techniques presented herein, for a first use case. This particular method is specifically tuned to address framing issues that often arise when a participant is occluded from an endpoint camera's view or not looking at a camera. Reference is also made toFIG.4while describing method500. Initially, at510, a direction and distance of a speaker is determined by analyzing audio data obtained at an endpoint. At520, faces and/or heads are detected in video data obtained in an endpoint to locate a speaker in the video data. Each of these steps may, for example, be performed as part of the analysis performed in step420of method400(ofFIG.4). At530, the direction and distance determined at410are compared to the face/head position detected at420to determine if the data matches. As discussed above, a mismatch may be considered a potential error (as is determined in step440of method400).

If the data matches at530, the endpoint, at540, continues to generate a framing for the speaker. This framing may be slightly adjusted as the speaker shifts or otherwise slightly moves. If, instead, the data does not match at530, the endpoint determines, at550, if the audio is in the same position as it was previously. This may indicate whether the speaker has changed (e.g., if a different participant is now talking) or if the speaker has moved. If the audio is still in the same location, this may cause the endpoint to determine that no potential error exists and the endpoint may, at555, continue to track speakers and generate framings based on current framing procedures (e.g., based on audio analysis and/or video analysis). For example, if the speaker has not changed, the data may be mismatched because the speaker is now occluded from a camera's field of view and, thus, a new framing may not be appropriate.

However, if the audio is determined to be emanating from a new location, the endpoint may, at560, evaluate distance sensor data (e.g., radar data), to determine how much motion is occurring in the room. Notably, if the audio is determined to be emanating from a new location, this does not necessarily mean the speaker has moved within the environment. Instead, the speaker might have turned his or her head (e.g., to look at someone entering a conference room), causing audio detection techniques to locate the source of the audio in a different location. Thus, distance sensor data is used to provide further context. The operations of steps570,580, and555may, for example, correspond to operations performed in steps450,460, and470of method400(ofFIG.4), respectively.

If the amount of motion is determined, at570, to be higher than a threshold, this may indicate that the speaker and/or other participants is/are moving in the room and, thus, speaker tracking framings, which are typically only optimized for a small amount of speaker movement may be suboptimal. Thus, a new overview framing (e.g., a wide-angle view of the environment) that attempts to capture all participants in the environment may be composed and applied to the video stream at580. If, instead, the amount of motion is determined, at570, to be lower than the threshold, the endpoint may, at555, continue to track speakers and generate framings based on current framing procedures (e.g., based on audio and/or video analysis). This is because a lower amount of movement may indicate that the speaker has not moved and is, instead, occluded or facing a new direction (perhaps only temporarily).

Notably, overall, method500processes distance sensor data once audio and/or video analysis identify an error, which reduces processing as compared to techniques that must constantly evaluate distance sensor data. Moreover, method500maintains its current framings unless: (1) video and audio data analyses produce mismatched results (e.g., as determined at530); (2) the audio source has been determined to have moved; and (3) distance sensor data indicates an amount of motion above a threshold. Thus, method500reduces or eliminates unnecessary reframing that might occur when a user is occluded or not facing a camera. Still further, since method500deactivates speaker tracking framings in favor of a new, best overview when motion in the environment is above a threshold, method500may ensure that the best overview is consistently updated when participants are moving or relocating in an environment. This may help avoid using overview framings that do not capture participants who have moved while another participant was speaking.

Reference is now made toFIG.6.FIG.6illustrates a flowchart of a method600executed by the distance-based framing techniques presented herein in a second use case. This particular method is specifically tuned to address framing issues that often arise when participants are disposed at edges of an endpoint camera's field of view, such as while interacting with a display. Reference is also made toFIG.4while describing method600. Initially, at610, the endpoint generates framings based on speaker tracking techniques, such as the example head, face, or eye techniques mentioned above, and/or overview techniques that attempt to capture all participants in an endpoint's environment. However, in this implementation, the endpoint also, at620, continually evaluates distance sensor data to detect motion near the endpoint (i.e., “near motion”). These two steps may, for example, be performed as part of the analysis performed in step420of method400(ofFIG.4).

If, at630, no near motion is detected, the endpoint may continue, at610, to generate framings based on speaker tracking techniques. Alternatively, if at630, near motion is detected, the endpoint will, at640, generate and apply a best overview (or new best overview) framing, which may contain the entire field of view of the camera. Thus, in this implementation, near motion may help an endpoint detect a potential error (e.g., at step440of method400) and may also confirm the potential error (e.g., at step450of method400). That is, in this implementation, near motion may be a self-confirming potential error. Advantageously, this may ensure that a participant positioned adjacent to the endpoint will not be ignored and omitted from the best overview (or new best overview) framing when his or her head/face/eyes are out of the field of view of the camera. This is advantageous because a participant standing adjacent an endpoint is often presenting (e.g., pointing to a slide show), writing on a display, or otherwise interacting with the display of the endpoint and, thus, is often the focal point of a meeting. Consequently, at least some of this participant (e.g., his or her hand) should be included in an overview (or new overview) framing, even if his or her head/face/eyes are not able to be captured in the camera's field of view. For example, it may be important to show this participant's hand writing on a display or pointing to a display even if this participant's head/face/eyes are out of the camera's field of view.

After generating and applying a best overview framing at640, the endpoint will continue to evaluate near motion at650. However, at650, the endpoint may utilize a different threshold as compared to630. For example, the endpoint may use a larger threshold at650than is used at630. As a more specific example, at630, the endpoint may determine near motion is detected when a participant is within 1 meter of the endpoint (as determined based on distance sensor data). Then, at650, the endpoint may determine there is no near motion when the distance sensor does not detect motion within, for example, 1.3 meters of the endpoint. These different thresholds are just examples, but overall, different thresholds will create hysteresis to avoid rapid switches between regular speaker tracking and a best overview framing (which contains the whole view of the camera's widest-angle lens). The thresholds can be determined dynamically based on historical patterns, current movement patterns, or can be based on predetermined distances (e.g., experimentally determined distances).

Moreover, although steps630and650discuss “near motion,” method600could, in at least some embodiments, detect peripheral motion at steps630and650, insofar as “peripheral motion” may indicate motion towards a boundary of a camera's field of view. Peripheral motion could be detected in addition to, or as an alternative to, near motion in order to attempt to capture at least a portion of participants that are partially outside of the camera's field of view in an endpoint's video stream.

Still referring toFIG.6, but with reference toFIG.5, in some embodiments, method600and method500could be combined. For example, an endpoint might allow a user to toggle between an “occluded participant mode” where the endpoint executes method500and a “near motion participant mode” where the endpoint executes method600. Alternatively, an endpoint might dynamically switch between these two modes/methods. For example, method500and/or method600could be triggered in response to different triggers, such as constant motion, peripheral motion, near end motion, continuous audio-video mismatches, etc.

Reference is now made toFIG.7.FIG.7illustrates an example motion detection plot700that can be utilized to analyze distance sensor data (e.g., radar data). As mentioned, in at least some embodiments, the distance sensor may be a radar sensor that receives five motion detections per second. Additionally or alternatively, the distance sensor may be any other distance sensor operating at any other frequency. In any case, once motion detections are received, the motion detections can plotted onto a plot700of an endpoint's environment710. InFIG.7, the environment710is illustrated schematically as a conference room with a table. An endpoint (not shown) is positioned adjacent to one end of the table so that its distance sensor712faces the table. Thus, the field of view of the distance sensor712spans over most of the table and over an angular swath of the environment710.

In environment710, the plot700is an angular mapping720. The mapping720includes radial columns, represented as720A,720B, and720C, and each of these columns is broken into segments at different distances from the distance sensor712. As an example, the segments of column720A are labelled as segments720A-1,720A-2,720A-3, and720A-4. With such a mapping (or any variation thereof), distance sensor data (e.g., motion detections) can be plotted over time to provide indications of movement or motion. The endpoint can track how many detections have been recorded in each segment (i.e., each cell) and, then, motion rates can be determined based on the number of detections in a cell (or in the plot) over the past X number of seconds. Notably, plot700ofFIG.7is only provided as an example and is not to be at scale. As example dimensions, angular columns (represented schematically as columns720A-C) might be each span 10 degrees and the segments of each column (e.g., segments720A-1to720A-4) may be 50 centimeters deep.

To summarize, in one form, a method is provided comprising: obtaining at least a video stream during an online conference session; analyzing the video stream, an audio stream received with the video stream, or both the video stream and the audio stream; based on the analyzing, composing a framing that either focuses on a speaker in the video stream or provides an overview of participants in the video stream, the framing being; detecting a potential error in the framing based on further analysis of at least one of the video stream, the audio stream, or distance sensor data received with the video stream; maintaining the framing if the distance sensor data contradicts the potential error; and generating a new framing if the distance sensor data confirms the potential error.

In another form, an apparatus is provided comprising: a network interface configured to provide network connectivity; a camera configured to capture video in an environment of the apparatus; one or more microphones configured to capture audio in the environment of the apparatus; a distance sensor configured to capture data representative of participant locations in the environment of the apparatus; a processor coupled to the network interface, the camera, the one or more microphones, and the distance sensor, the processor that: obtains video captured by the camera; analyzes the video, audio captured by the one or more microphones, or both the video and the audio; composes a framing that either focuses on a speaker in the video or provides an overview of participants in the video stream, the framing being, the framing being based on analysis of the video, the audio, or both the video and the audio; detects a potential error in the framing based on further analysis of at least one of the video, the audio, or distance sensor data received from the distance sensor; maintains the framing if the distance sensor data contradicts the potential error; and generates a new framing if the distance sensor data confirms the potential error.

In yet another form, a non-transitory computer-readable storage media is provided that is encoded with software comprising computer executable instructions and when the software is executed operable to: obtain at least a video stream during an online conference session; analyze the video stream, an audio stream received with the video stream, or both the video stream and the audio stream; compose a framing that either focuses on a speaker in the video stream or provides an overview of participants in the video stream, the framing being, the framing being based on analysis of the video stream, the audio stream, or both the video stream and the audio stream; detect a potential error in the framing based on further analysis of at least one of the video stream, the audio stream, or distance sensor data received with the video stream; maintain the framing if the distance sensor data contradicts the potential error; and generate a new framing if the distance sensor data confirms the potential error.

The above description is intended by way of example only. Although the techniques are illustrated and described herein as embodied in one or more specific examples, it is nevertheless not intended to be limited to the details shown, since various modifications and structural changes may be made within the scope and range of equivalents of the claims. In addition, various features from one of the embodiments may be incorporated into another of the embodiments. Accordingly, it is appropriate that the appended claims be construed broadly and in a manner consistent with the scope of the disclosure as set forth in the following claims.

It is also to be understood that the term “approximately” and terms of its family (such as “approximate,” etc.) should be understood as indicating values very near to those that accompany the aforementioned term. That is to say, a deviation within reasonable limits from an exact value should be accepted, because a skilled person in the art will understand that such a deviation from the values indicated is inevitable due to measurement inaccuracies, etc. The same applies to the terms “about” and “around” and “substantially.” Additionally, terms such as “transmit” and “receive” are broadly used herein to refer to techniques for providing and obtaining data in network environments. For example, data may be provided and obtained through packets transmitted and received through a network (e.g., Internet110ofFIG.1), but may also be provided and obtained through data communicated via out-of-band signaling or control channels used in computing environment100.