DETECTION OF AUDIO ISSUES ON A COMMUNICATION SESSION FROM VIDEO ANALYSIS

The technology disclosed herein enables detection of audio issues for a participant on a communication session from analysis of video of the participant. In a particular embodiment, a method includes receiving video of a first participant communicating over a communication session between a first endpoint of the participant and a second endpoint of a second participant. The method further includes determining from the video that the participant is speaking. In response to determining that the participant is speaking, the method includes determining an audio issue exists due to audio of the first participant not corresponding to the video and notifying the first participant about the audio issue.

TECHNICAL BACKGROUND

A relatively common occurrence on communication sessions, such as conference calls, over which multiple participants are communicating in real-time, is that a participant is unaware that they are not being heard by other participants. For instance, the participant may have forgotten to disable a local mute setting on their endpoint, may be having microphone problems, or may have a bad connection to the communication session. Unless another participant informs the speaking participant that they are not being heard (if the other participant even recognizes that the speaking participant is trying to speak), the speaking participant may fruitlessly continue speaking. Moreover, even if the other participant informs the speaking participant about their audio issue, the speaking participant may have difficulty pinpointing the cause of the audio issue (e.g., on mute, bad connection, etc.).

SUMMARY

The technology disclosed herein enables detection of audio issues for a participant on a communication session from analysis of video of the participant. In a particular embodiment, a method includes receiving video of a first participant communicating over a communication session between a first endpoint of the participant and a second endpoint of a second participant. The method further includes determining from the video that the participant is speaking. In response to determining that the participant is speaking, the method includes determining an audio issue exists due to audio of the first participant not corresponding to the video and notifying the first participant about the audio issue.

In some embodiments, notifying the first participant includes displaying a visual alert indicating the audio issue on a display of the first endpoint and/or playing an audible alert indicating the audio issue through a speaker of the first endpoint.

In some embodiments, determining the audio issue exists includes determining that a setting of the communication session with respect to the audio is causing the audio and the video to not correspond, determining that a hardware issue is causing the audio issue, determining that the audio is not synchronized in time with the video, and/or determining that the audio is below a threshold audio quality or is not being received.

In some embodiments, in response to determining the audio issue, the method includes determining that a secondary microphone of the first endpoint is capturing secondary audio of the first participant and, in response to determining that the secondary microphone is capturing the secondary audio, transferring the secondary audio over the communication session.

In some embodiments, the method includes notifying the second participant about the audio issue.

In some embodiments, receiving the video includes capturing the video from a camera of the first endpoint or receiving the video over a network connection with the first endpoint.

In another embodiment, an apparatus is provided having one or more computer readable storage media and a processing system operatively coupled with the one or more computer readable storage media. Program instructions stored on the one or more computer readable storage media, when read and executed by the processing system, direct the processing system to receive video of a first participant communicating over a communication session between a first endpoint of the participant and a second endpoint of a second participant. The program instructions further direct the processing system to determine from the video that the participant is speaking. In response to determining that the participant is speaking, the program instructions direct the processing system to determine an audio issue exists due to audio of the first participant not corresponding to the video and notify the first participant about the audio issue.

DETAILED DESCRIPTION

The examples provided herein enable detection of audio issues for a participant on a communication session based on video captured of the participant. In particular, video is captured of the participant and the video is analyzed to determine whether the participant is speaking. If the participant is speaking according to the video analysis but audio captured of the participant does not correspond to the video, then the participant is notified that an audio issue is present. For example, the audio may not exist at all (e.g., a microphone signal may not be received), the audio may be out of sync with the video, the audio may be of low quality, or some other type of issue may exist that adversely effects the audio's ability to be heard on the communication session. Notifying the participant about the audio issue allows the participant to troubleshoot a fix for the audio issue, if possible, and saves other participants on the call from having to notify the participant of the issue, if any of the other participants even recognize or care that an issue is occurring.

FIG. 1illustrates an implementation100for using video analysis to detect audio issues on a communication session. Implementation100includes communication session system101, endpoint102, and endpoint103. Participant122is a user that operates endpoint102and participant123is a user that operates endpoint103. Endpoint102and communication session system101communicate over communication link111. Endpoint103and communication session system101communicate over communication link112. Communication links111-112are shown as direct links but may include intervening systems, networks, and/or devices.

In operation, endpoint102and endpoint103may each respectively be a telephone, tablet computer, laptop computer, desktop computer, conference room system, or some other type of computing system capable of connecting to a communication session facilitated by communication session system101. Communication session system101facilitates real-time communication sessions between two or more endpoints, such as endpoint102and endpoint103. In some examples, communication session system101may be omitted in favor of a peer-to-peer communication session between endpoint102and endpoint103. A communication session may be audio only (e.g., a voice call) or may also include at least a video component (e.g., a video call). During a communication session, participant122and participant123are able to speak with, or to, one another by way of their respective endpoints102and103capturing their voices and transferring the voices in audio signals over the communication session. In some situations, there may be issues with the audio captured of participant122and participant123. Operation200is performed to detect an audio issue and notify one or more of participant122and participant123about the issue.

FIG. 2illustrates operation200to use video analysis to detect audio issues on a communication session. Operation200may be performed by any one of systems101-103and, in some cases, portions of operation200may be distributed among two or more of systems101-103. Operation200includes receiving video of participant122who is communicating over a communication session between endpoint102and endpoint103(201). While the communication session is only described to be between two endpoints, it should be understood that any number of two or more endpoints may be on the communication session. If the video is received by endpoint102, then receiving the video may include capturing the video using a camera of endpoint102(e.g., a camera built into endpoint102or otherwise connected to endpoint102as a peripheral). If communication session system101or endpoint103is receiving the video, then receiving the video may include receiving a signal including the video over communication link111and/or communication link112. In those examples, the video may be included in the user communications transmitted on the communication session (e.g., when the communication session is a video call) or may be sent out as an out of band signal associated with the communication session when the communication session does not include video of the participants (e.g., when the communication session is a voice call). The video may be encoded in any video format supported by systems101-103.

Upon receiving the video, operation200includes determining from the video that participant122is speaking (202). Facial recognition algorithms may be used to identify the face of participant122within the video image and, more specifically, may identify where on the face the mouth is located. Participant122may be determined to be speaking if the mouth is simply moving (e.g., lips of the mouth opening and closing) or is moving in a manner consistent with a person speaking. For instance, a machine learning algorithm may be trained to recognize speech movements rather than other types of mouth movements, such as fidgety mouth movements, where a participant is not actually speaking. In further examples, the algorithm may be trained to recognize the actual sounds (possibly even whole words) that would be made from certain mouth movements, which may then be used in the next step to determine whether a particular audio issue is occurring. In some examples, aspects of participant122other than mouth movement may be identified from the video. For instance, participant122may be moving their body or gesturing (e.g., hand movements, head movements, etc.) in a manner that is consistent with a person speaking.

In response to determining that the participant is speaking, operation200includes determining that an audio issue exists due to audio of participant122not corresponding to the video (203). The audio issue may be no audio being received of participant122(e.g., no audio captured by endpoint102and/or received over communication links111/112), audio being received that does not include speech, audio being received but the included speech is of low quality so as to make it difficult to comprehend by participant123when played at endpoint103(e.g., speech volume is too low, audio includes static, portions of the audio are cutting out, etc.), audio being received but the speech therein is not in sync with the video (e.g., speech sounds determined through the above video analysis do not match in time to the sounds included in the audio), or some other type of audio issue that would detract from participant123's ability to hear/comprehend speech from participant122over the communication session. The system performing operation200may be configured to only monitor for a certain type of audio issue (e.g., only one of the above audio issue examples) or may be configured to monitor for two or more types of audio issues (e.g., two or more of the above audio issue examples). In some examples, the audio issue may be determined based solely on the audio itself (or lack of audio) after speech is detected in the video (e.g., by determining whether speech is included in the audio or determining a quality of the audio) or may be determined based on the audio relative to other input information, such as the video (e.g., to determine if the audio is in sync with the video), audio received from another source (e.g., determine whether a microphone built into endpoint102captured speech while a microphone peripheral that was designated to capture the speech did not), setting information for the communication session (e.g., whether endpoint102has a local mute enabled or a moderator remotely has participant122on mute), or from some other source of information relevant to potential audio issues.

After determining the existence of the audio issue, operation200includes notifying the participant122about the audio issue (204). If endpoint102itself identified the audio issue, then endpoint102may display a visual alert (e.g., displaying a popup graphic/window on a graphical display, illuminating a notification lamp, or some other manner of visually providing information to a user) that informs participant122about the audio issue, may play an audible alert (e.g., a voice message, tone, pattern of tones, jingle, etc.) that inform the user about the audio issue, may produce a tactile alert (e.g., vibration or vibration pattern), provide some other type of indication to participant122that informs participant122about the audio issue—including combinations thereof. If communication session system101or endpoint103determined the existence of the audio issue, then, to notify participant122, the systems101or103may transmit a message, or other type of signal, to endpoint102that directs endpoint102to present an indicator or alert, like those described above, to inform participant122about the audio issue. In some examples, participant122may be notified with information about the type of audio issue identified. For example, endpoint102may display an alert that includes text reciting the type of audio issue (e.g., “Warning: audio out of sync with video” or “Alert: endpoint on mute”). The more information participant122is presented with, the better participant122can determine how to remedy the audio issue. In some examples, participant123may also be notified in a similar manner to inform participant123that an issue with audio from participant122has been detected (e.g., so that participant123does not need to notify participant122themselves about the issue or to indicate to participant123that the audio issue is not caused by endpoint103).

With operation200occurring in substantially real-time during the communication session, participant122is quickly notified of any audio issue when that audio arises. If speech is detected in the ongoing video and the audio without issue, then the communication session simply proceeds as normal for participant122. Once an audio issue is detected, then participant122is promptly notified so that remedial action, if any, can be taken. For example, upon notifying participant122about a low speech quality audio issue, participant122may switch microphones in hopes that a bad microphone was a cause of the issue.

FIG. 3illustrates operational scenario300for using video analysis to detect audio issues on a communication session. Operational scenario300is an example of endpoint102performing operation200during a voice communication session with endpoint103. Although, some or all of the steps performed by endpoint102may alternatively be performed in systems101or103. In some examples, a software client for participating in communication sessions facilitated by communication session system101may execute on endpoint102to instruct endpoint102to perform operation200. Endpoint102captures video301and audio302of participant122at step1. Video301may be captured by a camera of endpoint102and audio302may be captured by a microphone of endpoint102. Endpoint102identifies speech of participant122from within video301at step2even though video301is not sent over the communication session. Upon determining that speech is occurring in video301, endpoint102also identifies whether speech is included in audio302at step3. In some cases, endpoint102may determine that no audio was actually received and, if no audio was received, then audio302(which would be non-existent) inherently cannot include speech. Endpoint102may identify the speech by using a speech to text algorithm and/or a natural language processing algorithm to extract words from audio302, if words are present in audio302. In some cases, endpoint102may distinguish between words spoken by participant122or words spoken by another person at endpoint102(e.g., someone talking in the background).

The speech identified in step3is used to determine that an audio issue exists in audio302at step4. If no speech was identified in audio302at step3, then no further analysis of audio302may need to be performed at step4because endpoint102may simply indicate that the lack of speech is the audio issue. Although, in some examples, endpoint102may determine a reason for the lack of speech in audio302. For instance, endpoint102may determine whether a software setting of the communication session (e.g., local mute) causes speech to be absent from audio302(or causing audio302to be missing altogether) or whether a hardware issue is causing the speech to be absent from audio302(or causing audio302to be missing altogether). In some examples, if no software configuration related cause is found, endpoint102may automatically determine that a hardware issue exists or endpoint102may attempt to troubleshoot the hardware (e.g., by activating another available microphone or performing a test procedure on the current microphone). If endpoint102did identify speech in audio302at step3, then endpoint102may further process the speech to determine whether an audio issue exists (e.g., may determine whether there is an audio quality issue with the speech, or with audio302as a whole, or whether audio302, and the speech therein, is in sync in time with video301). For example, endpoint102may determine an audio issue exists when the speech to background noise ratio is below a threshold or may determine using natural language processing that words are being dropped from the speech.

Once an audio issue is determined to exist, endpoint102notifies participant122about the audio issue at step5using alert303. Alert303may be a visual, audible, tactile, or some other type of alert that can be produced by a computing system like endpoint102. Alert303may generically indicate that an audio issue exists or may provide additional information related to the audio issue, such as the type of audio issue, a cause of the audio issue, a recommended remedy for the audio issue, or some other type of relevant information.

Even though an audio issue was identified in this example, endpoint102still sends audio302over the communication session at step6, so that endpoint103can receive audio302at step7and play audio302to participant123at step8. In other examples, if audio302is never captured or the audio issue is determined to be severe enough (e.g., below a threshold quality), then endpoint102may determine not to send audio302. Similarly, in some examples, endpoint102may transfer an instructions to endpoint103directing endpoint103to present an alert similar to alert303that notifies participant123about the audio issue that occurring with respect to audio302(e.g., may present a message reciting “Audio of participant122currently has issues).

FIG. 4illustrates implementation400for using video analysis to detect audio issues on a communication session. Implementation400includes communication session system401, endpoints402-405, and communication network406. Communication network406includes one or more local area and/or wide area computing networks, including the Internet, over which communication session system401and endpoints402-405communicate. Endpoints402-405may each comprise a telephone, laptop computer, desktop workstation, tablet computer, conference room system, or some other type of user operable computing device. Though only endpoint402is shown to have primary microphone431, secondary microphone432, and camera433for the purposes of this example, endpoints403-405may include similar components. Communication session system401may be an audio/video conferencing server, a packet telecommunications server, a web-based presentation server, or some other type of computing system that facilitates user communication sessions between endpoints. Endpoints402-405may each execute a client application that enables endpoints402-405to connect to communication sessions facilitated by communication session system401and provide features associated therewith, such as the detection of audio issues described below.

In operation, a real-time video communication session is established between endpoints402-405, which are operated by respective participants422-425. The video communication session enables participants422-425to speak with, and see, one another in real time via their respective endpoints402-405. During the video communication session communication session system401determines whether audio of participants422-425from endpoints402-405is experiencing issues and notifies endpoints402-405about those issues. Using communication session system401to identify audio issues, as described below, allows resources of endpoints402-405to be used for other tasks. Likewise, communication session system401may be better suited to identify the audio issues. For instance, endpoint402may be a battery powered device, such as a smartphone, and the processing power thereon may be far less than what is available to communication session system401.

FIG. 5illustrates operational scenario500for using video analysis to detect audio issues on a communication session. Operational scenario500focuses on detecting an audio issue in audio from endpoint402, although, audio issues may be detected in audio from endpoints403-405in a similar manner. In some examples, audio issues may be determined at both an endpoint and communication session system401so that a source of an audio issue can better be identified (e.g., if no issue is found at endpoint402but an issue is found at communication session system401, then the issue may be caused by the connection between endpoint402and communication session system401.

In operational scenario500, endpoint402captures video501and audio502from participant422at step1. Video501is captured by camera433and audio502is captured by primary microphone431. In this example, primary microphone431is one of two microphones of endpoint402that are available for capturing audio. For example, primary microphone431may be a microphone in a headset worn by participant422and connected to endpoint402either wired or wirelessly while secondary microphone432may be a built-in microphone of endpoint402. Primary microphone431is considered primary because it is currently designated for capturing audio502. Participant422may designate primary microphone431via input into endpoint402(e.g., may select primary microphone431through a user interface of endpoint402), primary microphone431may be the default microphone for communication sessions, primary microphone431may be selected at random, or primary microphone431may become primary in some other manner.

Video501and audio502are transferred to communication session system401at step2and communication session system401passes video501and audio502to endpoints403-405at step3. In this example, audio502is passed to endpoints403-405regardless of whether communication session system401determines that an audio issue exists. In other examples, communication session system401may refrain from sending audio502to endpoints403-405upon determining that an audio issue exists. At step4, communication session system401determines from video501that participant422is speaking. Communication session system401may use a facial recognition algorithm on video501and the algorithm may output that participant422is speaking currently or may provide time stamps when participant422is speaking so that the time stamps can be aligned with audio502. In some examples, communication session system401may only analyze video501after it cannot be determined from audio502itself that participant422is speaking. That is, communication session system401may conserve the processing resources need to process video501when it is clear from processing audio502that participant422is speaking therein.

After determining from video501that participant422is speaking, communication session system401determines that an audio issue exists at step5by analyzing audio502to determine whether the speech in audio502sufficiently matches that shown in video501. In one example, communication session system401may determine that no sound exists in audio502(or audio502may not be received at all in some cases) or at least no sound exists in audio502that is consistent with the voice of participant422. In another example, communication session system401may determine that mouth movements of participant422in video501are not consistent with the sounds in audio502. For instance, the sounds may be offset by an amount of time (e.g., delayed by half a second), may be distorted due to poor audio capture by primary microphone431, may be distorted by a bad connection over which audio502is sent, or some other reason in which the sound represented in audio502does not match what is expected based on video501.

Upon identifying the audio issue, communication session system401transmits issue notification503to endpoint402at step6. In this example, issue notification503includes information about the type of audio issue that was determined by communication session system401. The information about the type of audio issue may indicate that the issue is caused by a software setting (e.g., local mute), that speech is present but of lower than a threshold quality, that speech is fully absent from audio502, that audio502was not actually received, or some other description of the identified audio issue. Endpoint402also sends issue notifications504to endpoints403-405at step7. Issue notifications504may also include information about the type of audio issue detected by communication session system401or may be more generic by indicating that audio of participant422is experiencing an issue without specifying further details.

Upon receiving issue notification503, endpoint402presents alert505to participant422at step8. Since issue notification503indicated a type of audio issue, alert505indicates the type of audio issue in alert505. Notifying participant422about the type of audio issue may help participant422better troubleshoot the issue. For example, if alert505notifies participant422that a local mute setting is enabled, then participant422will know relatively quickly that they should instruct endpoint402to turn off the mute setting. In another example, if alert505indicates a low speech volume, then participant422may be able to determine whether something is blocking primary microphone431. Upon receiving issue notifications504, endpoints403-405similarly present alerts506to their respective participants423-425at step9. Issue notification503and issue notifications504may instruct endpoints402-405to present alert505and alerts506, may instruct endpoints402-405on how alert503and alerts504should be presented (e.g., visually or audibly), or endpoints402-405may be preprogramed on how to handle received alerts. In some examples, participants422-425may each indicate to their respective endpoints402-405about how alerts should be presented. For instance, participant422may prefer alerts to be both visual and audible, while participant423may prefer alerts to be only visual. Regardless of how alert505and alerts506are presented, upon completion of operational scenario500, participant422are all aware that there is an issue with audio502from endpoint402and can proceed with the communication session accordingly.

FIG. 6illustrates operation600to represent communication session quality using words spoken on the session. Operation600is an example of how an issue with audio captured by primary microphone431from endpoint402on a communication session may automatically be remedied. Operation600is performed in endpoint402but communication session system401may perform operation600in other examples where communication session system401has access to audio captured by secondary microphone432.

In operation600, endpoint402captures video using camera433and primary audio using primary microphone431(601). The primary audio is captured via primary microphone431because endpoint402is currently configured (e.g., by the user, by default, or otherwise) to use audio captured by primary microphone431on the communication session. Endpoint402then determines that, while participant422is speaking in the video, the captured primary audio does not match the video (602). In this case, the primary audio does not include speaking while the video does. In some cases, endpoint402may think it is capturing the primary audio while no audio is actually being captured due to a faulty or non-existent primary microphone431. For instance, primary microphone431may actually be a headset (e.g., hearing aid) that only includes speakers for hearing audio playback from endpoint402and endpoint402inadvertently assumes the headset also has a microphone. Since there is no microphone, no audio is actually captured from primary microphone431. In other examples, endpoint402may determine that other types of audio issues are occurring (e.g., poor speech quality) rather than there simply being no speech at all.

Endpoint402then analyzes secondary audio being captured by secondary microphone432to determine that the secondary audio matches the video (603). Endpoint402may already be capturing the secondary audio so that endpoint402can go back and analyze the same time frame that was analyzed with respect to the primary audio. In other examples, endpoint402may begin capturing the secondary audio from secondary microphone432upon determining that the primary audio does not match the video. The secondary audio would then be compared to the corresponding video that is captured at the same time with the secondary audio. Since the secondary audio matches the video, endpoint402transmits the secondary audio over the communication session instead of the primary audio (604). In some examples, endpoint402may also notify participant422that the secondary audio is now being used on the communication session.

In some examples, the secondary audio may also be used to help determine the type of audio issue that is occurring. For instance, if the secondary audio is experiencing a similar audio issue as the primary audio (e.g., low quality), then the audio issue is likely not of a type that is caused by a microphone hardware issue. Endpoint402may then narrow down the list of possible issue types by removing the microphone hardware issue from the potential possibilities.

FIG. 7illustrates display system700for using video analysis to detect audio issues on a communication session. Display system700includes display711and camera721. Display711may be a cathode ray tube (CRT), Liquid Crystal Display (LCD), Light Emitting Diode display (LED), or some other type of display capable of presenting images described below. Camera721includes optics and an image sensor for capturing video of a participant viewing display711. Though not shown, display system700may be a display system for an endpoint described elsewhere herein.

Display711is displaying an example Graphical User Interface (GUI) for a client application connected to a video communication session between participants702-706, as shown in participant list734. Participant702is operating the endpoint of display system700. The GUI also shows participant windows731-733and end call button741, which removes the participant at display system700from the communication session when pressed. Real-time video of participant706is shown in participant window731, which is larger than participant window732and participant window733because participant706was recently speaking. Participant window732shows real-time video of participant703and participant window733shows real-time video of participant704. Video of the remaining participants on the communication session may not displayed because those participants are not one of the three most recent speakers, those participants do not have video enabled, or for some other reason.

In this example, participant702is now speaking on the communication session. Video and audio of participant702should, therefore, be displayed at the endpoints of participants703-706. Audio captured by the endpoint of participant702is determined to have an issue in accordance with the examples described above. As such, the client application directs the endpoint to display audio issue alert735on display711. The endpoint may also play an audible and/or provide a tactile alert to ensure participant702is aware of audio issue alert735. In this example, audio issue alert735notifies participant702that the audio issue is an audio quality issue. In other examples, audio issue alert735may provide additional details about the audio quality issue, such as low speech volume or dropped words. Participant702can then attempt to remedy the quality issue before continuing to speak since the other participants may have trouble comprehending participant702due to the quality issue.

FIG. 8illustrates computing architecture800for using video analysis to detect audio issues on a communication session. Computing architecture800is an example computing architecture for endpoints102,402and communication session systems101,401, although systems102,402,101, and401may use alternative configurations. Computing architecture800may also be used for other computing systems described herein. Computing architecture800comprises communication interface801, user interface802, and processing system803. Processing system803is linked to communication interface801and user interface802. Processing system803includes processing circuitry805and memory device806that stores operating software807.

Communication interface801comprises components that communicate over communication links, such as network cards, ports, RF transceivers, processing circuitry and software, or some other communication devices. Communication interface801may be configured to communicate over metallic, wireless, or optical links. Communication interface801may be configured to use TDM, IP, Ethernet, optical networking, wireless protocols, communication signaling, or some other communication format—including combinations thereof.

User interface802comprises components that interact with a user. User interface802may include a keyboard, display screen, mouse, touch pad, or some other user input/output apparatus. User interface802may be omitted in some examples.

Processing circuitry805comprises microprocessor and other circuitry that retrieves and executes operating software807from memory device806. Memory device806comprises a computer readable storage medium, such as a disk drive, flash drive, data storage circuitry, or some other memory apparatus. In no examples would a storage medium of memory device806be considered a propagated signal. Operating software807comprises computer programs, firmware, or some other form of machine-readable processing instructions. Operating software807includes audio issue module808. Operating software807may further include an operating system, utilities, drivers, network interfaces, applications, or some other type of software. When executed by processing circuitry805, operating software807directs processing system803to operate computing architecture800as described herein.

In particular, audio issue module808directs processing system803to receive video of a first participant communicating over a communication session between a first endpoint of the participant and a second endpoint of a second participant. Audio issue module808also directs processing system803to determine from the video that the participant is speaking. In response to determining that the participant is speaking, audio issue module808directs processing system803to determine an audio issue exists due to audio of the first participant not corresponding to the video and notify the first participant about the audio issue.

The descriptions and figures included herein depict specific implementations of the claimed invention(s). For the purpose of teaching inventive principles, some conventional aspects have been simplified or omitted. In addition, some variations from these implementations may be appreciated that fall within the scope of the invention. It may also be appreciated that the features described above can be combined in various ways to form multiple implementations. As a result, the invention is not limited to the specific implementations described above, but only by the claims and their equivalents.