Video conference audio/video verification

In one embodiment, a method implemented on a video conference participant device includes: receiving video conference data for a video conference session, performing at least one of the following diagnostic tests: verifying receipt of a presenter image from a presenter video conference device where the presenter video conference device is in use by a participant of the video conference meeting, the participant designated as a presenter for said video conference meeting, verifying receipt of shared desktop content from the presenter video conference device, or verifying receipt of audio data from the presenter video conference device; and transmitting results of said performing to said presenter video conference device.

FIELD OF THE INVENTION

The present invention generally relates to verification of audio/video data received during a video conference session.

BACKGROUND OF THE INVENTION

Video conferencing is known in the art. During a video conference session, one or more participant devices typically receive audio and/or video data provided by one or more other participant devices. The audio data typically represents voice and/or ambient noise picked up by a microphone used by one or more of the other participant devices; the video data typically represents camera images of the other participants and/or images of a shared desktop from a participant device used by a participant designated as a presenter for the video conference session. The audio/video data is then presented on the receiving participant device, thereby enabling its associated user, i.e., a “participant” of the video conference session, to see and hear the other participants of the session.

DESCRIPTION OF EXAMPLE EMBODIMENTS

Overview

A method implemented on a video conference participant device includes: receiving video conference data for a video conference session, performing at least one of the following diagnostic tests: verifying receipt of a presenter image from a presenter video conference device where the presenter video conference device is in use by a participant of the video conference meeting, the participant designated as a presenter for said video conference meeting, verifying receipt of shared desktop content from the presenter video conference device, or verifying receipt of audio data from the presenter video conference device; and transmitting results of said performing to said presenter video conference device.

A method implemented on a presenter video conference device includes: receiving verification results from at least one video conference participant device, the verification results indicating at least one of: whether a presenter image from the presenter video conference device has been received, whether shared desktop content from the presenter video conference device has been received, or whether audio data from the presenter video conference device has been received; and presenting at least an indication of the verification results on the presenter video conference device.

DETAILED DESCRIPTION OF EXAMPLE EMBODIMENTS

It's not uncommon that speakers or presenters in a video conference meeting are unsure whether the other participants in the meeting are actually able to hear them, see them, and/or to see the content they are sharing. Accordingly, speakers/presenters often interrupt themselves to inquire: “can everyone hear me?” or “are you seeing the slides yet?” Sometimes the other participants respond; sometimes they don't. When the other participants do respond, they often do so at the same time, rendering it difficult to determine whether or not all of the participants have indeed responded. And since any participant who indeed does not hear the speaker, will as a consequence also not hear the question, a participant's non-response may be presumed to indicate that the participant does not indeed hear the speaker. Accordingly, it will be appreciated that not only may such inquiries detract from the overall video conference experience, they may also be a rather ineffective way to actually determine whether or not the other participant devices are actually receiving and properly presenting audio data provided by the speaker's participant device.

Reference is now made toFIG. 1, which illustrates a multiplicity of video conference participant devices100participating in an exemplary video conference meeting via a video conference server200. In accordance with an exemplary embodiment, the user of video conference participant device100A has been designated as a presenter for the ongoing meeting depicted inFIG. 1. Audio/video data produced by video conference participant device100is forwarded to video conference server200via the Internet and/or any other suitable communication network, such as, for example but not limited to, a wide area network (WAN), a local area network (LAN), a mobile communication network, a landline telephone service, etc.

It will be appreciated that for a variety of possible reasons, video conference participant devices100B,100C and100D may not all necessarily receive all of the audio/video data generated by the presenter using video conference participant device100A. The quality of an IP connection to video conference server200may vary, resulting in loss of data. Furthermore, some or all of the data may be transmitted via different media with different levels of quality and/or service. For example, video conference participant device100B may be configured to receive audio data from video conference server200via a voice over IP (VOIP) connection; video conference participant device100C may be configured to receive audio data through a mobile network; and/or video conference participant device100D may be configured to receive audio data through “plain old telephone service” (POTS). Each of these delivery mechanisms may provide service of different levels of quality over time, thereby possibly affecting the quality of audio/video signals received by the individual video conference participant devices100in a not necessarily uniform manner.

It will similarly be appreciated that irrespective of the audio/video data actually received, the configuration and resources of each video conference participant device100may also affect the presentation of the video conference meeting. For example, other applications running in parallel may compete for resources for the video conference meeting. Also, the volume controls may be set such that even if the audio data is received, it may not be played loudly enough for the user to hear.

In accordance with embodiments described herein, video conference participant devices100are configured with diagnostic functionality to assess whether or not the audio/video data provided by the presenter is indeed received and accurately presented. In accordance with embodiments described herein the presenter may request that one or more of the video conference participant devices100use this functionality to verify that the audio/video presentation has been received and displayed accurately. For example, as depicted inFIG. 1, video conference participant device100C may not be displaying any video at all; video conference participant device100B may be displaying the image of the presenter; and video conference participant devices100D may be displaying a graph shared by the presenter.

Video conference server200comprises processor210; I/O module220; and video conference application260. It will be appreciated that video conference server200comprises hardware and software components, such as are well-known in the art. It will similarly be appreciated that video conference server200may comprise other components that are not depicted inFIG. 1. Video conference server200may comprise more than one processor210. For example, one such processor210may be a special purpose processor operative to execute video conference application260. Video conference application260is an application that is implemented in software and/or hardware on video conference server200. Video conference application260may be any suitable server-side application for video conferencing such as, for example, Cisco WebEx or GoToMeeting from Citrix Systems, Inc. In accordance with embodiments described herein, video conference application260comprises presenter verification module205which may be configured to support the performance of various verification processes performed by video conference participant devices100.

I/O module220is a hardware and/or software component that is configured to transmit and receive data to and from devices in communication with video conference server200. For example, I/O module may be implemented as a transceiver. Video conference server200is configured to use I/O module220to facilitate a video conference meeting between users of video conference participant devices100as per instructions received from video conference application260.

As described hereinabove, per the exemplary embodiment ofFIG. 1, the user of device100A has been designator as the presenter and the users of devices100B,100C and100D are listening and/or watching the audio/video presentation provided by device100A.

As depicted inFIG. 1, video conference participant device100A comprises display screen120, camera130and microphone150. It will be appreciated that while video conference participation device100A is depicted as a collection of peripheral devices attached to a personal computer, other configuration are also supported by the embodiments described herein. For example, video conference participant device100A may alternatively be implemented as a single integrated hardware device such as, for example, a personal TelePresence unit available commercially from Cisco Systems, Inc. Alternatively, video conference participant device100A may be implemented as a dedicated video conference suite with one or more installed display screens120, cameras130and/or microphones150.

It will be appreciated that each of video conference participant devices100B,100C and100D may also be configured and implemented in accordance with any of the configurations described hereinabove with relation to video conference participant device100A. It will therefore also be appreciated that while for ease of reference, each of video conference participation devices100are referred to as single integrated unit, in practice video conference participation units100may be comprised of a multiplicity of connected components.

Reference is now also made toFIG. 2which is a block diagram drawing of an exemplary video conference participation device100, constructed and operative in accordance with embodiments described herein. Video conference participation device100comprises processor110, I/O module120, display screen125, camera130, one or more speakers140, microphone150and video conference client160. Video conference client160is implementable as either a software application and/or hardware component and is executable by processor110to facilitate participation by video conference participant device100in a video conference meeting such as that depicted inFIG. 1. Processor110also operates I/O module120, display screen125, camera130, speaker(s)140and microphone150in support of video conference meetings as per instructions provided by video conference client160. For example, I/O module120sends and receives audio/video data between the participants of the video conference meeting; received video data is displayed on display screen125; camera130provides video data to be sent to other video conference participant devices100; speaker(s)140play(s) the received audio data; and microphone150provides audio input to be sent to other video conference participant devices100.

In accordance with embodiments described herein, video conference client160comprises presenter verification module165. Presenter verification module165is implementable as either a software application and/or hardware component that may be implemented as either an integrated component of video conference client160or as an independent module in communication with video conference client160. As will be described hereinbelow, presenter verification module165is configured to perform diagnostic tests on video conference participation device100to verify whether or not the audio/video data provided by another participant of a video conference meeting is indeed received and presented accurately.

Reference is now also made toFIG. 3, which is a flow chart illustration of a presenter AN data verification process300to be performed by video conference client160during a video conference meeting. Video conference client160receives (step310) a data stream including audio/video data associated with the ongoing video conference meeting. It will be appreciated that such data may be provided as a single data stream from video conference server200, which may aggregate video and audio data received from multiple video conference participant devices. Alternatively, video conference server may provide the video data and some or all of the audio data may be provided via one or more external communication networks such as mobile or POTS communication networks. The audio/video data is received by I/O module120and forwarded for presentation by video conference client160using display screen125and/or speaker(s)140.

It will be appreciated that the data stream may also include control data associated with the video conference meeting. For example, the data stream may also include a request submitted by the presenter to verify receipt and accurate presentation of the audio/video data. If a presenter verification request is not received in the data stream (step320), process control will return to step310. Alternatively, or in addition, video conference client160may be configured to periodically perform steps330,350,370and/or399without a specific request to do from the presenter.

If a presenter verification request is not received in the data stream (step320), video conference client160may invoke presenter validation module165to verify (step330) that the presenter's image has been received and properly displayed on display screen125.

Presenter verification module165may also verify (step350) that content shared from the presenter's desktop has been received and properly displayed on display screen125. It will be appreciated that while sharing of desktop content is a popular feature of video conference meetings; it may not always be relevant. If the presenter is not currently sharing and desktop content, the presenter verification request may indicate as much and step350may be skipped.

Presenter verification module165may also verify (step370) that audio data produced by the presenter, for example, the presenter's voice, has been properly received and played on speaker(s)140. It will be appreciated that the audio data may also include an audio track from shared desktop content. It will similarly be appreciated that the presenter may not be the only participant using the same video conference participant device100. For example, if the presenter's video conference participant device100(e.g., video conference participant device100A as per the exemplary embodiment ofFIG. 1) is a video conference room there may be multiple voices represented in the received audio data. It will similarly be appreciated that in any case, there may be other participants speaking at the same time as the presenter, regardless of their location.

Presenter verification module165may return (step399) the results of steps330,350and370to the presenter via I/O module120and process control may loop back to step310.

Reference is now made toFIG. 4, which is a flow chart illustration of the details of step330in process300. Presenter verification module165may compare observed packet loss of video data versus a defined packet loss threshold. If the packet loss is greater than the defined threshold (step332), it may be determined that the quality of the video data is unacceptable. Presenter verification module165may set (step340) the result to “FAILED” and return (step342) the result for inclusion in the results returned to the requester in step399(FIG. 3).

It is not uncommon for cameras130(FIG. 1) implemented in a personal computer based video conference participation devices100to have a lens cover designed to protect camera130when not in use. It will be appreciated that it is also not uncommon that a participant in a video conference meeting may forget to remove the lens cover, thereby producing an image consisting largely or entirely of black or at least dark grey pixels when the camera is operative during a video conference meeting. Similar results may be produced by the selection of the wrong camera for use in a video conference. For example, a computer based video conference participation device100may also be configured with two cameras, such as a laptop's integrated camera and an external camera for use when the laptop is used with a docking station. In such a case a participant may initially select the integrated camera for use even though the laptop is closed, thereby producing a black image. It will similarly be appreciated that similar results may be produced by a defective camera. Accordingly, presenter verification module165may determine whether the percentage of black or grey pixels in the presenter's image is greater than a defined threshold percentage. If the percentage of “dark” pixels is greater than the defined threshold (step334), it may be assumed that the presenter forgot to remove the lens cover, selected the wrong camera, or that the camera is defective. Presenter verification module165may set (step340) the result to “FAILED” and return (step342) the result for inclusion in the results returned to the requester in step399(FIG. 3).

It is also not uncommon for camera130to be positioned such that it does not capture the full image of the presenter. A mounted camera such as used with a personal computer based video conference participant device100may have been moved slightly such that it is no longer angled towards the presumed location of the presenter. Alternatively, the presenter may have moved from a coverage area for camera130. In cases where the presenter's video conference participation device100is a dedicated video conference room, there may be multiple cameras130, and the “wrong” image, i.e., not that of the presenter, may be forwarded to the other participants.

Accordingly, presenter verification module165may use an image processing library, for example, OpenCV, originally developed by Intel Research and now supported by Willow Garage and Itseez, to detect whether there the received video actually includes a facial image associated with the presenter. If a face is not detected in the video data (step336), it may be determined that camera130is incorrectly positioned. Presenter verification module165may set (step340) the result to “FAILED” and return (step342) the result for inclusion in the results returned to the requester in step399(FIG. 3).

Alternatively, or in addition face recognition software may be employed to verify that the detected facial image is indeed that of the presenter. If the face in the image is not recognized as that of the presenter verification module165may set (step340) the result to “FAILED” and return (step342) the result for inclusion in the results returned to the requester in step399(FIG. 3).

If step338is reached without the result being set to “FAILED” in step340, presenter verification module165may set (step340) the result to “PASSED” and return (step342) the result for inclusion in the results returned to the requester in step399(FIG. 3).

It will be appreciated that the specific order of execution for steps332,334and336may be exemplary. In accordance with embodiments described herein, presenter verification module165may perform these steps in any suitable order. Furthermore, in accordance with accordance with embodiments described herein, presenter verification module165may be configured to perform only a subset of these steps, e.g., steps332and334, but not336; steps334and336, but not332; only336, but not332and334; etc.

Still further, in accordance with accordance with embodiments described herein, presenter verification module may be configured to perform the verification tests of each of steps332,334and336regardless of their outcome; failure of one verification test may not necessarily preclude the performance of subsequent verification tests. In such embodiments, the result returned in step342would be an aggregated result including details for each of the individual verification tests.

It will be appreciated that steps334and/or336may also be performed on the presenter's video conference participant device100(e.g. device100A per the exemplary embodiment ofFIG. 1). Similarly, steps334and/or336may also be performed by presenter verification module205.

Reference is now made toFIG. 5, which is a flow chart illustration of the details of step350in process300. Presenter verification module165may verify that a video stream has been received for the shared desktop content. Otherwise, presenter verification module165may set (step362) the result to “FAILED” and return (step364) the result for inclusion in the results returned to the requester in step399(FIG. 3). It will be appreciated that in a typical video conference meeting, at any given time, only the current presenter may provide shared desktop content, such that if an associated video stream is received (step352) it may be assumed to be from the presenter; i.e., there is presumably no need to verify that the video stream is received specifically from the presenter.

Even if presenter verification module165confirms that a video stream has been received for the shared desktop content, it is possible that, for whatever reason, the video display of the shared content may be out of sync with the video display on the source device100. It is therefore not uncommon for a presenter to ask the other participants to confirm that they are seeing the correct slide in a presentation of shared desktop content; there may have been a problem with the continued reception of the video stream sometime after the data was first received by I/O module120.

In accordance with embodiments described herein, the source device100may be configured to embed serial numbers in the outgoing video stream of the shared content. For example, if the shared content is a deck of slides such as may be produced using Microsoft PowerPoint or similar applications, video conference client160may be configured to detect a slide change. A slide change may be detected via image comparisons (i.e., a significant change in the video image followed by little or no change), keyboard/mouse input (i.e., detection of an arrow or mouse click to advance a slide), and/or any other suitable method. Video conference client160may embed a serial number in the outgoing video stream whenever a slide change is detected. If the shared content is a video clip, video conference client160may embed a serial number or relative timestamp (e.g., every five, ten or twenty seconds from the start of the video clip) at intervals in the video clip. Alternatively, or in addition, pre-existing timeline metadata may be used instead of an explicit serial number or timestamp.

Accordingly, presenter verification module165may be configured to detect (step354) a serial number, timestamp and/or metadata embedded in the received video stream and forward (step356) a copy of the detected serial number, timestamp and/or metadata to the presenter for comparison to a most recent serial number, timestamp and/or metadata associated with a current image displayed at the source device (i.e., the video conference participant device100in use by the presenter). Alternatively, or in addition, a copy of the current serial number, timestamp and/or metadata may be included with the verification request received in step320(FIG. 3) and presenter verification module165may perform the comparison locally, with control proceeding to step362if there is no match between the detected and serial number, timestamp and/or metadata.

It is possible that a participant may not be able to see the shared desktop content simply because video conference client160has been minimized and/or another application on video conference participant device100has been brought to the foreground. It will be appreciated by one of ordinary skill in the art that functionality for the detection of the status of a video conference window is known; for example, the host device of a Cisco WebEx training session may be configured to detect the background/foreground window status for other devices participating in the training session. Accordingly, presenter verification module165may be configured to use similar functionality (for example, implemented using operating system APIs) to determine whether or not the window for video conference client160is currently displayed “on top”, i.e., in the foreground (step360). If not, presenter verification module165may set (step362) the result to “FAILED” and return (step364) the result for inclusion in the results returned to the requester in step399(FIG. 3).

If step360is reached without the result being set to “FAILED” in step362, presenter verification module165may set (step360) the result to “PASSED” and return (step364) the result for inclusion in the results returned to the requester in step399(FIG. 3).

It will be appreciated that the specific order of execution for steps352,354/356and358may be exemplary. In accordance with embodiments described herein, presenter verification module165may perform these steps in any suitable order. Furthermore, in accordance with accordance with embodiments described herein, presenter verification module165may be configured to perform only a subset of these steps, e.g., steps352and354/356, but not358; steps354/356and358, but not352; only354/356, but not352and358; etc.

Still further, in accordance with accordance with embodiments described herein, presenter verification module165may be configured to perform the verification tests of each of steps352,354/356and358regardless of their outcome; failure of one verification test may not necessarily preclude the performance of subsequent verification tests. In such embodiments, the result returned in step364would be an aggregated result including details for each of the individual verification tests.

Reference is now made toFIG. 6, which is a flow chart illustration of the details of step370in process300. Presenter verification module165may verify that an audio stream has been received for the ongoing video conference. Otherwise, presenter verification module165may set (step384) the result to “FAILED” and return (step386) the result for inclusion in the results returned to the requester in step399(FIG. 3). Presenter verification module165may also compare observed packet loss of audio data versus a defined packet loss threshold. If the packet loss is greater than the defined threshold it may be determined that the quality of the video data is unacceptable and the result in may nonetheless be set to “FAILED” in step384even though an audio stream was indeed received.

It will be appreciated by one of ordinary skill in the art that in a typical video conference meeting, more than one participant at any given time. Accordingly, presenter verification module165may verify that audio signals are being received by video conference participant device100, without verifying receipt of the presenter's audio per se.

Presenter verification module165may also check whether the received audio stream possesses speech-like properties (step374). For example, presenter verification module165my employ any suitable digital signal processing algorithm to detect speech-like properties in the received audio data. Alternatively, or in addition, speech recognition algorithms may be employed to detect and recognize a configurable minimum number of words to indicate that the audio data has speech-like properties. If the received audio data lacks speech-like properties, presenter verification module165may set (step384) the result to “FAILED” and return (step386) the result for inclusion in the results returned to the requester in step399(FIG. 3) if the audio stream does not possess speech-like properties.

It will be appreciated by one of skill in the art that the configuration of the receiving video conference participant device100may impact on the ability of presenter verification module165to assess whether or not the received audio signals are being provided accurately to the user. Specifically, as discussed hereinbelow, such verification may require audio feedback, i.e., that microphone150(FIG. 2) be open and capable of detecting sound from speaker(s)140(FIG. 2). If microphone150is muted, and/or speaker(s)140is/are implemented in a headset unit such that microphone150cannot reliably pick up sound from speaker(s)140, presenter verification module165may not be able to perform some verification tests.

Presenter verification module165may therefor determine whether or not microphone150and speaker(s) are configured to provide audio feedback (step376). For example, video conference client160may be configured to enable a user to use a menu option to mute microphone150. If microphone150is set on mute, there may be no audio feedback. Alternatively, or in addition, the user may indicate to video conference client160during a configuration or setup step that microphone150and/or speaker(s)140are being implemented using a headphone set which may generally prevent audio feedback. Alternatively, or in addition, echo cancellation and/or baseline ambient noise levels may be employed as discussed hereinbelow with regard to steps378and380may also be used to indicate whether or not video conference participant device100as configured provides audio feedback. Similarly, ambient noise in the form of audio signals received by microphone150when its associated user is not speaking may be used to determine whether or not the user is using a headset microphone which may presumptively indicate that speaker(s)140are also implemented in the same headset, such that audio feedback cannot be provided.

If there is audio feedback via microphone150and speaker(s)140, presenter verification module165may use any known echo cancellation method to detect whether or not the audio data received by video conference participant device100was echoed via speaker(s)140and cancelled out from audio received by microphone150(step178). Presenter verification module165may set (step384) the result to “FAILED” and return (step386) the result for inclusion in the results returned to the requester in step399(FIG. 3) if the audio stream does not possess speech-like properties.

Alternatively, or in addition, presenter verification module165may determine whether a return echo as captured by microphone150exceeds an ambient baseline level (step380). Presenter verification module165may establish an ambient noise baseline for video conference participant device100either as part of a configuration process for video conference client160or at the beginning of the video conference. If the audio input from microphone150does not exceed the baseline level by a configurable decibel level, presenter verification module165may set (step384) the result to “FAILED” and return (step386) the result for inclusion in the results returned to the requester in step399(FIG. 3).

If audio feedback is not detected (step376), the presenter verification module165may set (step377) a “no audio feedback” flag and set (step382) the result to “PASSED” and return (step386) the result for inclusion in the results returned to the requester in step399(FIG. 3); thereby indicating that not all of the verification tests were performed due to the configuration of video conference participant device100.

It will be appreciated that the specific order of execution for steps372,374,376,378and380may be exemplary. In accordance with embodiments described herein, presenter verification module165may perform these steps in any suitable order. Furthermore, in accordance with accordance with embodiments described herein, presenter verification module165may be configured to perform only a subset of the steps, e.g., steps372-378, but not380; steps374-380, but not372; only372and374, but not376-380; etc.

Still further, in accordance with accordance with embodiments described herein, presenter verification module165may be configured to perform the verification tests of each of steps372,374,376,378and/or380regardless of their outcome; failure of one verification test may not necessarily preclude the performance of subsequent verification tests. In such embodiments, the result returned in step386would be an aggregated result including details for each of the individual verification tests.

Video conference client160may be implemented with a user interface (UI) control such as, for example, a button or menu option, that may be used by the presenter to proactively request that process300be performed on the other video conference participant devices100. In accordance with embodiments described herein, the host and/or other designated participants may also use this UI control to proactively request that process300be performed on the other video conference participant devices100.

In accordance with embodiments described herein, video conference client160may be configured to provide the results of some or all of the diagnostic tests performed by process300in the form of audio and/or displayed messages in order to enable the user of device100to attempt to fix problems identified by the tests.

It is appreciated that software components of the present invention may, if desired, be implemented in ROM (read only memory) form. The software components may, generally, be implemented in hardware, if desired, using conventional techniques. It is further appreciated that the software components may be instantiated, for example: as a computer program product or on a tangible medium. In some cases, it may be possible to instantiate the software components as a signal interpretable by an appropriate computer, although such an instantiation may be excluded in certain embodiments of the present invention.

It will be appreciated by persons skilled in the art that the present invention is not limited by what has been particularly shown and described hereinabove. Rather the scope of the invention is defined by the appended claims and equivalents thereof.