Aggregation of video receiving capabilities

Video receiving capabilities of participants and source capabilities are compared and conference capabilities for providing different resolutions, frame rates, bit rate, and number of streams are determined by maintaining a conference receiving capability list updated as number and capability of participants' changes. Preferred receiving capabilities of participants are also taken into account in determining conference characteristics based on comparison with allowed capabilities.

BACKGROUND

Videoconferencing uses telecommunications of audio and video to bring people at different sites together for a meeting. This can be as simple as a conversation between two people in private offices (point-to-point) or involve several sites (multipoint) with more than one person in a number of rooms at different sites. Besides the audio and visual transmission of people, videoconferencing can be used to share documents, computer-displayed information, and whiteboards.

Videoconferencing among multiple remote points is sometimes facilitated employing Multipoint Control Unit (MCU). An MCU is a bridge that interconnects calls from several sources. All parties call the MCU, or the MCU may call the parties which are going to participate, for initiating the conference. MCUs may use various protocols such as Internet Protocol (IP), and be structured as software program(s), hardware, or combination of the two. One of the main tasks for an MCU is to organize the conference based on capabilities of the participating parties (e.g. receiving parties and source in a single source directed conference).

MCU controlled video conferences may be facilitated supporting a fixed resolution video stream or accommodating multiple video streams with different resolutions. The conference may be initiated directly by the MCU or escalate from a peer-to-peer chat, where each participant may be provided the ability to request and deliver multiple video sources. When multiple participants with different capabilities request varying resolutions, it is a challenge for the MCU to coordinate those and accommodate the video conference in an optimized fashion, especially if the number of participants is large.

SUMMARY

Embodiments are directed to accommodating a video conference for participants with varying capabilities with flexibility to meet different aggregation criteria, speed to handle large number of requests without degrading user experience, and ability to provide participants with multiple view sources.

DETAILED DESCRIPTION

While embodiments are described for video conference systems, they are not limited to strictly video conferencing. Network-based conferences combining various forms of communication such as audio, video, instant messaging, application sharing, and data sharing may be facilitated using the principles described herein.

Referring toFIG. 1, diagram100of an example video conferencing system is illustrated. At the core of a video conferencing system is a network (e.g. network110) enabling a number of participants (102,104) with audio/video transmission and reception capability to communicate with each other as a group. Participant machines102,104may be any computing device with audio/video capability such as desktop or laptop computers with a camera and microphone (as well as a speaker), specialized video conferencing equipment, or even mobile devices with audio/video capabilities.

Network110, as discussed in more detail below, may be any communication network or combination of networks. The video conference may be facilitated by a single device/program or by a combination of devices and programs. For example, audio/video server118, firewall server112, or mediation servers114may be involved with different aspects of the conference such as storage and processing of audio/video files, security, or interconnection of various networks for seamless communication. Any of these example tasks and others may be performed by software programs, hardware devices, and/or combination of the two.

According to one embodiment, MCU116may be the main facilitator of the video conference in coordination with one or more of the other devices and/or programs mentioned. MCU116may use various protocols such as Internet Protocol (IP), and be structured as software program(s), hardware, or combination of the two. MCU116may be a stand-alone hardware device, or it may be embedded into dedicated conferencing devices (e.g. audio/video server118or mediation servers114). Furthermore, MCU116may be structured as a “decentralized multipoint”, where each station in a multipoint call exchanges video and audio directly with the other stations with no central manager or other bottleneck.

As mentioned previously, an MCU controlled video conference may support receiving one video stream with fix resolution or receiving multiple video streams with different resolutions. MCU116may support, in addition to regular video conferences, multi-party conferences that escalate from a peer-to-peer chat through a mesh network.

To provide each participant with the ability to request multiple video sources and deliver the right streams, MCU116has to consider various factors including: receiver's capabilities (e.g. PC or mobile device's processing power, downlink bandwidth to the client during the meeting), sender's capabilities (e.g. PC or mobile device's processing power, uplink bandwidth from the client during the meeting), viewer's preferences (e.g. number of sources to view, display size of each source), and infrastructure administration (e.g. the need to limit the bandwidth consumed by video conferences).

Video capabilities may be defined as resolution, frame rate, bit rate, number of streams, and the like. One example scenario is when multiple people request the same source to send different video resolutions. This becomes challenging especially when the number of requesters is large (e.g. in hundreds), since the requests have to be aggregated into a single request to the sender.

An MCU according to one embodiment may execute an algorithm flexible to meet different aggregation criteria such as guaranteeing high resolution vs. guaranteeing as many sources as possible. In order to handle a high number of potential requests, the algorithm may be a fast algorithm and allow one participant to request multiple view sources in addition to allowing receiving capabilities to change anytime during the course of a meeting.

FIG. 2is a conceptual diagram illustrating an MCU coordinating video conference with a source and multiple receiving participants accommodating various resolutions.

In a single stream conference, at any given time one participant acts as sender (e.g. participant206) transmitting a single stream of video, which is distributed to a plurality of receivers (208) through MCU216in the video conference220. If the sender (e.g. participant206) can only send one video stream, the aggregated receiving capability does not need to have a capability group with more than one capability (e.g. resolution), but it may have multiple groups. For example, the aggregated capability may not have one group like: 1 VGA+1 CIF+1 QCIF, but it may have three groups: 1 VGA or 1 CIF or 1 QCIF. Since sender can only send one video stream, the number of streams for each group needs to be “1”.

According to one embodiment, the aggregation happens every time a new client (participant) joins or leaves the conference. To control aggregation complexity, MCU216compares the new client's capability or leaving client's capability with current MCU aggregated capability without having to browse through all the existing clients' capabilities.

The capabilities may be maintained in a table illustrated by reference numerals232and234, where column232lists the available capabilities, and column234lists the number of those based on the currently connected participants. For example, in the first instance of the table column234show that only one VGA capability and one CIF capability are listed for the first participant in the conference. After 1000 participants have joined, the table is updated to the second instance, where column234lists 990 HD capabilities, 995 VGA capabilities, and 980 CIF capabilities.

FIG. 3illustrates an example scenario of exchanging various resolution capabilities between video conference participants. An MCU according to an example embodiment may start with an empty conference receiving capability table344-1: HD=VGA=CIF=QCIF=0. When client1joins with HD and VGA and CIF and QCIF capability, as illustrated by table342, the capability information is transmitted to the MCU (343). Even though client1is capable of supporting multiple combinations of resolutions (such as one of each resolution, VGA+CIF+QCIF, CIF only, or QCIF only), the MCU is concerned about which capabilities are represented for the currently joined client only. Upon receiving the client information, the MCU updates its capability table344-2: HD=1, VGA=1, CIF=1, QCIF=1. The MCU also compares each resolution with number of clients. If equal, that means all the clients can support this resolution and it should be in the conference capabilities (in this case, one client).

When client2joins with CIF only and QCIF only capability as illustrated in table346, those capabilities are also transmitted to the MCU (345). The MCU may then update it conference receiving capability table344-3as HD=1, VGA=1, CIF2, QCIF=2. Even though client2does not support HD and VGA, the table maintains the actual number of clients that support available resolutions. When the MCU sends out aggregated conference receiving capability, HD and VGA do not appear, because client2does not support those resolutions. Thus, the aggregated result sent to the clients is different from table344-3maintained for tracking conference receiving capabilities of the clients.

For each additional client joining the conference, the same iterative update—comparison—update process may be performed enabling the MCU to maintain an up-to-date list of capabilities common to all participants and their number. The same process is repeated when a client leaves the conference. It should be noted, that even if same resolution appears more than once in receiving capability, it is counted once.

An MCU according to embodiments may execute two algorithms: the first one for aggregation of the receiving capability and the second for preferred receiving capability of each stream that a client requests. The receiving capability defines the overall receiving capability of a client. Both capabilities need to be aggregated. Of course, both tasks may also be performed by a single algorithm.

FIG. 4illustrates determination of allowed capabilities based on preferences in an example system according to embodiments. In a system according to some embodiments, each client may specify a preferred capability (PC) on each stream it subscribes to. Before processing the preferred capability, the MCU may first decide which receiving capability group from the same client should be used to work with. For example, a participant may have group1: 1 HD+1 VGA+1 CIF and group2: 1 CIF+3 QCIF. If the desire is to provide highest resolution, then group1may be selected. If the desire is to provide as many channels as possible, then group2may be selected.

Once the group is chosen, MCU may compare the selected group with conference receiving capability (as discussed above inFIG. 3) and generate an allowed capability (AC), which contains the lower resolution of the two sets. For example, the selected group may be 1 CIF+3 QCIF, and conference capability may be VGA or CIF or QCIF. Then, the allowed capability is 1 CIF+3 QCIF. This is to guarantee that no capability exceeds the conference receiving capability of the system. The MCU may then decide an allowed preference (AP) of each stream based on the lower resolution between the AC and the PC. As available streams are used up, no availability may be assigned to some of the requested streams.

Diagram400illustrates an example determination scenario for a video conference with four possible source streams (main video, spotlight1, spotlight2, and spotlight3). Client1is capable of receiving 1 HD or 1 HD+1 VGA, or 1 VGA+1 CIF+2 QCIF as illustrated in client1receiving capability table462. The MCU has conference receiving capability as 1 CIF+1 QCIF as indicated in table464. Thus, the client's maximum number of supported channels is 1 VGA+1 CIF+2 QCIF (472). Comparing this with the conference receiving capability, the MCU decides in step two (474), the allowed capability is 1 CIF+2 QCIF.

Client1may also indicate that its preferred capability468is VGA main video, CIF spotlight1, CIF spotlight2, and CIF spotlight3(1 VGA+3 CIF). Comparing that to the allowed capability (AC), the MCU decides in step3(476) the allowed preference (AP)466, which is CIF main video, QCIF spotlight1, QCIF spotlight2, and zero frame spotlight3(1 CIF+1 QCIF).

As the MCU determines the allowed preference, it does so by selecting for each stream the lower of the allowed capability and the preferred capability and then subtracting that allowed preference from the total allowed capabilities. Once all the allowed capabilities are exhausted, the rest of the allowed preferences (for remaining streams) are set to zero frame without specifying resolution. The reason for setting to zero frame is to allow the MCU to drop all the frames on the remaining stream(s).

In a multi-source conference, the MCU aggregates the allowed preference (AP) for each source whenever there is a change in preferred capability to avoid browsing through all the clients. The MCU starts with a conference allowed preference table assumed for main video stream. Before the meeting presenter starts to select other sources (spotlights), the clients (participants) do not know what to subscribe to, so the MCU assumes only the main video stream.

If a first client joins and sends VGA as preferred capability for main video, the MCU set main video to VGA. If another client joins with CIF as preferred capability, the MCU switches main video to CIF. At this point, the MCU chooses the lowest resolution to request to main video source.

Once the meeting presenter chooses the spotlight sources (e.g. through a conference document), each client is able to associate a spotlight source with a preferred capability. For example, when a client subscribes to stream spotlight source1with a preferred capability CIF, the MCU performs the above discussed analysis and determines allowed preference for this stream (e.g. QCIF). Then, the MCU creates a global conference allowed preference table for spotlight source1, containing HD=0, VGA=0, CIF=0, QCIF=1.

An example conference allowed preference table constructed by the MCU for each source may look like this:

SourceSpotlightSpotlightSpotlightSpotlightResolutionsource 1source 2source 3source 4Primary DSHD00001VGA00008CIF105892QCIF16300
The MCU selects the lowest resolution of each column. When a participant leaves the meeting, or un-subscribes from a particular video stream, its local preferred capability (PC) is subtracted from conference allowed preference table.

One special receiver to be considered is the Primary Dominant Speaker (also referred to as Primary Video Source “PVS”), who may request to Secondary (previous) Dominant Speaker (also referred to as Secondary Video Source “SVS”) as its main video source. Therefore, this one does not need to be aggregated with others. The MCU may just send the Primary Dominant Speaker's local main video preferred capability (PC) to the Secondary Dominant Speaker. The Primary Dominant Speaker's local main video preferred capability (PC) may be subtracted from main video conference allowed preference table, since that participant does not need to receive its own video.

The above described algorithms, capabilities, and parameters are for example purposes and do not constitute a limitation on embodiments. Aggregation of video receiving capabilities in a conference system may be performed and capabilities computed through additional or fewer steps, capabilities, and components using the principles described herein.

FIG. 5is an example networked environment, where embodiments may be implemented. Aggregating video receiving capabilities as described previously may be implemented locally or in a distributed manner over a number of physical and virtual clients and servers. Such a system may typically involve one or more networks such as communication network (s)580. The conference may also be implemented in un-clustered systems or clustered systems employing a number of nodes communicating over one or more networks.

A system according to embodiments may comprise any topology of servers, clients, Internet service providers, and communication media. Also, the system may have a static or dynamic topology. The term “client” may refer to a client application or a client device associated with a participant of the video conference. A system according to embodiments may involve many more components, typical and relevant ones are discussed in conjunction with this figure.

Video conference with aggregated receiving capabilities may be facilitated by MCU584alone or in conjunction with server586. Server586may provide complementary services such as storing and processing audio/video data. Data associated with the video conference (e.g. displayed documents, participant addresses, etc.) may be stored in one or more data stores such as data stores589, which may be directly accessed by the servers and/or clients of the system or managed through a database server588. Communication network(s)580provides the backbone of the video conference system and may employ a number of protocols such as SIP, RTP, and the like. Client devices (e.g581-583) provide platforms for participants to transmit and receive audio/video and other signals. Users may access the conference system using a client device or one or more client applications running on a client device.

Communication network(s)580provides communication between the nodes described herein. By way of example, and not limitation, communication network(s)580may include wired media such as a wired network or direct-wired connection, and wireless media such as acoustic, RF, infrared and other wireless media.

Many other configurations of computing devices, applications, data sources, data distribution systems may be employed to implement aggregation of video receiving capabilities. Furthermore, the networked environments discussed inFIG. 5are for illustration purposes only. Embodiments are not limited to the example applications, modules, or processes.

FIG. 6and the associated discussion are intended to provide a brief, general description of a suitable computing environment in which embodiments may be implemented. With reference toFIG. 6, a block diagram of an example computing operating environment is illustrated, such as computing device600. In a basic configuration, the computing device600may be a server executing a programs associated with the functionality of an MCU for facilitating a video conference. Computing device600may typically include at least one processing unit602and system memory604. Computing device600may also include a plurality of processing units that cooperate in executing programs. Depending on the exact configuration and type of computing device, the system memory604may be volatile (such as RAM), non-volatile (such as ROM, flash memory, etc.) or some combination of the two. System memory604typically includes an operating system605suitable for controlling the operation of the computing device, such as the WINDOWS® operating systems from MICROSOFT CORPORATION of Redmond, Wash. The system memory604may also include one or more software applications such as program modules606and video conferencing application622.

Video conferencing application622may be a separate application or an integral module of a hosted service application that provides advanced communication services through computing device600, as described previously. This basic configuration is illustrated inFIG. 6by those components within dashed line608.

The computing device600may have additional features or functionality. For example, the computing device600may also include additional data storage devices (removable and/or non-removable) such as, for example, magnetic disks, optical disks, or tape. Such additional storage is illustrated inFIG. 6by removable storage609and non-removable storage610. Computer storage media may include volatile and nonvolatile, removable and non-removable media implemented in any method or technology for storage of information, such as computer readable instructions, data structures, program modules, or other data. System memory604, removable storage609and non-removable storage610are all examples of computer storage media. Computer storage media includes, but is not limited to, RAM, ROM, EEPROM, flash memory or other memory technology, CD-ROM, digital versatile disks (DVD) or other optical storage, magnetic cassettes, magnetic tape, magnetic disk storage or other magnetic storage devices, or any other medium which can be used to store the desired information and which can be accessed by computing device600. Any such computer storage media may be part of device600. Computing device600may also have input device(s)612such as keyboard, mouse, pen, voice input device, touch input device, etc. Output device(s)614such as a display, speakers, printer, etc. may also be included. These devices are well known in the art and need not be discussed at length here.

FIG. 7illustrates a logic flow diagram for process700of aggregating video receiving capabilities according to embodiments. Process700may be implemented in an MCU device facilitating video conferencing.

Process700begins with operation702, where a participant's receiving capabilities are received by the MCU. At the beginning of a conference, the MCU may set default capabilities for the conference based on the source's (sender's) capabilities or assumed receiver capabilities. Processing moves from operation702to operation704.

At operation704, the conference receiving capabilities are incremented for each resolution as illustrated in the example ofFIG. 3. Processing advances from operation704to decision operation706, where a determination is made whether a count of conference receiving capabilities is equal to a number of participants. If the numbers are not the same, previous conference receiving capabilities are maintained at subsequent operation712and processing returns to operation702for receiving another participant's capabilities.

If the determination at decision operation706is affirmative, conference receiving capabilities are set to the aggregated participant receiving capabilities at operation708and the new capability information is transmitted to the current source for the conference at the subsequent operation710. After operation710, processing may return to operation702for receiving another participant's capabilities.

FIG. 8illustrates a logic flow diagram for process800of aggregating video receiving capabilities for multiple streams according to embodiments. Process800may also be implemented in an MCU device facilitating video conferencing.

Process800begins with operation802, where a participant's preferred capabilities are received by the MCU as sets of capabilities for different streams. A goal of the MCU may be to supply as many send streams as possible. Processing moves from operation802to decision operation804, where a determination is made whether a set of conference receiving capabilities larger than a number of available streams exists. If conference receiving capability sets larger than the number of streams exist, the allowed receiving capabilities is set to the receiving participant's capabilities at subsequent operation806. Otherwise, the set of capabilities with largest number of streams is selected at operation808.

Following operations806and808, receiving capabilities are set for each stream from highest resolution to lowest preferred resolution on each stream at operation810. In setting the receiving capabilities, if the allowed receiving capabilities have the participant's preferred capabilities, the capabilities for the particular stream may be decremented by one (indicating one participant is subscribed to the particular stream). If the participant's preferred capabilities are not supported for the particular stream, the capability may be set to next lower capability (e.g. resolution) and the capabilities for the particular stream decremented by one. Finally, if there are no more capabilities available, the particular stream may be marked as zero frame rate and not used for the conference. Processing moves from operation810to decision operation812.

At decision operation812, a determination is made whether the send stream is sourced by a manual stream (e.g. main video or spotlight). If the send stream is manually sourced, the preferred capabilities are incremented for the manual source at the subsequent operation814. If the send stream is sourced by a dominant speaker selection, dominant speaker's preferred capabilities are increments at operation816.

According to some embodiments, each send stream may have a preferred capability to request from the sender. If the stream is manually sourced, the preferred capability may be set to lowest resolution as discussed above for manual stream. If the stream is a Primary Video Source (PVS) stream, the preferred resolution may be set as discussed above and the preferred capability removed from the main video request. If the stream is a Secondary Video Source (SVS) stream, the preferred capability may be set to the PVS's main video preferred capability.

The allowed preferred capabilities for each stream are then send to the current source for the video conference at operation818. Processing may return to operation802for receiving another participant's capabilities.

The operations included in processes700and800are for illustration purposes. Aggregation of video receiving capabilities may be implemented by similar processes with fewer or additional steps, as well as in different order of operations using the principles described herein.