Encoding processing for conferencing systems

Optimization of conference call encoding processes is provided. A first client of a multi-party conference call may receive client capability data, including video scalability support, from each of the other clients to the conference call. Based on the client capability data and the transmission capabilities of the first client, including video scalability support, the first client may determine a total number of data streams and properties for each data stream, such that the total number of data streams and the plurality of properties for each data stream are optimized and supported by the respective client capability data and the transmission capabilities. Subsequently, the first client generates one or more data streams according to the total number of data streams and the properties that were determined for each data stream and transmits the one or more data streams to the other clients of the conference call.

TECHNICAL FIELD

The technical field relates generally to telecommunications and, more specifically, to conferencing systems operating over telecommunications networks.

BACKGROUND

Multi-party audio and/or video conference calls can involve participants or clients with a wide variety of preferences and capabilities. A client using a mobile phone to connect to the conference, for example, may have a low uplink and downlink bandwidth and support only low frame rate video. On the other hand, a client connecting to the conference using a desktop computer on a corporate intranet may have a high uplink and downlink bandwidth and support a high frame rate. A mobile phone client may, for example, only be able to encode and receive video at Common Intermediate Format (CIF) resolution (e.g., 320 by 240 pixels per frame) with a frame rate of 15 frames per second (fps), while the intranet client may be able to encode and play back video at Video Graphics Array (VGA) resolution (e.g., 640 by 480 pixels) with a frame rate of 30 fps. Consequently, the mobile phone client may not be able to send or receive the same quality video stream as the intranet client.

The conventional solution to the aforementioned problem involves degrading the video quality for all participating clients to a maximum level that the lowest performing client can handle. That is, the conferencing system may force a higher-capability client to compromise and sacrifice its conferencing capabilities by encoding/receiving video streams with a lower resolution and a lower frame rate as compared to a resolution and frame rate the higher-capability client can handle. Although this approach provides a system solution that supports lower-capability clients, the higher-capability clients are left with a sub-par conferencing experience that is below their abilities. Further, this approach is not optimally efficient since it leaves a certain amount of processing power and bandwidth unused.

SUMMARY

Optimization of conference call encoding processes is provided. This Summary is provided to introduce a selection of disclosed concepts in a simplified form that are further described below in the Detailed Description including the drawings provided. This Summary is not intended to identify key features or essential features of the claimed subject matter. Nor is this Summary intended to be used to limit the claimed subject matter's scope.

In one embodiment, a method of providing optimization of conference call encoding processes is provided that solves the above-described problem of serving audio and/or video to conference call clients with varying capabilities. A first client of a multi-party conference call may receive client capability data, including video scalability support, from each of the other clients to the conference call. Based on the client capability data and the transmission capabilities of the first client, including video scalability support, the first client may determine a total number of data streams and properties for each data stream, such that the total number of data streams and the plurality of properties for each data stream are optimized and supported by the respective client capability data and the transmission capabilities. Subsequently, the first client generates one or more data streams according to the total number of data streams and the properties that were determined for each data stream and transmits the one or more data streams to the other clients of the conference call.

DETAILED DESCRIPTION

Disclosed methods provide optimization of conference call encoding processes. The term conference call, as used herein, shall denote any type of telecommunication connection between parties wherein audio and/or video data is exchanged between parties of the conference call. One or more data streams, including audio and/or video data, are transmitted to receiving clients based on the sending client's ability to encode and transmit the data and the receiving clients' abilities to receive and process the data, while optimizing use of the available bandwidth and processing resources.

FIG. 1shows a block diagram of an operating environment100that supports conference calls, according to an example embodiment. The conference call environment100may comprise multiple conference call participants or clients120,130,140,150and a conference server110communicating via a communications network160. Each of the clients120,130,140,150and conference server110may be connected either wirelessly or in a wired or fiber optic form to the communications network160. Clients120,130,140,150and conference server110may each comprise a computing device600, described below in greater detail with respect toFIG. 6.FIG. 1shows that clients120,130,140,150may comprise a desktop computer, laptop, tablet computer or smart phone, for example. Communications network160may be a packet switched network, such as the Internet, or any local area network, wide area network, enterprise private network or the like.

Conference call environment100may be used when multiple clients want to share data streams amongst each other. Each client may connect directly to conference server110. The conference server110may receive and distribute a list of which clients are connected and what are each client's preferences and capabilities, which can include descriptions of uplink and download bandwidth, resolution preferences and restrictions, frame rate preferences and restrictions, video scalability supported and loss rate. Hereinafter, the term capability shall be used to refer to both preferences and capabilities, or constraints, associated with the computing device of a client.

Subsequently, conference server110may, from time to time (e.g., periodically), update to reflect any changes in the clients' capabilities. For example, conference server110may analyze the network connection of client150and determine that client150may have more bandwidth available than previously determined Conference server110may distribute that information for use by other clients in facilitating data sharing. Conference server110may further be responsible for receiving data streams from the clients and for processing and re-sending those data streams to the other clients. As known in the art and as used herein, a data stream may refer to a sequence of digitally encoded coherent signals (data packets) used to transmit or receive audio and/or video data.

Each client may encode, using an encoder, at least one data stream to send to conference server110. For example, in the case of a video conference call, each client may have an audio/video input such as a webcam and/or microphone connected to the client. The input may be used to generate one or more data streams (including audio and video data) and encode the data stream before sending the data stream to conference server110. The process of determining how many data streams an encoder will produce and the properties of each data stream is described in greater detail below with reference toFIG. 5. Conference server110may receive encoded data streams from each client, process the encoded data streams and decide which encoded data streams to relay to each client. For example, client150may encode a data stream as a series of video frames and send the encoded data stream to conference server110. The conference server110may receive the encoded data stream, process the encoded data stream and send the data stream to clients120,130and140. As a final point, each client may have the ability to decode the data stream(s) received from the conference server110and play or display the audio and/or video information embedded in the data stream(s).

FIG. 2illustrates a diagram showing the data flow of an example capability data collection process200, according to an example embodiment. As explained above, conference server110may receive and distribute data pertaining to each client's capabilities—hereinafter referred to as capability data. Further, on an ongoing basis, conference server110may update changes in the clients' capabilities in facilitating the data sharing. Although the system supports the distribution of client capability data among all of the clients in a conference call, for example purposes only,FIG. 2depicts the transfer of client capability data from clients130,140and150(designated as receivers of a data stream) to client120(designated as a sender of data stream). Note that althoughFIG. 2, by way of example, designates client120as a sender of a data stream and clients130,140and150as receivers of a data stream, the system supports the sending and receiving of data streams from any client to any other client.

FIG. 2shows that prior to, or during, a conference call session, one or more receiver clients130,140and150each send a request (requests202,204and206, respectively), or source request, to conference server110via the network160. A source request may include capability data for a client, including data pertaining to reception, processing and display of data, such as: the downlink or download bandwidth (usually quantified in bits per second) of the client, the frame rate (usually quantified in frames per second) supported by the client, the resolution (usually quantified in units of pixels by pixels) supported by the client, whether video scalability is supported by the client, and the loss rate of the client. A client may transmit a source request to the conference server110, for example, in the context of a Real-Time Transport Control Protocol (RTCP) formatted message before or during a conference call session. RTCP is a sister protocol for the Real-Time Transport Protocol (RTP) and provides out-of-band statistics and control information for an RTP data flow.

Video scalability may refer to the encoding of a video data stream that also contains one or more subset video data streams. A subset video data stream is derived by dropping packets from the larger video data stream to reduce the bandwidth required to transmit the subset video data stream. The subset video data stream can represent a lower spatial resolution (smaller screen), lower temporal resolution (lower frame rate), or lower quality video signal. Scalable Video Coding (SVC) is one example of a video scalability standard used in conjunction with the H.264/MPEG-4 video compression standard. When a video data stream has been encoded using video scalability processes, the video data stream can be said to include video scalability encoding. Similarly, when a sending client has the capability to generate a video data stream including video scalability encoding, or when a receiving client has the capability to receive and decode a video data stream having video scalability encoding, said client (sender or receiver) can be said to include video scalability support.

The loss rate of a client refers to the average number of bits that are lost during transfer of data to the client. Many communication channels are subject to channel noise, and thus errors may be introduced during transmission from the source to the receiver client. The loss rate of a client is used to calibrate forward error correction routines, as described more fully below.

FIG. 2shows the conference server110receiving the source requests202,204and206from receiver clients130,140and150, respectively. Subsequently, the conference server110produces the metadata208(i.e., source request metadata208ofFIG. 2) based on the source requests and transmits the metadata208to the sender client120. As used herein, the term metadata refers to data providing information about one or more aspects of other data, namely, the data included in the source requests202,204and206.

The metadata208may include at least the data that is included in the source requests202,204and206. In one embodiment, the metadata208further includes a summary of the data included in the source requests202,204and206organized by capabilities, such that receiver clients with identical or similar capabilities are grouped together. In another embodiment, the metadata208further includes data that can be formatted into at least one histogram or table that shows the distribution of receiver clients according to certain capability values, such as the download bandwidth in bit rate units. In yet another embodiment, the metadata208further includes data that can be formatted into at least one bitmap or lookup table for each client describing the range of values supported by that client for a particular capability, such as frame rate or resolution.

FIG. 3depicts a diagram showing the data flow of an example data stream transmission process300, according to an example embodiment. LikeFIG. 2,FIG. 3depicts, for example purposes only, data stream transmission from client120(designated as a sender of a data stream) to clients130,140and150(designated as receivers of the data stream). As explained above, a sender client120may encode at least one data stream to send to conference server110, which processes the data stream(s) and sends the data stream(s) to the receiver clients130,140and150. In the case of a video conference, a data stream may be encoded as a series of video frames wherein each frame may comprise a single image. Each frame may be represented by bits of data. The data stream may be encoded so that not every bit of data in every frame must be transmitted in order to represent the source of the data stream. Different frame types may be used to encode and/or compress the data stream. Frame types may comprise, for example, I-frames, P-frames, and B-frames.

A client, such as sender client120, can perform the encoding process using an encoder, which may include a hardware device, circuit, transducer, software program, or algorithm that captures, compresses and converts audio and/or video information from one format or code to another, for the purposes of standardizing the transmission of the data. Note that a client may include more than one encoder, which is shown as an encoder program module607in device600ofFIG. 6, according to one embodiment. Each encoder may produce one or more data streams. The process of determining how many data streams an encoder will produce and the properties of each data stream is described in greater detail below with reference toFIG. 5.

By way of example,FIG. 3shows that sender client120uses an encoder to produce two data streams302,304, which are transmitted to conference server110. The encoder of sender client120may employ the use of a video scalability standard, such as SVC, to reduce the number of data streams that are generated and transmitted. The conference server110receives the two data streams302,304and may process the data streams according to the capabilities of the receiver clients130,140and150, as described more fully below. As a result of the processing performed by the conference server110, the data stream302is split into two data streams306,308, while data stream304is passed through in an unprocessed manner as data stream310.

Consistent with embodiments described herein, conference server110may employ video scalability standard processes to alter or divide a data stream before sending it to one or more receiver clients. For example, receiver client140may support video at 30 frames per second, while receiver client130may have the same capabilities as receiver client140, except that it only supports video at 15 frames per second. In this case, sender client120may transmit to conference server110a data stream302that meets the higher capabilities of client140. Consequently, conference server110may relay the data stream302unaltered to receiver client140as data stream308. The conference server110may also copy the data stream302and, using video scalability processing, produce a processed data stream306, wherein every other frame is dropped. Processed data stream306, therefore, possesses the same properties as data stream308, except that processed data stream306includes video at a frame rate of 15 frames per second, which meets the capabilities of receiver client130. Consequently, with the aid of the conference server110, the sender client120may use a single data stream302to service two receiver clients (130,140) having different capabilities.

In addition to frame rate adjustments of a data stream as described above, video scalability encoding standard processes also support adjustments to picture size and quality. With regard to adjustments to picture size, the initial data stream received from sender client120may be encoded at multiple resolutions. Consequently, the conference server110may divide the initial data stream into two or more data streams having different resolutions. With regard to adjustments to picture quality, the initial data stream received from sender client120may be encoded at a single resolution but at different picture qualities. Accordingly, the conference server110may divide the initial data stream into two or more data streams having different picture qualities.

Subsequent to receiving and processing the data streams received from the sender client120, the conference server110sends data streams306,308,310to the receiver clients130,140and150, respectively. Finally, each receiver client130,140,150receives a data stream, which is decoded and displayed at the respective receiver client. A client, such as receiver client130, performs the decoding process using a decoder, which may include a hardware device, circuit, transducer, software program, or algorithm that receives encoded audio and/or video data and converts it to a format wherein the data that can be played or displayed for a user. A client's video decoder is shown as a decoder program module608in device600ofFIG. 6, according to one embodiment.

FIG. 4shows a flow chart setting forth the general stages involved in an example method400for providing conferencing services, including conference encoding processing, according to an example embodiment. Method400may be implemented by the various entities described inFIG. 1, such as conference server110and clients120,130,140and150. An example implementation of the stages of method400will be described in greater detail below. LikeFIG. 3,FIG. 4depicts, for example purposes only, data stream transmission from one sender client, such as client120, to multiple receiver clients, such as receiver clients130,140and150.

Method400may begin at starting block410and proceed to stage420wherein a conference call commences. A conference call may commence when at least one client, such as one or more of the clients120,130,140and150, register with the conference server110. For example, the client(s) may be operatively connected to conference server110via a network connection and may request participation in a conference call hosted by conference server110.

From stage420, method400may advance to stage425wherein conference server110may collect capability data from each registered client. As explained above, one or more receiver clients, such as clients130,140and150can each send a source request to conference server110, wherein the source request may include capability data for each client, such as: the download bandwidth, the frame rate supported by the client, the resolution supported by the client, the video scalability type supported by the client and the loss rate of the client. The conference server110may receive the source requests, produce metadata208based on the data in the source requests, and send the metadata208to the sender client120. The metadata208may include at least the data that is included in the source requests of the receiver clients.

In stage430, the sender client120may determine the data stream properties for at least one data stream to be generated based on: (i) the capability data of the receiver clients, which were received in the metadata208, and (ii) the capabilities of the sender client120. The process of stage430is described in greater detail below with reference toFIG. 5.

The capabilities of the sender client120, also referred to as transmission capabilities, may include the number of encoders of sender client120, the encoder processing capability or capacity of each encoder, usually quantified as the maximal macroblock processing rate (i.e., width in macroblocks times height in macroblocks times frames per second), and the uplink, or upload, bandwidth of the sender client, usually quantified in bits per second. The capabilities of the sender client120may further include the frame rate of the source video produced at the sender client120and the resolution of the source video. In the case of a video conference call, the properties of a data stream may include a resolution, a frame rate, a required bit rate of the data stream and whether video scalability is employed.

In summary, the data inputs for the determination of stage430can comprise: a) the capability data representing the capabilities of the receiver clients (i.e., download bandwidth, resolution, frame rate, video scalability support, etc.) and b) the data representing the capabilities of the sender client (i.e., the number of encoders, the processing capability of each encoder, uplink bandwidth, video source resolution, video source frame rate, video scalability support, etc.), while the data outputs of the stage430can comprise: a) a whole number representing a total number of data streams to generate and b) a set of properties for each data stream (i.e., required bit rate, resolution, frame rate, video scalability support and an identity of which encoder will produce each data stream).

From stage430, method400may advance to stage435where the one or more encoders of sender client120may generate one or more data streams based on the determinations of stage430, which determined the properties for each generated data stream. From stage435, method400may advance to stage440wherein sender client120may transmit the one or more data streams generated in stage435to the conference server110.

From stage440, method400may advance to stage445wherein the conference server110may process the one or more data streams received from sender client120according to the capabilities of the receiver clients130,140and150. As explained above with reference toFIG. 3, conference server110may alter or divide a data stream before sending it to a receiver client. For example, conference server110may receive a high frame rate data stream that meets the capabilities of a first receiver client but exceeds the capabilities of a second receiver client. Assuming all other properties of the data stream are supported by the first and second receiver client, using a video scalability standard such as SVC, conference server110may relay the high frame rate data stream unaltered to the first receiver client, but conference server110may also copy the high frame rate data stream and produce a processed data stream wherein a certain number of frames are dropped, thereby producing a data stream with a lower frame rate and meeting the lower capabilities of the second receiver client.

From stage445, method400may advance to stage450wherein the conference server110may send the data stream(s) it has received and processed in stage445to the receiver clients130,140and150. The receiver clients subsequently receive the data streams, which may be decoded and played or displayed at the respective receiver clients.

From stage450, method400may advance to stage455wherein it may be determined whether the capabilities of the receiver clients have changed, whether the receiver list of clients participating in the conference call have changed and/or whether the capabilities of the sender client have changed. Consistent with further embodiments, conference server110and/or the sender client120may re-evaluate the capabilities of the receiver clients and/or the sender client120, as well as the presence of the receiver clients in the conference call, and dynamically alter the number and type of data streams transmitted from sender client120, since such changes may alter the determination of stage430. For example, conference server110and/or the sender client120may determine that the downlink bandwidth available to receiver client150has dropped from 3 Mbps to 500 Kbps and sender client120may begin sending the CIF resolution data stream instead of the VGA resolution data stream to receiver client150. In another example, second receiver client140may be the only client receiving the HD resolution data stream at a high bit rate from sender client120. If receiver client140drops out of the conference call, sender client120may stop encoding and sending a high bit rate HD resolution data stream, which would free up processing and bandwidth resources that can be dedicated to other receiver clients still present in the conference call.

It should be noted that althoughFIG. 4shows stage460occurring in sequence after stage455, both stages455and460may occur either periodically or in response to triggering events, and both stages455and460may occur in parallel or in another thread from the steps of method400. Stage455, for example, may occur at predefined time intervals or may occur solely in response to detected changes in the capabilities of the clients or in response to the detection of clients joining or leaving the conference call. Stage460, for example, may also occur at predefined time intervals or may occur solely in response to the detection of clients leaving the conference call.

If the determination of stage455is affirmative, then control flows back to stage425wherein client capability data may be collected once more and the subsequent stages of method400are executed. In one embodiment, if the determination of stage455is affirmative, then control flows back to stage425wherein client capability data may be collected solely from those clients whose capabilities have changed since the initial determination of stage430.

If the determination of stage455is negative, then control flows to stage460where it may be determined whether the conference call is concluded. If the determination of stage460is negative, then control flows back to stage435wherein the data stream(s) continue to be generated and the subsequent stages of method400are executed. If the determination of stage460is affirmative, then control flows to stage470wherein the method400is concluded.

FIG. 5shows a flow chart setting forth the general stages involved in an example method500for optimizing encoding processes for conference calls, according to an example embodiment. Note that method500provides greater detail about the process430ofFIG. 4. Method500may be implemented using a computing device600as described in more detail below with respect toFIG. 6. An example implementation of the stages of method500will be described in greater detail below. LikeFIG. 4,FIG. 5depicts, for example purposes only, optimization routines for conference call encoding processes being executed by a sender client sending one or more data streams to multiple receiver clients, such as130,140and150. The term optimization, as used herein, refers to efforts to make the most effective use of all conference call resources—including uplink and downlink bandwidth, encoder and decoder processing resources, etc.—so as to produce to highest possible conference call experience for all receiver clients.

Method500may begin at starting block510and proceed to stage520wherein a first set of one or more candidate data streams is defined based on the receiver client capability data. Stage520entails defining a set of properties for one or more data streams. The properties for a data stream may include, but is not limited to, resolution, frame rate, required bit rate and video scalability support. In stage520, the sender client120may, for example, define one data stream for each receiver client, wherein the data stream allotted for a client is defined such that it can be supported by the client's capabilities. In stage520, the sender client120may further merge defined data streams corresponding to receiver clients with identical capabilities, so as to eliminate redundancy and reduce the number of candidate data streams.

From stage520, method500may advance to stage525wherein the sender client120may adjust and reduce the candidate data streams based on certain receiver client capabilities. Recall that sender client120received metadata208, which may include a table that shows the distribution of receiver clients according to certain capability values, such as the download bandwidth in bit rate units. Also recall the sender client120may receive a bitmap or lookup table for each receiver client describing the range of values supported by that client for a particular capability, such as frame rate or resolution.

In stage525, sender client120may review the lookup table associated with each receiver client and modify one or more properties of the candidate data streams according to the range of values in the lookup table, so as to reduce the number of candidate data streams. For example, the candidate data streams may include a first stream with a 30 fps frame rate that corresponds to receiver client130, and a second stream with a 15 fps frame rate that corresponds to receiver client140. In this example, the data in the lookup table associated with receiver client140may specify that the client140may be served with a 30 fps frame rate data stream that includes video scalability encoding, assuming the conference server110processes the data stream as outlined above with reference toFIG. 3. Assuming all other properties of the first and second candidate data streams are identical, in stage530the sender client120may produce a single data stream at 30 fps, wherein the data stream includes video scalability encoding that allows a conference server110to split the data stream into two data streams—a first data stream at 30 fps intended for client130and a second data stream at 15 fps for client140, as described more fully above with reference toFIG. 3. Consequently, the sender client120may reduce the number of candidate data streams by serving both clients130and140with the same data stream.

Further in this embodiment of stage525, sender client120may review the table that shows the distribution of receiver clients according to certain capability values, and modify the properties of the candidate data streams according to the table, so as to reduce the number of candidate data streams. For example, the table may show more than one receiver client has requested a data streams at a 1 Mbps bit rate. Assuming all other capabilities of the more than one receiver clients are identical, the more than one receiver clients may be served by one candidate data stream at a 1 Mbps bit rate, so as to reduce the number of candidate data streams. As a further example, the table may show a first receiver client has requested a data stream at a 1 Mbps bit rate and a second receiver client has requested a data stream at a 200 Kbps bit rate. Because the two requested bit rates are relatively far apart, the sender client120cannot serve the two clients with the same data stream, because doing so would be too far below or too far above the capabilities of one of the clients and would negatively affect the conference call experience.

As a last example, the table may show a first receiver client has requested a data stream at a 1 Mbps bit rate and a second receiver client has requested a data stream at a 900 Kbps bit rate. Because the two requested bit rates are relatively close in value, assuming all other capabilities of the two receiver clients are identical, the sender client120may serve the two clients one candidate data stream at the lower bit rate value (900 Kbps bit rate), so as to reduce the number of candidate data streams. Because the two requested bit rates are relatively close in value, the conference call experience of the client requesting the 1 Mbps bit rate data stream would not be substantially affected by a 900 Kbps bit rate data stream.

From stage525, method500may advance to stage530wherein the sender client120adjusts the candidate data streams based on certain sender client capabilities and recommended bit rates. Specifically, in stage530sender client120may reduce the frame rate and/or resolution of the candidate data streams to match the frame rate of the source video produced at the sender client120and/or to match the resolution of the source video.

Further in stage530, once adjustments to frame rate and/or resolution have occurred, candidate data streams may be left with bit rates that exceed the bandwidth necessary to transmit the data stream. For example, a candidate data stream that was initially defined at an HD resolution and a 2.5 Mbps bit rate may have been reduced to a VGA resolution video stream earlier in stage530. A VGA resolution data stream, however, doesn't require a 2.5 Mbps bit rate. Consequently, the sender client120may also reduce the bit rate of the candidate data stream to 500 Kbps, which may be adequate to support the VGA resolution data stream.

To aid in the performance of stage530, the sender client120may compare the properties of a candidate data stream, whose frame rate or resolution has been reduced, with a table that defines the recommended bit rates that correspond to each frame rate and/or resolution value. Using this table, the sender client120may identify an adequate reduced bit rate for a candidate data stream whose frame rate or resolution has been reduced. To the extent the reductions defined above result in identical candidate data streams, the sender client120may further merge defined data streams, so as to eliminate redundancy and reduce the number of candidate data streams.

It should be noted that the candidate data streams from which the client120chooses in step530can be the original set of candidate data streams defined in stage520above, the adjusted set of candidate data streams (i.e., with downgraded frame rate, resolution, etc.) defined after stage525above, a further adjusted or downgraded set of candidate data streams based on any of the aforementioned sets, or any combination of the above sets.

From stage530, method500may advance to stage540wherein the sender client120may adjust and reduce the candidate data streams based on sender client capabilities and an encoding algorithm. In stage540, the sender client120determines which of the candidate data streams to generate and send to the receiver clients, while abiding by the sender client capabilities. Consequently, state540generates one or more candidate data streams that may be optimized for one or more of the receiver clients. It should be noted that the candidate data streams from which the client120chooses in step540can be the original set of candidate data streams defined in stage520above, the adjusted set of candidate data streams defined after any of the stages525-530above, a further adjusted or downgraded set of candidate data streams based on any of the aforementioned sets or any combination of the above sets.

The objective of the calculation of stage540is to select a subset of the candidate data streams so as to optimize total profit p, while abiding by the maximum macroblock processing rate W and the maximum uplink (or upload) bandwidth bit rate B constraints. The first constraint W of the calculation is the encoding capability or capacity of the encoder of sender client120, quantified as the maximum macroblock processing rate. The second constraint B is the maximum uplink bandwidth of sender client120.

The encoder of sender client120can generate and send N data streams, with each data stream xi(1≦i≦N) having an associated profit pi, bit rate bi, and encoding cost wi, in units of macroblocks per second. The profit variable p may be a real number that represents the conference call experience of a receiver client. Higher bit rate and higher resolution data streams, for example, have higher profit values, since they result in a higher quality conference call experience for a receiver client.

In an example embodiment, the mathematical form of the calculation of stage540, in the case where the sender client120includes only one encoder, can be written as:

Maximize

subject to

∑i=1N⁢⁢wi⁢xi≤W⁢⁢and⁢⁢∑i=1N⁢⁢bi⁢xi≤Bxi∈{0,1},for⁢⁢all⁢⁢1≤i≤N
wherein profit piis the profit associated with a stream xi; wherein encoding cost wiis the encoding cost (quantified as a macroblock processing rate) associated with a data stream xi; and wherein bit rate biis the bandwidth (in bits per second) associated with a data stream xi. Note that the above formulation dictates that each candidate data stream is either selected or not selected (i.e., xiε{0,1}) by the calculation of stage540, so as to maximize total profit. Recall that the candidate data streams from which selection is made in the calculation of step540can be the original set of candidate data streams defined in stage520above, the adjusted set of candidate data streams defined after any of the stages525-530above, a further adjusted or downgraded set of candidate data streams based on any of the aforementioned sets or any combination of the above sets.

In the case where the sender client120includes multiple encoders K, each of which can generate and send up to k data streams, the mathematical form of the calculation of stage540is described below, in an example embodiment. The calculation in this case involves K+1 steps wherein the steps1through K involves calculating an optimal subset of candidate data streams (as they exist after any of the stages520-530, or any union of the aforementioned sets) that maximizes overall profit of the conference call, with the assumption of unlimited uplink bandwidth B. The last step K+1 involves calculating an optimal subset of candidate data streams, out of the data streams selected from the previous K steps, such that the aggregate bit rate of the selected data streams does not exceed the maximum uplink bandwidth B.

More specifically, assuming that the encoding capability of encoder k is denoted as Wk(quantified as a macroblock processing rate), the mathematical form of the calculation for the k-th stage (k≦K) is:

Maximize

subject to

Assuming that the encoding capability of encoder k is denoted as Wk(quantified as a macroblock processing rate), the mathematical form of the calculation for the last step K+1 is:

Maximize

subject to

Note that the above formulation dictates that each candidate video stream is either selected or not selected (i.e., xi(k+1)ε{0,1}) by the calculation of stage540, so as to maximize total profit. It should also be noted that unlike the calculation for instances where the sender client120includes only one encoder, as defined above, the calculation for instances where the sender client120includes multiple encoders K is fragmented into at least two discrete steps wherein the constraint for encoder processing capability is resolved in the first discrete step and the constraint for uplink bandwidth is resolved in the second discrete step. The separation of the aforementioned calculation into two discrete steps reduces the processing burden in performing the calculation. As such, in one embodiment, in cases where there is only one encoder, the multiple encoder calculation above can be utilized by assuming K=1 (since K represents the number of encoders). This allows the multiple encoder calculation to be utilized for a single encoder situation. Consequently, the use of the multiple encoder calculation imposes a lowered processing burden and may be favored for devices with a lowered processing capability such as a smart phone.

In summary, in stage540the sender client120identifies one or more of the candidate data streams to generate and send to the receiver clients (as well as the identity of the encoder which will be generating each data stream), while abiding by the sender client capabilities (number of video encoders, processing capability of each encoder, uplink bandwidth, video scalability support, etc.) and optimizing the conference call experience (i.e., “profit”) of the receiver clients.

From stage540, method500may advance to stage550wherein the sender client120may adjust the bit rate allocation in each candidate data stream according to the loss rate of the receiver client that receives that data stream. Recall that loss rate data for each receiver client was collected by the sender client120in step425of the method400. The loss rate of a receiver client refers to the average number of bits that are lost during transfer of a data stream to the receiver client. This data is used to calibrate forward error correction in the data stream sent to each receiver client. In stage550, the sender client120determines the portion of each data stream that is allotted to forward error correction bits, according to the receiver client's loss rate, and the portion of the data stream that is allotted for audio and/or video data. At the conclusion of stage550, the candidate data streams, as defined and modified by the method500, are deemed the data streams that shall be generated in step435of the method400. From stage550, method500may advance to stage560wherein the method500concludes.

FIG. 6is a block diagram of a system including an example computing device600and other computing devices. Consistent with the embodiments described herein, the aforementioned actions performed by clients120,130,140and150may be implemented in a computing device, such as the computing device600ofFIG. 6. Any suitable combination of hardware, software, or firmware may be used to implement the computing device600. The aforementioned system, device, and processors are examples and other systems, devices, and processors may comprise the aforementioned computing device. Furthermore, computing device600may comprise an operating environment for methods400and500as described above. Methods400and500may operate in other environments and are not limited to computing device600.

With reference toFIG. 6, a system consistent with an embodiment of the invention may include a plurality of computing devices, such as computing device600. In a basic configuration, computing device600may include at least one processing unit602and a system memory604. Depending on the configuration and type of computing device, system memory604may comprise, but is not limited to, volatile (e.g. random access memory (RAM)), non-volatile (e.g. read-only memory (ROM)), flash memory, or any combination or memory. System memory604may include operating system605, one or more programming modules606. Operating system605, for example, may be suitable for controlling computing device600's operation. In one embodiment, programming modules606may include, for example, an encoder program module607module and a decoding program module608. Furthermore, embodiments of the invention may be practiced in conjunction with a graphics library, other operating systems, or any other application program and is not limited to any particular application or system. This basic configuration is illustrated inFIG. 6by those components within a dashed line620.

Computing device600may have additional features or functionality. For example, computing device600may also include additional data storage devices (removable and/or non-removable) such as, for example, magnetic disks, optical disks, or tape. Such additional storage is illustrated inFIG. 6by a removable storage609and a non-removable storage610. Computer storage media may include volatile and nonvolatile, removable and non-removable media implemented in any method or technology for storage of information, such as computer readable instructions, data structures, program modules, or other data. System memory604, removable storage609, and non-removable storage610are all computer storage media examples (i.e. memory storage.) Computer storage media may include, but is not limited to, RAM, ROM, electrically erasable read-only memory (EEPROM), flash memory or other memory technology, CD-ROM, digital versatile disks (DVD) or other optical storage, magnetic cassettes, magnetic tape, magnetic disk storage or other magnetic storage devices, or any other medium which can be used to store information and which can be accessed by computing device600. Any such computer storage media may be part of device600. Computing device600may also have input device(s)612such as a keyboard, a mouse, a pen, a sound input device, a camera, a touch input device, etc. Output device(s)614such as a display, speakers, a printer, etc. may also be included. The aforementioned devices are only examples, and other devices may be added or substituted.

As stated above, a number of program modules and data files may be stored in system memory604, including operating system605. While executing on processing unit602, programming modules606(e.g. encoder and decoder modules607and608) may perform processes including, for example, one or more of method400's or method500's stages as described above. The aforementioned processes are examples, and processing unit602may perform other processes. Other programming modules that may be used in accordance with embodiments of the present invention may include electronic mail and contacts applications, word processing applications, spreadsheet applications, database applications, slide presentation applications, drawing or computer-aided application programs, etc.

Furthermore, embodiments of the invention may be practiced in an electrical circuit comprising discrete electronic elements, packaged or integrated electronic chips containing logic gates, a circuit utilizing a microprocessor, or on a single chip (such as a System on Chip) containing electronic elements or microprocessors. Embodiments of the invention may also be practiced using other technologies capable of performing logical operations such as, for example, AND, OR, and NOT, including but not limited to mechanical, optical, fluidic, and quantum technologies. In addition, embodiments of the invention may be practiced within a general purpose computer or in any other circuits or systems.