Negotiate multi-stream continuous presence

Described are embodiments for allowing the negotiation of a continuous presence layout. Specifically, in embodiments, an offer is generated by a client that includes attributes for displaying continuous presence video information. The attributes include, in some embodiments, one or more window identifiers, one or more bandwidth limit identifiers, one or more group numbers, and/or one or more ranks. The offer is sent to a server which transmits an answer to the offer. Once the attributes for the continuous presence layout has been negotiated, the server uses the attributes to format video content sent to the client.

BACKGROUND

Videoconferencing is a very powerful communication mode that allows people to be in remote locations and be able to see and speak to each other in real time. Typically, a video conference session is established by a client device (located at an endpoint where a participant will join) establishing a session with a conference server. Establishing a session between the client and the conference server can occur using a number of different protocols including, the Session Description Protocol (SDP), Session InitiationProtocol (SIP), and Real-Time Transport Protocol (RTP).

Once a session is established between the client and the conference server, video and audio is transmitted from each of the clients involved in the conference to the conference server. The conference server will then combine the video streams and transmit them to the clients for output at the client. The server typically controls the resolution of the video received by the client and changes the resolution based on bandwidth constraints, without any input from the client or consideration of the user experience on the client.

Although specific problems and issues have been identified in this background section, the embodiments described herein are not limited to solving these particular problems or issues. The embodiments may be applied to solve problems not described in this background section.

SUMMARY

It is with respect to the above issues and other problems that the embodiments presented herein were contemplated. Embodiments described in the present application provide for a client to negotiate attributes that affect a user experience during a multimedia session, such as a video conference. The client can negotiate attributes that affect, for example, the layout of continuous presence information displayed to a user, the video resolution of displayed information, shuffling of windows during the video conference, and the like.

In one embodiment, a method is provided that includes generating an offer, and indicating in the offer, window content specification, e.g., first window identifier for a first and second window, a bandwidth limit identifier for the first and second window, and a group identifier for the first and second window. The offer is then transmitted, e.g., to a conference server. An answer to the offer is then received. In some embodiments, the offer, and answer, is formatted according to a Session Description Protocol (SDP). After the answer is received, video content is received for displaying in the first window and the second window.

In embodiments, window content specification includes a first group identifier that is assigned a higher priority than a second group identifier. The higher priority indicates that resolution reductions should be applied to content for display in windows of the second group before resolution reductions are applied to content for display in windows in the first group. In embodiments, the first bandwidth limit identifier indicates a limit for reducing the resolution of video content for display in windows of the first group. The first bandwidth limit identifier may indicate a percentage of an original resolution for the window.

In some embodiments, in addition to the other identifiers, the offer includes a first rank identifier for the first window and/or a second rank identifier for the second window. The rank identifiers are used to control the shuffling of windows displaying continuous presence information to a user. For example, if a user is participating in a video conference, the rank can control the display of active speakers within various windows. In embodiments, the first window has a rank such that the most recent active speaker is displayed in the first window. Similarly, the second window can be ranked such that the second most recent active speaker is displayed. In some embodiments, a window can be ranked so that they are pinned, meaning that the same participant is always displayed in the window. In yet other embodiments, the shuffling of the windows that result from speakers coming in and out is minimized.

Another embodiment is directed to a communication device, e.g., a conference server, which includes a non-transitory computer readable medium, a processor, and an application stored in the computer readable medium and running on the processor. The application receives an offer for a multimedia communication session, the offer including, in embodiments, a first window identifier for a first window, a bandwidth limit identifier for the first window, a first group identifier for the first window, a second window identifier for a second window, a second bandwidth limit identifier for the second window, and a second group identifier for the second window. In embodiments, the application transmits an answer in response to receiving the offer. The answer in the offer are formatted according to SDP, some embodiments. The application then transmits video content for displaying in the first window and the second window. In some embodiments, the application reduces the resolution of video content, in response to a bandwidth constraint. The resolution reduction is based on the group identifier as well as the bandwidth limit identifier received in the offer. For example, the first group identifier may, in embodiments, have a lower priority than the second group identifier, in which case the resolution of video for display on windows associated with the first group identifier is reduced, up to the first bandwidth limit, before the resolution of video for display on windows associated with the second group identifier is reduced.

Other embodiments are directed to computer readable medium including computer executable instructions stored onto the computer readable medium which, when executed by one or more processors of a computer, causes the computer to perform a method for negotiating a multimedia session. The method includes generating an offer for a multimedia communication session. The offer includes in embodiments a plurality of window identifiers for a plurality of windows, a bandwidth limit identifier for each of the plurality of windows, and a group identifier for each of the plurality of windows, wherein a first group identifier for a first portion of the plurality of windows is different than a second group identifier for a second portion of the plurality of windows. The offer is then transmitted to a server. An answer to the offer is received from the server, and video content for displaying in the plurality of windows is received from the server.

The term “in communication with” as used herein refers to any coupling, connection, or interaction using electrical signals to exchange information or data, using any system, hardware, software, protocol, or format.

The term “computer-readable medium” as used herein refers to any tangible storage that participates in providing instructions to a processor for execution. Such a medium may take many forms, including but not limited to, non-volatile media, volatile media, and transmission media. Non-volatile media includes, for example, NVRAM, or magnetic or optical disks. Volatile media includes dynamic memory, such as main memory. Common forms of computer-readable media include, for example, a floppy disk, a flexible disk, hard disk, magnetic tape, or any other magnetic medium, magneto-optical medium, a CD-ROM, any other optical medium, punch cards, paper tape, any other physical medium with patterns of holes, a RAM, a PROM, and EPROM, a FLASH-EPROM, a solid state medium like a memory card, any other memory chip or cartridge, or any other medium from which a computer can read. When the computer-readable media is configured as a database, it is to be understood that the database may be any type of database, such as relational, hierarchical, object-oriented, and/or the like. Accordingly, embodiments are considered to include a tangible storage medium and prior art-recognized equivalents and successor media, in which the software implementations of the embodiments are stored.

The term “module” as used herein refers to any known or later developed hardware, software, firmware, artificial intelligence, fuzzy logic, or combination of hardware and software that is capable of performing the functionality associated with that element. Also, while exemplary embodiments are described, it should be appreciated that individual aspects of the embodiments can be separately claimed.

DETAILED DESCRIPTION

Embodiments described in the present application provide for a client to negotiate attributes that affect a user experience during a multimedia session, such as a video conference. The client is able to have some control of how audio/video data is output to a user, including without limitation, the video resolution of displayed information, shuffling of windows based on active speakers participating in a video conferences, and the layout of how the windows are displayed to a user. For example, the session may involve multi-stream continuous presence (CP) video sent as part of the video conference.

FIG. 1illustrates a system100that includes communication devices102A-102N, e.g., mobile phones, smart phones, mobile communications devices, telephones, soft phones, video displays, televisions, monitors, desktop computers, laptop computers, and the like. As illustrated inFIG. 1, communication devices102A-102N are connected to a network104that allows the communication devices102A-102N to communicate with each other. Also connected to the network104is a server108, which in embodiments is a conference server with video conference and/or multimedia capabilities.

Communication device102A includes, among other features, a memory112, which may store files and executing application(s) and/or modules such as SIP/SDP module116and video conference module120. As described in greater detail below, SIP/SDP module116and video conference module120are used to negotiate and engage in multimedia sessions between communication device102A and other communication devices (e.g.,102B-102N) or servers (e.g.,108).

In addition to memory112, communication device102A also includes additional hardware such as a processor124, and communication systems128. The processor124is used to execute the code of applications and modules such as SIP/SDP module116and video conference module120and other applications stored in memory108. A bus132provides a connection for transmitting signals among the memory112, processor124, and communication systems128. Communication device102A also includes a display136, which is configured to display audio/visual data that is received by communication device102A as part of a multimedia session. In the embodiment shown inFIG. 1, display136is displaying windows140,144, and148in which continuous presence information is displayed from participants of a video conference. In addition, communication device102A may also include other input/output devices, including but not limited to display(s), e.g., speakers, lights, keypads, and microphones.

It is noted that although SIP/SDP module116and video conference module120are shown inFIG. 1as stored in memory112of communication device102A, in other embodiments, at least portions of the modules are stored on a server(s), i.e., they may utilize distributed code. As one example, if video conference module120is stored, at least in part, on a server, communication device102A will communicate, using communications systems128, with the server to access information from the video conference module120, such as routines, subroutines, or other code, that may be stored on the server.

Server108includes among other features, memory152, where files and modules are stored such as multimedia module156, conference module160, and SIP/SDP module164. Server108also includes additional hardware such as a processor168, and communication systems172. The processor124is used to execute the code of applications and modules such as multimedia module156, conference module160, and SIP/SDP module164and other applications stored in memory152. A bus176provides a connection for transmitting signals among the memory152, processor168, and communication systems172. In addition, communication device102A may also include other input/output devices, including but not limited to display(s), e.g., speakers, lights, keypads, and microphones.

In embodiments, communication device102A engages in a number of different types of multimedia sessions with server108. One specific type of multimedia session is a video conference in which a number of participants at different points utilize communication devices, e.g.,102B-102N to communicate in real-time using both audio and video data. Server108serves as a central point for collecting audio and video data from the communication devices. Server108then transmits the audio video data to the communication devices for output. Although the description below focuses on videoconferencing, embodiments are not necessarily limited to this application. In other embodiments, the media sessions may involve previously recorded (or near real time) audio/video data that is displayed to a user for entertainment, security, information, or other reasons. Therefore, although the description below provides the specific example of videoconferencing, embodiments are not limited thereto.

FIG. 2illustrates one embodiment of communication device102A negotiating and establishing a multimedia session, specifically a video conference session, with server108, which is serving as the conference server for the video conference. In addition to communication device102A at least three other communication devices, and participants, are also participating in the video conference. For purposes of simplicity, the plurality of messages200exchanged during the negotiation between communication device102A and server108are shown as exchanged directly between communication device102A and server108. However, as can be appreciated, in actual operation, the plurality of messages200are transmitted through one or more networks which may be a LAN, a WAN, or other type of network. Additionally, it is noted that although specific messages are shown as being exchanged between communication device102A and server108, in other embodiments additional messages will be exchanged between device102A and server108. For example, messages for security protocols, transport protocols, and/or session initiation protocols will also be exchanged, in some embodiments. These additional messages may be exchanged before or after the plurality of messages200shown inFIG. 2.

As illustrated inFIG. 2, communication device102A initially sends an offer204to server108. The offer204may be formatted according to any appropriate protocol used to negotiate multimedia sessions. In one specific embodiment, the offer is formatted according to a Session Description Protocol (SDP), which provides a format for describing streaming media initialization parameters. Embodiments are not limited to SDP and the offer204may be in any suitable format. In addition, other protocols such as security protocols, transport protocols, and/or multimedia protocols may be used in generating and transmitting offer204. In one embodiment, Security Initiation Protocol (SIP) is used as a transport protocol when transmitting offer204. In this embodiment, one or more of SIP/SDP module116and video conference module120, are used to generate offer204.

In response to offer204, server108transmits an answer208, followed by video data212, which is multi-stream continuous presence (CP) video (e.g., multi Scalable Video Coding (SVC) or Advanced Video Coding (AVC) video streams) from participants in the videoconference. The communication device102A decodes the video data212and renders the CP video for display on various windows on communication device102A. In embodiments where SDP is used in formatting the offer204, the multi-stream video streams are negotiated using n video (e.g., “m lines”) in an SDP offer, where n>1. For example, n=4 indicates that the CP video will contain 4 participants/windows. A single video, referred to in an SDP request as an m line with an n=1, means no CP and typically display the most recent active speaker.

In conventional negotiations using SDP, only the codec used, bit rate (AVC and SVC), number of layers used (SVC) and direction (e.g., received only (recvonly), send and receive (sendrcv), send only (sendonly)) are negotiated. However, these do not address the user experience aspects of the CP layout (e.g. whether windows are displayed in a 2×2, 1+3 format), the grouping of CP windows, handling of bandwidth reduction/optimization and window shuffling algorithms. Therefore according to embodiments, offer204includes additional attributes that allow user experience aspects to be negotiated.FIG. 3illustrates an embodiment of offer204that is formatted according to SDP, consistent with one embodiment.FIG. 4illustrates an embodiment of an answer208that is formatted according to SDP, consistent with one embodiment.

As shown inFIG. 3, offer204includes a number of different attributes that may be referred to as window content specification. At line216, there are a string of attributes that, consistent with embodiments, address user experience aspects of the video that will be transmitted as part of the video conference being negotiated by communication device102A. The attributes include a first window identifier220, a first group identifier224, a first bandwidth limit identifier228, and a first rank identifier232. Each of these attributes provides information for how the client will output at least a portion of the video, transmitted as part of the video conference, in a first window. Line236also includes attributes including, a second window identifier240, a second group identifier244, a second bandwidth limit identifier248, and a second rank identifier252. These attributes provide information for how the client will output at least a second portion of the video in a second window. As noted above, these parameters (window identifiers, bandwidth limit identifiers, group identifiers, and rank identifiers) can be included in an offer and may be referred to as window content specification.

In some embodiments, the attributes are selected by a user or an administrator. The attributes can be selected to tailor the user experience to the multimedia session, or according to a particular preference. In other embodiments, there may be default values that are set if no input for the attributes are received. For example, the default values may be group identifier=1, bandwidth reduction limit identifier=100, vas-rank=1. These values are described in further detail below.

Additionally, line218includes an indication as to whether or not the attributes that precede line218are applicable to video content that is sent and/or received. As indicated inFIG. 3, line218indicates that the attributes noted above it, including in line216, are intended for bidirectional video, i.e., video that is both sent and received by communication device102A. Line238indicates that the attributes noted above it are only for video that is received. The client can therefore be flexible when negotiating the session with server108, such as by indicating that it will send video in high resolution but only receive video in lower resolution or vice versa.

FIG. 4illustrates an embodiment of an answer208that is formatted according to SDP. Answer208acknowledges the attributes that were sent in the offer204. In some embodiments, answer208may provide counter offers or may indicate that it cannot accommodate the attributes that have been requested by communication device102A. In these embodiments, communication device102A would then send another offer with different attributes in an attempt to negotiate attributes that are acceptable to server108.

Referring again toFIG. 3, window identifiers220and240are used to identify the windows on the client that will be used to display the video streams. Although only two window identifiers are shown in the offer204, in other embodiments, offer204may include more than two window identifiers. Each identifier is associated with a single window, which is used to display video from one video stream. In embodiments, each video stream is from one of the participants in the conference. As can be appreciated, any type of identifier may be used as window identifiers220and240, including any alphanumeric value. In one embodiment, group identifiers are one or two digit numbers that range from 1-99, with the lower number being of higher priority.

Group identifiers224and244are each associated with one or more windows. The group identifiers224and244are used to group windows together. Windows are grouped together for any number of reasons, for example to identify groups of windows with similar properties, i.e., windows with the same size and resolution, to change the properties of more than one window at a time, or for any other purpose. In one embodiment, group identifiers224and244are used in combination with bandwidth limit identifiers228and248as described in greater detail below. Although group identifiers224and244are shown in offer204as numeric values, in other embodiments, they may be any identifier including an alphanumeric value.

In embodiments, windows are grouped according to their layout as displayed on communication device102A.FIGS. 5 and 6illustrate four windows (304,308,312, and316) that are grouped differently according to their displayed layouts on communication device102A.FIG. 5illustrates that the four windows are each grouped within a single group, namely group1. As can be seen, the four windows (304,308,312, and316) are the same size when displayed on display136.

FIG. 6illustrates a second embodiment in which the four windows (304,308,312, and316) are grouped within two different groups. InFIG. 6, group1includes a single window304and group2includes three windows308,312, and316. The layout shown inFIG. 6can be used to highlight the participant that is currently actively speaking by displaying the video of the active speaker on window304. The remaining windows308,312, and316are be used to display video from other participants in the video conference.

It is noted thatFIGS. 5 and 6are provided to illustrate an example of window groupings consistent with embodiments. In other embodiments, however, the groupings and specific layouts of windows may vary. For example, the windows308,312, and316in group2may be displayed in a different layout, such as above window304, below window304, or on the left side of window304. In another embodiment, each window from group2can be displayed near a different corner of window304. These are merely some examples, and other groupings and/or layouts are possible. As can be appreciated, moderator and participants in the video conference can use different layouts and each can change their layout mid call, each can also have/change to a single Active Speaker window or just audio. This flexibility is not currently available.

Referring again toFIG. 3, bandwidth limit identifiers228and248are used to set a limit on the amount a video stream can be reduced in resolution. As can be appreciated, conference servers are under bandwidth constraints. Therefore, they can at any time reduce their bandwidth consumption by reducing the resolution of video it is streaming. Typically, the server reduces the resolution of streaming video based on its own preprogrammed algorithms without necessarily considering the user experience on the client. Bandwidth limit identifiers228and248allow the client in its negotiation of the multimedia session to limit the amount that particular video streams can be reduced in resolution. This allows the client to control the user experience. For example, if the client determines that video being displayed in one particular window will suffer too greatly from quality if reduced to below a predetermined resolution; it will provide a bandwidth limit identifier that does not allow for the resolution to fall below the predetermined resolution. On the other hand, there may be some video displayed in another window whose quality can be reduced by more than the predetermined resolution and still provide a suitable user experience. The client can therefore provide a lower bandwidth resolution limit.

Bandwidth limit identifiers228and248can be any suitable identifier that is understood by the server as a bandwidth limit. In the offer204shown inFIG. 3bandwidth limit identifier228indicates a percentage of an original resolution, namely 25% of an original resolution. In this embodiment, the server understands that the client has requested that the video stream associated with window1should not be reduced by more than 25% of the original resolution. The client in this embodiment has determined that reducing the resolution of window1by more than 25% affects the quality of the user experience too greatly and therefore 25% has been used as the bandwidth limit identifier228. On the other hand, bandwidth limit identifier248provides a limit of 50% of an original resolution. Thus, the video stream associated with window2can be reduced by as much as 50% of its original resolution. The client is therefore determined that the video being played in window2can be reduced up to as much as 50% of its original resolution and still provide an adequate user experience.

As one example, the display layout may be as shown as inFIG. 6, with window1corresponding to window304and window2corresponding to one of windows308,312, or316. Because window2is a smaller window, reducing the resolution does not affect the user experience as much as reducing the resolution of content displayed in window1. Thus, reducing the resolution up to 50% of the original resolution may be acceptable. In contrast window1, which is larger, will have a grainy appearance if the resolution is reduced too much.

The bandwidth limit identifiers228and248are defined as a percentage of a video's original resolution. Although in other embodiments, bandwidth limit identifiers can be defined differently. For example, offer204may refer to a specific resolution. In other embodiments, the bandwidth limit identifiers may be an alphanumeric value that is understood by the server to represent resolution limits. As can be appreciated, these are merely some examples and the bandwidth limit identifiers are not necessarily limited thereto.

As indicated above, group identifiers224and244can be used in combination with the bandwidth limit identifiers228and248to control the user experience at communication device102A. In some embodiments, the group identifiers224and244have a predetermined priority. For example, the group identified by group identifier224(“group1”) may have a higher priority than the group identified by group identifier228(“group2”). When used in combination with the bandwidth limit identifiers228and248, the server108understands that if there is a need to reduce resolution of video being streamed from server108to communication device102A, because of bandwidth constraints, the video content associated with windows in the group associated with group identifier228(namely “group2”) should be reduced first, up to the resolution limit identified by bandwidth limit identifier228. After the reduction in resolution of the video content associated with the windows in group2, if necessary, the video content associated with the windows in group1can then be reduced in resolution, up to the resolution indicated by bandwidth limit identifier248. In combination, the group identifiers and the bandwidth identifiers are used to control the user experience at communication device102A, which is not currently possible with the available versions of SDP.

Referring again toFIG. 3, rank identifiers232and252in offer204are provided to allow for windows to change based on participants speaking activity. These identifiers may be referred to as voice active rank identifiers. The rank identifiers indicate the desired assignment of active speakers to a window. The server will assign the window based on the active speaker history and the rank identifier provided by the client. The identifier with the highest rank, e.g., identifier232(“Rank1”) gets the most recently active speaker. In other words, the participant that is currently speaking, or most recently spoke, is displayed in the window with the highest rank, which in offer204is window1. The identifier with the next highest rank, e.g., identifier252(“Rank2”) gets the second most recently active speaker, and so on.

FIGS. 7A-10Billustrate various embodiments of using different rank identifiers for four windows (304,308,312, and316), and their behavior in response to their rank identifiers and speaking activity. These are provided for illustrative purposes only and embodiments are not necessarily limited thereto. For simplicity, only display136of communication device102A-is shown inFIGS. 7A-10B.

InFIG. 7A, communication device102A has sent an offer, such as offer204, indicating four windows (304,308,312, and316), each of which displays different continuous presence information for participants in the videoconference. Window304was associated with a rank identifier “rank1,” window308was associated with a rank identifier “rank2,” window312was associated with a rank identifier “rank3,” and window316was associated with a rank identifier “rank4.” Consistent with the description above, in this embodiment the window with rank1displays the most recently active speaker, the window would rank2displays the second most recently active speaker, the window with rank3displays the third most recent speaker, and the window with rank4displays the fourth most recent speaker. As shown inFIG. 7A, participant1is the most recently active speaker, participant2is the next most recent speaker, participant3is the third most recent speaker, and participant4is the fourth most recent speaker.

When participant5begins to speak, windows304,308,312, and316are shuffled in response, consistent with their rank.FIG. 7Billustrates, windows (304,308,312, and316), after they have been shuffled in response to participant5speaking. Participant5is shown in window304because she is the most recent speaker and window304has the rank identifier rank1. Windows308,312, and316are shuffled so that participant1is shown in window308, participant2is shown in window312, and participant3is shown in window316.

FIGS. 8A and 8Billustrate windows304,308,312, and316with the same rank identifier, namely rank1. This illustrates the embodiment where multiple windows have the same rank, which results in minimal shuffling. As shown inFIG. 8A, participant1is displayed in window304, participant2is displayed in window308, participant3is displayed in window312, and participant4is displayed in window316. When participant5begins to speak, instead of replacing participant1in window304, participant5replaces participant4and is displayed within window316, as shown inFIG. 8B. None of the other windows are changed. That is, shuffling is minimized so that only one window is changed to display the most recent speaker. In this embodiment, the server108decides which of the windows is changed so as to minimize the shuffling of windows304,308,312, and316. In other embodiments, the server108may decide to replace any of the other participants in the other windows, as long as the shuffling is minimized. This feature allows the communication device102A to control the user experience by assigning the same rank identifier for all of the windows, which results in the server having to minimize shuffling of windows when speakers switch in and out.

Some embodiments provide for selecting rank identifiers so that the behavior is a combination of having the most recently active speaker highlighted, but minimizing the shuffling of the other participants.FIGS. 9A and 9Billustrate windows304,308,312, and316where window304has a rank identifier of rank1, and windows308,312, and316have the same rank identifier, namely rank2. Window304will display the most recent active speaker. The other windows308,312, and316, will display the next 3 most recent active speakers. Because windows308,312, and316, all have the same rank identifier, the order for these windows will be determined by the server108to minimize shuffling. As shown inFIG. 9B, when participant5begins to speak, window304is changed to display participant5. To minimize shuffling, server108has changed window316to display participant1. This minimizes shuffling among windows308,312, and316, because308and312remain unchanged.

In some embodiments, rank identifiers can be used to pin a particular window. By “pinning” it is meant that the participant displayed in a window is never changed, even if there is speaking activity by other participants.FIGS. 10A and 10Billustrate windows304,308,312, and316where window304has a rank identifier of rank1, windows308has a rank identifier of rank2, window312has a rank identifier of rank3, and window316has a rank identifier of rank0. In this embodiment, rank0indicates that a window is pinned; therefore, participant4(shown in window316) is always displayed in window316. When participant5begins to speak, as shown inFIG. 10Bparticipant5replaces participant1in window304. Participant1is then displayed in window308, and participant2is displayed in window312. Because window316is associated with rank identifier rank0, it continues to display participant4even after participant5begins to speak. This embodiment may be useful in a number of situations. For example, in videoconferences where there will primarily be one speaker, the rank identifier can be selected so that the primary speaker is always displayed in a window even when not speaking. This embodiment is also useful in situations where there is an important participant in the videoconference, so even if not speaking the participant should be displayed in one of the windows.

In some embodiments, if a rank is not specified, in an offer, a default rank identifier is assigned. For example, a rank of 1 may be the default value for all windows in order to minimize shuffling. As can be appreciated, in other embodiments, the default value may be any value that is predetermined by an algorithm in the communication device102A, selected by a user of communication device102A, or preprogrammed by an administrator.

It is noted that although specific examples of offers and attributes in the offers are described above, any combination of attributes can be used to describe layouts for displaying video data, such as continuous presence information for a video conference. Additional examples of combination of attributes that describe various layouts (some of which are shown inFIGS. 5-10B) are provided below for illustrative purposes.

a=conteni: window1.1.100,1(this window gets the most recent speaker)

In Example 2, window1will display the most recent active speaker. The other 3 windows will get the next 3 most recent active speakers, minimum shuffling will be applied to windows 2,3,4.

In Example 3, all four windows will get switched with active speaker streams, minimum shuffling will be applied to all.

Referring now toFIG. 11, a flow diagram500, for negotiating a multimedia session, e.g., a videoconference. Flow500is in embodiments performed by a computing device such as communication device102A (FIGS. 1-2) or other client computing device. More specifically, one or more hardware or software components may be involved in performing flow500. For example, portions of flow500may be performed by a SIP/SDP module116and/or video conference120.

Flow500begins with step504where an offer for negotiating a multimedia session is generated. The offer is in embodiments formatted according to SDP, such as offer204described above. The offer may be formatted according to different protocols in other embodiments. Flow passes from step504to step508where an indication of a number of different attributes including a first window identifier, a first bandwidth limit identifier, and/or a first group identifier, are indicated in the offer. At optional step512, a rank identifier may also be included in the offer generated at step504. As indicated above, the rank identifier may be used in controlling the behavior of displayed windows in response to speaking activity.

At step516, a second group of attributes including a second window identifier, a second bandwidth limit identifier, and/or a second group identifier, are indicated in the offer. At optional step520, a second rank identifier may also be included in the offer generated at step504. As can be appreciated, flow500is limited to an offer that identifies two windows and attributes associated with the two windows. In some embodiments, the offer may include attributes of more than two windows, in which case, flow500will include additional steps for indicating attributes of the additional windows.

After the indications of window attributes have been made in the offer, for all of the windows, flow500passes to step524, where the offer is delivered to a network for transmission. In embodiments, step524includes in embodiments delivering the offer to a multimedia server and/or a videoconference server. Step524may entail the use of a number of different protocols, such as transport protocols, security protocols, and other multimedia protocols. Step524includes the necessary sub steps (e.g., generating headers, packets, etc.) for delivering the offer for transmission to the server. And answers that received at step528.

Flow passes from step528to step532where video content is received for display. The video content is then displayed at step536. Step536may involve a number of sub steps including decoding the video content received at step532before it is displayed. Display of the video content at step532involves displaying the video content on a number of different windows. The windows may be laid out in any desired manner, some examples shown inFIGS. 5 and 6, and described above.

At step540, lower resolution video content is received and at step544the lower resolution video content is displayed. Step540may be a result of the server having reached bandwidth constraints. In order for the server to comply with the bandwidth constraints, it must reduce the resolution of the video content. As indicated above however, the offer delivered at step524included attributes such as group identifiers and bandwidth limit identifiers. The reduced resolution content received at step540and displayed at step544will therefore be consistent with the bandwidth limit identifiers sent in the offer. As one example, the video content for display in windows identified by a group identifier of a lower rank will be reduced in resolution first, up to any bandwidth limit identifier associated with the group. Video content for display in windows identified by a second group identifier, of a higher rank, will then be reduced up to any bandwidth limit identifier associated with the second group.

As noted above, optional rank identifiers may be indicated in the offer at optional steps512and520. The rank identifiers are used to control the display of video content in windows as a result of speaking activity. In those embodiments, flow500will include optional step548, where modified data is received based on the rank and the speaking activity during the multimedia session. Step548is then followed by step552where the modified video content is displayed. Flow500then ends at556.

FIG. 5illustrates a flow diagram600for negotiating a multimedia session, such as a videoconference. Flow600is in embodiments performed by a computing device such as server108(FIGS. 1-3), or other multimedia and/or conference server. More specifically, one or more hardware or software components may be involved in performing flow600. For example, portions of flow500may be performed by multimedia module156, videoconference module160, and/or SIP/SDP module164described above.

Flow600begins with step604where an offer for a multimedia session is received. The offer includes indications of attributes including one or more window identifiers, one or more bandwidth limit identifiers, one or more group number identifiers, and/or one or more ranks. In embodiments, the offer is received from a client device, such as communication device102A. Flow passes from step604to step608where in response to the offer received at step604, an answer is transmitted. In embodiments, the answer may be formatted according to SDP. The answer may acknowledge the attributes of the offer received at step604, indicating that the attributes are acceptable. In other embodiments, the answer may provide a counter offer that includes different attributes than those included in the offer received at step604. In these embodiments, the offer and answer are one of several messages that are sent and transmitted during the negotiation of the multimedia session.

Flow600passes from step608to step612where video content is transmitted. It is noted that in order for the video content to be transmitted, there may be additional steps that are performed in parallel with and/or prior to step612. For example, video content can be received from various sources, i.e. client devices that are being utilized by participants of a videoconference. The video content can include combined video/audio streams that are then encoded before they are transmitted at step612.

Following step612, resolution of the video content is reduced at step616. Step616may be performed as a result of bandwidth constraints. For example, a server performing flow600may be utilizing bandwidth to perform other operations. As a result, the server must do something to reduce its bandwidth consumption. By reducing the resolution of video content at step616, the server can comply with the bandwidth constraints. Their reduction of resolution is performed in accordance with the attributes received in the offer received at step604. For example, if there are priorities set with respect to group identifiers, then the video content for display on those windows with a lower priority group is first reduced in resolution. Additionally, if there is a bandwidth limit identifier that limits the amount by which the resolution can be reduced, then the server will comply with those bandwidth limit identifiers. At step620, the reduced resolution video content is transmitted.

In those embodiments in which the offer includes a rank identifier, flow600includes additional optional steps624and628. As described above, the rank identifiers are used in shuffling windows on the communication device that is displaying the video content transmitted at steps612and620, in response to speaker activity. Therefore, at step624, a server can modify the video content in accordance with the ranks received any offer, and the speaking activity. The modified video content is then transmitted at step628. Flow then ends at632.

It is noted that although flows500and600illustrate steps in an order, other embodiments are not necessarily limited thereto. The steps shown inFIGS. 11 and 12may be performed in any order or in parallel. Additionally, there may be other steps performed that are not shown inFIGS. 11 and 12or described above. Also, although the flows500and600are described above as being performed in some embodiments by particular hardware and/or software components, other embodiments are not necessarily limited to the description above. As can be appreciated, steps500and600can be performed by other hardware or software not described above or shown inFIGS. 11 and 12.

FIG. 13illustrates one embodiment of a computer system700upon which servers or other systems described herein may be deployed or executed. The computer system700is shown comprising hardware elements that may be electrically coupled via a bus755. The hardware elements may include one or more central processing units (CPUs)705; one or more input devices710(e.g., a mouse, a keyboard, etc.); and one or more output devices715(e.g., a display device, a printer, etc.). The computer system700may also include one or more storage device720. By way of example, storage device(s)720may be disk drives, optical storage devices, solid-state storage device such as a random access memory (“RAM”) and/or a read-only memory (“ROM”), which can be programmable, flash-updateable and/or the like.

The computer system700may additionally include a computer-readable media reader725; a communications system730(e.g., a modem, a network card (wireless or wired), an infra-red communication device, etc.); and working memory740, which may include RAM and ROM devices as described above. In some embodiments, the computer system700may also include a processing acceleration unit735, which can include a DSP, a special-purpose processor and/or the like.

The computer-readable media reader725can further be connected to a computer-readable medium, together (and, optionally, in combination with storage device(s)720) comprehensively representing remote, local, fixed, and/or removable storage devices plus a computer-readable medium for temporarily and/or more permanently containing computer-readable information. The communications system730may permit data to be exchanged with the network520and/or any other computer described above with respect to the system700.

The computer system700may also comprise software elements, shown as being currently located within a working memory740, including an operating system745and/or other code750, such as application code implementing the servers or devices described herein. It should be appreciated that alternate embodiments of a computer system700may have numerous variations from that described above. For example, customized hardware might also be used and/or particular elements might be implemented in hardware, software (including portable software, such as applets), or both. Further, connection to other computing devices such as network input/output devices may be employed.

Furthermore, embodiments may be implemented by hardware, software, firmware, middleware, microcode, hardware description languages, or any combination thereof. When implemented in software, firmware, middleware or microcode, the application code or code segments to perform the necessary tasks may be stored in a machine readable medium such as storage medium. A processor(s) may perform the necessary tasks. A code segment may represent a procedure, a function, a subprogram, an application, a routine, a subroutine, a module, a software package, a class, or any combination of instructions, data structures, or application statements. A code segment may be coupled to another code segment or a hardware circuit by passing and/or receiving information, data, arguments, parameters, or memory contents. Information, arguments, parameters, data, etc. may be passed, forwarded, or transmitted via any suitable means including memory sharing, message passing, token passing, network transmission, etc.