Abstract:
When a multimedia conference between multiple users at multiple locations is established, a first cursor controlled by a first user in a first location is provided. A first event associated with the multimedia conference is detected. In response to detecting the first event associated with the multimedia conference, control of cursors in the multimedia conference can be handled in various ways to enhance the multimedia conference. For example, control of the first cursor can be switched from the first user to a second user at a second location, a second cursor can be provided to the multimedia conference that is controlled by the second user, or control of the first cursor can be merged so that the first cursor can be controlled by both the first user and the second user. This allows for enhanced control of the cursor in the multimedia conference that more closely resembles an in-person conference.

Description:
TECHNICAL FIELD 
       [0001]    The systems and methods disclosed herein relate to multimedia systems and in particular to control systems for multimedia communications. 
       BACKGROUND 
       [0002]    In face-to-face communications, people naturally and instinctively use not only voice, but also use gestures and movements, such as, pointing to direct a listener&#39;s focus while speaking. For example, each person involved in a face-to-face communication may use their hand or a laser pointer to draw attention to documents or diagrams being discussed during a presentation. 
         [0003]    In multimedia conferences, the mouse pointer and annotations are the instinctive virtual equivalents of the laser pointer and a whiteboard marker. However, in current systems, only the presenter&#39;s cursor is visible to other users and only the presenter can control the cursor or annotate a document in the presentation. The remaining participants in the multimedia communication currently do not have the ability to be involved in the same manner as in a face-to-face communication. 
       SUMMARY 
       [0004]    Systems and methods are provided to solve these and other problems and disadvantages of the prior art. When a multimedia conference between multiple users at multiple locations is established, a first cursor controlled by a first user in a first location is provided. A first event associated with the multimedia conference is detected. For example, a user (e.g., the first user or a user other than the first user) may issue a command via a menu in the multimedia conference to control the cursor. In response to detecting the first event associated with the multimedia conference, control of one or more cursors in the multimedia conference can be handled in various ways to enhance the multimedia conference. For example, control of the first cursor can be switched from the first user to a second user at a second location (which may or may not be the same as the first location), a second cursor can be provided to the multimedia conference that is controlled by the second user, or control of the first cursor can be merged so that the first cursor can be controlled by both the first user and the second user. These options allow for enhanced control of the cursor in the multimedia conference that more closely resembles an in-person meeting. 
     
    
     
       BRIEF DESCRIPTION OF THE DRAWINGS 
         [0005]      FIG. 1  is a block diagram of a first illustrative system for controlling a cursor in a multimedia conference. 
           [0006]      FIG. 2  is a diagram of a first view of a multimedia conference that uses a single cursor. 
           [0007]      FIG. 3  is a diagram of a second view of a multimedia conference that uses two or more cursors. 
           [0008]      FIG. 4  is a flow diagram of a process for controlling a cursor in a multimedia conference. 
           [0009]      FIG. 5  is a flow diagram of privately controlling a cursor in a multimedia conference based on a sidebar communication. 
       
    
    
     DETAILED DESCRIPTION 
       [0010]      FIG. 1  is a block diagram of a first illustrative system  100  for controlling a cursor in a multimedia conference. The first illustrative system  100  comprises communication endpoints  101 A- 101 N, a network  110 , and a communication system  120 . 
         [0011]    The communication endpoints  101 A- 101 N can be or may include any device that can communicate on the network  110  and is an endpoint in a communication session, such as, a Personal Computer (PC), a telephone, a video phone, a cellular telephone, a Personal Digital Assistant (PDA), a tablet device, a notebook device, and/or the like. As shown in  FIG. 1 , any number of communication endpoints  101 A- 101 N may be connected to the network  110 . In addition, one or more of the communication endpoints  101 A- 101 N may be directly connected to the communication system  120 . The communication endpoints  101 A- 101 N are typically at different locations. For example, the communication endpoint  101 A can be located in Denver, the communication endpoint  101 B can be located in New Jersey, and the communication endpoint  101 N can be located in Spain. A communication endpoint  101  may have a single user associated therewith or multiple users associated therewith. The latter type of communication endpoint  101  may also be referred to as a shared endpoint due to the fact that multiple users share input and/or output devices of the communication endpoint  101 . 
         [0012]    The network  110  can be or may include any collection of communication equipment that can send and receive electronic communications, such as the Internet, a Wide Area Network (WAN), a Local Area Network (LAN), a Voice over IP Network (VoIP), the Public Switched Telephone Network (PSTN), a packet switched network, a circuit switched network, a cellular network, a video network, a multimedia network, a combination of these, and the like. The network  110  can use a variety of electronic protocols, such as Ethernet, Internet Protocol (IP), Session Initiation Protocol (SIP), Integrated Services Digital Network (ISDN), video protocols, Extended Markup Language (XML) Hyper Text Markup Language (HTML) Requests, web sockets, Real Time Protocol (RTP), web protocols, and/or the like. Thus, the network  110  is an electronic communication network that allows for sending of messages via packets and/or circuit switched communications. 
         [0013]    In one embodiment, the network  110  comprises the PSTN (or other voice network) and a data network, such as the Internet, corporate network, and/or LAN. The PSTN is used for the audio portion of a multimedia conference. The Internet/corporate network/LAN is used for the multimedia portion of the multimedia conference. For example, a user may call via a telephone for the audio portion using the PSTN. The user then may use their PC for the multimedia portion/cursor control via the Internet or corporate network by accessing a web URL. 
         [0014]    The communication system  120  can be or may include any hardware/software that can establish and control a multimedia conference, such as a Private Branch Exchange (PBX), a communication manager, a video switch, a session manager, and/or the like. The communication system  120  further comprises a mixer  121 , a conferencing application  122 , a conference profile  124 , and a web server  125 . The communication system  120  can include custom hardware that can be used in management and control of the multimedia conference. For example, the communication system  120  may comprise a field programmable gate array or application specific processor, such as a digital signaling processor. 
         [0015]    The mixer  121  can be or may include any hardware/software that can mix voice, video, multimedia, and/or text communications. For example, the mixer  121  may mix audio signals/streams or video signals/streams for a discussion in a multimedia conference between users of the communication endpoints  101 A- 101 N. In  FIG. 1 , the mixer  121  is shown as part of the communication system  120 . However, in another embodiment, the mixer  121  may be distributed between the communication system  120  and the communication endpoints  101 A- 101 N. 
         [0016]    The conferencing application  122  can be any software/hardware that can manage a multimedia conference between the communication endpoints  101 A- 101 N. A multimedia conference is a conference that provides multimedia (e.g., display of documents) to two or more users. The conferencing application  122  is shown as part of the communication system  120 . However, in another embodiment, the conferencing application  122  may be distributed between the communication system  120  and the communication endpoints  101 A- 101 N. 
         [0017]    The conferencing application  122  also comprises a cursor control module  123 . The cursor control module  123  can be or may include any hardware/software that can control a cursor in a multimedia conference or enable various users to control the cursor in the multimedia conference. 
         [0018]    The conference profile  124  can be any file or set of information that contains information for controlling aspects of a multimedia conference, such as an American Standard Code for Information Exchange (ASCII) file, an Extensible Markup Language (XML) file, a text file, a binary file, and/or the like. The conference profile  124  can include user preferences and/or administrator preferences on how to control one or more cursors in a multimedia conference. 
         [0019]    The web server  125  can be or may any include hardware/software that can provide web services for the multimedia conference, such as an Apache™ web server, a Nginx™ web server, and Microsoft-IIS™ web server, and/or the like. The web server  125  can use a variety of protocols, such as XML HTTP Requests, web sockets, Java Server pages, and/or the like. 
         [0020]    The above descriptions are where the mixer  121 , the conferencing application  122 , the cursor control module  123 , the conference profile  124 , and the web server  125  are part of a centralized communication system  120 . However, in another embodiment, these elements  121 - 125  may be in one or more of the communication endpoints  101 A- 101 N. For example, in a peer-to-peer environment that does not have a centralized communication system  120 . 
         [0021]      FIG. 2  is a diagram of a first view  200  of a multimedia conference that uses a single cursor  220 A. The first view  200  is a first example of a document  210  being displayed as part of a multimedia conference. In this example, the document  210  is an individual slide in a presentation, which is an overview of revenue for the company ABC in 2014. The first view  200  includes a single cursor  220 A. The single cursor  220 A is shown as an arrow shaped movable icon. In this embodiment, movement of the single cursor  220 A can be controlled by two or more users of the communication endpoints  101 A- 101 N who are in the multimedia conference. For example, a first user at the communication endpoint  101 A can control the single cursor  220 A for a first period of time in the multimedia conference. Based on a defined event, such as a second user starting to speak, the second user at the communication endpoint  101 B can control the single cursor  220 A for a second period of time (that follows or coincides with the first period of time) in the multimedia conference. 
         [0022]    The displayed icon for the single cursor  220 A may change during the multimedia conference. For example, if the single cursor  220 A can only be controlled by one user at a time, the single cursor  220 A may also include a name (not shown) for the user currently controlling the single cursor  220 A (similar to the cursors  220 B and  220 C in  FIG. 3 ). Alternatively, when the first user controls the single cursor  220 A, the single cursor  220 A looks as shown with the arrow icon. When the second user controls the single cursor  220 A, the shape of the single cursor  220 A can change to a different shaped icon, such as a diamond shaped icon or the arrow may change from an upward pointing arrow to a downward pointing arrow. 
         [0023]    In another embodiment, a second icon of the single cursor  220 A may be displayed when two users can control the single cursor  220 A at the same time. For example, the single cursor  220 A&#39;s icon may change to a square or a two-headed arrow when two users can control the single cursor  220 A at the same time. When a first user moves the single cursor  220 A, the first user&#39;s name may be displayed by the square single cursor  220 A. When the second user moves the single cursor  220 A, the second user&#39;s name may be displayed by the square single cursor  220 A. 
         [0024]      FIG. 3  is a diagram of a second view of a multimedia conference that uses two or more cursors  220 . The second view  300  is a second example of the document  210  being displayed as part of the multimedia conference. In this embodiment, two cursors  220 B and  220 C are provided as part of the multimedia conference. Both of the cursors  220 A,  220 B, and/or  220 C may correspond to annotators. An annotator is a cursor  220  can allows a user to annotate a view of the multimedia conference  200 / 300 . In  FIG. 3 , the cursors  220 B and  220 C have a circle for an icon. 
         [0025]    In addition, the second view  300  includes a multimedia conference control section  330 . The multimedia conference control section  330  includes an add cursor button  331 , a remove cursor button  332 , and a conference participants section  333 . The conference participants section  333  shows each of the participants in the multimedia conference. The multimedia conference has the participants John, Sally, Sue, Fred, and Jim. The black dot by John indicates that John is the currently speaking participant. 
         [0026]    The add cursor button  331  allows an individual user to add a cursor  220 . The remove cursor button  332  allows the individual user to remove a cursor  220 . For example, if the second view  300  was for the user John at the communication endpoint  101 A, John could add a cursor  220  that he controls by clicking the add cursor button  331 . John could remove the cursor  220  that he controls by clicking on the remove cursor button  332 . 
         [0027]    The adding and removing of a cursor  220  may be accomplished in various ways in addition to using the add cursor button  331  and the remove cursor button  332 . For example, a user or moderator could select a user name in the conference participants section  333  and drag the user name onto the document  210  to add a cursor  220  for control by the selected user. Alternatively, a user could select a menu to add and/or remove a cursor  220  for an individual user. 
         [0028]    In one embodiment, a moderator of the multimedia conference may control the cursors  220  for each participant in the multimedia conference. For example, if the moderator was John, John can add and/or remove cursors  220  for Sally, Sue, Fred, and/or Jim to use in the multimedia conference. 
         [0029]    In  FIG. 3 , the cursor  220 B is controlled by John (as indicated by John&#39;s name by the cursor  220 B). John has annotated the item “COMPUTER REVENUE FOR 2014=$1.3 BILLION” in the document  210  as indicated by annotation  221 B. The cursor  220 C is controlled by Sally (as indicated by Sally&#39;s name by the cursor  220 C). Sally has drawn an annotation  221 C for the item “TELEVISION REVENUE FOR 2014=$1.35 BILLION” that states that this is “Too Low.” Although  FIG. 3  only shows two cursors  220 B and  220 C, additional cursors  220  may be presented and controlled by other users in the multimedia conference. 
         [0030]    The icon for the cursors  220 B and  220 C may be displayed differently depending on implementation. For example, the icon for the cursor  220 B may be different from the icon  220 C based on defined user preferences. 
         [0031]    In addition, the functionality of the cursors  220 B and  220 C may be different. For example, the cursor  220 B may always be controlled by a single user (John), while the cursor  220 C may be controlled by multiple users. The cursor  220 C may be controlled by the user Sally during the first ten slides of a presentation and then controlled by a third user Fred for the last ten slides of the presentation. The control of the cursor  220 B and  220 C may be defined in the conference profile  124  or administered at the start of the multimedia conference. 
         [0032]      FIG. 4  is a flow diagram of a process for controlling one or multiple cursors  220  in a multimedia conference. Illustratively, the communication endpoints  101 A- 101 N, the communication system  120 , the mixer  121 , the conferencing application  122 , the cursor control module  123 , and the web server  125  are stored-program-controlled entities, such as a computer or processor, which performs the method of  FIGS. 4-5  and the processes described herein by executing program instructions stored in a tangible computer readable storage medium, such as a memory or disk. Although the methods described in  FIGS. 4-5  are shown in a specific order, one of skill in the art would recognize that the steps in  FIGS. 4-5  may be implemented in different orders and/or be implemented in a multi-threaded environment. Moreover, various steps may be omitted or added based on implementation. 
         [0033]    The process starts in step  400 . The mixer  121  establishes voice, video, and/or text portion of the multimedia conference between the communication endpoints  101 A- 101 N in step  402 . The multimedia portion of conference is established by the conferencing application  122 . The process of establishing the voice/video/text and multimedia can occur at the same time or separately. For example, the communication endpoints  101 A- 101 N can all go to a Universal Resource Identifier (URI) provided by the web server  125  to be connected to the multimedia conference (for both audio/video and multimedia). Alternatively, the communication endpoints  101 A- 101 N can make a voice call to a conferencing telephone number to have the mixer  121  establish a voice conference. The users, via the URI served by the web server  125  can then receive the multimedia portion of the conference via the web server  125 . 
         [0034]    The cursor control module  123  provides a first cursor  220  in the multimedia conference controlled by a first user at the communication endpoint  101 A in step  404  (e.g., as shown in  FIG. 2 ). The conferencing application  122  determines in step  406  if an event was detected. An event may correspond to one or a series of actions or occurrences associated with the multimedia conference. The event may include one or more events that are defined in the conference profile  124 . For example, the event can be based on detection of who is currently speaking in the multimedia conference. When a new user speaks in the multimedia conference, the conferencing application  122  will detect the event (via the mixer  121 ) in step  406 . 
         [0035]    The conferencing application  122  can detect who is currently speaking in various ways. For instance, the conferencing application  122  can analyze voice data received via Real Time Protocol (RTP) or via the Public Switched Telephone Network (PSTN) from a user to determine if the user is speaking over a defined threshold. 
         [0036]    The conference profile  124  may contain individual user profiles. The individual user profiles can be defined by individual users. For example, a user may be able to define different communication endpoints  101  that will be used during the multimedia conference, such as a PC being used for the multimedia portion/mouse control and a telephone being used for the voice portion of the multimedia conference. The conferencing application  122  can detect the user&#39;s voice level (e.g., via the PSTN) to provide cursor  220  control on a separate communication device (e.g., the user&#39;s PC) connected via a different network, such as the Internet. 
         [0037]    A period of time may be associated with an event. For example, the event may be to switch the cursor  220  to the currently speaking user. However, the cursor  220  may not switch for a defined number of seconds after a user speaks before switching control of the cursor  220  to a new user. 
         [0038]    The event can be that a user is speaking a defined percentage of time in the multimedia conference. For example, if the user at the communication endpoint  101 B is speaking 30% of the time in the multimedia conference, the user of communication endpoint  101 B will be provided a second cursor  220  (e.g., was shown in  FIG. 3 ). 
         [0039]    The event can be based on who is speaking the most in a defined period. For example, if John as spoken the most over the last five minutes, then John will control the cursor  220 . The event can be based on amplitude. For instance, the loudest (or quietest person) may control the cursor  220 . 
         [0040]    The event may be a voice command spoken by a user in the multimedia conference. For example, the user at the communication endpoint  101 B can say “add cursor” or “remove cursor” to have a second cursor  220  added or removed from the multimedia conference. 
         [0041]    The event may be a user entering one or more Dual Tone Multi-Frequency (DTMF) tones during the multimedia conference. For example, the user may enter *9 to add a cursor  220  controlled by the user to the multimedia conference or *8 to remove a cursor  220 . 
         [0042]    In one embodiment, the event may be based on an agenda for the multimedia conference. For example, the agenda may be for a slide presentation in the multimedia conference. Alternatively, the agenda may define a lecture period where only the moderator controls a first cursor  220  and group discussion of where the group can control the first cursor  220  and/or a second cursor  220 . 
         [0043]    Other events may include a gesture, such as a user raising their hand or shaking their head. The event may be that the user is looking at a specific area of a presented document, the user pointing to the specific area of the presented document  210 , a stress level or mood of the user, the presence of the user being on camera (e.g., being in view of the user&#39;s camera), a command from the user, a sound level of the user (e.g., how loudly or softly the user is speaking, and/or the like. 
         [0044]    Other examples of events that can control a cursor  220  can include a user walking out of a conference room or office, the user walking away from the multimedia conference, the user not paying attention to the multimedia conference, and/or the like. 
         [0045]    In one embodiment, the event may be based on monitoring brain activity of a user, such as a user with disabilities. Based on detection of the brain activity of the user, a cursor  220  can be added into the multimedia conference. All the above events can be associated with switching control of a cursor  220 , adding a cursor  220 , merging control of a cursor  220 , and/or removing a cursor  220 . 
         [0046]    If a defined event is not detected in step  406 , the process goes to step  408 . The conferencing application  122  determines in step  408  if the multimedia conference is over (e.g., if the moderator ends the multimedia conference). If the multimedia conference is over in step  408 , the process ends in step  410 . Otherwise, if the multimedia conference is not over in step  408 , the process goes to step  406 . 
         [0047]    If an event is detected in step  406 , the cursor module  123  determines if the event results in switching control of the cursor  220  to a different user. If the event results in switching control of the cursor  220  to the different user in step  412 , the cursor control module  123  switches control of the cursor  220  to the different user in step  414 . For example, the second user at the communication endpoint  101 B may speak a voice command, select a button, select a menu, or enter one or more DTMF tones to switch control of the cursor  220  to the second user. The process then goes back to step  406  to check for another event. 
         [0048]    If the event does not result in switching control of the cursor  220  to a different user in step  412 , the cursor control module  123  determines if the event results in providing a new cursor  220  in step  416 . If the event results in providing a new cursor  220  in step  416 , the cursor control module  123  provides the new cursor  220  in step  418  and the process goes to step  406 . The event that results in providing a new cursor  220  can be based on various criteria. For example, the cursor control module  123  can provide a new cursor  220  for a second user based on the second user speaking more loudly or becoming agitated. 
         [0049]    If the event does not result in providing a new cursor  220  in step  416 , the cursor control module  123  determines, in step  420 , if the event results in removing one or more cursors  220 . If the event does result in removing the one or more cursors  220 , in step  420 , the cursor control module  123  removes the one or more cursors  220  in step  422 . For example, the multimedia conference may be in a group discussion mode where each user in the multimedia conference can control an individual cursor  220 . Based on a command from the presenter, the cursor control module  123  can remove all the cursors  220  for the group in step  422  and only leave the cursor  220  of the moderator. The process then goes to step  406 . 
         [0050]    If the event does not result in removing the one or more cursors  220 , the cursor control module  123  manages the cursor  220  as defined in step  424  and then goes to step  406 . For example, the cursor control module  123  can merge control of inputs received from two cursors  220  into a single cursor  220 . To merge control, the cursor control module  123  may remove a second cursor  220  controlled by a second user and make the first cursor  220  controllable by the first and second users at the same time. Alternatively, the cursor control module  123  can add a second user to control a single cursor  220 . Other types of events may be to unmerge control of a curser  220  by presenting a new cursor  220  and changing control of the previously merged cursor  220 . 
         [0051]    Other events may be related to controlling the cursor  220  and/or managing usage of the cursor  220 . For example, an event may be a command to erase all annotations associated with one or more cursors  220 . For example, an individual user may send a command via DTMF or voice to erase all annotations made by the user. Alternatively, a moderator may send a command via DTMF or voice to erase all annotations made by all users in the multimedia conference. The event may result in highlighting a particular user&#39;s annotations or changing colors of annotations. Other events may be to change a cursor  220 , a color of a cursor  220 , and/or to animate a cursor  220 . Still over events may be to record the annotations and/or cursor  220  movements as part of the multimedia conference. When the annotations and/or cursor  220  movements are recorded, information regarding who is controlling the cursor can also be captured and made available during playback of the multimedia conference. 
         [0052]    The above descriptions describe the steps  412 - 424  as being implemented in series. However, in other embodiments, an event may results in two or more of the steps  414 ,  418 ,  422 , and  424  being implemented based on a single event. For example, an event in an agenda (e.g., going to group discussion mode) may cause a first cursor  220  controlled by a first user (a presenter) to switch to a second user (a moderator) and also cause one or more additional cursors  220  (cursors for individuals in the group) to be provided in the multimedia conference. 
         [0053]      FIG. 5  is a flow diagram of privately controlling a cursor  220  in a multimedia conference based on a sidebar communication. The process of  FIG. 5  goes between steps  406  and  408  of  FIG. 4 . If an event is not detected in step  406 , the conferencing application  122  checks, in step  502 , if a request to establish a sidebar communication has been received. A sidebar communication can be a separate communication that occurs during the multimedia conference between two or more users that are involved in the multimedia conference. For instance, a user at the communication endpoint  101 A may have an Instant Messaging sidebar communication session with a user at the communication endpoint  101 B during a multimedia conference between the users at the communication endpoints  101 A- 101 N. The sidebar communication may be a voice communication, a video communication, a text communication, and/or the like. 
         [0054]    If a request to establish a sidebar communication is not received in step  502 , the process goes to step  408 . Otherwise, if a request to establish a sidebar communication has been received in step  502 , the conferencing application  122  establishes the sidebar communication session in step  504 . The conferencing application  122  determines in step  506  if a request to setup private control of the cursor(s)  220  has been received. A request to setup private control of the cursor(s)  220  can be any type of request, such as a DTMF tone(s), a voice command, a command from a graphical user interface, and/or the like. For example, a user, via a graphical user interface, may indicate to provide private control when establishing the sidebar communication. Private control of a cursor  220  can be for an individual user or two or more users to control one or more cursors  220  where at least one other participant in the conference cannot see the cursor(s)  220  and/or annotations associated with the cursor(s)  220 . 
         [0055]    If a request to setup private control of the cursor(s)  220  has not been received in step  506 , the process goes to step  408 . Otherwise, if a request to setup private control of the cursor(s)  220  has been received in step  506 , the cursor control module  123  sets up private control of the cursor(s)  220  per defined rules (or user commands) in step  508 . The rules can be stored in the conference profile  124 . The rules can be based on a variety of conditions, such as, which users can see the cursor(s)  220  (i.e., only those in the sidebar communication), which users can annotate, if two or more users can control a single cursor  220 , if each user has their own cursor  220 , and/or the like. 
         [0056]    The private control of the cursor  220  ends in step  510  when the sidebar communication ends. Step  510 , in one embodiment, is a separate thread that is spun off and checks for the sidebar communication to end. Once the sidebar session ends in step  510 , the process goes to step  408 . 
         [0057]    In one embodiment, the private control of the sidebar communication may end before the sidebar communication ends. For example, the private control may be terminated by a user, even though the sidebar communication is still ongoing. 
         [0058]    In another embodiment, private control of one or more cursors  220  may exist while one or more other cursors  220  may not be under private control. For example, while one cursor is under private control, another cursor controlled by a different or the same user may be displayed to all users in the multimedia conference. 
         [0059]    Of course, various changes and modifications to the illustrative embodiment described above will be apparent to those skilled in the art. These changes and modifications can be made without departing from the spirit and the scope of the system and method and without diminishing its attendant advantages. The following claims specify the scope of the invention. Those skilled in the art will appreciate that the features described above can be combined in various ways to form multiple variations of the invention. As a result, the invention is not limited to the specific embodiments described above, but only by the following claims and their equivalents.