Patent Publication Number: US-8121277-B2

Title: Catch-up playback in a conferencing system

Description:
TECHNICAL FIELD 
     The present invention relates generally to the fields of telephony and conferencing systems. 
     BACKGROUND 
     A participant to a conference held over a network may be interrupted, i.e., pulled out of real-time listening or participation, for a variety of reasons. For example, the participant may need to step away from the conference to answer another call, or briefly discuss a matter of immediate urgency with a co-worker. Alternatively, the participant may invoke some feature of the conference service that results in the conference audio being interrupted, such as playing out the roster, receiving whisper information, moving to a sidebar conversation, etc. Upon returning to the live conference the participant usually wants to listen to what transpired in the conference during the time he was gone. Simply listening to a recording of the missed portions of the conference, however, is inadequate insomuch as the participant is permanently shifted away from real-time participation. 
     A number of prior art conferencing systems include a feature that allows portions of a conference session to be recorded and then played back at an accelerated rate, in what is commonly referred to as “catch-up” mode. The problem with these systems is that the participant frequently forgets that he is listening to the conference in catch-up mode, and attempts to speak as a participant in the conference. Comments made during catch-up mode, however, cannot be mixed into the real-time (“live”) conference because the live conference has already progressed beyond the point in time of the catch-up mode playback. 
    
    
     
       BRIEF DESCRIPTION OF THE DRAWINGS 
       The present invention will be understood more fully from the detailed description that follows and from the accompanying drawings, which however, should not be taken to limit the invention to the specific embodiments shown, but are for explanation and understanding only. 
         FIG. 1  illustrates an example conference held over a network. 
         FIG. 2  illustrates an example of endpoint catch-up in the conferencing system shown in  FIG. 1 . 
         FIG. 3  illustrates an example method of handling speech during catch-up mode. 
         FIG. 4  illustrates an example method of handling speech during a moderated conference. 
         FIG. 5  illustrates an example method for triggering a comment for real-time playback in an open floor conference. 
         FIG. 6  illustrates an example method for triggering a comment for real-time playback in a moderated conference. 
         FIG. 7  illustrates an example method for non-real-time playback of a comment in the conferencing system of  FIG. 1 . 
         FIG. 8  illustrates an example method for playback of context when a comment is triggered in the conferencing system of  FIG. 1 . 
     
    
    
     DESCRIPTION OF EXAMPLE EMBODIMENTS 
     In the following description specific details are set forth, such as device types, system configurations, protocols, methods, etc., in order to provide a thorough understanding of the present invention. However, persons having ordinary skill in the relevant arts will appreciate that these specific details may not be needed to practice the present invention. 
       FIG. 1  illustrates an example illustrates an example conference held over a network. This lecture-hall type of conference (also known as a moderated or floor-controlled conference) is managed and handled by a conferencing system  19  (e.g., a server) coupled via an Internet protocol (IP) network  20  to the various meeting participant endpoint devices. In this example, communications between remote participants and the participants in lecture hall  12  are facilitated by video conference endpoint device  14 , which is shown including a loudspeaker  20 , microphone  17  and video camera  15 . (It is appreciated that multiple microphones and cameras may be used.) Remote participants  18  and  22  are shown attending the conference via respective endpoint devices  16  and  22 , each of which comprises a personal computer (PC) with built-in softphone capabilities and video conference endpoint device  14 , which is shown including a microphone  17  and video camera  15 . Using PCs  16  and  22 , respective participants  18  and  21  may each communicate with the other conference participants (i.e., members of the audience  13 A-D) and moderator/lecturer  11 . Although not explicitly shown in this example, endpoint device  14  may include a video display and PCs  16  &amp;  22  may include a camera for transmitting images of participants  18  and  21  to the participants in lecture hall  12 . 
     In the context of the present application, endpoint devices represent an end user, client, or person who wishes to initiate or participate in a conference session and via conferencing system  19  via IP network  20 . Other endpoint devices not specifically shown in  FIG. 1  that may be used to initiate or participate in a conference session include a personal digital assistant (PDA), a laptop or notebook computer, an IP telephone device, a non-IP telephone device, an audio/video appliance, a streaming client, a television device, programmable consumer electronics, or any other device, component, element, or object capable of initiating or participating in voice exchanges with conferencing system  19 . 
     In the example of  FIG. 1 , video conference data (e.g., audio and video data) is communicated to endpoint device  16  via IP network  20 . However, it should be understood that in different specific implementations the media path for the conference participants may include audio/video transmissions, e.g., Real-Time Transport Protocol (RTP) packets sent across a variety of different networks (e.g., Internet, intranet, PSTN, etc.), protocols (e.g., IP, Asynchronous Transfer Mode (ATM), Point-to-Point Protocol (PPP)), with connections that span across multiple services, systems, and devices. Alternative embodiments of the present invention may be implemented in software or hardware (firmware) installed in an IP communication systems, PBX, telephony, telephone, and other telecommunications systems. Similarly, the signaling path utilized for transmission of commands, messages, and signals for establishing, moderating, managing and controlling the conference session may be across any network resources. 
     Conferencing system  19  comprises a server with a processor subsystem for running a conferencing program including software (or firmware) plug-ins, modules, or enhancements that implement the various features and functions described herein. In a specific implementation, for example, conferencing system  19  may run a modified or enhanced IP communication system software product such as Cisco&#39;s MeetingPlace™ conferencing application that allows users to schedule and attend meeting conferences. In the embodiment shown, conferencing system  19  may also include a digital signal processor (DSP) or firmware/software-based system that mixes and/or switches audio/video signals received at its input ports under the control of the server. The audio signals received at the conference server ports originate from each of the conference or meeting participant endpoints (e.g., endpoint devices  16 ,  22  and  14 ), and possibly from an interactive voice response (IVR) system (not shown). In certain embodiments, conference server may also incorporate or be associated with a natural language automatic speech recognition (ASR) module for interpreting and parsing speech of the participants, and standard speech-to-text (STT) and text-to-speech (TTS) converter modules. 
     Practitioners in the art will appreciate that the actual media paths are established by conferencing system  19 . In other words, conferencing system  19  handles all of the control plane functions of the conference session, and is responsible for engaging the necessary media components/resources to satisfy the media requirements of each of the endpoints (i.e., endpoint  16 ) participating in a conference session. In operation, endpoint devices  14 ,  16  and  22  shown in  FIG. 1  may join the audio/video conference session by calling into a conferencing application running on the conferencing server. 
       FIG. 2  illustrates an example of endpoint catch-up in the conferencing system shown in  FIG. 1 . In the example shown, participant  18  associated with endpoint device (e.g., PC)  16  has stepped away from the conference session for a period of time and needs to catch up to the real-time conference that continues to proceed. This is achieved through conference control unit  23 , conference mixer  25 , and recording and playback server  27  components of system  19 . In operation, conference control  23  is a logical unit that provides media control (shown by path  40 ) of conference mixer  25  and recording and playback server  27 . Conference control unit  23  also performs all of the control plane functions involving endpoints  14 ,  16  and  22  via call signaling connections  33 ,  35 , and  37 , respectively. For instance, conference control unit  23  establishes communication channels for data packets transmitted between endpoints  16 ,  14  and  22 . 
     Conference mixer  25  mixes or switches the data (audio, video, and/or web collaboration) packet streams and also sends real-time media for recording (via path  45 ) to server  27 . As will be described in more detail below, server  27  may also transmit media playback of triggered comments to mixer  25  on request via connection path  46 . As participants talk during a conference, real-time media data transmissions occur via paths  39 ,  41  and  43  between conference mixer  25  and respective endpoints  14 ,  22  and  16 . Conference mixer  25  mixes the real-time media according to control signals/messages provided by conference control unit  23 . For instance, conference mixer  25  may be directed to apply any one of a wide variety of audio/video mixing or switching algorithms to produce an output media stream delivered to the endpoints devices. 
     Note that in certain embodiments, conference mixer  25  may filter out annoyances by dynamically suppressing audio streams that meet predefined annoyance criteria (e.g., white noise, pink noise, snoring, etc.). It is appreciated that for non-verbal audience reaction, the audio mix can be of relatively low-fidelity. However, for applications where verbal reaction is not to be filtered out as an annoyance, the verbal audience reaction should be of relatively high-fidelity. 
     Participant  18  may drop out of the real-time conference session by entering a command (e.g., pause button) on endpoint  16 . In response, recording and playback server  27  begins recording real-time media (via path  45 ) from the conference session. Assume, for example, that during the session participant  18  may have a question or a comment for moderator/lecturer  11 , with the moderator requesting that all questions be held until the end of the lecture. Participant  18  may click on an icon button of a user interface running on endpoint  16  to signal that he is dropping out of real-time listening mode to record his question/comment. Concurrently, server  27  begins recording the real-time media occurring in lecture hall  12  while endpoint  16  records his question/comment. That is, recording and playback server  27  may also record the question/comment received from endpoint  16  simultaneous with the recording of the real-time media. 
     Alternatively, detection of speech by the conferencing system may cause the participant to automatically drop out of real-time listening mode and into recording mode. That is, audible speech of a participant who otherwise would not be able to participate directly in the conference may trigger recording of that speech. When the speech stops for a predetermined period (e.g., 5 seconds of silence), the system may automatically stop the recording and enter catch-up mode for that participant. 
     In a specific implementation, recording and playback server  27  may also automatically record a predetermined time period (e.g., 5-10 seconds) of the conference session just prior to the point in time when endpoint  16  dropped out of the conference and attach it to his recorded question/comment. This additional recording provides a context for the recorded question/comment. The recorded speech and the recorded real-time media may be stored in a multimedia storage unit  50 . The combination of the context media and the question/comment may be placed in a moderated question and answer queue, allowing the lecturer to subsequently manage or trigger individual questions or comments. 
     When participant  18  has finished recording his question/comment (indicated by an appropriate command entered on endpoint  16 ), server  27  starts transmitting playback of the catch-up media recording (shown by path  52 ) to endpoint  16 . The catch-up media is played back at a faster rate (e.g., 1.5×-2× with pauses and silences removed or reduced) than real-time until endpoint  16  catches up to the real-time conference session. Later, after the lecture portion of the conference has ended and the moderator is ready to respond to questions/comments, server  27  retrieves the triggered questions/comments from multimedia storage  50 . The questions/comments are played back to conference mixer  25  and transmitted to each of endpoints  16 ,  14  and  22 . Note that the additional 5-10 seconds of recorded media preceding the question/comment may help the listeners to understand and identify the context of the question/comment. 
       FIG. 3  illustrates an example method of handling speech during catch-up mode. When speech is detected by the system in catch-up mode—such as when the listener interjects a question or comment—the conferencing system may first query whether the recording feature is disabled (block  60 ). If it is, then when speech is detected (block  61 ) the system may play a warning to the user (block  62 ), after which time playback of catch-up media resumes (block  63 ). The warning may be an audible tone or an IVR notice stating that recording of speech is currently disabled. On the other hand, if the recording feature is enabled, then playback of catch-up media is temporarily halted (block  64 ). The participant&#39;s comments are recorded (block  65 ) and a temporal mark is stored along with the recorded comments (block  66 ) indicating the context or point in the conference session where the participant&#39;s question/comment arose. The mark may be a simple indicator stating the comment occurred at “time 24:47” of the conference. 
     After the question/comment has been recorded, the system may optionally prompt the participant as to whether he wishes to edit the recording (block  67 ). If so, then the participant may edit his comments by recording over his previous comments (block  68 ). Other optional prompts may include the system asking whether the participant wants to attach any documents such as charts, diagrams, pictures, etc., (blocks  69  &amp;  70 ); or whether the participant wants to adjust the marked context point (blocks  71  &amp;  72 ), e.g., by changing the time mark when the comment is triggered in the conference session (for instance, the participant may indicate that he wants twenty seconds of the recorded real-time media played back before his comment). 
     The system may also give the participant the option of submitting/storing his question/comment (blocks  73  &amp;  75 ). If the participant declines (e.g., he reconsiders and determines that his comment is unnecessary or does not significantly add to the discussion), the system discards the recorded comment and resumes catch-up playback (block  74 ). In the case of a submitted question/comment, the system may further query whether the conference is a moderated conference (block  76 ). For moderated conferences, the comment may be entered into the floor-control queue (block  77 ); that is, when the floor is opened for questions and/or comments, the comment will be heard by the participants in on ordered manner. Once the comment has been submitted, the system may then resume catch-up playback ( 78 ). If the conference is not moderated, the comment may be stored for subsequent triggering (block  79 ) after which time the system resumes catch-up playback ( 80 ). 
       FIG. 4  illustrates an example method of handling speech during a moderated conference. As in the example of  FIG. 3 , when speech is detected the system may first query whether the recording feature is disabled (block  81 ). If so, and speech is detected (block  82 ) a warning is played to the user (block  83 ) and the system returns to real-time (block  84 ). If the recording feature is enabled, the system starts recording the real-time conference session for subsequent catch-up playback (block  85 ). The participant&#39;s comment/question is recorded (block  86 ) and a mark is stored indicating the context of the comment/question (block  87 ). In a manner similar to the example of  FIG. 3 , the system may then optionally prompt the participant to edit the comment (blocks  88  &amp;  89 ); attach documents (blocks  90  &amp;  91 ); attach or adjust the context (blocks  92  &amp;  93 ); and then either submit or discard the comment before beginning the catch-up process (blocks  94 - 99 ), i.e., playback of the portion of the conference that occurred while the comment/question was being made or asked. 
       FIG. 5  illustrates an example method for triggering a comment for real-time playback in an open floor conference. When a comment is ready for real-time playback in an open floor conference (block  101 ) a variety of triggering mechanisms may be used. The comment may be triggered for playback either by the participant who made the comment, e.g., using a command entered via graphical user interface (GUI) or voice user interface (VUI) once the participant has finished catching up (block  102 ). For example, the participant may have access to a list of comments that he has made (e.g., on a webpage or computer screen), and when there is a pause in the live conference the participant may manually trigger real-time playback of the comment through the GUI or VUI. Alternatively, the comment may be triggered by another participant using the GUI or VUI (block  103 ). The other participant may see a queue of comments on a computer screen, for instance, and click on a comment to trigger real-time playback of the comment. Yet another alternative is to have the comment auto-triggered by the conferencing system (block  104 ) when the participant has finished catching up and a period of silence has been detected. The triggered comment is then played back in real-time (block  105 ). 
       FIG. 6  illustrates an example method for triggering a comment for real-time playback in a moderated conference. In this scenario, the floor is opened for questions at the end of a floor-controlled conference or right before a change of agenda, etc. (block  106 ). A floor-control queue may be utilized to accumulate questions/comments generated by the participants throughout the conference session. The floor-control queue is then accessed or serviced at an appropriate time following the main presentation or discussion (block  107 ). When a comment makes it to the top of the queue, its playback is triggered (block  108 ). Alternatively, playback of a comment may also be triggered by a variety of ordinary floor-control algorithms. For example, comments do not necessarily have to be played back in order generated. In this example, when playback of a comment/question is triggered, the floor is granted to the participant who made the comment or asked the question (block  109 ). This gives that participant the ability to explain or further amplify on his comment if he wishes. 
       FIG. 7  illustrates an example method for non-real-time playback of an inserted comment when an off-line listener is playing back an entire conference recorded by the conferencing system of  FIG. 1 . The process in this example begins with a listener (or viewer) receiving a non-real-time playback of the conference session, i.e., a recording (block  110 ). As the playback progresses, the recording eventually reaches a point where a comment was previously inserted by a participant to the meeting (block  111 ). The comment need not have been played out in real-time—only that it has become part of the conference session recording. At the point where the comment is inserted, the system may first play a prompt to the listener indicating that the comment which follows was inserted by a non-real-time conference participant (block  112 ). For example, an IVR may state, “The following comment was made by a participant attending the conference session but was not part of the real time conversation.” Once the prompt has been played, the recorded comment is played (block  113 ). After playback of the comment, the non-real-time conference session may resume (block  114 ). 
       FIG. 8  illustrates an example method for playback of context when a comment is triggered in the conferencing system of  FIG. 1 . This example begins with the playing of a prompt or IVR script notifying the listener that a comment is about to be played back (block  115 ). Prior to playback of the comment itself, a predetermined amount of context is played ahead of where the comment occurred in the course of the conference session (block  116 ). For example, the context may comprise a preceding portion (e.g., 15 seconds) of the audio recording of the conference session just prior to the point where the comment occurred. In other cases, the context may comprise video, agenda, or web sharing context. The context playback may also comprise multiple different types of media, e.g., audio/video playback. 
     In a specific implementation, when the context is triggered to pop-up on a participant&#39;s web browser, the agenda context may appear (e.g., “the context for this comment is Agendum  53 ,” etc.). If the context involves web sharing, a slide that was used when the comment was made may automatically pop-up on the participant&#39;s computer screen. Additionally, an excerpted portion of the audio playback preceding the comment may accompany the information presented graphically on the user&#39;s PC monitor screen. After the full context has been played out, the actual comment is then played (block  117 ). 
     In certain embodiments, the system may be configured to allow user-controlled marking of context preceding a comment or question. This feature may apply to playback of recorded conferences, playback of catch-up media while the live, moderated conference session continues (i.e., in real-time), or during a real-time moderated conference. A variety of different options may be provided. For instance, in one embodiment a participant might hear something in a real-time conference session, or playback of a recorded session, and want to set a mark at that point using a GUI or VUI. In a specific example, a participant who has already recorded a comment/question may desire to mark a point in the session preceding where his comment/question occurred. In other words, instead of using a default context setting (e.g., 10 seconds of playback preceding the comment), the system may allow the commentator to determine the appropriate context and set the context accordingly. For instance, the commentator may determine that 25 seconds of preceding recording is needed to fully understand his comment. In other cases, the commentator may decide to attach a report, graph, or web link instead of, or in addition to, a selected number of seconds of contextual playback. 
     User-controlled marking of context may also be provided in the conferencing system of  FIG. 1 . User-controlled marking may be utilized in a situation where a user is listening to playback of catch-up media while the live, moderated conference session continues (i.e., in real-time). The user may want to control marking of the context preceding a question and/or comment that he has already recorded in the system. In order to do so, the user transitions from real-time to playback mode in the conferencing system. The user may then record his comment and precisely mark the context point preceding the comment using a GUI or VUI. For example, the user may listen to the context preceding his comment and then mark the context point to be where it makes the most sense. Alternatively, the user may simply choose a specified amount of time (e.g., 20 seconds) preceding the comment. 
     In another example of participant-controlled marking, the user may insert provisional marks while remaining a real time participant in the conference. For example, the participant may be listening to a lecture and begin to hear something that he doesn&#39;t understand. If misunderstood point is subsequently clarified, the participant can cancel the provisional mark. However, if the participant then leaves a comment or question, the provisional mark can act as the actual beginning of the context for the question. 
     It should be understood that elements of the present invention may also be provided as a computer program product which may include a machine-readable medium having stored thereon instructions which may be used to program a computer (e.g., a processor or other electronic device) to perform a sequence of operations. Alternatively, the operations may be performed by a combination of hardware and software. The machine-readable medium may include, but is not limited to, floppy diskettes, optical disks, CD-ROMs, and magneto-optical disks, ROMs, RAMs, EPROMs, EEPROMs, magnet or optical cards, or other type of machine-readable medium suitable for storing electronic instructions. 
     Additionally, although the present invention has been described in conjunction with specific embodiments, numerous modifications and alterations are well within the scope of the present invention. Accordingly, the specification and drawings are to be regarded in an illustrative rather than a restrictive sense.