Abstract:
The integration of data and video conference data allows presentations to be given more effectively and people to write down things which are hard to deliver using speech alone, for example mathematical formulae. Often when giving a presentation though it can be advantageous to be able to interact not only with a person verbally but also on a written level, for example a user may wish to draw on slides being used in a presentation to illustrate a point. At present only one user can transmit this type of data to the participants at any one time and only to participants having an H.239 compatible endpoint. The presentation invention enables multiple users to interact using written rather than spoken means without having to pass permission to transmit written data from one participant to another.

Description:
CROSS-REFERENCE TO RELATED APPLICATIONS 
     Not Applicable 
     STATEMENT RE: FEDERALLY SPONSORED RESEARCH/DEVELOPMENT 
     Not Applicable 
     FIELD OF THE INVENTION 
     This invention relates to apparatus enabling participants in a video conference to interact using data such as text or pictures. The invention is applicable to use within a video conference in which, for example, a presentation or lesson is taking place. 
     BACKGROUND OF THE INVENTION 
     When using video conferencing it is useful to enable participants in the conference to transmit text or other data simultaneously with the transmission of the video (i.e. the audio and image components of the conference). This may be of use, for example, if the purpose of the conference is to give a presentation. In the case of a presentation the data could be a slide show or photographs. Alternatively, the data may be formulae in mathematical proofs or any other data which may need to be modified in real time as the conference or presentation progresses. 
     One method of transmitting data is to use two separate video channels connecting two endpoints either directly or indirectly through a multi conference unit (MCU). This may be achieved using the ITU-T H.239 format that was developed to enable the use of two video channels simultaneously within a single video conference. 
     The two video channels may be configured in order to transmit different data types. For example, a first video channel may be adapted to transmit video data which is which is advantageously displayed at a high frame rate in order that movements can be seen smoothly. In order to avoid using a large amount of network bandwidth, data sent using a high frame rate is usually sent at a low resolution. The second video channel may be adapted to transmit presentation data which, because it can be shown at a low frame rate without detracting from the appearance of the data, can be displayed at a high resolution so that all the detail of the presentation data can readily be seen. 
     Video data is defined for the purposes of this description as data having an audio component, such as the voice of a person, and an image component such as the view of the person. Presentation data is defined for the purposes of this description as being data which is preferably displayed using a low frame rate but high resolution. It may be, for example, text, one or more PowerPoint slides or pages from a pdf file. 
       FIG. 1  illustrates a prior art configuration of a video conference that is being transmitted using H.239. It can be seen that the video conference is taking place between two endpoints  10 ,  12  that are H.239 enabled i.e. are able to display data transmitted using H.239 and transmit data using H.239. 
     Each of the H.239 enabled endpoints  10 ,  12  has two channels  14 ,  16  along which data is transmitted to an MCU  18 . The two channels may both transmit video data or presentation data. Alternatively, one channel may transmit video data and the other presentation data. The MCU  18  will process the data as required and transmit the processed data to the relevant endpoint  10 ,  12 . Data may also be transmitted directly between the two H.239 enabled endpoints. 
     The user may view the data either on separate screens; one for presentation data and one for the video data, or, alternatively, the user may view the data as a composite image on a single screen of an H.239 enabled endpoint. H.239 can therefore be used in a video conference to enable data received from two inputs such as a presentation input for presentation data and a video input to be output onto one screen at the endpoint. 
     In a conventional H.239 conference a token is created which is passed between conference participants. Only the participant in the conference holding the token can transmit presentation data to all the other participants. This means that there is no way for the conference participants to interact directly with any presentation data displayed unless they are holding the token. This restricts the interaction of the participants. 
     BRIEF SUMMARY 
     In accordance with a first aspect of the present invention there is provided a Multi-Conference Unit (MCU) for use in transmitting a video conference to endpoints connected to the video conference, the MCU comprising a video input for receiving a video data stream including video data, a presentation input for receiving a presentation data stream including presentation data, processing means adapted to combine the data in the video and presentation data streams to produce a combined data stream and an output adapted to transmit the combined data stream from the MCU to an endpoint connected to the video conference. 
     The endpoint receiving the combined data stream may include two display devices, the first device displaying the video data and the second device displaying the presentation data. Alternatively, the endpoint receiving the combined data stream includes one display device, wherein the presentation data and video data is displayed upon the same display device. 
     The MCU may further include a second presentation input for receiving a presentation data stream including mark-up data from an endpoint remote from that transmitting the presentation data stream received by the first presentation input, the MCU also comprising a second processing means adapted to combine the presentation data received from the first endpoint and the second endpoint. 
     Presentation data in the presentation data stream may be received by the MCU in an H.239 format. Alternatively, the presentation data in the presentation data stream may be received by the MCU in a VNC format. 
     The combined data stream may be transmitted in a format selected from a group including H.323, H.320 and SIP. 
     In accordance with a second aspect of the present invention there is provided a method for integrating data from two data channels during a video conference, the method comprising the steps of receiving a video data stream from a video input the video data stream including video data, receiving a presentation data stream from a presentation input the presentation data stream including presentation data, combining the data in the video and presentation data streams to produce a combined data stream and transmitting the combined data stream to an endpoint connected to the video conference. 
     The step of transcoding the presentation data stream from a first format into a second format may take place prior to combining the video and presentation data streams. 
     The combined data stream may be transmitted in one of the group of formats comprising H.323, H.320 and SIP. 
     In accordance with a third aspect of the present invention there is provided a non-transitory computer readable storage medium encoded with instructions that are executable on an MCU to cause the MCU to combine a video data stream and a presentation data stream to produce a combined data stream and transmit the combined data stream to an endpoint connected to the video conference wherein the video data stream includes video data and the presentation data stream includes presentation data. 
     In accordance with a fourth aspect of the present invention there is provided a multi-conference unit (MCU) for use in transmitting a video conference to endpoints connected to the video conference, the MCU comprising an input adapted to receive a presentation data stream using a first protocol from an endpoint, the presentation data stream including presentation data and being in a first format, encoding means adapted to convert the presentation data from a first format to a second format, processing means adapted to encapsulate the presentation data using a transmission protocol and an output adapted to transmit the converted presentation data using the transmission protocol from the MCU to an endpoint connected to the video conference. 
     The presentation data stream may be received from a VNC server. Additionally, the presentation data in the converted data stream may comprise a series of JPEG images. The transmission protocol may be H.239. 
     The MCU may further include a second input adapted to receive a video data stream and a second output adapted to transmit a video data stream. 
     The outputs may transmit the converted presentation data stream and video data stream to an endpoint including two display devices, the first display device displaying the presentation data in the converted data stream and the second display device displaying the video data in the video data stream. 
     The MCU preferably, further comprises a second input for receiving a presentation data stream including mark-up data from a second endpoint and a second processing means adapted to combine the presentation data received from the first endpoint and the second endpoint. 
     In accordance with a fifth aspect of the present invention there is provided a method for processing presentation data during a video conference the method comprising the steps of receiving a presentation data stream from a first endpoint, the presentation data stream including presentation data and being in a first format, converting the presentation data from the first format to a second format, encoding means adapted to encapsulate the presentation data using a transmission protocol and transmitting the converted presentation data using the transmission protocol from the MCU to an endpoint connected to the video conference. 
     A second data stream including mark-up data may also be received from a second endpoint and the MCU may combine the presentation data received from the first endpoint and the second endpoint. 
     The presentation data in the converted data stream may comprise a series of JPEG images. 
     In accordance with a sixth aspect of the present invention there is provided a non-transitory computer readable storage medium encoded with instructions that are executable on an MCU and adapted to cause an MCU to, on receipt of a presentation data stream in a first format, the presentation data stream including presentation data, convert the presentation data from the first format to a second format and encapsulate the presentation data using a transmission protocol and transmit a converted data stream comprising presentation data in the second format from the MCU to an endpoint connected to the video conference using the transmission protocol. 
     In accordance with a seventh aspect of the present invention there is provided a personal computer adapted to participate in a video conference comprising a first input adapted to receive video data, a presentation input adapted to receive presentation data, processing means adapted to detect mark-up data entered by a user and an output adapted to transmit the mark-up data to a multi conference unit. 
    
    
     
       BRIEF DESCRIPTION OF THE DRAWINGS 
       Embodiments of the invention will now be described by way of example and with reference to the following drawings in which: 
         FIG. 1  illustrates a prior art network in which two H.239 enabled endpoints are connected to a video conference; 
         FIG. 2  illustrates inputs and outputs with which the present invention may be implemented; 
         FIG. 3  illustrates an embodiment of the present invention in which two H.239 enabled endpoints and a non-H.239 enabled endpoint are connected to a video conference; 
         FIG. 4  illustrates an embodiment of the present invention in which two H.239 capable endpoints and a personal computer are connected to a video conference; and 
         FIG. 5  illustrates an embodiment of the present invention in which a personal computer is connected to an MCU using a VNC server. 
     
    
    
     DETAILED DESCRIPTION 
     For the purposes of the present description a video data stream is defined as a data stream which includes video data which is advantageously displayed at a high frame rate in order that movements can be seen smoothly. A presentation data stream is defined as a data stream which includes presentation data, such as a slide show. 
     Furthermore, for the purposes of the present description a combined data stream is defined as a data stream which incorporates data which has been produced from the merging of data received from two or more separate endpoints. The data to be merged may include mark-up which is defined, for the purpose of this description, as an input made by a user with reference to presentation data displayed to the user at an endpoint. The mark-up may, for example, be text typed onto a computer or hand-written marks made on a touch sensitive screen. 
     A converted data stream is defined as a data stream which has been converted from one data format to a second data format by the MCU before it is transmitted to an endpoint. An example of a data format is JPEG which is used for some presentation data. Formatted data is transmitted to an endpoint using a transmission protocol, such as SIP, H.323 or H.239. 
     Transmitting Data Received from Two Video Channels to a Conventional Video Endpoint 
       FIGS. 2 and 3  illustrate a video conference in which one embodiment of the present invention may be implemented. Multiple endpoints  10 ,  12 ,  19 ,  20  and  21  are connected to an MCU  18 . Endpoints  10  and  12  are H.239 enabled endpoints that may transmit data to other endpoints in the video conference using two channels  14 ,  16 . Endpoints  10  and  12  may comprise two screens, as illustrated in  FIG. 2 , where one screen is used to display video data (v) and the other screen displays presentation data (p) or, alternatively, the endpoints may have one display screen displaying both video and presentation data. 
     At least one endpoint  19 ,  20  or  21  participating in the video conference is not able to process video and presentation data for the same video conference that is received simultaneously from two video channels. The endpoint  19 ,  20  or  21  may be, for example, an H.323 endpoint, an H.320 endpoint, a SIP endpoint or any other endpoint that is suitable for participating in video conferences. 
     The endpoint may have two display devices each of which may display a different output, as illustrated by endpoints  19  and  20  in  FIG. 2 . In this instance endpoint  19  displays video data received from endpoints  10  and  12 , endpoint  20  displays the video and presentation data received from endpoint  10 . The endpoint may, on the other hand, only have one display device which displays both the presentation data and video data produced by the MCU as shown in the illustration of endpoint  21  in  FIG. 2 . 
     The endpoints are connected to a multi conference unit (MCU)  18  that receives, processes and transmit data received from any endpoint participating in a video conference. By transcoding, the endpoint is able to produce the detailed functionality described below. 
     Returning to  FIG. 3 , the users at the endpoints  10 ,  12 ,  20  connect to a video conference using any suitable means. For example, the user may dial into the MCU  18  and connect directly to a video conference. Dialing a number specifically assigned to the conference allows the user to connect to a specific video conference. Alternatively the user may connect to the MCU and then select the desired conference from a list displayed at their endpoint as described in U.S. patent application Ser. No. 10/962,912. 
     The MCU  18  determines the type of endpoint and its capabilities, for example, whether it can process two video channels simultaneously transmitted to it within a single video conference. Therefore, in the present example, the MCU  18  determines when the endpoints  10 ,  12  connect to it and that both endpoints  10  and  12  are able to transmit, receive and display data transmitted through two data channels. The MCU  18  also determines that endpoint  20  is not able to transmit, receive and display data transmitted through two data channels. 
     Once connected to a conference the user becomes a conference participant and is able to view data being transmitted by the other participant&#39;s endpoints  10 ,  12 ,  20  on the screen of their endpoint  10 ,  12 ,  20 . In the network of the present example H.239 enabled endpoint is the endpoint transmitting data to the MCU  18  through two video channels, for example it is transmitting video data through channel  14  and presentation data, such as power point presentation data, through channel  16 . The H.239 enabled endpoint  12  is only transmitting video data to the MCU  18  and the non-H.239 enabled endpoint  20  is transmitting video data to the MCU  18  using channel  22 . 
     The MCU  18  on receiving all the data transmitted by the endpoints  10 ,  12 ,  20  processes the data to combine all the video data received from the endpoints  10 ,  12 ,  20 . The MCU  18  has determined that H.239 enabled endpoint  21  can receive the data from the video conference using two channels. Therefore, the MCU transmits the combined video data to endpoint  12  through channel  14  and transmits the presentation data received from endpoint  10  to endpoint  12  using a separate channel  16 . 
     The MCU  18  has also determined that endpoint  20  cannot receive data transmitted through two data channels. Therefore, the MCU  18  decodes the data transmitted from all the endpoints and, in addition to combining the video data received from all the endpoints the MCU also combines the presentation data. The data is combined in such a way that the combined portions of data are displayed on different sections of the endpoint  20 &#39;s display. If desired the user may select to replace the image of a user at one of the endpoints with the image of the presentation data. 
     The MCU  18  then transcodes the combined data into a format which can be readily viewed at endpoint  20 . The data is then transmitted to the endpoint  20  through data channel  24  using any suitable protocol such as H.323, H.320 or SIP. The protocol may be selected according to the type of endpoint that MCU  18  determined that it is transmitting data to. 
     Transmitting Data Received from Two Video Channels to a Personal Computer 
       FIG. 4  illustrates a video conference in which a second embodiment of the present invention may be implemented. Multiple endpoints  10 ,  12 ,  26  are connected to an MCU  18 . Endpoints  10  and  12  are H.239 enabled endpoints that may transmit data to other endpoints in the video conference using two channels  14 ,  16 . At least one endpoint  26  participating in the video conference is a personal computer. 
     The personal computer is provided with at least one video endpoint application that enables the computer to process data that the personal computer  26  receives from the MCU  18 . The application may, for example, be a Java Applet or any other suitable application. 
     The video endpoint application may be present on the personal computer  26  before the MCU  18  contacts it. Alternatively, a user may manually download the video endpoint application to the computer, for example, from a website. Optionally, the MCU  18  may be provided with means for detecting the presence of the video endpoint application on the personal computer  26 , and causing the application to be updated or installed as required. The final option is described in the co-pending US Application entitled “Method for Adapting a Device to Participate in Video Conference Calls” the disclosure of which is herein incorporated by reference. 
     When the personal computer  26  connects to the MCU in order to connect to a video conference or to indicate availability for participation in video conferences the MCU determines the video endpoint applications present on the personal computer  26 . This may be done by transmitting a request message requesting details of video endpoint applications on the personal computer to the personal computer  26 . Alternatively, details of any changes to any video endpoint applications that have occurred since the personal computer last connected to an MCU  18 . As discussed above, the MCU  18  may, if required, cause one or more video endpoint applications to be installed on the personal computer  26 . 
     After the MCU  18  has determined whether a video endpoint application is present on the personal computer  26  it may then select a suitable codec for converting the data format into a data format that the personal computer  26  can process. It is preferable that the combined video data is sent using a separate channel,  28 , to the channel used to send presentation data, in this instance channel  30 . 
     If, for example, the personal computer has a Java applet viewer which is able to decode and process JPEG images then the MCU  18  may convert presentation data received from H.239 enabled endpoint  10  from the format to a series of JPEG images. The JPEG images are then transmitted to the personal computer  26  which processes the JPEGs using the Java applet and displays the JPEGs at the endpoint. 
     This invention is not limited to the MCU producing ‘JPEG’ images typically encoded according to the ISO/IEC IS 10918-1 or ITU-T Recommendation T.81 standard. Rather, for the purposes of this specification, the term JPEG may be taken to include, in addition to JPEG images, JPEG-like images or any other suitable compressed or uncompressed data format which may be processed by an applet and displayed by the personal computer  26 . Similarly, the video endpoint application is not limited to a Java applet but may be any suitable application that when executed by the personal computer  26  enables the personal endpoint to display images received from an MCU  18 . 
     The video data that has been combined into composite video data may be converted into a suitable format separately to the presentation data. For example, if a separate video endpoint application, for example a streaming application such as Quicktime is running to process the video data received by the personal computer  26 . The personal computer  26  may have the video endpoint applications for displaying the presentation data and video data set up so that the presentation and video data are both displayed on the same display of the personal computer  26 . 
     If desired, the video and presentation data may be transmitted to two separate display devices as illustrated in  FIG. 2  where the video output is transmitted to video endpoint  25  and the presentation data is transmitted from display on personal computer  26 . The video and presentation data may also be transmitted to two separate personal computers associated with each other such that the video data is shown to the user on one display and the presentation data is shown to the user on the other display. 
     Alternatively, the video and presentation data may be combined and transmitted to the personal computer using a single channel. In this case a single video endpoint application at the personal computer may decode and process the data received by the personal computer from the MCU. 
     Transmitting Presentation Data from a Personal Computer Connected to a Video Conference. 
     As discussed previously the conventional H.239 protocol only allows transmission of presentation data from one presentation data source. In the present invention the use of a video endpoint application at the personal computer may be used to allow any participant in a video conference to alter the appearance of the presentation data. 
     When a user using a personal computer  26  annotates the presentation data, for example by scribbling notes on the presentation data or typing or drawing on the screen of their endpoint (preferably a user is allowed to draw on the screen using a mouse or any other suitable method), this is registered by a video endpoint application on their endpoint. The user annotations are preferably made on the presentation data image such as the slides or text. 
     The personal computer encodes the annotations the user has entered into the personal computer and transmits the presentation data to the MCU. The presentation data received by the MCU from the personal computer is combined with presentation data received from the H.239 enabled endpoint. In this way the presentation data transmitted to endpoints connected to the conference is made up of presentation data from two or more endpoints. If more than one personal computer is connected to the video conference then presentation data may be received by the MCU from one or more of the personal computers connected to the video conference. 
     The personal computers may be considered as having a bidirectional virtual data connection with the MCU. This is because the personal computers are able to transmit presentation information to the MCU rather than just receiving presentation information from the MCU. 
     Alternatively, the presentation data may originate from a virtual network computer (VNC) server  32  as shown in  FIGS. 2 and 5 . The VNC server  32  creates a low bandwidth stream of the display of a PC desktop, and therefore can be used to transmit presentation data using a VNC format to the MCU through a VNC connection  34 . By using the VNC format to transmit the presentation data, the frame rate for the data is maintained without losing any data resolution. 
     The MCU may then transcode the data from the VNC format to the format in which the data will be transmitted to other endpoints in the video conference. As discussed above with reference to H.239 non-enabled endpoints the data transmitted by the MCU  18  may be displayed on a single display device  33 , as shown in  FIG. 2 . Alternatively, the endpoint  35  may have two display devices, one for displaying video data transmitted by the MCU  18  and the other for displaying presentation data transmitted by the MCU  18 . 
     If other endpoints are also transmitting presentation data to the MCU then the transcoded data that has been received from the VNC is combined with the presentation data received from the other endpoints and then transmitted by the MCU.