Patent Publication Number: US-2004045036-A1

Title: Delivery system and method of real-time multimedia streams

Description:
BACKGROUND OF THE INVENTION  
       [0001] 1. Field of the Invention  
       [0002] The present invention relates to delivery techniques of real-time data streams and in particular to system and method of delivery in real-time data streams over a network.  
       [0003] 2. Description of the Related Art  
       [0004] In general, a video transmission system may be divided into three main types of infrastructure: Television broadcasting such as Broadcasting Satellite (BS) or Communication Satellite (CS); Cable Television (CATV) broadcasting; and Internet broadcasting. A recent shift from analog-base to digital-base makes a wide variety of hitherto impossible services possible. In the field of Television broadcasting, for example, the following services become feasible:  
       [0005] Several hundred high-quality channels for providing programs;  
       [0006] EPG (Electronic Programming Guide);  
       [0007] Program reservations and watching using a home server;  
       [0008] Program-synchronization broadcasting such that, for example, a WWW homepage is synchronized to a program, which is realized in analog broadcasting systems such as ADAMS (TV-Asahi Data and Multimedia Service) or BitCast; and  
       [0009] Audience participation live program through Interactive (bidirectional) broadcasting.  
       [0010] In addition, there has been proposed a system such as Real Video G2, which is capable of presenting single content formed by combining content materials such as stored video data, audio data, and/or HTML document data. This conventional system allows the combined content materials to be synchronously displayed according to a predetermined scenario, achieving more widely various representations, compared with conventionally formed content.  
       [0011] Under these circumstances, a technique of multicasting live audio and video data, or real-time multimedia data, to a plurality of participants has received widespread attention, and therefore various multimedia content multicasting techniques have been proposed (for example, see U.S. Pat. No. 6,351,467 B1).  
       [0012] On the other hand, a live-video transmission system which transmits live video data from a transmitter to a receiver has been described in Japanese Patent Application Unexamined Publication No. P2001-45445A. More specifically, this live-video transmission system attaches picture ID and indexes to the live video data, allowing a user to select desired pictures of the live content. This conventional live-video transmission system is based on the unicast technology.  
       [0013] In the multicast delivery of real-time content streams, however, time-shifting operations of the real-time content, including pause, reverse, fast-forward, and time-shift replaying during recording, are not possible. In other words, it is impossible to play back awhile-precoding live video pictures. Taking an electronic conference using the multicasting technology as an example, when a participant is late for the electronic conference, the participant cannot reference previous presentation documents and therefore may not understand well a discussion that is currently made in the electronic conference.  
       SUMMARY OF THE INVENTION  
       [0014] An object of the present invention is to provide a system and method for delivering real-time data streams allowing time-shifting operations such as pause, reverse, fast-forward, and time-shift replaying.  
       [0015] According to the present invention, a transmitter delivers real-time multimedia data in multicast to the receivers while storing the real-time multimedia data into a first memory. When having received a time shift transition command from a receiver, the transmitter reads time-shifted multimedia data from the first memory depending on the time-shift transition command. The time-shifted multimedia data is then transmitted in unicast to the receiver which originated the time-shift transition command. The receiver receives the real-time multimedia data in multicast from the transmitter before transmitting the time-shift transition command and receives the time-shifted multimedia data in unicast from the transmitter after transmitting the time-shift transition command.  
       [0016] The transmitter preferably adds a time stamp to each transmission block size of the real-time multimedia data, stores the real-time multimedia data with time stamps into the first memory, and delivers the real-time multimedia data with time stamps to the receivers.  
       [0017] The receiver preferably creates a thumbnail picture from the real-time multimedia data received from the transmitter each time an amount of real-time multimedia data per unit time exceeds a predetermined level, and stores thumbnail pictures with corresponding time stamps into a second memory so as to designate a desired thumbnail picture, allowing a desired location of the real-time multimedia data to be designated. In addition, when a time-shift request occurs, the receiver creates a time-shift transition command based on the thumbnail pictures with the corresponding time stamps stored in the second memory, and transmits the time-shift transition command to the transmitter so as to receive time-shifted multimedia data from the transmitter in unicast.  
       [0018] The time-shift transition command may be one of a replay start location designation command, a pause command, a reverse command, a slow-replay command, and a fast-forward command.  
       [0019] According to another embodiment, each of the receivers stores the real-time multimedia data received from the transmitter into a third memory and, when a time-shift request occurs, reads time-shifted multimedia data from the third memory depending on the time-shift request.  
       [0020] According to another aspect of the present invention, when having received a start request command from a receiver, the transmitter delivers real-time multimedia data in multicast to the receiver while storing the real-time multimedia data into a first memory. When having received a time-shift transition command from the receiver, the transmitter reads time-shifted multimedia data from the first memory depending on the time-shift transition command, to transmit the time-shifted multimedia data in unicast to the receiver which originated the time-shift transition command. When having received a termination request command from the receiver, the transmitter terminates multimedia data delivery to the receiver.  
       [0021] The transmitter may include: an input section for inputting real-time multimedia data; a multicast transmitter for transmitting the real-time multimedia data to each of the receivers; a first unicast transceiver for receiving a command from a receiver and transmitting a response to the receiver; a command analyzer for analyzing a command received from the receiver to determine a type of the received command; a first memory for storing the real-time multimedia data; and a first controller controlling the multicast transmitter, the unicast transceiver and the first memory, such that the real time multimedia data is delivered in multicast to each of the receivers while storing the real-time multimedia data into the first memory, wherein, when having received a time-shift transition command from a receiver, time-shifted multimedia data is read from the first memory depending on the time-shift transition command and is transmitted in unicast to the receiver which originated the time-shift transition command.  
       [0022] The receiver may include: a multicast receiver for receiving the real-time multimedia data from the transmitter; a second unicast transceiver for transmitting a command to the transmitter and receiving a response to the command from the transmitter; and a second controller controlling such that the real-time multimedia data is received in multicast from the transmitter before transmitting the time-shift transition command, and the time-shifted multimedia data is received in unicast from the transmitter after transmitting the time-shift transition command.  
       [0023] As described above, the real-time multimedia stream delivery system according to the present invention is provided with a multicast-unicast changeover mechanism for changing between real-time multimedia data delivery and time-shifted multimedia data delivery, which allows time-shifted replay in real-time multimedia streaming delivery. In other words, the real-time multimedia data is transmitted in multicast and the time-shifted multimedia data is transmitted on demand to a receiver in unicast. Accordingly, each receiver having relatively small amount of memory capacity can provide time-shifted replay without storing real-time multimedia data.  
       [0024] Further, when the receiver creates thumbnail pictures from the received real-time multimedia data, time stamps attached to the thumbnail pictures are stored into the storage device. Accordingly, by just designating and sending a desired thumbnail picture to the transmitter, a user-desired scene of real-time multimedia data can be easily replayed. 
     
    
    
     BRIEF DESCRIPTION OF THE DRAWINGS  
     [0025]FIG. 1 is a block diagram showing a real-time stream delivery system according to a first embodiment of the present invention;  
     [0026]FIG. 2 is a block diagram showing a functional structure of a controller in the real-time stream delivery system according to the first embodiment;  
     [0027]FIG. 3 is a flow chart showing a transmission control operation of a transmitter in the real-time stream delivery system according to the first embodiment;  
     [0028]FIG. 4 is a flow chart showing a reception control operation of a receiver in the real-time stream delivery system according to the first embodiment;  
     [0029]FIG. 5 is a diagram showing a UDP format used in multicast transmission;  
     [0030]FIG. 6 is a diagram showing a TCP format used in unicast transmission;  
     [0031]FIG. 7 is a diagram showing the format of a command;  
     [0032]FIG. 8 is a diagram showing the time-varying amount of multimedia data for explanation of creating thumbnail images in the first embodiment;  
     [0033]FIG. 9 is a diagram showing a sequence of communication when a start request command is sent from the receiver to the transmitter in the first embodiment;  
     [0034]FIG. 10 is a diagram showing a sequence of communication when a time-shift transition command is sent from the receiver to the transmitter in the first embodiment;  
     [0035]FIG. 11 is a diagram showing a sequence of communication when a termination request command is sent from the receiver to the transmitter in the first embodiment;  
     [0036]FIG. 12 is a block diagram showing a real-time stream delivery system according to a second embodiment of the present invention; and  
     [0037]FIG. 13 is a block diagram showing a real-time stream delivery system according to a third embodiment of the present invention. 
    
    
     DESCRIPTION OF THE PREFERRED EMBODIMENTS  
     [0038] As shown in FIG. 1, it is assumed for simplicity that a delivery system of real-time multimedia streams according to a first embodiment of the present invention is composed of a transmitter  10  and a receiver  12 , which are connected by a network  11 . Needless to say, a plurality of receivers  12  may be connected to the network  11 . The network  11  may be a communication network such as the Internet or Intranet. As described later, the transmitter  10  delivers real-time multimedia data such as live video data to the receiver(s)  12  in multicast through the network. When a time-shift replaying request occurs, the transmitter  10  delivers requested time-shifted multimedia data in unicast through the network  11  to the receiver  12  which originates that request. The detailed structures and operations of the transmitter  10  and the receiver  12  will be described hereinafter.  
     [0039] Transmitter  
     [0040] In the transmitter  10 , an input section  101  inputs real-time multimedia data including still pictures, moving pictures, audio data, and/or text data from an input device such as still/video camera, microphone and the likes. Here, the multimedia data may be digital or analog.  
     [0041] A data processing section  102  receives the multimedia data from the input section  101  and performs data processing such as analog-to-digital conversion, format transform, and time stamp addition. The time stamp addition is to add time information (time stamp) to the multimedia data for each packet size. The multimedia data with time stamps added for each packet size are output to a controller  104  and stored in a storage device  103 , which may be a memory device such as a hard disk drive (HDD).  
     [0042] The controller  104  inputs the output of the data processing section  102  and the output of the storage device  103  and selects one of these outputs depending on a command analytical result inputted from a command analyzer  105 . The command analyzer  105  analyzes a command received by a command receiver  108  to output its analytical result to the controller  104  and a command response section  107 . Depending on the analytical result, the controller  104  selects one of the output of the data processing section  102  and the output of the storage device  103  to output selected multimedia data with time stamps to a transmission data processor  106 . Detailed structure and operation of the controller  104  will be described later.  
     [0043] The transmission data processor  106  assembles data packets from the multimedia data with time stamps and, if necessary, encrypts the payloads thereof. The data packets are outputted to a selected one of a multicast transmitter  111  and a unicast transmitter  112  of a transmission section  109 .  
     [0044] The command response section  107  creates a command response packet according to the analytical result inputted from the command analyzer  105  and outputs it to the unicast transmitter  112 .  
     [0045] The command receiver  108  receives a command packet from a certain receiver  12  through a unicast receiver  113  of a reception section  110  and outputs its command to the command analyzer  105 . In this embodiment, a multicast receiver  114  of the reception section  110  is not used.  
     [0046] Receiver  
     [0047] In the receiver  12 , the receiver  12  is provided with a reception section  115  including a multicast receiver  117  and a unicast receiver  118  and a transmission section  116  including a unicast transmitter  119  and a multicast transmitter  120 . The multicast receiver  117  receives real-time multimedia data from the multicast transmitter  111  of the transmitter  10 . The unicast receiver  118  receives time-shifted multimedia data or time-shift control data from the unicast transmitter  112  of the transmitter  10 . In this embodiment, the multicast transmitter  120  is not used.  
     [0048] A received data analyzer  121  analyzes packets received from he transmitter  10  through the unicast receiver  118  to determine whether a received packet is a data packet or a command response packet. When the data packet is received, the received data analyzer  121  outputs it as time-shifted multimedia data to a data reception selector  123 . When the command response packet is received, the received data analyzer  121  outputs it to a controller  124 .  
     [0049] A command transmitter  122  inputs a command from the controller  124  to assemble a command packet. The command packet is outputted to the transmitter  10  through the unicast transmitter  119 .  
     [0050] The data reception selector  123  selects one of the real-time multimedia data received from the multicast receiver  117  and the time-shifted multimedia data received from the received data analyzer  121  depending on an instruction inputted from the controller  124 . A selected multimedia data is outputted to a reception data processor  125 .  
     [0051] The controller  124  issues a command to a command transmitter  122 , instructs the data reception selector  123  to perform received data selection as described above, and controls reception flag (TRUE/FALSE) and reception mode (real-time/time-shift) as described later.  
     [0052] The reception data processor  125  performs digital-to-analog conversion and format transform of received multimedia data, analyzes the received multimedia data and its attached time stamps, and creates thumbnail pictures depending on the amount of multimedia data per unit time, which will be described later.  
     [0053] A storage device  126  stores thumbnail pictures created by the reception data processor  125  together with time stamps. The storage device  126  may be a memory device such as a hard disk drive (HDD). An output section  127  inputs multimedia data from the reception data processor  125  and the thumbnail pictures from the storage device  126  and outputs and, as necessary, combines them to an external monitor (not shown).  
     [0054] Multicast and Unicast  
     [0055] Multicast transmission and Unicast transmission are defined as standards in RFC (Request For Comments) issued by IETF (Internet Engineering Task Force).  
     [0056] Multicast transmission is defined in RFC/68: User Datagram Protocol (UDP), in which UDP is used to transfer data. UDP has a relatively simple format composed of: Source/Destination IP address fields; Protocol field (here, protocol number “17” is assigned), Packet length field; and Data field (see FIG. 5).  
     [0057] Unicast transmission is defined in RFC793: Transmission Control Protocol (TCP), in which TCP is used to transfer data. TCP has a format composed of Source port field, Destination port field, Sequence number field (indicating the location of transmitted data), Acknowledgement number field (indicating the sequence number of next data to be transmitted), Data offset field (indicating the start location of data), Reserved field, Control flag field, Window field (indicating the window size), Checksum field, Urgent pointer field, Options field, Padding field, and Data field (see FIG. 6). In Control flag field, URG flag set to 1 indicates that the data field includes urgent data, ACK flag set to 1 indicates that the acknowledgement field is valid (ACK flag is normally set to 1 other than connection establishment), PSH (Push) flag set to 1 indicates that buffering is not performed, RST (Reset) flat set to 1 indicates that a connection is forced to be disconnected, SYN (Synchronize) flag set to 1 indicates that a connection tries to be established, and FIN (Finish flag set to 1 indicates that the connection tries to be disconnected. The urgent pointer field indicates the length of urgent data included in data when URG flag is set to 1.  
     [0058] Commands  
     [0059] A command is composed of a command field and an option field as shown in FIG. 7. The command field defines a command depending on what value is written therein. For example, a  101  valve indicates Replay command, a  103  indicates Pause command, a  104  indicates Fast-Forward command, a  105  indicates Reverse command, and a  106  indicates Thumbnail-based Replay Start Location Designation command.  
     [0060] The option field defines the following options for each command defined in the command field. In the case of Replay command, replay speed (normal or slow) and current time stamp are defined in the option field. In the case of Pause command, current time stamp is defined. In the case of Fast-Forward command, fast-forward speed with replaying or not and current time stamp are defined. In the case of Reverse command, reverse speed with replaying or not and current time stamp are defined. In the case of Thumbnail-based Replay Start Location Designation command, the designated time stamp is set in the option field.  
     [0061] The command field and the option field as described above allow both real-time replay and time-shift replay.  
     [0062] Controller of Transmitter  
     [0063] As shown in FIG. 2, the controller  104  of the transmitter  10  implements a database  104   a , a device status manager  104   b , and a data selector  104   c . The database  104   a  stores a record composed of device information, transmission flag, transmission mode, and time stamp for each receiver  12 . The device status manager  104   b  receives an analytic result from the command analyzer  105  to store as a record the device information identifying the receiver  12  that has transmitted a corresponding command and the delivery information composed of transmission flag, transmission mode, and time stamp. The device information identifies the receiver  12 , which may be IP (Internet Protocol) address. The transmission mode includes “real-time”, “time-shift” and “not defined”. The transmission flag is one of TRUE and FALSE. The device information and the time stamp are extracted from a received command by the command analyzer  105 .  
     [0064] The, data selector  104   c  selects one of the real-time multimedia data received from the data processing section  102  and the time-shifted multimedia data received from the storage device  103  depending on the device information and the delivery information stored in the database  104   a.    
     [0065] In the case where a start request command for real-time multimedia delivery has been received from the receiver  12 , the device status manager  104   b  searches the database  104   a  for the device information included in the received start request command. In this case, since no device information identifying the receiver  12  is found, the device status manager  104   b  registers a new record for the receiver  12  into the database  104   a . For example, the device information is set to the IP address of the receiver  12 , the transmission flag is set to TRUE, the transmission mode is set to “real-time”, and the time stamp is set to 1000.  
     [0066] The data selector  104   c  monitors the database  104   a  at all times and, when no change in device status of the receiver  12  occurs in the database  104   a , selects the real-time multimedia data received from the data processing section  102  to output it to the transmission data processor  106 .  
     [0067] Thereafter, when a time-shift transition command has been received from the receiver  12 , the device status manager  104   b  searches the database  104   a  for the device information included in the received time-shift transition command. In this case, since the device information identifying the receiver  12  is found, the device status manager  104   b  sets the transmission mode of the record corresponding to the receiver  12  to “time-shift” and similarly sets the time stamp thereof to the shifted time stamp. When the transmission mode has been changed in the database  104   a , the data selector  104   c  detects such a change and therefore selects the time-shift multimedia data received from the storage device  103  to output it to the transmission data processor  106 .  
     [0068] When a termination command has been received from the receiver  12 , the device status manager  104   b  searches the database  104   a  for the device information included in the received termination command. In this case, since the device information identifying the receiver  12  is found, the device status manager  104   b  sets the transmission flag of the record corresponding to the receiver  12  to “FALSE”. When the data selector  104   c  detects such a change, the data selector  104   c  stops delivery of real-time or time-shifted multimedia stream.  
     [0069] It should be noted that the controller  104  may be composed of a program-controlled processor implementing the data selector  104   c  and the device status manager  104   b  and a memory (not shown) separately implementing the database  104   a.    
     [0070] Detailed control operations of the transmitter  10  and the receiver  12  will be described hereafter.  
     [0071] Operation of Transmitter  
     [0072] Referring to FIG. 3, the controller  104  first performs initialization of the database  104   a  such that the transmission flag is set to “FALSE” and the transmission mode to “real-time” (step A 1 ). Thereafter, it is determined whether the command receiver  108  receives a command packet from a receiver  12  through the unicast receiver  113  (step A 2 ). When no command has been received (No in step A 2 ), it is further determined whether the transmission flag of the database  104   a  is “TRUE” (step A 18 ). When the transmission flag is not “TRUE” (No in step A 18 ), it means that neither real-time nor time-shifted multimedia data is delivered and therefore control goes back to the step A 2 .  
     [0073] When the transmission flag is “TRUE” (Yes in step A 18 ), it means that real-time/time-shifted multimedia data is delivered to the receiver  12  identified by the device information corresponding to the transmission flag of TRUE and therefore the control goes to step A 12 , which will be described later.  
     [0074] When a command has been received (Yes in step A 2 ), the command analyzer  105  analyzes the received command and the controller  104  performs delivery control depending on the command analytic result (steps A 3 -A 11 ).  
     [0075] More specifically, when the received command is a start request command (Yes in step A 3 ), the command analyzer  105  instructs the command response section  107  to acknowledge receipt of the start request command (step A 4 ). The command response section  107  transmits a response to the start request command to the receiver  12  through the unicast transmitter  112 . Thereafter, the device status manager  104   b  of the controller  104  registers device information of the receiver  12  and writes corresponding delivery information into the database  104   a  such that its transmission flag is set to “TRUE” (step A 5 ) and its transmission mode is set to “real-time” (step A 6 ).  
     [0076] When the received command is not a start request command (No in step A 3 ) but a termination request command (Yes in step A 7 ), the command analyzer  105  instructs the command response section  107  to acknowledge receipt of the termination request command (step A 19 ). The command response section  107  transmits a response to the termination request command to the receiver  12  through the unicast transmitter  112 . Thereafter, the device status manager  104   b  of the controller  104  writes the corresponding delivery information into the database  104   a  such that its transmission flag is reset to “FALSE” (step A 20 ) and then the transmission process is terminated (step A 21 ).  
     [0077] When the received command is neither a start request command (No in step A 3 ) nor a termination request command (No in step A 7 ) but a time-shift transition command (Yes in step A 8 ), the command analyzer  105  instructs the command response section  107  to acknowledge receipt of the time-shift transition command (step A 9 ). The command response section  107  transmits a response to a time-shift transition command to the receiver  12  through the unicast transmitter  112 . Thereafter, the device status manager  104   b  of the controller  104  writes corresponding delivery information into the database  104   a  such that its transmission mode is set to “time-shift” (step A 10 ) and its time stamp is set to its designated value (step A 11 ). Such a time shift transition command includes a thumbnail based replay start location designation command, a pause command, a reverse command, a slow-replay command, or a fast-forward command. Hereinafter, it is assumed that a time shift transition command is a thumbnail-based replay start location designation command.  
     [0078] Subsequently, it is determined whether the current transmission mode of the corresponding receiver  12  indicates “real-time” (step A 12 ). When it indicates “real-time” (Yes in step A 12 ), the data selector  104   c  of the controller  104  receives real-time multimedia data from the data processing section  102  (step A 14 ) and transfers it to the transmission data processor  106 . The transmission data processor  106  assembles data packets from the real-time multimedia data with time stamps, and outputs them to the multicast transmitter  111  of the transmission section  109 . In this way, the real time multimedia data is transmitted to the receiver  12  (step A 15 ) and then the control goes back to the step A 2 .  
     [0079] In the case where the start request command is received after live performance or conference has started, the controller  104  preferably attaches preceding thumbnail pictures to the real-time multimedia data to be transmitted.  
     [0080] When the current transmission mode of the corresponding receiver  12  indicates “time-shift” (No in step A 12 ), the controller  104  checks the presence or absence of data stored in the storage device  103  (step A 13 ). When no time-shifted multimedia data is left in the storage device  103  (Yes in step A 13 ), the data selector  104   c  of the controller  104  receives real-time multimedia data from the data processing section  102  (step A 14 ) and transfers it to the transmission data processor  106 . The transmission data processor  106  assembles data packets from the real-time multimedia data with time stamps, and outputs them to the multicast transmitter  111  of the transmission section  109 . In this way, the real-time multimedia data is transmitted to the receiver  12  (step A 15 ) and then the control goes back to the step A 2 . As described before, when the real-time multimedia data is transmitted after live performance or conference has started, the controller  104  preferably attaches preceding thumbnail pictures to the real-time multimedia data to be transmitted.  
     [0081] When time-shifted multimedia data is left in the storage device  103  (No in step A 13 ), the data selector  104   c  of the controller  104  receives time-shifted multimedia data from the storage device  103  based on a time stamp value set in the database  104   a  corresponding to the receiver  12  (step A 16 ) and transfers it to the transmission data processor  106 . The transmission data processor  106  assembles unicast data packets from the time-shifted multimedia data with time-shifted time stamps and real-time time stamps, and outputs them to the unicast transmitter  112  of the transmission section  109 . In this way, the time-shifted multimedia data is transmitted to the receiver  12  (step A 17 ) and then the control goes back to the step A 2 .  
     [0082] Assuming that a time-shift transition command is a reverse command, at the step A 11 , the device status manager  104   b  of the controller  104  writes a current time stamp and a reverse interval received from the receiver  12  into the database  104   a . At the step A 16 , the data selector  104   c  of the controller  104  fetches time-shifted multimedia data from the storage device  103  based on time stamp values sequentially calculated from the current time stamp and the reverse interval and outputs them to the transmission data processor  106 . More specifically, the time stamp values are calculated by sequentially subtracting the reverse interval from the current time stamp. For example, assuming that the current time stamp is 100 and the reverse interval is 5, the data selector  104   b  sequentially fetches time-shifted multimedia data each corresponding to time stamps of 100, 95, 90, and so on from the storage device  103  at regular intervals (e.g. 1).  
     [0083] Assuming that a time-shift transition command is a slow-replay command, at the step A 11 , the device status manager  104   b  of the controller  104  writes a current time stamp and a slow replay interval received from the receiver  12  into the database  104   a . At the step A 16 , the data selector  104   c  of the controller  104  fetches time-shifted multimedia data from the storage device  103  based on time stamp values calculated from the current time stamp and the slow-replay interval and outputs them to the transmission data processor  106 . More specifically, the time stamp values are calculated by sequentially adding the slow-replay interval to the current time stamp. For example, assuming that the current time stamp is 100 and the slow-replay interval is 0.1, the data selector  104   b  sequentially fetches time-shifted multimedia data each corresponding to time stamps of 100, 100.1, 100.2 and so on from the storage device  103  at regular intervals (e.g. 1).  
     [0084] Assuming that a time-shift transition command is a fast-forward replay command, at the step A 11 , the device status manager  104   b  of the controller  104  writes a current time stamp and a fast-forward interval received from the receiver  12  into the database  104   a . At the step A 16 , the data selector  104   c  of the controller  104  fetches time-shifted multimedia data from the storage device  103  based on time stamp values calculated from the current time stamp and the fast-forward interval and outputs them to the transmission data processor  106 . More specifically, the time stamp values are calculated by sequentially adding the fast-forward interval to the current time stamp. For example, assuming that the current time stamp is 100 and the fast-forward interval is 5, the data selector  104   b  sequentially fetches time-shifted multimedia data each corresponding to the time stamps of 100, 105, 110 and so on from the storage device  103  at regular intervals (e.g. 1).  
     [0085] Assuming that a time-shift transition command is a pause command, at the step A 11 , the device status manager  104   b  of the controller  104  writes a current time stamp into the database  104   a . At the step A 16 , the data selector  104   c  of the controller  104  fetches time-shifted multimedia data from the storage device  103  based on a time stamp value calculated from the current time stamp and outputs it to the transmission data processor  106 . For example, assuming that the current time stamp is 100, the data selector  104   b  sequentially fetches time-shifted multimedia data corresponding to the time stamp of 100 from the storage device  103  at regular intervals (e.g. 1). When a pause release command has been received together with a current time stamp from the receiver  12 , the controller  104  sequentially receives the following time-shifted multimedia data from the storage device  103  to transmit them to the receiver  12 .  
     [0086] Operation of Receiver  
     [0087] Referring to FIG. 4, the controller  124  first performs initialization such that the reception flag is set to “FALSE” and the reception mode to “real-time” (step B 1 ). Thereafter, the controller  124  determines whether a command transmission request is received from outside user (step B 2 ). When a command transmission request has been received (Yes in step B 2 ), the controller  124  instructs the command transmitter  122  to transmit a requested command to the transmitter  10  through the unicast transmitter  110  of the transmission section  116  (step B 3 ). Such a requested command may be a start request command, a thumbnail-based replay start location designation command, a pause command, a reverse command, a fast-forward command, a slow-replay command, a termination request command or the likes.  
     [0088] Thereafter, it is determined whether the received data analyzer  121  receives a command response at the unicast receiver  118  (step B 4 ). When no command response has been received (No in step B 4 ), it is further determined whether the reception flag is “TRUE” (step B 18 ). When the reception flag is not “TRUE” (No in step B 18 ), it means that neither real-time nor time-shifted multimedia data is received and therefore control goes back to the step B 2 .  
     [0089] When the reception flag is “TRUE” (Yes in step B 18 ), it means that real-time/time-shifted multimedia data is received and therefore the control goes to step B 11 , which will be described later.  
     [0090] When a command response has been received (Yes in step B 4 ), the received data analyzer  121  analyzes the received command response to determined whether it is a response to the start request command (step B 5 ). When it is not a response to the start request command (No in step B 5 ), the control goes to a step B 8 . When it is a response to the start request command (Yes in step B 5 ), the controller  124  sets the reception flag to “TRUE” (step B 6 ) and the reception mode to “real-time” (step B 7 ).  
     [0091] Thereafter, the received data analyzer  121  determines whether the received command response is a response to the termination request command (step B 8 ). When the received command is a response to the termination request command (Yes in step B 8 ), the controller  124  sets the reception flag to “FALSE” (step B 19 ) and terminates the process (step B 20 ).  
     [0092] When the received command response is nota response to the termination request command (No in step B 8 ), the received data analyzer  121  determines whether the received command response is a response to the time-shift transition command (step B 9 ). When it is the response to the time-shift transition command (Yes in step B 9 ), the controller  124  sets the reception mode to “time-shift” (step B 10 ).  
     [0093] Subsequently, it is determined whether the current reception mode indicates “real-time” (step B 11 ). When it indicates “real-time” (Yes in step B 11 ), the data reception selector  123  selects real-time multimedia data received from the multicast receiver  117  of the reception section  115  (step B 15 ) and transfers it to the reception data processor  125 .  
     [0094] The reception data processor  125 , as necessary, performs thumbnail creation (step B 16 ) before back to the step B 2 . In the step B 16 , the reception data processor  125  monitors the amount of received real-time multimedia data and, when the amount of received data per unit time exceeds a predetermined threshold, creates thumbnail pictures from the received real-time multimedia data to store the thumbnail pictures with time stamps into the storage device  126 .  
     [0095] When the current reception mode indicates “time-shift” (No in step B 11 ), the data reception selector  123  gets a real-time time stamp of real-time multimedia data and a time-shifted time stamp of the time-shifted multimedia data attached to the real-time multimedia data (step B 12 ). Thereafter, the data reception selector  123  determines whether a difference between the time-shifted time stamp and the real-time time stamp is smaller than a predetermined threshold, to decide on whether a time-shifted picture catches up the real-time picture (step B 13 ).  
     [0096] When a time-shifted picture catches up the real-time picture (Yes in step B 13 ), the controller  124  sets the reception mode to “real-time” (step B 14 ). The data reception selector  123  selects real-time multimedia data received from the multicast receiver  117  of the reception section  115  (step B 15 ) and transfers it to the reception data processor  125 . The reception data processor  125  performs thumbnail creation (step B 16 ) before back to the step B 2 . When a time-shifted picture does not catch up the real-time picture (No in step B 13 ), the data reception selector  123  selects time-shifted multimedia data received from the received data analyzer  121  (step B 17 ) and then the control goes back to the step B 2 .  
     [0097] Taking a teleconference as an example, a presenter may use an electronic version of presentation documents to provide an explanation to conferees. In the teleconference application, there is seldom a case where an entire picture is changed during explanation of a single page. Accordingly, there is frequently used a delivery method such that difference data between a current frame and an immediately preceding frame or a several-second-old frame is delivered to reduce the load on the network as much as possible. In such a case, when breaking across pages, a picture is changed and thereby the amount of real-time multimedia data is increased as shown in FIG. 8. Therefore, the receiver  12  can easily detect page turning. At such page change timing, it is possible to create thumbnail pictures from the real-time multimedia data, by which the conferees can obtain an electronic version of presentation documents as the thumbnail pictures and further use the time stamps of the thumbnail pictures as indexes to designate replay start locations of the real-time multimedia data. Accordingly, when the present invention is applied to the teleconference application, it is possible for a conferee attending to the conference to check subjects on the agenda from the beginning of the conference. Further, when a conferee leaves the conference for a while, the conferee can check subjects on the agenda when the conferee is absent. Furthermore, a conferee can look at the foregoing several pages of the presentation documents delivered during the conference.  
     [0098] Sequential Operations  
     [0099] Sequential operations between the transmitter  10  and the receiver  12  will be described with reference to FIGS.  9 - 11 .  
     [0100] As shown in FIG. 9, when a start request command has been sent from the receiver  12  to the transmitter  10 , the transmitter  10  sends a response to the start request command back to the receiver  12  and sets the transmission mode to “real-time”. At the receiver  12 , when having received the response to the start request command, the reception mode is set to “real-time.” In the transmitter  10 , the controller  104  receives real-time multimedia data from the data processing section  102  and sends it to the receiver  12  in multicast.  
     [0101] As shown in FIG. 10, when a slow-replaying request command has been sent from the receiver  12  to the transmitter  10 , the transmitter  10  sends a response to the slow-replaying request command back to the receiver  12  and sets the transmission mode to “time-shift”. In the transmitter  10 , the controller  104  receives time-shifted multimedia data from the storage device  103  and sends it to the receiver  12  in unicast.  
     [0102] As shown in FIG. 11, when a termination request command has been sent from the receiver  12  to the transmitter  10 , the transmitter  10  sends a response to the termination request command back to the receiver  12  and terminates the transmission process.  
     [0103] Advantages  
     [0104] As described above, the real-time multimedia stream delivery system according to the first embodiment of the present invention is provided with a multicast unicast changeover mechanism for changing between real-time multimedia data delivery and time-shifted multimedia data delivery, which allows time-shifted replay in real-time multimedia streaming delivery.  
     [0105] Further, when the receiver creates thumbnail pictures from the received real-time multimedia data, time stamps attached to the thumbnail pictures are stored into the storage device. Accordingly, by sending a desired time stamp to the transmitter, a desired replay start location of real time multimedia data can be easily designated.  
     [0106] Referring to FIG. 12, a real-time multimedia stream delivery system according to a second embodiment of the present invention is composed of a transmitter  10  and a receiver  13 , which are connected by a network  11 . Needless to say, a plurality of receivers  13  may be connected to the network  11 . Blocks similar to those previously described with reference to FIG. 1 are denoted by the same reference numerals and detailed descriptions thereof will be omitted.  
     [0107] As described in the first embodiment, the transmitter  10  delivers real-time multimedia data to the receiver(s)  13  in multicast through the network  11 . When a time-shift replaying request occurs, the transmitter  10  delivers requested time-shifted multimedia data in unicast through the network  11  to the receiver  13  which originates that request.  
     [0108] According to the second embodiment, in addition, a storage device  130  of each receiver  13  is provided to store real-time multimedia data received from the transmitter  10  in multicast, which can be used for time-shift replaying under control of the controller  128 . More specifically, a controller  128  of the receiver  13  instructs a reception data processor  129  to store the received real-time multimedia data into the storage device  130 . Depending on user&#39;s instruction, the controller  128  can select one of an inside replay mode and an outside replay mode. The outside replay mode is that time-shift replaying is performed at the transmitter  10  as described in the first embodiment. The inside replay mode is that time-shift replaying is performed at the receiver  13  by reading time-shifted multimedia data from the storage device  130 . The time-shifted multimedia data is designated by inputting a time-shift transition command similar to the first embodiment.  
     [0109] In order to store the necessary amount of real-time multimedia data received from the transmitter  10 , the storage device  130  needs sufficient capacity, which may be implemented by hard disk drive (HDD). Further, the reception data processor  129  has functions of storing real-time multimedia data into the storage device  130  and instructing the storage device  130  to output time-shifted multimedia data to the output section  127  under the control of the controller  128 .  
     [0110] As described above, according to the second embodiment, the inside and outside replay modes can be selected at a receiver. Since the inside replay mode allows time-shifting replay without transferring multimedia data through the network  11 , the load on the network  11  can be dramatically reduced.  
     [0111] The above-described functions of the transmitter  10  may be implemented by transmitter control programs running on a program-controlled processor. Similarly, the above-described functions of the receiver  12  or  13  may be implemented by receiver control programs running on a program-controlled processor.  
     [0112] As shown in FIG. 13, the data processing section  102 , the controller  104 , the command analyzer  105 , transmission data processor  106 , the command response section  107 , and the command receiver  108  may be implemented by transmission control programs stored in a memory  202  running on a program-controlled processor  201 . Similarly, the received data analyzer  121 , the command transmitter  122 , data reception selector  123 , the controller  124 / 128 , and the reception data processor  125 / 129  may be implemented by reception control programs stored in a memory  204  running on a program-controlled processor  203 . The operations of the transmitter  10  and the receiver  12  or  13  are similar to those described with reference to FIGS. 3 and 4, and therefore details are omitted.