Patent Publication Number: US-2004059783-A1

Title: Multimedia cooperative work system, client/server, method, storage medium and program thereof

Description:
CROSS REFERENCE TO RELATED APPLICATION  
     [0001] This application is s continuation of International PCT Application No. PCT/JP01/01822 filed on Mar. 8, 2001. 
    
    
     
       BACKGROUND OF THE INVENTION  
       [0002] 1. Field of the Invention  
       [0003] The present invention generally relates to computer system and multimedia communication fields and in particular, relates to a multimedia cooperative work system for enabling a plurality of clients in a network to exchange opinions on an arbitrary multimedia data and realizing the improved efficiency of work, such as the co-editing work, commenting and the like, of multimedia data and the method thereof.  
       [0004] 2. Description of the Related Art  
       [0005] Owing to the advancement of computer technologies, the digital processing of entire multimedia data, such as character data, dynamic images and voice in a computer has become possible. In this way, a function for efficiently processing/operating multimedia data, which could not be possible by a conventional analog treatment, has been realized.  
       [0006] The electronic tag of an electronic document is one of such examples. Currently, markers/comments are attached to a printed document in order to misprint is pointed out (one type of co-editing work) or to refer to important items later (supplementary work for user&#39;s understanding/recognition). However, if a target document is another person&#39;s, no character can be directly written in it. Another person also cannot extract or use such comments.  
       [0007] An electronic memorandum can solve this problem by managing an original electronic document, an electronic tag and correspondence data between the original electronic document and electronic tag (for example, information that this comment is for line M of page N) as an individual piece of electronic data. By utilizing a variety of digital data processing technologies, such information can be displayed and presented to a user as if an electronic tag were embedded in an electronic document. As a publicly known case of such a prior art, there is Japanese Patent Laid-open No. 2000-163414 and the like.  
       [0008] In particular, recently, since dynamic image (moving image)/voice processing technology (storage, transmission, encryption/conversion and the like) has been improved, an environment in which a general user can utilize dynamic image/voice data lightheartedly, exists. For example, the following usages are available.  
       [0009] (1) Dynamic image/voice data that are compressed to several hours&#39; data and are stored on a CD (compact disc) or DVD (digital versatile disc) can be reproduced and appreciated in a TV monitor at home.  
       [0010] (2) Live images that are broadcast in real time in a network can be viewed lightheartedly using a computer connected to the Internet.  
       [0011] (3) AV data (AV; audio/visual, dynamic image data and the audio data to be synchronized with the dynamic image data and to be reproduced) taken by a home digital video camera can be enjoyed together with friends by sending the AV data to the friends by electronic mail and sharing the AV data with them.  
       [0012] As one of the prior art for attaching comments and the like in an environment where multimedia data, including such dynamic image (moving image) data can be transmitted/received through a network, there is a document editing device (Japanese Patent Application No.2-305770) (hereinafter called the “first prior art”) This editing device has a function to manage, edit and relate comments to realize the intra-group cooperative work of an electronic document composed of a variety of multimedia data, such as characters, static images, graphics, dynamic images and the like. A comment can also be attached to a comment.  
       [0013] Another prior art is a video message transmission system and the method thereof (Japanese Patent Application N. 11-368078)(hereinafter called the “second prior art”). This system/method enables a receiving user to access/process dynamic image data in units of segments by transmitting the dynamic image data together with the time sequence data and comment data of the dynamic image when a user transmits the captured dynamic image data to another user.  
       [0014] The applicant of the present invention has supposed that, for example, the following services should be realized.  
       [0015] As one example, there is a network appreciation service. For example, if one member of a local community (a group of neighborhood friends and the like) distributes/shares the AV data of an event, such as an athletic meeting at school, camp/drive and the like photographed by him to/with the members through a network, each member&#39;s comments (“A person photographed at this scene is the son of Mr.◯◯.”, “This scene is memorable.” and the like) can be exchanged between the members. In this way, he can comment on the AV data together with the members participating in the event as if they were together at his house and holding a video show.  
       [0016] As another example, there is the co-editing work supplementary service of AV data through a network. In this case, the comments are “This scene is re-arrayed after another scene.”, “Since this scene is important, the broadcast time should be extended.” and the like. Furthermore, final user comments can be used as automatically edited AV script by introducing a specific editing command as a kind of comment (this user comment corresponds to an electronic tag in an electronic document and, in particular, is called as a “multimedia electronic tag” in this specification).  
       [0017] However, the realization of such a service is not supposed in the prior arts described above and there is no technology for realizing such a service. For example, in the first prior art, a point (scene) in the time sequence of time-sequential data such as dynamic image data cannot be specified nor can a comment be attached. In the second prior art, the use of additional information by another user is not intended.  
       [0018] As described above, an object of the present invention is to provide a multimedia cooperative work system, the client/server, method, storage medium and program thereof enabling a plurality of clients in a network to exchange opinions on arbitrary multimedia data and realizing the improved efficiency of work, such as the co-editing work, commenting and the like, of multimedia data.  
       SUMMARY OF THE INVENTION  
       [0019] The multimedia cooperative work system of the present invention is configured to realize multimedia cooperative work by generating the model of a multimedia electronic tag in which the display of a comment and the attribute data thereof/comment input in hierarchical tree shape structure is possible for each scene of multimedia data, the registration of which is requested by an arbitrary client in a server, obtained by dividing the multimedia data in terms of time and exchanging comments on each scene among a plurality of clients, including the requesting client, using the multimedia electronic tag.  
       [0020] According to the multimedia cooperative work system described above, if an arbitrary client transmits arbitrary multimedia data (data, including dynamic image data and the like) to the server and requests the cooperative work, the model of the multimedia electronic tag is generated. A user of each client, including the requesting client (for example, a user doing the co-editing work, commenting and the like of multimedia data) can hold a video show through a network or doing co-editing work and the like as if he were exchanging opinions freely while viewing the AV data together with other users by repeating the input of a desired comment to an arbitrary scene, using the multimedia electronic tag and the input of a comment to another user&#39;s comment (when someone comments on someone else&#39;s comment is discovered by the attribute data described above). 
     
    
    
     BRIEF DESCRIPTION OF DRAWINGS  
     [0021]FIG. 1 shows the basic configuration of the present invention.  
     [0022]FIG. 2 shows the functional configuration of the entire multimedia cooperative work system.  
     [0023]FIG. 3 is a flowchart showing the operation of the entire multimedia cooperative work system.  
     [0024]FIG. 4 shows the internal data format of a management information DB.  
     [0025]FIG. 5 shows a specific example of the described content of a multimedia electronic tag (No. 1).  
     [0026]FIG. 6 shows a specific example of the described content of a multimedia electronic tag (No. 2).  
     [0027]FIG. 7 shows one example of the comment list display/comment input screen of a multimedia electronic tag displayed on the monitor of each client.  
     [0028]FIG. 8 is a flowchart showing the entire conversion process to a multimedia synchronization/reproduction format.  
     [0029]FIG. 9 is a flowchart showing the detailed tag &lt;video&gt; generation process in step S 12  shown in FIG. 8.  
     [0030]FIG. 10 is a flowchart showing the detailed tag &lt;text&gt; generation process in step S 13  shown in FIG. 8.  
     [0031]FIG. 11 shows the transition of the contents of a stack and stored tag &lt;MediaTime&gt; in the case where the process shown in FIG. 10 is applied to the multimedia electronic tag shown in FIG. 5.  
     [0032]FIG. 12 shows the result obtained by converting the format of a multimedia electronic tag shown in FIG. 5 into a multimedia synchronous reproduction format (in this example, SMIL format) by the processes described with reference to FIGS. 8 through 11 (No. 1).  
     [0033]FIG. 13 shows the result obtained by converting the format of a multimedia electronic tag shown in FIG. 5 into a multimedia synchronous reproduction format (in this example, SMIL) by the processes described with reference to FIGS. 8 through 11 (No. 2).  
     [0034]FIG. 14 shows a display example of a dynamic image/comments obtained by reproducing the SMIL documents shown in FIGS. 12 and 13 by a multimedia synchronous reproduction unit  27 .  
     [0035]FIG. 15 shows one example of the basic hardware configuration of a computer.  
     [0036]FIG. 16 shows the loading onto a computer of a program. 
    
    
     DESCRIPTION OF THE PREFERRED EMBODIMENT  
     [0037] The preferred embodiments of the present invention are described below with reference to the drawings.  
     [0038]FIG. 1 shows the basic configuration of the present invention.  
     [0039] In FIG. 1, a server  1  can communicate with each client  4  through a network  8  (for example, the Internet).  
     [0040] The server  1  comprises a multimedia electronic tag model generation unit  2  and a multimedia electronic tag modification/communication unit  3 .  
     [0041] The multimedia electronic tag model generation unit  2  generates the model of a multimedia electronic tag in which a comment and the attribute data thereof can be displayed/inputted in hierarchical tree shape for each scene of multimedia data, the registration of which is requested by an arbitrary client in a server, obtained by dividing the multimedia data in terms of time.  
     [0042] For the attribute data, for example, a comment writer name, a comment generation date, a comment destination (comment on whose comment) and the like, are used.  
     [0043] The publication destination or expiration date of a comment is described in the multimedia electronic tag as one kind of the attribute data of a comment.  
     [0044] The multimedia electronic tag modification/communication unit  3  deletes an overdue comment from a multimedia electronic tag or upon receipt of a multimedia electronic tag request from an arbitrary member client, the unit  3  transmits a multimedia electronic tag from which comments not belonging to this client as a publication destination are deleted, to the requesting client.  
     [0045] Each client  4  comprises a multimedia electronic tag editing unit  5 , a format conversion unit  6  and a multimedia synchronous reproduction unit  7  and the like.  
     [0046] The multimedia electronic tag editing unit  5  displays a comment with attribution data attached to each scene of multimedia data corresponding to the multimedia electronic tag, using the multimedia electronic tag obtained from a server or another client. Simultaneously, the unit  5  enables a comment to be inputted to an arbitrary scene or comment and updates the content of the multimedia electronic tag, based on the input.  
     [0047] The format conversion unit  6  converts the format of a multimedia electronic tag into a format in which multimedia data and the comments thereof are synchronized/reproduced.  
     [0048] The multimedia synchronous reproduction unit  7  synchronizes multimedia data with comments corresponding to each scene of the multimedia data and displays the multimedia data and comments, using the conversion result by the format conversion unit  6 .  
     [0049]FIG. 2 shows the configuration of an entire multimedia cooperative work system according to the preferred embodiment.  
     [0050] In FIG. 2, a multimedia server  10  provides a multimedia electronic tag service.  
     [0051] This multimedia server  10  comprises an electronic tag storage device  12  storing multimedia electronic tags, a multimedia storage device  13  storing multimedia data, a management information DB  14  storing member data, an electronic tag communication unit  15  exchanging a multimedia electronic tag with a client, a multimedia communication unit  16  exchanging multimedia data with a client, a mail server  17  distributing electronic mail to be exchanged between clients, a network I/F  18 , which interfaces the electronic tag communication unit  15 /multimedia communication unit  16 /mail server  17  with a network, and an initial electronic tag generation unit  11  generating an initial multimedia electronic tag, based on member data and multimedia data.  
     [0052] A client  20  is a terminal used for each user to obtain a multimedia electronic tag service. Although there are a plurality of clients  20  with the same configuration in the network, they are omitted in FIG. 1.  
     [0053] The client  20  comprises a multimedia communication unit  22  exchanging multimedia data with a server, a camera  23  used for a user to generate multimedia data, an electronic tag communication unit  24  exchanging a multimedia electronic tag with a server and/or a client, an electronic mail processing unit  25  performing a variety of electronic mail processes (the generation of electronic mail/display screen to be presented to a user, electronic mail exchange between clients, and the like), an electronic tag buffer  28  storing multimedia electronic tags, a format conversion device  26  converting the format of a multimedia electronic tag into a multimedia synchronization/reproduction format, a multimedia synchronization/reproduction unit  27  synchronizing multimedia data with the multimedia electronic tag, the format of which is converted by the format conversion device  26 , in terms of time and space, an electronic tag editing unit  31  performing a variety of multimedia electronic tag processes (the display of a multimedia electronic tag to be presented to a user, the generation of a comment input screen, the update of a multimedia electronic tag and the like), a display unit  29  displaying screens generated by the multimedia synchronization/reproduction unit  27 , electronic tag editing unit  31  and electronic mail processing unit  25 , and a user input unit  30  composed of input devices, such as a keyboard, a mouse and the like.  
     [0054] A network  40  is used to reciprocally connect a multimedia server  10  and a client  20  using a TCP/IP protocol.  
     [0055]FIG. 3 is a flowchart showing the operation of the entire multimedia cooperative work system shown in FIG. 2.  
     [0056] In FIG. 3, first, the multimedia generation process in step S 1  is described below.  
     [0057] First, in an arbitrary client  20 , multimedia data (in this specification, in particular, the AV data described above, including a time factor, such as dynamic image data) are generated, based on image data taken by the camera  23  shown in FIG. 2. It does not necessarily mean that the camera  23  must be used together with a client system at the time of photographing. It is acceptable even if data are taken only by the camera  23  and the camera  23  is connected to the client  20  at the time of multimedia registration. Alternatively, dynamic image data are stored in a storage medium which can be freely attached to/removed from the camera  23  and this storage medium can be connected to the client  20  later. For a specific connection method, for example, a DV (digital video) method and the like is used. However, the connection method is not limited to this method.  
     [0058] Next, the multimedia registration process in step S 2  is described below.  
     [0059] The client  20  transmits the multimedia data generated in step S 1  to the server  10  through the network  40  using the multimedia communication unit  22 , for example, in response to a user&#39;s registration request.  
     [0060] In the server  10 , multimedia data received through the multimedia communication unit  16  is stored in the multimedia storage device  13 . Although for a specific transmission method, an HTTP protocol, etc., is used, the method is not limited to this.  
     [0061] In the server  10 , after the reception/storage of multimedia data are completed, an identifier is assigned to the multimedia data. Then, the multimedia communication unit  16  returns the identifier of the stored multimedia data to the multimedia communication unit  22  of the client  20 , for example, using an HTTP protocol. This multimedia identifier is, for example, composed of a communication protocol, a server name and a file name. In this example, it is assumed that an identifier of, for example, http://www.mediaserv.com/data — 1.mpg is assigned.  
     [0062] The multimedia communication unit  16  of the server  10  generates a new entry in the management information DB  14 .  
     [0063]FIG. 4 shows the internal data format of the management information DB shown in FIG. 2.  
     [0064] In FIG. 4, an entire table storing data is represented by  50 .  
     [0065] This table  50  is composed of the entries of the multimedia file name  51 , registrant identifier  52 , electronic tag file name  53  and member data  54 .  
     [0066] In the entry of the multimedia file name  51 , the file name of the multimedia data stored in the multimedia storage device  13  shown in FIG. 2 (the multimedia identifier) is stored. In this example, the file name “/data — 1.mpg” and the like of the example identifier are shown.  
     [0067] In the entry of the registrant identifier  52 , the identifier of a client that registers the multimedia data, is stored. Although in this example, this is an electronic mail address, the identifier is not limited to this.  
     [0068] In the entry of the electronic tag file name  53 , the file name of a multimedia electronic tag corresponding to the multimedia data (the meta-information of the multimedia data) stored in the electronic tag storage device  14  shown in FIG. 2, is stored.  
     [0069] In the entry of the member data  54 , the client identifier of a user sharing the multimedia data and multimedia electronic tag data, is stored (Although in this example, this is the electronic mail address of each client, the identifier is not limited to this).  
     [0070] In the process of step S 2 , in the entry  51  “multimedia file name” shown in FIG. 4, the identifier assigned to the stored multimedia is inputted. In the entry  52  “registrant identifier”, the client identifier (email address and the like) of a user (the user in Step S 1 ) that makes a request for registering the multimedia data, is inputted. The storage of the multimedia electronic tag file name  53  and member data  54  are described later in the processes of steps S 3  and S 4 .  
     [0071] Next, the member notification process in step S 3  is described below.  
     [0072] After making the server  10  perform multimedia registration and receiving the identifier, a user in the client  20  notifies each member (the users of other clients  20 ) by electronic mail of the fact that multimedia is registered in a server. This member is another user with which the user making a registration request wants to exchange a comment on the multimedia data. Comment exchange means to freely exchange opinions on an arbitrary multimedia data through a network, such as to attach a comment to an arbitrary scene of multimedia data, which is described later, and to further attach a comment to another person&#39;s comment from time to time.  
     [0073] In this case, electronic mail embedding the multimedia identifier received by the multimedia communication unit  22  in step S 2  is notified.  
     [0074] The electronic mail is transmitted to the client  20  of each member through the mail server  17  of the server  10 .  
     [0075] In this case, in the server  10 , the electronic mail address of the member described in the destination field data of the electronic mail that is stored in the mail server  17  is extracted and the embedded multimedia identifier described above is also extracted from the mail body. Then, the electronic mail address and multimedia identifier are registered in the management information DB  14 . Specifically, the management information DB  14  is retrieved using the extracted multimedia identifier (or the destination field data of the electronic mail) as a key, and the electronic mail address of each member (and a transmitter) is inputted to the entry  54  “member data” corresponding to the corresponding entry  51  “multimedia file name” (although not shown in FIG. 4, a real name can also be inputted).  
     [0076] Next, the initial electronic tag generation process in step S 4  is described below.  
     [0077] In the multimedia server  10 , after the electronic mail is transferred, the initial electronic tag generation unit  11  generates the model of a multimedia electronic tag, based on both the information obtained in step S 3  and the multimedia data stored in step S 2 , and the electronic tag storage device  12  stores the model. This model is one provided with no comment, of the multimedia electronic tags shown in FIGS. 5 and 6, which is described later.  
     [0078] The initial electronic tag generation unit  11  is not automated so a person generates the model of the multimedia electronic tags using an existing editing device. In this case, the multimedia identifier  51  and member data  54  are read from the management information DB 14 , and also the entity of a multimedia data (AV data) corresponding to the multimedia identifier  51  read from the management information DB  14  is read from the multimedia storage device  13 . All the three pieces of data are inputted to the initial electronic tag generation unit  11  and are used to generate the model of a multimedia electronic tag.  
     [0079] Although the model of a multimedia electronic tag is described with reference to a specific example of the multimedia electronic tag shown in FIGS. 5 and 6, which is described later, a scene cutting method needed to generate segment data (to divide the entity of multimedia data into a plurality of scenes in terms of time and to manage the scenes in tree-shape structure) is assumed to be publicly known. Specifically, for this method, MPEG-7 (ISO/IEC 15938), which is currently being standardized by ISO/IEC, is used. The formal name of MPEG-7 is “Multimedia Content Description Interface”. MPEG-7 realizes the description of the internal structure (time sequence) of multimedia data, that is, the description of information of each scene which is obtained by dividing the multimedia data (description on when (what hour what minute what second) each scene starts at and when (what hour what minute what second) the scene ends).  
     [0080] Then, the intra-server identifier of a newly generated multimedia electronic tag is assigned to the model of a multimedia electronic tag and the model is linked to the identifier of the multimedia data. Then, the model is stored in the management information DB  14 . Specifically, the electronic tag storage device  12  stores/manages the data of the generated multimedia electronic tag model (initial electronic tag). An identifier is assigned to this initial electronic tag. This electronic tag identifier is transmitted to the management information DB  14  and is inputted to the corresponding entry  53  “electronic tag file name”.  
     [0081] After the processes in steps S 1  through S 4  are completed, each user (including a registrant) can refer to each comment, can attach a desired comment to an arbitrary scene at a desired time and can also attach a comment to a comment. In this way, a dynamic image with a comment that varies depending a scene can also be viewed. Processes for realizing such a user service (steps S 5  through S 8 ) are described below.  
     [0082] First, the electronic tag acquisition process in step S 5  is described.  
     [0083] Each user of another client  20  knows that the corresponding electronic tag is available by receiving the electronic mail in the process of above step S 3 , including information about the multimedia identifier.  
     [0084] In the client  20 , if, for example, the user makes a request for using an electronic tag, the electronic tag communication unit  24  issues a request to the electronic tag communication unit  15  of the multimedia server  10  for a multimedia electronic tag (for example, using an HTTP protocol) using the multimedia data identifier described in the electronic mail received in step S 3  as a key.  
     [0085] The electronic tag communication unit  15  of the multimedia server  10  makes an inquiry to the management information DB  14  for the identifier of the corresponding multimedia tag data, based on the received multimedia data identifier and reads multimedia electronic tag data from the electronic tag storage device  12 , using the obtained identifier. Then, the unit  15  transmits the multimedia electronic tag data to the client, for example, using an HTTP protocol. In this case, if the requesting client is not registered in the management information DB  14 , the request can also be refused.  
     [0086] In the requesting client  20 , the obtained multimedia tag data are stored in the electronic tag buffer  28 .  
     [0087] It is acceptable if, for example, multimedia electronic tag data obtained by this client attaching a comment can also be directly transmitted from the client using, for example, an HTTP protocol.  
     [0088] Next, the comment input process in step S 6  is described below.  
     [0089] The user of another client  20  can add his/her comment to an obtained multimedia electronic tag, as necessary. For this purpose, the electronic tag editing unit  31 , display unit  29 , and user input unit  30  are used. The editing result is stored in the electronic tag buffer  28 .  
     [0090] This process is described in detail later with reference to FIGS. 5, 6 and  7 .  
     [0091] Next, the multimedia synchronous reproduction in step S 7  is described below.  
     [0092] On each client  20  sides, a comment described in a multimedia electronic tag can be synchronized with a multimedia and be displayed, as necessary. For this purpose, both the format conversion device  26  and multimedia synchronous reproduction unit  27  are used.  
     [0093] The format conversion device  26  converts the format of a multimedia electronic tag stored in the electronic tag buffer  28 , for example, into the SMIL (Synchronized Multimedia Integration Language) of W3C standard (the conversion method is described later). The format conversion device  26  is, for example, an XSLT (Extensible Style Language Translator) processing system stipulated by W3C.  
     [0094] The multimedia synchronous reproduction unit  27  is, for example, an SMIL player, and synchronizes/reproduces multimedia data and comments thereof using time control data described in a multimedia electronic tag, the format of which is converted into SMIL by the format conversion device  26  in response to a user&#39;s synchronous reproduction request. The reproduction result is displayed in the display unit  29 .  
     [0095] The multimedia communication unit  22  obtains the multimedia data by communicating with the multimedia communication unit  16  of the server  10 .  
     [0096] More specifically, the multimedia communication unit  22  of the client  20  notifies the multimedia communication unit  16  of the server  10  of the “src” attribute (described later) of the tag &lt;video&gt; of the SMIL data inputted to the multimedia synchronous reproduction unit  27  as a multimedia identifier.  
     [0097] The multimedia communication unit  16  of the server  10  extracts the corresponding multimedia data from the multimedia storage device  13  using the multimedia identifier, and transmits the multimedia data to the multimedia communication unit  22  using, for example, an HTTP protocol.  
     [0098] Each of a specific example of a multimedia electronic tag, the format of which is converted into SMIL by the format conversion device  26  and a specific example of the synchronous reproduction of multimedia data and comments thereof using the multimedia electronic tag is described later.  
     [0099] Lastly, the electronic tag transmission process in step S 8  is described below.  
     [0100] The electronic tag communication unit  24  transmits the multimedia electronic tag, the content of which has been updated by a user adding comments and the like in the comment input process in step S 6 , to the electronic tag communication unit  15  of the server  10  together with the corresponding multimedia identifier (described in the electronic tag). Since, once receiving a multimedia electronic tag, each user can identify the identifier of the multimedia electronic tag, this electronic tag identifier can also be directly designated.  
     [0101] An electronic tag identifier can be obtained in the same way as in the electronic tag acquisition process in step S 5 , and the multimedia electronic tag data are stored in the electronic tag storage device  12 .  
     [0102] Alternatively, a multimedia electronic tag modified by a user can also be directly distributed to other members instead of distributing it through the server  10 , as necessary.  
     [0103] Next, it is assumed that a plurality of users perform the comment input/addition process shown in step S 6 , using the multimedia electronic tag model generated by the processes in step S 1  through S 4 . FIGS. 5 and 6 show a specific example of a multimedia electronic tag in this case. The electronic tag transmission process is described in more specific detail below with reference to FIGS. 5 and 6.  
     [0104] A multimedia electronic tag is, for example, described in XML (Extensible Markup Language), as shown in FIGS. 5 and 6. This is just one example, and the language is not limited to XML.  
     [0105]FIGS. 5 and 6 show the entire description of one multimedia electronic tag, which is divided into two portions for convenience&#39; sake and each of the two portions is shown in FIGS. 5 and 6.  
     [0106] The manager and the like of the multimedia server  10  side can basically determine the description of each tag described below arbitrarily. It is also assumed that the meaning (structure) of each tag described below is determined by the manager and the like of the multimedia server  10  side and is defined in DTD (Document Type Definition), which is not shown in FIGS. 5 and 6.  
     [0107] A multimedia electronic tag is largely composed of the following four descriptions (a) through (d).  
     [0108] (a) URL of multimedia entity  
     [0109] (b) Member data  
     [0110] A variety of information (name, electronic address, etc.) about users permitted to participate in the events (commenting, editing, opinion exchange, etc.) of a multimedia  
     [0111] (c) Description on the time sequence of multimedia data  
     [0112] Multimedia data are divided into time blocks (scenes) and the information of each scene is described. This described content is composed of the time data of all the scenes (offset from top, scene time, etc.). In order to collectively handle a plurality of scenes consecutive in terms of time as a high-order scene, description on scene data can also include description on a low-order scene or reference data about the scenes.  
     [0113] (d) Description of a user comment  
     [0114] Each user comment is configured so that the entity or reference data can be attached to the description on scene data. A user comment is comprised of a comment entity (which is also comprised of text, icons, static images, etc.), comment writer data (name, mail address, etc.) or reference data about comment writer data, reference data about a referred comment (information indicating the original comment to which a comment is made), comment time data (preparation date, expiration date, etc.) and comment publication scope data (publication is limited to special members). Of these items, a plurality of pieces of information except for the comment entity are called “(comment) attribute data”.  
     [0115] Basically each client has the multimedia electronic tag browser function and comment input operation function. In particular, using the input operation function, a user can input the addition destination scene, addition destination comment, publication scope, time data (expiration date, etc.). Using the browser function, the time data of each comment and the current time can be compared and only valid (non-overdue) comments can be displayed. Alternatively, the server  10  can also be provided with a function to delete overdue comments from a multimedia electronic tag.  
     [0116] When transmitting a multimedia electronic tag to a client, the multimedia server  10  can compare the user identifier of a client with comment publication scope data for each comment, and can transmit only comments, the publication of which is permitted.  
     [0117] Detailed descriptions of the multimedia electronic tags shown in FIGS. 5 and 6 are given.  
     [0118] In FIG. 5, portion A is route tag &lt;AVTag&gt; declaring that this XML document is a multimedia electronic tag. This route tag has an “updated_date” attribute indicating the latest modification date (date when this XML document has been modified last) and a “modifier” attribute indicating the intra-system identifier of the modifier (in this example, electronic mail address). In the example shown in FIG. 5, a user, Suzuki@aaa.bbb.jp has modified the content of the XML document at 11 o&#39;clock, Dec. 1, 2000.  
     [0119] Portion B is a tag aggregate indicating member data. Tag &lt;UserList&gt; at top is a “wrapper” used to describe member data.  
     [0120] Tag &lt;User&gt; is used to describe individual member data, and has an “id” attribute used to refer to member data in another place of the XLM document. An individual “id” attribute value shall be unique in an XML document. In the example shown in FIG. 5, as this “id” attribute of member data, id=“u1”, id=“u2”, and id=“u3” are assigned to Ichiro Tanaka, Taro Suzuki, and Shiro Sato, respectively.  
     [0121] Tag &lt;Name&gt; is used to describe the name of a user. A first name and a family name are described in tags &lt;FirstName&gt; and &lt;FamilyName&gt;, respectively. Although a family name and a first name must not always be described separately, in this example, they are separated in relation to an example display, which is described later, (in which only a family name is displayed). Therefore, only the family name of a user, only the first name or both the family and first names can be described using only tag &lt;Name&gt;.  
     [0122] Tag &lt;Email&gt; is used to describe a user identifier in the system (in this example, electronic mail address).  
     [0123] The contents of tags &lt;User&gt; and &lt;Email&gt; are described referring to the member data  54  in the management data DB  14  in the process of step S 4  shown in FIG. 3 (generation of a multimedia electronic tag model). In the example shown in FIG. 5, it is a multimedia electronic tag corresponding to a multimedia identifier=http://www.mediaserv.com/data — 1.mpg, and the corresponding member data  54  in FIG. 4 is obtained in this way. As a result, the real member names of Ichiro Tanaka, Taro Suzuki and Shiro Sato, and their electronic mail addresses are described.  
     [0124] Portion C is tag &lt;MediaURI&gt; used to describe a multimedia identifier corresponding to the multimedia electronic tag. In this example, the corresponding multimedia is a file name, “datal.mpg” (MPEG-1 dynamic image) that is stored in a server, www.mediaserv.com, and it means that it can be obtained using an HTTP protocol. This is also described in the model generation of the process in step S 4  using the information of the multimedia file name  51  in the management information DB  14 .  
     [0125] Portion D is composed of tag &lt;Segment&gt; describing the highest-order segment in the time sequence of multimedia data (id of the segment=“root_seg”) and user comments attached to the highest-order segment. User comments are not described in the model generation step.  
     [0126] Tag &lt;Image&gt; is used to describe the URL of the representative image of an attached segment. When a multimedia electronic tag is displayed in the client  20  for comment input, representative image data are obtained from the server  10  and are displayed using, for example, an HTTP protocol  
     [0127] Tag &lt;UserLabel&gt; is the “wrapper” of a comment attached to this segment. Each comment is described using tag &lt;Label&gt;.  
     [0128] Tag &lt;Label&gt; has an “id” attribute indicating a comment identifier, a “userref” attribute indicating the reference of a comment writer (the reference destination of which is stored in tag &lt;UserList&gt;) and an “expiration_date” attribute indicating the expiration date of a comment.  
     [0129] In the comment identifier, for example, the “id” attribute of “comment No. 2” is id=“com — 1”. This indicates that “comment No. 2” is comment relation to the comment of id=“com — 1” (the comment of “comment No. 1”). This is just one example, and description on “id” attribute is not limited to this example.  
     [0130] Tag &lt;Comment&gt; is used to describe a specific comment content (in a text format). Although in FIG. 5, it is described “comment No. 1”, “comment No. 2” and the like, in reality, some comment sentences inputted by each user are described.  
     [0131] Although in this example, a comment content is in a text format, the format is not limited to text. For example, icon data (entity or referrer) and the like can be used.  
     [0132] In this case, at the time of the generation of the multimedia electronic tag model shown in step S 4 , tags &lt;Label&gt; and &lt;Comment&gt; are not described. These portions will be added and updated every time a user attaches a comment in each client  20 .  
     [0133] At the time of the model generation, tags &lt;Segment&gt; and &lt;Image&gt; are described, and tag &lt;UserLabel&gt;, which is a comment “wrapper”, is set.  
     [0134] For example, in the example shown in FIG. 5, although the URL of a representative image=http://www.mediaserv.com/root_seg.jpg is described in tag &lt;Segment&gt;, for example, in steps S 1  and S 2 , the user of a client requesting the registration of multimedia data arbitrarily determines this representative image (a static image extracted from multimedia data) and transmits the representative image to the server  10  together with the multimedia data. Then, the server  10  assigns an identifier (URL, etc.) to this representative image file. Although the process also applies to a representative image in a low-order segment, which is described later, in that case, a user instructs the server  10  how to divide multimedia data and also selects a representative image for each divided scene. Then, the user also transmits information indicating which scene each representative image represents, to the server  10  together with the multimedia data.  
     [0135] Alternatively, at the time of the process of step S 4 , for example, the operator of the server  10  can refer to multimedia data (dynamic image) read from the multimedia storage device  13  and can arbitrarily select a screen (static image) that should become a representative image. Then, the operator can arbitrarily determine the file name (URL) of this static image.  
     [0136] In this case, the operator also arbitrarily specifies the time sequence (tree-shape structure) of the multimedia data as in tag &lt;Segment&gt;, and the low-order segment (descriptions in portions F and G, which are described later).  
     [0137] Tag &lt;TargetUser&gt; is an optional tag. A default state where there is no tag &lt;TargetUser&gt;(specifically, a comment with the “id” attribute of “com1” and “com2” in portion D) means that this comment should be made public to all members.  
     [0138] If users to which multimedia data should be made public are designated by tag &lt;TargetUser&gt; like a comment with the “id” attribute of “com1 — 1” in portion D, it means that this comment data should be transmitted to only the users. In this example, it means that the comment with the “id” attribute of “com1 — 1” (comment No. 2) is directed to only a member, the member data “id” attribute of which is id=“u1”, that is, Ichiro Tanaka.  
     [0139] The electronic tag storage device  12  stores in advance, for example, a multimedia electronic tag, including such tag &lt;TargetUser&gt;. In response to a user&#39;s request, the electronic tag communication unit  15  of the multimedia server  10  transmits this entire multimedia electronic tag to users Tanaka (publication destination user) and Suzuki (comment writer), and transmits a multimedia electronic tag without “comment No. 2” to user Sato.  
     [0140] When a client directly transmits an edited multimedia electronic tag to another client (in this example, if the client of user Suzuki transmits the multimedia electronic tag shown in FIG. 5 to users Tanaka and Sato), the electronic tag communication unit  24  of the client of user Suzuki transmits the multimedia electronic tag shown in FIG. 5 to the multimedia server  10  and the client of user Tanaka without deleting “comment No. 2”. However, the electronic tag communication unit  24  transmits the multimedia electronic tag shown in FIG. 5 without “comment No. 2”.  
     [0141] Portion E is a tag aggregate used to describe the time data of a segment “root_seg”. Tag &lt;MediaTime&gt; at top is a “wrapper”. Tag &lt;Offset&gt; indicates the start time of a segment (offset from the beginning of data). In this example, it indicates that the start time of the segment is the beginning of data (that is, offset is 0). Tag &lt;Duration&gt; indicates the time length of a segment. In this example, it indicates that the time length is 10 minutes 20 seconds.  
     [0142] The description of F portion, G portion, etc., shown in FIG. 6 follows the description of the E portion shown in FIG. 5.  
     [0143] In FIG. 6, each of F portions and G is tag &lt;Segment&gt;describing one of two low-order segments, included in the highest-order segment “root_seg” (the respective “id” attributes of the segments are id=“seg — 0” and id=“seg1) and a user comment attached to the respective two segments, respectively. In other words, they are a description off each scene obtained by dividing multimedia data in terms of time and a description on a user comment attached to each scene, respectively. In the example shown in FIG. 6, they indicate that the multimedia data have two layers and the number of the second layer is two.  
     [0144] Such a hierarchical structure is indicated by a range relation specified in each tag &lt;Segment&gt;(so-called “nest relation”). Specifically, the start tag of the highest-order segment “root_seg” is described at the top of portion D, and an end tag (/Segment) is described below portion G (immediately above tag &lt;/AVTag&gt; that is described last). Other tags &lt;Segment&gt;described between the start and end tags are low-order segments, as shown in FIG. 6.  
     [0145] Therefore, in order to generate a further lower-order segment below the first low-order segment (to generate three-layer structure), it is acceptable if a new tag &lt;Segment&gt; is described between the start tag (&lt;Segment id=“seg — 0”&gt;) and end tag (&lt;/Segment&gt;described at the end of portion F).  
     [0146] As shown in FIGS. 5 and 6, the relation between comments can also be expressed by so-called “parentage” and “brotherhood”.  
     [0147] Since the descriptive method of tags &lt;Segment&gt; in portions F and G is basically the same as that of the highest segment “roor_seg” in portion D, it is only briefly described here.  
     [0148] First, as described in a tag aggregate (tags &lt;MediaTime&gt;, &lt;Office&gt; and &lt;Duration&gt;) used to describe time data described near the tail, the segment of a segment id=“seg — 0” in portion F (hereinafter called the “first low-order segment) indicates that the first low-order segment starts from data top (offset is “0h0m0s”) and has the time length of 5 minutes 20 seconds.  
     [0149] Similarly, as described in the tag aggregate used to describe time data, the segment of a segment id=“seg — 1” in portion G (hereinafter called the “second low-order segment) indicates that the second low-order segment starts from a point 5 minutes  20  top (offset is “0h5m20s”) seconds away from the beginning of data and has the time length of 5 minutes (in other words, the second low-order segment covers a time range between 5 minutes 20 seconds and 10 minutes 20 seconds).  
     [0150] In the example shown in FIG. 6, there is no time overlapping between two low-order segments, and time range covered by them is the same as that of a parent segment (in this case, the highest-order segment). However, this is just one example, and the setting is not limited to this. As described above, the operator and the like of the server  10  can determine what is the time range, how many low-order segments should be provided, or how many layers the hierarchy should have, arbitrarily (or based on the requesting user&#39;s desire).  
     [0151] As described above, in the example shown in FIG. 6, the URL of the representative image of the first and second low-order segment are http://www.mediaserv.com/seg — 1.jpg and http://www.mediaserv.com/seg — 2.jpg, respectively.  
     [0152] “Comment No. 4” and “comment No. 5” are attached to the first and second low-order segment, respectively. Therefore, as described above, “comment No. 4” is displayed while multimedia data are reproduced between top and 5 minutes 20 seconds, and “comment No. 5” is displayed between 5 minutes 20 seconds and 10 minutes and 20 seconds. “Comment No. 1” through “comment No. 3” are always displayed while multimedia data are reproduced, since they are attached to the highest segment.  
     [0153] In this way, according to the present invention, a comment can be attached to the entire multimedia data or an arbitrary one of the scenes obtained by dividing multimedia data in terms of time (or another comment). A comment writer name, a comment generation date, a comment destination (to which scene or whose comment a comment is attached) and the like can also be displayed.  
     [0154] Furthermore, a specific example of the comment display/input screen is described below.  
     [0155]FIG. 7 shows one example of the comment list display/comment input screen of a multimedia electronic tag displayed in each client. A case where the server  10  receives and displays a multimedia electronic tag with the contents shown in FIGS. 5 and 6 is shown. It is assumed that each client  20  is provided with a browser function to display an XML document (there is such an existing tool). It is assumed that as in a prior art, a screen, including buttons and a comment input column as shown in FIG. 7 is displayed, which is not shown nor described in FIG. 7 and are not described, using an HTML document specifying the display format, XSL (XSLT) and the like. In the example, it is assumed that the format of a multimedia electronic tag received from the server  10  is converted into a prescribed display format by the electronic tag editing unit  31  shown in FIG. 2, and a screen as shown in FIG. 7 is displayed by the display unit  29 .  
     [0156] In FIG. 7, the entire comment display/input screen is represented by  60 .  
     [0157] A high-order segment display area  61  displays comments attached to the highest-order segment and the representative image thereof. Information about the highest-order segment corresponds to a portion beginning with tag &lt;Segment&gt; in portion D shown in FIG. 5.  
     [0158] Buttons  62  are used to designate a target comment to which a new comment is attached. The button  62  is not limited to the example display, and the display format varies depending the content of the HTML document, XSL (XSLT) and the like.  
     [0159] If a user clicks a desired button  62  using, for example, a mouse, the designation of a comment corresponding the button  62  is displayed (in the example shown in FIG. 7, check is marked) and it is interpreted that a new comment inputted to a comment input area  68 , which is described later, corresponds to a comment to be attached to the comment designated by the button  62 . Then, the corresponding description is attached to the multimedia electronic tag. In this way, the content of a multimedia electronic tag continues to be updated every time a new comment is attached. In the example shown in FIG. 7, it means that a new comment is attached to “comment No. 1” given by user Tanaka.  
     [0160] The name of a comment writer is represented by  63 . This is generated using the “userref” attribute of tag &lt;Label&gt; in portion D and information about tag &lt;Name&gt;in portion B that are shown in FIG. 5 (although in this example, only a family name is displayed using information about tag &lt;FamilyName&gt; and not using information about tag &lt;FirstName&gt;, it is not limited to this).  
     [0161] In this example, a comment writer name is displayed as one example of the comment attribute data, and attribution data is not limited to this. Therefore, for example, a comment generation date and the like can also be displayed instead.  
     [0162] The content of a comment is represented by  64 . This is generated using the information of each tag &lt;Comment&gt;in portion D shown in FIG. 5.  
     [0163] Each of  62 ,  63  and  64  is generated for each comment, and they are displayed in their addition order from top to bottom on the screen. As shown in FIGS. 5 and 6, a comment on a comment is indented and displayed. In the example shown in FIG. 7, it is indicated that on user Suzuki&#39;s comment “comment No. 2” is attached to user Tanaka&#39;s comment “comment No. 1”.  
     [0164] An image  65  is a representative image attached to a segment. The display image is reproduced using data referenced using an URL described in tag &lt;Image&gt; in portion D shown in FIG. 5.  
     [0165] Display areas  66  and  67  display the comment contents of the low-order segments (first and second low-order segments) of a segment “root_seg” described in the respective tags &lt;Segment&gt; in portions F and G. The structure is the same as that of the display area  61  of a high-order segment. Each of the areas  66  and  67  displays the representative image of each low-order segment and the comment thereof. Each of the areas  66  and 67 also displays a comment on a comment like the high-order segment display area  61 .  
     [0166] The respective display positions of the areas  66  and  67  are below the high-order segment display area  61  in the example shown in FIG. 7. If there are a plurality of low-order segments, they shall be displayed from left to right in time sequence order.  
     [0167] In order to attach a comment to each segment instead of a comment in the high-order segment display area  61 , display area  66  and display area  67 , it is acceptable, for example, if an area where the representative image is displayed is clicked using a mouse and the like.  
     [0168] In a comment input area  68 , a user viewing the comment display/input screen  60  attaches a new comment to the designated segment or comment after designating a desired segment or comment in the high-order segment display area  61 , display area  66  or display area  67 .  
     [0169] In a publication user designation area  69 , the publication destination of a newly attached comment is selected and inputted. Selection buttons and the name of each member are represented by  69   a  and  69   b , respectively. If a user clicks a desired button  69   a  using, for example, a mouse and the like, the selection is displayed (in the example shown in FIG. 7, check is marked) and the selection result is reflected (specifically, if a specific user is designated as the publication destination, tag &lt;TargetUser&gt; shown in FIG. 5 is attached to the newly attached comment). In the example shown in FIG. 7, all-member publication is selected and no tag &lt;TargetUser&gt; is attached.  
     [0170] A “send” button  70  is used to start an operation to transmit an edited multimedia electronic tag to a multimedia server or client.  
     [0171] A “reproduce” button  71  is used to start an operation to synchronize/reproduce an edited multimedia electronic tag and the corresponding multimedia.  
     [0172] If this “reproduce” button is designated, the format conversion device  26  converts the format of a multimedia electronic tag into a multimedia synchronous reproduction format.  
     [0173] The process operation of this format conversion device  26  is described below with reference to FIGS. 8 through 13.  
     [0174] In this example, it is assumed that this conversion into a multimedia synchronous reproduction format is performed by SMIL format conversion.  
     [0175]FIG. 8 is a flowchart showing the summary of the entire SMIL conversion process.  
     [0176] First, portions A and B of a multimedia electronic tag shown in FIG. 5 are outputted (step S 11 ). The contents are fixed.  
     [0177] Then, portion J (tag &lt;video&gt;) shown in FIG. 12, which is described later, is generated/outputted (step S 12 ). The details of this process are described later with reference to FIG. 9.  
     [0178] Then, portion K (tag &lt;text&gt;) shown in FIG. 12, which is described later, is generated/outputted (step S 13 ). The details of this process are described later with reference to FIG. 10.  
     [0179] Lastly, the remaining portions are outputted (step S 14 ). The contents are fixed.  
     [0180]FIG. 9 is a flowchart showing the detailed process in step S 12  of FIG. 8.  
     [0181] In FIG. 9, first, tag &lt;media URI&gt; is retrieved from a conversion source file (multimedia electronic tag) and the information (URI of the multimedia data) is obtained. Then, the “src” attribute of tag &lt;video&gt; is generated (step S 21 ).  
     [0182] Since in the example shown in FIG. 5, the URI of the multimedia data is http://www.mediaserv.com/data — 1.mpg as shown in portion C, the “src” attribute of tag &lt;video&gt; becomes as shown in portion J of FIG. 12.  
     [0183] Then, the tag &lt;MediaTime&gt; of the highest-order segment (tag &lt;MediaTime&gt; of portion E shown in FIG. 5) is retrieved, and the values of “begin” attribute (Offset data) and “end” attribute (a value obtained by adding the value of tag “Duration” to the value of tag “Offset”) of tag &lt;video&gt; are generated using the information of tags &lt;Offset&gt; and &lt;Duration&gt; of tag &lt;MediaTime&gt; (step S 22 ).  
     [0184] Lastly, tag &lt;video&gt; is completed by adding the value (fixed) of “region” attribute (in the example shown in FIG. 12, region=“video — 0”) to each of the attribute values (step S 23 ).  
     [0185]FIG. 10 is a flowchart showing the detailed process in step S 13  shown in FIG. 8.  
     [0186] First, a stack temporarily storing comment data, which is not shown in FIG. 10, is cleared (initialized) (step S 31 ).  
     [0187] Then, tag &lt;Segment&gt; is retrieved from the top of an electronic tag (step S 32 ). If tag &lt;Segment&gt; is discovered, the process proceeds to step S 33 . If tag &lt;Segment&gt; is not discovered, the electronic tag is not legal. Therefore, the process is stopped.  
     [0188] In step S 33 , first, comment data are generated based on information of tag &lt;UserLabel&gt; appearing immediately after the discovered tag &lt;Segment&gt;. A comment character string is obtained from tag &lt;comment&gt; in each tag &lt;Label&gt; of tag &lt;UserLabel&gt;, and the family name of a user is obtained from “userref” attribute, and the tags &lt;Name&gt;/&lt;FamilyName&gt; of tag &lt;UserLabel&gt;. Then, a final comment character string is generated by combining the comment character string and the family name. If tag &lt;Label&gt; is included in another tag &lt;Label&gt;, a plurality of blanks are inserted in the top of the comment character string depending on the depth (nesting stage). The comment character string obtained in this way (for the number of tags &lt;Label&gt;) are “pushed” into the stack, as comment information. In order to separate the comment from the comment of another layer (in order to separate the comment from a comment obtained by applying the process in step S 33  to a low-order segment that is discovered in the process in steps S 34  or S 36 , which are described later), a character string for separation, such as “------” is additionally “pushed” into the stack.  
     [0189] Lastly, the content of tag &lt;MediaTime&gt; appearing immediately after tag &lt;/UserLabel&gt; (tags &lt;Offset&gt; and &lt;Duration&gt;) is stored.  
     [0190] Then, tag &lt;Segment&gt; or &lt;/Segment&gt; is retrieved from the current position in the direction of the file tail (step S 34 ). If tag &lt;Segment&gt; is discovered (there is a low-order segment), the process returns to step S 33 . If tag &lt;/Segment&gt; is discovered, the process proceeds to step S 35 .  
     [0191] In step S 35 , first, the current stack content is stored in a file. The file name is assumed to be unique. Then, tag &lt;text&gt; is generated based on the file name and the content of the stored tag &lt;MediaTime&gt;. If there is the “pushed” comment data on the low-order segment, the comment data are discarded as “pop”. The boundary between the “pushed” comment data on the low-order segment and the “pushed” comment data on the high-order segment can be recognized by a separation character string, such as “-----” described above.  
     [0192] The details are described later with reference to a specific example shown in FIG. 11.  
     [0193] Then, in step S 36 , tag &lt;Segment&gt; is retrieved from the current position in the direction of the file tail. If tag &lt;Segment&gt; is discovered, the process moves to step S 33 . If tag &lt;Segment&gt; is not discovered, the process is terminated.  
     [0194]FIG. 11 shows the transition of the stack and content of the stored tag &lt;MediaTime&gt; that is obtained by applying the process shown in FIG. 10 to the multimedia electronic tag shown in FIG. 5.  
     [0195] First, the first process target in step S 33  after the start of the process is the highest segment in portion D shown in FIG. 5.  
     [0196] As shown in portion D of FIG. 5, “comment No. 1”. “Comment No. 2” and “comment No. 3” are attached to this highest-order segment, and each of these is sequentially “pushed” into the stack. Lastly, a separation character string, such as “-----”, is additionally “pushed” into the stack. As a result, the stack content shown in line  71  of FIG. 11 is obtained.  
     [0197] Since the content of tag &lt;MediaTime&gt; stored lastly in the first step S 33  is the same as the described content of portion E shown in FIG. 5, the content becomes as shown in line  71  of FIG. 11.  
     [0198] If the first step S 33  is completed and in succession the process in step S 34  is performed, the tag &lt;Segment&gt; of portion F shown in FIG. 6 (&lt;Segment id=“seg — 0”&gt;) is discovered. Therefore, the process returns to step S 33  (line  72  of FIG. 11).  
     [0199] Then, in the second step S 33 , “comment No. 4” is “pushed” into the stack and the stack content becomes as shown in line  73  of FIG. 11. Since the stored content of tag &lt;MediaTime&gt; is replaced with the content of the tag &lt;MediaTime&gt; in portion F in the first step S 33 , the content becomes as shown in line  73  of FIG. 11.  
     [0200] Then, in the second step S 34 , tag &lt;/Segment&gt; lastly described in portion F is discovered, the process proceeds to step S 35 .  
     [0201] In the second step S 35 , as described above, first, the current stack content (stack content described in line  73  of FIG. 11, that is, “comment No. 1” through “comment No. 4”) is stored in a file. The file is assumed to be named “comment — 1.txt” in relation to the example shown in portion K of FIG. 12. Then, tag &lt;text&gt; is generated based on the file name and the content of the stored tag &lt;MediaTime&gt;. In this example, tag &lt;text&gt; representing the upper half of portion K shown in FIG. 12 is generated. Specifically, tag &lt;text&gt; in which “src” attribute is the file name “comment — 1.txt” and “begin”/“end” attributes are the “Offset” value (0h0m0s), which is the content of the stored tag &lt;Media Time&gt;/this “Offset” value plus “Duration” value (0h5m20s), respectively, is generated (“region” attribute is fixed).  
     [0202] Lastly, the content stored up to the separation character string “-----” of the stack (in this example, only “comment No. 4”) is “popped” and discarded from the stack. As a result, the stored content of the stack at the time of the completion of the second step S 35  becomes as shown in line  74  of FIG. 11.  
     [0203] Then, since in the second step S 36 , tag &lt;Segment&gt; in portion G of FIG. 6 ((&lt;Segment id=“segl”&gt;) is discovered, the process returns to step S 33  (line  75  in FIG. 11).  
     [0204] Then, in the third step S 33 , “comment No. 5” is “pushed” into the stack. As a result, the stack content becomes as shown in line  76  of FIG. 11.  
     [0205] The stored content of tag &lt;MediaTime&gt; is replaced with the content of tag &lt;MediaTime&gt; in the portion G. As a result, the stored content becomes as shown in line  76  of FIG. 11.  
     [0206] Then, since in the third step S 34 , tag &lt;/Segment&gt; lastly described in portion G is discovered, the process proceeds to the third step S 35 .  
     [0207] In the third step S 35 , as described above, first, the current stack content (stack content described in line  76  of FIG. 11, that is, “comment No. 1” through “comment No. 3” and “comment No. 5”) is stored in a file. The file is assumed to be named “comment — 2.txt” in relation to portion K shown in FIG. 11. Then, tag &lt;text&gt; is generated based on the file name and the content of the stored tag &lt;MediaTime&gt;. In this example, tag &lt;text&gt;representing the lower half of the portion K shown in FIG. 11. Specifically, tag &lt;text&gt; in which the “src” attribute is the file name “comment — 2.txt” and the “begin”/“end” attributes are the “Offset” value (0h5m20s) of the content of the stored tag &lt;MediaTime&gt;/the “Offset” value plus “Duration ” value (0h10m20s), respectively, is generated (“region” attribute is fixed).  
     [0208] Lastly, the content stored up to the separation character string “----” of the stack (in this example, only “comment No. 5”) is popped and discarded. As a result, the stored content of the stack at the time of completion of step S 35  becomes as shown in line  77  of FIG. 11.  
     [0209] Then, if in the third step S 36 , tag &lt;/Segment&gt;described immediately after portion G shown in FIG. 6 (end tag corresponding to the highest-order segment), the entire process shown in FIG. 10 is terminated.  
     [0210]FIG. 12 shows the result of converting the format of the multimedia electronic tag shown in FIGS. 5 and 6 into a multimedia synchronous reproduction format (in this example, SMIL format) by the processes described with reference to FIGS. 8 through 11.  
     [0211] In FIG. 12, description enclosed by a frame  81  is a SMIL main body.  
     [0212] In FIG. 12, SMIL document declaration by tag &lt;smil&gt; and screen layout designation by tag &lt;layout&gt; are described in portion H. In the example shown in FIG. 12, it is assumed that a text display area “text — 0” and a dynamic-image display area “video — 0” are declared and the content is predetermined.  
     [0213] Portion I is the top of each synchronous reproduction control data of a dynamic image and text that are described in tag &lt;body&gt;.  
     [0214] In portion J, first, tag &lt;par&gt; means to reproduce an object in parallel (to simultaneously reproduce a plurality of objects with a different display area). Tag &lt;video&gt; declares a dynamic image object (comment). “Src” attribute, “region” attribute, “begin” attribute and “end” attribute describe the URL of a dynamic image (including voice), a plot position, a reproduction start time and a reproduction end time, respectively. In K portion, tag &lt;seq&gt; means to reproduce an object in series (to sequentially reproduce a plurality of objects with the same display area in terms of time). Tag &lt;text&gt; declares a text object (comment). The meaning of the attribute is the same as that of tag &lt;video&gt;. “Comment — 1.txt” and “comment — 2.txt” are files generated in the course of a multimedia electronic tag conversion process, as described above, and the contents of the files are shown in portions enclosed by frames  82  and  83  in FIGS. 13A and 13B, respectively. This has been already described with reference to FIG. 11.  
     [0215] If this SMIL file is reproduced, dynamic images/voice and the content of “comment — 1.txt” are displayed for the first 5 minutes 20 seconds. Dynamic images/voice and the content of “comment — 2.txt” are displayed for 5 minutes from 5 minutes 20 seconds until 10 minutes 20 seconds.  
     [0216]FIG. 14 shows this reproduction screen display. A dynamic image display portion and a comment display portion are represented by  91  and  92 , respectively.  
     [0217] Lastly, the respective hardware configurations of the client  10  and multimedia server  20  are described.  
     [0218] The client  10  can be implemented by a general-purpose computer.  
     [0219]FIG. 15 shows one example of the basic hardware configuration of such a computer.  
     [0220] The data processing device  100  shown in FIG. 15 comprises a CPU  101 , a memory  102 , an input device  103 , an output device  104 , a storage device  105 , a medium driving device  106  and a network connection device  107 , and these components are connected to one another by a bus  108 . The configuration shown in FIG. 15 is just an example and the configuration is not limited to this.  
     [0221] The CPU (central processing unit)  101  controls the entire data processing device  100 .  
     [0222] The memory  102  temporarily stores a program and data that are usually stored in the storage device  105  (or a portable storage medium  109 ) and are read, for example, in order to execute the program and to update the data, respectively. For the memory  102 , for example, a RAM is used. The CPU  102  performs a variety of the processes described above using the program and data read from the memory  102 .  
     [0223] The input device  103  is a user interface used to input the user&#39;s instruction and data described above. For the input device  103 , for example, a keyboard, a pointing device and a touch panel are used.  
     [0224] The output device  104  is a user interface displaying the comment input screen, images/comments and the like. For the output device  104 , for example, a display is used.  
     [0225] The storage device  105  stores the program/data used to enable the data processing device  100  to realize a variety of the processes/functions described above. For the storage device  105 , for example, an HDD (hard disc drive), a variety of magnetic disc devices, optical disc devices and magneto-optical disc devices are used.  
     [0226] These program/data can also be stored in the portable storage medium  109 . In this case, the program/data stored in the portable storage medium  109  are read by the medium driving device  106 . For the portable storage medium  109 , for example, an FD (floppy disc)  109   a , a CD-Rom  109   b , a DVD, a magneto-optical disc are used.  
     [0227] Alternatively, the program/data can be downloaded from an external storage device through a network  40  connected to the network connection device  107 . The program/data can be read from a storage medium storing them (portable storage medium  109 , etc.), can be downloaded from a network transmitting them (transmission medium) or can be read from a signal transmitted through this transmission medium (transmission signal) when they are downloaded.  
     [0228] The network connection device  107  corresponds to the network I/F (interface)  21  shown in FIG. 2.  
     [0229] The multimedia server  20  has almost the same basic configuration as that shown in FIG. 15.  
     [0230]FIG. 16 shows the loading onto the computer of the program.  
     [0231] In FIG. 16, the data processing device (computer)  100  realizes the operations shown in the flowcharts, for example, by reading the program from the storage device  105  to the memory  102 , and executing it. The operations can also be realized by downloading the program onto the data processing device  100  from the portable storage medium  109  storing it that is put and distributed in the market.  
     [0232] Alternatively, the operations can realized by downloading the program onto the data processing device  100  from the data processing device (storage device)  110  of an external program provider through a network  120 . In this case, the software program can be executed by transmitting a transmission signal obtained by modulating a data signal representing the program with a carrier wave from the data processing device  110  of the program provider through the network  120 , which is a transmission medium, and reproducing the program.  
     [0233] As described above, by using the multimedia electronic tag of the present invention, a comment with a variety of attributes, such as a writer user and the like on multimedia data with a time sequence, such as dynamic image and the like can be shared/exchanged among members through a network. In this way, the smooth cooperative work of arbitrary multimedia data can be realized among the members. For example, the network commenting service, AV data co-editing work supplementary service through a network and the like can be provided.