METHOD AND APPARATUS FOR INSERTING INFORMATION INTO MULTIMEDIA DATA

A method of inserting information into multi-media data, the method constituted of: retrieving multimedia data from at least one multimedia storage device; extracting identifying parameters from the retrieved multimedia data; responsive to the extracted identifying parameters, identifying a particular one of a plurality of accompanying records associated with the retrieved multimedia data; creating synchronization information for the identified accompanying record; inserting the identified particular accompanying record into the retrieved multimedia data responsive to the created synchronization information; and outputting the retrieved multimedia data comprising the inserted identified particular accompanying record.

BACKGROUND OF THE INVENTION

The invention relates generally to the field of video data streams and in particular to a method and apparatus for inserting information into a video data stream.

Video entertainment, such as TV shows and home viewed movies, has been very popular for several decades. In recent years, with the advancement of transmission technologies, particularly satellite and cable broadcast technologies, and the development of the Internet, consumers have been exposed to a wide variety of video entertainment sources from all over the world. However, a consumer may not understand the language of the video entertainment coming from a different country. Traditionally, subtitles or dubbing have been supplied with video entertainment destined for a country whose primary language differs from the language of the video entertainment.

Unfortunately, several disadvantages exist in the current system, such as: consumers may connect to a video entertainment source which is located in a different country and does not supply subtitles or dubbing in the language of the country of the consumer; even if subtitles or dubbing are supplied in the primary language of the country of the consumer, the consumer may prefer a different language; the video presentation may be provided with one of subtitles and dubbing and the consumer prefers the one that wasn't provided; and in order to receive any type of translation, the consumer usually needs to have a subscription or other financial commitment with the provider of the particular video presentation. Additionally, data other than subtitles may be desired by the consumer, the desired data typically not being supplied by the provider of the video presentation. For example, commentary produced by other consumers may be desired to be embedded within the video presentation instead of the subtitles or dubbing supplied by the provider of the video presentation.

User created accompanying records, such as subtitles in a plurality of languages, typically are found on the Internet, however it is not possible to display any of these accompanying records unless a consumer is in possession of the entire file of the particular video presentation and has sufficient knowledge and means to insert the accompanying record into the video presentation. In the event a consumer is viewing a video presentation from a multimedia data provider, typically the video presentation is being viewed contemporaneously with its reception, such as a broadcast displayed on a TV or a web site providing online viewing. In such a case, accompanying records cannot be inserted into the incoming video presentation by the user.

SUMMARY OF THE INVENTION

Accordingly, it is a principal object of the present invention to overcome at least some of the disadvantages of present and prior art methods of inserting information into a video data stream.

This is accomplished in certain embodiments by providing a system arranged to insert information into multimedia data. In one embodiment, the multimedia data comprises a video data stream and the system comprises: at least one electronic device in communication with a database having stored thereon a plurality of accompanying records, the at least one electronic device arranged to receive the video data stream, the at least one electronic device comprising an output module; an extracting functionality arranged to extract identifying parameters from the received video data stream; an identifying functionality arranged to, responsive to the extracted identifying parameters, identify a particular one of the plurality of accompanying records associated with the received video data stream; a time marker functionality arranged to determine a time marker for the received video data stream; a synchronization functionality arranged to create synchronization information responsive to the determined time marker; and an insertion functionality arranged to insert the identified particular accompanying record into the received multimedia data stream responsive to the created synchronization information, wherein the output module is arranged to output the received multimedia data stream comprising the inserted accompanying record, and wherein the extraction of identifying parameters, the identification of the particular accompanying record, the periodic determination, the creation of synchronization information, and the insertion of the identified particular accompanying record and the outputting are contemporaneous with the receiving.

In one embodiment, the system further comprises a server, the server comprising the database, wherein the communication with database comprises communication with the server. In one further embodiment, the at least one electronic device comprises a plurality of electronic devices, the system further comprising a user synchronization adjustment functionality arranged to: receive a user input from a first of the plurality of electronic devices; and adjust the temporal position of the inserted accompanying records in the received video stream of a second of the plurality of electronic devices responsive to the received user input, the second of the plurality of electronic devices different from the first of the plurality of electronic devices.

In one independent embodiment, a non-transitory computer readable medium is provided, the non-transitory computer readable medium having instructions stored thereon, which, when executed by one or more processors, causes the one or more processors to perform operations, the operations comprising: retrieving multimedia data from at least one multimedia storage device; extracting identifying parameters from the retrieved multimedia data; responsive to the extracted identifying parameters, identifying a particular one of a plurality of accompanying records associated with the retrieved multimedia data; creating synchronization information for the identified accompanying record; inserting the identified particular accompanying record into the retrieved multimedia data responsive to the created synchronization information; and outputting the retrieved multimedia data comprising the inserted identified particular accompanying record.

In one embodiment, the multimedia data comprises a multimedia data stream being received by the at least one multimedia storage device, wherein the extracting identifying parameters, the identifying the particular one of the plurality of accompanying records, the creating synchronization information, the inserting and the outputting are contemporaneous with the receiving of the multimedia data stream by the at least one multimedia storage device. In another embodiment, the at least one multimedia storage device comprises a first multimedia storage device and a second multimedia storage device different from the first multimedia storage device, the retrieving comprising retrieving multimedia data from each of the first and second multimedia storage devices, the operations further comprising: receiving a user input from an electronic device, the electronic device comprising the first multimedia storage device; and adjusting the temporal position of the inserted accompanying record in the retrieved multimedia data from the second multimedia storage device responsive to the received user input.

In one embodiment, the operations further comprise receiving a time marker for the retrieved multimedia data, and wherein the creating synchronization information is responsive to the received time marker. In another embodiment, the operations further comprise determining a time marker for the retrieved multimedia data, and wherein the creating synchronization information is responsive to the determined time marker.

In one further embodiment, the retrieved multimedia data comprises embedded display information and wherein the determining the time marker is responsive to the embedded display information. In another further embodiment, the determining a time marker is performed periodically.

In one embodiment, the retrieved multimedia data comprises a video presentation, wherein in the event the video presentation is interrupted with material not of the video presentation, the operations further comprise: identifying the interruption in the video presentation; interrupting the inserting of the identified particular accompanying record into the retrieved multimedia data responsive to the identified interruption; identifying the resumption of the video presentation; and resuming the inserting of the identified particular accompanying record responsive to the identified resumption. In another embodiment, the operations further comprise receiving information associated with the at least one multimedia storage device, wherein the identifying the particular one of the plurality of accompanying records is further responsive to the received information associated with the at least one multimedia storage device.

In one embodiment, the retrieved multimedia data comprises embedded electronic program guide information and wherein the extracting identifying parameters from the retrieved multimedia data comprises extracting a portion of the embedded electronic program guide information.

In another independent embodiment, a method of inserting information into multimedia data is provided, the method comprising: retrieving multimedia data from at least one multimedia storage device; extracting identifying parameters from the retrieved multimedia data; responsive to the extracted identifying parameters, identifying a particular one of a plurality of accompanying records associated with the retrieved multimedia data; creating synchronization information for the identified accompanying record; inserting the identified particular accompanying record into the retrieved multimedia data responsive to the created synchronization information; and outputting the retrieved multimedia data comprising the inserted identified particular accompanying record.

In one embodiment, the multimedia data comprises a multimedia data stream being received by the at least one multimedia storage device, wherein the extracting identifying parameters, the identifying the particular one of the plurality of accompanying records, the creating synchronization information, the inserting and the outputting are contemporaneous with the receiving of the multimedia data stream by the at least one multimedia storage device. In another embodiment, the at least one multimedia storage device comprises a first multimedia storage device and a second multimedia storage device different from the first multimedia storage device, the retrieving comprising retrieving multimedia data from each of the first and second multimedia storage devices, the method further comprising: receiving a user input from an electronic device, the electronic device comprising the first multimedia storage device; and adjusting the temporal position of the inserted accompanying record in the retrieved multimedia data from the second multimedia storage device responsive to the received user input.

In one embodiment, the method further comprises: receiving a time marker for the retrieved multimedia data, wherein the creating synchronization information is responsive to the received time marker. In another embodiment, the method further comprises: determining a time marker for the retrieved multimedia data, wherein the creating synchronization information is responsive to the determined time marker.

In one further embodiment, the retrieved multimedia data comprises embedded display information and wherein the determining the time marker is responsive to the embedded display information. In another further embodiment, the determining a time marker is performed periodically.

In one embodiment, the retrieved multimedia data comprises a video presentation and wherein in the event the video presentation is interrupted with material not of the video presentation, the method further comprises: identifying the interruption in the video presentation; interrupting the inserting of the identified particular accompanying record into the retrieved multimedia data responsive to the identified interruption; identifying the resumption of the video presentation; and resuming the inserting of the identified particular accompanying record responsive to the identified resumption. In another embodiment, the method further comprises: receiving information associated with the at least one multimedia storage device, wherein the identifying the particular one of the plurality of accompanying records is further responsive to the received information associated with the at least one multimedia storage device.

In one embodiment, the retrieved multimedia data comprises embedded electronic program guide information and wherein the extracting identifying parameters from the retrieved multimedia data comprises extracting a portion of the embedded electronic program guide information.

In one independent embodiment, a system arranged to insert information into multimedia data is provided, the system comprising: a retriever in communication with at least one multimedia storage device having stored thereon multimedia data, the retriever arranged to retrieve multimedia data from the at least one multimedia storage device; an extractor arranged to extract identifying parameters from the retrieved multimedia data; an identifier in communication with an accompanying record storage device having stored thereon a plurality of accompanying records, the identifier arranged to, responsive to the extracted identifying parameters, identify a particular one of the plurality of accompanying records associated with the retrieved multimedia data, wherein the timing relationship between the identified accompanying record and the retrieved multimedia data is unknown; a synchronizer arranged to create synchronization information for the identified accompanying record; an inserter arranged to insert the identified particular accompanying record into the retrieved multimedia data responsive to the created synchronization information; and an output module arranged to output the retrieved multimedia data comprising the inserted accompanying record.

In one embodiment, the system further comprises: a processor; a memory, having instructions stored thereon, wherein the processor is arranged to execute the instructions stored on the memory thereby performing the operations of one of the retriever, extractor, identifier, synchronizer, inserter and output module. In another embodiment, the multimedia data comprises a multimedia data stream being received by the at least one multimedia storage device, wherein the extraction of identifying parameters, the identification of the particular accompanying record, the creation of synchronization information, and the insertion of the identified particular accompanying record and the outputting are contemporaneous with the receiving of the multimedia data stream by the at least one multimedia storage device.

In one embodiment, the system further comprises: a server comprising the accompanying record storage device, wherein the communication with the accompanying record storage device comprises communication with the server. In one further embodiment, the at least one multimedia storage device comprises a plurality of multimedia storage devices, the system further comprising: a plurality of electronic devices, each electronic device comprising a particular one of the plurality of the multimedia storage devices; and a user synchronizer arranged to: receive a user input from a first of the plurality of electronic devices; and adjust the temporal position of the inserted accompanying record in the received video stream of a second of the plurality of electronic devices responsive to the received user input, the second of the plurality of electronic devices different from the first of the plurality of electronic devices.

In one embodiment, the retriever is arranged to retrieve from the at least one multimedia storage device a time marker for the retrieved multimedia data, and wherein the synchronizer is responsive the retrieved time marker. In another embodiment, the system further comprises: a time marker determiner arranged to determine a time marker for the retrieved multimedia data, wherein the synchronizer is responsive to the determined time marker.

In one embodiment, the retrieved multimedia data comprises embedded display information and wherein the time marker determiner is responsive to the embedded display information. In one further embodiment, the time marker determiner is arranged to periodically determine the time marker for the received video data stream.

In one embodiment, the retrieved multimedia data received multimedia data stream comprises a video presentation and wherein the system further comprises: an interrupter; and a resumer, wherein in the event the video presentation is interrupted with material not of the video presentation the interrupter is arranged to: identify the interruption in the video presentation; and interrupt the insertion of the identified particular accompanying record into the retrieved multimedia data, and wherein at the cessation of the interruption of the video presentation, the resumer is arranged to: identify the cessation of the interruption in the video presentation, and resume the insertion of the identified particular accompanying record into the retrieved multimedia data. In another embodiment, the retrieved multimedia data comprises embedded electronic program guide information and wherein the extractor is arranged to extract a portion of the embedded electronic program guide information.

DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

FIG. 1Aillustrates a high level block diagram of a system10arranged to insert information into a video data stream, system10comprising: a server20; an electronic device30; a multimedia data receiver35; and a display40, optionally comprising a speaker (not shown). A multimedia data provider180is further illustrated for clarity. Server20comprises: a database50, comprising an accompanying records storage52and an optional electronic device information storage54; an identifier60; a time marker determiner70; a synchronizer80; an optional interrupter90; an optional resumer100; an optional user synchronizer110; and a communication module120. Electronic device30comprises: a retriever and extractor130; an inserter140; an output module150; an optional user input device160; and a communication module170. Identifier60, time marker determiner70, synchronizer80, optional interrupter90, optional resumer100, optional user synchronizer110, retriever and extractor130, inserter140, output module150and optional user input device160can each be implemented by any of: a dedicated functionality; computer readable instructions for a general purpose computing device or processor, the readable instructions stored on a memory associated with the respective one of server20and electronic device30; dedicated hardware; and a dedicated control circuitry, without limitation. Database50is in one embodiment part of server20and in another embodiment is external of database50, including, without limitation, a cloud computing environment in communication with server20.

As will be described below in relation toFIG. 2, any of multimedia data receiver35, identifier60, time marker determiner70, synchronizer80, optional interrupter90and optional resumer100are in another embodiment situated within electronic device30instead of within server20without exceeding the scope. Additionally, any of retriever and extractor130and inserter140are in another embodiment situated within server20instead of within electronic device30without exceeding the scope.

FIG. 1Billustrates a high level block diagram of system10, further illustrating server20and a plurality of electronic devices30, each in communication with a respective multimedia data receiver35and a respective display40. Thus, server20as illustrated in greater detail inFIG. 1Ais illustrated inFIG. 1Bin communication with a plurality of electronic devices30. Electronic device30may be a portion of a device which provides additional features and/or functionality without exceeding the scope, including without limitation a general computing device.

Multimedia data receiver35is arranged to receive a multimedia data stream from multimedia data stream provider180and each electronic device30is arranged to receive the multimedia data stream from the respective multimedia data receiver35. The received multimedia data stream comprises any of a plurality of multimedia data stream types such as video and music, without limitation. In one non-limiting embodiment, multimedia data stream provider180is a cable head end and in another embodiment multimedia data stream provider180is a direct broadcast satellite. In one further embodiment, multimedia data receiver35is arranged to receive a multimedia data stream of one or more of a plurality of technologies including, but not limited to, Internet Protocol Television (IPTV) and Digital Video Broadcasting (DVB). In another non-limiting embodiment, multimedia data stream provider180is a web site. In another non-limiting embodiment, multimedia data stream provider180is a media player having a video data file in a removable medium.

Accompanying records storage52is arranged to have stored thereon a plurality of accompanying records, each associated with a particular multimedia presentation. The accompanying records comprise any of a plurality of accompanying record types, such as subtitles or dubbing information. Optional electronic device information storage54has stored thereon information associated with one or more electronic devices30. In one non-limiting embodiment, the information comprises any of: geographical location; language preference; and display preferences.

In one embodiment, accompanying records storage52is arranged to have stored thereon a plurality of accompanying records associated with each multimedia presentation. For example, a plurality of subtitle accompanying records each corresponding to a particular video presentation, each subtitle accompanying record in a different language, may be stored on accompanying records storage52. In one further embodiment, the accompanying records associated with each video presentation are synchronized with each other, i.e. the accompanying records are adjusted such that the beginnings of the accompanying records correspond to the same temporal point in the associated video presentation. In another embodiment the time shift between the beginnings of the accompanying records are determined and stored on accompanying records storage52.

Optional user input device160comprises an apparatus arranged to receive a user input, including, but not limited to: a remote control; a touch screen; a plurality of push buttons; a keyboard; a mouse; a camera; or a voice command.

Server20is in communication with electronic device30via respective communication modules120,170. Internal communication links with respective communication modules120,170are not shown for the sake of simplicity. In an exemplary embodiment a control circuitry is provided for each of server20and electronic device30, as is known to those skilled in the art, and is not shown for the sake of simplicity. Identifier60and time marker determiner70are each in communication with database50. Identifier60is further in communication with time marker determiner70. Time marker determiner70is further in communication with synchronizer80, optional interrupter90and optional resumer100. Optional interrupter90is further in communication with optional resumer100. User synchronizer110is in communication with synchronizer80. Electronic device30is further in communication with multimedia data receiver35and display40. Preferably, retriever and extractor130is in communication with multimedia data receiver35and output module150is in communication with display40. Inserter140is in communication with output module150and optional user input device160is optionally in communication with inserter140.

FIG. 1Cillustrates a high level flow chart of the method of operation of system10,FIGS. 1A-1Cbeing described together.

In operation, in stage1000, electronic device30receives a multimedia data stream from multimedia data receiver35, the multimedia data stream comprising a video presentation. In stage1010, retriever and extractor130retrieves the received multimedia data stream and extracts identifying parameters from the received multimedia data stream of stage1000. In one embodiment, the identifying parameters are extracted from a text layer of the received video data stream. In one embodiment, wherein the received multimedia data stream comprises embedded electronic program guide (EPG) information, the extracted identifying parameters are responsive to at least a portion of the EPG information. The extracted identifying parameters are then transmitted to server20via communication module170and received via communication module120. Communication between communication module120and communication module170is preferably bidirectional and in an exemplary embodiment is implemented over the Internet, however this is not meant to be limiting in any way.

In stage1020, identifier60identifies at least one of the accompanying records stored on accompanying records storage52which corresponds to the extracted identifying parameters of stage1010, i.e. at least one accompanying record which is associated with the video presentation of the received multimedia data stream of stage1000. In one embodiment, the extracted identifying parameters are compared to identifying parameters associated with each of the stored accompanying records to find the corresponding accompanying record. In another embodiment, a function of the extracted identifying parameters is compared to identifying parameters associated with each of the stored accompanying records to find the corresponding accompanying record. In one further embodiment, the function is a translation of the extracted identifying parameters. In one embodiment, identifier60adjusts the extracted identifying parameters according to a data cleansing algorithm such that the extracted identifying parameters may be compared to identifying parameters associated with the stored accompanying records. In one further embodiment, the data cleansing algorithm is utilized in conjunction with a stored list of titles of video presentations in a plurality of languages. In one particular embodiment, the data cleansing algorithm is arranged to remove non-relevant data, such as data added to a title.

In one embodiment, stage1010as described above is periodically performed and the extracted identifying parameters are compared to the accompanying records stored on accompanying records storage52until a corresponding accompanying record is identified. In one embodiment, the extracted identifying parameters are particular symbols from subtitles embedded in the video presentation. The extracted symbols are compared with symbols from each stored accompanying record and in the event a correspondence is identified between the extracted symbols and the symbols of a stored accompanying record, the accompanying record is identified as corresponding with the video presentation. In one particular embodiment, the extracted identifying parameters are any of: words; letters; characters; punctuation marks; the length of the lines of the displayed subtitles; and the time difference between the display of each line of the displayed subtitles.

Optionally, as described above, there may be a plurality of similar accompanying records corresponding to the extracted identifying parameters stored on accompanying records storage52. In one embodiment, identifier60further identifies the desired accompanying record responsive to the identification information of electronic device30stored on optional electronic device information storage54. Additionally or alternately, in one embodiment identifier60is further arranged to receive identification information from electronic device30, the desired accompanying record identified responsive to the received identification information. The identification information may be transmitted by communication module170as part of the transmission of identifying parameters described above. Additionally or alternately, in one embodiment identifier60is further arranged to receive user preference information responsive to a user input at optional user input device160, the desired accompanying record identified responsive to the received identification information. In one embodiment, a plurality of accompanying records is identified responsive to the identification information of electronic device30.

In optional stage1030, time marker determiner70is arranged to determine a time marker for the received multimedia data stream of stage1000. In one embodiment, in the event that the extracted identification parameters of stage1010comprise EPG information, time marker determiner70is arranged to determine the amount of time elapsed from the start of the video presentation responsive to the EPG information. Additionally or alternately, in one embodiment retriever and extractor130is further arranged to extract embedded data from the received data stream of stage1000and transmit the extracted embedded data to time marker determiner70and time marker determiner70is arranged determine a time marker responsive to the extracted embedded data. In one embodiment, the embedded data is a text layer embedded in the received multimedia data stream of stage1000. In one embodiment, the embedded data is embedded display data, and time marker determiner70is arranged to compare a function of the extracted display data with one of the identified at least one accompanying record of stage1020to determine the temporal point in the accompanying record which corresponds to the current temporal point of the received video presentation of stage1000. Thus, the determined time marker is associated with the temporal point of the accompanying record which corresponds to the present temporal point of the video presentation.

In one embodiment, the extracted embedded data comprises subtitles for the video presentation and one of the identified at least one accompanying records comprises subtitles of the video presentation in a different language. In one further embodiment, time marker determiner70is arranged to translate the extracted subtitles into the language of the subtitles of the accompanying record and compare the translated subtitles to the subtitles of the accompanying record. Time marker determiner70then determines which portion of the subtitles of the accompanying record corresponds to the portion of subtitles extracted from the received video data stream, the end of the extracted portion of subtitles representing the current temporal point of the video presentation.

In another further embodiment, in the event that an accompanying record associated with the video presentation, comprising subtitles of the video presentation in the same language as the extracted subtitles is stored on accompanying records52, time marker determiner70determines which portion of the subtitles of the accompanying record corresponds to the portion of subtitles extracted from the received video data stream, a predetermined point of the extracted portion of subtitles representing the current temporal point of the video presentation. In one particular embodiment, the predetermined point is the end. As described above, in one embodiment a plurality of accompanying records associated with the same video presentation are synchronized with each other. Therefore, by comparing the subtitles of the video presentation with one of the stored accompanying records, the time marker is determined in relation to all of the accompanying records associated with the video presentation. Advantageously, the time marker is determined by comparison of subtitles in the same language thereby allowing for synchronization of an accompanying record containing subtitles in a different language or an accompanying record containing other display material, as will be described below.

In another further embodiment, as described above, particular portions are identified from the extracted subtitles. The identified portions are compared with portions from the identified accompanying record to determine the current temporal point of the video presentation in relation to the identified accompanying record. As described above, in one embodiment the particular portions are any of: words; letters; characters; punctuation marks; the length of the lines of the displayed subtitles; and the time difference between the display of each line of the displayed subtitles.

In one embodiment, the operation of time marker determiner70as described above is performed periodically, each subsequent determined time marker preferably achieving greater accuracy. In one embodiment, in the event that in stage1020a plurality of accompanying records were identified, time marker determiner70is arranged to determine the temporal point in each of the identified accompanying records which corresponds to the current temporal point of the video presentation which is represented by the extracted information. In another embodiment, as will be described below, synchronizer80creates synchronization information for the plurality of accompanying records responsive to the determined temporal point of one of the plurality of accompanying records.

In optional stage1035a time marker, as described above, is transmitted to synchronizer80by electronic device30. In one embodiment, the time marker was created by multimedia data stream provider180and embedded in the video presentation. In one embodiment, the time marker is extracted by retriever and extractor130. In one embodiment, a time marker is periodically transmitted to synchronizer80.

In stage1040, synchronizer80is arranged to create synchronization information responsive to the determined time marker of optional stage1030or the received time marker of optional stage1035. In the embodiment where time marker determiner70is arranged to periodically determine a time marker, synchronizer80is arranged to periodically create synchronization information responsive to the periodically determined time markers. In one embodiment, the created synchronization information indicates the temporal point of the identified at least one accompanying record of stage1020which is desired to be currently displayed. In the event that in stage1020a plurality of accompanying records were identified, synchronizer80creates synchronization information for each of the plurality of identified accompanying records. In the embodiment where the corresponding temporal point in each identified accompanying record is determined, synchronization information is created for each accompanying record responsive to the determined corresponding temporal point. In the embodiment where only the corresponding temporal point of one of the plurality of accompanying records was determined, synchronization information is created for each accompanying record responsive to the single determined corresponding temporal point. In the embodiment where, as described above, the plurality of accompanying records were synchronized with each other, the created synchronization information is preferably the same for each of the plurality of accompanying records. As described above, in one embodiment the time shifts between the various accompanying records associated with the same video presentation are determined. Additionally, as described above, in one embodiment the time marker of stage1030is determined in relation to an accompanying record different than the accompanying record desired to be displayed, such as an accompanying record containing subtitles in the same language of the displayed subtitles of the video presentation. In such a combined embodiment, the created synchronization information is adjusted according to the determined time shifts.

The identified at least one accompanying record of stage1020and the created synchronization information of stage1040are transmitted to electronic device30via communication module120and received by communication module170.

In stage1050, inserter140is arranged to insert the identified at least one accompanying record of stage1020into the received multimedia data stream of stage1000, the insertion of the accompanying record beginning at the desired temporal point, as described above, responsive to the created synchronization information of stage1040, responsive to the communication received by communication module170. In the embodiment where synchronization information is periodically created, the periodically created synchronization information is periodically transmitted to inserter140via communication modules120,170. Inserter140adjusts the temporal point of the accompanying record being inserted into the received video data stream, responsive to the received periodic synchronization information.

In another embodiment, server20transmits subsequent portions of the identified at least one accompanying record, beginning at the desired temporal point, to inserter140via communication modules120,170, and inserter140inserts the received portions of display data into the received data stream as it is received from server20. Preferably, the rate of transmission of the identified at least one accompanying record from server20is adjusted such that the synchronization of the accompanying record with the video presentation is maintained.

In stage1060, output module150is arranged to output the received multimedia data stream comprising the inserted accompanying record to display40and to the optional speaker in the event the accompanying record is a vocal record.

In optional stage1070, optional interrupter90is arranged to identify an interruption in the video presentation of the received multimedia data stream of stage1000by material not of the video presentation, such as a commercial message, as will be described further.

In one embodiment, as described above in relation to stage1030, time marker determiner70is arranged to periodically determine a time marker for the received multimedia data stream by comparing a function of extracted display data from the received multimedia data stream with an identified accompanying record. Optional interrupter90monitors the comparison of stage1030and in the event the function of the extracted display data is not consonant with the identified accompanying record for a predetermined length of the identified accompanying record, or if no data is received from retriever and extractor130for a predetermined time period, optional interrupter90interrupts the insertion of stage1050by transmitting an interrupt command to inserter140via communication modules120,170. In the embodiment where inserter140inserts the identified at least one accompanying record responsive to synchronization information received from synchronizer80, optional interrupter90thus terminates the operation of inserter140. In the embodiment where inserter140inserts the portion of the identified at least one accompanying record received from server20, optional interrupter90terminates the transmission of the identified at least one accompanying record from server20to inserter140, and thus inserter140ceases operation due to the lack of input.

In another embodiment, retriever and extractor130is arranged to periodically extract a predetermined layer of data from the received video data stream, the predetermined layer of data comprising an indication of whether the multimedia data stream currently comprises the video presentation or material not of the video presentation. The periodically extracted layer of data is transmitted by extractor30to optional interrupter90via communication modules120,170. In the event that optional interrupter90identifies, responsive to the extracted layer of data, that an interruption in the video presentation has occurred, the insertion of the identified at least one accompanying record is terminated, as described above.

In another embodiment, an interruption in the video presentation is detected by methods known to the prior art, such as the method described in U.S. Pat. No. 5,151,788 issued on Sep. 29, 1992 to Blum, the entire contents of which are incorporated herein by reference. In the event that optional interrupter90identifies that an interruption in the video presentation has occurred, the insertion of the identified at least one accompanying record is terminated, as described above, until the resumption of the video presentation is detected.

In optional stage1080, optional resumer100is arranged to identify a resumption of the video presentation of the received multimedia data stream of stage1000. In the embodiment where optional interrupter90identified the interruption of the video presentation by monitoring the periodic comparison of stage1030, optional resumer100begins monitoring the periodic comparison of stage1030after an interruption has been identified by optional interrupter90. In the event the function of the extracted display data is consonant with the identified accompanying record for a predetermined length of the identified accompanying record, the insertion of the identified at least one accompanying record into the received multimedia data stream is resumed, as described above in relation to stage1050.

In optional stage1090, in the event a user of the electronic device of stage1000notices a discrepancy in the synchronization of the video presentation and the inserted accompanying record of stage1050, i.e. the displayed data is not properly aligned with the displayed video presentation, the user may provide input at optional user input device160indicating the alignment correction necessary. In one embodiment, inserter140is arranged to adjust the synchronization of the inserted accompanying record with the video presentation responsive to the user input, and the user may continue to provide input at optional user input device160until the video presentation and the inserted accompanying record are properly aligned.

In another embodiment, the user input is received by synchronizer80via communication modules17,120and synchronizer80is arranged to adjust the created synchronization information for the respective accompanying record of stage1040responsive to the received user input. As described above, the accompanying record is inserted into the received multimedia data stream of stage1000responsive to the created synchronization information. The user may continue to provide input at optional user input device160until the video presentation and the inserted accompanying record are properly aligned. In one embodiment, the user may enter an additional input indicating that the accompanying record is properly synchronized with the video presentation.

As described above, and as illustrated inFIG. 1B, a plurality of electronic devices30may be in communication with server20. In optional stage1100, in the event the identified accompanying record of stage1020, or another accompanying record associated with the same video presentation, is being displayed to a plurality of electronic devices30, optional user synchronizer110is arranged to adjust the temporal position of the inserted accompanying record of stage1050for each of the plurality of electronic devices30displaying the associated accompanying record. In the embodiment where synchronizer80is arranged to periodically create synchronization information, optional user synchronizer110is arranged to control synchronizer80to adjust subsequent created synchronization information responsive to the user input. In another embodiment, optional user synchronizer110is arranged to control inserter140to adjust the insertion of the accompanying record responsive to the user input, as described above.

FIG. 2illustrates a high level block diagram of a system200arranged to insert information into a video data stream, system200comprising: a server210; an electronic device220; and a display40. Multimedia data provider180is further illustrated as described above in relation toFIG. 1A. Server220comprises: a database50, comprising an accompanying records storage52and an optional electronic device information storage54; an identifier60; a time marker determiner70; an optional user synchronizer110; and a communication module120. Electronic device230comprises: a multimedia data receiver35; a synchronizer80; an optional interrupter90; an optional resumer100; a retriever and extractor130; an inserter140; an output module150; an optional user input device160; and a communication module170. Multimedia data receiver35, identifier60, time marker determiner70, synchronizer80, optional interrupter90, optional resumer100, optional user synchronizer110, retriever and extractor130, inserter140and output module150can each be implemented by any of: a dedicated functionality; computer readable instructions for a general purpose computing device or processor, the readable instructions stored on a memory associated with the respective one of server220and electronic device230; dedicated hardware; and a dedicated control circuitry, without limitation. Database50is in one embodiment part of server210and in another embodiment is external of database50, including, without limitation, a cloud computing environment in communication with server220.

Server220is in communication with electronic device230via respective communication modules120,170. Identifier60and time marker determiner70are each in communication with database50. Internal communication links with respective communication modules120,170are not shown for the sake of simplicity. In an exemplary embodiment a control circuitry is provided for each of server220and electronic device230, as is known to those skilled in the art, and is not shown for the sake of simplicity. Identifier60and time marker determiner70are each in communication with database50. Identifier60is further in communication with time marker determiner70. Synchronizer80is in communication with inserter140. Optional interrupter90is in communication with optional resumer100and with inserter140. Resumer100is further in communication with inserter140. Retriever and extractor130is in communication with multimedia data receiver35and multimedia data receiver35is in communication with a multimedia data provider180. Inserter140is further in communication with output module150and output module150is in communication with display40.

The operation of system200is in all respects similar to the operation of system10ofFIGS. 1A-1C, and in the interest of brevity will not be further described, differing primarily in that certain functions are performed by electronic device230which inFIG. 1Aare performed by server20.

FIG. 3Aillustrates a high level block diagram of a system300arranged to insert information of an accompanying record into a multimedia data file andFIG. 3Billustrates a high level flow chart of the operation of system300,FIGS. 3A-3Bbeing described together. System300is in all respects similar to system10ofFIG. 1A, with the exception that a retriever310and an extractor315is provided in electronic device30in place of retriever and extractor130, and multimedia data receiver35is replaced with an electronic device320comprising a multimedia storage330. Retriever310is in communication with multimedia storage330and extractor315is in communication with retriever310. Multimedia storage330contains therein one or more multimedia files, such as video files and music files.

In stage2000, retriever310retrieves a multimedia data file from multimedia storage330. In stage2010, extractor315is arranged to extract identifying parameters from the retrieved multimedia data file of stage2000and transmit the identifying parameters to server20, as described above in relation to stage1010ofFIG. 1C. In stage2020, identifier60is arranged to identify an accompanying record stored on accompanying records storage52corresponding to the retrieved multimedia data file of stage2000, responsive to the extracted identifying parameters of stage2010, as described above in relation to stage1020. As described above, in the event a plurality of accompanying records are identified as corresponding to the retrieved multimedia data file, identifier60is arranged to identify a particular accompanying record according to identifying information of electronic device30or electronic device320stored on optional electronic device information storage54. The timing relationship between the identified accompanying record and the retrieved multimedia data file is typically, a priori, unknown. For example, the multimedia data file may contain a multimedia presentation and the identified accompanying record may begin at a point in time of the multimedia presentation which is not the beginning.

In stage2030, synchronization information is created for the identified accompanying record of stage2020, as described above in relation to stage1040and optional stages1020-1030. In stage2040, the identified accompanying record of stage2020is inserted by inserter140into the retrieved multimedia data file of stage2000responsive to the created synchronization of stage2030. In one embodiment, the accompanying record is inserted in the multimedia data file during the output thereof, as described above in relation to the data stream of stage1050. In another embodiment, the entire accompanying record is inserted into the multimedia data file prior to output thereof. In stage2050, the multimedia file comprising the inserted accompanying record of stage2040is output by output module150to display40and to the optional speaker in the event the accompanying record is a vocal record. In one embodiment, optional stages1070-1100are performed, as described above. In one embodiment, the above stages may be accomplished at any point during the output of the retrieved multimedia data file. Particularly, a portion of the data from the multimedia data file may be output to display40and an accompanying record is identified for the remaining portion of the data.

FIG. 4illustrates a high level schematic diagram of a system400arranged to insert information into multimedia data. System400comprises: a server410, comprising a processor430and a memory440; an electronic device420, comprising a processor450and a memory460; and a display40. Memory440has stored thereon instructions, processor430being arranged to execute the stored instructions. Memory460has stored thereon instructions, processor450being arranged to execute the stored instructions. The arrangement and operation of system400is in all respects similar to any of systems10,200, and300, described above, with the exception that processors430and450are arranged to perform the operations of one or more of the above described functionalities, modules and components of systems10,200and300.

The terms “include”, “comprise” and “have” and their conjugates as used herein mean “including but not necessarily limited to”.