Patent Description:
Streaming digitally encoded audiovisual (AV) programs, such as feature length films and television programs, over the Internet has become increasingly popular as the availability of high-bandwidth Internet connections has expanded. Streaming media services typically include a content server, a content player, and a communications network connecting the content server to the content player. The content server is configured to store media files (or "streams") made available to end-users. Each media file may provide a digital version of a movie, a television program, a sporting event, a staged or live event captured by recorded video, etc..

Oftentimes, audiovisual material is edited as part of a post-production process in order to convert such material into one or more media files for distribution to end-users. Those media files are usually distributed as part of a streaming media service or through more conventional physical media channels. Audiovisual material is edited for a variety of reasons, including, for example, to convey the author's creative intent, to delete certain scenes in order to conform to ratings, or to include credits that are not a part of the feature presentation. Metadata relating to such edits is typically provided with the audiovisual material in order to facilitate conversion of the audiovisual material into media file(s) that can then be distributed to end-users.

One drawback of the above approach is that a given processing edit specified by an author can introduce temporal drift between the audio and video tracks making up the audiovisual material when the edit enters or exits a constituent track at a non-sample boundary. When accumulated over the duration of the resulting media file, multiple edits to the audiovisual material can produce a perceptible lag or lead between the audio track and the video track, which degrades quality and can result in a poor user experience.

<CIT> discloses a media editing system which provides an editor with full visibility and editing capability for synchronous data that is adjunct to audio and video. <CIT> relates to the editing and switching of digital television signals consisting of video and associated sound components.

As the foregoing illustrates, improved techniques for editing audiovisual material would be useful.

One embodiment of the present invention sets forth a method for mitigating drift in audiovisual assets. The method includes determining that an edit associated with a presentation timeline is within boundaries of a video frame. The method further includes calculating a temporal drift associated with the edit, where the temporal drift comprises a duration of time between the edit and a boundary of the video frame. The method further includes determining whether to include the video frame in the presentation timeline based on the temporal drift and an accumulated temporal drift associated with the presentation timeline.

Further embodiments provide, among other things, a computer program product and a control server configured to implement the method set forth above.

At least one advantage of the disclosed techniques is that edits associated with a presentation timeline are processed to reduce temporal drift between an audio track and a video track included in the presentation timeline. In addition, the negative impact of any resulting temporal drift associated with non-sample boundary edits on the end-user viewing experience can be mitigated by transmitting metadata specifying the accumulated temporal drift to an endpoint device when the corresponding portions of the media file are transmitted to the endpoint device. The endpoint device can parse the metadata and compensate for the accumulated temporal drift, for example, by modifying an audio delay parameter associated with the endpoint device.

So that the manner in which the above recited features of the present invention can be understood in detail, a more particular description of the invention, briefly summarized above, may be had by reference to exemplary configurations, some of which are illustrated in the appended drawings.

In the following description, numerous specific details are set forth to provide a more thorough understanding of the embodiments of the present invention.

<FIG> illustrates a network infrastructure <NUM> configured to implement one or more aspects of the present invention. As shown, the network infrastructure <NUM> includes content servers <NUM>, control server <NUM>, and endpoint devices <NUM>, each of which are connected via a communications network <NUM>.

Each endpoint device <NUM> communicates with one or more content servers <NUM> (also referred to as "caches" or "nodes") via the network <NUM> to download content, such as textual data, graphical data, audio data, video data, and other types of data. The downloadable content, also referred to herein as a "media file," is then presented to a user of one or more endpoint devices <NUM>. In various embodiments, the endpoint devices <NUM> may include computer systems, set top boxes, mobile computer, smartphones, tablets, console and handheld video game systems, digital video recorders (DVRs), DVD players, connected digital TVs, dedicated media streaming devices, (e.g., the Roku® set-top box), and/or any other technically feasible computing platform that has network connectivity and is capable of presenting content, such as text, images, video, and/or audio content, to a user.

Each content server <NUM> may include a web-server, database, and server application <NUM> configured to communicate with the control server <NUM> to determine the location and availability of various media files that are tracked and managed by the control server <NUM>. Each content server <NUM> may further communicate with a fill source <NUM> and one or more other content servers <NUM> in order "fill" each content server <NUM> with copies of various media files. In addition, content servers <NUM> may respond to requests for media files received from endpoint devices <NUM>. The media files may then be distributed from the content server <NUM> or via a broader content distribution network. In some embodiments, the content servers <NUM> enable users to authenticate (e.g., using a username and password) in order to access media files stored on the content servers <NUM>. Although only a single control server <NUM> is shown in <FIG>, in various embodiments multiple control servers <NUM> may be implemented to track and manage media files.

In various embodiments, the fill source <NUM> may include an online storage service (e.g., Amazon® Simple Storage Service, Google® Cloud Storage, etc.) in which a catalog of files, including thousands or millions of media files, is stored and accessed in order to fill the content servers <NUM>. Although only a single fill source <NUM> is shown in <FIG>, in various embodiments multiple fill sources <NUM> may be implemented to service requests for media files.

<FIG> is a block diagram of the content server <NUM> of <FIG>, according to various exemplary configurations of the present invention. As shown, the content server <NUM> includes, without limitation, a central processing unit (CPU) <NUM>, a system disk <NUM>, an input/output (I/O) devices interface <NUM>, a network interface <NUM>, an interconnect <NUM>, and a system memory <NUM>.

The CPU <NUM> is configured to retrieve and execute programming instructions, such as server application <NUM>, stored in the system memory <NUM>. Similarly, the CPU <NUM> is configured to store application data (e.g., software libraries) and retrieve application data from the system memory <NUM>. The interconnect <NUM> is configured to facilitate transmission of data, such as programming instructions and application data, between the CPU <NUM>, the system disk <NUM>, I/O devices interface <NUM>, the network interface <NUM>, and the system memory <NUM>. The I/O devices interface <NUM> is configured to receive input data from I/O devices <NUM> and transmit the input data to the CPU <NUM> via the interconnect <NUM>. For example, I/O devices <NUM> may include one or more buttons, a keyboard, a mouse, and/or other input devices. The I/O devices interface <NUM> is further configured to receive output data from the CPU <NUM> via the interconnect <NUM> and transmit the output data to the I/O devices <NUM>.

The system disk <NUM> may include one or more hard disk drives, solid state storage devices, or similar storage devices. The system disk <NUM> is configured to store non-volatile data such as media files <NUM> (e.g., audio files, video files, subtitles, etc.). The media files <NUM> can then be retrieved by one or more endpoint devices <NUM> via the network <NUM>. In some embodiments, the network interface <NUM> is configured to operate in compliance with the Ethernet standard.

The system memory <NUM> includes a server application <NUM> configured to service requests for media files <NUM> received from endpoint device <NUM> and other content servers <NUM>. When the server application <NUM> receives a request for a media file <NUM>, the server application <NUM> retrieves the corresponding media file <NUM> from the system disk <NUM> and transmits the media file <NUM> to an endpoint device <NUM> or a content server <NUM> via the network <NUM>.

<FIG> is a block diagram of the control server <NUM> of <FIG>, according to various exemplary configurations of the present invention. As shown, the control server <NUM> includes, without limitation, a central processing unit (CPU) <NUM>, a system disk <NUM>, an input/output (I/O) devices interface <NUM>, a network interface <NUM>, an interconnect <NUM>, and a system memory <NUM>.

The CPU <NUM> is configured to retrieve and execute programming instructions, such as control application <NUM>, stored in the system memory <NUM>. Similarly, the CPU <NUM> is configured to store application data (e.g., software libraries) and retrieve application data from the system memory <NUM> and a database <NUM> stored in the system disk <NUM>. The interconnect <NUM> is configured to facilitate transmission of data between the CPU <NUM>, the system disk <NUM>, I/O devices interface <NUM>, the network interface <NUM>, and the system memory <NUM>. The I/O devices interface <NUM> is configured to transmit input data and output data between the I/O devices <NUM> and the CPU <NUM> via the interconnect <NUM>. The system disk <NUM> may include one or more hard disk drives, solid state storage devices, and the like. The system disk <NUM> is configured to store a database <NUM> of information associated with the content servers <NUM>, the fill source(s) <NUM>, and the media files <NUM>.

The system memory <NUM> includes a control application <NUM> configured to access information stored in the database <NUM> and process the information to determine the manner in which specific media files <NUM> will be replicated across content servers <NUM> included in the network infrastructure <NUM>. The system memory <NUM> further includes a media processing application <NUM> configured to generate one or more media files <NUM> based on audiovisual material and information specifying how the audiovisual material is to be edited to generate media file(s) <NUM>. For example, in various embodiments, edits made to an audio track and/or a video track in order to generate a media file <NUM> may be specified in metadata that is included in, or otherwise provided with, the audio track and/or the video track.

<FIG> is a block diagram of the endpoint device <NUM> of <FIG>, according to various exemplary configurations of the present invention. As shown, the endpoint device <NUM> may include, without limitation, a CPU <NUM>, a graphics subsystem <NUM>, an I/O device interface <NUM>, a mass storage unit <NUM>, a network interface <NUM>, an interconnect <NUM>, and a memory subsystem <NUM>.

In some embodiments, the CPU <NUM> is configured to retrieve and execute programming instructions stored in the memory subsystem <NUM>. Similarly, the CPU <NUM> is configured to store and retrieve application data (e.g., software libraries) residing in the memory subsystem <NUM>. The interconnect <NUM> is configured to facilitate transmission of data, such as programming instructions and application data, between the CPU <NUM>, graphics subsystem <NUM>, I/O devices interface <NUM>, mass storage <NUM>, network interface <NUM> and memory subsystem <NUM>.

In some embodiments, the graphics subsystem <NUM> is configured to generate frames of video data and transmit the frames of video data to display device <NUM>. In some embodiments, the graphics subsystem <NUM> may be integrated into an integrated circuit, along with the CPU <NUM>. The display device <NUM> may comprise any technically feasible means for generating an image for display. For example, the display device <NUM> may be fabricated using liquid crystal display (LCD) technology, cathode-ray technology, and light-emitting diode (LED) display technology. An input/output (I/O) device interface <NUM> is configured to receive input data from user I/O devices <NUM> and transmit the input data to the CPU <NUM> via the interconnect <NUM>. For example, user I/O devices <NUM> may comprise one of more buttons, a keyboard, and a mouse or other pointing device. The I/O device interface <NUM> also includes an audio output unit configured to generate an electrical audio output signal. User I/O devices <NUM> includes a speaker configured to generate an acoustic output in response to the electrical audio output signal. In alternative embodiments, the display device <NUM> may include the speaker. A television is an example of a device known in the art that can display video frames and generate an acoustic output.

A mass storage unit <NUM>, such as a hard disk drive or flash memory storage drive, is configured to store non-volatile data. A network interface <NUM> is configured to transmit and receive packets of data via the network <NUM>. In some embodiments, the network interface <NUM> is configured to communicate using the well-known Ethernet standard. The network interface <NUM> is coupled to the CPU <NUM> via the interconnect <NUM>.

In some embodiments, the memory subsystem <NUM> includes programming instructions and application data that comprise an operating system <NUM>, a user interface <NUM>, and a playback application <NUM>. The operating system <NUM> performs system management functions such as managing hardware devices including the network interface <NUM>, mass storage unit <NUM>, I/O device interface <NUM>, and graphics subsystem <NUM>. The operating system <NUM> also provides process and memory management models for the user interface <NUM> and the playback application <NUM>. The user interface <NUM>, such as a window and object metaphor, provides a mechanism for user interaction with endpoint device <NUM>. Persons skilled in the art will recognize the various operating systems and user interfaces that are well-known in the art and suitable for incorporation into the endpoint device <NUM>.

In some embodiments, the playback application <NUM> is configured to request and receive content from the content server <NUM> via the network interface <NUM>. Further, the playback application <NUM> is configured to interpret the content and present the content via display device <NUM> and/or user I/O devices <NUM>.

As described above, audiovisual material oftentimes is edited as part of a post-production process in order to convert the audiovisual material into a media file <NUM> for distribution to end-users. Editing of audiovisual material could be performed for various reasons, such as to more accurately convey the author's creative intent, to delete certain scenes in order to conform to ratings standards, and/or to include credits that are not a part of the feature presentation. Metadata relating to such edits is typically provided with the audiovisual material in order to facilitate processing of the audiovisual material into one or more media files <NUM> that convey the creative intent of the author.

However, under certain circumstances, conventional techniques for processing edits to audiovisual material can introduce temporal drift between an audio tracks and a video track. In particular, when an edit fails to enter (or exit) an audio track and/or a video track at a sample boundary (e.g., when the edit falls within the boundaries of an audio frame and/or a video frame), drift is introduced into the presentation timeline and, thus, into the media file <NUM> generated based on the presentation timeline. Further, when accumulated over the duration of the presentation timeline, multiple non-sample boundary edits to the audiovisual material may produce a perceptible lag or lead between an audio track and a video track, resulting in a poor user experience.

Accordingly, in various embodiments, the media processing application <NUM> may reduce the magnitude of temporal drift that is accumulated over the duration of a presentation timeline by determining, for each non-sample boundary edit, whether the corresponding audio frame and/or a video frame should be included in or excluded from the presentation timeline. Additionally, in some embodiments, the media processing application <NUM> further mitigates the impact of accumulated temporal drift on the user experience by generating metadata specifying the accumulated temporal drift for each portion of a media file <NUM>. The metadata is then transmitted to an endpoint device <NUM> on which the media file <NUM> is to be played, enabling the endpoint device <NUM> to compensate for the accumulated temporal drift, such as by adjusting an audio delay parameter associated with the endpoint device <NUM>.

Because, in a typical media file <NUM> (e.g., having a video framerate of <NUM> to <NUM> frames/second and an audio framerate of <NUM>,<NUM> to <NUM>,<NUM> frames/second), the duration of each video frame is orders of magnitude longer than the duration of each audio frame, the techniques described below primarily focus on including or excluding video frames. However, each of the techniques described below also is applicable to determining whether to include or exclude audio frames associated with non-sample boundary edits. More specifically, an edit that falls within the boundaries of an audio frame is likely to be relatively close (e.g., approximately <NUM> microseconds or less for a <NUM> audio sampling rate) to one of the boundaries of the audio frame. By contrast, an edit that falls within the boundaries of a video frame may be relatively far (e.g., up to approximately <NUM> milliseconds for <NUM> video frames/second) from one of the boundaries of the video frame. Accordingly, the techniques described below may be implemented to determine whether to include or exclude a video frame and/or an audio frame intersected by a particular edit associated with a presentation timeline.

<FIG> is a conceptual illustration of a presentation timeline <NUM> generated by editing a video track <NUM> and an audio track <NUM>, according to various embodiments of the present invention. As shown, the presentation timeline <NUM> includes a first video portion <NUM> and a second video portion <NUM> of the video track <NUM> and a first audio portion <NUM> and a second audio portion <NUM> of the audio track <NUM>. Each of the first video portion <NUM>, the second video portion <NUM>, the first audio portion <NUM>, and the second audio portion <NUM> includes an entry edit <NUM> (e.g., entry edit <NUM>-<NUM>, entry edit <NUM>-<NUM>, etc.) and an exit edit <NUM> (e.g., exit edit <NUM>-<NUM>, exit edit <NUM>-<NUM>, etc.).

In general, entry edits <NUM> and exit edits <NUM> may be specified in any technically feasible manner. For example, each entry edit <NUM> and exit edit <NUM> may be specified in metadata associated with the presentation timeline <NUM> and/or specified in metadata associated with a video track <NUM> and/or an audio track <NUM>. For clarity of explanation, <FIG> includes only a single video track <NUM> and a single audio track <NUM>. However, in other exemplary configurations, each portion of the presentation timeline <NUM> may specify edits associated with any number of video tracks <NUM> and/or audio tracks <NUM>. Furthermore, for clarity of explanation, the exemplary configurations described below assume that the video track <NUM> has a rate of <NUM> frames/second. However, in other embodiments, any technically feasible framerate, including variable framerates, may be implemented.

<FIG> depicts each entry edit <NUM> and exit edit <NUM> of the video track <NUM> (e.g., <NUM>-<NUM> and <NUM>-<NUM>) as being temporally aligned with the corresponding entry edit <NUM> and exit edit <NUM> of the audio track <NUM> (e.g., <NUM>-<NUM> and <NUM>-<NUM>). However, in some embodiments, an entry edit <NUM> and/or an exit edit <NUM> of the video track <NUM> may be temporally misaligned with a corresponding entry edit <NUM> and/or exit edit <NUM> of the audio track <NUM>. For example, with reference to <FIG>, video portion <NUM> may correspond to a first time interval of the video track <NUM> (e.g., from t=<NUM> seconds to t=<NUM> seconds), and audio portion <NUM> may correspond to a second time interval of the audio track <NUM> that is partially overlapping (e.g., from t=<NUM> seconds to t=<NUM> seconds) or non-overlapping (e.g., from t=<NUM> seconds to t=<NUM> seconds) with the first time interval. Further, one or both of the video portion <NUM>, <NUM> and audio portion <NUM>, <NUM> may correspond to multiple time intervals of the video track <NUM> and/or audio track <NUM>. For example, with reference to <FIG>, video portion <NUM> may correspond to a single time interval of the video track <NUM> (e.g., from t=<NUM> seconds to t=<NUM> seconds), and audio portion <NUM> may correspond to multiple, discontinuous time intervals of the audio track <NUM> (e.g., from t=<NUM> seconds to t=<NUM> seconds and from t=<NUM> seconds to t=<NUM> seconds). Accordingly, entry edits <NUM> and exit edits <NUM> may be specified in a flexible manner when generating the presentation timeline <NUM>.

<FIG> is a conceptual illustration of non-frame boundary edits made to the video track <NUM> when generating the presentation timeline <NUM> of <FIG>. As shown in <FIG>, exit edit <NUM>-<NUM> of the first video portion <NUM> falls within the boundaries of video frame <NUM>-<NUM>. Consequently, if a constant video framerate is maintained, then the media processing application <NUM> must determine whether the video frame <NUM>-<NUM> should be included in or excluded from the presentation timeline <NUM> when processing the exit edit <NUM>-<NUM>.

If the video frame <NUM>-<NUM> is included in the presentation timeline <NUM>, then a lag time <NUM>-<NUM> of approximately <NUM> milliseconds (e.g., a temporal drift of +<NUM> milliseconds) will be generated in the presentation timeline <NUM>. If, on the other hand, the video frame <NUM>-<NUM> is excluded from the presentation timeline <NUM>, then a lead time <NUM>-<NUM> of approximately <NUM> milliseconds (e.g., a temporal drift of -<NUM> milliseconds) will be generated in the presentation timeline <NUM>. Although the techniques discussed herein are described with reference to seconds (e.g., milliseconds), in other exemplary configurations, the techniques may implement any technically feasible unit of time (e.g., ticks).

In various exemplary configurations, for each edit (e.g., entry edit <NUM> and exit edit <NUM>), the media processing application <NUM> determines whether to include or exclude a video frame <NUM> intersected by the edit based on the location of the edit relative to the boundaries of the video frame <NUM>. Additionally, when determining whether to include or exclude a video frame <NUM> intersected by the edit, the media processing application <NUM> may further consider the accumulated temporal drift at the location of the edit in the presentation timeline <NUM>. For example, the media processing application <NUM> may determine whether to include or exclude a video frame <NUM> based on whether including (or excluding) the video frame <NUM> would cause the cause the accumulated temporal drift to exceed a threshold value (e.g., a lag time and/or lead time of <NUM> to <NUM> milliseconds).

Alternatively, the media processing application <NUM> may determine whether to include or exclude a video frame <NUM> based on which outcome would result in an accumulated temporal drift having a lower magnitude. In a specific example, with reference to exit edit <NUM>-<NUM>, the media processing application <NUM> could determine that the video frame <NUM>-<NUM> should be included, since including the video frame <NUM>-<NUM> will result in an accumulated temporal drift having a lower magnitude (e.g., a lag time <NUM>-<NUM> of approximately <NUM> milliseconds) than the accumulated temporal drift that would be generated by excluding the video frame <NUM>-<NUM> (e.g., a lead time <NUM>-<NUM> of approximately <NUM> milliseconds).

Further, in some exemplary configurations, the media processing application <NUM> may be biased towards maintaining either a positive accumulated temporal drift (a lag time <NUM>) or a negative accumulated temporal drift (a lead time <NUM>) when processing each edit. Such a bias may be implemented based on whether a user is more likely to notice a positive accumulated temporal drift or a negative accumulated temporal drift. In order to implement such a bias, the media processing application <NUM> could assign a first threshold value to the positive accumulated temporal drift (a threshold lag time) and assign a second threshold value having a lower magnitude to the negative accumulated temporal drift (a threshold lead time). Then, for each edit, the media processing application <NUM> would determine whether including or excluding the corresponding video frame <NUM> would cause the accumulated temporal drift to exceed the first threshold value or the second threshold value.

Returning to <FIG>, the media processing application <NUM> determines that the video frame <NUM>-<NUM> should be included. Accordingly, exit edit <NUM>-<NUM> generates an accumulated temporal drift of approximately <NUM> milliseconds (a lag time <NUM>-<NUM> of approximately <NUM> milliseconds) when the first video portion <NUM> is added to the presentation timeline <NUM>, as shown in <FIG>, which is a conceptual illustration of the accumulated temporal drift produced when generating in the presentation timeline <NUM> of <FIG>, according to various exemplary configurations of the present invention. As described above, the decision to include video frame <NUM>-<NUM> when adding the first video portion <NUM> to the presentation timeline <NUM> could be based on the media processing application <NUM> determining that including the video frame <NUM>-<NUM> would result in an accumulated temporal drift having a lower magnitude than if the video frame <NUM>-<NUM> were excluded.

As shown in <FIG>, entry edit <NUM>-<NUM> associated with the first video portion <NUM> is aligned with a video frame boundary (i.e., the leftmost boundary of the first video frame <NUM> included in the video track <NUM>). Consequently, as shown in <FIG>, no temporal delay is incurred in the presentation timeline <NUM> when the media processing application <NUM> processes entry edit <NUM>-<NUM>.

In contrast to entry edit <NUM>-<NUM>, entry edit <NUM>-<NUM> of the second video portion <NUM> falls within the boundaries of video frame <NUM>-<NUM>, as shown in <FIG>. Consequently, if a constant video framerate is to be maintained, then the media processing application <NUM> must determine whether the video frame <NUM>-<NUM> should be included in or excluded from the presentation timeline <NUM>. If the video frame <NUM>-<NUM> is included in the presentation timeline <NUM>, then a lag time <NUM>-<NUM> of approximately <NUM> milliseconds (e.g., a temporal drift of +<NUM> milliseconds) will be added to the accumulated temporal drift associated with the presentation timeline <NUM>. If, on the other hand, the video frame <NUM>-<NUM> is excluded from the presentation timeline <NUM>, then a lead time <NUM>-<NUM> of approximately <NUM> milliseconds (e.g., a temporal drift of -<NUM> milliseconds) will be subtracted from the accumulated temporal drift associated with the presentation timeline <NUM>.

In some exemplary configurations, the media processing application <NUM> may determine whether inclusion of the video frame <NUM>-<NUM> would cause the accumulated temporal drift to exceed a threshold lag time <NUM>. If inclusion of the video frame <NUM>-<NUM> would not cause the accumulated temporal drift to exceed a threshold lag time <NUM>, then the video frame <NUM>-<NUM> is included in the presentation timeline <NUM>. This outcome is illustrated in <FIG>, which shows that inclusion of both video frame <NUM>-<NUM> and video frame <NUM>-<NUM> in the presentation timeline <NUM> generates an accumulated temporal drift of approximately <NUM> milliseconds.

Next, as shown in <FIG>, the media processing application <NUM> determines that exit edit <NUM>-<NUM> of the second video portion <NUM> falls within the boundaries of video frame <NUM>-<NUM>. Consequently, if a constant video framerate is to be maintained, then the media processing application <NUM> must determine whether the video frame <NUM>-<NUM> should be included in or excluded from the presentation timeline <NUM>. If the video frame <NUM>-<NUM> is included in the presentation timeline <NUM>, then a lag time <NUM>-<NUM> of approximately <NUM> milliseconds (e.g., a temporal drift of +<NUM> milliseconds) will be added to the accumulated temporal drift associated with the presentation timeline <NUM>. If, on the other hand, the video frame <NUM>-<NUM> is excluded from the presentation timeline <NUM>, then a lead time <NUM>-<NUM> of approximately <NUM> milliseconds (e.g., a temporal drift of -<NUM> milliseconds) will be subtracted from the accumulated temporal drift associated with the presentation timeline <NUM>.

In some exemplary configurations, the media processing application <NUM> may determine that inclusion of the video frame <NUM>-<NUM> would cause the accumulated temporal drift to exceed the threshold lag time <NUM>, as shown in <FIG>. Accordingly, in such exemplary configurations, the video frame <NUM>-<NUM> would be excluded from the presentation timeline <NUM>, reducing the accumulated temporal drift from approximately <NUM> milliseconds to approximately <NUM> milliseconds.

Although not shown in <FIG>, the presentation timeline <NUM> may further include a third video portion that is adjacent to the second video portion and a fourth video portion that is adjacent to the third video portion. Then, as shown in <FIG>, the media processing application <NUM> may determine that a video frame <NUM> associated with the entry edit <NUM> of the third video portion will be included, causing a lead time <NUM>-<NUM> of approximately <NUM> milliseconds to be subtracted from the accumulated temporal drift. Accordingly, after processing of the entry edit <NUM> associated with the third video portion, the presentation timeline <NUM> would have a negative accumulated temporal drift (a lead time).

Additionally, as further shown in <FIG>, the exit edit <NUM> associated with the third video portion and the entry edit <NUM> associated with the fourth video portion may further decrease the accumulated temporal drift of the presentation timeline <NUM> by lead time <NUM>-<NUM> and lead time <NUM>-<NUM>, respectively. Then, upon processing the exit edit <NUM> associated with the fourth video portion, the media processing application <NUM> determines that exclusion of the corresponding video frame <NUM> would incur a lead time <NUM>-<NUM> of approximately <NUM> milliseconds, causing the accumulated temporal drift to fall below the lead time threshold <NUM>. Consequently, the media processing application <NUM> determines that the video frame <NUM> associated with the exit edit <NUM> of the fourth video portion should be included in the presentation timeline <NUM>, causing the magnitude of the accumulated temporal drift to be reduced by a lag time <NUM>-<NUM> of approximately <NUM> milliseconds. The media processing application <NUM> then continues to process each entry edit <NUM> and exit edit <NUM> associated with subsequent video portions in a similar manner.

Although the techniques implemented in <FIG> proceed either towards the lag threshold time <NUM> (e.g., by including video frames <NUM>) or towards the lead threshold time <NUM> (e.g., by excluding video frames <NUM>) until either the lag threshold time <NUM> or the lead threshold time <NUM> would be exceeded, in other exemplary configurations, the media processing application <NUM> may decide to include or exclude a video frame <NUM> depending upon whether the accumulated temporal drift is currently positive or negative (e.g., above or below a threshold value of zero). For example, if the accumulated temporal drift is currently positive when a particular entry edit <NUM> or exit edit <NUM> is being processed, then the media processing application <NUM> could determine that the corresponding video frame <NUM> should be excluded. If, on the other hand, the accumulated temporal drift is currently negative when a particular entry edit <NUM> or exit edit <NUM> is being processed, then the media processing application <NUM> could determine that the corresponding video frame <NUM> should be included. In some exemplary configurations, this process of determining whether to include or exclude a video frame <NUM> based on whether the accumulated temporal drift is currently positive or negative could be performed for each edit processed by the media processing application <NUM>. Accordingly, such a technique could enable the media processing application <NUM> to maintain an accumulated temporal drift that is close to zero.

In other exemplary configurations, the media processing application <NUM> may decide to include or exclude a video frame <NUM> depending upon whether the accumulated temporal drift is currently above or below a particular threshold value. For example, if the accumulated temporal drift is currently above a threshold value (e.g., <NUM> milliseconds) when a particular entry edit <NUM> or exit edit <NUM> is being processed, then the media processing application <NUM> could determine that the corresponding video frame <NUM> should be excluded. If, on the other hand, the accumulated temporal drift is currently below the threshold value when a particular entry edit <NUM> or exit edit <NUM> is being processed, then the media processing application <NUM> could determine that the corresponding video frame <NUM> should be included. In some exemplary configurations, this process of determining whether to include or exclude a video frame <NUM> based on whether the accumulated temporal drift is currently above or below a threshold value could be performed for each edit processed by the media processing application <NUM>. Accordingly, such a technique could enable the media processing application <NUM> to maintain an accumulated temporal drift that is close to the threshold value.

In various exemplary configurations, the media processing application <NUM> also may process dwell edits and empty edits by implementing techniques that are similar to the techniques implemented to process the entry edits <NUM> and exit edits <NUM> described above. For example, when a dwell edit - specifying that a particular image should be displayed in the presentation timeline <NUM> for a certain period of time - has a duration that is not an integer multiple of a video frame duration (e.g., approximately <NUM> milliseconds for <NUM> frames/second), then the media processing application <NUM> may include a video frame <NUM> by rounding up to the nearest integer multiple. Alternatively, the media processing application <NUM> may exclude a video frame <NUM> by rounding down to the nearest integer multiple. Similarly, when an empty edit - specifying a duration of time by which the presentation timeline <NUM> should be delayed or stalled - has a duration that is not an integer multiple of a video frame duration, then the media processing application <NUM> may include a video frame <NUM> by rounding up to the nearest integer multiple or exclude a video frame <NUM> by rounding down to the nearest integer multiple.

In still other exemplary configurations, instead of (or in addition to) including and excluding entire video frames <NUM>, the media processing application <NUM> may include or exclude portions of video frames <NUM> by adjusting the framerate of the presentation timeline <NUM>. For example, upon encountering an entry edit <NUM> that falls within the boundaries of a video frame <NUM>, the media processing application <NUM> could determine a video framerate that would enable the video frame <NUM> to be played back for the duration of time between the location of the entry edit <NUM> and the rightmost boundary of the video frame <NUM>. Additionally, upon encountering an exit edit <NUM> that falls within the boundaries of a video frame <NUM>, the media processing application <NUM> could determine a video framerate that would enable the video frame <NUM> to be played back for the duration of time between the location of the exit edit <NUM> and the leftmost boundary of the video frame <NUM>. The media processing application corresponding portion(s) of the presentation timeline <NUM> and/or resulting media file <NUM>, enabling a client device to play the media file <NUM> at a variable framerate.

<FIG> illustrates a flow diagram of method steps for computing the accumulated temporal drift in a media file <NUM>.

As shown in <FIG>, a method <NUM> begins at step <NUM>, where the media processing application <NUM> receives an edit (e.g., an entry edit <NUM> or an exit edit <NUM>) associated with a presentation timeline <NUM>. At step <NUM>, the media processing application <NUM> determines whether the edit falls within the boundaries of a video frame <NUM>. If the media processing application <NUM> determines that the edit falls within the boundaries of a video frame <NUM>, then the method <NUM> proceeds to step <NUM>, where the media processing application <NUM> calculates one or more temporal drifts associated with the edit. For example, the media processing application <NUM> may calculate both a first temporal drift that would result if the video frame <NUM> was included in the presentation timeline <NUM> and a second temporal drift that would result if the video frame <NUM> was excluded from the presentation timeline <NUM>.

At step <NUM>, the media processing application <NUM> determines whether to include the video frame <NUM> intersected by the edit based on the temporal drift(s) calculated at step <NUM>, an accumulated temporal drift associated with the presentation timeline <NUM> proximate to the location of the edit, and one or more optional threshold values (e.g., a lag time threshold <NUM> and/or a lead time threshold <NUM>). In general, at step <NUM>, the media processing application <NUM> may implement any of the techniques described, such as determining whether a sum of the temporal drift and the accumulated temporal drift exceeds the lag time threshold <NUM> or the lead time threshold <NUM>.

At step <NUM>, the media processing application <NUM> updates the accumulated temporal drift based on whether the video frame <NUM> was included in or excluded from the presentation timeline <NUM>. For example, the media processing The remainder of the method <NUM> then proceeds to steps <NUM> through <NUM> as described above.

<FIG> illustrates a flow diagram of method steps for transmitting accumulated temporal drift information associated with a media file <NUM> to an endpoint device <NUM>.

As shown in <FIG>, a method <NUM> begins at step <NUM>, where the media processing application <NUM> and/or server application <NUM> receives a request for one or more portions of a media file <NUM>. At step <NUM>, the media processing application <NUM> and/or server application <NUM> retrieves the one or more portions of the media file <NUM> and accumulated temporal drift information associated with the one or more portions of the media file <NUM>. As described above, accumulated temporal drift information may be associated with portions of the presentation timeline <NUM> and/or media file <NUM> having duration of time (e.g., <NUM> second fragments of a media file <NUM>).

Next, at step <NUM>, the media processing application <NUM> transmits the portion(s) of the media file <NUM> and the corresponding accumulated temporal drift information to a content server <NUM> and/or an endpoint device <NUM> via the network <NUM>. Additionally or alternatively, at step <NUM>, the server application <NUM> transmits the portion(s) of the media file <NUM> and the corresponding accumulated temporal drift information to an endpoint device <NUM> via the network <NUM>.

At step <NUM>, the media processing application <NUM> determines whether an additional media file <NUM> (or additional portions of a media file <NUM>) are to be transmitted. If the media processing application <NUM> determines that an additional media file <NUM> or an additional portion of a media file <NUM> are to be transmitted, then the method <NUM> returns to step <NUM>. If the media processing application <NUM> determines that no additional media files <NUM> or portions of a media file <NUM> are to be transmitted, then the method <NUM> terminates.

<FIG> illustrates a flow diagram of method steps for receiving accumulated temporal drift information associated with a media file <NUM> from a The remainder of the method <NUM> then proceeds to steps <NUM> through <NUM> as described above.

<FIG> illustrates a flow diagram of method steps for transmitting accumulated temporal drift information associated with a media file <NUM> to an endpoint device <NUM>, according to various exemplary configurations of the present invention. Although the method steps are described in conjunction with the systems of <FIG>, persons skilled in the art will understand that any system configured to perform the method steps, in any technically feasible order, falls within the scope of the present invention.

<FIG> illustrates a flow diagram of method steps for receiving accumulated temporal drift information associated with a media file <NUM> from a content server <NUM>, according to various exemplary configurations of the present invention. Although the method steps are described in conjunction with the systems of <FIG>, persons skilled in the art will understand that any system configured to perform the method steps, in any technically feasible order, falls within the scope of the present invention.

As shown in <FIG>, a method <NUM> begins at step <NUM>, where the playback application <NUM> transmits a request for one or more portions of a media file <NUM> to the content server <NUM> and/or to the control server <NUM> via the network <NUM>. At step <NUM>, in response to the request, the playback application <NUM> receives the one or more portions of the media file <NUM> and accumulated temporal drift information associated with the one or more portions of the media file <NUM> via the network <NUM>.

Next, at step <NUM>, the playback application <NUM> plays back the portion(s) of the media file <NUM> based on the corresponding accumulated temporal drift information. For example, in some exemplary configurations, the playback application <NUM> could read the accumulated temporal drift from metadata included in a portion of the media file <NUM>. The playback application <NUM> could then adjust a delay parameter (e.g., an audio delay parameter and/or a video delay parameter) based on the accumulated temporal drift when playing back to the portion of the media file <NUM>.

At step <NUM>, the playback application <NUM> determines whether an additional media file <NUM> (or additional portions of a media file <NUM>) is to be requested. If the playback application <NUM> determines that an additional media file <NUM> or portion of a media file <NUM> is to be requested, then the method <NUM> returns to step <NUM>. If the playback application <NUM> determines that no additional media files <NUM> or portions of a media file <NUM> are to be requested, then the method <NUM> terminates.

In sum, a media processing application receives a presentation timeline specifying edits associated with an audio track and/or a video track. Then, for each non-sample boundary edit included in the presentation timeline, the media processing application determines whether to include the corresponding audio frame and/or video frame. This determination may be based on a temporal drift associated with including the audio frame and/or video frame, an accumulated temporal drift associated with the presentation timeline, and/or one or more temporal drift thresholds. The media portion of a media file and transmit the portions of the media file <NUM> to a content server <NUM> and/or endpoint device <NUM>.

At least one advantage of the disclosed techniques is that edits associated with a presentation timeline may be processed to reduce or eliminate temporal drift between an audio track and a video track included in the presentation timeline. Additionally, the effect of non-sample boundary edits on the end-user experience may be mitigated by transmitting metadata specifying an accumulated temporal drift to an endpoint device when the corresponding portions of the media file are transmitted to the endpoint device. Accordingly, the endpoint device is able to compensate for the accumulated temporal drift, such as by modifying an audio delay parameter associated with the endpoint device.

Aspects of the present exemplary configurations may be embodied as a system, method or computer program product. Accordingly, aspects of the present disclosure may take the form of an entirely hardware embodiment, an entirely software embodiment (including firmware, resident software, micro-code, etc.) or an embodiment combining software and hardware aspects that may all generally be referred to herein as a "circuit," "module" or "system. " Furthermore, aspects of the present disclosure may take the form of a computer program product embodied in one or more computer readable medium(s) having computer readable program code embodied thereon.

These computer program instructions may be provided to a processor of a general purpose computer, special purpose computer, or other programmable data processing apparatus to produce a machine, such that the instructions, which execute via the processor of the computer or other programmable data processing apparatus, enable the implementation of the functions/acts specified in the flowchart and/or block diagram block or blocks. Such processors may be, without limitation, general purpose processors, special-purpose processors, application-specific processors, or field-programmable processors or gate arrays.

The flowchart and block diagrams in the figures illustrate the architecture, functionality, and operation of possible implementations of systems, methods and computer program products according to various exemplary configurations of the present disclosure.

Claim 1:
A method (<NUM>), comprising:
determining (<NUM>) that an edit associated with a presentation timeline is within boundaries of a video frame;
calculating (<NUM>) a temporal drift between an audio track and a video track included in the presentation timeline associated with the edit, wherein the temporal drift comprises a duration of time between the edit and a boundary of the video frame when the edit enters or exits a constituent track at a non-sample boundary;
determining (<NUM>) whether to include the video frame in the presentation timeline based on the temporal drift and an accumulated temporal drift associated with the presentation timeline; and
updating (<NUM>) the accumulated temporal drift based on whether the video frame is to be included in the presentation timeline.