Reduced video player start-up latency in HTTP live streaming and similar protocols

Technology for processing an ordered sequence of frames of a media program intended for play in an HTTP Live Streaming (HLS) player. Processing a first ordered subset of frames of the sequence. Processing a second ordered subset of frames of the sequence separate from the first subset. The second subset following on the first forms a third ordered subset of frames of the media program. Processing the second subset includes creating a manifest file of the subset. The playing time of processed frames of the first subset is at least equal to the processing time of the second subset. Processing the first subset can include creating a manifest file of the subset. The media program can include ads (at least a pre-roll ad) and content. The first ordered subset can be the pre-roll ad. The second ordered subset can be the content and remaining ads (e.g., mid-roll and post roll).

FIELD OF THE TECHNOLOGY

The technology disclosed herein relates to on-line audio media and audiovisual media, hereinafter “on-line media.” In exemplary embodiments, the technology relates to decreasing the time from a play request to playing of streaming media transported under HTTP Live Streaming (HLS) and similar protocols.

DETAILED DESCRIPTION

Reference will now be made in detail to implementations of the technology. Each example is provided by way of explanation of the technology only, not as a limitation of the technology. It will be apparent to those skilled in the art that various modifications and variations can be made in the present technology without departing from the scope or spirit of the technology. For instance, features described as part of one implementation can be used on another implementation to yield a still further implementation. Thus, it is intended that the present technology cover such modifications and variations that come within the scope of the technology.

Content delivery describes the delivery of media “content” such as audio or video or computer software and games over a delivery medium such as broadcasting or the Internet. Content delivery has two parts: delivery of finished content for digital distribution, with its accompanying metadata; and delivery of the end product to the end-user.

Streaming media is media that is received by and presented to an end-user while being delivered by a streaming provider. The name refers to the delivery method of the medium rather than to the medium itself. The distinction is usually applied to media that are distributed over telecommunications networks, e.g., “on-line,” as most other delivery systems are either inherently streaming (e.g., radio, television) or inherently non-streaming (e.g., books, video cassettes, audio CDs). Hereinafter, on-line media and on-line streaming will be referred to as “media” and “streaming.” The verb ‘to stream’ is also derived from this term, meaning to deliver media in this manner. Internet television is a commonly streamed medium.

HTTP Live Streaming (also known as HLS) is an HTTP-based streaming media communications protocol implemented by Apple Inc. as part of their QuickTime X and iPhone software systems. HLS works by breaking the overall media stream into a sequence of small HTTP-based file downloads, each download loading one short “chunk” of an overall potentially unbounded transport stream. As the stream is played, the client (e.g., the media player) may select from a number of different alternate streams containing the same material encoded at a variety of data rates, allowing the streaming session to adapt to the available data rate. At the start of the streaming session, the player downloads/receives a manifest containing the metadata for the various sub-streams which are available. Since its requests use only standard HTTP transactions, HTTP Live Streaming is capable of traversing a firewall or proxy server that lets through standard HTTP traffic, unlike UDP-based protocols such as RTP. This also allows a content delivery network (CDN) to readily be implemented for any given stream.

Media distribution via the Internet is a source of problems for many reasons, including video encoding formats, bandwidth, playback device capabilities, different screen sizes and resolutions, and so forth. Some manufacturers provide a set of guidelines for media distributors as a standard or generally accepted way to stream media to that manufacturer's devices, such as a standard chunk size for streaming media. Certain video playback devices, such as the Apple IPAD, prefer video content to be streamed not continuously, but as a series of discrete “chunks” of video outlined in a manifest file, such as an m3u8 file, e.g., HLS. The IPAD receives the manifest file that contains links to each of the chunks of video content, and processes the manifest file to retrieve and play back each chunk in turn. However, the m3u8 format imposes several limitations, including the inability to embed other manifest files. This inability to embed other manifest files in the m3u8 format is a hurdle to delivering advertisements and other dynamic interstitial content in streaming video to such devices. In this disclosure, HLS represents the range of protocols that chunk content and employ a playlist/manifest file to manage playback.

Having disclosed some components of a computing system, the disclosure now turns toFIG. 1B, which illustrates an example media streaming system embodiment1000. The communications between the entities depicted inFIG. 1Bcan occur via one or more wired or wireless networks. Further, the devices can communicate directly, via the World Wide Web, or via an application programming interface (API). A playback device1002, such as a tablet device, smartphone, desktop or portable computer, set-top box, Internet-enabled television, media center PC, or any other suitable device, first makes a request to a media server1004for playback of media content, such as an episode of Star Trek. Typically, the media server1004resides in a network, such as the Internet, for example a third-party content distribution network such as Akami or Limelight.

In HLS, the media server1004receives the request and generates or fetches a manifest file1006to send to the playback device1002in response to the request. Example formats for the manifest file1006include the m3u and m3u8 formats. An m3u8 file is a specific variation of an m3u encoded using UTF-8 Unicode characters. The m3u file format was initially used in the WINAMP Media Player for only audio files, but has since become a de facto playlist standard on many media devices for local and/or streaming media, including music and other media types. Many media devices employ variations of the m3u file format, any of which can be used according to the principles set forth herein. A manifest file can include links to media files as relative or absolute paths to a location on a local file system, or as a network address, such as a Uniform Resource Identifier (URI) path. The m3u8 format is used herein as a non-limiting example to illustrate the principles of manifest files including non-standard variants.

The manifest file1006includes a list of pointers to sequential chunks of the requested media content. The contents of an example m3u8 manifest file for a 32-second video asset is provided below:

Before or at the time of the request, the media server1004generates or identifies the chunks of the requested media content as streaming media content1010. The chunks of the streaming media content1010are generated, either by the media server1004, the content producer, or some other entity, by splitting the original media content1008. Upon receiving the manifest file1006, the playback device1002can fetch a first chunk for playback from the streaming media content1010, and, during playback of that chunk, fetch a next chunk for playback after the first chunk, and so on until the end of the media content. The functionality of the entities depicted inFIG. 1Bcan be split or merged across entities. For example, an first ingestion server can ingest the original media content1008to produce the streaming media content1010, while a second customer-facing server can service requests for manifest files, and a third media streaming server streams the actual chunks indicated by the links in the manifest file.

The term “video advertising” is generally accepted to refer to advertising that occurs on Internet television. It is served before, during and after a video stream on the internet. The typical types of advertising units used are pre-roll, mid-roll and post-roll, with these ad units/pods like the traditional spot advertising seen on television, although often they are shorter than their TV counterparts if they are run online.FIG. 2illustrates an HLS-formatted media program200of content chunks210, a pre-roll ad chunk220, mid-roll ad chunks230, and post-roll ad chunks240. An example manifest file generated by technology for segmenting content files and ad files for compatibility with HLS is provided below:

As this example illustrates, while a standard manifest file is unable to contain other manifest files, the manifest file can contain links to chunks from multiple sources. For instance, the manifest file can include links to chunks on a single-content server and one or more different advertising servers. Alternatively, the manifest file can include links to chunks on a single server that stores both content and advertising. The content chunks, for example, can be stored at different locations, although in a common scenario, a content producer stores all the content chunks on a single server or cluster of servers. When the playback device receives the manifest file, the playback device fetches a first chunk and begins playing that chunk. As that chunk is playing, the playback device fetches the next chunk, and so on.

In some situations, it is advantageous for the content provider to determine the ads to deliver with the content upon request of the content. For example, when ads are selected based on characteristics of the end user. As described above, the ads to accompany content may not come from the same URL as the content. Further, there are many end user devices that employ HLS for playing video content, e.g., the IPAD. There is much content not in HLS format, e.g., content formatted for Adobe Flash. Longer content not in HLS format is preferably formatted in HLS format before playing at an HLS end user device. In some situations, it is advantageous for the content provider to format the content for delivery only upon a play request from the end user device. For example, in streaming a live event.

For such non-HLS content to be delivered to an HLS player with ads determined at the time of the request to play the content, it takes time to: retrieve ads and content, format the ads and content into HLS chunks, and create and deliver a manifest file for the combined content and ads. For example retrieving a single ad identifier from Google's DoubleClick system can take more than 500 milliseconds (or half a second). When ad-supported content and the ads associated with that content are retrieved, formatted, and specified by a single HLS-compliant manifest, the video player can not start until it receives the manifest. This can increase the start-up time of the video player to a point where it becomes unacceptable for the viewer. Potential loss of viewers can occur. Note that some content and some ads may already be in HLS format. However, joining both content and ads into a media program covered by a manifest still requires time, e.g., to retrieve ad-supported content and the supporting ads, and to create the manifest joining them.FIG. 3is a notional timeline300illustrating typical media programming200processing310for playing320as an HLS stream, resulting in T_STARTUP—0330.

Implementations of the technology process (e.g., identify, retrieve, segment and manifest if necessary, and deliver with manifest) and play the HLS pre-roll ad while the remaining content and ads are processed. This approach can be especially useful in at least those situations where both 1) the pre-roll ad can be processed, and play of the pre-roll ad can be started, in less time than the entire content and all the ads can be processed, and 2) the remaining content and ads can be processed before the end of play of the pre-roll ad. The technology provides utility even where 2) is not met, when the delay between the end of the pre-roll and the beginning of play of the remaining content and ads is tolerably short.

FIG. 4illustrates methods400for processing an ordered sequence of frames of a media program intended for play in a device operative to play the program from an HLS stream, and where a first ordered subset of frames is a pre-roll ad410, and a second ordered subset of frames includes content420and the remaining ads430of the media program. In such methods, the first ordered subset is processed independently440from processing the second ordered subset460. In some embodiments, processing the first ordered subset, e.g., pre-roll ad410not in HLS format, can include segmenting the subset and creating a manifest file. In other embodiments, non-HLS pre-roll ads can be played directly by the destination player without re-formatting in to HLS form. Playing the pre-roll ad450can provide sufficient time to process the content and remaining ads. At some time no later than the end of play of the first ordered subset, the second ordered subset of frames of the media program can be processed. In cases such as those illustrated inFIG. 4, the content420and remaining ads430are processed460as the second subset in HLS format. At the end of play of the processed first ordered subset450, the processed second ordered subset can be played470.FIG. 5illustrates how the pre-roll ad220can be the first subset, and the content210and remaining ads230,240can be the second subset. Dividing the stream as shown inFIG. 5can result in two manifest files (where both the first and second subsets are in HLS format) as follows:

FIG. 6illustrates a notional timeline comparing typical media programming200processing310for playing320as an HLS stream to notional timelines for processing a stream500of the present technology. As withFIG. 3, the time for processing310the entire stream200can be so long that some viewers are lost. In implementations of the present technology, the time to process the second subset460is less than the time for processing440and playing450the first subset. Note that the second subset can begin processing460at T=0 or as late as a time that allows the second subset to be processed460no later than the first subset finishing play450. The second subset can then be played470at the end of playing the first subset450, resulting in substantially continuous play610with startup latency T_STARTUP—1620<T_STARTUP—0330.

In some implementations, the technology assesses the ads included in the media program. If there is a pre-roll ad, and the technology determines that separately processing the pre-roll ad will improve the viewer experience, then the technology performs the functions described herein using the pre-roll ad as the first subset of frames. The technology can determine that separately processing the pre-roll ad will improve the viewer experience in a variety of ways. For example, the technology can count the total number of ads to be manifested into a single manifest; if more than a threshold of ads is to be manifested into a single manifest, then the technology can divide the frames of the media program into at least two manifest files, e.g., the first manifest file including the pre-roll ad. The threshold can be programmable, including settable in real time (e.g., as a function of the quality and bandwidth of the communications link), determined by a profile (e.g., of the user, of the content, the playing device, the quality of the communications link). For example, if the media program includes a pre-roll ad, and the total number of ads is greater than five (5), then the technology disclosed herein can be employed.

Assessing ads can include estimating or determining the processing time associated with a single-manifest versus multiple manifest approaches. Again, a threshold can be used. Also, the time for processing single versus multiple manifest files can be compared to determine whether the viewing experience is improved by single-manifest versus multiple-manifest approaches. Note that the viewing experience can be considered “improved” even if the total latency of the multiple manifest approach is equal to or greater than the total latency of the single manifest approach. For example, while a single manifest approach might result in six (6) seconds of latency, and the corresponding double manifest approach results in three (3) seconds of initial latency and then four (4) second of latency from the pre-roll ad to the content, this can be considered a better viewing experience if the criteria is that no latency be greater than four (4) seconds, even though the overall latency of the multiple manifest approach is seven (7) seconds versus six (6) seconds for the single manifest approach.