Methods, devices, and systems for publishing key pictures

Techniques for publishing key pictures at a server and using the key picture information for operations on a client device are described herein. In accordance with various embodiments, the server creates an intermediate unit including key picture(s) from media content. The server then generates map(s) specifying for each key picture, a type, properties, and an association to the intermediate unit. The server also packages the intermediate unit into key picture unit(s) according to the map(s). The client device obtains a manifest for the media content referencing the key picture(s) and the type of each key picture. Upon receiving a request to perform operation(s) on a respective key picture, the client device obtains a corresponding key picture unit and parses the manifest to determine a respective type. Based on the respective type matches a criterion, the client device performs the operation(s) on the key picture unit.

CROSS-REFERENCE TO RELATED APPLICATION

This application claims the benefit of priority from Indian Patent Application No. 202121050765, filed on Nov. 5, 2021, the contents of which are hereby incorporated by reference.

TECHNICAL FIELD

The present disclosure relates generally to multimedia content streaming and, more specifically, to publishing key pictures for streaming multimedia content.

BACKGROUND

Some previously existing packagers store a key picture segment as a separate entity to maintain the compatibility between outputs in various formats, e.g., producing outputs according to specifications such as Dynamic Adaptive Streaming over HTTP (DASH), HTTP Live Streaming (HLS), Smooth Streaming, HTTP Dynamic Streaming (HDS), etc. When storing the media segments and the key picture segments separately, extracting the key pictures and storing them as separate entities drive up processing and storage costs. Further, even though a single media segment often includes more than one key picture (e.g., several I-pictures and/or IDR pictures), previously existing packagers merely publish the first key picture to the client. Without publishing every key picture within a segment, some key picture(s) at the sub segment level are not accessible by end users. Additionally, previously existing packagers do not specify whether a key picture is an I-picture or an IDR picture. Consequently, features that require access to a particular type of key picture cannot be offered, thus impacting the user experience.

DESCRIPTION OF EXAMPLE EMBODIMENTS

Overview

In accordance with some embodiments, a method for publishing key pictures described herein solves the aforementioned addressability and cost issues by making every key picture addressable and/or accessible to end users. In some embodiments, an intermediate packager at a server (e.g., a headend) creates and maintains mapping(s) between a parent intermediate segment and key pictures within. A just-in-time (JIT) packager at the headend then publishes the type of each key picture to the clients, e.g., using attributes, tags, parameters, and/or values in a manifest. Publishing the type of each key picture maintains backward compatibility and involves minimal changes on the client side. For client devices that are capable of interpreting the new parameter in the manifest, such client devices can utilize the key picture information for improved user experience. For other client devices, the new attributes, tags, parameters, and/or values are ignored without causing errors.

In accordance with various embodiments, a key picture packaging and publishing method is performed at a server that includes including one or more processors and a non-transitory memory. The method includes creating an intermediate unit from media content, where the intermediate unit includes one or more key pictures. The method further includes generating one or more maps specifying a type of each of the one or more key pictures, properties of the one or more key pictures within the intermediate unit, and an association of each of the one or more key pictures to the intermediate unit. The method also includes packaging the intermediate unit into one or more key picture units according to the one or more maps.

In accordance with various embodiments, a method of using the key picture information for operations is performed at a client device that includes including a processor and a non-transitory memory. The method includes obtaining a manifest for media content, where the manifest references one or more key pictures in the media content and specifies a type of each of the one or more key pictures. The method further includes receiving a request to perform at least one operation on a respective key picture of the one or more key pictures. The method also includes obtaining a key picture unit within the media content corresponding to the respective key picture and parsing the manifest to determine a respective type of the respective key picture in response to the request. The method additionally includes performing the at least one operation on the key picture unit in accordance with a determination that the respective type of the respective key picture matches a criterion.

EXAMPLE EMBODIMENTS

Methods, devices, and systems in accordance with various embodiments described herein include a headend that generates mapping(s) and publishes key picture information according to the mapping(s). Publishing every key picture information solves the aforementioned cost and addressability issues in previously existing systems. By making every key picture addressable and publishing the key picture information to be accessible by the end user, key pictures are not stored separately, thus lowering storage and processing cost. Further, because each key picture is addressable, for trick mode, thumbnail scrubbing, seeking within a segment, and/or targeted content substitution, each individual key picture can be accessed and utilized to provide more access points and access points that are closer to the point of interest, thus improving user experience.

Reference is now made toFIG.1, which is a block diagram of an exemplary multimedia content delivery system100in accordance with some embodiments. In some embodiments, the multimedia content delivery system100includes a server110(e.g., a headend), a content delivery network (CDN)130, and a client device140. Although a single server110, a single CDN130, and a single client device140are illustrated inFIG.1, the system100can include one or more servers110as well as one or more client devices140, and can include zero, one, or more CDNs130. For instance, the CDN(s)130can be included in the system100for scalability. As such, the server110provides multimedia content to the client device(s)140, optionally via the CDN(s)130. For the sake of simplicity, the subject matter will be described hereinafter for the most part with reference to a single server110, a single client device140, and a single CDN130.

In some embodiments, the server110includes an encoder112for encoding multimedia content from a content storage114(e.g., live or video-on-demand (VOD) content), an intermediate packager116for receiving the encoded content from the encoder112and creating intermediate units117and corresponding metadata118, and a just-in-time (JIT) packager120for identifying relevant portion(s) within the intermediate units117according to request(s) from the client device140and converting the relevant portion(s) to a suitable format for adaptive bitrate (ABR) streaming.

As used herein, the multimedia content (also referred to hereinafter as “media content”, “media content item(s)”, “media asset”, or “content”) provided by the server110and received by the client device140can include any multimedia data, such as visual data, audio data, and/or text, etc. Further, the content prepared by the server110and received by the client device140can be in a variety of encoding and/or packaging formats. For example, the encoder112can encode the content from the content storage114and output content including video and/or audio encoding types such as MPEG 2, MPEG 1, advanced video coding (AVC), AC3, high efficiency video coding (HEVC), etc. In another example, the intermediate packager can package the encoded content to create the intermediate units117according to Common Interchange Format (CIF) (e.g., for MPEG 2 transport stream (TS)) or Common Media Application Format (CMAF) (e.g., for FMP4 segment according to the ISO Base Media File Format (ISO_BMFF)). In yet another example, based on the request from the client device140, the JIT packager120can package the intermediate units117content according to Dynamic Adaptive Streaming over HTTP (DASH), HTTP Live Streaming (HLS), Smooth Streaming, or HTTP Dynamic Streaming (HDS) format and construct manifest in accordance with HLS or DASH.

In some embodiments, the intermediate packager116identifies key pictures in the encoded content from the encoder112when creating the intermediate units117. For example, the intermediate packager116can parse an MPEG 2 TS from the encoder112and identify a transport stream packet header with payload_unit_start_indicator bit set and the adaptation header field with random_access_indicator bit set. The combination of the payload_unit_start_indicator bit and the random_access_indicator bit indicates the start of a key picture. In another example, for BMFF content, the key picture can be identified based on the information in the trun box, which includes details about the samples.

Upon identifying the key pictures while packaging the intermediate units, the intermediate packager116also creates the metadata118, e.g., CIF or CMAF manifest, and records mappings as part of the metadata118in accordance with some embodiments. For example, the intermediate packager116(e.g., a CIF packager) receives the MPEG 2 TS content from the encoder112and creates fixed duration MPEG 2 TS segments as the intermediate units117. The intermediate packager116, while creating CIF segments as the intermediate units117, also creates mapping file(s) depicting the association between every key picture and the CIF segment (e.g., the parent intermediate segment) the key picture resides in. In some embodiments, the mapping file includes the type of the key picture (e.g., an I-picture or an instantaneous decoding refresh (IDR) picture) and properties of the key picture. Such properties include, but are not limited to, the start offset of the key picture within the parent intermediate segment, the size of the key picture, the parent intermediate segment, and/or the timestamp associated with the key picture, etc. As shown inFIG.1, in some embodiments, the intermediate packager116maintains one mapping file for each bandwidth to ensure the proper start offsets of the key pictures for the bandwidth. Exemplary associations of an intermediate unit with key pictures are shown inFIGS.2A-2Cand described in detail below.

On the receiving end, in some embodiments, the client device140includes a playlist controller142for requesting and obtaining playlist and/or manifest files and a player144for playing the content from the server110via the CDN130according to the playlist and/or the manifest. In some embodiments, the client device140also includes a splicer146for splicing targeted content from a targeted content store148and/or inserting the target content, e.g., advertisement substitution and/or splicing.

It should be noted that one or more components and/or functions of the server110and/or the client device140may be distributed and/or re-arranged. For example, the content storage114and/or the encoder112can be on a different and distinct server from the server hosting the intermedia packager115and/or the JIT packager120. As such, the server110and/or the client device140in the exemplary content delivery system100can include more, less, and/or different elements than shown inFIG.1. Each of the components in the content delivery system100can include appropriate hardware, software, and/or firmware to perform the operations attributed to the element herein. Operation(s) attributed to an element in the content delivery system100herein should not be considered binding and in some embodiments, other element(s) in the exemplary system100may additionally or alternatively perform such operation(s).

FIGS.2A-2Care diagrams200A-200C illustrating the association of an exemplary intermediate unit with key pictures in accordance with some embodiments. InFIG.2A, the intermediate packager (e.g., the intermediate packager116,FIG.1) identifies three key pictures within an exemplary intermediate unit210, e.g., IDR picture 1220, an I-picture230, and IDR picture 2240. For example, the intermediate unit210can be a MPEG 2 TS segment 143974041678726.ts. Within the MPEG 2 TS segment, IDR picture 1220is located at offset564, I-picture230is located at offset 564000, and IDR picture 2240is located at offset 1128000.

Upon identifying the key pictures220-240, in some embodiments, the intermediate packager creates a map as shown inFIG.2B. The map illustrates the intermediate unit210being the parent unit of the three key pictures220-240. In some embodiments, the intermediate packager creates a mapping file based on the map. An exemplary map file is shown inFIG.2C. In some embodiments, the intermediate packager stores and maintains the map inFIG.2Band/or the mapping file inFIG.2Cas the metadata118(FIG.1).

As shown inFIG.2C, in some embodiments, the exemplary mapping file identifies IDR picture 1220(e.g., “segment”: “143974041678726”), I-picture230(e.g., “segment”: “143974041858726”), and IDR picture 2240(e.g., “segment”: “143974042038726”) as virtual entities within the parent segment210(e.g., “parent_segment”: “143974041678726.ts”). Further, the mapping file specifies the type of each of the key pictures, e.g., IDR picture 1220and IDR picture 2240with “type”: “IDR” and I-picture220with “type”: “I”. Additionally, in some embodiments, the mapping file also specifies properties of each of the key pictures, such as the location of the key picture in terms of the start offset within the parent media segment210, the size of the key picture in terms of bytes, and/or the timestamp associated with the key picture, etc.

In some embodiments, for linear content (e.g., live manifest), the intermediate packager maintains the map and/or the mapping file to ensure that key pictures specified in the map and/or the mapping file pertaining to the parent segments are within the time shift buffer depth to accommodate playbacks. In some embodiments, for VOD content (e.g., non-growing manifest), the intermediate packager maintains the map and/or the mapping file to ensure that the key pictures specified in the map and/or the mapping file are in the media segments of the VOD content. With the map and/or the mapping file, the intermediate packager does not store key picture segments as separate physical entities, thus reducing storage and processing costs.

In some embodiments, the JIT packager (e.g., the JIT packager120,FIG.1) receives a manifest request from the client device (e.g., the client device140,FIG.1). The manifest being requested can be a Media Presentation Description (MPD) file for DASH or an I-frame manifest for HLS in accordance with various embodiments. In response to the manifest request, the JIT packager fetches the latest and the appropriate mapping file (e.g., for the proper bandwidth). In some embodiments, having obtained the mapping file, the JIT packager parses the mapping file, extracts the key picture information, and places the key picture information at appropriate locations in the manifest, e.g., in the I-frame adaptation set section in the MPD for DASH as shown inFIG.3Bor in the I-frame manifest/playlist for HLS as shown inFIG.4B.

FIG.3Aillustrates an exemplary manifest file300A without exposing the key picture information for every key picture, andFIG.3Billustrates an exemplary manifest file300B with the key picture information for every key picture in accordance with some embodiments. InFIG.3A, an I-frame adaptation set section in an exemplary MPD file for DASH includes a first entry310-1and a second entry310-2along the segment timeline. Two key pictures at the start of the entries310-1and310-2can be accessed by the client, e.g., one key picture with timestamp “143974041678726” and another key picture with timestamp “143974042218726”. As shown inFIG.3A, in previously existing systems, other than the first key picture within each segment, information of subsequent key pictures within a segment is not published and not accessible by end users.

Assuming there is a key picture every 2 s within each segment, as shown inFIG.3B, for each segment, the MPD includes the key picture information for three key pictures. For example, instead of having one entry310-1for the 6 s segment “143974041678726” as shown inFIG.3A, inFIG.3B, as indicated by a first box320-1, the JIT packager places three entries representing the key picture information for three key pictures within the segment. Further, the JIT packager indicates the type for each of the key pictures, e.g., specifying ty=“IDR” for key picture “143974041678726”, ty=“I” for key picture “143974041858726”, and ty=“IDR” for key picture “143974042038726”. In another example, instead of having one entry310-2for the 6 s segment “143974042218726” as shown inFIG.3A, inFIG.3B, as indicated by a second box320-2, the JIT packager places three entries representing the key picture information for three key pictures within the segment. Further, the JIT packager indicates the type for each of the key pictures, e.g., specifying ty=“IDR” for key picture t=“143974042218726”, ty=“I” for key picture t=“143974042398726”, and ty=“I” for key picture t=“143974042578726”.

In some embodiments, the custom attribute s@ty shown inFIG.3Bfor DASH specifies whether a key picture is a safe seek point, e.g., whether the key picture is an I-picture or an IDR picture. In the case of the key picture being an IDR picture, the key picture is a safe seek point, i.e., any subsequent picture to the IDR picture can be decoded without referencing any picture prior to the IDR picture. As such, with the key picture information in the MPD, the client devices have access to any key picture and to the proper type of key picture. Further, since DASH MPD is an XML document, the client devices can use a standard XML parser to parse the MPD as shown inFIG.3B. In the case of a client device not recognizing the custom s@ty attribute, the client device ignores the attribute in accordance with some embodiments, thus ensuring backward compatibility. Accordingly, the MPD as shown inFIG.3Bcan be used by any client device capable of processing XML documents.

FIG.4Aillustrates an exemplary manifest file400A without exposing key picture information for every key picture andFIG.4Billustrates an exemplary manifest file400B with the key picture information for every key picture in accordance with some embodiments. An HLS fragmented MP4 I-frame variant manifest identifies a plurality of media files, with each media file being identified by a media file URL preceded by an EXTINF tag and a duration of the media file. InFIG.4A, a first entry410-1identifies a media file 143694618050455.m4f that has a duration of 6 s, and a second entry410-2identifies a media file 143694618590455.m4f that also has a duration of 6 s. Each of these media files corresponds to a key picture segment. However, between the first entry410-1and the second entry410-2, there are multiple key pictures. As shown inFIG.4A, in previously existing systems, other than the first key picture being packaged into I-frame segment 143694618050455.m4f and the second key picture being packaged into I-frame segment 143694618590455.m4f, key pictures in between the first key picture and the second key picture are not published and not accessible by end users.

Assuming there is a key picture every 2 s within each fragment, as shown inFIG.4B, the manifest for HLS includes the key picture information for three key pictures within the fragment. For example, instead of having one entry410-1for the 6 s media file 143694618050455.m4f as shown inFIG.4A, inFIG.4B, as indicated by a first box420-1, the JIT packager places three entries representing the key picture information for three key pictures within the fragment. Further, the JIT and indicates the type for each of the key pictures, e.g., a 2 s media file 143694618050455.m4f with EXT-X-KEY-FRAME-TYPE: “IDR”, followed by a 2 s media file 143694618230455.m4f with EXT-X-KEY-FRAME-TYPE: “I”, and followed by a 2 s media file 143694618410455.m4f with EXT-X-KEY-FRAME-TYPE: “IDR”. In another example, instead of having one entry410-2for the 6 s media file 143694618590455.m4f as shown inFIG.4A, inFIG.4B, as indicated by a second box420-2, the JIT packager places three entries representing the key picture information for three key pictures within the fragment and indicates the type for each of the key pictures, e.g., a 2 s media file 143694618590455.m4f with EXT-X-KEY-FRAME-TYPE: “IDR”, followed by a 2 s media file 143694618770455.m4f with EXT-X-KEY-FRAME-TYPE: “IDR”, and followed by a 2 s media file 143694618950455.m4f with EXT-X-KEY-FRAME-TYPE: “I”.

In the exemplary HLS manifest400B, similar to the custom attribute s@ty described above with reference toFIG.3B, the EXT-X-KEY-FRAME-TYPE tag is a custom tag in some embodiments. In the case of a client device not recognizing the EXT-X-KEY-FRAME-TYPE tag, the client device ignores the custom tag, thus ensuring backward compatibility. Even if a client device does not recognize the custom attribute s@ty and/or the custom tag EXT-X-KEY-FRAME-TYPE, because more key pictures are available, the client device benefits from features such as more fine grained trick mode playback and/or thumbnail scrubbing. Accordingly, the manifest as shown inFIGS.3B and4Bcan be used by any client device for improved user experience and the key picture publishing method described herein makes every key picture available without storing the key pictures as separate entities.

FIG.5is a sequence diagram500illustrating generating and publishing key picture information in accordance with some embodiments. In some embodiments, in step510, the intermediate packager116obtains the encoded content from the encoder112. In some embodiments, in step520, the intermediate packager116packages the encoded content into intermediate units and generates maps (e.g., as shown inFIG.2B) and/or mapping files (e.g., as shown inFIG.2C) to record key picture information and to establish the associations of the key pictures with a respective parent intermediate unit as described above with reference toFIGS.2A-2C. When the JIT packager120receives a manifest request in step530, the JIT packager generates at least a portion of the manifest (e.g., the I-frame adaptation set portion of the MPD file for DASH as shown inFIG.3Bor adding the custom tag EXT-X-KEY-FRAME-TYPE to the manifest file for HLS as shown inFIG.4B) based on the maps and/or the mapping files in step540.

Further, when the JIT packager120receives a key picture request from the client device140in step550, the JIT packager120obtains the bandwidth information along with the key picture request. Accordingly, in step560, the JIT packager120ensures it has the up-to-date mapping file corresponding to the bandwidth (e.g., one of the mapping files from the metadata117inFIG.1) and based on the mapping file, obtains the corresponding intermediate unit from the intermediate packager116. Further, in step570, based on the mapping file, the JIT packager120fetches the required portion for the requested key picture from the parent intermediate unit, constructs a key picture unit in accordance with some embodiments. In some embodiments, in step580, the JIT packager120sends the key picture unit to the client device140.

In some embodiments, instead of the JIT packager120obtaining the intermediate unit and the mapping file in step560and fetching the required portion in step570, the JIT packager120forwards the key picture request to the intermediate packager116. In such embodiments, the intermediate packager116keeps the mapping file within the intermediate packager116, e.g., stored as the metadata118(FIG.1). When the key picture request comes to the JIT packager120, the JIT packager120forwards the key picture request to the upstream intermediate packager116along with the bandwidth requirement. In some embodiments, the intermediate packager116then locates the mapping file corresponding to the bandwidth, uses the mapping file to fetch the required portion from its parent intermediate unit, and returns the required portion to the JIT packager120in an intermediate format, e.g., CIF or CMAF format. In some embodiments, the JIT packager120converts the required portion to the key picture unit, e.g., into an HLS or DASH segment.

In some embodiments, a key picture unit is a key picture segment. As used herein, a key picture segment refers to a segment that includes one I-picture or one IDR picture. In previously existing solutions, the key picture segments are created by the intermediate packager and stored within the intermediate packager to be retrieved by the JIT packager in response to the request from the client device. The storage and processing overhead is high for such previously existing solutions. With the key picture publishing methods, devices, and systems described herein, the key pictures are not created and stored at the intermediate level. The maps and the mapping files allow the intermediate packager116to create virtual entities corresponding to the key pictures within each intermediate unit. As shown inFIG.5, when the key picture request from the client device140comes to the JIT packager120, e.g., in step550, the JIT packager120obtains the key picture data based on the mapping file from the parent intermediate unit and then repackages the key picture data to the key picture segment. As such, relative to previously existing solutions where physical entities for the key pictures are created and stored at the intermediate level, creating virtual entities and references corresponding to the key pictures by the intermediate packager116reduces processing and storage costs.

Publishing the key picture information for every available key picture enhances ABR content viewing experience. For example, when previewing the content through thumbnail scrubbing, no important scene is missed, e.g., every key picture can be previewed. In another example, because each key picture is accessible, I-frame trick mode is smoother and seeking at a sub segment level is possible. For live content, tuning to and/or starting to play from the last IDR picture within the latest segment available within a live manifest reduce the latency to the live point.

FIGS.6A and6Bare diagrams600A and600B illustrating targeted content substitution in accordance with some embodiments. In previously existing OTT advertisement splicing or substitution solutions, substitution is possible at the segment level, i.e., substituting segment(s) with advertisement segments. Using the key picture publishing method described herein, once the key picture information for every key picture and the corresponding key picture type are published, the client device (e.g., the client device140,FIG.1) can perform client-side targeted content insertion, substitution, and/or splicing at the sub segment level.

InFIG.6A, a media content item includes a plurality of units, e.g., unit 1610-1, unit 2610-2, unit 3610-3, unit 4610-4, etc. Within a given segment, e.g., unit 3610-3, there are three IDR pictures, e.g., IDR picture 1620-1with timestamp t1, IDR picture 2620-2with timestamp t2, and IDR picture 3620-3. The gap between the start of IDR picture 1620-1at t1and the start of IDR picture 2620-2at t2is, for example, 2 seconds. InFIG.6B, the client device stores a targeted content item635, e.g., an advertisement that is 2 seconds long and stored in the targeted content store148(FIG.1). With appropriate signaling, e.g., scte-35 signaling, the client device (e.g., the splicer146,FIG.1) can replace a portion of the content from the start of IDR picture 1620-1at time t1till the start of IDR picture 2620-2at time t2with pre-stored targeted content. Once replaced, because the transition to the media content item is at the start of IDR picture 2620-2, e.g., any subsequent picture to IDR picture 2620-2can be decoded without referencing any picture prior to IDR picture 2620-2, the playback works smoothly without disruption.

The targeted content635has such a short duration that it does not impact the viewing experience of the sports content provided by unit 3610-3. Inserting small chunks of targeted content at the sub segment level allows context-sensitive advertisement insertion, which is more appealing and relatable to end users. For example, during a sports game, e.g., after a well-known player scores a goal, a short sports equipment advertisement featuring the player is more appealing and relatable to the viewers. Such contextual advertisements in small chunks are less irritating, thus potentially influencing the users more effectively and increasing the chance of receiving a response.

FIGS.7A and7Bare flowcharts illustrating a method700of packaging and publishing key pictures in accordance with some embodiments. In some embodiments, as represented by block710, the method700is performed at a server, e.g., the server110inFIG.1hosting a headend. In some embodiments, the server includes one or more processors (e.g., one or more processors for the encoder112, the intermediate packager116, and/or the JIT packager120inFIG.1) and a non-transitory memory (e.g., a non-transitory memory for storing the content in the content storage114, the intermediate units117, and/or the metadata118inFIG.1).

The method700begins with the server creating an intermediate unit from media content as represented by block720, e.g., the intermediate packager116receiving encoded media content from the encoder112and creating the intermediate units117from the encoded media content. In some embodiments, the intermediate unit includes one or more key pictures. For example, the intermediate unit can be a segment of CIF MPEG 2 or CMAF BMFF format, and each CIF segment or CMAF fragment can have one or more I-pictures or IDR pictures. InFIG.2, the intermediate unit210includes three pictures210-240.

The method700continues, as represented by block730, with the server (e.g., the intermediate packager116,FIG.1) generating one or more maps specifying a type of each of the one or more key pictures, properties of the one or more key pictures within the intermediate unit, and an association of each of the one or more key pictures to the intermediate unit. For example, as shown inFIG.1and in step520ofFIG.5, while creating the intermediate segments, the intermediate packager116creates maps and/or stores the maps in the metadata118. In another example, inFIGS.2B and2C, the intermediate packager116generates a map and/or a mapping file depicting the association between every key picture and the parent intermediate segment that the key pictures reside in.

In some embodiments, as represented by block732, the type of each of the one or more key pictures indicates an I-picture or an instantaneous decoding refresh (IDR) picture, e.g., in the mapping file as shown inFIG.2C, each of the key pictures has a type value of “IDR” or “I”. In some embodiments, as represented by block734, the properties of the one or more key pictures include one or more of a start offset of a respective key picture, a size of the respective key picture, and a timestamp associated with the respective key picture. For example, in the mapping file as shown inFIG.2C, the intermediate packager specifies the start offset of each key picture within the parent intermediate unit, the size of the key picture, and the timestamp associated with the key picture. In some embodiments, for trick mode, as represented by block736, when the media content is linear content, the one or more key pictures specified in the one or more maps are within a time shift buffer depth for the linear content. Also for trick mode, in some other embodiment, as represented by block738, when the media content is a video-on-demand (VOD) media content item that includes a plurality of segments, the one or more maps further include mappings of a plurality of key pictures to a plurality of intermediate units corresponding to the plurality of segments.

In some embodiments, as represented by block740, generating the one or more maps includes generating a mapping file for a bandwidth, where the properties in the mapping file specify a start offset for a respective key picture of the one or more key pictures packaged according to the bandwidth. For example, inFIG.1, the intermediate packager116maintains the mapping file for each video bandwidth to capture the proper start offset for a given bandwidth. In such embodiments, as represented by block742, when the server receives a key picture request for a respective key picture of the one or more key pictures that includes bandwidth parameters, the server locates the mapping file corresponding to the bandwidth parameters. As such, each key picture would have the proper start offset for the bandwidth.

Turning toFIG.7B, as represented by block750, the method700continues with the server (e.g., the JIT packager120,FIG.1) packaging the intermediate unit into one or more key picture units according to the one or more maps. As such, the JIT packager picks up the relevant intermediate unit and converts it for ABR streaming, e.g., packaging the content in formats such as DASH or HLS and constructing manifest. In some embodiments, as represented by block752, packaging the intermediate unit into the one or more key picture units includes receiving a request for a respective key picture of the one or more key pictures, fetching a portion including the respective key picture from the intermediate unit based on the one or more maps in response to the request, and constructing a key picture unit based on the portion and providing the key picture unit. Further, in such embodiments, as represented by block754, the request referencing the respective key picture is originated from a client device and includes one or more of a trick mode play, a thumbnail scrubbing, seeking a safe seek point, seeking to a latest unit of live content, and inserting targeted content in accordance with some embodiments.

For example, when a client device sends a key picture request during trick mode play, thumbnail scrubbing, seeking a safe seek point, seeking to a latest unit of live content, and/or inserting targeted content, in steps550and560ofFIG.5, the JIT packager120receives the key picture request and locates the up-to-date mapping file for the request. Further, in step570ofFIG.5, based on the mapping file, the JIT packager120fetches the required portion for the key picture from the parent intermediate unit and constructs the key unit, e.g., according to DASH or HLS format, and responds to the client device with the created key unit.

In some embodiments, as represented by block760, the method700further includes publishing the type of the key picture for each of the one or more key pictures in a manifest according to the one or more maps. In some embodiments, the publishing includes inserting in the manifest a reference to each of the one or more key pictures and the type of each of the one or more key pictures. In such embodiments, as represented by block762, the manifest is an HTTP Live Streaming (HLS) playlist or a Dynamic Adaptive Streaming over HTTP (DASH) Media Presentation Description (MPD) file.

For example, in steps530and540ofFIG.5, upon being requested by the client device for a manifest, e.g., a request for a DASH MPD file or a request for an I-frame manifest for HLS, the JIT packager120obtains the up-to-date and the appropriate mapping file along with the manifest for the intermediate units (e.g., a CIF manifest or a CMAF manifest). The JIT packager then parses the mapping file and includes the key picture information at the appropriate location. For example, inFIG.3B, the JIT packager inserts the key picture information in the I-frame adaptation set for DASH MPD. In another example, inFIG.4B, the JIT packager inserts the key picture information in the I-frame manifest or playlist for HLS.

FIGS.8A and8Bare flowcharts illustrating a method800of using published key picture information for operations on a client device in accordance with some embodiments. In some embodiments, as represented by block810, the method800is performed at a client device, e.g., the client device140inFIG.1. In some embodiments, the client device includes a processor (e.g., the processor for executing instructions for the playlist controller142, the player144, and/or the splicer146,FIG.1) and a non-transitory memory (e.g., a non-transitory memory for storing the targeted content in the targeted content store148,FIG.1).

The method800begins, as represented by block820, with the client device (e.g., the playlist controller142,FIG.1) obtaining a manifest for media content, where the manifest references one or more key pictures in the media content and specifies a type of each of the one or more key pictures. For example, inFIG.3B, the DASH MPD references the key pictures identified by the intermediate packagers and the type of each of the key pictures with the custom attribute s@ty. In another example, inFIG.4B, the HLS manifest references the key pictures and the type of each of the key pictures with EXT-X-KEY-FRAME-TYPE tag.

As represented by block840, the method800continues with the client device (e.g., the player144,FIG.1) receiving a request to perform at least one operation on a respective key picture of the one or more key pictures, e.g., trick mode, thumbnail scrubbing, seeking within segment, seeking to the start of the last IDR or the closest point to live content, and/or targeted content substitution, etc. As represented by block850, the method800continues with the client device (e.g., facilitated by the playlist controller142and/or the player144,FIG.1) obtaining a key picture unit within the media content corresponding to the respective key picture and parsing the manifest to determine a respective type of the respective key picture in response to the request. As represented by block860, the method800continues with the client device (e.g., facilitated by the player144,FIG.1) performing the at least one operation on the key picture unit in accordance with a determination that the respective type of the respective key picture matches a criterion.

In some embodiments, as represented by block852, the request to perform the at least one operation includes a trick mode play or a thumbnail scrubbing to access the respective key picture. In such embodiments, the criterion specifies the respective type of the respective key picture is an I-picture or an IDR picture. In other words, for trick mode play and/or thumbnail scrubbing, each individual key picture is accessible and the player can utilize any key picture for improved user experience.

In some embodiments, as represented by block854, the request to perform the at least one operation includes seeking a safe seek point. In such embodiments, the criterion specifies the respective type of the respective key picture is an IDR picture. In other words, for seeking within a unit such as a segment, the start of an IDR picture can be used as a safe seek point. As such, seeking within segment is possible when an IDR picture exists within the segment after the start of the segment. For example, without publishing every key picture within the segment, relying on the DASH MPD inFIG.3A, one IDR picture at the start of the segment with the timestamp 143974041678726 can be used by a seeking operation as a safe seek point. In contrast, once the key picture information for every key picture is published, inFIG.3B, either the IDR picture with the timestamp 143974041678726 at the start of the segment or the IDR picture with the timestamp 143974042038726 within the same segment can be used by a seek operation as a safe seek point. In another example, without publishing every key picture within the segment, relying on the HLS manifest inFIG.4A, one IDR picture at the start of 143694618050455.m4f can be used by a seeking operation as a safe seek point. In contrast, once publishing every key picture information, inFIG.4B, either the IDR picture at the start of 143694618050455.m4f or the IDR picture referenced by 143694618410455.m4f within the same unit can be used by a seek operation as a safe seek point.

Turning toFIG.8B, in some embodiments, as represented by block856, when the media content is live content, the request to perform the at least one operation includes seeking to a latest unit of the live content. In such embodiments, the criterion specifies the respective type of the respective key picture is an IDR picture. Further, in such embodiments, performing the at least one operation includes identifying the respective key picture being a last key picture within the latest unit, and seeking to the last key picture within the latest unit in accordance with the determination that the respective type of the respective key picture is the IDR picture. In other words, when seeking to the closest point of live content, seeking to the start of the last IDR picture in the latest segment of the live content is closer than seeking to the start of the latest segment, thus providing a closer to live viewing experience.

In some embodiments, as represented by block858, the request to perform the at least one operation includes inserting targeted content starting at the respective key picture. In such embodiments, the criterion specifies the respective type of the respective key picture is an IDR picture. Further, in such embodiments, performing the at least one operation includes identifying a first timestamp of the respective key picture and a second timestamp of an IDR picture within a unit according to the manifest, and inserting the targeted content within the unit between the first timestamp and the second timestamp. For example, as shown inFIGS.6A and6B, the targeted content635can be inserted within unit 3610-3between the start of two IDR pictures620-1and620-2to ensure smooth playback.

In some embodiments, as represented by block860, the method800further includes receiving the key picture unit within the media content according to the manifest, where the key picture unit includes the respective key picture and is generated by a packager at a server fetching a portion from an intermediate unit including the respective key picture according to a map and packaging the portion into the key picture unit.

FIG.9is a block diagram of a computing device900for packaging and publishing key pictures in accordance with some embodiments. In some embodiments, the computing device900corresponds to the server110inFIG.1and performs one or more of the functionalities described above with respect to the server110. While certain specific features are illustrated, those skilled in the art will appreciate from the present disclosure that various other features have not been illustrated for the sake of brevity, and so as not to obscure more pertinent aspects of the embodiments disclosed herein. To that end, as a non-limiting example, in some embodiments the computing device900includes one or more processing units (CPUs)902(e.g., processors), one or more input/output interfaces903(e.g., input devices, sensors, a network interface, a display, etc.), a memory906, a programming interface908, and one or more communication buses904for interconnecting these and various other components.

In some embodiments, the communication buses904include circuitry that interconnects and controls communications between system components. The memory906includes high-speed random access memory, such as DRAM, SRAM, DDR RAM or other random access solid state memory devices; and, in some embodiments, include non-volatile memory, such as one or more magnetic disk storage devices, optical disk storage devices, flash memory devices, or other non-volatile solid state storage devices. The memory906optionally includes one or more storage devices remotely located from the CPU(s)902. The memory906comprises a non-transitory computer readable storage medium. Moreover, in some embodiments, the memory906or the non-transitory computer readable storage medium of the memory906stores the following programs, modules and data structures, or a subset thereof including an optional operating system930, a content storage module933, an encoder940, an intermediate packager950, and a JIT packager960. In some embodiments, one or more instructions are included in a combination of logic and non-transitory memory. The operating system930includes procedures for handling various basic system services and for performing hardware dependent tasks.

In some embodiments, the content storage module933(e.g., the content storage114inFIG.1) stores multimedia content. To that end, the content storage module933includes a set of instructions935aand heuristics and metadata935b.

In some embodiments, the encoder940(e.g., the encoder112inFIG.1) is configured to encoder the content from the content storage module933. To that end, the encoder940includes a set of instructions941aand heuristics and metadata941b.

In some embodiments, the intermediate packager950(e.g., the intermediate packager116inFIG.1) is configured to package the encoded content from the encoder940into intermediate units951(e.g., the intermediate units117inFIG.1) and generate the corresponding metadata952(e.g., the metadata118inFIG.1). To that end, the intermediate packager950includes a set of instructions953aand heuristics and metadata953b.

In some embodiments, the JIT packager960(e.g., the JIT packager120inFIG.1) is configured to package the intermediate units951and the corresponding metadata952to formats for streaming to the client. To that end, the JIT packager960includes a set of instructions961aand heuristics and metadata961b.

Although the content storage module933, the encoder940, the intermediate packager950, and the JIT packager960are illustrated as residing on a single computing device900, it should be understood that in other embodiments, any combination of the content storage module933, the encoder940, the intermediate packager950, and the JIT packager960can reside in separate computing devices in various embodiments. For example, in some embodiments, each of the content storage module933, the encoder940, the intermediate packager950, and the JIT packager960resides on a separate computing device.

FIG.10is a block diagram of a computing device1000for using the published key picture information for content consumption in accordance with some embodiments. In some embodiments, the computing device1000corresponds to the client device140inFIG.1and performs one or more of the functionalities described above with respect to the client device140. While certain specific features are illustrated, those skilled in the art will appreciate from the present disclosure that various other features have not been illustrated for the sake of brevity, and so as not to obscure more pertinent aspects of the embodiments disclosed herein. To that end, as a non-limiting example, in some embodiments the computing device1000includes one or more processing units (CPUs)1002(e.g., processors), one or more input/output interfaces1003(e.g., input devices, sensors, a network interface, a display, etc.), a memory1006, a programming interface1008, and one or more communication buses1004for interconnecting these and various other components.

In some embodiments, the communication buses1004include circuitry that interconnects and controls communications between system components. The memory1006includes high-speed random access memory, such as DRAM, SRAM, DDR RAM or other random access solid state memory devices; and, in some embodiments, include non-volatile memory, such as one or more magnetic disk storage devices, optical disk storage devices, flash memory devices, or other non-volatile solid state storage devices. The memory1006optionally includes one or more storage devices remotely located from the CPU(s)1002. The memory1006comprises a non-transitory computer readable storage medium. Moreover, in some embodiments, the memory1006or the non-transitory computer readable storage medium of the memory1006stores the following programs, modules and data structures, or a subset thereof including an optional operating system1030, a targeted content store1033, a playlist controller1040, a player1050, and a splicer1060. In some embodiments, one or more instructions are included in a combination of logic and non-transitory memory. The operating system1030includes procedures for handling various basic system services and for performing hardware dependent tasks.

In some embodiments, the targeted content store1033(e.g., the targeted content store148inFIG.1) stores targeted content (e.g., advertisements). To that end, the targeted content store1033includes a set of instructions1035aand heuristics and metadata1035b.

In some embodiments, the playlist controller1040(e.g., the playlist controller142inFIG.1) is configured to obtain playlist and/or manifest from the server. To that end, the playlist controller1040includes a set of instructions1041aand heuristics and metadata1041b.

In some embodiments, the player1050(e.g., the player144inFIG.1) is configured to obtain the content from the server according to the playlist and/or manifest and play the content. To that end, the player1050includes a set of instructions1051aand heuristics and metadata1051b.

In some embodiments, the splicer1060(e.g., the splicer146inFIG.1) is configured to substitute the content with targeted content from the targeted content store1033and/or insert the targeted content into the content. To that end, the splicer1060includes a set of instructions1061aand heuristics and metadata1061b.

Although the targeted content store1033, the playlist controller1040, the player1050, and the splicer1060are illustrated as residing on a single computing device1000, it should be understood that in other embodiments, any combination of the playlist controller1040, the player1050, and the splicer1060can reside in separate computing devices in various embodiments. For example, in some embodiments, each of the playlist controller1040, the player1050, and the splicer1060resides on a separate computing device.

It will also be understood that, although the terms “first,” “second,” etc. may be used herein to describe various elements, these elements should not be limited by these terms. These terms are only used to distinguish one element from another. For example, a first device could be termed a second device, and, similarly, a second device could be termed a first device, which changing the meaning of the description, so long as all occurrences of the “first device” are renamed consistently and all occurrences of the “second device” are renamed consistently. The first device and the second device are both devices, but they are not the same device.