Patent ID: 12192546

DETAILED DESCRIPTION

The accompanying drawings, which form a part hereof, show examples of the disclosure. It is to be understood that the examples shown in the drawings and/or discussed herein are non-exclusive and that there are other examples of how the disclosure may be practiced.

FIG.1shows an example communication network100in which features described herein may be implemented. The communication network100may comprise one or more information distribution networks of any type, such as, without limitation, a telephone network, a wireless network (e.g., an LTE network, a 5G network, a WiFi IEEE 802.11 network, a WiMAX network, a satellite network, and/or any other network for wireless communication), an optical fiber network, a coaxial cable network, and/or a hybrid fiber/coax distribution network. The communication network100may use a series of interconnected communication links101(e.g., coaxial cables, optical fibers, wireless links, etc.) to connect multiple premises102(e.g., businesses, homes, consumer dwellings, train stations, airports, etc.) to a local office103(e.g., a headend). The local office103may send downstream information signals and receive upstream information signals via the communication links101. Each of the premises102may comprise devices, described below, to receive, send, and/or otherwise process those signals and information contained therein.

The communication links101may originate from the local office103and may comprise components not illustrated, such as splitters, filters, amplifiers, etc., to help convey signals clearly. The communication links101may be coupled to one or more wireless access points127configured to communicate with one or more mobile devices125via one or more wireless networks. The mobile devices125may comprise smart phones, tablets or laptop computers with wireless transceivers, tablets or laptop computers communicatively coupled to other devices with wireless transceivers, and/or any other type of device configured to communicate via a wireless network.

The local office103may comprise an interface104, such as a termination system (TS). The interface104may comprise a cable modem termination system (CMTS) and/or other computing device(s) configured to send information downstream to, and to receive information upstream from, devices communicating with the local office103via the communications links101. The interface104may be configured to manage communications among those devices, to manage communications between those devices and backend devices such as servers105-107and/or122, and/or to manage communications between those devices and one or more external networks109. The local office103may comprise one or more network interfaces108that comprise circuitry needed to communicate via the external networks109. The external networks109may comprise networks of Internet devices, telephone networks, wireless networks, wireless networks, fiber optic networks, and/or any other desired network. The local office103may also or alternatively communicate with the mobile devices125via the interface108and one or more of the external networks109, e.g., via one or more of the wireless access points127.

The push notification server105may be configured to generate push notifications to deliver information to devices in the premises102and/or to the mobile devices125. The content server106may be configured to provide content to devices in the premises102and/or to the mobile devices125. This content may comprise, for example, video, audio, text, web pages, images, files, etc. The content server106(or, alternatively, an authentication server) may comprise software to validate user identities and entitlements, to locate and retrieve requested content, and/or to initiate delivery (e.g., streaming) of the content. The application server107may be configured to offer any desired service. For example, an application server may be responsible for collecting, and generating a download of, information for electronic program guide listings. Another application server may be responsible for monitoring user viewing habits and collecting information from that monitoring for use in selecting advertisements. Yet another application server may be responsible for formatting and inserting advertisements in a video stream being transmitted to devices in the premises102and/or to the mobile devices125. The local office103may comprise additional servers, such as a transcoding server122(described below), additional push, content, and/or application servers, and/or other types of servers. Although shown separately, the push server105, the content server106, the application server107, the transcoding server122, and/or other server(s) may be combined. The servers105,106,107, and122, and/or other servers or other types of devices, may be computing devices and may comprise memory storing data and/or storing computer executable instructions that, when executed by one or more processors, cause the server(s) or other devices to perform steps described herein.

An example premises102amay comprise an interface120. The interface120may comprise circuitry used to communicate via the communication links101. The interface120may comprise a modem110, which may comprise transmitters and receivers used to communicate via the communication links101with the local office103. The modem110may comprise, for example, a coaxial cable modem (for coaxial cable lines of the communication links101), a fiber interface node (for fiber optic lines of the communication links101), twisted-pair telephone modem, a wireless transceiver, and/or any other desired modem device. One modem is shown inFIG.1, but a plurality of modems operating in parallel may be implemented within the interface120. The interface120may comprise a gateway111. The modem110may be connected to, or be a part of, the gateway111. The gateway111may be a computing device that communicates with the modem(s)110to allow one or more other devices in the premises102ato communicate with the local office103and/or with other devices beyond the local office103(e.g., via the local office103and the external network(s)109). The gateway111may comprise a set-top box (STB), digital video recorder (DVR), a digital transport adapter (DTA), a computer server, and/or any other desired computing device.

The gateway111may also comprise one or more local network interfaces to communicate, via one or more local networks, with devices in the premises102a. Such devices may comprise, e.g., display devices112(e.g., televisions), STBs or DVRs113, personal computers114, laptop computers115, wireless devices116(e.g., wireless routers, wireless laptops, notebooks, tablets and netbooks, cordless phones (e.g., Digital Enhanced Cordless Telephone—DECT phones), mobile phones, mobile televisions, personal digital assistants (PDA)), landline phones117(e.g. Voice over Internet Protocol—VoIP phones), and any other desired devices. Example types of local networks comprise Multimedia Over Coax Alliance (MoCA) networks, Ethernet networks, networks communicating via Universal Serial Bus (USB) interfaces, wireless networks (e.g., IEEE 802.11, IEEE 802.15, Bluetooth), networks communicating via in-premises power lines, and others. The lines connecting the interface120with the other devices in the premises102amay represent wired or wireless connections, as may be appropriate for the type of local network used. One or more of the devices at the premises102amay be configured to provide wireless communications channels (e.g., IEEE 802.11 channels) to communicate with one or more of the mobile devices125, which may be on- or off-premises.

The mobile devices125, one or more of the devices in the premises102a, and/or other devices may receive, store, output, and/or otherwise use content items. A content item may comprise a video, a game, one or more images, software, audio, text, webpage(s), and/or other content.

FIG.2shows hardware elements of a computing device200that may be used to implement any of the computing devices shown inFIGS.1and3(e.g., the mobile devices125, any of the devices shown in the premises102a, any of the devices shown in the local office103, any of the wireless access points127, any devices with the external network109, any of elements301-303,306,310,315, and318, etc.) and any other computing devices discussed herein. The computing device200may comprise one or more processors201, which may execute instructions of a computer program to perform any of the functions described herein. The instructions may be stored in a read-only memory (ROM)202, random access memory (RAM)203, removable media204(e.g., a USB drive, a compact disk (CD), a digital versatile disk (DVD)), and/or in any other type of computer-readable medium or memory. Instructions may also be stored in an attached (or internal) hard drive205or other types of storage media. The computing device200may comprise one or more output devices, such as a display device206(e.g., an external television and/or other external or internal display device) and a speaker214, and may comprise one or more output device controllers207, such as a video processor. One or more user input devices208may comprise a remote control, a keyboard, a mouse, a touch screen (which may be integrated with the display device206), microphone, etc. The computing device200may also comprise one or more network interfaces, such as a network input/output (I/O) interface210(e.g., a network card) to communicate with an external network209. The network I/O interface210may be a wired interface (e.g., electrical, RF (via coax), optical (via fiber)), a wireless interface, or a combination of the two. The network I/O interface210may comprise a modem configured to communicate via the external network209. The external network209may comprise the communication links101discussed above, the external network109, an in-home network, a network provider's wireless, coaxial, fiber, or hybrid fiber/coaxial distribution system (e.g., a DOCSIS network), or any other desired network. The computing device200may comprise a location-detecting device, such as a global positioning system (GPS) microprocessor211, which may be configured to receive and process global positioning signals and determine, with possible assistance from an external server and antenna, a geographic position of the computing device200.

AlthoughFIG.2shows an example hardware configuration, one or more of the elements of the computing device200may be implemented as software or a combination of hardware and software. Modifications may be made to add, remove, combine, divide, etc. components of the computing device200. Additionally, the elements shown inFIG.2may be implemented using basic computing devices and components that have been configured to perform operations such as are described herein. For example, a memory of the computing device200may store computer-executable instructions that, when executed by the processor201and/or one or more other processors of the computing device200, cause the computing device200to perform one, some, or all of the operations described herein. Such memory and processor(s) may also or alternatively be implemented through one or more Integrated Circuits (ICs). An IC may be, for example, a microprocessor that accesses programming instructions or other data stored in a ROM and/or hardwired into the IC. For example, an IC may comprise an Application Specific Integrated Circuit (ASIC) having gates and/or other logic dedicated to the calculations and other operations described herein. An IC may perform some operations based on execution of programming instructions read from ROM or RAM, with other operations hardwired into gates or other logic. Further, an IC may be configured to output image data to a display buffer and/or perform any other functionality in support of any of the features described herein.

FIG.3shows an example block diagram of how content and manifests may be provided to one or more premises such as the premises102a. In this example, content may be stored or otherwise accessible via a content library, referred to herein by way of example as a mezzanine content library301. The content may comprise one or more content items313, which may each be any type of content. For example, a content item may comprise multimedia content and/or may comprise video, audio, text, graphics, animation, subtitles, metadata, and/or other information. Examples of the content items313include a video program, movie, television show, advertisement, livestream, broadcast sporting event, and the like. A content item313may be of any length, e.g., about a half hour, about an hour, any length between those two lengths, or any length less than or greater than either of those two lengths.

The mezzanine content library301may organize and store one or more of the content items313via one or more storage devices that may be physically in the same location or that may be distributed across multiple locations. For example, the content items313of the mezzanine content library301may be stored in devices of one or more of the external networks109, in devices of the local office (for example, in the content server106), and/or in devices of the premises102a(for example, in the gateway111). The mezzanine content library301may also include or otherwise be in communication with one or more computing devices (for example, the content server106, the application server107, and/or the gateway111), referred to herein as a content retriever302, that handles various aspects of content item retrieval and/or delivery. Any of the blocks301-303,306,310,315, and318may be partially or fully implemented by any one or more devices, such as one or more data storage devices and/or one or more computing devices (e.g., one or more computing devices such as shown inFIG.2). For example, any of the blocks301-303,306,315, and/or318may be implemented, in whole or in part, by one or more of the interface104, the push server105, the content server106, the application server107, the transcoding server122, the network interface108, devices in the external network(s)109, and/or the interface120. The block310(e.g., a device, such as a client, that can receive content) may be implemented, in whole or in part, by one or more of the interface120, the display device112, the set top box/DVR113, the personal computer114, the laptop computer115, the wireless device116, the landline phone117, the mobile device(s)125, or any other computing device(s). Moreover, any of the blocks301-303,306, and/or315may be physically implemented by the same device(s) as others of the blocks301-303,306,315, and/or318, or subdivided into being implemented by multiple devices. For example, the content retriever302and the transcoder303may both be partially or fully implemented by the application server107(which itself may be physically made up of one or more computing devices). Or, the transcoder303and the packager306may be partially or fully implemented by the transcoding server122, the application server107, and/or the content server106(each of which may be physically made up of one or more computing devices).

In operation, the content retriever302may retrieve one or more of the content items313from the mezzanine content library301, and pass those retrieved content items313to the transcoder303. Each content item313, as stored in the mezzanine content library301, may or may not already be divided into a sequential time series of a plurality of segments. If not, or if a different division is desired, then the retrieved content items313may be divided into a sequential time series of a plurality of segments by the content retriever302and/or by the transcoder303. A segment may be any portion of a content item. The transcoder303may then transcode each of the segments, of a content item from the mezzanine content library301, into one or more transcoded versions (referred to herein by way of example as one or more representations314) of each of the segments. The transcoder303may be configured to transcode each of the segments into any one or more of a plurality of formats, as desired. For example, the transcoder303may be able to transcode a segment into multiple different transcoded bitrates, multiple different levels of content quality (e.g., video quality), and/or multiple different data formats and/or streaming protocols (for example, AVC, HEVC, AV1, MPEG-DASH, etc.), one variation for each of the representations314. For example, a first representation314(Representation 1) of a particular transcoded segment of a content item may have a first bitrate (for example, a maximum bitrate for that segment), a first content quality, a first data format, and/or via a first streaming protocol. A second representation314(Representation 2) of that same transcoded segment of the same content item may have a different second bitrate (for example, a maximum bitrate for that segment), a different second content quality, a different second data format, and/or a different second streaming protocol. Where a plurality of the representations314are generated for a given segment, each of the representations314may have at least one characteristic (e.g., bitrate of the transcoded segment) different from others of the representations314for that segment. In the example of Representation 1 and Representation 2 discussed above, one representation for a given segment may have a higher transcoded bitrate (for example, a higher maximum transcoded bitrate) than the other representation for that same segment.

Some of all of the transcoded segments, for a given one of the representations314for a given one of the content items313, may be of constant length based on playback time of that segment, relative to other segments. For example, if the content item313is a one-hour video program, then for each of the representations314associated with that content item314, the video program may be divided and then transcoded (or transcoded and then divided) into 1,800 two-second segments (totaling 3,600 seconds of video per representation). Moreover, two or more segments within a given representation314may be of different time lengths. For example, one of the segments may be two seconds in length and another of the segments may be one second in length. Also or alternatively, two or more segments for a given representation314may have constant or varying lengths based on quantities of data in the segments. For example, one of the segments may contain 1 MB (megabyte) of content data, and another of the segments may contain 1.5 MB of content data. Each segment may be of any fixed or variable time length, e.g., on the order of milliseconds, seconds, or minutes. In further examples, each segment may be about 2 seconds in length, or each segment may be about 1 second in length, or each segment may be less than 5 seconds in length. Segments may be of any data length, e.g., on the order of one or more KB (kilobytes), one or more MB, or one or more GB (gigabytes) of content data.

The transcoder303may output each of the representations314in the form of one or more packets305that may each comprise a descriptor305a(such as a packet header) and a payload305b. Each payload305bmay include at least some of the data representing the transcoded segment(s). The transcoder303may then store each of the packets305in a transcoded content library315. The transcoded content library315may be physically different from or the same as the mezzanine content library301and/or a manifest storage318, and may be in one or more different geographical locations or the same geographical location(s) as the mezzanine content library301and/or the manifest storage318.

The transcoder303may also determine, for each transcoded segment of each of the representation314, one or more characteristics of the transcoded segment. Examples of the characteristic(s) that may be determined include bitrate (for example, one or more maximum transcoded bitrates or other transcoded bitrates for the segment), content quality (for example, picture resolution), segment length (time length and/or data size), and/or complexity of the transcoded content item or portion thereof. The transcoder303may determine the characteristic(s) for a given transcoded segment based on indirectly or directly measuring the characteristics, calculating the characteristics, estimating the characteristics, or the like, based on the content item313associated with the transcoded segment and/or based on the transcoded segment itself. For example, during the transcoding process, following the allocation of bits to each picture in a given segment, the transcoder303may calculate the sum of all bits allocated per picture and include the resulting summated value into one or more descriptors305afor the segment so that this information could be parsed further downstream, such as by the packager306. Also, the transcoder303may determine segment length and/or resolution of a transcoded segment prior to or during transcoding, by interrogating a transcoder configuration (e.g., XML, JSON, etc.) that may be used to tell the transcoder303what the segment length, resolution, etc. are supposed to be. The one or more characteristics may be determined on a segment-by-segment basis for an entire transcoded content item313or for any one or more portions of the transcoded content item313. For example, the transcoder303may determine, for each representation314for a given content item313, the bitrate (for example, maximum bitrate) for each transcoded segment or for each group of n>1 transcoded segments. Each descriptor305amay comprise one or more indications of the one or more characteristics determined by the transcoder303. Thus, after the packet(s)305are received by the packager306for a given transcoded segment, the packager306may parse the associated descriptor(s)305aand read the indication(s) of the one or more characteristics for that segment.

To initiate or continue delivery of content to the device310, the device310may send the request312, which may indicate or otherwise be associated with one or more requested segments associated with a desired content item313. Based on the request312, the content retriever302may retrieve the data packet(s)305containing the desired one or more segments, and possibly also containing one or more further segments, such as one or more segments later in the content item313.

To determine the one or more characteristics associated with the transcoded content item305, the transcoder303may transcode one or more segments and determine the one or more characteristics based on those transcoded segments. The transcoder303may additionally or alternatively determine one or more of the characteristics for one or more segments prior to transcoding (or prior to fully transcoding) the one or more segments. For example, the transcoder303may determine the one or more characteristics based on a pre-transcoded version of the segment, and/or based on metadata stored at, for example, the content library301. In the example discussed above, the transcoder303may determine the transcoded bitrates for each representation314of a given segment after actually transcoding the segment into each of the representations314, or prior to any transcoding of the segment, or after transcoding the segment into a subset of (for example, only a single one of) the representations314, and deriving the characteristics for other representations for the segment based on the characteristics of the subset (e.g., one) of the representations.

As discussed above, the representations314may be stored in the transcoded content library315, such as in the form of the one or more packets305for each transcoded segment. One or more of those packets305may be subsequently retrieved for delivery to a device such as to the device310. The device310may be any device, such as any computing device, for example any of the devices110-117and125inFIG.1. To initiate delivery of a content item (or to continue delivery of further segments of a content item that has already begun delivery) to the device310, the device310(or another device associated with the device310) may send a request312associated with (e.g., identifying) the content item, one or more transcoded segments of the content item, and/or one or more packets305associated with the content item. The request312may be received by, for example, the content retriever302, or by any other device. In response to or otherwise after the request312, the content retriever302may retrieve, from the transcoded content library315, the packets305for selected transcoded segments, and pass those data packets305to the packager306. The packager306may be responsible for parsing and reading the descriptors305aof the passed packets305, and for packaging and delivering (such as via one or more transport streams such as MPEG transport streams) the content in the payloads305bto the device310in the form of fragments320each containing one or more of the segments314.

The packager306may also receive or otherwise determine information319, which may include, for example, data indicating the current and/or expected state of the network (e.g., network congestion, available network bandwidth, a fragment length preference, time of day, day of the week, and/or data provided by device310such as the type or speed of network connection being used by device310to receive fragments320). The information319may be determined based on one or more real-time or delayed measurements (for example, a measurement relating to video quality, such as a measurement of or relating to network congestion, of or relating to available network bandwidth, and/or of or relating to actual received video by the device310or by an intervening device between the packager306and the device310), where the measurement(s) may be performed anywhere in the network, such as at the packager306side (e.g., measurement M1) or at the device310side (e.g., measurement M2). Where one or more of the measurements are at the device310side, the device310may perform the measurements (e.g., perform measurement M2) and directly or indirectly report the information319to the packager306. The packager306may use the information319to determine how to package one or more of the segments314into fragments320. For example, the packager306may determine the length of each fragment320(how many segments are contained in a fragment) based on the information319. For example, if the information319indicates a higher level of network congestion and/or less available network bandwidth (which may be a result of measurement, as discussed above), packager306(and/or another element in the network, such as the content retriever302) may determine that each fragment320should contain fewer segments315. Likewise, if the information319indicates a lower level of network congestion and/or more available network bandwidth, packager306(or another element in the network) may determine that each fragment320should contain a relatively larger number of segments315.

The packager306(or another device) may also be responsible for updating a manifest308based on the descriptors305aof the packets305retrieved from the transcoded content library315. For example, based on each descriptor305a, the packager306may generate data indicating one or more of the characteristics identified by the descriptor305. The data, referred to herein by way of example as a manifest308, may be created and delivered to a device (e.g., the device310) without being stored, or may be stored in the manifest storage318. The manifest308may be made available to a device (e.g., the device310), and may be any type and format of data (e.g., any data set in any desired format), such as one or more data files, tables, lists, collections, etc. The manifest storage318may be physically different from or the same as the mezzanine content library301and/or the transcoded content library315, and may be in one or more different geographical locations or the same geographical location(s) as the mezzanine content library301and/or the transcoded content library315. While the fragments320are being delivered, the manifest308may be stored and/or provided separately from the fragments320, or the manifest308may be packaged with the fragments320. For example, one or more of the transport streams may comprise the manifest308. Or, the manifest308may be stored in a location for retrieval by the device310. The manifest308may be updated309by the packager306, and may be read311by the device310. For the device310to obtain the manifest308, the manifest308may be pushed to the device310or it may be retrieved (for example, requested, or pulled) by the device310. If the manifest308is pushed to the device310, the manifest308may be updated (e.g., modified in place or entirely replaced), and pushed on a periodic basis or on an as-needed basis. For example, the updated manifest308may be pushed every segment, or every nth segment (n>1). Or, the updated/replaced manifest308may be pushed on a regular or irregular time period basis, such as every half second, every second, every nth second (n>1), or at variable time periods. The manifest308may have a particular published or otherwise known location in the manifest storage318, which may be identified by a location identifier, such as a universal resource locator (URL), that the device310can access to retrieve the manifest308. If the manifest308is retrieved from the manifest storage318by the device310, the manifest308may be electronically published, and the published manifest308may be updated (for example, modified, or entirely replaced and re-published) in the manifest storage18periodically or as needed, such as prior to each time the manifest308is published, sent, or otherwise made available for access by the device310. For example, the manifest308may be updated or replaced for every segment, or for every nth segment. Or, the manifest308may be updated or replaced on a regular or irregular time period basis, such as every half second, every second, every nth second (n>1), or at variable time periods.

The manifest308may comprise or indicate (e.g., may comprise an index of) information associated with each of a plurality of representations of a content item. The information may indicate, or otherwise be based on, the one or more characteristics discussed above. For example, the manifest308may comprise bandwidth information and/or other information for a first representation (e.g., Representation 1) of a given segment and/or fragment containing the segment (for example, an indication of maximum bitrate, complexity, etc. of the Representation 1 fragment containing the segment), and bandwidth information and/or other information for a second representation (e.g., Representation 2) of that given segment and/or fragment containing the segment (for example, an indication of maximum bandwidth, complexity, etc. of Representation 2 of the segment and/or fragment). Various examples of information that may be in the manifest308are shown inFIGS.6-8. To obtain the bandwidth information for a given segment and/or fragment, the packager306may translate the bitrate information for the segment and/or fragment (indicated by, for example, the descriptor305a), into bandwidth information that is to be included in the manifest308.

FIG.4Ashows an example of how selected transcoded segments of representations314may be provided as one or more fragments320to the device310, such as by the process and system described in connection withFIG.3. InFIG.4A, a content item313may have been transcoded and divided into a plurality of segments314for each of three representations (Representation 1, Representation 2, and Representation 3). The number of representations created and/or available for a given content item may be determined based on, for example, the capabilities of the transcoder303, the capabilities of the device310(which may be communicated to the content provider such as via the request312), the throughput (for example, measured throughput) of the communication link(s)101, one or more characteristics of the content item to be transcoded (such as the quality, length, and/or bitrate of the content item to be transcoded), the expected characteristics of the content item after transcoding (such as the quality and/or bitrate of the content item after transcoding), an indication of the number of representations or a range of desired characteristics (e.g., qualities and/or transcoded bitrates of the representations) (which may be communicated to the content provider such as via the request312), and/or any other factors.

InFIG.4A, for each of the segments314, Representation 1 may have a first bitrate (and/or quality) for the transcoded segment, Representation 2 may have a second bitrate (and/or quality) for the transcoded same segment lower than the first bitrate, and Representation 3 may have a third bitrate (and/or quality) for the transcoded same segment lower than the second bitrate. For example, each of the transcoded segments314a1,314a2, and314a3, while representing the same segment, may have different characteristics such as different transcoded bitrates and/or different qualities. For example, the transcoded segment314a1may have a first transcoded bitrate and/or a first transcoded quality, the transcoded segment314a2may have a second transcoded bitrate and/or a second transcoded quality lower than the first transcoded bitrate and/or the first transcoded quality, and the transcoded segment314a3may have a third transcoded bitrate and/or a third transcoded quality lower than the second transcoded bitrate and/or the second transcoded quality. These differences in characteristics between representations of the same segment may be found amongst any of the triplets of segment representations, such as amongst segments314b1,314b2, and314b3, amongst314c1,314c2, and314c3, etc. For example, the transcoded segment314b1may have a fourth bitrate and/or a fourth quality, the transcoded segment314b2may have a fifth bitrate and/or a fifth quality lower than the fourth bitrate and/or the fourth quality, and the transcoded segment314b3may have a sixth bitrate and/or a sixth quality lower than the fifth bitrate and/or the fifth quality.

In the example ofFIG.4A, the segments314are each two seconds in length, and the device310may, based on the manifest308, select Representation 1 to be used for the segments at time T=0 and 6, select Representation 2 to be used for the segments at time T=2, 4, and 8, and select Representation 3 to be used for the segment at time T=10. Each of these selected transcoded segments is indicated inFIG.4Awith a thick border. The device310may send an indication of which representations and/or segments are to be used via one or more of the requests312. Based on the requests312, the content retriever302may retrieve the appropriate transcoded segments from the transcoded content library315, and pass these transcoded segments (such as in the form of packets305) to the packager306. The transcoded segments retrieved in this example are transcoded segments314a1,314b2,314c2,314d1,314e2, and314f3. The packager306may use information from the selected transcoded segments314(such as from the descriptors305aof the packets305) and/or from the information319to update the manifest308and/or to package the selected transcoded segments314, and may package the selected transcoded segment(s)314into one of the fragments320as discussed previously, which would then be delivered to the device310. In the example shown inFIG.4A, a fragment320amay include segment314a1, a fragment320bmay include segments314b2and314c2, a fragment320cmay include fragment314d1, a segment320dmay include fragment314e2, and a fragment320emay include segment314f3. Each fragment320may have a length in time equal to the sum of the length of the fragments it contains. For example, where each segment is two seconds in length, a fragment320containing one segment may also be two seconds in length, a fragment320containing two segments may be four seconds in length, a fragment320containing three segments may be six seconds in length, and so on.

FIG.4Bshows another example of selected transcoded segments314being provided as one or more fragments320to the device310, such as by the process and system described in connection withFIG.3. In this example, Representations 1-3 may be the same Representations 1-3 for the same content item as inFIG.4A, except occurring later in the content item (here, time starts at T=12 seconds whereasFIG.4Aexplicitly shows segments up through T=10 seconds). In this example, segment314g3is selected and sent to the device310in fragment320e, segments314h2and314i2are selected and sent to the device310as part of fragment320f, and segments314j2and314k1are selected and sent to the device310as part of fragment320g. In both the examples ofFIGS.4A and4B, the length of each fragment320may depend on a determination, such by the packager306, and that determination may be based on information such as the requests312and/or the information319. Moreover, the length of each fragment320may sometimes or always be static over time (e.g., always one segment, or always two segments), and/or may sometimes or always be dynamically changing based on the requests312and/or the information319, which may also be dynamically changing over time. For example, as the values measured by measurements M1 and/or M2 change over time, the information319may change in response, and the packager306may likewise use this changed information319to determine a different length for the next fragment.

In the examples ofFIGS.4A and4B, all of the segments314for all of the representations may have been previously created via transcoding by the transcoder303, even though only a subset of those segments may be actually sent to the device310as part of fragments320. The segments that are actually sent to the device310may depend on, for example, which segments and/or representations are requested by the device310via one or more of the requests312. The system may alternatively operate in an on-demand manner, such that the transcoder303may provide just-in-time transcoding (based on the requests312) to generate only those segments for those representations needed for delivery to the device310.

FIG.5is a flow diagram showing example communications related to systems and methods ofFIG.3. The communication flow ofFIG.5may be used to implement the examples ofFIGS.4A and4B. After the appropriate content items313have been segmented and transcoded by transcoder303and stored in the transcoded content library315, the device310may send a request312a(for example, one of the requests312ofFIG.3), which may be a first request from the device310for a communications session, which may identify a certain content item (and/or one or more segments thereof) desired to be received by the device310, and/or which may be used to authenticate the device310. The request312amay further provide information about the device310, such as an indication of capabilities of the device310(e.g., maximum resolution, maximum bandwidth, user preferences, and/or the like). The request312amay be received by, for example, the packager306and/or the content retriever302. In response to the request312a, the packager306may provide an initial manifest308a(for example, one of the manifests308ofFIG.3) to the device310(e.g., by sending the manifest308adirectly to the device310or by publishing the manifest308afor retrieval by the device310). The indication of the initial manifest308amay comprise, for example, an indication of a location (for example, a URL) in the manifest storage318from which the initial manifest308amay be retrieved. Additionally or alternatively, the initial manifest308aitself may be sent to the device310from the manifest storage318. In response to the indication of the initial manifest308a, the device310may retrieve the initial manifest308afrom the indicated location. As explained previously, one or more of the manifests308may be updated (for example, modified or entirely replaced) over time in any of a number of ways. The initial manifest308a, for any given content item (e.g., a video-on-demand (VOD) asset or Linear stream), may exist in a Common Interchange Format (CIF), such as MPEG-DASH, before the device310begins selection or playback of the content item. As will be described below, during a playback session for the content item, the information in packet descriptors305a(FIG.3) may be interrogated (e.g., periodically) by packager306to update the manifest308as the device310continues to request additional segments of the video/audio over time. Therefore, updated manifests (e.g., manifest308b, manifest308c, etc.) may be made available over time to the device310so that the device310can be better informed as to what the actual bandwidth requirement to stream any representation in the relevant manifest308might be.

The initial manifest308amay indicate one or more representations of the requested content item. As explained previously, each of the representations may have one or more different characteristics for a given one or more segments, such as different segment transcoded bitrates, different transcoded segment qualities of the represented content, and/or different transcoded segment lengths. Continuing the example in the previous paragraph, the initial manifest308amay indicate a plurality of different representations for the first segment in the content item identified by the request312a, and/or for one or more additional segments such as for one or more immediately subsequent segments in the content item.

An example of at least a portion of the initial manifest308ais shown below, in which the initial manifest308acomprises information, for each of Representation 1 and Representation 2, for the first transcoded segment and/or fragment of a content item:

<Period id=″1″ start=″PTOHOM0.000S″><AdaptationSet id=″2″ contentType=″video″ mimeType=″video/mp4″><SegmentTemplate initialization=″manifest/init.mp4″media=″manifest/init.mp4″ timescale=″90000″ startNumber=″1″presentationTimeOffset=″2790000″><SegmentTimeline><S t=″2790000″ d=″180180″ r=″0″/></SegmentTimeline></SegmentTemplate><!-- QOE signals to the player additional, xcoder measurementof video quality to help it judge tradeoffs between quality and bit rate -- ><Representation id=″video00″ bandwidth=″800000″ QOE=″53″codecs=″avc1.4d401f″ width=″320″ height=″180″/><Representation id=″video01″ bandwidth=″1000000″ QOE=″82″codecs=″avc1.4d401f″ width=″640″ height=″360″/></AdaptationSet></Period>

In this example, the content item has a plurality of transcoded segments that are each two seconds in length (for example, segments that begin at intervals of every two seconds). However, the content item may have transcoded segments of any fixed or variable length. To simplify the drawing, Representation 3 is not explicitly shown inFIG.6(or inFIGS.7and8), however it will be understood that Representation 3 (and additional other representations) may also be indicated in any of the manifests308a-c. Thus, initial manifest308amay comprise, for example, information for each of Representation 1, Representation 2, Representation 3, and/or any other representations of the content item as desired. The initial manifest308ain this example indicates that, for Representation 1, the fragment length is one segment, and the associated transcoded segment for that fragment (the transcoded segment314a1) would have a maximum bandwidth of 1 megabit per second (Mbps). The manifest308afurther indicates that, for Representation 2, the fragment length is also one segment, and the associated transcoded segment for that fragment (the transcoded segment314a2) would have a maximum bandwidth of 800 kilobits per second (Kbps). The maximum bandwidths here may be based on the determined characteristic(s) discussed previously. In other variations, the initial manifest308amay additionally or alternatively indicate other information for a transcoded segment, such as another type of bandwidth (for example, average bandwidth for the transcoded segment and/or fragment containing the segment), content quality (for example, content resolution, bits-per-pixel for the transcoded segment and/or fragment, and/or value from a video quality measurement such as structural similarity (SSIM) for the transcoded segment and/or fragment), a segment length, a fragment length, and/or another characteristic for each of the transcoded segments314a1,314a2and/or fragment(s) containing those segments. The manifest308amay further comprise the same indications for one or more further transcoded segments and/or fragments thereof, such as for the transcoded segment314a2.

The device310may select, based on the manifest308a, based on a measured network performance, and/or based on any other factors, which of the representations (and/or segment(s) thereof) are desired. According to the flow ofFIG.5, the device310may then send a request312bindicating the selected representation. Because the device310has already been authenticated (using, for example, the request312a), the request may be sent to the content retriever302rather than the packager306. However, the request312bmay be sent to, and/or received by, any one or more of the blocks ofFIG.3. The request312bmay indicate one of the representations (and/or segments thereof). For example, assume that the device310determines (e.g., measures) that it has up to 2 Mbps of bandwidth available at the moment. In that case, the device310may compare the available bandwidth (2 Mbps) with the information in the manifest308afor one or more of the representations, and determine based on the comparison that the available bandwidth is higher than the maximum bandwidth (1 Mbps) of the relevant segment (the transcoded segment314a1) for Representation 1. Thus, the device310may generate the request312bto indicate Representation 1, corresponding to the transcoded segment314a1, which corresponds to the original content segment314a. Put another way, the device310may select and request the representation indicated by the manifest308aas having the segment with the highest maximum bandwidth that does not exceed the bandwidth available to the device310. While this example uses a characteristic of maximum bandwidth for each transcoded segment, the same process may be performed for other types of characteristics. For example, if the manifest308ahad indicated the quality of the transcoded segment for each representation, the device310may compare the indicated quality for each representation with the available bandwidth.

If, instead, a manifest was used that was not updated regularly such as on a segment-by-segment basis (for example, a static manifest for the entire content representing the highest bandwidth across all segments of Representation 1), the manifest might have indicated that the maximum bandwidth of Representation 1 is 3.4 Mbps and the maximum bandwidth of Representation 2 is 2.0 Mbps (which are respectively the maximum bandwidth of any segment shown in these representations). In such a case, the indicated maximum bandwidth of Representation 1 would have been higher than the available bandwidth of 2 Mbps. Thus, if the manifest were not updated such as in the manner described herein, the device310might have chosen Representation 2 for any of the segments, a lower quality representation than Representation 1. This may have resulted in a less desirable selection for at least some segments of the content as compared with using a manifest that is dynamically updated, for example, on a segment-by-segment basis. For example, the examples of the dynamic manifest308described herein may allow the device310to make better choices, as exemplified by the device310selecting the higher-quality Representation 1 for the first segment of the example content.

Based on the request312b, the content retriever302may request (for example, retrieve) one or more transcoded segments (in this example, at least segment314a1, which would have a maximum bandwidth of 1 Mbps). The content retriever302(or another element such as the packager306) may determine a quantity of the segments to retrieve in response to the request312b. The number of segments retrieved may depend on one or more factors, for example the information319and/or an explicit number of segments indicated by the request312b. In this example, it is determined that only one segment (segment314a1) is to be retrieved in response to the request312b. The retrieved transcoded segment314a1may be forwarded by the content retriever302to the packager306. The retrieved transcoded segment314a1may be in one or more packets305, which as explained above may include one or more descriptors305aindicating one or more characteristics of one or more subsequent transcoded segments, as this information may be used to generate the updated next version of the manifest308b.

The packager306may package the retrieved transcoded segment314a1into fragment320a, and send fragment320ato the device310. The packager306may further generate the next updated manifest308b, based on the characteristic(s) indicated by the descriptor(s)305ain the packet(s)305and/or the information319. The packager306may also send manifest308bto the device310and/or publish the manifest308bto the manifest storage318for retrieval by the device310. The packager306may further send directly to the device310and/or publish a link (for example, a URL) to the location of the published and updated manifest308b. The packager306may update the manifest308from the manifest308ato the manifest308bin any of a number of ways. For example, the packager306may wholly replace the manifest308awith the manifest308b, modify without deleting the manifest308ato result in the manifest308b, and/or supplement the manifest308awith the manifest308bsuch that both manifests308aand308bsimultaneously exist. Moreover, the manifest308bmay be published to the same location as, or to a location different from, the location at which the manifest308awas previously published. An example of at least a portion of the manifest308bis as follows:

<Period id=“2” start=“PT0H0M2.000S”><AdaptationSet id=“2” contentType=“video” mimeType=“video/mp4”><Role schemeIdUri=“urn:mpeg:dash:role:2011” value=“main”/><SegmentTemplate initialization=“manifest/init.mp4”media=“manifest/init.mp4” timescale=“90000” startNumber=“2”presentationTimeOffset=“2790000”><SegmentTimeline><S t=“2790000” d=“180180” r=“0”/> <!-- This timestamp is just fordemo purposes, also the presentationTimeOffset above --><S t=“459185940” d=“37536” r=“0”/></SegmentTimeline></SegmentTemplate><Representation id=“video00” bandwidth=“1000000” QOE=“55”codecs=“avc1.4d401f” width=“320” height=“180”/><Representation id=“video01” bandwidth=“1200000” QOE=“84”codecs=“avc1.4d401f” width=“640” height=“360”/></AdaptationSet></Period>

In this example, the manifest308bindicates that the Representation 1 transcoded version of the segment314b(the transcoded segment314b1) has a fragment length of one segment and has a maximum segment bandwidth of 1.2 Mbps, and that Representation 2 has a fragment length of two segments, and that the transcoded versions of the segments314band314c(the transcoded segments314b2and314c2) have a maximum bandwidth bitrate of 1.0 Mbps and 700 Kbps, respectively.

The communication flow ofFIG.5may continue as described above, but with the client using the manifest308bto determine which representation to select. The client sends the request312cbased on and/or indicating this selection, and further appropriate segments are retrieved, analyzed, and/or transcoded, and then packaged and sent to the device310. For example, the device310may select, based on the manifest308b, current measured network conditions, and/or capabilities of the device310(e.g., the device310buffer status and maximum bandwidth capabilities), Representation 2. In particular, the device310may determine that it would like to receive a transcoded version of segment314bin accordance with Representation 2device310. This would correspond to segment314b2, consistent with the example ofFIG.4A, in which the fragments320contain the transcoded segment314a1followed by the transcoded segment314b2. For example, assume that the device310determines (e.g., measures, such as via measurement M2 or elsewhere) that it has up to 1.1 Mbps of bandwidth available at the moment. In that case, the device310may compare the available bandwidth (1.1 Mbps) with the information in the manifest308bfor one or more of the representations, and determine based on the comparison that the available bandwidth is higher than the maximum bandwidth (1 Mbps) of the relevant segment (the transcoded segment314b2) for Representation 2 and lower than the maximum bandwidth (1.2 Mbps) of the transcoded segment314b1for Representation 1. Thus, the device310may generate the request312bto indicate Representation 2, corresponding to the transcoded segment314b2, which corresponds to the original content item segment314b. Put another way, the device310may again select and request the representation indicated by the manifest308bas having the segment with the highest maximum bandwidth that does not exceed the bandwidth available to the device310. In addition to segment314b2, the content retriever302may determine, based on, for example, the information319and/or the request312c, that it should retrieve two segments, and so the content retriever302also retrieves segment314c2. The packager306may take both segments314b2and314c2, and package them into a single fragment320bfor sending to the device310.

Thus, the device310may repeatedly ask for the manifest308for each segment and/or fragment of the content item, and the packager306(and/or other element) may repeatedly update the manifest308over time (such as for each segment and/or for each fragment). Moreover, the manifest308may dynamically update the segment and/or fragment bandwidths, and/or dynamically update associated fragment lengths for each representation. An example of information that may be updated in the manifest308over time is shown in the table ofFIG.9. The table also shows examples selections by the device310, indicated by thick borders, of representations based on the available bandwidth and/or other factors. For example, at segment start time 0 seconds, the device310may determine that its available bandwidth is 2 Mbps, and the current iteration of the manifest308may indicate that for this segment as transcoded, the maximum bandwidth for Representation 1 would be 1.0 Mbps, the maximum bandwidth for Representation 2 would be 800 Kbps, and the maximum bandwidth for Representation 3 would be 700 Kbps. The device310may compare its determined available bandwidth with one or more of the maximum bandwidths indicated by the current version of the manifest308, and determine that it should select Representation 1 for this particular segment, because Representation 1 has the highest maximum bandwidth that does not exceed the available bandwidth.

All of the selections by the device310need not be based on the highest maximum bandwidth not exceeding the available bandwidth. The device310may take other factors into consideration when making such a selection. For example, the device310may use its current or predicted receive buffer status as at least one factor in making a selection of a particular representation, and may take into account the fragment lengths advertised by the manifest308. For example, for the segment starting at 24 seconds, the available network bandwidth to the device310is 3.3 Mbps. Even though this is sufficient to receive Representation 1 of only 3.2 Mbps, the device310may determine that there is some reason (such as based on a determination that the client310has a nearly full receive buffer or based on a preferred fragment length shorter or longer than the fragment length indicated by the manifest308) that it should select a different representation that may have a lower bandwidth. In this example, the device310selects Representation 2 having a bandwidth of 1.8 Mbps and a fragment length of 1, rather than Representation 1 having a bandwidth of 3.2 Mbps and a fragment length of 3. The device310may make this determination in order to prevent a threshold buffer fullness from being realized, or to reserve excess current or future bandwidth for other reasons such as in anticipation of another overlapping use of the bandwidth (such as for receiving associated or even unrelated data). Thus, because each of the representations, at any given time, may have different fragment lengths advertised by the manifest308, the device310may also determine which representation to select based on the advertised fragment lengths.

The device310may purposely choose a representation having a lower bandwidth (and/or shorter fragment length) than it can actually handle (as it did for the segment starting at 24 seconds), so that, for example, the device310will have extra bandwidth to pre-fetch one or more subsequent (future) segments (and/or a portion of a subsequent segment) of the same content item. To allow the device310to have sufficient information to make an informed decision to perform such pre-fetching, the manifest308may indicate the maximum bandwidths and/or other characteristics for not only the upcoming segment, but also for one or more subsequent segments after the upcoming segment. For example, for the segment starting at 24 seconds, the manifest308may indicate, in addition to the maximum bandwidths for representations of that segment, maximum bandwidths for representations of the segments starting at 26 seconds, 28 seconds, and 30 seconds. Thus, the device310may use the manifest308to obtain a preview of what bandwidths may be needed, and to determine whether to request not only one of the representations of the segment starting at 26 seconds, but also to request one of the representations of any of those subsequent segments. Thus, the request312may additionally identify such subsequent segment(s) and/or representation(s) for those subsequent segment(s). Pre-fetching of subsequent segments may occur in an overlapping manner (for example, at least partially simultaneously) with loading of the upcoming segment. For example, the device310may request (via one or more requests312) Representation 2 for the segment starting at 24 seconds and also Representation 1 for the segment starting at 26 seconds. In response, the content retriever302may retrieve both requested segments from the transcoded content library315. In addition, the packager306may package and send both of the transcoded versions of the two segments to the device310. In sending both transcoded segments, the packager306may send both one after the other, in a partially overlapping manner in time, and/or in a fully overlapping manner in time. When both segments are sent in a partially or fully overlapping manner, the two segments may be packaged together in the same one or more packets and/or in separate packets, and may be sent at least partially simultaneously with each other to the device310.

Referencing again the example flow ofFIG.5, any of the requests312a,312b, and/or312cmay therefore request only a single representation for a single segment, or may request more than one representation for more than one segment. After receiving the manifest308b, the device310may generate, based on the manifest308b, the request312cindicating (in this example) the selection of Representation 2 for the second segment of the content item, and the request312cmay be received by the content retriever302. Based on the request312c, the content retriever302may retrieve an appropriate one or more transcoded segments in the same manner described earlier with respect to this figure.

The packager306may then send the appropriate retrieved transcoded segment(s) to the device310and/or update the manifest308, to generate and publish the manifest308cin the same manner as discussed above for the manifest308b. The manifest308cmay be based on the one or more characteristics of the transcoded segments as indicated by the one or more associate descriptors305a. Manifest308c, in this example, would indicate that the Representation 1 transcoded segment314c1has a maximum segment bandwidth of 900 Kbps, and that the Representation 2 transcoded segment314c2has a maximum segment bandwidth of 700 Kbps. The device310may determine (e.g., measure) that it only has up to 800 Kbps of bandwidth available (the network experience for the device310may have been degrading over the last several seconds). The device310may compare the available bandwidth (800 Kbps) with the information in the manifest308cfor one or more of the representations, and determine based on the comparison that the available bandwidth is higher than the maximum bandwidth (700 Kbps) of the relevant segment (the transcoded segment314c2) for Representation 2. The device310may generate the request312cto indicate Representation 2, corresponding to the transcoded segment314c2. Put another way, the device310may again select and request the representation indicated by the manifest308cas having the segment with the highest maximum bandwidth that does not exceed the bandwidth available to the device310.

The communication flow and process ofFIG.5may continue as desired. For example, this communication flow and process may continue until the content item has ended and/or until the communication session terminates. The manifest308may be updated, as the content item progresses, to indicate the one or more characteristics of upcoming one or more segments of the content. The device310may use the updated manifest308to determine, on a segment-by-segment basis (or every group of multiple segments, as desired) which representation to use to obtain a transcoded version of the upcoming segment(s) in the content item. Such an updated, dynamic manifest308may allow the device310to make better decisions as compared with a manifest that statically indicates, for example, a maximum bandwidth of an entire item of content and/or a large portion thereof. By knowing the one or more characteristics (for example, maximum bandwidth requirement) of one or more upcoming segments, the device310may be able to obtain an overall higher quality version of the transcoded content305.

Given a dynamically adjusting bandwidth value in the updated manifests308, and/or given dynamically adjusting video quality scores in the same manifest (e.g., generated by a video quality measurement algorithm such as SSIM, PSNR, etc.), the device310could, knowing its own on-network bandwidth, look at the required bandwidth of all available representations of the content in the manifest308. Out of all of the potential options not exceeding the available bandwidth on the network, the device310could look at the video quality score in the manifest308for each representation. If the video quality (e.g., SSIM scores) per each representation are within a certain threshold of each other (for example, within a one-point difference on a 100-point scale), it may be difficult for the Human Visual System (HVS) to discern a noticeable difference between one or another representation. In this case, the device310could use the dynamically updated bandwidth values and video quality scores in the manifest308to make a decision to, e.g., request a lower or the lowest bandwidth representation out of the available options in the manifest308. For example, there may be moments in certain content items where there is very little content complexity (e.g., all black video being one of the least complex, but static text on a solid color background could also be considered rather low-complexity, etc.). In these instances, a user of the device310may notice very little perceivable difference between black video at 1080p displayed on a 1080p-native device vs black video at 360p, which may need to be scaled up for display where the device310might include, e.g., a 1080p-native display device. In this case, the device310could decide to request the 360p representation of the content item until a new dynamic manifest308suggests a video quality score delta has increased substantially between representations. Upon determining this substantial increase (e.g., beyond a predetermined threshold increase), the device310may take this video quality score into account along with the device's310bandwidth requirement and/or available bandwidth on the network, and may make an informed decision as to what would be a desirable user experience. Thus, in this case, the device310may request the corresponding representation having the desired video quality. This type of process may be considered a quality-driven adaptive bit-rate (ABR) content delivery process.

Although examples are described above, features and/or steps of those examples may be combined, divided, omitted, rearranged, revised, and/or augmented in any desired manner. Various alterations, modifications, and improvements will readily occur to those skilled in the art. Such alterations, modifications, and improvements are intended to be part of this description, though not expressly stated herein, and are intended to be within the spirit and scope of the disclosure. Accordingly, the foregoing description is by way of example only, and is not limiting.