Patent Description:
<CIT> discloses a method for transmitting multimedia content to a client comprising active management of the client's receive buffer comprising adjusting the server's transmission rate based on the client's consumption rate to manage the buffer efficiently and avoid depletion or overflow of the buffer.

<CIT> discloses a method and system for near real-time delivery of variable bit rate (VBR) media streams. It addresses challenges in streaming media, particularly related to maintaining quality with variable bandwidth. The method involves identifying a media asset for streaming, determining available bandwidth, and calculating a delay for decoding the media stream based on bandwidth and a minimum transmission rate. This also involves the setting of VBR coding parameters, bandwidth, and transmission rate.

<CIT> discloses a segment based media transmission system wherein data is gathered about current conditions of the network and/or the media player, and encoding of the media stream is adjusted during operation to adapt to changing conditions. The encoding parameter is suitably adjusted in response to changes in available bandwidth and/or segment transmit time, as appropriate.

The invention is defined in the independent claims, while advantageous embodiments are set out in the dependent claims.

Some features are shown by way of example, and not by limitation, in the accompanying drawings. In the drawings, like numerals reference similar elements.

The accompanying drawings, which form a part hereof, show examples of the disclosure. It is to be understood that the examples shown in the drawings and/or discussed herein are non-exclusive and that there are other examples of how the disclosure may be practiced.

<FIG> shows an example communication network <NUM> in which features described herein may be implemented. The communication network <NUM> may comprise one or more information distribution networks of any type, such as, without limitation, a telephone network, a wireless network (e.g., an LTE network, a <NUM> network, a WiFi IEEE <NUM> network, a WiMAX network, a satellite network, and/or any other network for wireless communication), an optical fiber network, a coaxial cable network, and/or a hybrid fiber/coax distribution network. The communication network <NUM> may use a series of interconnected communication links <NUM> (e.g., coaxial cables, optical fibers, wireless links, etc.) to connect multiple premises <NUM> (e.g., businesses, homes, consumer dwellings, train stations, airports, etc.) to a local office <NUM> (e.g., a headend). The local office <NUM> may send downstream information signals and receive upstream information signals via the communication links <NUM>. Each of the premises <NUM> may comprise devices, described below, to receive, send, and/or otherwise process those signals and information contained therein.

The communication links <NUM> may originate from the local office <NUM> and may comprise components not illustrated, such as splitters, filters, amplifiers, etc., to help convey signals clearly. The communication links <NUM> may be coupled to one or more wireless access points <NUM> configured to communicate with one or more mobile devices <NUM> via one or more wireless networks. The mobile devices <NUM> may comprise smart phones, tablets or laptop computers with wireless transceivers, tablets or laptop computers communicatively coupled to other devices with wireless transceivers, and/or any other type of device configured to communicate via a wireless network.

The local office <NUM> may comprise an interface <NUM>, such as a termination system (TS). The interface <NUM> may comprise a cable modem termination system (CMTS) and/or other computing device(s) configured to send information downstream to, and to receive information upstream from, devices communicating with the local office <NUM> via the communications links <NUM>. The interface <NUM> may be configured manage communications among those devices, to manage communications between those devices and backend devices such as servers <NUM>-<NUM> and <NUM>, and/or to manage communications between those devices and one or more external networks <NUM>. The local office <NUM> may comprise one or more network interfaces <NUM> that comprise circuitry needed to communicate via the external networks <NUM>. The external networks <NUM> may comprise networks of Internet devices, telephone networks, wireless networks, fiber optic networks, and/or any other desired network. The local office <NUM> may also or alternatively communicate with the mobile devices <NUM> via the interface <NUM> and one or more of the external networks <NUM>, e.g., via one or more of the wireless access points <NUM>.

The push notification server <NUM> may be configured to generate push notifications to deliver information to devices in the premises <NUM> and/or to the mobile devices <NUM>. The content server <NUM> may be configured to provide content to devices in the premises <NUM> and/or to the mobile devices <NUM>. This content may comprise, for example, video, audio, text, web pages, images, files, etc. The content server <NUM> (or, alternatively, an authentication server) may comprise software to validate user identities and entitlements, to locate and retrieve requested content, and/or to initiate delivery (e.g., streaming) of the content. The application server <NUM> may be configured to offer any desired service. For example, an application server may be responsible for collecting, and generating a download of, information for electronic program guide listings. Another application server may be responsible for monitoring user viewing habits and collecting information from that monitoring for use in selecting advertisements. Yet another application server may be responsible for formatting and inserting advertisements in a video stream being transmitted to devices in the premises <NUM> and/or to the mobile devices <NUM>. The local office <NUM> may comprise additional servers, such as the transcoder <NUM> (described below), additional push, content, and/or application servers, and/or other types of servers. Although shown separately, the push server <NUM>, the content server <NUM>, the application server <NUM>, the transcoder <NUM> (which may also be an encoder), encoder and/or other server(s) may be combined. The servers <NUM>, <NUM>, <NUM>, and <NUM>, and/or other servers, may be computing devices and may comprise memory storing data and also storing computer executable instructions that, when executed by one or more processors, cause the server(s) to perform steps described herein.

An example premises 102a may comprise an interface <NUM>. The interface <NUM> may comprise circuitry used to communicate via the communication links <NUM>. The interface <NUM> may comprise a modem <NUM>, which may comprise transmitters and receivers used to communicate via the communication links <NUM> with the local office <NUM>. The modem <NUM> may comprise, for example, a coaxial cable modem (for coaxial cable lines of the communication links <NUM>), a fiber interface node (for fiber optic lines of the communication links <NUM>), twisted-pair telephone modem, a wireless transceiver, and/or any other desired modem device. One modem is shown in <FIG>, but a plurality of modems operating in parallel may be implemented within the interface <NUM>. The interface <NUM> may comprise a gateway <NUM>. The modem <NUM> may be connected to, or be a part of, the gateway <NUM>. The gateway <NUM> may be a computing device that communicates with the modem(s) <NUM> to allow one or more other devices in the premises 102a to communicate with the local office <NUM> and/or with other devices beyond the local office <NUM> (e.g., via the local office <NUM> and the external network(s) <NUM>). The gateway <NUM> may comprise a set-top box (STB), digital video recorder (DVR), a digital transport adapter (DTA), a computer server, and/or any other desired computing device.

The gateway <NUM> may also comprise one or more local network interfaces to communicate, via one or more local networks, with devices in the premises 102a. Such devices may comprise, e.g., display devices <NUM> (e.g., televisions), STBs or DVRs <NUM>, personal computers <NUM>, laptop computers <NUM>, wireless devices <NUM> (e.g., wireless routers, wireless laptops, notebooks, tablets and netbooks, cordless phones (e.g., Digital Enhanced Cordless Telephone-DECT phones), mobile phones, mobile televisions, personal digital assistants (PDA)), landline phones <NUM> (e.g. Voice over Internet Protocol-VoIP phones), and any other desired devices. Example types of local networks comprise Multimedia Over Coax Alliance (MoCA) networks, Ethernet networks, networks communicating via Universal Serial Bus (USB) interfaces, wireless networks (e.g., IEEE <NUM>, IEEE <NUM>, Bluetooth), networks communicating via in-premises power lines, and others. The lines connecting the interface <NUM> with the other devices in the premises 102a may represent wired or wireless connections, as may be appropriate for the type of local network used. One or more of the devices at the premises 102a may be configured to provide wireless communications channels (e.g., IEEE <NUM> channels) to communicate with one or more of the mobile devices <NUM>, which may be on- or off-premises.

The mobile devices <NUM>, one or more of the devices in the premises 102a, and/or other devices may receive, store, output, and/or otherwise use assets. An asset may comprise a video, a game, one or more images, software, audio, text, webpage(s), and/or other content.

<FIG> shows hardware elements of a computing device <NUM> that may be used to implement any of the computing devices shown in <FIG> (e.g., the mobile devices <NUM>, any of the devices shown in the premises 102a, any of the devices shown in the local office <NUM>, any of the wireless access points <NUM>, any devices with the external network <NUM>) and any other computing devices discussed herein (e.g., the transcoder <NUM>). The computing device <NUM> may comprise one or more processors <NUM>, which may execute instructions of a computer program to perform any of the functions described herein. The instructions may be stored in a read-only memory (ROM) <NUM>, random access memory (RAM) <NUM>, removable media <NUM> (e.g., a USB drive, a compact disk (CD), a digital versatile disk (DVD)), and/or in any other type of computer-readable medium or memory. Instructions may also be stored in an attached (or internal) hard drive <NUM> or other types of storage media. The computing device <NUM> may comprise one or more output devices, such as a display device <NUM> (e.g., an external television and/or other external or internal display device) and a speaker <NUM>, and may comprise one or more output device controllers <NUM>, such as a video processor. One or more user input devices <NUM> may comprise a remote control, a keyboard, a mouse, a touch screen (which may be integrated with the display device <NUM>), microphone, etc. The computing device <NUM> may also comprise one or more network interfaces, such as a network input/output (I/O) interface <NUM> (e.g., a network card) to communicate with an external network <NUM>. The network I/O interface <NUM> may be a wired interface (e.g., electrical, RF (via coax), optical (via fiber)), a wireless interface, or a combination of the two. The network I/O interface <NUM> may comprise a modem configured to communicate via the external network <NUM>. The external network <NUM> may comprise the communication links <NUM> discussed above, the external network <NUM>, an in-home network, a network provider's wireless, coaxial, fiber, or hybrid fiber/coaxial distribution system (e.g., a DOCSIS network), or any other desired network. The communication device <NUM> may comprise a location-detecting device, such as a global positioning system (GPS) microprocessor <NUM>, which may be configured to receive and process global positioning signals and determine, with possible assistance from an external server and antenna, a geographic position of the communication device <NUM>.

Although <FIG> shows an example hardware configuration, one or more of the elements of the computing device <NUM> may be implemented as software or a combination of hardware and software. Modifications may be made to add, remove, combine, divide, etc. components of the computing device <NUM>. Additionally, the elements shown in <FIG> may be implemented using basic computing devices and components that have been configured to perform operations such as are described herein. For example, a memory of the computing device <NUM> may store computer-executable instructions that, when executed by the processor <NUM> and/or one or more other processors of the computing device <NUM>, cause the computing device <NUM> to perform one, some, or all of the operations described herein. Such memory and processor(s) may also or alternatively be implemented through one or more Integrated Circuits (ICs). An IC may be, for example, a microprocessor that accesses programming instructions or other data stored in a ROM and/or hardwired into the IC. For example, an IC may comprise an Application Specific Integrated Circuit (ASIC) having gates and/or other logic dedicated to the calculations and other operations described herein. An IC may perform some operations based on execution of programming instructions read from ROM or RAM, with other operations hardwired into gates or other logic. Further, an IC may be configured to output image data to a display buffer.

<FIG> is a timeline <NUM> of content that may be transmitted via a media content stream. The timeline <NUM> comprises a television show 302a, a first advertisement <NUM>, a second advertisement <NUM>, and a television show 302b. As shown on the timeline <NUM>, the television show 302a stops and the first advertisement <NUM> begins around <NUM>:<NUM>, the first advertisement <NUM> stops and the second advertisement <NUM> begins around <NUM>:<NUM>, and the second advertisement <NUM> stops and the television show 302b begins around <NUM>:<NUM>. The time periods associated with when first media content (e.g., the television show 302a) switches to second media content (e.g., the first advertisement <NUM>) may be referred to as a transition period and/or a transition point. For example, a transition period may correspond to a few seconds of a television show wherein it fades to black, a few seconds where a display remains black, and then a few seconds corresponding to the beginning of an advertisement. The television show 302a and the television show 302b may be two different portions of the same television show. The media content stream may be transmitted, e.g., from the content server <NUM>, to one or more user devices, which may be configured to decode and display the media content stream. Transmission of the media content stream may comprise transmission of one or more packets, which may correspond to one or more frames of the television show 302a, the first advertisement <NUM>, the second advertisement <NUM>, and/or the television show 302b. Transmission of media content need not correlate to display of the media content. For example, the first advertisement <NUM> and the second advertisement <NUM> may be transmitted to the user device much earlier than their display.

<FIG> is a timeline 401a showing a delay in transmission and display of media content. The timeline 401a shown may be a subset of the timeline <NUM> in <FIG>. The horizontal axis of the timeline 401a corresponds to time. As shown in <FIG>, one or more last frames of the TV show are transmitted to a user device (box 402a) from <NUM>:<NUM> to <NUM>:<NUM>, are decoded by the user device (box 402b) from <NUM>:<NUM> to <NUM>:<NUM>, and are displayed by the user device (box 402c) from <NUM>:<NUM> to <NUM>:<NUM>. An ending delay <NUM> may comprise a transmission time and/or decoding time (e.g., an arrival time of a last packet of a last frame of media content in a decoder buffer). Techniques described herein may be applied to packets in generally the same manner as applied to frames, such as the one or more last frames of the TV show. One or more first frames of the advertisement are transmitted to the user device (box 403a) from <NUM>:<NUM> to <NUM>:<NUM> or in advance and stored in a user device buffer, are decoded by the user device (box 403b) from <NUM>:<NUM> to <NUM>:<NUM>, and are displayed by the user device (box 403c) from <NUM>:<NUM> to <NUM>:<NUM>. A startup delay <NUM> may comprise a transmission time and/or decoding time (e.g., an arrival time of ad first frame first packet in the decoder buffer). The startup delay may be a constant predetermined value set, e.g., by an transcoding device (which may alternatively be referred to as an encoding device or encoder) during an encoding process. Though only two sets of frames are depicted in <FIG>, other frames (not shown) may be transmitted to, decoded by, and/or displayed by the user device. As a result of the timing of the ending delay <NUM>, the startup delay <NUM>, and the periods when frames are displayed (represented by box 402c and box 403c), a period where nothing is displayed by the user device (represented by box <NUM>) exists from <NUM>:<NUM> to <NUM>:<NUM>. This period where nothing is displayed by the user device may comprise the user device displaying a buffering indication, a loading screen, a single frame of the television show, display artifacts, or other similar indications of a delay by the user device. The period may also be associated with audio artifacts or clipping.

The length of the period where nothing is displayed by the user device (represented by box <NUM>) may depend on the encoding of media content transmitted via a media content stream. The length may depend on the on the ending delay <NUM> and startup delay <NUM>, which may be associated with the bitrate, encoding format, and/or bit distribution of content transmitted, which in turn can affect how long one or more frames take to be transmitted, decoded, and/or presented (e.g., as represented by boxes 402a, 402b, 402c, 403a, 403b, and/or 403c). For example, a television show may be encoded using CBR, such that a certain amount of time of media content (e.g., <NUM> second) may be associated with a quantity of bits (e.g., <NUM> kilobits). A user device may, based on the predictability of this bit rate, buffer a quantity of the bits of a show for display such that no interruptions occur during decoding and/or display of the television show. An example of an effect of a video buffer is discussed below with respect to <FIG>. In contrast, an advertisement may be encoded using VBR, such that different quantities of bits may correspond to different lengths of the advertisement. For example, a first half of an advertisement may require a low bit rate, causing a user device to buffer, e.g., using a video buffer in RAM, relatively little of the transmission of the advertisement, but a second half of the advertisement may correspond to a large number of bits, causing the user device to pause and download/decode the large number of bits and potentially interrupt display of the advertisement.

The length of the period where nothing is displayed by the user device (represented by box <NUM>) may depend on network conditions (e.g., network delay, network latency, and/or network bandwidth). For example, if the content server <NUM> is transmitting content to the personal computer <NUM>, then variations in network conditions may delay transmissions (e.g., boxes 402a, 403a). This may, in turn, lengthen the period where nothing is displayed by the user device.

<FIG> shows a timeline 401b similar to that of <FIG>, but without a period where nothing is displayed because the ending delay <NUM> and the startup delay <NUM> have been made approximately equal. Interruptions in the display of sequential content may be minimized by making the ending delay <NUM> and the startup <NUM> match within a threshold. Though the ending delay <NUM> and the startup delay <NUM> shown in <FIG> are not exactly equal, their difference may be so small (e.g., nanoseconds) that their impact on the display of content via a user device may not be readily noticed by a viewer. Thus, the ending delay <NUM> and the startup delay <NUM> need only match within a threshold (e.g., a threshold determined based on the likelihood that the delay would be noticed by the average viewer, result in visual artifacts, or the like).

To shorten the startup delay <NUM>, the size of the advertisement frames transmitted may be lowered by, for example, retrieving a lower quality version of the advertisement frames and/or by re-encoding one or more advertisement frames in a different encoding format. This may, for example, avoid a circumstance where a user device may be unable to download a first portion of an advertisement because previous portions of other media content were large (e.g., such that one or more decoders and/or buffers may be full). One or more frames of the advertisement may be removed and/or compressed, such that the length of the corresponding portion of the advertisement remains the same but transmission may be performed more quickly. Similar steps may be taken with respect to the ending delay <NUM>. For example, one or more frames of the television show may be removed, the ending frames of the television show may be re-encoded and/or compressed, or the like.

<FIG> shows a timeline 401c using buffer periods. A user device and/or intermediary devices in a network may implement a buffer to cache a duration of media content prior to display to avoid interruptions in the display of content resulting from delays (e.g., the ending delay <NUM> and the startup delay <NUM>). Buffer periods (e.g., buffer period 407a and buffer period 407b) may be introduced so that the display of the television show frames (box 402c) and the advertisement frames (box 403c) are sequential, and no delay occurs. The buffer period 407a and the buffer period 407b need not be the same. For example, the buffer period 407a is longer than the buffer period 407b such that display of the television show frames (box 402c) and the advertisement frames (box 403c) are approximately sequential, even though the particular size and/or length of the frames may be different.

The length of the period where nothing is displayed by the user device (represented by box <NUM> in <FIG>, not shown in <FIG>) may depend on a buffer size (e.g., of a user device and/or transcoding device) and/or an availability (e.g., of the user device and/or transcoding device). A user device and/or transcoding device may, to prevent display interruptions, modify the size and/or availability of a buffer such that frames are received and/or decoded long before they are displayed. If a user device has a small buffer and/or where buffer size of the user device is limited, display interruptions may result. For example, for a CBR video encoded at <NUM> megabits per second and a user device having a buffer of <NUM> megabits, the user device may be capable of buffering up to ten seconds of video. Should the user device take longer than ten seconds to decode subsequent frames, an interruption may occur.

<FIG> shows a timeline 401d using buffer periods, but involving unexpectedly large advertisement frames. If, for example, the television show was encoded using CBR, the user device may be configured to select a CBR bit rate for the television show (e.g., 100kbps) such that television show frames may be properly received, buffered, decoded, and displayed without interruption. For example, the user device may be configured to maintain a ten second buffer and have a maximum buffer size of <NUM> megabits, such that it may retrieve a <NUM> megabits per second version of the television show. If, however, the advertisement frames are encoded using VBR and/or are only available in higher bit rate CBR (e.g., the first few seconds of the advertisement are <NUM> megabits per second in CBR or VBR), the user device may be unable to select a bit rate and/or allocate buffer size to avoid interruption. As such, as shown in <FIG>, it may take so long to receive the advertisement frames that an interruption occurs and nothing is displayed (represented by box <NUM>).

<FIG> shows a timeline 401e where the ending delay <NUM> exceeds the startup delay <NUM> and a delay period <NUM> is added. The delay period <NUM> is added such that transmission (box 403a) and/or decoding (box 403b) of the advertisement frames are delayed, and display of the ad frames (box 403c) occurs immediately after the television show frames (402c). Delays, such as the delay period <NUM>, may be before/after transmission/receipt of one or more frames, may be before/after decoding of one or more frames, and/or may be before/after display of one or more frames. The scenario depicted in <FIG> may occur where, for example, last frames of a television show require a long time to transmit, decode, and/or display, but the first frames of an advertisement require a relatively shorter time to transmit, decode, and/or display.

<FIG> is a flow chart showing the method for encoding media content. The steps shown in <FIG> may be performed by a computing device, such as the computing device <NUM>, the transcoder <NUM>, the app server <NUM>, and/or one or more of the devices described in the household 102a. For example, steps shown in <FIG> may be performed by a transcoder (e.g., the transcoder <NUM>), and/or may be performed by a user device (e.g., the laptop computer <NUM>).

In step <NUM>, initial configuration settings is determined. Information about user devices, such as their buffer size and availability, is determined and stored. Network condition information, such as bandwidth, latency, and reliability, may be determined and stored. Available encoders are determined and selected based on their abilities. For example, a plurality of encoders may be queried to determine their availability, including their ability to encode content in VBR, CBR, and CVBR. One or more encoders capable of removing frames from media content may be identified. Maximum acceptable delay times may be determined. It is determined that the maximum acceptable delay time between two pieces of media content may not exceed a second because any larger delay may be easily noticed by the average viewer.

In step <NUM>, first media content and second media content are identified. The first media content and second media content may be transmitted in a media content stream. Media content, such as the first media content and second media content, may be any audiovisual content, such as a television show, movie, advertisement, and/or a live broadcast. Media content may be encoded (e.g., in CBR, VBR, or CVBR) using one or more encoding standards, such as MPEG-<NUM> Part <NUM>, Advanced Video Coding (MPEG-<NUM> AVC). The first media content and/or second media content may be identified based on metadata, e.g., a manifest that provides timecodes for the end of the first media content and/or the beginning of the second media content and/or information about the encoding of the first media content and/or the second media content. Additionally and/or alternatively, the first media content and the second media content may be identified by determining a portion of the media content stream that switches from a first type of encoding (e.g., CBR) to a second type of encoding (e.g., VBR). For example, a computing device may be configured to buffer a certain amount of a media content stream before it is transmitted to user devices, such that the differences in types of encoding may be detected by analyzing the media content stream as stored in the buffer. Additionally and/or alternatively, the first media content and/or second media content may be determined based on cue tones in streams. For example, as described in Society of Cable Telecommunications Engineers (SCTE) standard <NUM>, one or more cue tones may be inserted into a telecommunications stream to indicate a time period wherein an advertisement may be inserted. The presence of these tones may be used to distinguish between the first media content (wherein no cue tone may be present) and second media content (wherein the tone may be present an advertisement may be displayed). As another example, first media content may be determined based on a cue tone indicating an end of content being distributed, whereas second media content may be determined based on a cue tone indicating a beginning of content.

The first media content and second media content may come from different sources and have different encoding properties, such as different encoding formats. For example, the first media content and second media content may be at a different resolution, bit rate, have different dynamic ranges, and/or may require different levels of decoding prior to display. For example, the content server <NUM> may store television shows at 720p and in CBR, whereas advertisements may be stored on the app server <NUM> at 1080p and in VBR.

In step <NUM>, an ending delay is determined. The ending delay comprises one or more time periods associated with transmission of, receipt of, and/or decoding of one or more ending frames of the first media content. For example, it may take <NUM> for the content server <NUM> to transmit, and user equipment (e.g., a set-top box or other forms of the gateway <NUM>) to receive, process, and/or cause display of, one or more last frames of the first media content, it may take <NUM> for the display device <NUM> to decode the last frames, and the last frames may be for <NUM> of the first media content. The ending delay in that example may be <NUM>, with a corresponding display time of <NUM>. The ending delay may be projected and/or estimated. For example, the amount of time it may take to transmit, decode, and/or display frames may be predicted based on network conditions, processing capabilities of the user device, or similar considerations to generate a projected ending delay.

In step <NUM>, a startup delay is determined. The startup delay comprises one or more time periods associated with transmission of, receipt of, and/or decoding of one or more frames of the second media content. For example, it may take <NUM> for the content server <NUM> to transmit, and the display device <NUM> to receive, first frames of the second media content, it may take <NUM> for the display device <NUM> to decode the first frames, and the first frames may be for <NUM> of the first media content. The startup delay in that example may be <NUM>, with a corresponding display time of <NUM>. Like the ending delay, the startup delay may be projected and/or estimated based on network conditions, processing capabilities of the user device, or similar considerations.

In step <NUM>, the startup delay and the ending delay are compared, and differences between the delays are determined. The transmission times of the last frames of the first media content may be compared with the transmission times of the first frames of the second media content. The decoding times of the last frames of the first media content may be compared with the decoding times of the first frames of the second media content. For example, the last frames of the first media content may take a comparatively longer amount of time to decode than the first frames of the second media content because the former may have a higher resolution than the latter. The display times of the last frames of the first media content may be compared with the display times of the first frames of the second media content. For example, the last frames may correspond to the last thirty seconds of the television show, but the first frames may correspond to only five seconds of the advertisement.

In step <NUM>, the differences between the delays are compared to a threshold. The threshold is a maximum acceptable delay time and may be based on an amount of difference between the delays that may be acceptable. The difference between the delays may be acceptable if the last frame of the first media content and the first frame of the second media content are displayed without interruption and/or with a sufficiently short interruption (e.g., such that the interruption is barely detectable by the average viewer). The difference between the delays may be acceptable if the last frame of the first media content and the first frame of the second media content are displayed without artifacts and/or with an acceptably low number of artifacts. The threshold may be based on the genre and/or length of the first media content and the second media content. For example, a relatively large delay may be acceptable after the end of a movie or television show, but may be unacceptable between relatively short advertisements. If the difference satisfies the threshold, for example where the difference between the delays is lower than the acceptable delay, the flow chart may end. Otherwise, the flow chart may proceed to step <NUM>.

In step <NUM>, the first media content and the second media content are compared to determine whether there is a difference in a format of the first media content and a format of the second media content. For example, the first media content may be encoded in CBR, but the second media content may be encoded in VBR, or vice versa. The first media content and the second media content may also differ in that they may be encoded at different maximum bit rates, resolutions, with different quantization parameter (QP), or the like. These differences may cause delay because, for example, a user device may be unable to properly predict and transmit VBR or CBR content where it has an unexpectedly high bit rate. Such a delay may be avoided if, for example, all or portions of the first media content and the second media content are transmitted using CBR or CVBR and at a particular bit rate such that the user device may allocate bandwidth to and properly buffer the content. It thus may be desirable to convert VBR portions of the first media content and/or the second media content to CBR or CVBR. As another example, a switch from low resolution to high resolution content may cause undesirable lag in a user device, and thus user experience may be improved if the high resolution content is re-encoded into a lower resolution. If there is a difference in format, the flow chart proceeds to step <NUM>. Otherwise, the flow chart proceeds to step <NUM>.

In step <NUM>, all or portions of the first media content are encoded. The all or portions of the first media content are encoded such that a projected difference in startup delay and ending delay is less than a maximum acceptable delay time. As the first media content and/or the second media content may already be encoded, this step may comprise re-encoding the first media content and/or the second media content. An encoder, such as the transcoder <NUM>, is instructed to encode the first media content using one or more encoding parameters selected based on the comparison in step <NUM>. Such encoding parameters may comprise, for example, an indication to encode all or portions of the first media content and/or the second media content in CBR or CVBR and at a particular bit rate. The particular bit rate may additionally and/or alternatively be selected based on the capabilities of the user device and availability of the network. For example, if network conditions are poor and/or if a user device has a relatively small buffer, the bit rate selected may be relatively low.

Where the first media content and/or the second media content are encoded in VBR, it may be desirable to encode the VBR content into CBR or CVBR. By re-encoding the first media content and/or the second media content in a manner that limits the maximum number of bits per second to a predetermined bit rate, buffers may be accordingly adjusted based on the predetermined bit rate such that no interruption occurs. For example, VBR media content having an initial bit rate of <NUM>-<NUM> megabits per second may be re-encoded in CVBR or CBR of <NUM> megabits per second. Additionally and/or alternatively, a predetermined quantization parameter (QP) may be used to affect the size of the first media content and/or the second media content. For example, the first media content may be encoded using a relatively small QP so that more detail is retained, but the first media content may be re-encoded using a relatively larger QP such that less detail is retained and the overall file size of the first media content is lowered. The predetermined bit rate and/or predetermined QP may be based on network conditions, capabilities of one or more user devices (e.g., the projected speed in which a user device may decode media content at a particular bit rate and/or QP), or the like.

In step <NUM>, the first media content and/or the second media content may be analyzed to determine whether unnecessary frames are present in the first media content and/or the second media content. The first media content and/or the second media content may comprise one or more entirely black frames (e.g., frames after the content has faded out) and/or frames that are repeated (e.g., the same image repeated multiple times). A computing device may detect such frames by comparing two or more adjacent frames in the first media content and/or the second media content such that, if the two frames are identical (e.g., as determined by hashing both frames and comparing the hashes) and/or sufficiently similar (e.g., as determined by comparing the two or more frames and determining that the color and brightness of the pixels are <NUM>% similar across the frames), a match is determined to exist. Such frames may be removed (and, if desired, replaced by repeating an existing frame) without being easily noticed by a viewer. This removal may reduce the overall size of the first media content and/or the second media content, decreasing transmission and/or decoding time. If there are unnecessary frames, the flow chart proceeds to step <NUM>. Otherwise, the flow chart proceeds to step <NUM>.

The one or more unnecessary frames may be based on determining that the unnecessary frames have little or no visual content. Solid-colored frames that do not contain any content may have no or minimal visual content, whereas frames of an exciting chase scene in a movie may have significant amounts of visual content. Such visual content may be determined based on metadata (e.g., metadata indicating that a particular number of frames are all black) and/or an analysis of one or more frames (e.g., noticing that a frame comprises nothing but a single color). Visual content may be determined based on a scene detection algorithm. A manifest file associated with the first media content and/or the second media content may indicate one or more portions of first media content and/or second media content that have little to no visual content.

In step <NUM>, one or more unnecessary frames may be removed from the first media content and/or the second media content. Removing the one or more unnecessary frames may comprise re-encoding the first media content and/or the second media content, as described with respect to step <NUM>. For example, the transcoder <NUM> may be instructed to re-encode the first media content and/or the second media content while simultaneously removing certain frames from the first media content and/or the second media content. Removing the unnecessary frames may additionally and/or alternatively comprise using the trim functionality available in video editing tools. Removing the unnecessary frames may additionally and/or alternatively comprise causing one or more frames to repeat. For example, all but one entirely black frame may be removed from media content, and the remaining black frame may be repeated for a duration corresponding to the removed frames. Additionally and/or alternatively, the removed frames may be replaced with different frames, e.g., blank frames encoded at a lower bitrate.

In step <NUM>, the system may determine whether a buffer (e.g., in a user device) is too small to store a necessary amount of the first media content and/or the second media content. A buffer may be configured in user devices (e.g., within the RAM of the personal computer <NUM>) and/or may be located in one or more devices on a network (e.g., between the content server <NUM> and the display device <NUM>). The buffer may be associated with a buffering algorithm, e.g., executing on the user device. The buffer may be configured to store a certain quantity (e.g., a number of frames, a number of bits, a period of time) of a media content stream. But given the ending delay and/or the startup delay and/or current network conditions, the buffer may be too small: for example, the second media content may be in VBR and have a high bit rate early on, such that a relatively small buffer may overflow. As another example, network conditions may be so poor that a high bit rate VBR content may take an undesirably long time to be downloaded by a user device, such that early retrieval and buffering of the high bit rate VBR content may be necessary to avoid visual interruption. As such, increasing the size of a buffer may be desirable where doing so would reduce the possibility of an interruption in the display of content. Decreasing the size of a buffer may also be desirable where the first media content and/or the second media content are live, such that an excessively large buffer may excessively delay the display of the first media content and/or the second media content. If the buffer is too small, the flow chart proceeds to step <NUM>. Otherwise, the flow chart returns to step <NUM>.

In step <NUM>, based on determining that the buffer is too small, a size of the buffer may be enlarged so that the buffer may be capable of storing a necessary amount of the first media content and/or the second media content. The size may be increased in order to provide additional time for the transmission and/or receipt of content, such that display of the content is not interrupted. For example, a user device may be instructed to temporarily increase the size of its buffer (e.g., from ten seconds to twenty seconds) based on determining that a delay of approximately ten seconds is likely.

<FIG> is a flow chart depicting how, in an example implementation of some aspects described herein, a transcoding device may processes described herein may be performed by a transcoding device, such as the transcoder <NUM>, based on cue tones in a content stream. In step <NUM>, the transcoder may process a content stream. Processing may comprise transcoding the content stream using VBR, CBR, or CVBR. For example, processing may comprise encoding media content using default encoding settings, where the default encoding settings are selected to maximize quality. Processing may comprise listening for specific cue tones, such as those described in SCTE <NUM>. For example, a transcoder may analyze an audio waveform in order to detect specified in cue tones and out cue tones. Processing may comprise retrieving a manifest corresponding to the content stream. For example, a transcoder may encode different portions of a content stream based on different encoding settings specified in a manifest corresponding to the content stream.

In step <NUM>, the transcoding device may detect an out cue tone. An out cue tone may indicate the end of first media content and/or the beginning of the second media content. The cue tone may be a cue tone of the type described in SCTE <NUM>. If an out cue tone is detected, the flow chart may continue to step <NUM>. Otherwise, the flow chart may proceed to step <NUM>.

In step <NUM>, the transcoding device may detect an in cue tone. An in cue tone may indicate the end of second media content and/or the beginning of the next media content. As with the out cue tone, the in cue tone may be a cue tone of the type described in SCTE <NUM>. If an in cue tone is detected, the flow chart may proceed to step <NUM>. Otherwise, the flow chart may continue to step <NUM>.

In step <NUM>, based on detecting the out cue tone or the in cue tone, the content stream may be encoded using a constant bit rate for a period of time. The period of time may be selected such that at least a portion of first media content and at least a portion of second media content are encoded using a constant bit rate. For example, the processing performed in step <NUM> may encode first media content and second media content in VBR; however, for two seconds at the end of the first media content and for four seconds at the beginning of the second media content, both the first media content and the second media content may be encoded using CBR. Additionally and/or alternatively, in step <NUM>, one or more frames may be removed and/or repeated, and/or a buffer associated with the content stream may be modified.

In step <NUM>, it is determined whether the end of the content stream has been reached. If so, the flow chart ends. Otherwise, the flow chart returns to step <NUM> and continues to process the content stream.

<FIG> is a flow chart depicting how, in an example implementation of some aspects described herein, the processes described herein may be performed by a transcoding device, such as the transcoder <NUM>, by analyzing a starting delay period and an ending delay period. Step <NUM>, step <NUM>, and step <NUM> are similar to step <NUM>, step <NUM>, and step <NUM>, respectively. In step <NUM>, a content stream is processed. If an out cue is detected in step <NUM>, the flow chart proceeds to step <NUM>. Otherwise, the flow chart continues to step <NUM>. If an in cue is detected in step <NUM>, the flow chart proceeds to step <NUM>. Otherwise, the flow chart continues to step <NUM>.

In step <NUM>, an ending delay period is determined. In step <NUM>, a starting delay period is determined. Both the ending delay period and the starting delay period may be projected and/or estimated based on network conditions, the processing capability of user devices and/or intermediary devices, the length of one or more portions of the first media content and/or the second media content, or similar considerations. If an out cue is detected, then the ending delay period may be more reliably estimated than the startup delay period, as frames corresponding to the second media content may not have yet been received. If an in cue is detected, then the starting delay period may be more reliably determined than the ending delay period, as the actual ending delay period resulting from one or more previous frames may not be known.

In step <NUM>, it is determined whether the starting delay is greater than the ending delay. If the starting delay is greater than the ending delay, then the first media content may end sooner than the second media content may be retrieved, decoded, and displayed. Such a circumstance may result in nothing being displayed, as represented by box <NUM> of <FIG>. If the starting delay is greater than the ending delay, the flow chart may proceed to step <NUM>. Otherwise, the flow chart continues to step <NUM>.

In step <NUM>, one or more frames of the first media content and/or the second media content may be processed to cause the startup delay to be approximately less than or equal to the ending delay. For example, the one or more frames of the first media content and/or the second media content may be re-encoded such that the startup delay is a predetermined amount of time less than the ending delay. Processing may comprise re-encoding all or portions of the first media content and/or the second media content, removing one or more unnecessary frames from the first media content and/or the second media content, causing one or more frames of the first media content and/or the second media content to repeat, and/or modifying a buffer associated with a device configured to receive the first media content and/or the second media content. For example, the transcoder may select CBR encoding parameters (e.g., a constant bit rate budget) based on a projection that, when all or portions of the first media content and/or the second media content are encoded using the CBR encoding parameters, the startup delay will be approximately less than or equal to the ending delay.

In step <NUM>, it is determined whether the end of the stream has been reached. If so, the flow chart ends. Otherwise, the flow chart returns to step <NUM>.

<FIG> is a flow chart depicting how, in an example implementation of some aspects described herein, user equipment (e.g., a set-top box or other forms of the gateway <NUM>) may retrieve lower bitrate content. In step <NUM>, a content stream may be received. For example, a set-top box may receive a content stream corresponding to a television channel. The received stream may be monitored (e.g., by the user device) for one or more cue tones and/or one or more indications of a transition period between first media content and/or second media content. The user device may retrieve a manifest corresponding to the received media content stream and, based on the manifest, determine a transition period between first media content and second media content.

Step <NUM> and step <NUM> are similar to step <NUM> and step <NUM>, respectively. In step <NUM>, if an out cue is detected, the flow chart may proceed to step <NUM>. Otherwise, the flow chart proceeds to step <NUM>. In step <NUM>, if an in cue is detected, the flow chart may proceed to step <NUM>. Otherwise, the flow chart may continue to step <NUM>.

In step <NUM>, if an out cue or an in cue are detected, an ending delay period is determined. In step <NUM>, a starting delay period is determined. Both step <NUM> and step <NUM> are similar to step <NUM> and step <NUM>, respectively. In step <NUM>, it is determined whether the starting delay is greater than the ending delay. If so, the flow chart proceeds to step <NUM>. Otherwise, the flow chart continues to step <NUM>.

In step <NUM>, if the starting delay period is greater than the ending delay period, lower bitrate content may be retrieved. Retrieval of lower bitrate content may comprise retrieving a lower bit rate portion of the first media content and/or the second media content. For example, based on determining that the starting delay period associated with an advertisement is greater than the ending delay period associated with a television show, a lower bit rate version of the ad may be retrieved when switching from a television show to an advertisement. Additionally and/or alternatively, retrieval of lower bitrate content may comprise causing a transcoder to encode all or portions of the content stream at a lower bitrate by, for example, transmitting a request for lower bit rate content to the transcoder. Additionally and/or alternatively, a buffer may be modified to make the starting delay approximately less than or equal to the ending delay. Additionally and/or alternatively, one or more frames may be skipped and/or repeated in order to make the starting delay approximately less than or equal to the ending delay.

Claim 1:
A method performed by a computing device, the method comprising:
determining initial configuration settings (<NUM>) including information about available encoders and at least one user device (<NUM>);
identifying (<NUM>) a first media content and a subsequent second media content;
determining (<NUM>) a duration of a projected and/or estimated ending delay comprising time periods associated with receipt of and decoding of one or more ending frames of the first media content, that has been encoded using at least one first encoding parameter;
determining (<NUM>) a duration of a projected and/or estimated startup delay comprising time periods associated with receipt of and decoding of one or more frames of the second media content, that has been encoded using at least one second encoding parameter;
determining (<NUM>) a first time difference between the durations of the startup delay and the ending delay;
comparing (<NUM>) the first time difference to a threshold indicating a maximum acceptable delay time;
if the first time difference exceeds the threshold, determining (<NUM>) whether there is a difference in a format of the first media content and a format of the second media content;
if there is a difference in format, selecting at least one new encoding parameter for at least the first media content based on the comparison, such that a projected difference in startup delay and ending delay is less than a maximum acceptable delay time;
causing the encoder to encode (<NUM>), using the at least one new encoding parameter, at least a portion of the first media content.