Patent Description:
Internet-based video technologies may rely on Hypertext Transfer Protocol (HTTP) based adaptive streaming. This class of protocols has been standardized under the umbrella of Dynamic Adaptive Streaming over HTTP (DASH). In DASH systems, video content is divided into multiple segments or chunks, each segment or chunk corresponding to a period of playback time. The video content is encoded at multiple discrete bit-rates and the segments or chunks from different bit-rate streams are aligned so that a video player can switch to a different bit-rate at a chunk boundary in response to changes in network bandwidth conditions. <CIT> relates to communication of data and, more specifically, to a technique for controlling streaming data packet transmissions. <CIT> relates to a system for a mobile wireless device to receive and display a video stream while preventing overflow or starvation of its receive buffer by requesting changes to the video streaming or encoding rates and by controlling the video playback frame rate. The current receive buffer level is used to make comparisons with several thresholds, the results of which are used to trigger actions. If the current receive buffer level has risen above a start level, then playback of the video can begin. If the current receive buffer level rises above an early detection threshold, then the video streaming device is requested to slow its streaming rate. If the current receive buffer level rises above a high level threshold, then the video streaming device is requested to stop streaming the video. If the current receive buffer level drops below a low level threshold, then the play-back frame rate is slowed. <CIT> relates to a method for estimating bandwidth, the method comprising: receiving a server response comprising a plurality of chunks of a segment of a video; identifying the plurality of chunks by identifying the presence of a chunk delineator in the response; identifying at least a subset of the chunks that have a size that is equal to or greater than a Maximum Transfer Unit (MTU) value; and outputting an estimated bandwidth based on a total size of chunks in the subset, and a total download time for the chunks in the subset.

The present invention is set forth in the independent claims. Further advantageous embodiments are set forth in the dependent claims. For purposes of explanation, several embodiments of the subject technology are set forth in the following figures.

The detailed description set forth below is intended as a description of various configurations of the subject technology and is not intended to represent the only configurations in which the subject technology may be practiced. The appended drawings are incorporated herein and constitute part of the detailed description. The detailed description includes specific details for providing a thorough understanding of the subject technology. However, the subject technology is not limited to the specific details set forth herein and may be practiced without one or more of the specific details. In some instances, structures and components are shown in a block-diagram form in order to avoid obscuring the concepts of the subject technology.

Internet-based video technologies may rely on Hypertext Transfer Protocol (HTTP) based adaptive streaming. This class of protocols has been standardized under the umbrella of Dynamic Adaptive Streaming over HTTP (DASH). In DASH systems, video content is divided into multiple segments or chunks, each segment or chunk containing a portion of video content data corresponding to a period of playback time (e.g., <NUM> seconds, <NUM> seconds, <NUM> seconds, etc.). The term "playback" refers to the presentation of decoded video content, which may include both visual content and audio content, on a display device which may be a television, a laptop, a tablet, a smartphone, etc. The video content is encoded at multiple discrete bit-rate levels and the segments or chunks from different bit-rate streams are aligned so that a video player can switch to a different bit-rate level at a chunk boundary in response to changes in network bandwidth conditions. The bit-rate level indicates an amount of data per unit time (e.g., megabits per second) at which the video content is encoded and which should be accommodated by available network bandwidth for uninterrupted streaming of the video content. The increased amount of data provided by higher bit-rate levels may be used to improve the quality of the streamed video content by increasing resolution and/or increasing frame rate, for example. The terms "segment" and "chunk" are used interchangeably herein.

For example, <FIG> is a diagram illustrating adaptive bit-rate streaming of video content according to aspects of the subject technology. As depicted in <FIG>, M copies of video content are stored on a server. Each copy of the video content is divided into K segments or chunks and is encoded at a different respective bit-rate level from a set of available bit-rate levels <IMG>= {r<NUM>, r<NUM>,. , rM} resulting in M streams of the video content being stored on the server encoded at different respective bit-rate levels. The bold line drawn through the various streams of the video content represents an example path illustrating changes in bit-rate levels made while streaming the video content. The streamed K segments or chunks of the video content encoded at the different bit-rate levels are arranged along the timeline illustrated at the bottom of <FIG>.

The Quality of Experience (QoE) plays a critical role in Internet video applications, as it ultimately affects revenue streams for content providers. Specifically, metrics such as the duration of rebuffering (i.e., the player's playback buffer does not have content to play), startup delay (i.e., the lag between the user clicking vs. the time to begin playback), the average playback bit-rate, and the variability of the bit-rate delivered have emerged as key factors. Among all those factors, the rebuffering time is of top importance as too long or too frequent rebuffering may cause users to abandon watching the current channel and switch to other programs.

The subject technology proposes control algorithms designed to reduce both the amount of rebuffering time and the frequency of rebuffering events during adaptive bit-rate streaming of video content. According to aspects of the subject technology, a control algorithm is introduced on top of an existing ABR control algorithm that may replace bit-rate level decisions made by the ABR control algorithm based on fill levels of a playback buffer in order to reduce rebuffering time and frequency. Alternatively, or in addition to, another control algorithm may be employed that controls a playback speed of the video content during streaming based on fill levels of the playback buffer in order to reduce rebuffering time and frequency. These control algorithms and their associated improvements are discussed in more detail below.

<FIG> illustrates an example of a network environment <NUM> in which an ABR video streaming system may be implemented in accordance with aspects of the subject technology. Not all of the depicted components may be required, however, and one or more implementations may include additional components not shown in the figure. Additional components, different components, or fewer components may be employed.

The example network environment <NUM> includes content delivery network (CDN) <NUM> that is communicably coupled to electronic device <NUM>, such as by network <NUM>. CDN <NUM> may include, and/or may be communicably coupled to, content server <NUM>, antenna <NUM>, and/or satellite transmitting device <NUM>. Content server <NUM> can encode and/or transmit encoded data streams, such as MPEG AVC (Advanced Video Coding)/ITU-T H. <NUM> encoded video streams, MPEG HEVC (High-Efficiency Video Coding)/ITU-T H. <NUM> encoded video streams, VP9 encoded video streams, AOM AV1 encoded video streams, and/or MPEG VVC (Versatile Video Coding)/ITU-T H. <NUM> encoded video streams, over network <NUM>. Antenna <NUM> transmits encoded data streams over the air, and satellite transmitting device <NUM> can transmit encoded data streams to satellite <NUM>.

Electronic device <NUM> may include, and/or may be coupled to, satellite receiving device <NUM>, such as a satellite dish, that receives encoded data streams from satellite <NUM>. In one or more implementations, electronic device <NUM> may further include an antenna for receiving encoded data streams, such as encoded video streams, over the air from antenna <NUM> of the CDN <NUM>. Content server <NUM> and/or electronic device <NUM> may be, or may include, one or more components of the electronic system discussed below with respect to <FIG>, <FIG>, <FIG>, and/or <NUM>.

Network <NUM> may be a public communication network (such as the Internet, a cellular data network or dial-up modems over a telephone network) or a private communications network (such as private local area network (LAN) or leased lines). Network <NUM> may also include, but is not limited to, any one or more of the following network topologies, including a bus network, a star network, a ring network, a mesh network, a star-bus network, a tree or hierarchical network, and the like. In one or more implementations, network <NUM> may include transmission lines, such as coaxial transmission lines, fiber optic transmission lines, or generally any transmission lines, that communicatively couple content server <NUM> and electronic device <NUM>.

Content server <NUM> may include, or may be coupled to, one or more processing devices, data store <NUM>, and/or an encoder. The one or more processing devices execute computer instructions stored in data store <NUM>, for example, to implement a content delivery network. Data store <NUM> may store the computer instructions on a non-transitory computer-readable medium. Data store <NUM> may further store multiple copies of video content encoded at different respective bit-rate levels that are delivered by CDN <NUM>. An encoder may use a codec to encode video streams, such as an AVC/H. <NUM> codec, an HEVC/H. <NUM> codec, a VP9 codec, an AV1 codec, a VVC/H. <NUM> codec, or any other suitable codec.

In one or more implementations, content server <NUM> may be a single computing device such as a computer server. Alternatively, content server <NUM> may represent multiple computing devices that are working together to perform the actions of a server computer (such as a cloud of computers and/or a distributed system). Content server <NUM> may be coupled with various databases, storage services, or other computing devices, such as an adaptive bit rate (ABR) server, that may be collocated with content server <NUM> or may be disparately located from content server <NUM>.

Electronic device <NUM> may include, or may be coupled to, one or more processing devices, a memory, and/or a decoder, such as a hardware decoder. Electronic device <NUM> may be any device that is capable of decoding an encoded data stream, such as a VVC/H. <NUM> encoded video stream.

In one or more implementations, electronic device <NUM> may be, or may include all or part of, a laptop or desktop computer, a smartphone, a tablet device, a wearable electronic device such as a pair of glasses or a watch with one or more processors coupled thereto and/or embedded therein, a set-top box, a television or other display with one or more processors coupled thereto and/or embedded therein, video game console, or other electronic devices that can be used to receive and decode an encoded data stream, such as an encoded video stream.

In <FIG>, electronic device <NUM> is depicted as a set-top box, e.g., a device that is coupled to, and is capable of displaying video content on display <NUM>, such as a television, a monitor or any device capable of displaying video content. In one or more implementations, electronic device <NUM> may be integrated into display <NUM> and/or display <NUM> may be capable of outputting audio content in addition to video content. Electronic device <NUM> may receive streams from CDN <NUM>, such as encoded data streams, that include video content items, such as television programs, movies, or generally any content items. Electronic device <NUM> may receive the encoded data streams from the CDN <NUM> via antenna <NUM>, via network <NUM>, and/or via satellite <NUM>, and decode the encoded data streams, e.g., using a hardware decoder.

<FIG> is a block diagram illustrating components of an electronic device, such as electronic device <NUM> represented in <FIG>, according to aspects of the subject technology. Not all of the depicted components may be required, however, and one or more implementations may include additional components not shown in the figure. Depicted or described connections between components are not limited to direct connections and may be implemented with one or more intervening components.

The electronic device (e.g., adaptive video player) depicted in <FIG> includes HTTP engine <NUM>, playback buffer <NUM>, throughput predictor <NUM>, ABR controller <NUM>, and decoder <NUM>. According to aspects of the subject technology, HTTP engine <NUM> issues requests ("GET") to server <NUM> via network <NUM> (e.g., the Internet) for chunks of video content encoded at a selected bit-rate level. HTTP engine <NUM> downloads the requested chunks from server <NUM> via network <NUM> and stores the chunks in playback buffer <NUM>. HTTP engine <NUM> also may report the network throughput (i.e., bandwidth) experienced while downloading the chunks from server <NUM>.

ABR controller <NUM> selects bit-rate levels for the next chunk(s) to be downloaded by HTTP engine <NUM> and notifies HTTP engine <NUM> of the selections. ABR controller <NUM> makes the bit-rate level selections based on one or more inputs received from other components of the electronic device. For example, throughput predictor <NUM> estimates the network bandwidth expected to be available for downloading the next chunk based on the previous bandwidth measures delivered by HTTP engine <NUM> to throughput predictor <NUM>. Playback buffer <NUM> reports or makes available for querying a fill level of the playback buffer in terms of the amount of playback time available from the chunks of video content buffered in the playback buffer. Other metrics such as the number of chunks or video segments buffered in the playback buffer, for example, may be used to measure the fill level of the playback buffer. One or both of the estimated network bandwidth and the fill level of the playback buffer may be used by ABR controller <NUM> to make bit-rate level selections. ABR controller <NUM> also may use other inputs in addition to or in place of the two inputs described above.

Decoder <NUM> consumes and decodes chunks of video content from playback buffer <NUM> and provides the decoded video content to display <NUM> for playback of the video content to viewer. Decoder <NUM> also may report user-perceived Quality-of-Experience (QoE) scores to assist the decision-making logic in ABR controller <NUM>. Examples of the processes summarized above are explained in more detail in the description provided below.

Each of the components depicted in <FIG>, or one or more portions thereof, may be implemented in software (e.g., instructions, subroutines, code), may be implemented in hardware (e.g., an Application Specific Integrated Circuit (ASIC), a Field Programmable Gate Array (FPGA), a Programmable Logic Device (PLD), a controller, a state machine, gated logic, discrete hardware components, or any other suitable devices) and/or may be implemented as a combination of both.

<FIG> is a flowchart illustrating an example adaptive bit-rate streaming process according to aspects of the subject technology. For explanatory purposes, the blocks of the process illustrated in <FIG> are described herein as occurring in serial, or linearly. However, multiple blocks of the process may occur in parallel. In addition, the blocks of the process need not be performed in the order shown and/or one or more blocks of the process need not be performed and/or can be replaced by other operations.

For purposes of describing the process illustrated in <FIG>, downloaded video content is modeled as a set of consecutive video segments or chunks, V = {<NUM>, <NUM>,· · · , K}, where L is the segment time (i.e., each segment contains L seconds of video). Multiple copies of the video content are stored on the server where each copy is encoded at a different bit-rate selected from the set of available bit rate levels, <IMG>= {r<NUM>, r<NUM>,. , rM}, where r<NUM> < r<NUM> < ··· , < rM. The adaptive video player may choose to download video segment k at a selected bit-rate level Rk ∈ <IMG>. B(t) ∈ [<NUM>, Bmax] is the fill level at time t (i.e., the play time of the video content remaining in the playback buffer), Bmax is the maximum amount of playback time of video content that can be buffered in the playback buffer. Bk = B(tk) represents the fill level of the playback buffer at the time tk, which is the time at which the adaptive video player begins to download video segment k from the server.

At the start of the process in <FIG>, tracking parameters total_rebuffering_time (tr_time) and total_rebuffering_events (tr_events) are both reset to zero. Total_rebuffering_time keeps track of the total amount of time the adaptive video player spends rebuffering during playback of the video content, and therefore not playing back video, to add one or more video segments to the playback buffer in response to the playback buffer being emptied by the decoder. Total_rebuffering_events keeps track of the total number of instances during playback of the video content that rebuffering occurs. In addition, the current video segment number is set to one (k=<NUM>), the selected bit-rate level for the first video segment is set to the lowest bit-rate level available (R<NUM> = r<NUM>), and the fill level of the playback buffer is initialized to zero (B<NUM> = <NUM>). With these values in place, the HTTP engine issues a request for video segment <NUM> encoded at bit-rate level r<NUM> and the fill level is updated to B<NUM> = B<NUM> + L. After the initialization of the parameters outlined above, playback by the decoder is started.

After the process has been started, the occupancy level of the playback buffer is checked to determine if there is room in the playback buffer to append video segment k (block <NUM>). If there is not sufficient space in the playback buffer for video segment k, the occupancy level of the playback buffer is reduced using Playback((Bk+<NUM> + L - Bmax)+), where Playback(t) is a function that consumes video content of length t in the playback buffer and (x)+ = max (x, <NUM>) ensures that the term can never be negative (block <NUM>). The fill level of the playback buffer is then updated to Bk+<NUM> = Bk+<NUM> - (Bk+<NUM> + L - Bmax)+.

Δtk represents a waiting time before the HTTP engine may start to download chunk k + <NUM> after the downloading of chunk k has been completed. Δtk is employed in some ABR control algorithms to improve fairness of multi-player video streaming. If Δtk is being employed, the fill level of the playback buffer is reduced using Playback(Δtk) and the fill level is updated to Bk+<NUM> = Bk+<NUM> - Δtk (block <NUM>).

A bit-rate level for a next video segment (Rk+<NUM>) is determined by the ABR controller (block <NUM>). For example, the bit-rate level for the next video segment may be determined using the function Rk+<NUM> = f(Ĉk+<NUM>, Bk+<NUM>) where Rk+<NUM> ∈ <IMG>, Ĉx+<NUM> is the predicted bandwidth for downloading the next video segment, and Bk+<NUM> is the fill level of the playback buffer at the start of downloading the video segment k + <NUM>. The predicted throughput or bandwidth may rely on the previous measures of network bandwidth. For example, C = {Ck-N+<NUM>, Ck-N+<NUM>,. , Ck-<NUM>, Ck} may represent the measured network bandwidth experienced during the download of the last N video segments and C̃ = {C̃k-N+<NUM>, C̃k-N+<NUM>,. , C̃k-<NUM>, C̃k} may represent the estimated network bandwidths determined for last N video segments. The predicted bandwidth C̃k+<NUM> for downloading the next video segment, i.e. video segment k + <NUM>, is a function of C and C̃, i.e. Ĉk+<NUM> = g(C, C̃). For example, the predicted bandwidth Ĉk+<NUM> may be determined by taking the harmonic average of the previously measured bandwidths as laid out in the equations below. <MAT> <MAT> <MAT>.

The subject technology is not limited to algorithms that use both the predicted bandwidth and the fill level. For example, some algorithms may use only the predicted bandwidth to select and set a bit-rate level for the next video segment. Alternatively, other algorithms may use only the fill levels to select and set a bit-rate level for the next video segment.

With the bit-rate level set for the next video segment, k is incremented (k = k + <NUM>) (block <NUM>) and the HTTP engine issues a request to the server for the next video segment (block <NUM>).

The ABR controller may periodically monitor the network bandwidth experienced during the downloading of the video segment and may make a decision to abandon the downloading of the video segment at the current bit-rate level and restart the downloading of the video segment at a new lower bit-rate level if certain conditions are met (block <NUM>). The conditions may include if the elapsed downloading time at the current bit-rate level satisfies a first pre-defined threshold, if the remaining downloading time at the current bit-rate level satisfies another pre-defined threshold, if the bit-rate level estimated based on the network bandwidth experienced so far during the downloading of the video segment is less than the current bit-rate level set for the video segment, and/or an estimated size of the video segment at a lower bit-rate level is smaller than the size of the video segment at the current bit-rate level remaining to be downloaded. The subject technology is not limited to these conditions for the evaluation of whether to abandon the downloading of a video segment and start downloading the video segment at a lower bit-rate level.

If the download of the video segment at the current bit-rate level is abandoned (block <NUM>), an amount of rebuffering time and a number of rebuffering events are determined for the period of time before the downloading was abandoned and the occupancy level of the playback buffer is updated to reflect the amount of video content in the playback buffer that was consumed by the decoder (block <NUM>). These updates may be made based on the following equations: <MAT> <MAT> <MAT> <MAT> where sdk(Rk) is the size of the portion of the video segment that was downloaded at the current bit-rate level before the download was abandoned, SCk is the average network bandwidth (throughput) experienced during the download of the video segment at the current bit-rate level before the download was abandoned, and <MAT>.

With the download of the video segment at the current bit-rate level Rk abandoned, a new bit-rate level R̂k is determined and Rk is set to R̂k for the video segment (block <NUM>). For example, the new bit-rate level R̂k (with R̂k < Rk) may be a function of the downloaded video segment size so far sdk(Rk), the current video segment size dk(Rk), the network bandwidth experienced so far SCk, and the current bit-rate level Rk, namely: <MAT> The subject technology is not limited to any particular function for determining the new bit-rate level R̂k. In addition, the function may be based on fewer factors or more factors than those outlined above. With the new bit-rate level set for the video segment k, the process returns to the HTTP engine issuing a request to the server for the video segment encoded at the new bit-rate level (block <NUM>).

If the download of video segment k is allowed to complete rather than be abandoned (block <NUM>), an amount of rebuffering time and a number of rebuffering events are determined for the period of time the downloading occurred and the occupancy level of the playback buffer is updated to reflect the amount of video content in the playback buffer that was consumed by the decoder during that period of time (block <NUM>). These updates may be made based on the following equations: <MAT> <MAT> <MAT> <MAT> where dk(Rk) be the size of the video segment k encoded at bit-rate level Rk and Ck is the average network bandwidth (throughput) experienced during downloading the video segment k.

The process continues by determining whether any video segments of the video content remain to be downloaded from the server (k < K) (block <NUM>). If video segments remain on the server to be downloaded, the process returns to check the occupancy level of the playback buffer to determine if there is room in the playback buffer to append video segment k (block <NUM>). If all K video segments of the video content V have been downloaded from the server, the video segments remaining in the playback buffer are decoded by the decoder for playback and the playback of the video content ends.

As noted above, rebuffering is an important factor in evaluating the QoE in Internet video applications. <FIG> is a graph illustrating an occurrence of rebuffering during operation of an adaptive video player according to aspects of the subject technology. At time tk, the adaptive video player starts to download chunk (video segment) k. The download time for this chunk will be dk(Rk)/Ck, i.e., it depends on the size of selected chunk with bit-rate Rk, as well as average download bandwidth (throughput) Ck. Once chunk k is completely downloaded, the video player waits for Δtk and starts to download the next chunk k + <NUM> at time tk+<NUM>. The fill level B(t) evolves as the chunks are being downloaded and the video is being played. Specifically, the fill level increases by L seconds after chunk k is downloaded and decreases by dk(Rk)/Ck seconds as the player consumes chunks from the playback buffer. The buffer dynamics can be formulated as follows: <MAT> <MAT> As illustrated in <FIG>, if <MAT>, the playback buffer becomes empty while the adaptive video player is still downloading chunk k, leading to a rebuffering event. The total amount of rebuffering time experienced during the playback of video content V may be defined as: <MAT> Similarly, the total number of rebuffering events experienced during the playback of video content V may be defined as: <MAT>.

One example of a QoE metric that may be used to evaluate the performance of an adaptive video system considers the user experience with both the average video quality over all downloaded chunks and the penalty caused by the average quality variations from one chunk to another, the rebuffering time and the startup delay. The QoE metric is defined as: <MAT> where q(. ) is a nondecreasing function which maps selected bit-rate level Rk to video quality q(Rk) perceived by user, and where α, β, γ are non-negative weighting parameters corresponding to video quality variations, rebuffering time and startup delay, respectively.

By assuming the quality mapping function q(Rk) = Rk and setting α = <NUM>, β = <MAT> and γ = <NUM>, the QoE metric can be simplified as the effective bit-rate (i.e., the average playback bit-rate minus the penalty of rebuffering time and streaming video quality variations). The effective bit-rate for streaming K chunks may then be defined as: <MAT> Here the initial startup delay is not included in the effective bit-rate metric as it is often a fixed amount of time, irrespective of which ABR control algorithm is used.

<FIG> is a block diagram illustrating components of an electronic device according to aspects of the subject technology. Not all of the depicted components may be required, however, and one or more implementations may include additional components not shown in the figure. Depicted or described connections between components are not limited to direct connections and may be implemented with one or more intervening components.

The electronic device depicted in <FIG> is similar to the electronic device described above in connection with <FIG>. For example, both electronic devices include HTTP engine <NUM>, playback buffer <NUM>, throughput predictor <NUM>, ABR controller <NUM>, and decoder <NUM>, the description of which will not be repeated. However, the electronic device depicted in <FIG> has been modified to include insufficient and sufficient buffer controller <NUM> that applies a control algorithm on top of the control algorithm applied by ABR controller <NUM> to set the bit-rate levels for the video segments requested and downloaded from server <NUM> via network <NUM>. The new control algorithm applied by insufficient and sufficient buffer controller <NUM> reduces rebuffering time and events and improves QoE using the fill level of playback buffer <NUM> to identify conditions to replace the bit-rate level set by ABR controller <NUM> with more appropriate bit-rate levels for the corresponding fill level. The process performed by insufficient and sufficient buffer controller <NUM> is described in more detail below in connection with <FIG>.

<FIG> is a flowchart illustrating an example process for setting a bit-rate level for a video segment according to aspects of the subject technology. For explanatory purposes, the blocks of the process illustrated in <FIG> are described herein as occurring in serial, or linearly. However, multiple blocks of the process may occur in parallel. In addition, the blocks of the process need not be performed in the order shown and/or one or more blocks of the process need not be performed and/or can be replaced by other operations.

The process illustrated in <FIG> replaces or supplements the operations described above in connection with block <NUM> of <FIG>. In the process of <FIG>, the ABR controller determines and sets a bit-rate level for the next video segment to be downloaded by the HTTP engine in the manner described above with respect to block <NUM> in <FIG> (block <NUM>). In particular, the bit-rate level for the next video segment may be determined using the function Rk+<NUM> = f (Ĉk+<NUM>, Bk+<NUM>) where Rk+<NUM> ∈ <IMG>, Ĉk+<NUM> is the predicted bandwidth for downloading the next video segment, and Bk+<NUM> is the fill level of the playback buffer at the start of downloading the video segment k + <NUM>.

The fill level Bk+<NUM> is compared against a first threshold, Thlow, representing a relatively low fill level for the playback buffer. If the fill level Bk+<NUM> is less than the first threshold Thlow (block <NUM>), the first bit-rate level for the next video segment set by the ABR controller is replaced with a second bit-rate level that is predetermined (block <NUM>). For example, the ABR controller may write the set first bit-rate level in a control register that is accessible to the HTTP engine. When the fill level satisfies the first threshold, the second bit-rate level may replace the first bit-rate level in the control register. The predetermined second bit-rate level is a relatively low bit-rate level that will allow the next video segment to be downloaded more quickly into the playback buffer and hopefully avoid a rebuffering event. For example, the predetermined second bit-rate level may be the lowest bit-rate level from <IMG>= {r<NUM>, r<NUM>,. , rM} which results in Rk+<NUM> = r<NUM>.

If the fill level Bk+<NUM> is not less than the first threshold Thlow (block <NUM>), the fill level Bk+<NUM> is compared against a second threshold, Thhigh, representing a relatively high fill level for the playback buffer. If the fill level Bk+<NUM> is greater than the second threshold Thhigh (block <NUM>), the first bit-rate level for the next video segment set by the ABR controller is replaced with a third bit-rate level that is predetermined (block <NUM>). The predetermined third bit-rate level is a relatively high bit-rate level that will take advantage of a cushion of playback time provided by the relatively high fill level to increase the bit-rate level for the next video segment and thereby increase the quality of the next video segment during playback. For example, the predetermined third bit-rate level may be the highest bit-rate level from <IMG>= {r<NUM>, r<NUM>,. , rM} which results in Rk+<NUM> = rM.

If the fill level Bk+<NUM> does not satisfy the second threshold (block <NUM>), the bit rate level for the next video segment set by the ABR controller is left in place. In summary, the insufficient and sufficient buffer controller modifies the adaptive video player such that the bit-rate level set for the next video segment Rk+<NUM> is set according to the following criteria: <MAT> The first and second thresholds, Thlow and Thhigh, may be configurable. According to aspects of the subject technology, the two thresholds may be set as follows: <MAT> <MAT>.

The electronic device depicted in <FIG> is similar to the electronic devices described above in connection with <FIG> & <FIG>. For example, all three electronic devices include HTTP engine <NUM>, playback buffer <NUM>, throughput predictor <NUM>, ABR controller <NUM>, and decoder <NUM>, the description of which will not be repeated. In addition, the electronic device depicted in <FIG> includes insufficient and sufficient buffer controller <NUM>, the description of which also will not be repeated. However, the electronic device depicted in <FIG> has been modified to include playback speed controller <NUM>. It is noted that the incorporation of insufficient and sufficient buffer controller <NUM> in the electronic device of <FIG> is optional and playback speed controller <NUM> may be incorporated into the electronic device with or without insufficient and sufficient buffer controller <NUM>.

According to aspects of the subject technology, playback speed controller <NUM> implements a control algorithm that adjusts a playback speed for a video segment based on a fill level of the playback buffer. The control algorithm leverages a property of the human visual system where changes in playback speed of video content within about <NUM>% of the real-time playback speed (above or below) are generally not noticeable to the viewer. For example, if the fill level is less than a pre-defined low threshold, i.e., Thlow, or if the fill level is less than a pre-defined middle threshold, i.e., Thmid, and the previous video segment downloading was abandoned, the decoder is set to a slow-playback mode (e.g., <NUM>% of real-time playback speed) for the video segment. Slowing the playback speed of the decoder for a video segment provides more time to download the next video segment into the playback buffer before playback of the current video segment completes and therefore reduces the chance of a rebuffering event or potentially shortens the duration of a rebuffering event. If the fill level is higher than a pre-defined high threshold, i.e., Thhigh, and the playback lags behind the real-time playback speed due to slow playback mode being previously used, the decoder is switched to a fast-playback mode (e.g., <NUM>% of real-time playback speed) for the video segment to catch up playback speed. Otherwise, the decoder is set to normal real time playback mode. The operation of playback speed controller <NUM> is described in further detail below in connection with <FIG> & <FIG>.

<FIG> is a flowchart illustrating an example process according to the invention using playback speed control for a video segment according to aspects of the subject technology. For explanatory purposes, the blocks of the process illustrated in <FIG> are described herein as occurring in serial, or linearly. However, multiple blocks of the process may occur in parallel. In addition, the blocks of the process need not be performed in the order shown and/or one or more blocks of the process need not be performed and/or can be replaced by other operations.

The process illustrated in <FIG> replaces or supplements the process described above for block <NUM> of <FIG>. As discussed above, this stage of the process follows a decision to abandon the downloading of a video segment in order to start over using a lower bit-rate level for the video segment. Initially, the fill level of the playback buffer is compared against the threshold PThlow (block <NUM>). If the fill level is less than PThlow, playback speed controller <NUM> changes the playback speed set for the video segment by reducing the playback speed by a specified amount (e.g., <NUM>%, <NUM>%, <NUM>%, etc.) changing the playback speed from a first playback speed to a second playback speed (block <NUM>).

If the fill level of the playback buffer does not satisfy the threshold PThlow, the fill level of the playback buffer is compared against the threshold PThmid, which is greater than PThlow (block <NUM>). In addition, the process determines whether downloading of the previous video segment was abandoned and restarted at a lower bit-rate level (block <NUM>). This determination may be made by checking whether a flag in a specified memory location has been set. If the fill level is determined to be less than the threshold PThmid and downloading of the previous video segment was abandoned, playback speed controller <NUM> changes the playback speed for the video segment by reducing the playback speed by a specified amount (e.g., <NUM>%, <NUM>%, <NUM>%, etc.) changing the first playback speed to the second playback speed (block <NUM>). The amount by which the playback speed is reduced may be the same as for the condition where the fill level satisfies the threshold PThlow, or it may be reduced by a different amount.

If the fill level of the playback buffer does not satisfy the threshold PThmid, the fill level of the playback buffer is compared against the threshold PThhigh, which is greater than PThmid (block <NUM>). In addition, the process makes a determination on whether an amount of playback lag time is greater than zero (block <NUM>). The playback lag time tracks the amount of time playback of the current video content lags behind the real-time playback of the video content due to the playback speed for one or more previous video segments being reduced. If the fill level is greater than the threshold PThhigh and the current amount of playback lag time is greater than zero, playback speed controller <NUM> changes the playback speed for the video segment by increasing the playback speed by a specified amount (e.g., <NUM>%, <NUM>%, <NUM>%, etc.) changing the playback speed from the first playback speed to a third playback speed (block <NUM>). If the fill level of the playback buffer is not greater than PThhigh, or there is no playback lag time, no change is made to the playback speed of the video segment.

After reducing the playback speed for the video segment (block <NUM>), increasing the playback speed for the video segment (block <NUM>), or if no change is made to the playback speed the process proceeds to updating the amount of rebuffering time, the number of rebuffering events, the fill level, and the amount of playback lag time for the period during which video segment k started downloading until abandonment (block <NUM>). The manner in which these parameters are updated varies depending on whether the playback speed for the video segment was reduced, increased, or left unchanged.

If the playback speed for the video segment was reduced, the following equations are used to update the parameters: <MAT> <MAT> <MAT> <MAT> <MAT> where δ is the fractional amount by which the playback speed is changed (e.g., <NUM>, <NUM>, <NUM>, etc.) and pt_lag is the amount of playback lag time accumulated during playback of the video content. The playback lag time is initialized to zero at the start of playing back video content, such as the beginning of the process represented in <FIG>. δ may be a configurable value. If the playback speed for the video segment was increased and the playback lag time was greater than zero, the following equations are used to update the parameters: <MAT> <MAT> <MAT> <MAT> <MAT> If no changes were made the playback speed for the video segment, the parameters are updated in the manner described above with respect to block <NUM> in <FIG>. The process then sets the previous video segment abandoned flag to true to reflect the abandonment of downloading the video segment (block <NUM>).

<FIG> is a flowchart illustrating an example process using playback speed control for a video segment according to aspects of the subject technology. For explanatory purposes, the blocks of the process illustrated in <FIG> are described herein as occurring in serial, or linearly. However, multiple blocks of the process may occur in parallel. In addition, the blocks of the process need not be performed in the order shown and/or one or more blocks of the process need not be performed and/or can be replaced by other operations.

The process illustrated in <FIG> replaces or supplements the process described above for block <NUM> of <FIG>. As discussed above, this stage of the process follows a decision not to abandon downloading of the video segment k. The portion of the process that adjusts the playback speed for the video segment k based on the fill level of the playback buffer (blocks <NUM>, <NUM>, <NUM>, <NUM>, and <NUM>) follows the same corresponding portion of the process described above with respect to <FIG> (blocks <NUM>, <NUM>, <NUM>, <NUM>, and <NUM>) and the description of this portion of the process will not be repeated.

After reducing the playback speed for the video segment (block <NUM>), increasing the playback speed for the video segment (block <NUM>), or if no change is made to the playback speed the process proceeds to updating the amount of rebuffering time, the number of rebuffering events, the fill level, and the amount of playback lag time after video segment k has completed downloading (block <NUM>). Similar to what was discussed above with respect to <FIG>, the manner in which these parameters are updated varies depending on whether the playback speed for the video segment was reduced, increased, or left unchanged.

If the playback speed for the video segment was reduced, the following equations are used to update the parameters: <MAT> <MAT> <MAT> <MAT> <MAT> If the playback speed for the video segment was increased and the playback lag time was greater than zero, the following equations are used to update the parameters: <MAT> <MAT> <MAT> <MAT> <MAT> If no changes were made the playback speed for the video segment, the parameters are updated in the manner described above with respect to block <NUM> in <FIG>. The process then sets the previous video segment abandoned flag to false to reflect the completion of downloading video segment k (block <NUM>).

The thresholds PThlow, PThmid, and PThhigh may be configurable. For example, these thresholds may be set at follows: <MAT> <MAT> <MAT> While the values for PThlow and PThhigh indicated above are the same as the values for Thlow and Thhigh used for the implementations described above in connection with <FIG>, the subject technology is not limited to using the same threshold values for these different control algorithms and may implement the different control algorithms with different threshold values.

Changing the playback speeds used by the decoder in decoding video segments from the playback buffer may rely on trick modes to either reduce the playback speed or increase the playback speed while matching the target display frame-rate. For example, frame-rate conversion (FRC) may be turned on in the decoder when the playback speed is reduced to insert additional frames between the existing frames in the video segment. Similarly, the decoder may periodically drop one or more existing frames from the video segment during playback when the playback speed is increased. In addition, audio/speech pitch correction may be applied to audio portions of the video segment to match the expected sound pitch experienced during unchanged playback speeds.

<FIG> conceptually illustrates an electronic system <NUM> with which one or more implementations of the subject technology may be implemented. The electronic system <NUM>, for example, can be a network device, a media converter, a desktop computer, a laptop computer, a tablet computer, a server, a phone, or generally any electronic device that is capable of communicating signals over a network and implementing an adaptive video player in the manner described above. Such an electronic system <NUM> includes various types of computer readable media and interfaces for various other types of computer readable media. In one or more implementations, the electronic system <NUM> is, or includes, one or more of server <NUM> and/or electronic device <NUM>. The electronic system <NUM> includes a bus <NUM>, one or more processing unit(s) <NUM>, a system memory <NUM>, a read-only memory (ROM) <NUM>, a permanent storage device <NUM>, an input device interface <NUM>, an output device interface <NUM>, and a network interface <NUM>, or subsets and variations thereof.

The one or more processing unit(s) <NUM> can be a single processor or a multicore processor in different implementations.

The ROM <NUM> stores static data and instructions that are needed by the one or more processing unit(s) <NUM> and other modules of the electronic system. The permanent storage device <NUM>, on the other hand, is a read-and-write memory device. The permanent storage device <NUM> is a non-volatile memory unit that stores instructions and data even when the electronic system <NUM> is off. One or more implementations of the subject disclosure use a mass-storage device (such as a solid-state drive, or a magnetic or optical disk and its corresponding disk drive) as the permanent storage device <NUM>.

Other implementations use a removable storage device (such as a flash memory drive, optical disk and its corresponding disk drive, external magnetic hard drive, etc.) as the permanent storage device <NUM>. Like the permanent storage device <NUM>, the system memory <NUM> is a read-and-write memory device. However, unlike the permanent storage device <NUM>, the system memory <NUM> is a volatile read-and-write memory, such as random access memory. System memory <NUM> stores any of the instructions and data that the one or more processing unit(s) <NUM> needs at runtime.

The bus <NUM> also connects to the input device interface <NUM> and the output device interface <NUM>. The input device interface <NUM> enables a user to communicate information and select commands to the electronic system. Input devices used with the input device interface <NUM> include, for example, alphanumeric keyboards and pointing devices (also called "cursor control devices"). The output device interface <NUM> enables, for example, the display of images generated by the electronic system <NUM>. Output devices used with the output device interface <NUM> include, for example, printers and display devices, such as a liquid crystal display (LCD), a light emitting diode (LED) display, an organic light emitting diode (OLED) display, a flexible display, a flat panel display, a solid state display, a projector, or any other device for outputting information. One or more implementations include devices that function as both input and output devices, such as a touchscreen.

Finally, as shown in <FIG>, the bus <NUM> also couples the electronic system <NUM> to one or more networks (not shown) through one or more network interfaces <NUM>. In this manner, the computer can be a part of one or more network of computers (such as a local area network (LAN), a wide area network (WAN), or an Intranet, or a network of networks, such as the Internet). Any or all components of the electronic system <NUM> can be used in conjunction with the subject disclosure.

In some implementations, the tangible computer-readable storage medium can be directly coupled to a computing device, while in other implementations, the tangible computer-readable storage medium can be indirectly coupled to a computing device, e.g., via one or more wired connections, one or more wireless connections, or any combination thereof.

Claim 1:
A method comprising:
issuing a first request to a server (<NUM>, <NUM>) for a next video segment encoded at a first bit-rate level;
initiating downloading of the next video segment encoded at the first bit-rate level from the server (<NUM>, <NUM>) and storing the next video segment in a playback buffer (<NUM>);
comparing (<NUM>, <NUM>) a fill level of the playback buffer (<NUM>) to a first threshold (PThlow) and to a second threshold (PThhigh);
determining (<NUM>) if a current playback lag time is greater than zero;
if the fill level of the playback buffer (<NUM>) satisfies the first threshold (PThlow), changing (<NUM>) a first playback speed set for the next video segment to a second playback speed, or if the fill level of the playback buffer (<NUM>) satisfies the second threshold (PThhigh) and if the current playback lag time is greater than zero, changing (<NUM>) the first playback speed set for the next video segment to a third playback speed; and
decoding the next video segment from the playback buffer (<NUM>) for playback at the first playback speed or, if the fill level of the playback buffer satisfies the first threshold (PThlow), at the second playback speed, on a display device (<NUM>) after the next video segment has been downloaded and stored in the playback buffer (<NUM>),
wherein the second threshold (PThhigh) is greater than the first threshold (PThlow) and the third playback speed is greater than the second playback speed, the method further comprising:
comparing (<NUM>) the fill level of the playback buffer (<NUM>) to a third threshold (PThmid), wherein the third threshold (PThmid) is greater than the first threshold (PThlow) and less than the second threshold (PThhigh);
determining (<NUM>) if downloading a previous video segment was stopped; and
if the fill level of the playback buffer (<NUM>) satisfies the third threshold (PThmid) and downloading the previous video segment was stopped, changing (<NUM>) the first playback speed for the next video segment to the second playback speed.