Intrasegment adjustment of video transmission rate

A method includes determining a first transmission rate for transmitting an entirety of a segment of a video stream to a client device. The segment includes a set of frames. In some implementations, the method includes, after transmitting a first subset of the set of frames at the first transmission rate, detecting that a network connectivity of the client device has reduced below a connectivity threshold associated with the first transmission rate. In some implementations, the method includes transmitting an entirety of the segment at a second transmission rate that is less than the first transmission rate. In some implementations, the method includes triggering the client device to present a second subset of the set of frames that corresponds to a temporal position in the video stream after the first subset of the set of frames while foregoing re-presentation of the first subset of the set of frames.

TECHNICAL FIELD

The present disclosure generally relates to intrasegment adjustment of video transmission rate.

BACKGROUND

Some devices are capable of receiving video content at various transmission rates. After a device starts receiving video content at a particular transmission rate, there may be a degradation in a network being used to transmit the video content. The degradation may require a change in the transmission rate.

DESCRIPTION OF EXAMPLE EMBODIMENTS

OVERVIEW

Various implementations disclosed herein include devices, systems, and methods for adjusting a transmission rate of a video stream while a segment is being transmitted. In some implementations, a method includes determining a first transmission rate for transmitting an entirety of a segment of a video stream to a client device. The segment includes a set of frames. In some implementations, the method includes, after transmitting a first subset of the set of frames at the first transmission rate, detecting that a network connectivity of the client device has reduced below a connectivity threshold associated with the first transmission rate. In some implementations, the method includes transmitting an entirety of the segment at a second transmission rate that is less than the first transmission rate. In some implementations, the method includes triggering the client device to present a second subset of the set of frames that corresponds to a temporal position in the video stream after the first subset of the set of frames while foregoing re-presentation of the first subset of the set of frames.

In accordance with some implementations, a device includes one or more processors, a non-transitory memory, and one or more programs. In some implementations, the one or more programs are stored in the non-transitory memory and are executed by the one or more processors. In some implementations, the one or more programs include instructions for performing or causing performance of any of the methods described herein. In accordance with some implementations, a non-transitory computer readable storage medium has stored therein instructions that, when executed by one or more processors of a device, cause the device to perform or cause performance of any of the methods described herein. In accordance with some implementations, a device includes one or more processors, a non-transitory memory, and means for performing or causing performance of any of the methods described herein.

EXAMPLE EMBODIMENTS

A video stream that is being transmitted to a client device includes various segments. Each segment includes numerous frames. While the video stream is being transmitted, a transmission rate of the video stream can be changed. However, the change goes into effect after a current segment finishes to play at the client device. If the transmission rate deteriorates significantly while the current segment is being played at the client device, the client device may drop packets corresponding to a remainder of the current segment or not receive the packets fast enough thereby leading to a loss of video at the client device and a detracted user experience.

The present disclosure provides methods, systems and/or devices for changing a transmission rate of a video stream while a current segment is being transmitted and using the new transmission rate to present a remainder of the current segment at the client device. Since the new transmission rate is applied to the current segment, frames in the current segment are not dropped due to a network degradation. By contrast, if the new transmission rate is not applied to the current segment, some frames in the current segment may be dropped if the network degrades.

In operation, an edge device adjusts a transmission rate of a video stream being transmitted to a client device. The edge device detects that network conditions have deteriorated while frames of a higher bitrate version of a segment are being transmitted to the client device. Since the edge device cannot change the transmission rate for transmitting the remainder of the frames in the higher bitrate version of the segment, the edge device transmits a lower bitrate version of the entire segment at a lower transmission rate. Since the client device has already presented a first part of the higher bitrate version of the segment, the edge device instructs the client device to decode a corresponding first part of the lower bitrate version of the segment at a faster decode rate (e.g., at a second decoder) and to skip presentation of the first part of the lower bitrate version of the segment. Since the client device did not receive a second part of the higher bitrate version of the segment, the client device has not presented the second part of the higher bitrate version of the segment. As such, the edge device instructs the client device to start presenting a corresponding second part of the lower bitrate version of the segment.

FIG.1Ais a diagram that illustrates an example operating environment10in accordance with some implementations. While pertinent features are shown, those of ordinary skill in the art will appreciate from the present disclosure that various other features have not been illustrated for the sake of brevity and so as not to obscure more pertinent aspects of the example implementations disclosed herein. To that end, as a non-limiting example, the operating environment10includes a client device20, an edge device40, and a content store110. In some implementations, the edge device40is a part of an edge computing network (not shown). Although the content store110is shown as being separate from the edge device40, in some implementations, the content store110is integrated into the edge device40. In some implementations, the content store110includes a distributed storage system. In some implementations, the client device20is coupled with a display. For example, in some implementations, the client device20includes a dongle that is connected to a television. Alternatively, in some implementations, the client device20is integrated into a display (e.g., a television). As shown inFIG.1A, the client device20includes a buffer30.

The content store110stores a set of one or more media content items120. Each media content item120includes a set of content frames. The content store110can store different versions of a media content item120. For example, the content store110can store a high bit rate version of the media content item120and a low bit rate version of the media content item120. In the example ofFIG.1A, the content store110stores higher bit rate content frames122a,122b, . . . , and122n, and lower bit rate content frames122a′,122b′, . . . , and122n′ for the same media content item120. The edge device40retrieves the higher bit rate content frames122a,122b, . . . , and122nfrom the content store110when the edge device40is transmitting the media content item120to the client device20at a first transmission rate130. By contrast, the edge device40retrieves the lower bit rate content frames122a′,122b′, . . . , and122n′ from the content store110when the edge device40is transmitting the media content item120to the client device20at a transmission rate that is lower than the first transmission rate130.

As shown inFIG.1A, a group of content frames can form a segment.FIG.1Aillustrates a first higher bit rate segment124a, a second higher bit rate segment124b, . . . , and an nth higher bit rate segment124n.FIG.1Aalso illustrates a first lower bit rate segment124a′, a second lower bit rate segment124b′, . . . , and an nth lower bit rate segment124n′. In the example ofFIG.1A, the client device20is currently presenting the first higher bit rate segment124a. Specifically, the client device20is presenting a first higher bit rate content frame122aon a display. The buffer30is storing a second higher bit rate content frame122bthat will be presented after the first higher bit rate content frame122a. A third higher bit rate content frame122cis being transmitted to the client device20at the first transmission rate130.

Referring toFIG.1B, the transmission of the third higher bit rate content frame122cfails or is delayed, for example, because a connection between the edge device40and the client device20degrades. The transmission of the third higher bit rate content frame122cmay have failed because a bandwidth of the client device20may have dropped to a value that does not support the first transmission rate130. The transmission of the third higher bit rate content frame122cmay have failed because of network congestion (e.g., too many other client devices may be requesting content from the edge device40). As shown inFIG.1B, the client device20is now presenting the second higher bit rate content frame122bthat was stored in the buffer30. However, the buffer30does not store the third higher bit rate content frame122cbecause its transmission failed. In some examples, the buffer30may store a set of frames that are to be presented next in a sequential manner, and the edge device40may detect that a subsequent frame that is not stored in the buffer30may not reach the client device20in time due to network degradation. For example, the buffer30may store the next ten frames and the edge device40may detect that an eleventh frame may not reach the client device20in time due to network degradation.

Referring toFIG.1C, if the network degradation persists, then re-transmitting the third higher bit rate content frame122cat the first transmission rate130will likely fail. Hence, instead of re-transmitting the third higher bit rate content frame122cat the first transmission rate130, the edge device40transmits a lower bit rate version of the first higher bit rate segment124aat a lower transmission rate. As shown inFIG.1C, the edge device40transmits the first lower bit rate segment124a′ at the second transmission rate132that is lower than the first transmission rate130. Specifically,FIG.1Cshows the first lower bit rate content frame122a′ being transmitted to the client device20and being stored in the buffer30. Since the first higher bit rate content frame122ahas already been presented at the client device20, the edge device40sends a skip presentation instruction150to the client device20that instructs the client device20to skip presentation of the first lower bit rate content frame122a′. In some implementations, the client device20skips presentation of the first lower bit rate content frame122a′ in response to determining that the client device20has already presented the corresponding higher bit rate content frame122a. The edge device40also sends an accelerate decode instruction152to the client device20instructing the client device20to decode the first lower bit rate content frame122a′ at an accelerated decoding rate (e.g., at a multiple of a default decoding rate, for example, at 1.25× the default decoding rate, at 5× the default decoding rate, at 10× the default decoding rate, at 35× the default decoding rate, etc.). In some implementations, the client device20determines to decode the first lower bit rate content frame122a′ at the accelerated decoding rate. The client device20can apply the accelerated decoding rate and skip presentation of the first lower bit rate content frame122a′ based on a timestamp associated with the first lower bit rate content frame122a′. For example, the client device20may maintain a pointer that points to a timestamp of a current playback position, and the client device20can skip presentation of the first lower bit rate content frame122a′ if its corresponding timestamp is smaller than the timestamp of the current playback position.

In operation, the client device20receives the first lower bit rate content frame122a′, and the client device20decodes the first lower bit rate content frame122a′ at an accelerated decoding rate. However, the client device20does not present the first lower bit rate content frame122a′ because the client device20has already presented the first higher bit rate content frame122a. Referring toFIG.1D, the edge device40transmits the second lower bit rate content frame122b′ at the second transmission rate132. As shown inFIG.1D, the edge device40can include the skip presentation instruction150and the accelerate decode instruction152with the second lower bit rate content frame122b′. The client device20receives the second lower bit rate content frame122b′, and the client device20decodes the second lower bit rate content frame122b′ at the accelerated decoding rate. However, the client device20does not present the second lower bit rate content frame122b′ because the client device20has already presented the second higher bit rate content frame122b.

Referring toFIG.1E, the edge device40transmits the third lower bit rate content frame122c′ at the second transmission rate132. Since the third higher bit rate content frame122cwas not presented at the client device20, the edge device40sends a presentation instruction160that instructs the client device20to present the third lower bit rate content frame122c′. The edge device40may also send the accelerate decode instruction152instructing the client device20to decode the third lower bit rate content frame122c′ at the accelerated decoding rate. As shown inFIG.1E, the client device20decodes the third lower bit rate content frame122c′ and presents the third lower bit rate content frame122c′. In some implementations, the client device20determines to present the third lower bit rate content frame122c′ based on a timestamp associated with the third lower bit rate content frame122c′. For example, the client device20can present the third lower bit rate content frame122c′ if its corresponding timestamp matches the timestamp of the current playback position.

Transmitting a lower bit rate version of the first higher bit rate segment124aallows the edge device40to lower the transmission rate while the first higher bit rate segment124ais being presented at the client device20. Lowering the transmission rate while the client device20is presenting the first higher bit rate segment124aprovides an appearance that the client device20successfully presented the first higher bit rate segment124a. Since the user of the client device20probably cannot differentiate between the first higher bit rate segment124aand the first lower bit rate segment124a′, starting with the first higher bit rate segment124aand finishing with the first lower bit rate segment124a′ provides a perception that the client device20successfully received and presented the first higher bit rate segment124a. In other words, starting with the first higher bit rate segment124aand finishing with the first lower bit rate segment124a′ ensures that there is no interruption in the presentation of the media content item120at the client device20due to network degradation.

FIG.1Fillustrates an alternative to the implementation illustrated inFIGS.1C-1E. Referring toFIG.1F, in some implementations, an encoder170generates a lower bit rate version of a content frame based on a higher bit rate version of a reference frame (e.g., an I-frame). In the example ofFIG.1F, the encoder170can generate the third lower bit rate content frame122c′ based on the first higher bit rate content frame122a. The edge device40transmits the third lower bit rate content frame122c′ that gets stored in the buffer30and is presented by the client device20after the second higher bit rate content frame122b. In this implementation, the content store110may not have to store lower bit rate content frames because the encoder170can generate the lower bit rate content frames based on a higher bit rate content frame. Hence, the content store110uses less memory. Furthermore, the edge device40does not have to send lower bit rate versions of content frames that have already been presented. For example, the edge device40does not have to send the first lower bit rate content frame122a′ and the second lower bit rate content frame122b′ (shown inFIGS.1C and1D). Not having to send the lower bit rate versions of content frames that have already been presented reduces congestion on a network that may already be congested. In this implementation, the client device20does not have to decode lower bit rate versions of content frames that the client device20has already presented.

FIG.2is a flowchart representation of a method200for changing a transmission rate of a video stream. In various implementations, the method200is performed by a device (e.g., by the edge device40shown inFIGS.1A-1F). In some implementations, the method200is performed by processing logic, including hardware, firmware, software, or a combination thereof. In some implementations, the method200is performed by a processor executing code stored in a non-transitory computer-readable medium (e.g., a memory).

As represented by block210, in various implementations, the method200includes determining a first transmission rate for transmitting an entirety of a segment of a video stream to a client device. For example, as shown inFIG.1A, the edge device40is transmitting the third higher bit rate content frame122cat the first transmission rate130. The segment includes a set of frames. For example, as shown inFIG.1A, the first higher bit rate segment124aincludes the higher bit rate content frames122a,122b,122c,122dand122e. The first transmission rate can be selected based on network status data that indicates a status of the network. For example, the first transmission rate can be selected based on a congestion level of the network. The first transmission rate can also be selected based on a bandwidth of the client device20. For example, the higher the bandwidth, the greater the first transmission rate.

As represented by block220, in some implementations, the method200includes after transmitting a first subset of the set of frames at the first transmission rate, detecting that a network connectivity of the client device has reduced below a connectivity threshold associated with the first transmission rate. For example, as described in relation toFIG.1B, a communication channel between the edge device40and the client device20may degrade due to network congestion and/or due to a sudden decrease in the bandwidth of the client device20.

As represented by block220a, in some implementations, detecting that the network connectivity has reduced includes detecting that an amount of bandwidth available to the client device has reduced to a value less than a threshold amount of bandwidth to support the first transmission rate. For example, the first transmission rate may require a certain amount of client bandwidth, and the client bandwidth may have dropped to a value that is less than the required bandwidth. As such, sending frames at the first transmission rate may result in the frames being dropped and not being received by the client device.

As represented by block220b, in some implementations, the method200includes periodically polling the client device for status data that indicates the network connectivity of the client device. For example, the edge device40periodically obtains a report from the client device20. The report can indicate network status data that the client device20is collecting. For example, the report can indicate a current bandwidth of the client device20. The edge device40can also generate the status data on its own. For example, the edge device40can monitor a communication channel between the edge device40and the client device20to determine a health (e.g., bandwidth, congestion, etc.) of the communication channel. In some implementations, the status data indicates a number of frames that the client device20successfully received. The status data can indicate a timestamp of the last frame that the client device20successfully received.

As represented by block230, in various implementations, the method200includes transmitting an entirety of the segment at a second transmission rate that is less than the first transmission rate. For example, as shown inFIG.1C, the edge device40starts transmitting the first lower bit rate segment124a′ from the beginning at the second transmission rate132. As represented by block230a, in some implementations, the method200includes selecting the second transmission rate based on the network connectivity of the client device. For example, the second transmission rate can be proportional to the client bandwidth. As another example, the second transmission rate can be inversely proportional to the network congestion level.

As represented by block230b, in some implementations, the method200includes transmitting subsequent segments of the video stream at the second transmission rate until the network connectivity satisfies the connectivity threshold associated with the first transmission rate. For example, referring toFIG.1E, the edge device40can send the second lower bit rate segment124b′ at the second transmission rate132instead of sending the second higher bit rate segment124bat the first transmission rate130if the network conditions do not improve. In some implementations, the method200includes switching from the second transmission rate to the first transmission rate when the network connectivity satisfies the connectivity threshold associated with the first transmission rate. For example, if the network conditions improve, the edge device40can send the second higher bit rate segment124bat the first transmission rate130instead of sending the second lower bit rate segment124b′ at the second transmission rate132.

As represented by block230c, in some implementations, the video stream is associated with a first version that is adapted for the first transmission rate and a second version that is adapted for the second transmission rate. For example, as shown inFIGS.1A-1E, the media content item120includes higher bit rate content frames and lower bit rate content frames. In some implementations, transmitting the first subset at the first transmission rate includes transmitting the first version that is adapted for the first transmission rate. For example, as described in relation toFIGS.1A and1B, the edge device40successfully sent the first higher bit rate content frame122aand the second higher bit rate content frame122bto the client device20. In some implementations, transmitting the entirety of the segment at the second transmission rate includes transmitting the second version that is adapted for the second transmission rate. For example, as described in relation toFIGS.1C-1D, the edge device40transmits the first lower bit rate segment124a′ at the second transmission rate132when sending the first higher bit rate segment124aat the first transmission rate130is not feasible.

In some implementations, transmitting the entirety of the segment at the second transmission rate includes generating a lower bit rate version of the second subset of the set of frames based on a higher bit rate version of a reference frame. In some implementations, the method200includes transmitting the lower bit rate version of the second subset of the set of frames while foregoing transmission of a lower bit rate version of the first subset of the set of frames. For example, as shown inFIG.1F, the edge device40transmits the lower bit rate content frame122c′ to the client device20without transmitting the lower bit rate content frames122a′ and122b′ shown inFIG.1E. In some implementations, the method200includes transmitting lower bit rate versions of a reference frame and the second subset of the set of frames while foregoing transmission of a lower bit rate version of the first subset of the set of frames.

As represented by block240, in some implementations, the method200includes triggering the client device to present a second subset of the set of frames that corresponds to a temporal position in the video stream after the first subset of the set of frames while foregoing re-presentation of the first subset of the set of frames. For example, as shown inFIGS.1C-1E, the edge device40instructs the client device20to skip presentation of the first lower bit rate content frame122a′ and the second lower bit rate content frame122b′ because the client device20has already presented the first higher bit rate content frame122aand the second higher bit rate content frame122b.

As represented by block240a, in some implementations, triggering the client device to present the second subset of the set of frames includes instructing the client device to decode the first subset of the set of frames at an accelerated decode rate that is greater than a default decode rate, and forgo re-presentation of the first subset of the set of frames. For example, as shown inFIG.1C, the edge device40sends the accelerate decode instruction152to the client device20instructing the client device20to decode the first lower bit rate content frame122a′ at an accelerated decoding rate. Moreover, as shown inFIG.1C, the edge device20sends the skip presentation instruction150to the client device20instructing the client device20to skip presentation of the first lower bit rate content frame122a′ because the client device20has already presented the first higher bit rate content frame122a.

In some implementations, transmitting the entirety of the segment at the second transmission rate includes including, in each frame of the first subset, an instruction to decode the frame at the accelerated decode rate and to forgo re-presentation of the frame. For example, as shown inFIG.1C, the edge device40sends the skip presentation instruction150and the accelerate decode instruction152to the client device20. The skip presentation instruction150and the accelerate decode instruction152can be integrated into the first lower bit rate content frame122a′ (e.g., into the metadata of the first lower bit rate content frame122a′).

In some implementations, the accelerated decode rate is a multiple of the default decode rate. In some implementations, the edge device40specifies the accelerated decoding rate. Referring toFIG.1C, the accelerate decode instruction152can specify the accelerated decoding rate. Alternatively, the client device20can determine the accelerated decode rate. For example, the client device20can decode the frames as fast as possible.

In some implementations, triggering the client device to present the second subset includes instructing the client device to decode frames in the second subset at the default decoding rate. For example, referring toFIG.1E, the edge device40can instruct the client device20to decode the fourth lower bit rate content frame122d′ at the default decoding rate instead of the accelerated decoding rate.

FIG.3is a block diagram of a device300in accordance with some implementations. In some implementations, the device300implements the edge device40shown inFIGS.1A-1F. While certain specific features are illustrated, those of ordinary skill in the art will appreciate from the present disclosure that various other features have not been illustrated for the sake of brevity, and so as not to obscure more pertinent aspects of the implementations disclosed herein. To that end, as a non-limiting example, in some implementations the device300includes one or more processing units (CPUs)301, a network interface302, a programming interface303, a memory304, one or more input/output (I/O) devices310, and one or more communication buses305for interconnecting these and various other components.

In some implementations, the network interface302is provided to, among other uses, establish and maintain a metadata tunnel between a cloud hosted network management system and at least one private network including one or more compliant devices. In some implementations, the one or more communication buses305include circuitry that interconnects and controls communications between system components. The memory304includes high-speed random access memory, such as DRAM, SRAM, DDR RAM or other random access solid state memory devices, and may include non-volatile memory, such as one or more magnetic disk storage devices, optical disk storage devices, flash memory devices, or other non-volatile solid state storage devices. The memory304optionally includes one or more storage devices remotely located from the one or more CPUs301. The memory304comprises a non-transitory computer readable storage medium.

In some implementations, the memory304or the non-transitory computer readable storage medium of the memory304stores the following programs, modules and data structures, or a subset thereof including an optional operating system306, a data obtainer320, a transmission rate determiner330and a data transmitter340. In various implementations, the device300performs the method200shown inFIG.2.

In some implementations, the data obtainer320includes instructions320a, and heuristics and metadata320bfor obtaining network status data. For example, the data obtainer320can periodically poll the client device for a bandwidth value. The data obtainer320can analyze a communication channel between the device300and the client device to generate the network status data. In some implementations, the data obtainer320performs at least some of the operation(s) represented by block220inFIG.2.

In some implementations, the transmission rate determiner330includes instructions330a, and heuristics and metadata330bfor determining a transmission rate for transmitting content frames to the client device. The transmission rate determiner330can determine the transmission rate based on the network status data (e.g., the client bandwidth) obtained by the data obtainer320. For example, the transmission rate determiner330can reduce the transmission rate when the client bandwidth reduces and/or when the network congestion increases. As another example, the transmission rate determiner330can increase the transmission rate when the client bandwidth increases and/or when the network congestion decreases. In some implementations, the transmission rate determiner330performs at least some of the operation(s) represented by blocks210and230inFIG.2.

In some implementations, the data transmitter340transmits content frames to the client device at the transmission rate determined by the transmission rate determiner330. In some implementations, the data transmitter340selects different versions of the video content based on the transmission rate determined by the transmission rate determiner330. For example, the data transmitter340sends a lower bit rate version of the video content when the transmission rate determiner330reduces the transmission rate. In some implementations, the data transmitter340performs at least some of the operation(s) represented by block240inFIG.2.

In some implementations, the one or more110devices310include a receiver for receiving network status data and a transmitter for transmitting content frame data.

It will be appreciated thatFIG.3is intended as a functional description of the various features which may be present in a particular implementation as opposed to a structural schematic of the implementations described herein. As recognized by those of ordinary skill in the art, items shown separately could be combined and some items could be separated. For example, some functional blocks shown separately inFIG.3could be implemented as a single block, and the various functions of single functional blocks could be implemented by one or more functional blocks in various implementations. The actual number of blocks and the division of particular functions and how features are allocated among them will vary from one implementation to another and, in some implementations, depends in part on the particular combination of hardware, software, and/or firmware chosen for a particular implementation.