Audio packet loss concealment via packet replication at decoder input

A system includes a server to generate a real-time stream of audio packets and a client device to decode and playback the audio content of the stream. The client device includes a network interface configured to receive a stream of audio packets via a network and a buffer configured to temporarily buffer a subset of audio packets of the stream. The client device further includes an audio decoder having an input to receive audio packets from the buffer and an output to provide corresponding segments of a decoded audio data stream. The client device also includes a stream monitoring module configured to provide an audio packet of the subset in the buffer which was previously decoded by the decoder to the input of the decoder again for a repeated decoding in place of a decoding of an audio packet that is lost or late.

BACKGROUND

Real-time media streaming services often are susceptible to network issues, such as packet loss or otherwise delayed packets. For audio streams, the loss or late arrival of an audio packet can be problematic because the audio decoder at the receiving client device typically cannot simply pause audio playback as this will degrade the audio quality and thus impact the quality of service. To mitigate the impact of late/lost audio packets, many systems employ a conventional packet loss concealment (PLC) technique, such as forward error correction (FEC), silence insertion, interpolative analysis, or post-decoding segment replication.

Typically, FEC-based PLC techniques rely on the transmission of an increased number of at least partially redundant packets, with this redundancy allowing the client device to reconstruct a lost or late audio packet. However, this redundancy consumes extra bandwidth and thus often reduces the overall bitrate of the audio stream. Moreover, this approach is limited to only those client devices that have built-in FEC capabilities. Silence insertion techniques, in effect, provide either zero output or a default defined output for a time slot corresponding to a lost/late audio packet. While relatively simple to implement, this default insertion often introduces perceptible audio artifacts. In contrast, interpolative analysis-based techniques, which analyze the prior audio signal and attempt to reconstruct the content of the lost/late packet through signal estimation, can provide improved audio signal replication and thus fewer perceptible audio artifacts. Post-decoding replication techniques rely on replication of a slice of the audio output of the audio decoder to compensate for a late/lost audio packet, along with additional processing to analyze the reconstructed audio signal and effectively “stitch” it together in a manner that reduces audio artifacts. However, while post-decoding replication techniques and interpolative analysis techniques may provide improved signal fidelity, these techniques often require a complexity and compute resource capability that is impracticable for many client devices.

SUMMARY OF EMBODIMENTS

In a first embodiment an electronic device includes a network interface configured to receive a stream of audio packets via a network, a buffer configured to temporarily buffer a subset of audio packets of the stream, and an audio decoder having an input to receive audio packets from the buffer and an output to provide corresponding segments of a decoded audio data stream (i.e., segments whose decoded audio data respectively corresponds to an audio packet from the buffer). The electronic device further includes a stream monitoring module configured to provide an audio packet of the subset in the buffer that was previously decoded by the decoder to the input of the decoder again for a repeated decoding in place of a decoding of an audio packet that is lost or late. The stream monitoring module can be configured to provide a packet loss concealment (PLC) process by providing the audio packet which was previously decoded by the decoder to the input of the decoder again for a repeated decoding, further responsive to determining that a number of times that the audio packet has been decoded in a row has not exceeded a specified threshold, and further may be configured to trigger a fault responsive to the number of times that an audio packet has been decoded in a row has exceeded the specified threshold. The stream monitoring module also can be further configured to implement an alternative packet loss concealment (PLC) process for compensating for an audio packet that is lost or late responsive to the number of times that an audio packet has been decoded in a row has exceeded the specified threshold.

Triggering a fault may included triggering generation of a fault signal, e.g., a fault signal resulting in a stop of playback of the audio stream (and, in an exemplary embodiment, also of a corresponding video stream if there is one) and/or in initiating a system performance check.

A proposed electronic device may thus include a network interface to receive the audio stream from a network, a buffer, e.g., a jitter buffer, to temporarily buffer a subset of the audio packets as they are received at the electronic device, and an audio decoder to decode audio packets from the buffer to generate corresponding segments of a decoded audio signal. The decoded audio signal can then be mixed with other audio sources, repacketized and further transmitted, or otherwise further processed to ultimately generate one or more analog audio, signals used to drive one or more corresponding speakers. In at least one embodiment, the electronic device may employ the proposed packet loss concealment (PLC) technique based on one or more repeat decodings of a previously received audio packet to compensate for one or more subsequent audio packets that have been lost or excessively delayed (that is, “late”) by the network. To this end, the electronic device may employ the stream monitoring module to monitor the received audio stream to detect when an audio packet of the stream is, late or lost. In response to detecting such a late or lost audio packet, the stream monitoring module provides an audio packet of the subset in the buffer which was previously decoded by the decoder for a previous time slot to the input of the decoder again for a repeated decoding for the current time slot in place of a decoding of the audio packet that is lost or late and which was intended for the current time slot. By repeating the decoding of a previously-decoded audio packet to fill in the gap caused by the late/lost packet, this PLC technique leverages the decoder's audio synthesis (by following a typical decoding processing path) to facilitate the continuity of the decoded audio signal during packet decoding replication, and thus reducing or eliminating the distortion or artifacts that otherwise would occur in the absence of this continuity.

In an example embodiment, the stream monitoring module can be configured to, for a first audio packet timely received at the electronic device, provide the first audio packet to the audio decoder for decoding into a corresponding first segment of a decoded audio signal for a first time slot; for a second audio packet that is late or lost, provide the first audio packet to the audio decoder for decoding into a corresponding second segment of the decoded audio signal for a second time slot; and wherein the second segment has continuity with the first segment. The stream monitoring module also can be further configured to, for a third audio packet that is late or lost, provide the first audio packet to the audio decoder for decoding into a corresponding third segment of the decoded audio signal for a third time slot; and wherein the third segment has continuity with the second segment.

In any of the above instances, the stream of audio packets can be generated from audio content generated by a video game application executed at a server connected to the electronic device via the network.

Other embodiments may include a method of operating the electronic device of any of the instances described above, as well as a system including a server to generate the stream of audio packets and the electronic device of any of the instances described above.

In another embodiment, a proposed computer-implemented method includes receiving a stream of audio packets from a network, temporarily buffering a subset of the audio packets, decoding, at an audio, decoder, a first audio packet of the subset to generate a first segment of a decoded audio data stream, and responsive to detecting that a second audio packet following the first audio packet in the stream is lost or late, decoding, at the audio decoder, the first audio packet again to generate a second segment of the decoded audio signal, the second segment following the first segment in the decoded audio signal. The method also can include, responsive to detecting that a third audio packet following the second audio packet in the stream is lost or late, decoding, at the audio decoder, the first audio packet a third time to generate a third segment of the decoded audio signal, the third segment following the second segment in the decoded audio signal. Decoding the first audio packet again can be further responsive to determining that a number of times that the first audio packet has been decoded in a row has not exceeded a specified threshold. The method further can include triggering a fault responsive to the number of times that the first audio packet has been decoded in a row has exceeded the specified threshold. The method also can further include implementing an alternative packet loss concealment process for compensating the second audio packet being lost or late responsive to the number of times that the first audio packet has been decoded in a row has exceeded the specified threshold. In any of the above-described instances, the audio decoder can generate the second segment to have continuity with the first segment. Likewise, in any of the above-described instances, the stream of audio packets can be generated from audio content generated by a video game application executed at a server.

Other embodiments may include an electronic device comprising a buffer to buffer audio packets and a decoder to decode audio packets, the electronic device configured to perform the example method.

Another embodiment includes a non-transitory computer-readable medium storing a set of executable instructions configured to manipulate a processor to decode a first audio packet of a temporarily-buffered subset of audio packets of a stream received via a network to generate a first segment of a decoded audio signal, and responsive to detecting that a second audio packet following the first audio packet in the stream is lost or late, decoding, at the audio decoder, the first audio packet again to generate a second segment of the decoded audio signal, the second segment following the first segment in the decoded audio signal and having continuity with the first segment.

DETAILED DESCRIPTION

FIGS.1-3illustrate various systems and techniques for mitigating the impact of packet loss or packet delays in a real-time audio stream transmitted from a server to a user's electronic device (that is, “client device”) via one or more networks. The client device includes a network interface to receive the audio stream from a network, a jitter buffer to temporarily buffer a subset of the audio packets as they are received at the client device, and an audio decoder to decode audio packets from the jitter buffer to generate corresponding segments of a decoded audio signal. For example, the decoded audio signal can then be mixed with other audio sources, repacketized and further transmitted, or otherwise further processed to ultimately generate one or more analog audio signals used to drive one or more corresponding speakers. In at least one embodiment, the client device employs a packet loss concealment (PLC) technique based on one or more repeat decodings of a previously received audio packet to compensate for one or more subsequent audio packets that have been lost or excessively delayed (that is, “late”) by the network. To this end, the client device employs a stream monitoring module to monitor the received audio stream to detect when an audio packet of the stream is late or lost. In response to detecting such a late or lost audio packet, the stream monitoring module provides an audio packet of the subset in the buffer which was previously decoded by the decoder for a previous time slot to the input of the decoder again for a repeated decoding, for the current time slot in place of a decoding of the audio packet that is lost or late and which was intended for the current time slot. By repeating the decoding of a previously-decoded audio packet to fill in the gap caused by the late/lost packet, this PLC technique leverages the decoders audio synthesis (by following a typical decoding processing path) to facilitate the continuity of the decoded audio signal during packet decoding replication, and thus reducing or eliminating the distortion or artifacts that otherwise would occur in the absence of this continuity.

FIG.1illustrates, a real-time media streaming system100employing a PLC technique using pre-decoder packet replication in accordance with some embodiments. The system100includes a server102coupled to a user's electronic device104(hereinafter, “client device104”) via one or more networks106. The one or more networks106can include, for example, the Internet or other public-access network, wired or wireless wide area network (WAN), wired, or wireless local area network (LAN), wired or wireless personal area network (PAN), or combinations thereof.

The server102includes a network interface108coupled to the network106, a real-time media source110, and an audio encoder112. The real-time media source110generates or otherwise provides real-time media content for transmission to the client device104. To illustrate, the real-time media source110can include, for example, a cloud-based video game being executed at the server102based on player inputs received from the client device104via the network106, with the video game generating both a stream of video frames and a stream of accompanying audio frames for transmission to the client. As another example, the real-time media source110can include a video conferencing application that distributes video and audio streams among the various participant's client devices. As yet another example, the real-time media source110can include the forwarding transmission of the voice content of a Voice-over-internet Protocol (VoIP) or other packet-based voice call in a mobile cellular system. The audio encoder112operates to encode the audio content stream from the real-time media source110and provide the resulting encoded audio stream to the network interface108, whereupon the network interface108packetizes the encoded audio stream and transmits the resulting audio packets to the client device104via the network106as part of a packetized audio stream114.

The client device104represents any of a variety of electronic devices utilized to playback the audio content of the audio stream114, or to decode and forward the audio content for playback by yet another electronic device. Examples of the client device104include a mobile telephone, a desktop computer, a laptop computer, a tablet computer, a game console, a “smart” television, a “smart” watch, an automotive informational/entertainment system, and the like. The client device104includes a network interface116to receive the audio packets of the audio stream114, a jitter buffer118(e.g., a circular buffer) to temporarily buffer a sliding subset of the recently received audio packets, and an audio decoder120that operates to sequentially decode audio packets from the jitter buffer118in a specified order (e.g., received order, sequential order based on time stamp, etc.) to generate a corresponding decoded audio segment of an output decoded audio signal122(e.g., a pulse-code-modulation (PCM) digital signal) that can be either directly converted to one or more analog audio signals used to drive at least one speaker124(e.g., via a digital-to-analog converter, or DAC) or processed further, such as by a digital amplifier/mixer127, before being converted to one or more analog speaker signals for driving the at least one speaker124. In one embodiment, the audio decoder120is implemented as one or more processors126executing audio decoding software128stored in system memory130or other non-transitory computer-readable medium. To illustrate, the audio decoding software128can be implemented as, for example, an Opus Interactive Audio Codec or other well-known or proprietary software-based codec. In other embodiments, the audio decoder120can be implemented as hardcoded or programmable logic, such as an application-specific integrated circuit (ASIC) or field-programmable gate array (FPGA) configured to perform the functionality described herein. In still other embodiments, the audio decoder120can be implemented as a combination of a processor executing software and specific hardcoded/programmable logic.

In at least one embodiment the network106is a combination of one or more packet-switched networks, and thus is subject to congestion, routing errors, buffer overflows, and other network issues that can result in one or more of the audio packets of the audio stream114being lost (that is, never received by the client device104) or late (that is, not received by the client device104in time to be processed for playback in its corresponding decoding timeslot). As a late audio packet has the same result as a lost audio packet in that neither can be used to provide the playback of the represented audio content in the corresponding time slot and therefore is effectively “lost” unless otherwise noted reference to a “lost audio packet” herein is intended to include either an audio packet lost in the network106or an audio packet that arrived too late at the client device104. Without a compensating mechanism, a lost audio packet means that the audio decoder120will not have the intended audio content to decode to generate a corresponding segment of the decoded audio signal122for the corresponding time slot, and thus introducing a significant discontinuity in the resulting decoded audio signal122that will detract from the listener's experience. While conventional PLC techniques such as FEC, silence insertion, interpolation analysis, and decoded signal segment repetition attempt to mitigate the impact of lost audio packets, these conventional techniques are either overly complex and resource-intensive or do not sufficiently eliminate discontinuities in the resulting decoded audio signal.

Accordingly, in at least, one embodiment the client device104employs a PLC technique based on pre-decoder packet replication (that is, replication of audio packet content at the input of the decoder). To this end, the client device104further includes a stream monitoring module132coupled to the network interface116and the jitter buffer118and further coupled to a packet selector134that operates to select and provide audio packets buffered in the jitter buffer118to an input136of the audio decoder120. In some embodiments, one or both of the stream monitoring module132and the packet selector134are implemented at least in part as one or more processors126executing software138stored in the system memory130or other non-transitory computer-readable medium. In other embodiments, one or both of the stream monitoring module132and the packet selector134are implemented at least in part as hardcoded or programmable logic, or a combination of processor-executed software and programmable/hardcoded logic.

As a general operational overview, the stream monitoring module132monitors the received audio stream114via the network interface116or the jitter buffer118to detect lost audio packets. While the audio packet for a corresponding decoding timeslot has been received in time, the stream monitoring module132controls the packet selector134to access the received audio packet from the jitter buffer118and provide the access audio packet to the audio decoder120for decoding of its audio content to generate the segment of the decoded audio signal122for the corresponding time slot. In contrast, in response to detecting a lost audio packet, when the decoding timeslot for the lost packet is approaching the stream monitoring module132controls the packet selector134to access the audio packet that was decoded by the audio decoder120for the previous decoding timeslot and provide this same audio packet to the input136of the audio decoder120for repeated decoding so that the resulting decoded segment is used in the decoded audio signal122as a replacement for the decoded segment, that would have otherwise been generated by the audio decoder120for the lost audio packet had it not been lost.

To illustrate by way of example, assume that the server102transmits audio packets140,141and142in the audio stream114via the network106in that order. In this example, audio packets140and142are successfully received and buffered on time at the client device104, but audio packet141is lost in the network106. Accordingly, with the audio packet140being timely received and buffered, the packet selector134provides the audio packet140from the jitter buffer118to the input136of the audio decoder120at the corresponding time slot X, whereupon the audio decoder120decodes the audio content of the audio packet140to generate a segment Y of the decoded audio signal122. For the next time slot X+1, in response to determining that the next audio packet, audio packet141, is “lost”, the stream monitoring module132controls the packet selector134to again provide the previously-decoded audio packet, that is, audio packet140, to the input136of the audio decoder120, whereupon it is again decoded to generate a corresponding segment Y+1 of the decoded audio signal122. Note that the typical audio synthesis properties exhibited by the audio decoder120result in the segment Y+1 being generated by the audio decoder120to stitch seamlessly with the segment Y previously generated by the audio decoder120from the same audio content, and thus ensuring continuity in the decoded audio signal122between segment and segment Y+1 even though both segments were generated from the same input packet (audio packet140).

Subsequently, as audio packet142is timely received, for time slot X+2 the packet selector134provides the audio packet142to the input136of the audio decoder120, whereupon the audio decoder120decodes the audio packet142to generate a segment Y+2 for the decoded audio signal122, which again is stitched seamlessly with the segment Y+2 by the audio decoder120due to standard audio synthesis procedures employed by the audio decoder120.

FIG.2illustrates a method200representing the pre-decoder packet repetition PLC technique employed by the client device104of the system100ofFIG.1in greater detail in accordance with some embodiments. As represented by block202, during initialization of the client device104for processing the audio stream114, the stream monitoring module132sets a number of parameters. One such parameter initialization includes setting a variable REP_PACKET_COUNT representing an ongoing count of number of repeated packets to zero or some other initialization value. Another parameter initialization includes setting a variable THRESHOLD to a specified number representing the maximum number of times an audio packet can be replicated for decoding (or the number of times replicated decoding is performed) in a row before either triggering a fault or a switch to a different PLC process, depending on implementation. Thus, THRESHOLD is, a tuning parameter that can be used to balance between robustness in view of packet losses and audio quality. The value for THRESHOLD thus can be set by a user, by the provider of the audio stream114, by a provider of the client device104, and the like.

At block204, the chant device104begins receiving the audio packets of the audio, stream114from the server102via the network interface116and buffering a sliding subset of the audio packets in the jitter buffer118(where the maximum size of the current buffered subset is based on the number of entries in the jitter buffer118). As audio packets are received and buffered, at block206the stream monitoring, module132monitors the incoming audio packets to determine whether an audio packet has been lost. When no lost packet is detected, at block208the stream monitoring module132sets or otherwise maintains the variable REP_PACKET_COUNT at zero and for the next decoding time slot the packet selector134accesses the buffered audio packet corresponding to that decoding time slot from the jitter buffer118and provides the accessed audio packet as the input audio packet to the input136of the audio decoder120at block210.

To illustrate, in some implementations each audio packet is assigned a sequence number identifying the position of the corresponding audio packet in an intended playback sequence during the encoding process at the server102. Consequently, when the stream monitoring module132accesses the next audio packet from the jitter buffer118, the stream monitoring module132compares the sequence number of the accessed audio packet with, the expected sequence number. When these numbers match, the stream monitoring module132determines that the audio packet for the corresponding decoding time slot is timely. Conversely, if the actual sequence number of the accessed audio packet does not match the expected sequence number, the stream monitoring module132identifies the audio packet for the corresponding decoding time slot as lost.

Returning to block206, if a lost audio packet is detected, then at block212the stream monitoring module132determines whether the maximum number of packet decoding replications in a row for the same audio packet have been performed by comparing the variable REP_PACKET COUNT to THRESHOLD. If so (that is, REP_PACKET_COUNT=THRESHOLD), then no further packet decoding repetitions for an interrupted sequence of lost packets is permitted, and thus at block214the stream monitoring module132either triggers a fault to stop playback of the audio stream (and corresponding video stream if there is one) and to initiate a system performance check. Alternatively, rather than trigger a fault, the client device104instead can switch to using a different PLC technique, such as silence insertion, interpolation analysis, and the like. However, if fewer than the maximum packet decoding repetitions in a row have been performed (that is, REP_PACKET_COUNT<THRESHOLD), then at block216the stream monitoring module132directs the packet selector134to access the audio packet that was previously decoded for the most-recent time slot from the jitter buffer118and provide this accessed audio packet as the input audio packet to the input136of the audio decoder for the upcoming decoding time slot. The stream monitoring module132also increments REP_PACKET_COUNT to reflect that a packet decoding replication of this audio packet has been performed.

At block218, the audio decoder120decodes the input audio packet selected via the process represented by blocks206-216, that is, either the audio packet associated with the current decoding time slot (if this packet was not lost) or the audio packet that was previously decoded for a previous time slot (if the audio packet intended for this time slot was lost). The audio decoder120generates a corresponding segment of the decoded audio signal122from the encoded audio content of the input audio packet. As explained above, the audio synthesis techniques typically employed by the audio decoder120result in the seamless stitching of each generated segment with the next, and thus providing a decoded audio signal122free of substantial discontinuities even in the event of one or more lost packets in a row. Concurrent with the decoding of audio packets to generate corresponding segments of the decoded audio signal122, at block220the decoded audio signal122can be further processed (e.g., by mixing with other audio signals) and then, converted to one or more analog signals used to drive the one or more speakers124to affect, playback of the audio content represented by the audio stream114.

FIG.3illustrates a chart300depicting an example operation of the pre-decoder packet repetition PLC process represented by method200ofFIG.2. The seven vertical rows of chart300represent seven decoding time slots for the audio decoder120of the client device104, identified as time slots A, B, C, D, E, F, and G, with time slot A being the earliest time slot and time, slot G being the latest time slot. Row310represents the factions of the server102in generating audio packets for each corresponding time slot A-G, row312represents the actions of the client device104in receiving audio packets from the server102for each corresponding time slot, and row314represents the actions of the client device in decoding audio packets and providing for playback of the resulting decoded audio signal segments.

As illustrated by row310, the server generates and transmits audio packets301,302,303,304,305,306, and307for time slots A, B, C, D, E, F, and G, respectively. As represented by rows312and314, on the client device side, for time slot A, audio packet301is received (without loss or late arrival), buffered, decoded, and the resulting segment of the decoded audio signal played, back per normal operation. Likewise, for time slot B, audio packet302is received (without loss or late arrival), buffered, decoded, and the resulting segment of the decoded audio signal played back per normal operation. However, for time slot C, the associated audio packet303is lost by the network106. Accordingly, the stream monitoring module132notes the lost status of audio packet303and thus directs the packet selector134to access the audio packet decoded for the previous time slot B, that is audio packet302, and to provide this audio packet302to the audio decoder120for decoding again into a corresponding segment of the decoded audio signal for time, slot C. Thus, the audio content of audio packet302is decoded twice to generate two successive segments of the resulting decoded audio signal, once for its associate time slot B and then again to fill in for the lost audio packet303for the following time slot C to reconstruct the audio signal for the segments corresponding to time slots B and C.

Thereafter, for time slot D, the associated audio packet304is received on time and thus provided for decoding by the audio decoder120into a corresponding segment of the decoded audio signal. For the following two time slots E and F, the associate audio packets305and306are not received on time for decoding in their respective time slots, but instead are received late during the following time slot G. Accordingly, for time slot E, the client device104reuses the audio content of audio packet304to decode a corresponding segment of the decoded audio signal, and the audio packet304is again selected for decoding a third time to generate another segment of the decoded audio signal for time slot F. Then, while all three of audio packets305,306, and307are received in time for time slot G, audio packets305and306are associated with earlier time slots and thus are discarded from the jitter buffer118, and the audio decoder120decodes the audio packet307to generate a segment of the decoded audio signal corresponding to time slot G.

Benefits, other advantages, and solutions to problems have been described above with regard to specific embodiments. However, the benefits, advantages, solutions to problems, and any feature(s) that may cause any benefit, advantage, or solution to occur or become more pronounced are not to be construed as a critical, required, or essential feature of any or all the claims. Moreover, the particular embodiments disclosed above are illustrative only, as the disclosed subject matter may be modified and practiced in different but equivalent manners apparent to those skilled in the art having the benefit of the teachings herein. No limitations are intended to the details of construction or design herein shown, other than as described in the claims below.

It is therefore evident that the particular embodiments disclosed above may be altered or modified and all such variations are considered within the scope of the disclosed subject matter. Accordingly, the protection sought herein is as set forth in the claims below.