Patent Description:
Although the capacity in telecommunication networks is continuously increasing, it is still of great interest to limit the required bandwidth per communication channel. In mobile networks smaller transmission bandwidths for each call yields lower power consumption in both the mobile device and the base station. This translates to energy and cost saving for the mobile operator, while the end user will experience prolonged battery life and increased talk-time. Further, with less consumed bandwidth per user, the mobile network can service a larger number of users in parallel.

Through modern music playback systems and movie theaters most listeners are accustomed to high quality immersive audio. In mobile telecommunication services, the constraints on radio resources and processing delay have kept the quality at a lower level and most voice services still deliver only monaural sound. Recently, stereo and multi-channel sound for communication services has gained momentum in the context of Virtual/Mixed/Augmented Reality which requires immersive sound reproduction beyond mono. To render high quality spatial sound within the bandwidth constraints of a telecommunication network still presents a challenge. In addition, the sound reproduction also needs to cope with varying channel conditions where occasional data packets may be lost due to e.g. network congestion or poor cell coverage.

In a typical stereo recording the channel pair shows a high degree of similarity, or correlation. Some embodiments of stereo coding schemes [<NUM>] may exploit this correlation by employing parametric coding, where a single channel is encoded with high quality and complemented with a parametric description that allows reconstruction of the full stereo image. The process of reducing the channel pair into a single channel is often called a down-mix and the resulting channel is often called the down-mix channel. The down-mix procedure typically tries to maintain the energy by aligning inter-channel time differences (ITD) and inter-channel phase differences (IPD) before mixing the channels. To maintain the energy balance of the input signal, the inter-channel level difference (ILD) may also be measured. The ITD, IPD and ILD are then encoded and may be used in a reversed up-mix procedure when reconstructing the stereo channel pair at a decoder. The ITD, IPD, and ILD parameters describe the correlated components of the channel pair, while a stereo channel pair may also include a non-correlated component which cannot be reconstructed from the down-mix. This non-correlated component may be represented with an inter-channel coherence parameter (ICC). The non-correlated component may be synthesized at a stereo decoder by running the decoded down-mix channel through a decorrelator filter, which outputs a signal which has low correlation with the decoded down-mix. The strength of the decorrelated component may be controlled with the ICC parameter.

While the parametric stereo reproduction gives good quality at low bitrates, the quality tends to saturate for increasing bitrates due to the limitation of the parametric model. To overcome this issue, the non-correlated component can be encoded. This encoding is achieved by simulating the stereo reconstruction in the encoder and subtracting the reconstructed signal from the input channel, producing a residual signal. If the down-mix transformation is revertible, the residual signal can be represented by only a single channel for the stereo channel case. Typically, the residual signal encoding is targeted to the lower frequencies which are psycho-acoustically more relevant while the higher frequencies can be synthesized with the decorrelator method. <FIG> is a block diagram depicting an embodiment of a conventional setup for a parametric stereo codec including a residual coder. In <FIG>, the encoder receives input signals, performs the processing described above in the stereo processing and down-mix block <NUM>, encodes the mono output via mono encoder <NUM>, encodes the residual signal via residual encoder <NUM>, and encodes the ITD, IPD, ILD, and ICC parameters. The decoder receives the encoded mono output, the encoded residual signal, and the encoded parameters. The decoder decodes the residual signal via residual decoder <NUM> and decodes the mono signal via mono decoder <NUM>. The parametric synthesis block <NUM> receives the decoded mono signal and the decoded residual signal and based on the parameters, outputs stereo channels CH1 and CH2.

Similar principles apply for multichannel audio such as <NUM> and <NUM>. <NUM>, and spatial audio representations such as Ambisonics or Spatial Audio Object Coding. The number of channels can be reduced by exploiting the correlation between the channels and bundling the reduced channel set with metadata or parameters for channel reconstruction or spatial audio rendering at the decoder.

To overcome the problem of transmission errors and lost packages, telecommunication services make use of Packet Loss Concealment (PLC) techniques. In the case that data packets are lost or corrupted due to poor connection, network congestion, etc., the missing information of lost or corrupt data packets in the receiver side may be substituted by the decoder with a synthetic signal to conceal the lost or corrupt data packet. Some embodiments of PLC techniques are often tied closely to the decoder, where the internal states can be used to produce a signal continuation or extrapolation to cover the packet loss. For a multi-mode codec having several operating modes for different signal types, there are often several PLC technologies that can be implemented to handle the concealment of the lost or corrupt data packet.

For linear prediction (LP) based speech coding modes, a technique that may be used is based on adjustment of glottal pulse positions using estimated end-of-frame pitch information and replication of pitch cycle of the previous frame [<NUM>]. The gain of the long-term predictor (LTP) converges to zero with the speed depending on the number of consecutive lost frames and the stability of the last good frame [<NUM>]. Frequency domain (FD) based coding modes are typically designed to handle general or complex signals such as music. For such signals, different techniques may be used depending on the characteristics of the last received frame. Such analysis may include the number of detected tonal components and periodicity of the signal. If the frame loss occurs during a highly periodic signal such as active speech or single instrumental music, a time domain PLC similar to the LP based PLC may be suitable for implementation. In this case the FD PLC may mimic an LP decoder by estimating LP parameters and an excitation signal based on the last received frame [<NUM>]. In case the lost frame occurs during a non-periodic or noise-like signal, the last received frame may be repeated in spectral domain where the coefficients are multiplied to a random sign signal to reduce the metallic sound of a repeated signal. For a stationary tonal signal, it has been found advantageous in some embodiments to use an approach based on prediction and extrapolation of the detected tonal components. More details about the above-described techniques can be found in [<NUM>].

One concealment method operating in the frequency domain is the Phase ECU [<NUM>]. It can be implemented as a stand-alone tool operating on a buffer of the previously decoded and reconstructed time signal. Its framework is based on a sinusoidal analysis and synthesis paradigm. In this technique, the sinusoid components of the last good frame are extracted and phase shifted. When a frame is lost, the sinusoid frequencies are obtained in DFT domain from the past decoded synthesis. First the corresponding frequency bins are identified by finding the peaks of the magnitude spectrum plane. Then, fractional frequencies of the peaks are estimated using peak frequency bins. The peak frequency bins and corresponding fractional frequencies may be stored for use in creating a substitute for a lost frame. The frequency bins corresponding to the peaks along with the neighbors are phase shifted using fractional frequencies. For the remaining frequency bins of the frame, the magnitude of the past synthesis is retained while the phase may be randomized. The burst error may also be handled such that the estimated signal can be smoothly muted by converging it to zero. More detail of Phase ECU can be found in [<NUM>].

There are many different terms used for the packet loss concealment techniques, including Frame Error Concealment (FEC), Frame Loss Concealment (FLC), and Error Concealment Unit (ECU).

The PLC techniques described above are techniques designed for single-channel audio codecs. For a stereo or multi-channel decoder, one solution for error concealment may be to apply any of the above described PLC techniques on each channel. However, this solution does not provide any control of the spatial characteristics of the signal. It is likely the use of this solution will create non-correlated signals, which would give a stereo or multi-channel output that sounds unnatural or too wide. For the stereo case depicted in <FIG>, this translates to using a single channel PLC separately on the down-mix signal and on the residual signal component.

Error concealment of the residual signal component may be particularly sensitive, since the residual component may be added to the side signal which is spatially unmasked. Discontinuities result in dramatic changes in character of the side signal and are therefore easily detected and found to be disturbing when heard.

According to some embodiments of inventive concepts, a method is provided to approximate a lost or corrupted multichannel audio frame of a received multichannel audio signal in a decoding device. The method includes generating a down-mix error concealment frame and transforming the down-mix error concealment frame into a frequency domain to generate a transformed down-mix error concealment frame The method further includes decorrelating the transformed down-mix concealment frame to generate a decorrelated concealment frame. The method further includes obtaining a residual signal spectrum of a stored residual signal of a previously received multichannel audio signal. The method further includes generating an energy adjusted decorrelated residual signal concealment frame using the residual signal spectrum and providing the transformed down-mix error concealment frame, the energy-adjusted decorrelated residual concealment frame, and multi-channel audio parameters from the previously received multichannel audio signal frame to a parametric multi-channel audio synthesis component to generate a synthesized multichannel audio frame. The method further includes performing an inverse frequency domain transformation of the synthesized multichannel audio frame to generate a substitution frame for the lost or corrupted multichannel audio frame.

A potential advantage of combining the phase evolution error concealment method for the peaks of the spectrum with a noise spectrum coming from the error concealed down-mix signal passed through a decorrelator, is that the operation avoids discontinuities in the periodic signal components by phase adjusting the peaks. Moreover, the noise spectrum keeps the desired relation to the down-mix signal, e.g. the desired level of correlation. Another potential advantage is that the operation keeps the energy level of the residual signal at a stable level during frame loss.

According to other embodiments of inventive concepts, an apparatus configured to approximate a lost or corrupted multichannel audio frame of a received multichannel audio signal. The apparatus includes at least one processor and memory communicatively coupled to the processor, said memory comprising instructions executable by the processor, which cause the processor to perform operations. The operations include generating a down-mix error concealment frame and transforming the down-mix error concealment frame into a frequency domain to generate a transformed down-mix error concealment frame The operations further include decorrelating the transformed down-mix concealment frame to generate a decorrelated concealment frame. The operations further include obtaining a residual signal spectrum of a stored residual signal of a previously received multichannel audio signal. The operations further include generating an energy adjusted decorrelated residual signal concealment frame using the residual signal spectrum and providing the transformed down-mix error concealment frame, the energy-adjusted decorrelated residual concealment frame, and multi-channel audio parameters from the previously received multichannel audio signal frame to a parametric multi-channel audio synthesis component to generate a synthesized multichannel audio frame. The operations further include performing an inverse frequency domain transformation of the synthesized multichannel audio frame to generate a substitution frame for the lost or corrupted multichannel audio frame.

According to other embodiments of inventive concepts, a decoder is configured to perform operations. The operations include generating a down-mix error concealment frame and transforming the down-mix error concealment frame into a frequency domain to generate a transformed down-mix error concealment frame The operations further include decorrelating the transformed down-mix concealment frame to generate a decorrelated concealment frame. The operations further include obtaining a residual signal spectrum of a stored residual signal of a previously received multichannel audio signal. The operations further include generating an energy adjusted decorrelated residual signal concealment frame using the residual signal spectrum and providing the transformed down-mix error concealment frame, the energy-adjusted decorrelated residual concealment frame, and multi-channel audio parameters from the previously received multichannel audio signal frame to a parametric multi-channel audio synthesis component to generate a synthesized multichannel audio frame. The operations further include performing an inverse frequency domain transformation of the synthesized multichannel audio frame to generate a substitution frame for the lost or corrupted multichannel audio frame.

According to other embodiments of inventive concepts, an computer program product comprising a non-transitory computer readable medium storing computer program code which when executed by at least one processor causes the at least one processor to: generate a down-mix error concealment frame; transform the down-mix error concealment frame into a frequency domain to generate a transformed down-mix error concealment frame; decorrelate the transformed down-mix concealment frame to generate a decorrelated concealment frame; obtain a residual signal spectrum of a stored residual signal of a previously received multichannel audio signal; generate an energy adjusted decorrelated residual signal concealment frame using the residual signal spectrum; provide the transformed down-mix error concealment frame, the energy-adjusted decorrelated residual concealment frame, and multi-channel audio parameters from the previously received multichannel audio signal frame to a parametric multi-channel audio synthesis component to generate a synthesized multichannel audio frame; and perform an inverse frequency domain transformation of the synthesized multichannel audio frame to generate a substitution frame for the lost or corrupted multichannel audio frame.

According to some other embodiments of inventive concepts, a method is provided to approximate a lost or corrupted multichannel audio frame of a received multichannel audio signal in a decoding device comprising a processor, the method comprising the following operations performed by the processor. The operations include generating a down-mix error concealment frame and transforming the down-mix error concealment frame into a frequency domain to generate a transformed down-mix error concealment frame The operations further include decorrelating the transformed down-mix concealment frame to generate a decorrelated concealment frame. The operations further include obtaining a residual signal spectrum of a stored residual signal of a previously received multichannel audio signal. The operations further include generating an energy adjusted decorrelated residual signal concealment frame using the residual signal spectrum. The operations further include obtaining a set of multi-channel audio substitution parameters. The operations further include performing an inverse frequency domain transformation of the transformed down-mix error concealment frame, the energy-adjusted decorrelated residual concealment frame, and multi-channel audio parameters from the previously received multichannel audio signal frame to generate a transformed down-mix error concealment time-domain frame, an energy-adjusted decorrelated residual concealment time domain frame, and multi-channel audio time domain parameters. The operations further include providing the transformed down-mix error concealment time-domain frame, the energy-adjusted decorrelated residual concealment time-domain frame, and the multi-channel audio time-domain parameters to a parametric multi-channel audio synthesis component to generate a synthesized multichannel audio substitute frame.

According to some other embodiments of inventive concepts, a computer program product comprising a non-transitory computer readable medium storing computer program code which when executed by at least one processor causes the at least one processor to: generate a down-mix error concealment frame; transform the down-mix error concealment frame into a frequency domain to generate a transformed down-mix error concealment frame; decorrelate the transformed down-mix concealment frame to generate a decorrelated concealment frame; obtain a residual signal spectrum of a stored residual signal of a previously received multichannel audio signal frame; generate an energy adjusted decorrelated residual signal concealment frame using the residual signal spectrum; obtain a set of multi-channel audio time-domain substitution parameters; perform an inverse frequency domain transformation of the transformed down-mix error concealment frame, the energy-adjusted decorrelated residual concealment frame to generate a transformed down-mix error concealment time-domain frame and an energy-adjusted decorrelated residual concealment time domain frame; and provide the transformed down-mix error concealment time-domain frame, the energy-adjusted decorrelated residual concealment time-domain frame, and the multi-channel audio time-domain substitution parameters to a parametric multi-channel audio synthesis component to generate a synthesized multichannel audio substitute frame.

According to some other embodiments of inventive concepts, an apparatus configured to approximate a lost or corrupted multichannel audio frame of a received multichannel audio signal is provided. The apparatus includes at least one processor and memory communicatively coupled to the processor, said memory comprising instructions executable by the processor, which cause the processor to perform operations. The operations include generating a down-mix error concealment frame and transforming the down-mix error concealment frame into a frequency domain to generate a transformed down-mix error concealment frame The operations further include decorrelating the transformed down-mix concealment frame to generate a decorrelated concealment frame. The operations further include obtaining a residual signal spectrum of a stored residual signal of a previously received multichannel audio signal. The operations further include generating an energy adjusted decorrelated residual signal concealment frame using the residual signal spectrum. The operations further include obtaining a set of multi-channel audio substitution parameters. The operations further include performing an inverse frequency domain transformation of the transformed down-mix error concealment frame, the energy-adjusted decorrelated residual concealment frame, and multi-channel audio parameters from the previously received multichannel audio signal frame to generate a transformed down-mix error concealment time-domain frame, an energy-adjusted decorrelated residual concealment time domain frame, and multi-channel audio time domain parameters. The operations further include providing the transformed down-mix error concealment time-domain frame, the energy-adjusted decorrelated residual concealment time-domain frame, and the multi-channel audio time-domain parameters to a parametric multi-channel audio synthesis component to generate a synthesized multichannel audio substitute frame.

<FIG> illustrates an example of an operating environment of a decoder <NUM> that may be used to decode multichannel bitstreams as described herein. The decoder <NUM> may be part of a media player, a mobile device, a set-top device, a desktop computer, and the like. The decoder <NUM> receives encoded bitstreams. The bitstreams may be sent from an encoder, from a storage device <NUM>, from a device on the cloud via network <NUM>, etc. During operation, decoder <NUM> receives and processes the frames of the bitstream as described herein. The decoder <NUM> outputs multi-channel audio signals and transmits the multi-channel audio signals to a multi-channel audio player <NUM> having at least one loudspeaker for playback of the multi-channel audio signals. Storage device <NUM> may be part of a storage depository of multi-channel audio signals such as a storage repository of a store or a streaming music service, a separate storage component, a component of a mobile device, etc. Multichannel audio player may be a Bluetooth speaker, a device having at least one loudspeaker, a mobile device, a streaming music service, etc..

<FIG> is a block diagram illustrating elements of decoder <NUM> configured to decode multi-channel audio frames and provide concealment for lost or corrupt frame according to some embodiments of inventive concepts. As shown, decoder <NUM> may include a network interface circuit <NUM> (also referred to as a network interface) configured to provide communications with other devices/entities/functions/etc. The decoder <NUM> may also include a processor circuit <NUM> (also referred to as a processor) coupled to the network interface circuit <NUM>, and a memory circuit <NUM> (also referred to as memory) coupled to the processor circuit. The memory circuit <NUM> may include computer readable program code that when executed by the processor circuit <NUM> causes the processor circuit to perform operations according to embodiments disclosed herein.

According to other embodiments, processor circuit <NUM> may be defined to include memory so that a separate memory circuit is not required. As discussed herein, operations of the decoder <NUM> may be performed by processor <NUM> and/or network interface <NUM>. For example, processor <NUM> may control network interface <NUM> to transmit communications to multichannel audio players <NUM> and/or to receive communications through network interface <NUM> from one or more other network nodes/entities/servers such as encoder nodes, depository servers, etc. Moreover, modules may be stored in memory <NUM>, and these modules may provide instructions so that when instructions of a module are executed by processor <NUM>, processor <NUM> performs respective operations.

In one embodiment, the multi-channel decoder of a multi-channel encoder and decoder system as outlined in <FIG> may be used. In more detail, the encoder can be described with reference to <FIG>. In the description that follows, two channels will be used to describe the embodiments. These embodiments may be used with more than two channels. The multi-channel encoder processes the input left and right channels (designated as CH1 and CH2 in <FIG> and denoted L and R in <FIG>) in segments referred to as frames. For a given frame m the two input channels may be written <MAT> where l denotes the left channel, r denotes the right channel, n = <NUM>,<NUM>,<NUM>,. ,N denotes the sample number in frame m and N is the length of the frame. In an embodiment, the frames may be extracted with an overlap in the encoder such that the decoder may reconstruct the multi-channel audio signals using an overlap add strategy. The input channels are windowed with a suitable windowing function w(n) and transformed to the Discrete Fourier Transform (DFT) domain. Note that other frequency domain representations may be used here, such as a Quadrature Mirror Filter (QMF) filter bank, a Hybrid QMF filter bank or an odd DFT (ODFT) representation which is composed of the MDCT and MDST transform components.

The signals are then analyzed in parametric analysis block <NUM> to extract the ITD, IPD and ILD parameters. In addition, the channel coherence may be analyzed, and an ICC parameter may be derived. The set of multi-channel audio parameters for frame m may be denoted P(m), which contains the complete set of ITD, IPD, ILD and ICC parameters used in the parametric representation. The parameters are encoded by a parameter encoder <NUM> and added to the bitstream to be stored and/or transmitted to a decoder.

Before producing a down-mix channel, in one embodiment, it may be beneficial to compensate for the ITD and IPD to reduce the cancellation and maximize the energy of the down-mix. The ITD compensation may be implemented both in time domain before the frequency transform or in frequency domain, but it essentially performs a time shift on one or both channels to eliminate the ITD. The phase alignment may be implemented in different ways, but the purpose is to align the phase such that the cancellation is minimized. This ensures maximum energy in the down-mix. The ITD and IPD adjustments may be done in frequency bands or be done on the full frequency spectrum and it should preferably be done using the quantized ITD and IPD parameters to ensure that the modification can be inverted in the decoder stage.

The embodiments described below are independent of the realization of the IPD and ITD parameter analysis and compensation. In other words, the embodiments are not dependent on how the IPD and ITP are analyzed or compensated In such embodiments, the ITD and IPD adjusted channels are denoted with an asterisk: <MAT>.

The ITD and IPD adjusted input channels are then down-mixed by the parametric analysis and down-mix block <NUM> to produce a mid/side representation, also called a down-mix/side representation. One way to perform the down-mix is to use the sum and difference of the signals.

The down-mix signal Xm(m, k) is encoded by down-mix encoder <NUM> to be stored and/or transmitted to a decoder. This encoding may be done in frequency domain, but it may also be done in time domain. In that case a DFT synthesis stage is required to produce a time domain version of the down-mix signal, which is in turn provided to the down-mix encoder <NUM>. The transformation to time domain may, however, introduce a delay misalignment with the multi-channel audio parameters that would require additional handling. In one embodiment, this is solved by introducing additional delay or by interpolating the parameters to ensure that the decoder synthesis of the down-mix and the multi-channel audio parameters are aligned.

The complementary side signal XS(m, k) may be generated from the down-mix and the obtained multi-channel audio parameters by a local parametric synthesis block <NUM>. A side signal prediction XS̃(m, k) can be derived using the down-mix signal <MAT> where p(·) is a predictor function and may be implemented as a single scaling factor α which minimizes the mean squared error (MSE) between the side signal and the predicted side signal. Further, the prediction may be applied on frequency bands and involve a prediction parameter for each frequency band b.

If the coefficients of band b are designated as column vectors XS̃,b(m) and XM,b(m), the minimum MSE predictor can be derived as <MAT>.

However, this expression may be simplified to produce a more stable prediction parameter. The prediction parameter αb can be used as an alternative implementation of the ILD parameter. Further details are described in the prediction mode of reference [<NUM>].

Given the predicted side signal, a prediction residual XR(m, k) can be created [<NUM>].

The prediction residual may be inputted in to a residual encoder <NUM>. The encoding may be done directly in DFT domain or it could be done in time domain. Similarly, as for the down-mix encoder, a time domain encoder would require a DFT synthesis which may require alignment of the signals in the decoder. The residual signal represents the diffuse component which is not correlated with the down-mix signal. If a residual signal is not transmitted, a solution in one embodiment may be to substitute a signal for the residual signal in the stereo synthesis state in the decoder with the signal coming from a decorrelated version of the decoded down-mix signal. The substitute is typically used for low bitrates where the bit budget is too low to represent the residual signal with any useful resolution. For intermediate bit rates, it is common to encode a part of the residual. In this case the lower frequencies are often encoded, since they are perceptually more relevant. For the remaining part of the spectrum, the decorrelator signal is used as a substitute for the residual signal in the decoder. This approach is often referred to as a hybrid coding mode [<NUM>]. Further details are provided in the decoder description below.

The representation of the encoded down-mix, the encoded multi-channel audio parameters, and the encoded residual signal is multiplexed into a bitstream <NUM>, which may be transmitted to a decoder or stored in a medium for future decoding.

In one embodiment, a multi-channel decoder is used in DFT domain as outlined in <FIG>. <FIG> illustrates an embodiment of a decoder in which the blocks of <FIG> that generate a residual signal in case of a lost frame. <FIG> illustrates an embodiment of a combination of the blocks of <FIG> and <FIG>. In the description that follows, the blocks of <FIG> shall be used. However, it should be noted that the demux <NUM> of <FIG> provides at least the same functions as demux <NUM> of <FIG>, the down mix decoder <NUM> of <FIG> provides at least the same functions as the down mix decoder <NUM> of <FIG>, the stereo parameters decoder <NUM> of <FIG> provides at least the same functions of stereo parameters <NUM> of <FIG>, decorrelator <NUM> of <FIG> provides at least the same functions of decorrelator <NUM> of <FIG>, residual decoder <NUM> of <FIG> provides at least the same functions as residual decoder <NUM> of <FIG>, parametric synthesis block <NUM> of <FIG> provides at least the same functions of parametric synthesis block <NUM> of <FIG>. Similarly, the down-mix PLC <NUM> of <FIG> provides at least the same functions of down-mix PLC <NUM> of <FIG>, the decorrelator <NUM> of <FIG> provides at least the same functions of decorrelator <NUM> of <FIG>, memory <NUM> of <FIG> provides at least the same functions of memory <NUM> of <FIG>, spectral shaper <NUM> of <FIG> provides at least the same functions of spectral shaper <NUM> of <FIG>, phase-ecu <NUM> of <FIG> provides at least the same functions as phase-ecu <NUM> of <FIG>, signal combiner <NUM> of <FIG> provides at least the same functions as signal combiner <NUM> of <FIG>, and parametric synthesis block <NUM> of <FIG> provides at least the same functions of parametric synthesis block <NUM> of <FIG>.

Turning now to <FIG>, a down-mix decoder <NUM> provides a reconstructed down-mix signal M̂(m, n) which is segmented into DFT analysis frames m and n = <NUM>,<NUM>,<NUM>,. , N - <NUM> denote the sample numbers within frame m. The analysis frames are typically extracted with an overlap which permits an overlap-add strategy in the DFT synthesis stage. The corresponding DFT spectra may be obtained through a DFT transform <MAT> where w(n) denotes a suitable windowing function. The shape of the windowing function can be designed using a trade-off between frequency characteristics and algorithmic delay due to length of the overlapping regions. Similarly, a residual decoder <NUM> produces a reconstructed residual signal R̂(m,n) for frame m and time instances n = <NUM>,<NUM>,<NUM>,. Note that the frame length NR may be different from N since the residual signal may be produced at a different sampling rate. Since the residual coding may be targeted only for the lower frequency range, it may be beneficial to represent it with a lower sampling rate to save memory and computational complexity. A DFT representation of the residual signal XR̂(m, k) is obtained. Note that if the residual signal is upsampled in DFT domain to the same sampling rate as the reconstructed down-mix, the DFT coefficients will need to be scaled with N/NR and the XR̂(m, k) would be zero-padded to match the length N. To simplify the notation, and since the embodiment is not affected by the use of different sampling rates, for purposes of better understanding of the method, the sampling rates shall be equal and NR = N in the following description. Thus, no scaling or zero-padding shall be shown.

It should be noted that the frequency transform by means of a DFT is not necessary in case the down-mix and/or the residual signal is encoded in DFT domain. In this case, the decoding of the down-mix and/or residual signal provides the DFT spectrum that are necessary for further processing.

In an error free frame, often referred to as a good frame, the multi-channel audio decoder produces the multi-channel synthesis using the decoded down-mix signal together with the decoded multi-channel audio parameters in combination with the decoded residual signal. The DFT spectrum of the residual signal XR̂(m, k) is stored in memory <NUM>, such that the variable XR̂,mem(k) always holds the residual signal spectrum of the last received frame.

In some embodiments, a relevant subpart of the spectrum may be stored in order to save memory, e.g. only the lower frequency bins. In other embodiments, the residual signal may be stored in the time domain and the DFT spectrum may be obtained only when error occurs. This could reduce the peak computational complexity since the error concealment operation typically has lower complexity than the decoding of a correctly received frame. In the description that follows, the residual signal is already transformed to DFT domain during normal operation and the residual signal is stored as a DFT spectrum. In other embodiments, the residual signal is stored in the time domain. In these embodiments, the residual signal spectrum is obtained by transforming the residual signal to the DFT domain.

The decoded down-mix M̂(m, n) is fed to the decorrelator <NUM> to synthesize a non-correlated signal component D(m,n), and the resulting signal is transformed to DFT domain XD(m, k). Note that the decorrelation may also be carried out in the frequency domain. The decoded down-mix XM̂(m, k), the decorrelated component XD(m, k), and the residual signal XR̂(m, k) is fed together with the multi-channel audio parameters P(m) to the parametric multi-channel synthesis block <NUM> to produce the reconstructed multi-channel audio signal. After the multi-channel synthesis in DFT domain has been applied, the left and right channels are transformed to time domain and output from the stereo decoder.

Turning to <FIG>, operations the decoder <NUM> may perform when the decoder <NUM> detects a lost or corrupted multichannel audio frame (i.e., a bad frame) of an encoded multichannel audio signal. When the decoder detects a lost or corrupted frame, i.e., a bad frame (as represented by the bad frame indicator (BFI) in <FIG>), the PLC technique is performed. In operation <NUM>, the PLC of the down-mix decoder <NUM> is activated and generates an error concealment frame for the down-mix M̂ECU(m, n). The down-mix error concealment frame is frequency transformed to produce the corresponding DFT spectrum XM̂,ECU(m, n) in operation <NUM>. In operation <NUM>, the transformed down-mix error concealment frame may be input into the same decorrelator function <NUM> that is used for the down-mix to generate the decorrelated concealment frame DECU(m, n) or input to a different decorrelator function and then frequency transformed to produce a decorrelated down-mix concealment frame XD,ECU(m, k).

The decorrelator function may be done in time domain before transformation, in the form of an all-pass filter, a delay, or a combination thereof. It may also be done in frequency domain after the frequency transform, in which case it would operate on frames, likely including past frames.

In operation <NUM>, a residual signal spectrum is obtained. The residual signal spectrum may be retrieved from storage when it has been previously stored. In situations where the residual signal is stored prior to DFT transformation operations, then the residual signal spectrum is obtained by performing a DFT operation on the stored residual signal. To generate a concealment frame for the residual signal, an energy adjusted decorrelated residual signal is generated in operation <NUM>. In operation <NUM>, a Phase ECU <NUM> performs a phase extrapolation or phase evolution strategy on a residual signal from the past synthesis which is stored in memory <NUM> as previously described. See also [<NUM>].

Turning to <FIG>, the phase extrapolation or phase evolution strategy phase-shifts the peak sinusoids of the residual signal spectrum (see sinusoid component of <FIG>) in operation <NUM> and the energy of the noise spectrum of non-peak sinusoids (see noise spectrum of <FIG>) is adjusted in operation <NUM>. Further details of these operations are provided in <FIG>.

Turning to <FIG>, in operation <NUM>, the residual signal spectrum XR̂,mem(k), which may also be referred to as a "prototype signal" is first input to a peak detector circuit that detects peak frequencies on a fractional frequency scale. A set of peaks <MAT> may be detected which are represented by their estimated fractional frequency fi and where Npeaks is the number of detected peaks. Here the fractional frequency is expressed as a fractional number of DFT bins, such that e.g. the Nyquist frequency is found at f = N/<NUM> + <NUM>. In operation <NUM>, each detected peak is then associated with a number of frequency bins representing the detected peak. The number of frequency bins may be found by rounding the fractional frequency to the closest integer and including the neighboring bins, e.g. the Nnear peaks on each side: <MAT> where [·] represents the rounding operation and Gi is the group of bins representing the peak at frequency fi. The number Nnear is a tuning constant that is determined when designing the system. A larger Nnear gives higher accuracy in each peak representation, but also introduces a larger distance between peaks that may be modeled. A suitable value for Nnear may be <NUM> or <NUM>. A concealment spectrum XR,ECU (m, k) for the residual signal is formed by inserting the group of bins, including a phase adjustment operation <NUM>, which is based on the fractional frequency and the number of samples between the analysis frame of the previous frame and where the current frame would start.

The phase adjustment for each peak frequency fiis applied to each corresponding group of bins Gi according to the phase adjustment <MAT> which is applied to the corresponding bins of the concealment spectrum for the residual signal <MAT>.

In operation <NUM>, the remaining bins of XR,ECU(m, k), which are not occupied by the peak bins Gi, which may be referred to as the noise spectrum or the noise component of the spectrum, are populated using the spectral coefficients of the decorrelated concealment frame XD,ECU(m, k). To ensure the coefficients have the appropriate energy level and overall spectral shape, the energy may be adjusted to match the energy of the noise spectrum of the residual spectrum memory XR̂,mem(k). This may be done by setting all peak bins Gi to zero in a calculation buffer and matching the energy of the remaining noise spectrum bins. The energy matching may be done on a band basis as shown in <FIG>.

Turning to <FIG>, a band b is designated in operation <NUM> that spans the range of bins kstart(b). In operation <NUM>, the energy matching gain factor gb can be calculated as: <MAT>.

In operation <NUM>, the noise spectrum bins k are filled with the energy adjusted decorrelated residual concealment frame using the energy matching gain factor: <MAT>.

Note that it may also be possible to apply the scaling on wide or narrow bands or even for each frequency bin. In the case of scaling for each bin, the magnitude spectrum of the residual memory XR̂,mem(k) is kept while the phase is applied from the spectrum of the decorrelated concealment frame XD,ECU(m, k). For example, the scaling may be achieved either by a magnitude adjustment of XD,ECU(m, k) to match the magnitude of XR̂,mem(k), or by a phase adjustment of XR̂,mem(k) to match the phase of XD,ECU(m,k). However, performing the scaling on a band basis retains some of the spectral fine structure which may be desirable.

In an embodiment in the case of scaling for each frequency bin, applying the phase from the spectrum of the decorrelated concealment frame XD,ECU(m,k) may use an approximation of the phase. This may reduce the complexity of the scaling. The energy matching gain factor gk can be calculated as: <MAT>.

The noise spectrum bins k are filled with the energy adjusted decorrelated residual concealment frame using the energy matching gain factor: <MAT>.

The computation of gk involves a square root and a division, which may be computationally complex. In an embodiment, an approximate phase adjustment is used that matches the sign and the order of the absolute values of the real and imaginary components of the phase target such that the phase is moved within π/<NUM> of the phase target. This embodiment may skip the gain scaling with the energy matching gain factor gk̇· XR,ECU(m,k) may be written as <MAT> <MAT> where (c,d) is <MAT> in the case where the order of the absolute values of the real and imaginary components is the same, i.e. <MAT> and otherwise <MAT>.

The approximate phase adjustment is illustrated in <FIG>. In <FIG>, the phase target is given by XD,ECU(m,k) illustrated at <NUM>. The non-phase adjusted ECU synthesis XR̂,mem(k) is illustrated at <NUM>. The ECU synthesis XR,ECU(m, k) after the approximate phase adjustment has been applied is illustrated at <NUM>. The approximate phase adjustment can be used on a band basis and/or on a per frequency bin basis.

Note that if no tonal components are found, i.e. no peaks are detected, the entire concealment frame will be composed of the decorrelated concealment frame with spectral shaping applied, XR,ECU(m,k). This is illustrated in <FIG>. Turning to <FIG>, in operation <NUM>, the decoder <NUM> detects whether there are peak signals in the residual signal spectrum on a fractional frequency scale. If there are peak signals, operations <NUM> to <NUM> are performed. Specifically, each peak frequency is associated with a number of peak frequency bins in operation <NUM>. Operation <NUM> is similar in operation to operation <NUM>. In operation <NUM>, a phase adjustment is applied to each of the number of peak frequency bins. Operation <NUM> is similar in operation to operation <NUM>. In operation <NUM>, the remaining bins are populated using spectral coefficients of the decorrelated concealment frame and the energy level of the remaining bins is adjusted to match the energy level of the noise spectrum of the residual spectrum memory. Operation <NUM> is similar in operation to operation <NUM>. If there are no peak signals, operation <NUM> is performed, which populates all bins using spectral coefficients of the decorrelated concealment frame and the energy level of the bins is adjusted to match the energy level of the noise spectrum of the residual spectrum memory.

To complete the stereo synthesis of the error concealment frame, the multi-channel parameters needs to be estimated for the lost frame. This concealment may be done with various methods, but one way that was found to give reasonable results was to just repeat the stereo parameters from the last received frame to produce the multi-channel audio substitution parameters P̂(m).

The final spectrum of the conceal residual spectrum is found by combining the spectral peaks with the energy adjusted noise spectrum in signal combiner <NUM>. An example of the combination is illustrated in <FIG>.

Returning to <FIG>, in operation <NUM>, the down-mix error concealment frame <MAT>, the decorrelated down-mix concealment frame XD,ECU(m, k) and the energy adjusted decorrelated residual concealment frame XR,ECU(m, k) is fed together with the multichannel audio parameters P̂(m) to the parametric synthesis block <NUM> to produce the reconstructed signal. After the synthesis in DFT domain has been applied, the multichannel signal is transformed to time domain (e.g., left and right channels) in operation <NUM> and output from the stereo decoder.

For example, in operation <NUM> of <FIG>, multichannel audio signals are generated based on the reconstructed signal (i.e., substitution frame). In operation <NUM>, the multichannel audio signals are output towards at least one loudspeaker for playback.

Turning to <FIG>, DFTs and IDFTs are illustrated. The IDFTs serve to decouple the down-mix decoding and the residual decoding from the DFT analysis stage. In other embodiments, the IDFTs are not used. In yet other embodiments where the signal processing described above is performed in the time domain, the DFTs are only used to provide the a decorrelated down-mix concealment frame XD,ECU(m, k) and a residual signal spectrum XR̂,mem(k) while the IDFTs are used to provide their time domain counterparts.

Turning, to <FIG> and <FIG>, flowcharts are illustrated depicting how the operations of concealment of residual signal of <FIG> may be performed in serial or in parallel. In case of an error-free frame, the DFT spectrum of the residual signal XR̂(m, k) is stored in memory and updated in every error-free frame in operation <NUM>. This memory is later used in the concealment of the "lost frame". When the decoder detects or is notified of frame loss/corruption, the PLC algorithm, designed for down-mix part, is activated and generates the down-mix signal M̂ECU(m, n) in operation <NUM>. PLC algorithm for down-mix can be chosen from the techniques described above. Then, M̂ECU(m, n) can be fed to the decorrelator in operation <NUM> to extract a non-correlated signal XD,ECU(m, k). Decorrelation can also be carried out in time domain as well. Also, the memory of down-mix, which holds the down-mix signal of the past frame, may be included in the input to the decorrelator. Then sinusoid components of residual memory, residual from last good XR̂,mem(k), are phase shifted in operation <NUM>. Note that operations <NUM> and <NUM> are independent from each other and can be carried out the other way around. To keep the shape of the residual signal close to the residual of last good frame, the spectrum of decorrelator signal is reshaped in operation <NUM> based on the residual signal of the last good frame. The phase-shifted sinusoid components of residual signal of the last good frame and the reshaped decorrelated signal are combined in operation <NUM> and the concealment frame for residual signal XR,ECU(m, k) is generated. In another embodiment, the decoder may process operations <NUM> and <NUM> in parallel with operation <NUM>. This is illustrated in <FIG>.

<FIG> and <FIG> show an example of how the decorrelator signal is shaped. <FIG> illustrates a residual signal spectrum (labeled as prototype) and a decorrelator output. <FIG> illustrates a concealment frame for the residual signal XR,ECU(m, k) derived as described above.

As previously indicated, the input to the parametric synthesis block <NUM> may alternatively be in the time domain. <FIG> illustrates the operation of decoder <NUM> when the input to the parametric synthesis block <NUM> is in the time domain and the parametric synthesis block synthesizes the signals in the time domain. Operations <NUM> to <NUM> are the same operations as operations <NUM> to <NUM> of <FIG> as described above. In operation <NUM>, the decoder <NUM> performs an inverse frequency domain (IFD) transformation on the decorrelated concealment frame and the concealment frame for the residual signal. In operation <NUM>, the resulting IFD transformed signals and the parametric multi-channel audio time-domain substitution parameters are provided to the multi-channel audio synthesis component <NUM>, which generates the output channels in the time domain.

Explanations for abbreviations from the above disclosure are provided below.

Citations for references from the above disclosure are provided below.

Many variations and modifications can be made to the embodiments without substantially departing from the principles of the present inventive concepts. All such variations and modifications are intended to be included herein within the scope of present inventive concepts. Accordingly, the above disclosed subject matter is to be considered illustrative, and not restrictive, and the examples of embodiments are intended to cover all such modifications, enhancements, and other embodiments. Thus, to the maximum extent allowed by law, the scope of present inventive concepts are to be determined by the broadest permissible interpretation of the present disclosure including the examples of embodiments and their equivalents, and shall not be restricted or limited by the foregoing detailed description.

Claim 1:
A method of approximating a lost or corrupted multi-channel audio frame of a received multi-channel audio signal in a decoding device comprising a processor, the method comprising the following operations performed by the processor:
generating a down-mix error concealment frame (<NUM>, <NUM>, <NUM>, <NUM>);
transforming the down-mix error concealment frame into a frequency domain to generate a transformed down-mix error concealment frame (<NUM>);
decorrelating the transformed down-mix error concealment frame to generate a decorrelated concealment frame (<NUM>, <NUM>, <NUM>,<NUM>);
obtaining a residual signal spectrum (<NUM>) of a stored residual signal of a previously received multi-channel audio signal frame;
detecting peak frequencies of the residual signal spectrum (<NUM>, <NUM>) of the stored residual signal on a fractional frequency scale;
associating (<NUM>, <NUM>) each peak frequency with a number of peak frequency bins representing the peak frequency;
applying a phase adjustment (<NUM>, <NUM>, <NUM>, <NUM>, <NUM>) to each of the number of peak frequency bins according to a phase adjustment to form a residual signal concealment spectrum;
populating remaining bins (<NUM>, <NUM>) of the residual signal concealment spectrum using spectral coefficients of the decorrelated concealment frame and adjusting an energy level of the remaining bins to match an energy level of a noise spectrum of the residual signal spectrum to generate an energy-adjusted decorrelated residual signal concealment frame;
obtaining a set of multi-channel audio substitution parameters;
providing (<NUM>) the transformed down-mix error concealment frame, the energy-adjusted decorrelated residual signal concealment frame, and the set of multi-channel audio substitution parameters to a parametric multi-channel audio synthesis component to generate a synthesized multi-channel audio frame; and
performing (<NUM>) an inverse frequency domain transformation of the synthesized multi-channel audio frame to generate a substitution frame for the lost or corrupted multi-channel audio frame.