Patent Description:
The sound or soundfield within the listening environment of a listener that is placed at a listening position may be described using an ambisonics signal. The ambisonics signal may be viewed as a multi-channel audio signal, with each channel corresponding to a particular directivity pattern of the soundfield at the listening position of the listener. An ambisonics signal may be described using a three-dimensional (3D) cartesian coordinate system, with the origin of the coordinate system corresponding to the listening position, the x-axis pointing to the front, the y-axis pointing to the left and the z-axis pointing up.

A multi-channel audio signal may comprise active sections of active speech or audio and other inactive sections without speech or audio, which typically only comprise background noise. The background noise may have spatial characteristics, such as an air conditioning noise coming from a particular direction.

<CIT> discloses a method for generating a comfort noise (CN) parameter. The method includes receiving an audio input; detecting, with a Voice Activity Detector (VAD), a current inactive segment in the audio input; as a result of detecting, with the VAD, the current inactive segment in the audio input, calculating a CN parameter CN used; and providing the CN parameter CN used to a decoder. The CN parameter CN used is calculated based at least in part on the current inactive segment and a previous inactive segment.

<CIT> discloses a multichannel audio signal processing method and an encoder. The encoder includes a signal detection unit and a signal encoding unit. The signal encoding unit is configured to: when the signal detection unit detects that an N-frame downmixed signal includes a speech signal, encode the N-frame downmixed signal; or when the signal detection unit detects that the N-frame downmixed signal does not include a speech signal: encode the N-frame downmixed signal if the signal detection unit determines that the N-frame downmixed signal satisfies a preset audio frame encoding condition, or skip encoding the N-frame downmixed signal if the signal detection unit determines that the N-frame downmixed signal does not satisfy a preset audio frame encoding condition.

The present document addresses the technical problem of modelling spatial background noise in an efficient manner, in particular for providing perceptually pleasing transitions between active and inactive sections of a multi-channel audio signal. The technical problem is solved by the independent claims. Preferred examples are described in the dependent claims.

According to an aspect, a method for encoding a multi-channel input (audio) signal which comprises N different channels, with N><NUM>, in particular N><NUM>, is described. The method comprises determining whether a current frame of the multi-channel input signal is an active frame or an inactive frame, using a signal and/or a voice activity detector. Furthermore, the method comprises determining a downmix signal based on the multi-channel input signal and/or based on a target bitrate for encoding the multi-channel input signal, wherein the downmix signal comprises less than or equal to N channels. The method further comprises determining upmixing metadata comprising a set of (spatial) parameters for generating, based on the downmix signal, a reconstructed multi-channel signal comprising N channels. The upmixing metadata may be determined in dependance of whether the current frame is an active frame or an inactive frame. In addition, the method comprises encoding the upmixing metadata into a bitstream. The method further comprises determining a covariance, in particular a covariance matrix, for the current frame of the multi-channel input signal, by performing temporal smoothing over multiple frames from the sequence of frames, in dependance of whether the current frame is an active frame or an inactive frame; and determining one or more parameters of the upmixing metadata based on the covariance for the current frame.

According to a further aspect, a software program is described. The software program may be adapted for execution on a processor and for performing the method steps outlined in the present document when carried out on the processor.

According to another aspect, a storage medium is described. The storage medium may comprise a software program adapted for execution on a processor and for performing the method steps outlined in the present document when carried out on the processor.

According to a further aspect, a computer program product is described. The computer program may comprise executable instructions for performing the method steps outlined in the present document when executed on a computer.

According to another aspect, an encoding unit for encoding a multi-channel input signal which comprises N different channels, with N being a positive integer greater than one, preferably N><NUM>, is described. The encoding unit is configured to determine whether a current frame of the input signal is an active frame or an inactive frame using a signal and/or a voice activity detector. Furthermore, the encoding unit is configured to determine a downmix signal based on the multi-channel input signal and/or based on the bitrate, wherein the downmix signal comprises less than or equal to N channels. In addition, the encoding unit is configured to determine upmixing metadata comprising a set of parameters for generating, based on the downmix signal, a reconstructed multi-channel signal comprising N channels. The upmixing metadata may be determined in dependance of whether the current frame is an active frame or an inactive frame. The encoding unit is further configured to encode the upmixing metadata into a bitstream. The encoding unit is further configured to determine a covariance, in particular a covariance matrix, for the current frame of the multi-channel input signal, by performing temporal smoothing over multiple frames from the sequence of frames, in dependance of whether the current frame is an active frame or an inactive frame; and determine one or more parameters of the upmixing metadata (<NUM>) based on the covariance for the current frame.

It should be noted that the methods, devices and systems including its preferred embodiments as outlined in the present patent application may be used stand-alone or in combination with the other methods, devices and systems disclosed in this document. Furthermore, all aspects of the methods, devices and systems outlined in the present patent application may be arbitrarily combined. In particular, the features of the claims may be combined with one another in an arbitrary manner.

As outlined above, the present document relates to an efficient coding of spatial comfort noise for immersive audio signals such as First Order Ambisonics (FOA) or HOA (Higher Order Ambisonics) signals. Notably FOA or HOA signals are referred to herein more generally as soundfield representation (SR) signals. An SR signal may comprise a relatively high number of channels or waveforms, wherein the different channels relate to different panning functions and/or to different directivity patterns. By way of example, an Lth-order 3D FOA or HOA signal comprises (L+<NUM>)<NUM> channels.

<FIG> illustrates an encoding unit <NUM> and a decoding unit <NUM> for encoding and decoding a multi-channel input signal <NUM>, which may comprise an SR signal. In particular, the multi-channel input signal <NUM> may comprise (possibly a combination of) one or more mono signals, one or more stereo signals, one or more binaural signal, one or more (conventional) multi-channel signals (such as a <NUM> or a <NUM> signal), one or more audio objects, and/or one or more SR signals. The different signal components may be considered to be individual channels of the multi-channel input signal <NUM>.

The encoding unit <NUM> comprises a spatial analysis and downmix module <NUM> configured to downmix the multi-channel input signal <NUM> to a downmix signal <NUM> comprising one or more channels. The downmix signal <NUM> may itself be an SR signal, notably a first order ambisonics (FOA) signal, if the input signal <NUM> comprises a HOA signal. Downmixing may be performed in the subband domain or QMF domain (e.g., using <NUM> or more subbands).

The spatial analysis and downmix module <NUM> is further configured to determine SPAR, Spatial Audio Resolution Reconstruction or Spatial Reconstruction, metadata <NUM> that is configured to reconstruct the multi-channel input signal <NUM> from the downmix signal <NUM>. The spatial analysis and downmix module <NUM> may be configured to determine the SPAR metadata <NUM> in the subband domain. The SPAR metadata <NUM> is referred to herein also as upmixing metadata.

In addition, the encoding unit <NUM> may comprise a coding module <NUM> which is configured to perform waveform encoding (e.g., EVS encoding) of the downmix signal <NUM>, thereby providing coded audio data <NUM>. Each channel of the downmix signal <NUM> may be encoded using a mono waveform encoder, thereby enabling an efficient encoding. Furthermore, the encoding unit <NUM> comprises a quantization module <NUM> which is configured to quantize the SPAR metadata <NUM> and to perform entropy encoding of the (quantized) SPAR metadata <NUM>, thereby providing coded metadata <NUM>. The coded audio data <NUM> and the coded metadata <NUM> may be inserted into a bitstream. Coding of the downmix signal <NUM> and/or of the SPAR metadata <NUM> is typically controlled using a mode and/or bitrate control module <NUM>.

Typically, operation of the spatial analysis and/or downmix module <NUM> is dependent on the target bitrate. In particular, the number of channels of the downmix signal <NUM> may be dependent on the target bitrate. A higher number of downmix channels typically allows an increased performance as it allows more waveform reconstruction than parametric reconstruction. On the other hand, a higher number of downmix channels typically leads to an increased bitrate, as an increased number of channels need to be coded by the audio coding module <NUM>.

At low bitrates, it may be preferable to only use a single channel downmix. On the other hand, the number of downmix channels may be increased with increasing bitrate. This also applies to the DTX operation described in the present document.

The decoding unit <NUM> of <FIG> comprises a decoding module <NUM> which is configured to derive a reconstructed downmix signals <NUM> from the coded audio data <NUM>. Furthermore, the decoding unit <NUM> comprises a metadata decoding module <NUM> which is configured to derive the SPAR metadata <NUM> from the coded metadata <NUM>.

In addition, the decoding unit <NUM> comprises a reconstruction module <NUM> which is configured to derive a reconstructed multi-channel signal <NUM> from the SPAR metadata <NUM> and from the reconstructed downmix signal <NUM>. The reconstructed multi-channel signal <NUM> may comprise a reconstructed SR signal. In particular, the reconstructed multi-channel signal <NUM> may comprise the same types of channels as the multi-channel input signal <NUM>. The reconstructed multi-channel signal <NUM> may be used for speaker rendering, for headphone rendering and/or for SR rendering.

<FIG> illustrates an example reconstruction module <NUM>. The reconstruction module <NUM> takes as input one or more channels of the reconstructed downmix signal <NUM>. A first mixer <NUM> may be configured to upmix the one or more channels of the reconstructed downmix signal <NUM> to an increased number of signals. The first mixer <NUM> depends on the SPAR metadata <NUM>.

The reconstruction module <NUM> may comprise one or more decorrelators <NUM> which are configured to produce signals from the representation of the W channel of the reconstructed downmix signal <NUM> that are processed in a second mixer <NUM> to produce an increased number of channels. The second mixer <NUM> depends on the SPAR metadata <NUM>. The output of the first mixer <NUM> and the output of the second mixer <NUM> are summed to provide the reconstructed multi-channel signal <NUM>. The reconstructed multi-channel signal <NUM> may be of the same type (in particular, may comprise the same number of channels) as the multi-channel input signal <NUM>.

The SPAR metadata <NUM> may be composed of data that represents the coefficients of upmixing matrices used by the first mixer <NUM> and/or by the second mixer <NUM>. The mixers <NUM>, <NUM> may operate in the subband domain (notably in the QMF domain). In this case, the SPAR metadata <NUM> comprises data that represents the coefficients of upmixing matrices used by the first mixer <NUM> and by the second mixer <NUM> for a plurality of different subbands (e.g., <NUM> or more subbands).

<FIG> illustrates the example of a one channel downmix signal <NUM> which comprises a representation of the W channel (referred to as W'). The first mixer <NUM> generates reconstructions of the predictable portions of the X, Y and/or Z channels based on W' (using the prediction coefficients from the SPAR metadata <NUM>). The second mixer <NUM> generates representations of the uncorrelated residual signals X', Y' and/or Z'. In other words, the second mixer <NUM> uses the SPAR metadata <NUM> to spectrally shape signals which are uncorrelated with regards to W' in accordance to the original residual signals X', Y' and/or Z', respectively, thereby providing representations or reconstructions of the residual signals X', Y' and/or Z'.

The encoding unit <NUM> may be configured to convert an FOA input signal <NUM> into a downmix signal <NUM> and parameters, i.e., SPAR metadata <NUM>, used to regenerate the input signal <NUM> at the decoding unit <NUM>. The number of channels of the downmix signal <NUM> may vary from <NUM> to <NUM> channels. The parameters may include prediction parameters Pr, cross-prediction parameters C and/or decorrelation parameters P. These parameters may be calculated from the covariance matrix of a windowed input signal <NUM>. Furthermore, the parameters may be calculated in a specified number of subbands. In the case of comfort noise, a reduced number of subbands (also referred to as frequency bands) may be used, e.g., <NUM> subbands instead of <NUM> subbands.

An example representation of SPAR parameter extraction may be as follows (as described with reference to <FIG>):.

The prediction parameters may be determined within the prediction module <NUM> shown in <FIG>, thereby providing the residual channels Y', Z' and X' <NUM>.

In an exemplary implementation, W may be an active channel (or in other words, with active prediction, hereinafter referred to as W'). As an example (but not as limitation), an active W' channel that allows some kind of mixing of the X, Y, Z channels into the W channel may be defined as follows: <MAT>.

Here, f is the mixing factor and can be static or dynamic across time and/or frequency. In an implementation, f may vary between active and inactive frames. In other words, the mixing factor may be dependent on whether the current frame is an active frame or an inactive frame. In yet other words, the mixing of the X, Y and/or Z channel into the W channel may be different for active frames and for inactive frames. Hence, a representation of the W channel, i.e., the W' channel, may be determined by mixing the initial W channel with one or more of the other channels. By doing this, the perceptual quality may be further increased.

Remix the W and predicted (Y', Z', X') channels from most to least acoustically relevant. Remixing may correspond to reordering or re-combining signals, based on a defined methodology. An example methodology for remixing may be re-ordering of the input signals W, Y', X', Z', given the assumption that audio cues from left and right are more important than front-back, which are more important to up-down cues. The remix methodology may be known at the corresponding decoding unit <NUM>, such that no parameters need to be transmitted within the bitstream. Remixing may be performed within the remixing module <NUM> of <FIG>, thereby providing the remixed channels <NUM>.

Calculate the covariance of the <NUM> channel post-prediction and remixing downmix <NUM>. <MAT> <MAT> where dd represents the one or more extra downmix channels beyond W (i.e. the <NUM>nd to nth channels), and u represents the one or more channels that need to be wholly regenerated (i.e. n+<NUM>)th to <NUM>th channels), because they are not comprised within the downmix signal <NUM> (wherein n is the number of channels within the downmix signal <NUM>).

For the example of a WABC remix <NUM> with <NUM>-<NUM> channels, d and u represent the following channels:.

wherein n is the number of channels within the downmix signal <NUM>. Of main interest to the calculation of SPAR metadata <NUM> are the Rdd, Rud and Ruu covariances. The SPAR metadata may be determined within the calculation module <NUM> of <FIG>.

From this, determine whether it is possible to cross-predict any remaining portion of the fully parametric channels (i.e., the u channels) from the residual channels being sent (i.e., from the d channels).

The cross-prediction coefficients C may be calculated as follows: <MAT>.

Therefore, C may have the shape (<NUM>×<NUM>) for a <NUM>-channel downmix, and (<NUM>×<NUM>) for a <NUM>-channel downmix.

Calculate the remaining energy in parameterized channels that should be filled in by decorrelators <NUM>. The residual energy in the upmix channels Resuu is the difference between the actual energy Run (post-prediction) and the regenerated cross-prediction energy Reguu. <MAT> <MAT> <MAT>.

The scale parameter is a constant, with <NUM> ≤ scale ≤ <NUM>. Th scale parameter may be frequency dependent. In particular, different values of the scale parameter may be used for different frequency bands. The decorrelation parameter matrix P may be a covariance matrix, which is is Hermitian symmetric, and thus only decorrelation parameters from the upper or lower triangle of the decorrelation matrix may need to be included into the SPAR metadata <NUM>. The diagonal entries are real-valued, while the off-diagonal elements may be complex-valued. The P coefficients dictate how much decorrelated components of the W channel are used to recreate the A, B and C channels, before un-prediction and un-mixing is performed.

In some implementations, only diagonal values of P are computed and sent to the decoding unit <NUM>. These diagonal values may be computed as follows <MAT>.

<FIG> illustrates further components of the reconstruction module <NUM> of the decoding unit <NUM>. In particular, the reconstruction module <NUM> may comprise a remixing module <NUM> configured to remix the channels according to the above-mentioned remixing methodology. Furthermore, the prediction parameters may be used within the un-prediction module <NUM> to generate the channels of the reconstructed multi-channel signal <NUM> from the reconstructed residual channels.

The present document addresses the technical problem of modelling the spatial parameters of background noise in a voice communication system, such that the decoding unit <NUM> can generate high quality spatial comfort noise using these spatial parameters and a multi-channel excitation signal. The multi-channel excitation signal may be a multi-channel white noise signal where all channels are generated with different seed and are uncorrelated with each other.

For this purpose, the encoding unit <NUM> may be configured to sample the ambient noise and to encode one or more parameters which describe the ambient noise. The one or more parameters for the spatial aspect of the noise may be estimated, encoded, and sent during periods of "No voice activity" to the decoding unit <NUM>. The decoding unit <NUM> may be configured to recreate the noise to be played out based on the one or more parameters.

In other words, a method of encoding spatial background noise with a relatively small set of parameters is described, in order to allow transmission of the one or more parameters to the decoding unit <NUM>. The one or more parameters may be used to create spatial comfort noise at the receiving end of a transmission system. The term spatial means that the noise has ambiance associated with left/right, up/down, and/or front/back. The noise may be anisotropic such that there may be a stronger ambient noise in one direction than in another direction. The spatial noise may be constructed to be played out in conjunction with headtracking at the decoding unit <NUM>, when played out over loudspeakers.

The general methodology of generating spatial comfort noise at the decoder is addressed in <CIT>.

The present document addresses the technical problem on how to model spatial comfort noise at the encoding unit <NUM>. This document specifically addresses a procedure for modeling and/or quantizing one or more spatial parameters for spatial comfort noise at the encoding unit <NUM> so that spatial comfort noise can be generated at the decoding unit <NUM>.

The method described in this document characterizes the spatial characteristics of the ambient noise as being distinct from the mechanisms of characterizing the frequency spectrum of ambient noise. Specifically, this document describes the creation of one or more spatial comfort noise parameters when utilizing a mono codec comfort noise encoder, wherein the spatial comfort noise parameters characterize the spatial component of the noise. The spatial and spectral components may be used at the decoding unit <NUM> to generate spatial comfort noise.

An input audio signal <NUM> may be provided to the encoding unit <NUM>, wherein the input audio signal <NUM> comprises a series of frames. The frames may e.g., have a temporal length of <NUM>. The series of frames may comprise a subset of audio or voice frames and a subset of frames which consist only of background noise. An example sequence of audio frames may be considered
<IMG>
wherein "A" indicates an active speech and/or audio frame, and wherein "S" indicates a silence frame (also referred to herein as inactive frame).

For a discontinuous transmission (DTX) system, where the actual bitrate of the codec may be substantially reduced during inactive frames by only sending noise shaping parameters and by assuming that background noise characteristics do not change as frequent as active speech or audio frames, the above sequence may be translated into the following sequence of frames by the encoding unit <NUM>:
<IMG>
wherein "AB" indicates an encoder bitstream for an active frame, wherein "SID" indicates a silence indicator frame, which comprises a series of bits for comfort noise generation, and wherein "ND" indicates no data frames, i.e., nothing is transmitted to the decoding unit <NUM> during these frames.

Hence, the encoding unit <NUM> may be configured to classifying the different frames of the input signal <NUM> into active (A) or silent (S) frames (which are also referred to as inactive frames). Furthermore, the encoding unit <NUM> may be configured to determine and encode data for comfort noise generation within a "SID" frame (which corresponds e.g., to the current S frame of a series of S frames). The SID frames may be sent repeatedly, in particular periodically, for a series of S frames. By way of example, a SID frame may be sent every <NUM>th frame (which corresponds to a time interval of <NUM> between subsequent SID frames, when using <NUM> frames). No data may be transmitted during the one or more following S frames of the series of S frames. Hence, the encoding unit <NUM> may be configured to perform DTX (discontinuous transmission) or to switch to a DTX mode.

In other words, the encoding unit <NUM> may be configured to send audio data <NUM> and encoded metadata <NUM> to the decoding unit <NUM> for every active frame. On the other hand, the encoding unit <NUM> may be configured to send only encoded metadata <NUM> (and no audio data <NUM>) for a fraction of the inactive frames (i.e., for the SID frames). For the remaining inactive frames (i.e., for the ND frames), no data may be sent at all (not even encoded metadata <NUM>).

The encoded metadata <NUM> which is sent for a SID frame may be reduced and/or compressed with regards to the encoded metadata <NUM> which is sent for an active frame.

The encoding unit <NUM> may comprise a voice activity detector which is configured to switch the encoder to DTX mode. If the DTX flag (e.g., the CombinedVAD flag mentioned below) is set, then packets may be generated in a discontinuous mode based on an input frame, otherwise a frame may be coded as a speech and/or audio active frame.

The encoding unit <NUM> may be configured to determine a mono downmix signal <NUM> and the mono downmix signal <NUM> may be used to detect an inactive frame by operating a Signal Activity Detector or Voice Activity detector (SAD/VAD) on the mono downmix signal <NUM>. For the example of a soundfield B-format input signal <NUM>, the SAD/VAD may be operated on the representation of the W channel signal. In an alternative example, the SAD/VAD may be operated on multiple (notably all) channel signals of the input signal <NUM>. The individual results for the individual channel signals may then be combined into a single CombinedVAD flag. If the CombinedVAD flag is set, a frame may be considered to be inactive. On the other hand, if the CombinedVAD flag is not set, the frame may be considered to be active.

Hence a VAD and/or SAD may be used to classify the frames of a sequence of frames into active frames or inactive frames. Encoding and/or generating comfort noise may be applied to the inactive frames. The encoding of the comfort noise (notably the encoding of noise shaping parameters) within the encoding unit <NUM> may be performed such that the decoding unit <NUM> is enabled to generate high quality comfort noise for a soundfield. The comfort noise that is generated by the decoding unit <NUM> preferably matches the spectral and/or spatial characteristics of the background noise within the input signal <NUM>. This does not necessarily imply the waveform reconstruction of the input background noise. The comfort noise generated by a soundfield decoding unit <NUM> for a series of inactive frames is preferably such that the comfort noise sounds continuous with regards of the noise within the directly preceding active frames. Hence, the transition between active and inactive frames at the decoding unit <NUM> is preferably smooth and non-abrupt.

The decoding unit <NUM> may be configured to generate random white noise as an excitation signal. The excitation signal may comprise multiple channels of white noise, wherein the white noise in the different channels is typically uncorrelated from one another. The bitstream from the encoding unit <NUM> may only comprise noise shaping parameters (as encoded metadata <NUM>), and the decoding unit <NUM> may be configured to shape the random white noise within the different channels (spectrally and spatially) using the noise shaping parameters that have been provided within the bitstream. By doing this, spatial comfort noise may be generated in an efficient manner.

In the following, a method for determining spatial and/or spectral comfort noise parameters at the encoding unit <NUM> is described. As indicated above, an "active frame" may be a frame, within which a voice and/or audio signal is detected. A "non- or inactive frame" may be a frame, within which no voice and no speech is detected, such that only background noise is present. The frames may be classified into active and inactive frames using a voice and/or speech detection (VAD and/or SAD) algorithm.

WYZX may be the four channels of a B-format soundfield signal. The abbreviation CNG may refer to Comfort Noise Generation. The abbreviation SPAR may refer to SPAtial Reconstructor. As outlined in the context of <FIG>, the SPAR scheme may be an algorithm that takes an input <NUM> with N channels (preferably FOA input with N = <NUM>) and generates a downmix <NUM> such that the number n of downmix channels (dmx_ch) are <NUM>≤ n ≤N, as well as spatial metadata <NUM> (the so-called SPAR parameters). Furthermore, the SPAR scheme reconstructs a multi-channel signal <NUM> with N channels from the (reconstructed) downmix <NUM>, <NUM> with n channels <NUM>, using the set of SPAR parameters <NUM>.

The SPAR parameters <NUM> for an FOA input signal <NUM> may comprise PR (prediction coefficients), C (Cross term) coefficients, and P (decorrelator) coefficients. Ypred or Y' may be the residual of the Y channel signal after linear prediction from one or more other channel signals (notably from the W channel signal). Xpred or X' may be the residual of the X channel signal after linear prediction from one or more other channel signals (notably from the W channel signal). Zpred or Z' may be the residual of the Z channel signal after linear prediction from one or more other channel signals (notably from the W channel signal).

It should be noted that even though the present document primarily refers to the modelling of spatial noise in a soundfield signal, the methods which are described herein are not limited to soundfield signals and may be applied to any multiple dimension noise field and/or to any kind of multi-channel signal. In particular, the methods described herein may apply to channel and soundfield based encoders for input signals <NUM> having a dimensionality N of greater than <NUM> and preferably greater than <NUM> (i.e., more than stereo).

The method for encoding spatial comfort noise may comprise VAD and/or SAD for a frame of the mono downmix signal <NUM> (e.g., the W channel signal for a soundfield signal). The encoding of spatial comfort noise parameters may be performed, if the frame is detected to be an inactive frame.

The covariance of the different channel signals of the frame may be determined for a plurality of different frequency bands or subbands. The covariance matrix may be determined as outlined above, however for a reduced number of frequency bands. The smoothing of the covariance, i.e., the smoothing of the covariance matrix, may be performed across a plurality of subsequent frames. Hence, the covariance may be (low pass) filtered across multiple frames using a filter. Different filters may be used for the different frequency bands. In other words, different smoothing functions and/or different forgetting factors may be used in different frequency bands. Hence, temporal smoothing of the covariance may be performed. As a result of this, the data rate for encoding parameters for the background noise may be reduced. Furthermore, audible discontinuities of the spatial comfort noise may be avoided. An example of temporal covariance smoothing is described in <CIT>.

The covariance estimate Rdtx for comfort noise generation preferably makes use of relatively extensive covariance smoothing. By increasing the forgetting factor for comfort noise modelling, the perception of the spatial characteristics of the noise may be stabilized, thereby avoiding noise that sounds non-stationary which may be perceptually annoying. Hence, the smoothed covariance Rdtx for a frame may be determined, and the SPAR metadata <NUM> may be determined based on smoothed covariance Rdtx.

In order to allow for a smooth transition between an active frame and a subsequence inactive frame, the covariance calculation scheme may be switched from the normal covariance estimate (used for active frames) to the noise covariance estimate, as soon as the SAD and/or VAD detects an inactive frame.

A frame may comprise a relative short burst or transient (e.g., a voice burst or a knock). Such a transient may be relatively short, e.g., <NUM>. Such a burst may interrupt a sequence of inactive frames. The interruption would normally lead to a reset of the calculation of the covariance (because the short burst of active frames would be calculated using no smoothing or a modified smoothing scheme). The encoding unit <NUM> may be configured to remove frames that carry a transient or a short burst from the smoothing scheme for determining the covariance estimates. Once a transient or voice burst is detected, then the covariance estimate may be held constant until the burst or transient has ended. For the case, where the burst or transient is relatively short, e.g., <NUM> or less, the covariance smoothing may continue subsequent to the transient (based on a stored covariance prior to the burst or transient). Hence, covariance smoothing may be continued using a stored covariance for the frame that directly preceded the burst or transient. In other words, weighted averaging of the covariances may be continued by excluding the one or more active frames that correspond to a transient or burst.

On the other hand, if the burst lasts for a relatively long time, the covariance estimate may be reset and covariance estimates prior to the reset may not be used for averaging.

By doing this, it may be taken into account that the spatial aspects of background noise are relatively stable through time. By removing transients, knocks, bumps, ticks, etc. from the smoothed covariance calculation, perceivable disruptions of the spatial aspects of the background noise may be avoided in a reliable manner. In particular, a reset of the covariance due to the detection of a relatively short sequence of active frames, which may lead to a noticeable artifact, may be avoided.

The SPAR metadata <NUM> calculation may comprise the calculation of prediction coefficients (Pr), of cross-term coefficients C if <NUM>< n < <NUM>, and/or of decorrelator coefficients (P) (as outlined above), wherein n is the number of channels within the downmix signal <NUM>.

As indicated above, for an inactive frame, the covariance may be computed for a reduced number of bands compared to the case of an active frame (e.g., <NUM> bands instead of <NUM> bands). The assumption behind reducing the number of bands for inactive frames is that typically less frequency resolution is required for capturing noise parameters, due to the broadband nature of background noise. By reducing the number of bands, the data rate, i.e. the bitrate, may be reduced. Furthermore, only real valued coefficients may be sent from the encoding unit <NUM> to the decoding unit <NUM>. Hence, a reduced model comprising a reduced number of frequency bands may be used for modelling the background noise. This model allows modelling the spatial characteristics of broadband noise in an efficient manner.

For the case of a two channel downmix <NUM> (which comprises e.g., the representation of W channel signal and the Ypred or Y' channel signal), the data comprised within the bitstream from the encoding unit <NUM> to the decoding unit <NUM> may comprise (for a frame of the input signal <NUM>):.

The processing which may be performed by the encoding unit <NUM> in case of a two channel downmix signal <NUM> is illustrated in <FIG>.

For the case of a full parametric W downmix <NUM> (comprising only the W channel signal as the downmix signal <NUM>), the data comprised within the bitstream from the encoding unit <NUM> to the decoding unit <NUM> may comprise (for a frame of the input signal <NUM>):.

The processing which may be performed by the encoding unit <NUM> in case of a one channel downmix signal <NUM> is illustrated in <FIG>. <FIG> illustrates the case of a three channel downmix signal <NUM>.

For the case of a downmix to a single mono channel (for example a <NUM>-<NUM>-<NUM> or <NUM>-<NUM>-<NUM> downmix, indicating that a multi-channel input signal <NUM> with <NUM> or <NUM> channels is downmixed to <NUM> channel and upmixed to <NUM> or <NUM> channels), only audio data <NUM> regarding the mono (W') channel downmix signal <NUM> (which may be a representation of the W channel) is included into the bitstream. The SPAR metadata <NUM> may be used to generate a FOA spatial comfort noise based on the reconstructed downmix signal <NUM>. When the encoding unit <NUM> detects an inactive frame, the calculation of the SPAR metadata <NUM> is adapted, as outlined in the present document. In particular, the first inactive frame subsequent to an active frame may be encoded to provide a SID frame. For this purpose, the encoded SPAR metadata <NUM> may be provided within the bitstream for the first inactive frame. Furthermore, during a series of inactive frames, further SID frames may be provided repeatedly, in particular periodically.

For the one channel downmix example, the decoding unit <NUM> may be configured to generate a reconstructed downmix signal <NUM> based on the audio data <NUM>. This reconstructed downmix signal <NUM> may be referred to as WCNG, which, during inactive frames, may include a parametric reconstruction of background noise present in the uncoded representation of the W channel in the downmix using white noise as an excitation signal and using spectral shaping parameters coded by a mono audio codec (e.g., EVS). The three decorrelated channels for reconstructing the Y, X and Z channel signals may be generated from WCNG using decorrelators <NUM> (e.g., time domain or filterbank domain decorrelators). Alternatively, three decorrelated channels for reconstructing the Y, X and Z channel signals may be generated by generating uncorrelated comfort noise with different seeds and spectrally shaping the uncorrected comfort nois according to WCNG. The SPAR metadata <NUM> may be applied to WCNG and the decorrelated channels to generate comfort noise in a soundfield format, having the spectral and spatial characteristics of the original background noise.

For the two channel downmix example (<NUM>-<NUM>-<NUM>, for a first order soundfield), the comfort noise parameters for the mono dowmmix (W') channel and for one prediction channel may be provided to the decoding unit <NUM>. The decoding unit <NUM> may apply a method for generating FOA spatial comfort noise from a two channel downmix <NUM> and from the SPAR metadata <NUM>. The two downmix channels may be uncorrelated comfort noise signals, one having the spectrum shaped according to the original W channel representation and the other one having the spectrum shaped according to the original residual channel.

For the SID frames, two independent encoder module <NUM> instances encode spectral information regarding the mono (W') channel and spectral information regarding the residual channel, respectively. Furthermore, two independent instances of the decoding unit <NUM> may generate uncorrelated comfort noise signals with different seeds. The uncorrelated comfort noise signals may be spectrally shaped based on the representation of the W channel and the residual channel in the uncoded downmix, respectively. The reconstructed W channel may be referred to as WCNG and the reconstructed residual channel may be referred to as PCNG.

PCNG typically is a better approximation of the original uncorrelated residual channel compared to decorrelating WCNG and applying decorrelating coefficients (as done in the full parametric approach, which makes use of a single downmix channel only). As a result of this, the perceptual quality of the background noise is typically higher, when using a multi-channel downmix signal <NUM>.

The two decorrelated channels may be created by running WCNG through time domain or filterbank domain decorrelators or by generating uncorrelated comfort noise with different seed and by spectrally shaping the uncorrelated comfort noise channels as per WCNG. WCNG, PCNG comfort noise signals and the two decorrelated signals may then be upmixed to an FOA output using the SPAR metadata <NUM>.

Since the downmix signals <NUM> are continuously running with the same downmix configuration in active and inactive frames, background noise typically sounds smooth even during transition frames. Furthermore, since the decoding unit <NUM> is using the prediction coefficients and the decorrelation coefficients computed by the SPAR encoder <NUM>, spatial properties are replicated in the comfort noise which is generated by the SPAR decoder <NUM>.

The C coefficients or cross prediction coefficients may be set to <NUM>, thereby further reducing the data rate for inactive frames during DTX. An alternate implementation of comfort noise modelling may include the C parameters.

In an example implementation of the comfort noise modelling the upper or lower triangle parameters of the P matrix are not used. Only the real-valued diagonal elements of the P matrix may be used to model the comfort noise. In an alternative implementation of the comfort noise modelling the upper or lower triangle parameters of the P matrix may be taken into account.

For the case of a <NUM>-<NUM>-<NUM> downmix, mono codec CNG parameters may be generated and sent for the mono representation of the W downmix channel and for two prediction channels. The FOA spatial comfort noise may then be generated from a three channel downmix <NUM> and from the SPAR metadata <NUM>. The three downmix channels may be uncorrelated comfort noise signals, one having the spectrum shaped as uncoded W channel representation in the downmix and the other two having the spectrum shaped as uncoded Y residual channel and X residual channel in the downmix.

<FIG>, <FIG> and <FIG> illustrate a frame-by-frame calculation of the spatial comfort noise parameters when encoding an FOA input signal <NUM> with one, two, or three channel downmixes <NUM>, respectively. Y' indicates the residual after predicting the Y channel from the W channels. X' indicates the prediction of the X channel from the W channel.

In a preferred example, uniform quantization of the parameters Pr (prediction coefficients) and/or P (decorrelator coefficients) may be performed. The quantization scheme may depend on the direction of the noise. In particular, the number of quantization points which is allocated to the different channels may be dependent on the direction of the noise.

Quantization of the prediction coefficients (Pr) may be performed as follows:.

Quantization of the decorrelation coefficients (P) may be as follows;.

It should be noted that the notation N-n-N indicates the number N of channels of the input signal <NUM>, the number n of channels of the downmix signal <NUM> and the number N of channels of the reconstructed signal <NUM>.

<FIG> shows a flow chart of an example method <NUM> according to the claimed invention for encoding a multi-channel input signal <NUM> which comprises N different channels, with N being a positive integer greater than one, preferably N><NUM>. The multi-channel input signal <NUM> may be an SR signal, in particular an ambisonics signal, such as a FOA signal. The input signal <NUM> comprises a sequence of frames, wherein each frame may cover a certain temporal length of the input signal <NUM>, e.g., between <NUM> and <NUM>. A frame of the multi-channel input signal <NUM> typically comprises a block of multi-channel PCM data. In particular, a frame may comprise audio samples (at a certain sampling rate of e.g., <NUM> or more, or <NUM> or more, in particular <NUM>) for a certain temporal segment (of e.g., <NUM>) for each channel of the multi-channel input signal <NUM>.

The method <NUM> may be repeated for each frame of the sequence of frames. The method <NUM> comprises for a current frame of the sequence of frames, determining <NUM> whether the current frame is an active frame or an inactive frame using a signal and/or a voice activity detector. In other words, the current frame may be classified in being an active frame (which comprises a signal, notably speech) or being an inactive frame (which comprises (only) background noise). For this purpose, the samples of the current frame may be analyzed, e.g., with regards to the energy of the audio signal within the current frame and/or with regards to the spectral composition of the audio signal within the current frame. The SAD and/or VAD may be performed based on a single channel (e.g., the W channel) of the multi-channel input signal <NUM> or based on multiple, in particular all, channels of the multi-channel input signal <NUM>.

Furthermore, the method <NUM> comprises determining <NUM> a downmix signal <NUM> based on the multi-channel input signal <NUM> and/or based on the operating and/or target bitrate, wherein the downmix signal <NUM> typically comprises less than or equal to N channels. In particular, the downmix signal <NUM> comprises n channels, with typically n≤N, preferably n<N. The number n of channels of the downmix signal <NUM> may be equal to the number N of channels of the multi-channel input signal <NUM>, in particular for relatively high bit rates. The downmix signal <NUM> may be generated by selecting one or more channels from the multi-channel input signal <NUM>. The downmix signal <NUM> may e.g., comprise the W channel of a FOA signal. Furthermore, the downmix signal <NUM> may comprise one or more residual channels of the FOA signal (which may be derived using the prediction operations described herein).

The downmix signal <NUM>, in particular the number n of channels of the downmix signal <NUM>, is typically determined in dependence on the target data rate for the bitstream.

The method <NUM> further comprises determining <NUM> upmixing metadata <NUM>, in particular SPAR metadata, comprising a set of parameters. The upmixing metadata <NUM> may be determined such that it allows generating a reconstructed multi-channel signal <NUM> comprising N channels based on the downmix signal <NUM> (or based on a corresponding reconstructed downmix signal <NUM>). The set of parameters of the upmixing metadata <NUM> may describe and/or model one or more spatial characteristics of audio content, in particular of noise, comprised within the current frame of the multi-channel input signal <NUM>.

As indicated above, the multi-channel input signal <NUM> may comprise an ambisonics signal, notably an FOA signal, with a W channel, a Y channel, a Z channel and an X channel. The set of parameters of the upmixing metadata <NUM> may comprise prediction coefficients for predicting the Y channel, the Z channel and the X channel based on the W channel, thereby providing residual channels, referred to as Y' channel, Z' channel and X' channel, respectively. The prediction coefficients are referred to herein as Pr or PR. The downmix signal <NUM> may comprise a representation of W channel and one or more residual signals (in particular, the one or more residual signals having the highest energy).

If the downmix signal comprises a first residual channel in addition to the W channel representation, a cross-prediction parameter may be determined (possibly as part of the upmixing metadata <NUM>), wherein the cross-prediction parameter is dependent of the covariance between the one or more residual channels included in the downmix signal and the one or more remaining residual channels (which are not included in the downmix signal <NUM>). The one or more cross-prediction coefficients are referred to herein as C.

Furthermore, a decorrelation parameter for generating a decorrelated channel when reconstructing a residual channel that has not been included into the downmix signal <NUM> may be determined based on a covariance of the residual channel. The decorrelation parameter may be referred to herein as P and may be part of the upmixing metadata.

The upmixing metadata <NUM> are determined in dependance of whether the current frame is an active frame or an inactive frame. In particular, the set of parameters, which is comprised within the upmixing metadata <NUM> may depend on whether the current frame is an active frame or an inactive frame. If the current frame is an active frame, the set of parameters of the upmixing parameters <NUM> may be larger and/or may comprise a higher number of different parameters than if the current frame is an inactive frame.

In particular, the cross-prediction parameter may not be part of the upmixing metadata <NUM> for the current frame, if the current frame is an inactive frame. On the other hand, the cross-prediction parameter may be part of the upmixing metadata <NUM> for the current frame, if the current frame is an active frame.

Alternatively, or in addition, if more than one residual channel is included into the downmix signal <NUM>, the set of parameters of the upmixing metadata <NUM> for the current frame may comprise a decorrelation parameter for each possible combination of a non-included residual channel either with itself or with another one of the non-included residual channels, if the current frame is an active frame. On the other hand, the set of parameters of the upmixing metadata <NUM> for the current frame may comprise a decorrelation parameter only for the combination of a non-included residual channel with itself, if the current frame is an inactive frame.

Hence, the type of parameters which are included into the upmixing metadata <NUM> may be different for an active frame and for an inactive frame. In particular, one or more parameters which are less relevant for reconstructing the spatial characteristics of background noise may be omitted for an inactive frame. As a result of this, the data rate for encoding background noise may be reduced without impacting the perceptional quality.

The set of parameters may comprise corresponding parameters for a number of different frequency bands. In other words, the parameters of a given type (e.g., the Pr, the C and/or the P parameters) may be determined for a plurality of different frequency bands (also referred to herein as subbands). The number of different frequency bands, for which the parameters are determined, may depend on whether the current frame is an active frame or an inactive frame. In particular, if the current frame is an active frame, the number of different frequency bands may be higher than if the current frame is an inactive frame.

Hence, the frequency resolution of the set of parameters, which describes the spatial characteristics of the multi-channel input signal may be adapted in dependence of whether the current frame is an active frame or an inactive frame. In particular, a high frequency resolution may be used for an active frame than for an inactive frame. By doing this, the data rate for encoding background noise may be reduced without impacting the perceptional quality.

The method <NUM> may further comprise encoding <NUM> the upmixing metadata <NUM> into a bitstream (wherein the bitstream may be transmitted or provided to a corresponding decoding unit <NUM>). The set of parameters of the upmixing metadata <NUM> may be entropy encoded to provide coded metadata <NUM> to be inserted into the bitstream. As a result of this, an efficient encoding of spatial background noise is provided.

The method <NUM> further comprises encoding the downmix signal <NUM> into the bitstream, if, in particular only if, the current frame is an active frame. The one or more channels of the downmix signal <NUM> may be encoded individually using (one or more instances of) a single channel audio encoder (such as an EVS (enhanced voice services) encoder) to provide audio data <NUM> to be inserted into the bitstream.

The method <NUM> may comprise quantizing the parameters from the set of parameters for encoding <NUM> the upmixing metadata <NUM> for the current frame into the bitstream, using a quantizer. In other words, a quantizer may be used to quantize the set of parameters, which is to be encoded into the bitstream. The quantizer, in particular the quantization step size and/or the number of quantization steps of the quantizer, may be dependent on whether the current frame is an active frame or an inactive frame. In particular, the quantization step size may be lower and/or the number of quantization steps may be higher for an active frame than for an inactive frame. Alternatively, or in addition, the quantizer, in particular the quantization step size and/or the number of quantization steps of the quantizer, may be dependent on the number of channels of the downmix signal. By doing this, the efficiency of encoding spatial background noise at high perceptual quality may be further increased.

Alternatively, or in addition, the quantizer, in particular the quantization step size and/or the number of quantization steps of the quantizer, may be dependent on the channel and/or on the type of the channel, for which upmixing metadata <NUM> is encoded. Different channels and/or different types of channels may relate to different directions. By way of example, the quantization step size may be smaller and/or the number of quantization steps may be higher for the Y channel compared to the X channel and/or the Z channel, because left/right cues may be perceptually more important than front/back or up/down cues.

The method comprises determining a covariance, in particular a covariance matrix and/or a set of covariance values, for the current frame of the multi-channel input signal, by performing temporal smoothing over multiple frames from the sequence of frames. The temporal smoothing is performed in dependance of whether the current frame is an active frame or an inactive frame. By way of example, the forgetting factor for the temporal smoothing may be dependent on whether the current frame is an active frame or an inactive frame. In particular, if the current frame is an active frame, the forgetting factor may be shorter than if the current frame is an inactive frame. In other words, less temporal smoothing may be performed for an active frame compared to an inactive frame. In an example, the covariance for the current frame may be determined without temporal smoothing, if the current frame is an active frame. On the other hand, the covariance for the current frame may be determined with temporal smoothing, if the current frame is an inactive frame.

The forgetting factor may be frequency dependent. In particular, different forgetting factors, i.e., different values of the forgetting factor, may be used at least for some of the plurality of different frequency bands. Hence, the temporal smoothing of the covariance may be different for different frequencies and/or frequency bands. By doing this, the perceptual quality of the spatial background noise may be further increased.

More smoothing may be performed during inactive frames (compared to active frames) because spatial characteristics of background noise are typically assumed to be more stable as compared to active speech or general audio. This may also provide a more stable spatial comfort noise experience at the decoder output.

The forgetting factor may be dependent on the bandwidth of the different frequency bands. In particular, the forgetting factor for a frequency band may depend on the bandwidth of the frequency band. The forgetting factor may increase with the frequency resolution of the frequency bands (and vice-versa).

One or more parameters of the upmixing metadata (notably the one or more prediction parameters, the one or more cross-prediction coefficients and/or the one or more decorrelation parameters) are determined in a precise manner based on the covariance for the current frame.

Hence, temporal smoothing of the covariance (which is the basis for describing the spatial characteristics) may be performed for inactive frames comprising background noise. By doing this, the perceptual quality of the reconstructed background noise at the decoding unit <NUM> may be improved. Furthermore, the data rate efficiency may be improved (due to an improved performance of entropy encoding).

The method <NUM> may comprise identifying one or more frames from the sequence of frames, which correspond to a transient noise or talk burst. A transient noise and/or a talk burst may be acoustic events which last at the most for a pre-determined burst duration threshold. The burst duration threshold may e.g., be set between <NUM> and <NUM>, e.g., at <NUM>. It may be of perceptual importance to capture spatial parameters for a transient noise and/or talk burst without temporal smoothing so that the decoding unit <NUM> can reconstruct the transient noise and/or talk burst at its correct spatial location. However, it may also be of perceptual importance to maintain the covariance smoothing filter state to model spatial background noise (without the impact of transient noise and/or talk bursts), as abrupt changes in covariance of background noise may cause audible discontinuity and/or artifacts.

The method <NUM> may comprise ignoring the one or more identified frames (which comprise the transient noise and/or talk burst) when performing temporal smoothing, and/or removing the one or more identified frames from the temporal smoothing calculation for determining the covariance for the current frame. In other words, one or more intermediate active frames which comprise a relatively short transient noise and/or talk burst may be removed from the temporal smoothing operation of the covariance. As a result of this, the perceptual quality of the spatial background noise may be improved. As a result, abrupt changes in the spatial characteristics of the reconstructed spatial background noise at the decoding unit <NUM> can be avoided which results in improved perceptual quality.

The method <NUM> may comprise determining that the current frame is an active frame (directly) following a set of one or more preceding inactive frames from the sequence of frames by less than the pre-determined burst duration threshold. The covariance for the current frame may then be determined without taking into account the current frame. The covariance for the current frame may then be determined based on a reference covariance which has been determined based on the set of one or more preceding inactive frames. Hence, an active frame following a sequence of inactive frames may be ignored during temporal smoothing of the covariance, thereby increasing the perceptual quality of the spatial background noise at the decoding unit <NUM>.

Alternatively, the method <NUM> may comprise determining that the current frame is an active frame (directly) following a set of one or more preceding inactive frames from the sequence of frames by more than the pre-determined burst duration threshold. In other words, it may be determined that the current frame is not part of a relatively short transient and/or burst, but comprises speech and/or a signal. The covariance for the current frame may then be determined based on samples, in particular only based on samples, from the current frame. Hence, a reset of the covariance calculation may be performed, subsequent to detecting the onset of a speech and/or a signal within the multi-channel input signal <NUM>. As a result of this, the perceptual quality of the reconstructed multi-channel signal <NUM> may be improved.

The method <NUM> may comprise determining that the current frame is an inactive frame. In this case, the covariance for the current frame may be stored as the reference covariance for performing temporal smoothing for determining the covariance for one or more subsequent frames from the sequence of frames, which (directly) follow the current frame. By storing the covariance of an inactive frame as a reference covariance for the calculation of the covariance of one of more subsequent (inactive) frames, temporal smoothing may be performed in an efficient manner.

Furthermore, the method <NUM> may comprise, for the subsequent frame from the sequence of frames, which (directly) follows the current frame, and if the current frame is an inactive frame, determining whether the subsequent frame is an active frame or an inactive frame using the signal and/or the voice activity detector. If the subsequent frame is an inactive frame, the covariance for the subsequent frame may be determined based on samples of the current frame, in particular based on the covariance for the current frame, and based on samples of the subsequent frame (while taking into account a forgetting factor). By doing this, temporal smoothing may be performed in a reliable and efficient manner.

The method <NUM> may comprise determining a covariance, in particular a covariance matrix, for the current frame of the multi-channel input signal <NUM> within a number of different frequency bands. As indicated above, the number of frequency bands may be dependent on whether the current frame is an active frame or an inactive frame. In particular, if the current frame is an active frame, the number of frequency bands may be higher than if the current frame is an inactive frame. One or more parameters of the upmixing metadata <NUM> may then be determined based on the covariance for the current frame. In particular, a plurality of parameters may be determined for the plurality of frequency bands based on the corresponding plurality of covariances for the plurality of frequency bands. As a result of this, the frequency-dependent parameters may be determined in a reliable and precise manner.

If it is determined that the current frame is an inactive frame, it may be determined whether a preceding frame from the sequence of frames, which directly precedes the current frame, has been an inactive frame. In other words, it may be determined whether the current frame is a SID (silence indicator) frame or an ND (no data) frame. If the preceding frame has been an inactive frame and the number of consecutive inactive frames since the last SID frame is less than a pre-computed SID frame interval (i.e., the current frame is an ND frame), it may be refrained from encoding <NUM> the downmix signal <NUM> and/or the upmixing metadata <NUM> for the current frame into the bitstream, thereby further reducing the data rate for signaling spatial background noise. On the other hand, if the preceding frame has been an active frame or the number of consecutive inactive frames since the last SID frame is equal to the pre-computed SID frame interval (i.e., the current frame is a SID frame), the downmix signal <NUM> and the upmixing metadata <NUM> for the current frame may be encoded into the bitstream, thereby providing a precise signaling of spatial background noise.

If the current frame is an active frame, each channel of the downmix signal <NUM> may be encoded individually using an instance of a mono audio encoder (such as EVS), wherein the mono audio encoder may be configured to encode the audio signal within a channel of the downmix signal <NUM> into an (encoded) excitation signal and into (encoded) spectral data.

If the current frame is an inactive (SID) frame, each channel of the downmix signal <NUM> may be encoded individually using an instance of a mono audio encoder (such as EVS). However, in this case, the (encoded) excitation signal may be not be inserted into the bitstream. On the other hand, the (encoded) spectral data may be inserted into the bitstream. As a result of this, spectral data regarding the spectral characteristics of the background noise may be inserted into the bitstream in an efficient manner.

At the corresponding decoding unit <NUM>, the excitation signals of the one or more channels of the downmix signal <NUM> may be replaced by white noise (wherein the white noise for the different channels is typically independent from one another). Furthermore, the (encoded) spectral data may be used to add the spectral characteristics to the one or more channels of the downmix signal <NUM> to provide the reconstructed downmix signal <NUM>.

The method <NUM> may comprise variable rate DTX, wherein the frequency of transmission of SID frames is variable. The frequency of transmission of SID frames may vary in dependence of the change in spatial and/or spectral characteristics of the background noise or signal-to-noise ratio of the input signal <NUM>. As outlined above, the frames of the input signal <NUM> may be classified into active frames or inactive frames. The sequence of frames of the input signal <NUM> may comprise a subsequence of inactive frames. In order to provided information regarding the spatial and/or spectral characteristics of the noise comprised within the subsequence of inactive frames, the encoding unit <NUM> may be configured to repeatedly, in particular periodically, insert SID frames in the bitstream. Each SID frame may comprise the upmixing metadata <NUM> described in the present document. The frequency or repetition rate for the SID frames may be adapted in dependence of the input signal <NUM>.

In particular, the current inactive frame may be analyzed, in order to determine whether a chance in spatial and/or spectral characteristic of the noise within the current inactive frame has occurred with respect to the previous inactive frame. By way of example, it may be determined whether the value of a distance measure between the upmixing metadata <NUM> for the current inactive frame and the upmixing metadata <NUM> of the previous inactive frame is greater than a pre-determined distance threshold. If this is the case, a SID frame may be inserted for the current inactive frame, in order to signal the changed upmixing metadata <NUM> to the decoding unit <NUM>. If, on the other hand, the value of the distance measure is smaller than the distance threshold, the current inactive frame may be treated as an ND frame.

As outlined above, an input audio signal <NUM> may be provided to the encoding unit <NUM>, wherein the input audio signal <NUM> comprises a series of frames. The frames may e.g., have a temporal length of <NUM>. The series of frames may comprise a subset of audio or voice frames and a subset of frames which consist only of background noise. An example sequence of audio frames may be considered
<IMG>
wherein "A" indicates an active speech and/or audio frame, and wherein "S" indicates a silence frame (also referred to herein as inactive frame) and "ST" indicates a silence transmitted frame, for which a change in spectral and/or spatial characteristic of background noise is detected and hence spatial and/or spectral parameters are to be coded and sent to the decoding unit <NUM>.

For a discontinuous transmission (DTX) system, for which the actual bitrate of the codec is significantly reduced during inactive frames by only sending noise shaping parameters and assuming that background noise characteristics do not change as frequent as active speech or audio frames, the above sequence may be translated into the following sequence of frames by the encoding unit <NUM>:
<IMG>
wherein "AB" indicates an encoder bitstream for an active frame, wherein "SID" indicates a silence indicator frame, which comprises a series of bits for comfort noise generation, and wherein "ND" indicates no data frames, i.e., nothing is transmitted to the decoding unit <NUM> during these frames. Note that the frequency of transmission of SID frames in the above sequence is not pre-determined and is dependent on change in spectral and/or spatial characteristics of input background noise.

Hence, the method <NUM> may comprise, if the current frame is an inactive frame, detecting a change in spatial and/or spectral characteristics of the background noise based on a change in the covariance matrix for the current frame. A SID frame may only be sent, if the change is greater than a pre-determined threshold.

In other words, the method <NUM> may comprise determining that the current frame is an inactive frame following a subsequence of one or more previous inactive frames (which is directly preceding the current frame). Furthermore, the method <NUM> may comprise determining whether a spatial and/or spectral characteristic of the background noise comprised within the current frame and/or whether a signal-to-noise ratio of the current frame has changed with regards to the corresponding spatial and/or spectral characteristic of the background noise in the subsequence of one or more previous inactive frames and/or with regards to the signal-to-noise ratio of the subsequence of one or more previous inactive frames.

The method <NUM> may further comprise encoding <NUM> the upmixing metadata <NUM> for the current frame into the bitstream, if, in particular only if, it is determined that the spatial and/or spectral characteristic of background noise comprised within the current frame and/or the signal-to-noise ratio of the current frame has changed with regards to the subsequence of one or more previous inactive frames. Hence, the current frame may be encoded as a SID frame if, in particular, only if, the spatial and/or spectral characteristic of background noise comprised within the current frame and/or the signal-to-noise ratio of the current frame has changed.

Alternatively, or in addition, the method <NUM> may comprise determining that the current frame is an inactive frame following one or more previous inactive frames. In addition, the method <NUM> may comprise determining a value of a distance measure (e.g., a mean square error) between the covariance and/or the upmixing metadata <NUM> for the current frame and a previous covariance and/or previous upmixing metadata <NUM> for the one or more previous inactive frames. In other words, it may be determined by how much the covariance for the current frame deviates from the corresponding previous covariance for the one or more previous inactive frames, and/o by how much the upmixing metadata <NUM> for the current frame deviates from the previous upmixing metadata <NUM> for the one or more previous inactive frames. The previous upmixing metadata <NUM> may be the upmixing metadata that has been sent in the last SID frame. The previous covariance may be the covariance that has been used for generating the previous upmixing metadata <NUM>.

The method <NUM> may further comprise determining whether the value of the distance measure is greater than a pre-determined distance threshold. Encoding <NUM> the upmixing metadata <NUM> for the current frame into the bitstream may be performed, if, in particular only if, the value of the distance measure is greater than the pre-determined distance threshold. Alternatively, it may be refrained from encoding <NUM> the upmixing metadata <NUM> for the current frame into the bitstream, if, in particular only if, the value of the distance measure is smaller than the pre-determined distance threshold.

Hence, the transmission of background noise parameters (i.e., of upmixing metadata <NUM>) may be non-periodic and may be dependent on a change in spatial and/or spectral characteristics of the input background noise and/or of the signal-to-noise ratio. In particular, only if the change in spatial and/or spectral characteristics and/or in signal-to-noise ratio is higher than a pre-determined threshold, background noise parameters may be updated and sent to the decoding unit <NUM>.

By enabling an adaptive transmission of SID frames, the data rate can be reduced further without impacting the perceptual quality of comfort noise.

The corresponding decoding method may comprise, if the current frame is an inactive frame, determining at the decoding unit <NUM> if the current frame is a SID frame or NO_DATA frames. Crossfading the spatial background noise parameters in the current SID frame with the spatial background noise parameters in the one or more previous SID frame(s) over a number of NO_DATA frames. For e.g., if Pcurr_sid are parameters for the current SID frame, and Pprev_sid are the parameters for the previous SID frame, then the change in background noise parameters Pcurr_sid - Pprev_sid may be applied over a number x of frames (with x><NUM>), wherein spatial parameters in every subsequent frame are changed by (Pcurr_sid - Pprev_sid)/x. This ensures a smooth transition of spatial comfort noise from one spatial location to another.

The method <NUM> may comprise, if the (directly) preceding frame of the current frame has been an active frame, determining the number of channels of the downmix signal <NUM> for the preceding claim and maintaining the same number of channels of the downmix signal <NUM> for the current frame, as for the downmix signal <NUM> for the preceding claim. In other words, the number of channels within the downmix signal <NUM> may be maintained during a transition between an active frame and an inactive frame. As a result of this, the perceptual quality of spatial background noise may be further increased.

As outlined above, the multi-channel input signal <NUM> may comprise an ambisonics signal with a W channel, a Y channel, a Z channel and an X channel. In particular, the multi-channel input signal <NUM> may be an FOA signal. The method <NUM> may comprise mixing the W channel with the Y channel, the Z channel and/or the X channel using a mixing factor f, to generate a channel of the downmix signal <NUM>. As a result of the mixing process, a representation of the W channel may be obtained, referred to herein as the W' channel. By mixing information from the Y, Z and/or X channel into the W channel, a modified W' channel is obtained, which comprises more spatial information than the original W channel. As a result of this, the perceptual quality of the encoding scheme may be improved and/or the number of downmix channels within the downmix signal <NUM> may be reduced.

The mixing factor (which defines the extent to which the Y, Z and/or X channels are mixed into the W channel) may be dependent on whether the current frame is an active fame or an inactive frame. In particular, if the current frame is an active frame, the mixing factor may be higher than if the current frame is an inactive frame. In other words, the Y, Z and/or X channels may be mixed into the W channel at a higher extent for active frames than for inactive frames. By doing this, the perceptual quality of the encoding scheme may be further improved.

<FIG> shows a flow chart of an example method <NUM> for decoding a bitstream which is indicative of a reconstructed multi-channel signal <NUM> comprising N channels. The bitstream may comprise audio data <NUM> and/or encoded upmixing metadata <NUM>. The reconstructed signal <NUM> comprises a sequence of frames.

The method <NUM> comprises, for a current frame of the sequence of frames, comprise determining <NUM>, based on the bitstream, whether the current frame is an active frame or an inactive frame. This may e.g., be determined based on the encoded upmixing metadata <NUM> comprised within the bitstream. Furthermore, the method <NUM> may comprise determining <NUM> a reconstructed downmix signal <NUM>. The reconstructed downmix signal <NUM> typically comprises less than N channels (but may comprise N channels). The reconstructed downmix signal <NUM> may be generated based on audio data <NUM> comprised within the bitstream, if the current frame is an active frame. On the other hand, the bitstream typically does not comprise audio data <NUM> for an inactive frame. The reconstructed downmix signal <NUM> may be generated using a random noise generator, if the current frame is an inactive frame. Statistically independent noise generators may be used for the one or more different channels of the reconstructed downmix signal <NUM>.

The method <NUM> further comprises generating <NUM> the reconstructed multi-channel signal <NUM> based on the reconstructed downmix signal <NUM> and based on the upmixing metadata <NUM> comprising a set of parameters. The reconstructed multi-channel signal <NUM> may be generated in dependance of whether the current frame is an active frame or an inactive frame.

In particular, the method <NUM> may comprise generating a decorrelated channel for a channel of the reconstructed multi-channel signal <NUM> that is not included in the reconstructed downmix signal <NUM>. The decorrelated channel may be generated based on the reconstructed downmix signal <NUM> (notably based on the reconstructed W channel). Furthermore, the method <NUM> may comprise adding a spatial property to the decorrelated channel based on the upmixing metadata <NUM>, to generate a reconstructed channel of the reconstructed multi-channel signal <NUM>, other than the one or more reconstructed channels comprised within the reconstructed downmix signal <NUM>.

By adapting the reconstruction of the reconstructed multi-channel signal <NUM>, notably of one or more reconstructed channels of the multi-channel signal <NUM>, in dependence of whether a frame of the signal <NUM> is an active frame or an inactive frame, high quality, spatial background noise may be provided.

As outlined in the context of the corresponding encoding method <NUM>, upmixing metadata <NUM> is typically only transmitted for a fraction of the inactive frames. The decoding unit <NUM> may be configured to use the upmixing metadata <NUM> that has been transmitted within a SID frame for a plurality of inactive frames following the SID frame. In particular, the upmixing metadata <NUM> may be used until a new SID frame is received. By doing this, high quality comfort noise may be generated in a data rate efficient manner.

The method <NUM> may comprise determining that the current frame is an inactive frame following one or more previous inactive frames, wherein previous upmixing metadata <NUM> was used for generating the reconstructed multi-channel signal <NUM> for the one or more previous inactive frames. Furthermore, the method <NUM> may comprise determining that the bitstream comprises updated upmixing metadata <NUM> for the current frame. In other words, it may be determined that the current frame is a SID frame (which comprises updated upmixing metadata <NUM>, for signaling an updated spatial and/or spectral characteristic of the background noise).

The reconstructed multi-channel signal <NUM> (i.e., the spatial background noise) for the current frame may then be generated based on the updated upmixing metadata <NUM>. By receiving SID frames with updated upmixing metadata <NUM>, changes in the background noise at the encoding unit <NUM> may be tracked in an efficient manner.

The method <NUM> may comprise performing metadata smoothing based on the updated upmixing metadata <NUM> and the previous upmixing metadata <NUM>, to determine smoothed upmixing metadata <NUM>. Metadata smoothing may comprise crossfading from the previous upmixing metadata <NUM> to the updated upmixing metadata <NUM> within one or more inactive frames starting with the current frame. The reconstructed multi-channel signal <NUM> for the current frame may then be determined based on the smoothed upmixing metadata <NUM>.

In other words, spatial metadata smoothing may be performed by the decoding unit <NUM> and the spatial parameters sent in a current SID frame may be crossfaded with spatial parameters sent in one or more previous SID frames over a number of NO_DATA (ND) frames.

Hence, a smooth transition between the previous spatial and/or spectral characteristics to the updated spatial and/or spectral characteristics of the background noise may be performed within one or more frames. By doing this, the perceptual quality of comfort noise may be further increased.

Furthermore, an encoding unit <NUM> corresponding to the encoding method <NUM> and/or a decoding unit <NUM> corresponding to the decoding method <NUM> are described in the present document.

Hence, a method <NUM> of using spatial parameters <NUM> and same or different downmixes <NUM> used for active frames to model spatial characteristics of noise are described, thereby allowing comfort noise generation at the decoder <NUM> that is spatially consistent between active and non-active frames. The method <NUM> may comprise determining whether a voice signal is present in one or more frames of an audio input <NUM>. In response to determining that no voice signal is present, a covariance may be estimated using frame to frame averaging. Furthermore, spatial noise parameters <NUM> may be calculated and entropy coding of the spatial noise parameters <NUM> may be performed. The entropy coded spatial noise parameters <NUM> may be packed into the bitstream for the one or more frames.

The method <NUM> may comprise, in response to detecting transients in a frame of the one or more frames, removing the frame from covariance averaging. Calculating the spatial noise parameters <NUM> may be performed with a smoothed covariance estimation that smoothens across multiple frames to avoid spatial variability in the noise. The method <NUM> may comprise smoothing covariance across transients and short talk bursts and removing these from the calculation. Alternatively, or in addition, the method <NUM> may comprise using a limited set of bands and/or limited set of parameters to reduce parameter bit rate for noise and switching back to a full set when audio is present. Alternatively, or in addition, the method <NUM> may comprise calculating spatial elements separately from spectral elements of the noise to allow re-use of existing comfort noise generators.

<FIG> is a mobile device architecture for implementing the features and processes described in reference to <FIG>. The mobile device may e.g., comprise the decoding unit <NUM>. Architecture <NUM> can be implemented in any electronic device, including but not limited to: a desktop computer, consumer audio/visual (AV) equipment, radio broadcast equipment, mobile devices (e.g., smartphone, tablet computer, laptop computer, wearable device). In the example embodiment shown, architecture <NUM> is for a smart phone and includes processor(s) <NUM>, peripherals interface <NUM>, audio subsystem <NUM>, loudspeakers <NUM>, microphone <NUM>, sensors <NUM> (e.g., accelerometers, gyros, barometer, magnetometer, camera), location processor <NUM> (e.g., GNSS receiver), wireless communications subsystems <NUM> (e.g., Wi-Fi, Bluetooth, cellular) and I/O subsystem(s) <NUM>, which includes touch controller <NUM> and other input controllers <NUM>, touch surface <NUM> and other input/control devices <NUM>. Other architectures with more or fewer components can also be used to implement the disclosed embodiments.

Memory interface <NUM> is coupled to processors <NUM>, peripherals interface <NUM> and memory <NUM> (e.g., flash, RAM, ROM). Memory <NUM> stores computer program instructions and data, including but not limited to: operating system instructions <NUM>, communication instructions <NUM>, GUI instructions <NUM>, sensor processing instructions <NUM>, phone instructions <NUM>, electronic messaging instructions <NUM>, web browsing instructions <NUM>, audio processing instructions <NUM>, GNSS/navigation instructions <NUM> and applications/data <NUM>. Audio processing instructions <NUM> include instructions for performing the audio processing described in reference to <FIG>.

Aspects of the systems described herein may be implemented in an appropriate computer-based sound processing network environment for processing digital or digitized audio files. Portions of the adaptive audio system may include one or more networks that comprise any desired number of individual machines, including one or more routers (not shown) that serve to buffer and route the data transmitted among the computers. Such a network may be built on various different network protocols, and may be the Internet, a Wide Area Network (WAN), a Local Area Network (LAN), or any combination thereof.

One or more of the components, blocks, processes or other functional components may be implemented through a computer program that controls execution of a processor-based computing device of the system. It should also be noted that the various functions disclosed herein may be described using any number of combinations of hardware, firmware, and/or as data and/or instructions embodied in various machine-readable or computer-readable media, in terms of their behavioral, register transfer, logic component, and/or other characteristics. Computer-readable media in which such formatted data and/or instructions may be embodied include, but are not limited to, physical (non-transitory), non-volatile storage media in various forms, such as optical, magnetic or semiconductor storage media.

Claim 1:
A method (<NUM>) for encoding a multi-channel input signal (<NUM>) which comprises N different channels; wherein the input signal (<NUM>) comprises a sequence of frames; wherein the method (<NUM>) comprises, for a current frame of the sequence of frames,
- determining (<NUM>) whether the current frame is an active frame or an inactive frame using a signal and/or a voice activity detector;
- determining (<NUM>) a downmix signal (<NUM>) based on the multi-channel input signal (<NUM>); wherein the downmix signal (<NUM>) comprises less than or equal to N channels;
- determining (<NUM>) upmixing metadata (<NUM>) comprising a set of parameters for generating, based on the downmix signal (<NUM>), a reconstructed multi-channel signal (<NUM>) comprising N channels; wherein the upmixing metadata (<NUM>) is determined in dependance of whether the current frame is an active frame or an inactive frame; and
- encoding (<NUM>) the upmixing metadata (<NUM>) into a bitstream,
wherein the method (<NUM>) further comprises
- determining a covariance, in particular a covariance matrix, for the current frame of the multi-channel input signal (<NUM>), by performing temporal smoothing over multiple frames from the sequence of frames, in dependance of whether the current frame is an active frame or an inactive frame; and
- determining one or more parameters of the upmixing metadata (<NUM>) based on the covariance for the current frame.