Patent Description:
The present disclosure generally relates to audio signal processing. In particular, the present disclosure relates to methods of processing a compressed representation of a spatial audio signal for generating a reconstructed representation of the spatial audio signal.

Human hearing enables listeners to perceive their environment in the form of a spatial audio scene - whereby the term "spatial audio scene" is used here to refer to the acoustic environment around a listener, or the perceived acoustic environment in the mind of the listener.

While the human experience is attached to spatial audio scenes, the art of audio recording and reproduction involves the capture, manipulation, transmission and playback of audio signals, or audio channels. The term "audio stream" is used to refer to a collection of one or more audio signals, particularly where the audio stream is intended to represent a spatial audio scene.

An audio stream may be played back to a listener, via electro-acoustic transducers or by other means, to provide one or more listeners with a listening experience in the form of a spatial audio scene. It is commonly a goal of audio recording practitioners and audio artists to create audio streams that are intended to provide a listener with the experience of a specific spatial audio scene.

An audio stream may be accompanied by associated data, referred to as metadata, that assists in the playback process. The accompanied metadata may include time-varying information that may be used to affect modifications in the processing that is applied during the playback process. In the following, the term "captured audio experience" may be used to refer to an audio stream plus any associated metadata.

In some applications, the metadata consists solely of data indicative of the intended loudspeaker arrangement for playback. Often, this metadata is omitted, on the assumption that the playback speaker arrangement is standardized. In this case, the captured audio experience consists solely of an audio stream. An example of one such captured audio experience is a <NUM>-channel audio stream, recorded on a compact disc, where the intended playback system is assumed to be in the form of two loudspeakers arranged in front of the listener.

Alternatively, a captured audio experience in the form of a scene-based multichannel audio signal may be intended for presentation to a listener by processing the audio signals, via a mixing matrix, so as to generate a set of speaker signals, each of which may be subsequently played back to a respective loudspeaker, wherein the loudspeakers may be arbitrarily arranged spatially around the listener. In this example, the mixing matrix may be generated based on prior knowledge of the scene-based format and the playback speaker arrangement.

An example of a scene-based format is Higher Order Ambisonics (HOA), and an example method for computing suitable mixing matrices is given in "<NPL>.

Typically, such scene-based formats include a large number of channels or audio objects, which leads to comparatively high bandwidth or storage requirements when transmitting or storing spatial audio signals in these formats.

Thus, there is a need for compact representations of spatial audio signals representing spatial audio scenes. This applies to both channel-based and object-based spatial audio signals.

<CIT> describes a frequency-domain spatial audio coding framework based on the perceived spatial audio scene rather than on the channel content. In one embodiment, time-frequency spatial direction vectors are used as cues to describe the input audio scene.

<CIT> describes a method for spatial audio signal processing, comprising: determining, for two or more playback audio signals, at least one spatial audio parameter for providing spatial audio reproduction; determining between the two or more playback audio signals at least one audio signal relationship parameter, the at least one audio signal relationship parameter being associated with a determination of inter-channel signal relationship information between the two or more playback audio signals and for at least two frequency bands, such that the two or more playback audio signals are configured to be reproduced based on the at least one spatial audio parameter and the at least one audio signal relationship parameter.

<CIT> describes spatial audio signal processing where at least one audio signal is received before determining at least one spatial parameter associated with the at least one audio signal and generating an adaptive smoothing parameter based on the at least one spatial parameter. This is followed by determining panning gains for applying to a first part of the at least one audio signals, applying the adaptive smoothing parameter to the panning gains to generate associated smoothed panning gains, before applying the smoothed panning gains to the first part of the at least one audio signal to generate a positioned audio signal. Preferably decorrelation is applied to a second part of the audio signal to generate an ambient signal. Preferably the adaptive smoothing parameter is based upon a direction smoothness parameter. Preferably the direction smoothness parameter is averaged based on the energy of the audio signal to generate the adaptive smoothing parameter.

The present disclosure proposes methods of processing a compressed representation of a spatial audio signal for generating a reconstructed representation of the spatial audio signal, corresponding apparatus, programs, and computer-readable storage media.

Parts of the disclosure not forming part of the invention relate to a method of processing a spatial audio signal for generating a compressed representation of the spatial audio signal. The spatial audio signal may be a multichannel signal or an object-based signal, for example. The compressed representation may be a compact or size-reduced representation. The method may include analyzing the spatial audio signal to determine directions of arrival for one or more audio elements in an audio scene (spatial audio scene) represented by the spatial audio signal. The audio elements may be dominant audio elements. The (dominant) audio elements may relate to (dominant) acoustic objects, (dominant) sound sources, or (dominant) acoustic components in the audio scene, for example. The one or more audio elements may include between one and ten audio elements, such as four audio elements, for example. The directions of arrival may correspond to locations on a unit sphere indicating the perceived locations of the audio elements. The method may further include, for at least one frequency subband (e.g., for all frequency subbands) of the spatial audio signal, determining respective indications of signal power associated with the determined directions of arrival. The method may further include generating metadata including direction information and energy information, with the direction information including indications of the determined directions of arrival of the one or more audio elements and the energy information including respective indications of signal power associated with the determined directions of arrival. The method may further include generating a channel-based audio signal with a predefined number of channels based on the spatial audio signal. The channel-based audio signal may be referred to as an audio mixture signal or audio mixture stream. It is understood that the number of channels of the channel-based audio signal may be smaller than the number of channels or the number of objects of the spatial audio signal. The method may yet further include outputting, as the compressed representation of the spatial audio signal, the channel-based audio signal and the metadata. The metadata may relate to a metadata stream.

Thereby, a compressed representation of a spatial audio signal can be generated that includes only a limited number of channels. Still, by appropriate use of the direction information and energy information, a decoder can generate a reconstructed version of the original spatial audio signal that is a very good approximation of the original spatial audio signal as far as the representation of the original spatial audio scene is concerned.

Analysing the spatial audio signal may be based on a plurality of frequency subbands of the spatial audio signal. For example, the analysis may be based on the full frequency range of the spatial audio signal (i.e., the full signal). That is, the analysis may be based on all frequency subbands.

Analysing the spatial audio signal may involve applying scene analysis to the spatial audio signal. Thereby, the (directions of) the dominant audio elements in the audio scene can be determined in a reliable and efficient manner.

The spatial audio signal may be a multichannel audio signal. Alternatively, the spatial audio signal may be an object-based audio signal. In this case, the method may further include converting the object-based audio signal to a multichannel audio signal prior to applying the scene analysis. This allows to meaningfully apply scene analysis tools to the audio signal.

An indication of signal power associated with a given direction of arrival may relate to a fraction of signal power in the frequency subband for the given direction of arrival in relation to the total signal power in the frequency subband.

The indications of signal power may be determined for each of a plurality of frequency subbands. In this case, they may relate, for a given direction of arrival and a given frequency subband, to a fraction of signal power in the given frequency subband for the given direction of arrival in relation to the total signal power in the given frequency subband. Notably, the indications of signal power may be determined in a per-subband manner, whereas the determination of the (dominant) directions of arrival may be performed on the full signal (i.e., based on all frequency subbands).

Analysing the spatial audio signal, determining respective indications of signal power, and generating the channel-based audio signal may be performed on a per-time-segment basis. Accordingly, the compressed representation may be generated and output for each of a plurality of time segments, with a downmixed audio signal and metadata (metadata block) for each time segment. Alternatively or additionally, analyzing the spatial audio signal, determining respective indications of signal power, and generating the channel-based audio signal may be performed based on a time-frequency representation of the spatial audio signal. For example, the aforementioned steps may be performed based on a discrete Fourier transform (such as a STFT, for example) of the spatial audio signal. That is, for each time segment (time block), the aforementioned steps may be performed based on the time-frequency bins (FFT bins) of the spatial audio signal, i.e., on the Fourier coefficients of the spatial audio signal.

The spatial audio signal may be an object-based audio signal that includes a plurality of audio objects and associated direction vectors. Then, the method may further include generatingthe multichannel audio signal by panning the audio objects to a predefined set of audio channels. Therein, each audio object may be panned to the predefined set of audio channels in accordance with its direction vector. Further, the channel-based audio signal may be a downmix signal generated by applying a downmix operation to the multichannel audio signal. The multichannel audio signal may be a Higher Order Ambisonics signal, for example.

The spatial audio signal may be a multichannel audio signal. Then, the channel-based audio signal may be a downmix signal generated by applying a downmix operation to the multichannel audio signal.

An aspect of the disclosure relates to a method of processing a compressed representation of a spatial audio signal for generating a reconstructed representation of the spatial audio signal. The compressed representation includes a channel-based audio signal with a predefined number of channels and metadata. The metadata includes direction information and energy information. The direction information includes indications of directions of arrival of one or more audio elements in an audio scene (spatial audio scene). The energy information Z r includes, J for at least one frequency subband, respective indications of signal power associated with the directions of arrival. The method includes generating audio signals of the one or more audio elements based on the channel-based audio signal, the direction information, and the energy information. The method further includes generating a residual audio signal from which the one or more audio elements are substantially absent, based on the channel-based audio signal, the direction information, and the energy information. The residual signal may be represented in the same audio format as the channel-based audio signal, e.g., may have the same number of channels.

In some embodiments, an indication of signal power associated with a given direction of arrival may relate to a fraction of signal power in the frequency subband for the given direction of arrival in relation to the total signal power in the frequency subband.

In some embodiments, the energy information may include indications of signal power for each of a plurality of frequency subbands. Then, an indication of signal power may relate, for a given direction of arrival and a given frequency subband, to a fraction of signal power in the given frequency subband for the given direction of arrival in relation to the total signal power in the given frequency subband.

In some embodiments, the method may further include panning the audio signals of the one or more audio elements to a set of channels of an output audio format. The method may yet further include generating a reconstructed multichannel audio signal in the output audio format based on the panned one or more audio elements and the residual signal. The output audio format may relate to an output representation, for example, such as HOA or any other suitable multichannel format. Generating the reconstructed multichannel audio signal may include upmixingthe residual signal to the set of channels of the output audio format. Generating the reconstructed multichannel audio signal may further include adding the panned one or more audio elements and the upmixed residual signal.

According to the invention, generating audio signals of the one or more audio elements includes determining coefficients of an inverse mixing matrix M for mapping the channel-based audio signal to an intermediate representation including the residual audio signal and the audio signals of the one or more audio elements, based on the direction information and the energy information. The intermediate representation may also be referred to as a separated or separable representation, or a hybrid representation.

According to the invention, determining the coefficients of the inverse mixing matrix M includes determining, for each of the one or more audio elements, a panning vector Pandown(dir) for panning the audio element to the channels of the channel-based audio signal, based on the direction of arrival dir of the audio element. Said determining the coefficients of the inverse mixing matrix M further includes determining a mixing matrix E that would be used for mapping the residual audio signal and the audio signals of the one or more audio elements to the channels of the channel-based audio signal, based on the determined panning vectors. Said determining the coefficients of the inverse mixing matrix M further includes determining a covariance matrix S for the intermediate representation based on the energy information. Determination of the covariance matrix S may be further based on the determined panning vectors Pandown. Said determining the coefficients of the inverse mixing matrix M further includes a determining the coefficients of the inverse mixing matrix M based on the mixing matrix E and the covariance matrix S.

In some embodiments, the mixing matrix E may be determined according to E = (IN|Pandown(dir<NUM>)|. |Pandown(dirP)). Here, IN may be an N × N identitiy matrix, with N indicating the number of channels of the channel-based signal, Pandown(dirp) may be the panning vector for the p-th audio element with associated direction of arrival dirp that would pan (e.g., map) the p-th audio element to the N channels of the channel-based signal, with p = <NUM>,. , P indicating a respective one among the one or more audio elements and P indicating the total number of the one or more audio elements. Accordingly, the matrix E may be a N × P matrix. The matrix E may be determined for each of a plurality of time segments k. In that case, the matrix E and the directions of arrival dirp would have an index k indicating the time segment, e.g., Ek = (IN|Pandown(dirk,<NUM>)|. |Pandown(dirk,P)). Even though the proposed method may operate in a band-wise manner, the matrix E may be the same for all frequency subbands.

In some embodiments, the covariance matrix S may be determined as a diagonal matrix according to <MAT> for <NUM> ≤ n ≤ N, and {S}N+p,N+p = ep for <NUM> ≤ p ≤ P. Here, ep may be the signal power associated with the direction of arrival of the p-th audio element. The matrix S may be determined for each of a plurality of time segments k, and/or for each of a plurality of frequency subbands b. In that case, the matrix S and the signal powers ep would have an index k indicating the time segment and/or an index b indicating the frequency subband, e.g., <MAT> for <NUM> ≤ n ≤ N, and {Sk,b}N+p,N+p = ek,p,b for <NUM> ≤ p ≤ P.

In some embodiments, determining the coefficients of the inverse mixing matrix M based on the mixing matrix E and the covariance matrix S may involve determining a pseudo inverse based on the mixing matrix E and the covariance matrix S.

In some embodiments, the inverse mixing matrix M may be determined according to M = S × E* × (E × S × E*)-<NUM>. Here, "×" indicates the matrix product and "*" indicates the conjugate transpose of a matrix. The inverse mixing matrix M may be determined for each of a plurality of time segments k, and/or for each of a plurality of frequency subbands b. In that case, the matrices M and S would have an index k indicating the time segment and/or an index b indicating the frequency subband, and the matrix E would have an index k indicating the time segment, e.g., <MAT>.

In some embodiments, the channel-based audio signal may be a first-order Ambisonics signal. Another aspect relates to an apparatus including a processor and a memory coupled to the processor, wherein the processor is adapted to carry out all steps of the methods according to any one of the aforementioned aspects and embodiments.

Another aspect of the disclosure relates to a program including instructions that, when executed by a processor, cause the processor to carry out all steps of the aforementioned methods.

Yet another aspect of the disclosure relates to a computer-readable storage medium storing the aforementioned program.

It will also be appreciated that apparatus features and method steps may be interchanged in many ways. In particular, the details of the disclosed method(s) can be realized by the corresponding apparatus, and vice versa, as the skilled person will appreciate. Moreover, any of the above statements made with respect to the method(s) are understood to likewise apply to the corresponding apparatus, and vice versa.

Example embodiments of the disclosure are illustrated by way of example in the accompanying drawings, in which like reference numbers indicate the same or similar elements and in which:.

Generally, the present disclosure relates to enablingstorage and/or transmission, using a reduced amount of data, of a spatial audio scene.

Concepts of audio processing that may be used in the context of the present disclosure will be described next.

A multichannel audio signal (or audio stream) may be formed by panning individual sonic elements (or audio elements, audio objects) according to a linear mixing law. For example, if a set of R audio objects are represented by R signals, {or(t): <NUM> ≤ r ≤ R}, then a multichannel panned mixture, {zn(t): <NUM> ≤ n ≤ N} may be formed by <MAT>.

The panning function, Pan(θr), represents a column vector containing N scale-factors (panning gains) indicative of the gains that are used to mix the object signal, or(t), to form the multichannel output, and where θr is indicative of the location of the respective object.

One possible panning function is a first-order Ambisonics (FOA) panner. An example of an FOA panning function is given by <MAT>.

An alternative panning function is a third-order Ambisonics panner (30A). An example of a 30A panning function is given by <MAT>.

It is understood that the present disclosure is not limited to FOA or HOA panning functions, and that use of other panning functions may be considered, as the skilled person will appreciate.

An audio stream, consisting of one or more audio signals, may be converted into short-term Fourier transform (STFT) form, for example. To this end, a discrete Fourier transform may be applied to (optionally windowed) time segments of the audio signals (e.g., channels, audio object signals) of the audio stream. This process, applied to an audio signal x(t), may be expressed as follows <MAT>.

It is understood that the STFT is an example of a time-frequency transform and that the present disclosure shall not be limited to STFTs.

In Equation (<NUM>), the variable Xc,k(f) indicates the short-term Fourier transform of channel c (<NUM> ≤ c ≤ NumChans), for audio time segment k ( <MAT>), at frequency bins f (<NUM> ≤ f ≤ F), where F indicates the number of frequency bins produced by the discrete Fourier transform. It will be appreciated that the terminology used here is by way of example, and that specific implementation details of various STFT methods (including various window functions) may be known in the art. Audio time segment k may be defined for example as a range of audio samples centered around t = k × stride + constant, so that time segments are uniformly spaced in time, with a spacing equal to stride.

The numeric values of the STFT (such as Xc,k(<NUM>), Xc,k(<NUM>),. , Xc,k(F)) may be referred to as FFT bins.

Further, the STFT form may be converted into an audio stream. The resulting audio stream may be an approximation to the original input and may be given by <MAT>.

Characteristic data may be formed from an audio stream where the characteristic data is associated with a number of frequency bands (frequency subbands), where a band (subband) is defined by a region of the frequency range.

By way example, the signal power in channel c of a stream, in frequency band b (where the number of bands is B and <NUM> ≤ b ≤ B), where band b spans FFT bins fmin ≤ f ≤ fmax, may be computed according to <MAT>.

According to a more general example, the frequency band b may be defined by a weighting vector, FRb(f), that assigns weights to each frequency bin, so that an alternative calculation of the power in a band may be given by <MAT>.

In a further generalization of Equation (<NUM>), the STFT of a stream that is composed of C audio signals may be processed to produce the covariance in a number of bands, where the covariance, Rb,k is a C × C matrix, and where element {Rb,k}i,j is computed according to <MAT> where Xj,k(f) represents the complex conjugate of Xj,k(f).

In another example, band-pass filters may be employed to form filtered signals representative of the original audio stream in frequency bands according to the band-pass filter responses. For example, an audio signal xc(t) may be filtered to produce x'c,b(t), representing a signal with energy predominantly derived from band b of xc(t), and hence an alternative method for computing the covariance of a stream in band b for time block k (corresponding to time samples tmin ≤ t ≤ tmax) may be expressed by <MAT>.

An audio stream composed of N channels may be processed to produce an audio stream composed of M channels according to an M × N linear mixing matrix, Q, so that <MAT> which may be written in matrix form as <MAT> where x̂(t) refers to the column-vector formed from the N elements: x<NUM>(t), x<NUM>(t),.

Further, an alternative mixing process may be implemented in the STFT domain, wherein the matrix, Q, may take on different values in each time block, k, and in each frequency band, b. In this case, the processing may be considered to be approximately given by <MAT> or, in matrix form <MAT>.

It will be appreciated that alternative methods may be employed to produce an equivalent behavior to the processing described in Equation (<NUM>).

Next, example implementations of methods and apparatus according to embodiments of the disclosure will be described in more detail.

Broadly speaking, methods not forming part of the invention represent a spatial audio scene in the form of an audio mixture stream and a direction metadata stream, where the direction metadata stream includes data indicative of the location of directional sonic elements in the spatial audio scene and data indicative of the power of each directional sonic element, in a number of subbands, relative to the total power of the spatial audio scene in that subband. Methods according to embodiments of the disclosure relate to determining the direction metadata stream from an input spatial audio scene, and to creating a reconstituted (e.g., reconstructed) audio scene from a direction metadata stream and associated audio mixture stream.

Examples of methods not forming part of the invention are efficient (e.g., in terms of reduced data for storage or transmission) in representing a spatial sound scene. The spatial audio scene may be represented by a spatial audio signal. Said methods may be implemented by defining a storage or transmission format (e.g., the Compact Spatial Audio Stream) that consists of an audio mixture stream and a metadata stream (e.g., direction metadata stream).

The audio mixture stream comprises a number of audio signals that convey a reduced representation of the spatial sound scene. As such, the audio mixture stream may relate to a channel-based audio signal with a predefined number of channels. It is understood that the number of channels of the channel-based audio signal is smaller than the number of channels or the number of audio objects of the spatial audio signal. For example, the channel-based audio signal may be a first-order Ambisonics audio signal. In other words, the Compact Spatial Audio Stream may include an audio mixture stream in the form of a first-order Ambisonics representation of the soundfield.

The (direction) metadata stream comprises metadata that defines spatial properties of the spatial sound scene. Direction metadata may consist of a sequence of direction metadata blocks, wherein each direction metadata block contains metadata that indicates properties of the spatial sound scene in a corresponding time segment in the audio mixture stream.

In general, the metadata includes direction information and energy information. The direction information comprises indications of directions of arrival of one or more (dominant) audio elements in the audio scene. The energy information comprises, for each direction of arrival, an indication of signal power associated with the determined directions of arrival. In some implementations, the indications of signal power may be provided for one, some, or each of a plurality of bands (frequency subbands). Moreover, the metadata may be provided for each of a plurality of consecutive time segments, such as in the form of metadata blocks, for example.

In one example, the metadata (direction metadata) includes metadata that indicates properties of the spatial sound scene over a number of frequency bands, where the metadata defines:.

Details on the determination of the direction information and the energy information will be provided below.

<FIG> schematically shows an example of an arrangement not forming part of the invention. Specifically, the figure shows an arrangement <NUM> wherein a spatial audio scene <NUM> is input to a scene encoder <NUM> that generates an audio mixture stream <NUM> and a direction metadata stream <NUM>. The spatial audio scene <NUM> may be represented by a spatial audio signal or spatial audio stream that is input to the scene encoder <NUM>. The audio mixture stream <NUM> and the direction metadata stream <NUM> together form an example of a compact spatial audio scene, i.e., a compressed representation of the spatial audio scene <NUM> (or of the spatial audio signal).

The compressed representation, i.e., the mixture audio stream <NUM> and the direction metadata stream <NUM> are input to scene decoder <NUM> which produces a reconstructed audio scene <NUM>. Audio elements that exist within the spatial audio scene <NUM> will be represented within the audio mixture stream <NUM> accordingto a mixture panning function.

<FIG> schematically shows another example of an arrangement not forming part of the invention. Specifically, the figure shows an alternative arrangement <NUM> wherein the compact spatial audio scene, composed of audio mixture stream <NUM> and a direction metadata stream <NUM>, is further encoded by providing the audio mixture stream <NUM> to audio encoder <NUM> to produce a reduced bit-rate encoded audio stream <NUM>, and by providing the direction metadata stream <NUM> to a metadata encoder <NUM> to produce an encoded metadata stream <NUM>. The reduced bit-rate encoded audio stream <NUM> and the encoded metadata stream <NUM> together form an encoded (reduced bit-rate encoded) spatial audio scene.

The encoded spatial audio scene may be recovered by first applying the reduced bit-rate encoded audio stream <NUM> and the encoded metadata stream <NUM> to respective decoders <NUM> and <NUM> to produce a recovered audio mixture stream <NUM> and a recovered direction metadata stream <NUM>. The recovered streams <NUM>, <NUM> may be identical to or approximately equal to the respective streams <NUM>, <NUM>. The recovered audio mixture stream <NUM> and the recovered direction metadata stream <NUM> may be decoded by decoder <NUM> to produce a reconstructed audio scene <NUM>.

<FIG> schematically illustrates an example of an arrangement for generating a reduced bit-rate encoded audio stream and an encoded metadata stream from an input spatial audio scene. Specifically, the figure shows an arrangement <NUM> of scene encoder <NUM> providing a direction metadata stream <NUM> and audio mixture stream <NUM> to respective encoders <NUM>, <NUM> to produce an encoded spatial audio scene <NUM> which includes reduced bit-rate encoded audio stream <NUM> and the encoded metadata stream <NUM>. Encoded spatial audio stream <NUM> is preferably arranged to be suitable for storage and/or transmission with reduced data requirement, relative to the data required for storage/transmission of the original spatial audio scene.

<FIG> schematically illustrates an example of an arrangement for generating a reconstructed spatial audio scene from the reduced bit-rate encoded audio stream and the encoded metadata stream. Specifically, the figure shows an arrangement <NUM> wherein an encoded spatial audio stream <NUM>, composed of reduced bit-rate encoded audio stream <NUM> and encoded metadata stream <NUM>, is provided as input to decoders <NUM>, <NUM> to produce audio mixture stream <NUM> and direction metadata stream <NUM>, respectively. Streams <NUM>, <NUM> are then processed by scene decoder <NUM> to produce a reconstructed audio scene <NUM>.

Details of generating the compact spatial audio scene, i.e., the compressed representation of the spatial audio scene (or of the spatial audio signal / spatial audio stream) will be described next.

<FIG> is a flowchart of an example of a method <NUM> of processing a spatial audio signal for generating a compressed representation of the spatial audio signal. The method <NUM> comprises steps S510 through S550.

At step S510, the spatial audio signal is analyzed to determine directions of arrival for one or more audio elements (e.g., dominant audio elements) in an audio scene (spatial audio scene) represented by the spatial audio signal. The (dominant) audio elements may relate to (dominant) acoustic objects, (dominant) sound sources, or (dominant) acoustic components in the audio scene, for example. Analyzing the spatial audio signal may involve or may relate to applying scene analysis to the spatial audio signal. It is understood that a range of suitable scene analysis tools are known to the skilled person. The directions of arrival determined at this step may correspond to locations on a unit sphere indicating the (perceived) locations of the audio elements.

In line with the above description of frequency-banded analysis, analyzing the spatial audio signal at step S510 can be based on a plurality of frequency subbands of the spatial audio signal. For example, the analysis may be based on the full frequency range of the spatial audio signal (i.e., the full signal). That is, the analysis may be based on all frequency subbands.

At step S520, respective indications of signal power associated with the determined directions of arrival are determined for at least one frequency subband of the spatial audio signal.

At step S530, metadata comprisingdirection information and energy information is generated. The direction information comprises indications of the determined directions of arrival of the one or more audio elements. The energy information comprises respective indications of signal power associated with the determined directions of arrival. The metadata generated at this step may relate to a metadata stream.

At step S540, a channel-based audio signal with a predefined number of channels is generated based on the spatial audio signal.

Finally, at step S550, the channel-based audio signal and the metadata are output as the compressed representation of the spatial audio signal.

It is understood that the above steps may be performed in any order or in parallel to each other, as long as the order of steps ensures that the necessary input for each step is available.

Typically, a spatial scene (or spatial audio signal) may be considered to be composed of a summation of acoustic signals that are incident on a listener from a set of directions, relative to the listening position. The spatial audio scene may therefore be modeled as a collection of R acoustic objects, where object r (<NUM> ≤ r ≤ R) is associated with an audio signal or(t) that is incident at the listening position from a direction of arrival defined by the direction vector θr. The direction vector may also be a time-varying direction vector θr(t).

Hence, accordingto some implementations, the spatial audio signal (spatial audio stream) may be defined as an object-based spatial audio signal (object-based spatial audio scene), in the form of a set of audio signals and associated direction-vectors <MAT>.

Further, according to some implementations, the spatial audio signal (spatial audio stream) may be defined in terms of short-term Fourier transform signals, Or,k(f), according to Equation (<NUM>), and direction-vectors may be specified according to block-index, k, so that: <MAT>.

Alternatively, the spatial audio signal (spatial audio stream) may be represented in terms of a channel-based spatial audio signal (channel-based spatial audio scene). A channel based stream consists of a collection of audio signals, wherein each acoustic object from the spatial audio scene is mixed into the channels accordingto a panningfunction (Pan(θ)), accordingto Equation (<NUM>). By way of example, a Q-channel channel-based spatial audio scene, {Cq,k(f): <NUM> ≤ q ≤ Q}, may be formed from an object-based spatial audio scene accordingto <MAT> <MAT>.

It will be appreciated that many characteristics of a channel-based spatial audio scene are determined by the choice of the panning function, and in particular the length (Q) of the column-vector returned by the panning function will determine the number of audio channels contained in the channel-based spatial audio scene. Generally speaking, a higher-quality representation of a spatial audio scene may be realized by a channel-based spatial audio scene containing a larger number of channels.

As an example, at step S540 of the method <NUM> the spatial audio signal (spatial audio scene) may be processed to create a channel-based audio signal (channel-based stream) according to Equation (<NUM>). The panning function may be chosen so as to create a relatively low-resolution representation of the spatial audio scene. For instance, the panning function may be chosen to be the First Order Ambisonics (FOA) function, such as that defined in Equation (<NUM>). As such, the compressed representation may be a compact or size-reduced representation.

<FIG> is a flowchart providing another formulation of a method <NUM> of generating a compact representation of a spatial audio scene. The method <NUM> is provided with an input stream, in the form of a spatial audio scene or a scene-based stream, and produces a compact spatial audio scene as the compact representation. To his end, method <NUM> comprises steps S610 through S660. Therein, step S610 may be seen as correspondingto step S510, step <NUM> may be seen as corresponding to step S520, step S630 may be seen as corresponding to step S540, step S650 may be seen as correspondingto step S530, and step S660 may be seen as corresponding to step S550.

At step S610, the input stream is analyzed to determine dominant directions of arrival.

At step S620, for each band (frequency subband), a fraction of energy allocated to each direction is determined, relative to a total energy in the stream in that band.

At step S630, a downmix stream is formed, containing a number of audio channels representing the spatial audio scene.

At step S640, the downmixed stream is encoded to form a compressed representation of the stream.

At step S650, the direction information and energy-fraction information are encoded to form encoded metadata.

Finally, at step S660 the encoded downmixed stream is combined with the encoded metadata to form a compact spatial audio scene.

<FIG> schematically illustrate examples of details of generating a compressed representation of a spatial audio scene, not forming part of the invention.

It is understood that the specifics of, for example, analyzing the spatial audio signal for determining directions of arrival, determining indications of signal power associated with the determined directions of arrival, generating metadata comprisingdirection information and energy information, and/or generating the channel-based audio signal with a predefined number of channels as described below may be independent of the specific system arrangement and may apply to, for example, any of the arrangements shown in <FIG>, or any suitable alternative arrangements.

<FIG> schematically illustrates a first example of details of generating the compressed representation of the spatial audio scene. Specifically, <FIG> shows a scene encoder <NUM> in which a spatial audio scene <NUM> is processed by a downmix function <NUM> to produce an N-channel audio mixture stream <NUM>, in accordance with, for example, steps S540 and S630. In some examples, the downmix function <NUM> may include the panning process according to Equation (<NUM>) or Equation (<NUM>), wherein a downmix panningfunction is chosen: <MAT>. For example, a first order Ambisonics panner may be chosen as the downmix panning function: <MAT> and hence N = <NUM>.

For each audio time segment, scene analysis <NUM> takes as input the spatial audio scene, and determines the directions of arrival of up to P dominant acoustic components within the spatial audio scene, in accordance with, for example, steps S510 and S610. Typical values for P are between <NUM> and <NUM>, and a preferred value for P is P ≈ <NUM>. Accordingly, the one or more audio elements determined at step S510 may comprise between one and ten audio elements, such as four audio elements, for example.

Scene analysis <NUM> produces a metadata stream <NUM> composed of direction information <NUM> and energy band fraction information <NUM> (energy information). Optionally, scene analysis <NUM> may also provide coefficients <NUM> to the downmix function <NUM> to allow the downmix to be modified. Without intended limitation, analyzing the spatial audio signal (e.g., at step S510), determining respective indications of signal power (e.g., atstep S520), and generatingthe channel-based audio signal (e.g., atstep S540) may be performed on a per-time-segment basis, in line with, for example, the above description of STFTs. This implies that the compressed representation will be generated and output for each of a plurality of time segments, with a downmixed audio signal and metadata (metadata block) for each time segment.

For each time segment, k, direction information <NUM> (e.g., embodied by the directions of arrival of the one or more audio elements) can take the form of P direction vectors, {dirk,p: <NUM> ≤ p ≤ P}. Direction vector p indicates the direction associated with dominant object index p, and may be represented in terms of unit-vectors,
where: <MAT> or in terms of spherical coordinates,
where: <MAT>.

In some examples, the respective indications of signal power determined at step S520 take the form of a fraction of signal power. That is, an indication of signal power associated with a given direction of arrival in the frequency subband relates to a fraction of signal power in the frequency subband for the given direction of arrival in relation to the total signal power in the frequency subband.

Further, in some examples the indications of signal power are determined for each of a plurality of frequency subbands (i.e., in a per-subband manner). Then, they relate, for a given direction of arrival and a given frequency subband, to a fraction of signal power in the given frequency subband for the given direction of arrival in relation to the total signal power in the given frequency subband. Notably, even though the indications of signal power may be determined in a per-subband manner, the determination of the (dominant) directions of arrival may still be performed on the full signal (i.e., based on all frequency subbands).

Yet further, in some examples analyzing the spatial audio signal (e.g., at step S510), determining respective indications of signal power (e.g., at step S520), and generating the channel-based audio signal (e.g., at step S540) are performed based on a time-frequency representation of the spatial audio signal. For example, the aforementioned steps and other steps as suitable may be performed based on a discrete Fourier transform (such as a STFT, for example) of the spatial audio signal. For example, for each time segment (time block), the aforementioned steps may be performed based on the time-frequency bins (FFT bins) of the spatial audio signal, i.e., on the Fourier coefficients of the spatial audio signal.

Given the above, for each time segment, k, and for each dominant object index p (<NUM> ≤ p ≤ P), energy band fraction information <NUM> can include a fraction value ek,p,b for each band b of a set of bands (<NUM> ≤ b ≤ B). The fraction value ek,p,b is determined for the time segment k according to: <MAT>.

The fraction value ek,p,b may represent the fraction of energy in a spatial region around the direction dirk,p, so that the energy of multiple acoustic objects in the original spatial audio scene may be combined to represent a single dominant acoustic component assigned to direction dirk,p. In some examples, the energy of all acoustic objects in the scene may be weighted, using an angular difference weighting function w(θ) that represents a larger weighting for a direction, θ, that is close to dirk,p, and a smaller weighting for a direction, θ, that is far from dirk,p. Directional differences may be considered to be close for angular differences less than, for example, <NUM>° and far for angular differences greater than, for example, <NUM>°. In alternative examples, the weighting function may be chosen based on alternative choices of the close/far angular differences.

In general, the input spatial audio signal for which the compressed representation is generated may be a multichannel audio signal or an object-based audio signal, for example. In the latter case, the method for generatingthe compressed representation of the spatial audio signal would further comprise a step of converting the object-based audio signal to a multichannel audio signal prior to applying the scene analysis (e.g., prior to step S510).

In the example of <FIG>, the input spatial audio signal may be a multichannel audio signal. Then, the channel-based audio signal generated at step S540 would be a downmix signal generated by applying a downmix operation to the multichannel audio signal.

<FIG> schematically illustrates another example of details of generating the compressed representation of the spatial audio scene. The input spatial audio signal in this case may be an object-based audio signal that comprises a plurality of audio objects and associated direction vectors. In this case, the method of generatingthe compressed representation of the spatial audio signal comprises generating a multichannel audio signal, as an intermediate representation or intermediate scene, by panning the audio objects to a predefined set of audio channels, wherein each audio object is panned to the predefined set of audio channels in accordance with its direction vector. Thus, <FIG> shows an alternative example of a scene encoder <NUM> wherein spatial audio scene <NUM> is input to a converter <NUM> that produces the intermediate scene <NUM> (e.g., embodied by the multichannel signal). Intermediate scene <NUM> may be created according Equation (<NUM>) where the panning function is selected so that the dot-product of panning gain vectors Pan(θ<NUM>) and Pan(θ<NUM>) approximately represents an angular difference weighting function, as described above.

In some examples, the panning function used in converter <NUM> is a third order Ambisonics panning function, <MAT>, as shown in Equation (<NUM>). Accordingly, the multichannel audio signal may be a higher-order Ambisonics signal, for example.

The intermediate scene <NUM> is then input to scene analysis <NUM>. Scene analysis <NUM> may determine the directions, dirk,p, of dominant acoustic objects in the spatial audio scene from analysis of the intermediate scene <NUM>. Determination of the dominant directions may be performed by estimating the energy in a set of directions, with the largest estimated energy representing the dominant direction.

Energy band fraction information <NUM> for time segment k may include a fraction value ek,p,b for each band b that is derived from the energy in band b of the intermediate scene <NUM> in each direction dirk,p, relative to the total energy in band b of the intermediate scene <NUM> in time segment k.

The audio mixture stream <NUM> (e.g., channel-based audio signal) of the compact spatial audio scene (e.g., compact representation) in this case is a downmix signal generated by applying the downmix function <NUM> (downmix operation) to the spatial audio scene.

<FIG> shows an alternative arrangement of a scene encoder including a converter <NUM> to convert spatial audio scene <NUM> into a scene-based intermediate format <NUM>. The intermediate format <NUM> is input to scene analysis <NUM> and to downmix function <NUM>. In some examples, downmix function <NUM> may include a matrix mixer with coefficients adapted to convert intermediate format <NUM> into the audio mixture stream <NUM>. That is, the audio mixture stream <NUM> (e.g., channel-based audio signal) of the compact spatial audio scene (e.g., compact representation) in this case may be a downmix signal generated by applying the downmix function <NUM> (downmix operation) to the intermediate scene (e.g., multichannel audio signal).

In an alternative example, shown in <FIG>, spatial encoder <NUM> may take input in the form of a scene-based input <NUM>, wherein acoustic objects are represented according to a panning rule, Pan(θ). In some examples, the panning function may be a higher-order Ambisonics panning function. In one example the panning function is a third-order Ambisonics panning function.

In another alternative example, illustrated in <FIG>, a spatial audio scene <NUM> is converted by converter <NUM> in spatial encoder <NUM> to produce an intermediate scene <NUM> which is input to downmix function <NUM>. Scene analysis <NUM> is provided with input from the spatial audio scene <NUM>.

<FIG> schematically illustrates an example of details of decoding a compressed representation of a spatial audio scene to form a reconstituted audio scene, according to embodiments of the disclosure. Specifically, the figure shows a scene decoder <NUM> including a demixer <NUM> that takes an audio mixture stream <NUM> and produces a separated spatial audio stream <NUM>. Separated spatial audio stream <NUM> is composed of P dominant object signals <NUM> and a residual stream <NUM>. Residual decoder <NUM> takes input from residual stream <NUM> and creates a decoded residual stream <NUM>. Object panner <NUM> takes input from dominant object signals <NUM> and creates panned object stream <NUM>. Decoded residual stream <NUM> and panned object stream <NUM> are summed <NUM> to produce reconstituted audio scene <NUM>.

Further, <FIG> shows direction information <NUM> and energy band fraction information <NUM> input to a demix matrix calculator <NUM> that determines a demix matrix <NUM> (inverse mixing matrix) to be used by demixer <NUM>.

Details of processing the compact spatial audio scene (e.g., the compressed representation of the spatial audio signal) for generating the reconstructed representation of the spatial audio signal will be described next.

<FIG> is a flowchart of an example of a method <NUM> of processing a compressed representation of a spatial audio signal for generating a reconstructed representation of the spatial audio signal. It is understood that the compressed representation comprises a channel-based audio signal (e.g., embodied by the audio mixture stream <NUM>) with a predefined number of channels and metadata, the metadata comprising direction information (e.g., embodied by direction information <NUM>) and energy information (e.g., embodied by energy band fraction information <NUM>), with the direction information comprising indications of directions of arrival of one or more audio elements in an audio scene and the energy information comprising, for at least one frequency subband, respective indications of signal power associated with the directions of arrival. The channel-based audio signal may be a first-order Ambisonics signal, for example. The method <NUM> comprises steps S1310 and S1320, and optionally, steps S1330 and S1340. It is understood that these steps may be performed by the scene decoder <NUM> of <FIG>, for example.

At step S1310, audio signals of the one or more audio elements are generated based on the channel-based audio signal, the direction information, and the energy information.

At step S1320, a residual audio signal from which the one or more audio elements are substantially absent is generated, based on the channel-based audio signal, the direction information, and the energy information. Here, the residual signal may be represented in the same audio format as the channel-based audio signal, e.g., may have the same number of channels as the channel-based audio signal.

At optional step S1330, the audio signals of the one or more audio elements are panned to a set of channels of an output audio format. Here, the output audio format may relate to an output representation, for example, such as HOA or any other suitable multichannel format.

At optional step S1340, a reconstructed multichannel audio signal in the output audio format is generated based on the panned one or more audio elements and the residual signal. Generating the reconstructed multichannel audio signal may include upmixingthe residual signal to the set of channels of the output audio format. Generating the reconstructed multichannel audio signal may further include adding the panned one or more audio elements and the upmixed residual signal.

In line with the above description of methods of processing the spatial audio scene for generating the compressed representation of the spatial audio scene, an indication of signal power associated with a given direction of arrival may relate to a fraction of signal power in the frequency subband for the given direction of arrival in relation to the total signal power in the frequency subband.

Moreover, in some embodiments, the energy information may include indications of signal power for each of a plurality of frequency subbands. Then, an indication of signal power may relate, for a given direction of arrival and a given frequency subband, to a fraction of signal power in the given frequency subband for the given direction of arrival in relation to the total signal power in the given frequency subband.

Generating audio signals of the one or more audio elements at step S1310 may comprise determining coefficients of an inverse mixing matrix M for mapping the channel-based audio signal to an intermediate representation comprising the residual audio signal and the audio signals of the one or more audio elements, based on the direction information and the energy information. The intermediate representation can also be referred to as a separated or separable representation, or a hybrid representation.

Details of said determining the coefficients of the inverse mixing matrix M will be described next with reference to the flowchart of <FIG>. Method <NUM> illustrated by this flowchart comprises steps S1410 through S1440.

At step S1410, for each of the one or more audio elements, a panning vector Pandown(dir) for panning the audio element to the channels of the channel-based audio signal is determined, based on the direction of arrival dir of the audio element.

At step S1420, a mixing matrix E that would be used for mapping the residual audio signal and the audio signals of the one or more audio elements to the channels of the channel-based audio signal is determined, based on the determined panning vectors.

At step S1430, a covariance matrix S for the intermediate representation is determined based on the energy information. Determination of the covariance matrix S may be further based on the determined panning vectors Pandown.

Finally, at step S1440, the coefficients of the inverse mixing matrix M are determined based on the mixing matrix E and the covariance matrix S.

Returning to <FIG>, demix matrix calculator <NUM> computes the demix matrix <NUM> (inverse mixing matrix), Mk,b according to a process that includes the following steps:.

where "×" indicates the matrix product and "*" indicates the conjugate transpose of a matrix. The calculation accordingto Equation (<NUM>) may correspond to step S1440, for example.

The demix matrix M may be determined for each of a plurality of time segments k, and/or for each of a plurality of frequency subbands b. In that case, the matrices M and S would have an index k indicating the time segment and/or an index b indicating the frequency subband, and the matrix E would have an index k indicating the time segment, e.g., <MAT>.

In general, determining the coefficients of the inverse mixing matrix M based on the mixing matrix E and the covariance matrix S may involve determining a pseudo inverse based on the mixing matrix E and the covariance matrix S. One example of such pseudo inverse is given in Equations (<NUM>) and (20a).

In Equation (<NUM>), the matrix Ek (mixing matrix) is formed by stacking together an N × N identity matrix (IN) and the P columns formed by the panning function applied to the directions of each of the P dominant acoustic components: <MAT>.

In Equation (<NUM>), IN is an N × N identitiy matrix, with N indicating the number of channels of the channel-based signal. Pandown(dirp) is the panning vector for the p-th audio element with associated direction of arrival dirp that would pan the p-th audio element to the N channels of the channel-based signal, with p = <NUM>,. , P indicating a respective one among the one or more audio elements and P indicating the total number of the one or more audio elements. The vertical bars in Equation (<NUM>) indicate a matrix augmentation operation. Accordingly, the matrix E is a N × P matrix.

Further, the matrix E may be determined for each of a plurality of time segments k. In that case, the matrix E and the directions of arrival dirp would have an index k indicating the time segment, e.g., <MAT>.

If the proposed method operates in a band-wise manner, the matrix E may be the same for all frequency subbands.

In accordance with step S1420, matrix Ek is the mixing matrix that would be used for mapping the residual audio signal and the audio signals of the one or more audio elements to the channels of the channel-based audio signal. As can be seen from Equations (<NUM>) and (21a), the matrix Ek is based on the panning vectors Pandown(dir) determined at step S1410.

In Equation (<NUM>), the matrix S is a (N + P) × (N + P) diagonal matrix. It can be seen as a covariance matrix for the intermediate representation. Its coefficients can be calculated based on the energy information, in accordance with step S1430. The first N diagonal elements are given by <MAT> for <NUM> ≤ n ≤ N, and the remaining P diagonal elements are given by <MAT> for <NUM> ≤ p ≤ P, where ep is the signal power associated with the direction of arrival of the p-th audio element.

The covariance matrix S may be determined for each of a plurality of time segments k, and/or for each of a plurality of frequency subbands b. In that case, the covariance matrix S and the signal powers ep would have an index k indicating the time segment and/or an index b indicating the frequency subband. The first N diagonal elements would be given by <MAT> and the remaining P diagonal elements would be given by <MAT>.

In a preferred embodiment, the demix matrix Mk,b is applied, by demixer <NUM>, to produce a separated spatial audio stream <NUM> (as an example of the intermediate representation), in accordance with the above-described implementation of step S1310, wherein the first N channels are the residual stream <NUM> and the remaining P channels represent the dominant acoustic components.

The N + P channel separated spatial stream <NUM>, Yk(f), the P channel dominant object signals <NUM> (as examples of the audio signals of the one or more audio elements generated at step S1310), Ok(f), and the N channel residual stream <NUM> (as an example of the residual audio signal generated at step S1320), Rk(f), are computed from the N-channel audio mixture <NUM>, Xk(f), according to: <MAT> wherein the signals are represented in STFT form, the expression {Yk(f)}<NUM>. N indicates an N-channel signal formed from channels <NUM>. N of Yk(f), and {Yk(f)}N+<NUM>. N+P indicates a P-channel signal formed from channels N+<NUM>. N+P of Yk(f). It will be appreciated by those skilled in the are that the application of the matrix Mk,b may be achieved according to alternative methods, known in the art, that provide an equivalent approximate function to that of Equation (<NUM>).

In addition to the above, in some embodiments, the number of dominant acoustic components P may be adapted to take a different value for each time segment, so that Pk may be dependent on the time segment index, k. For example, the scene analysis <NUM> in the scene encoder <NUM> may determine a value of Pk for each time segment. In general, the number of dominant acoustic components P may be time-dependent. The choice of P (or Pk) may include a trade-off between the metadata data-rate and the quality of the reconstructed audio scene.

Returning to <FIG>, the spatial decoder <NUM> produces an M-channel reconstituted audio scene <NUM> wherein the M-channel stream is associated with an output panner <MAT>. This may be done in accordance with step S1340 described above. Examples of output panners include stereo panningfunctions, vector-based amplitude panningfunctions as known in the art, and higher-order Ambisonics panning functions, as known in the art.

For example, object panner <NUM> in <FIG> may be adapted to create the M-channel panned object stream <NUM>, Zp, according to <MAT>.

<FIG> is a flowchart providingan alternative, not forming part of the invention, of a method <NUM> of decoding a compact spatial audio scene to produce a reconstituted audio scene. Method <NUM> comprises steps S1510 through S1580.

At step S1510, a compact spatial audio scene is received and the encoded downmix stream and the encoded metadata stream are extracted.

At step S1520, the encoded downmix stream is decoded to form a downmix stream.

At step S1530, the encoded metadata stream is decoded to form the direction information and the energy fraction information.

At step S1540, a per-band demixing matrix is formed from the direction information and the energy fraction information.

At step S1550, the downmix stream is processed according to the demixing matrix to form a separated stream.

At step S1560, object signals are extracted from the separated stream and panned to produce panned object signals according to the direction information and a desired output format.

At step S1570, residual signals are extracted from the separated stream and processed to create decoded residual signals according to the desired output format.

Finally, at step S1580, panned object signals and decoded residual signals are combined to form a reconstituted audio scene.

Methods of processing a spatial audio signal for generating a compressed representation of the spatial audio signal, as well as methods of processing a compressed representation of a spatial audio signal for generating a reconstructed representation of the spatial audio signal have been described above. Additionally, the present disclosure also relates to an apparatus for carrying out these methods. An example of such apparatus <NUM> is schematically illustrated in <FIG>. The apparatus <NUM> may comprise a processor <NUM> (e.g., a central processing unit (CPU), a graphics processing unit (GPU), a digital signal processor (DSP), one or more application specific integrated circuits (ASICs), one or more radio-frequency integrated circuits (RFICs), or any combination of these) and a memory <NUM> coupled to the processor <NUM>. The processor may be adapted to carry out some or all of the steps of the methods described throughout the disclosure. If the apparatus <NUM> acts as an encoder (e.g., scene encoder), it may receive, as input <NUM>, the spatial audio signal (i.e., the spatial audio scene), for example. The apparatus <NUM> may then generate, as output <NUM>, the compressed representation of the spatial audio signal. If the apparatus <NUM> acts as a decoder (e.g., scene decoder), it may receive, as input <NUM>, the compressed representation. The apparatus may then generate, as output <NUM>, the reconstituted audio scene.

The apparatus <NUM> may be a server computer, a client computer, a personal computer (PC), a tablet PC, a set-top box (STB), a personal digital assistant (PDA), a cellular telephone, a smartphone, a web appliance, a network router, switch or bridge, or any machine capable of executing instructions (sequential or otherwise) that specify actions to be taken by that apparatus. Further, while only a single apparatus <NUM> is illustrated in <FIG>, the present disclosure shall relate to any collection of apparatus that individually or jointly execute instructions to perform any one or more of the methodologies discussed herein.

The present disclosure further relates to a program (e.g., computer program) comprising instructions that, when executed by a processor, cause the processor to carry out some or all of the steps of the methods described herein.

Yet further, the present disclosure relates to a computer-readable (or machine-readable) storage medium storing the aforementioned program. Here, the term "computer-readable storage medium" includes, but is not be limited to, data repositories in the form of solid-state memories, optical media, and magnetic media, for example.

Unless specifically stated otherwise, as apparent from the following discussions, it is appreciated that throughout the disclosure discussions utilizing terms such as "processing," "computing," "calculating," "determining", analyzing" or the like, refer to the action and/or processes of a computer or computing system, or similar electronic computing devices, that manipulate and/or transform data represented as physical, such as electronic, quantities into other data similarly represented as physical quantities.

The methodologies described herein are, in one example embodiment, performable by one or more processors that accept computer-readable (also called machine-readable) code containing a set of instructions that when executed by one or more of the processors carry out at least one of the methods described herein. Any processor capable of executing a set of instructions (sequential or otherwise) that specify actions to be taken are included. Thus, one example is a typical processing system that includes one or more processors. Each processor may include one or more of a CPU, a graphics processing unit, and a programmable DSP unit. The processing system further may include a memory subsystem including main RAM and/or a static RAM, and/or ROM. A bus subsystem may be included for communicating between the components. The processing system further may be a distributed processing system with processors coupled by a network. If the processing system requires a display, such a display may be included, e.g., a liquid crystal display (LCD) or a cathode ray tube (CRT) display. If manual data entry is required, the processing system also includes an input device such as one or more of an alphanumeric input unit such as a keyboard, a pointing control device such as a mouse, and so forth. The processing system may also encompass a storage system such as a disk drive unit. The processing system in some configurations may include a sound output device, and a network interface device. The memory subsystem thus includes a computer-readable carrier medium that carries computer-readable code (e.g., software) including a set of instructions to cause performing, when executed by one or more processors, one or more of the methods described herein. Note that when the method includes several elements, e.g., several steps, no ordering of such elements is implied, unless specifically stated. The software may reside in the hard disk, or may also reside, completely or at least partially, within the RAM and/or within the processor during execution thereof by the computer system. Thus, the memory and the processor also constitute computer-readable carrier medium carrying computer-readable code. Furthermore, a computer-readable carrier medium may form, or be included in a computer program product.

In alternative example embodiments, the one or more processors operate as a standalone device or may be connected, e.g., networked to other processor(s), in a networked deployment, the one or more processors may operate in the capacity of a server or a user machine in server-user network environment, or as a peer machine in a peer-to-peer or distributed network environment. The one or more processors may form a personal computer (PC), a tablet PC, a Personal Digital Assistant (PDA), a cellular telephone, a web appliance, a network router, switch or bridge, or any machine capable of executing a set of instructions (sequential or otherwise) that specify actions to be taken by that machine.

Note that the term "machine" shall also be taken to include any collection of machines that individually or jointly execute a set (or multiple sets) of instructions to perform any one or more of the methodologies discussed herein.

Thus, one example embodiment of each of the methods described herein is in the form of a computer-readable carrier medium carrying a set of instructions, e.g., a computer program that is for execution on one or more processors, e.g., one or more processors that are part of web server arrangement. Thus, as will be appreciated by those skilled in the art, example embodiments of the present disclosure may be embodied as a method, an apparatus such as a special purpose apparatus, an apparatus such as a data processing system, or a computer-readable carrier medium, e.g., a computer program product. The computer-readable carrier medium carries computer readable code including a set of instructions that when executed on one or more processors cause the processor or processors to implement a method. Accordingly, aspects of the present disclosure may take the form of a method, an entirely hardware example embodiment, an entirely software example embodiment or an example embodiment combining software and hardware aspects. Furthermore, the present disclosure may take the form of carrier medium (e.g., a computer program product on a computer-readable storage medium) carrying computer-readable program code embodied in the medium.

The software may further be transmitted or received over a network via a network interface device. While the carrier medium is in an example embodiment a single medium, the term "carrier medium" should be taken to include a single medium or multiple media (e.g., a centralized or distributed database, and/or associated caches and servers) that store the one or more sets of instructions. The term "carrier medium" shall also be taken to include any medium that is capable of storing, encoding or carrying a set of instructions for execution by one or more of the processors and that cause the one or more processors to perform any one or more of the methodologies of the present disclosure. A carrier medium may take many forms, including but not limited to, non-volatile media, volatile media, and transmission media. Non-volatile media includes, for example, optical, magnetic disks, and magneto-optical disks. Volatile media includes dynamic memory, such as main memory. Transmission media includes coaxial cables, copper wire and fiber optics, including the wires that comprise a bus subsystem. Transmission media may also take the form of acoustic or light waves, such as those generated during radio wave and infrared data communications. For example, the term "carrier medium" shall accordingly be taken to include, but not be limited to, solid-state memories, a computer product embodied in optical and magnetic media; a medium bearing a propagated signal detectable by at least one processor or one or more processors and representing a set of instructions that, when executed, implement a method; and a transmission medium in a network bearing a propagated signal detectable by at least one processor of the one or more processors and representing the set of instructions.

It will be understood that the steps of methods discussed are performed in one example embodiment by an appropriate processor (or processors) of a processing (e.g., computer) system executing instructions (computer-readable code) stored in storage. It will also be understood that the disclosure is not limited to any particular implementation or programming technique and that the disclosure may be implemented using any appropriate techniques for implementing the functionality described herein. The disclosure is not limited to any particular programming language or operating system.

Reference throughout this disclosure to "one example embodiment", "some example embodiments" or "an example embodiment" means that a particular feature, structure or characteristic described in connection with the example embodiment is included in at least one example embodiment of the present disclosure. Thus, appearances of the phrases "in one example embodiment", "in some example embodiments" or "in an example embodiment" in various places throughout this disclosure are not necessarily all referring to the same example embodiment. Furthermore, the particular features, structures or characteristics may be combined in any suitable manner, as would be apparent to one of ordinary skill in the art from this disclosure, in one or more example embodiments.

It should be appreciated that in the above description of example embodiments of the disclosure, various features of the disclosure are sometimes grouped together in a single example embodiment, figure, or description thereof for the purpose of streamlining the disclosure and aiding in the understanding of one or more of the various inventive aspects. This method of disclosure, however, is not to be interpreted as reflecting an intention that the claims require more features than are expressly recited in each claim. Rather, as the following claims reflect, inventive aspects lie in less than all features of a single foregoing disclosed example embodiment. Thus, the claims following the Description are hereby expressly incorporated into this Description, with each claim standing on its own as a separate example embodiment of this disclosure.

Furthermore, while some example embodiments described herein include some but not other features included in other example embodiments, combinations of features of different example embodiments are meant to be within the scope of the disclosure, and form different example embodiments, as would be understood by those skilled in the art.

However, it is understood that example embodiments of the disclosure may be practiced without these specific details.

Claim 1:
A method (<NUM>) of processing a compressed representation of a spatial audio signal for generating a reconstructed representation of the spatial audio signal, wherein the compressed representation comprises a channel-based audio signal with a predefined number of channels and metadata, the metadata comprising direction information and energy information, with the direction information comprising indications of directions of arrival of one or more audio elements in an audio scene and the energy information comprising, for at least one frequency subband, respective indications of signal power associated with the directions of arrival, the method comprising:
generating (S1310) audio signals of the one or more audio elements based on the channel-based audio signal, the direction information, and the energy information; and
generating (S1320) a residual audio signal from which the one or more audio elements are substantially absent, based on the channel-based audio signal, the direction information, and the energy information,
wherein generating audio signals of the one or more audio elements comprises:
determining (<NUM>) coefficients of an inverse mixing matrix M for mapping the channel-based audio signal to an intermediate representation comprising the residual audio signal and the audio signals of the one or more audio elements, based on the direction information and the energy information,
wherein determining the coefficients of the inverse mixing matrix M comprises:
determining (S1410), for each of the one or more audio elements, a panning vector Pandown(dir) for panning the audio element to the channels of the channel-based audio signal, based on the direction of arrival dir of the audio element;
determining (S1420) a mixing matrix E that would be used for mapping the residual audio signal and the audio signals of the one or more audio elements to the channels of the channel-based audio signal, based on the determined panning vectors;
determining (S1430) a covariance matrix S for the intermediate representation based on the energy information; and
determining (S1440) the coefficients of the inverse mixing matrix M based on the mixing matrix F and the covariance matrix S.