Patent Description:
Higher Order Ambisonics (HOA) offers one possibility to represent three-dimensional sound among other techniques like wave field synthesis (WFS) or channel based approaches like the <NUM> multichannel audio format. In contrast to channel based methods, the HOA representation offers the advantage of being independent of a specific loudspeaker set-up. This flexibility, however, is at the expense of a decoding process which is required for the playback of the HOA representation on a particular loudspeaker set-up. Compared to the WFS approach, where the number of required loudspeakers is usually very large, HOA signals may also be rendered to set-ups consisting of only few loudspeakers. A further advantage of HOA is that the same representation can also be employed without any modification for binaural rendering to headphones.

HOA is based on the representation of the spatial density of complex harmonic plane wave amplitudes by a truncated Spherical Harmonics (SH) expansion. Each expansion coefficient is a function of angular frequency, which can be equivalently represented by a time domain function. Hence, without loss of generality, the complete HOA sound field representation actually can be assumed to consist of O time domain functions, where O denotes the number of expansion coefficients. These time domain functions will be equivalently referred to as HOA coefficient sequences or as HOA channels in the following. The spatial resolution of the HOA representation improves with a growing maximum order N of the expansion. Unfortunately, the number of expansion coefficients O grows quadratically with the order N, in particular O = (N + <NUM>)<NUM>. For example, typical HOA representations using order N = <NUM> require O = <NUM> HOA (expansion) coefficients. According to the previously made considerations, the total bit rate for the transmission of HOA representation, given a desired single-channel sampling rate fs and the number of bits Nb per sample, is determined by O · fs · Nb. Consequently, transmitting an HOA representation of order N = <NUM> with a sampling rate of fs = <NUM> employing Nb = <NUM> bits per sample results in a bit rate of <NUM>. 2MBits/s, which is very high for many practical applications like e.g. streaming. Thus, compression of HOA representations is highly desirable. The compression of HOA sound field representations is proposed in <CIT>, <CIT> and <CIT>. These processings have in common that they perform a sound field analysis and decompose the given HOA representation into a directional component and a residual ambient component. On one hand the final compressed representation is assumed to consist of a number of quantised signals, resulting from the perceptual coding of the directional signals and relevant coefficient sequences of the ambient HOA component. On the other hand it is assumed to comprise additional side information related to the quantised signals, which side information is necessary for the reconstruction of the HOA representation from its compressed version.

An important part of that side information is a description of a prediction of portions of the original HOA representation from the directional signals. Since for this prediction the original HOA representation is assumed to be equivalently represented by a number of spatially dispersed general plane waves impinging from spatially uniformly distributed directions, the prediction is referred to as spatial prediction in the following.

The coding of such side information related to spatial prediction is described in ISO/IEC JTC1/SC29/WG11, N14061, "Working Draft Text of MPEG-H 3D Audio HOA RM0", November <NUM> , Geneva, Switzerland. However, this state-of-the-art coding of the side information is rather inefficient.

A problem to be solved by the invention is to provide a more efficient way of decoding side information related to that spatial prediction.

This problem is solved by the method disclosed in claim <NUM>. An apparatus that utilises this method is disclosed in claim <NUM>. A corresponding computer program product is disclosed in claim <NUM>.

A bit is prepended to the coded side information representation data ζCOD, which bit signals whether or not any prediction is to be performed. This feature reduces over time the average bit rate for the transmission of the ζCOD data. Further, in specific situations, instead of using a bit array indicating for each direction if the prediction is performed or not, it is more efficient to transmit or transfer the number of active predictions and the respective indices. A single bit can be used for indicating in which way the indices of directions are coded for which a prediction is supposed to be performed. On average, this operation over time further reduces the bit rate for the transmission of the ζCOD data.

In principle, the inventive method is suited for improving the coding of side information required for coding a Higher Order Ambisonics representation of a sound field, denoted HOA, with input time frames of HOA coefficient sequences, wherein dominant directional signals as well as a residual ambient HOA component are determined and a prediction is used for said dominant directional signals, thereby providing, for a coded frame of HOA coefficients, side information data describing said prediction, and wherein said side information data can include:.

In principle the inventive apparatus is suited for improving the coding of side information required for coding a Higher Order Ambisonics representation of a sound field, denoted HOA, with input time frames of HOA coefficient sequences, wherein dominant directional signals as well as a residual ambient HOA component are determined and a prediction is used for said dominant directional signals, thereby providing, for a coded frame of HOA coefficients, side information data describing said prediction, and wherein said side information data can include:.

Advantageous additional embodiments of the invention are disclosed in the respective dependent claims.

Exemplary embodiments of the invention are described with reference to the accompanying drawings, which show in:.

In the following, the HOA compression and decompression processing described in patent application <CIT> is recapitulated in order to provide the context in which the inventive coding of side information related to spatial prediction is used.

In <FIG> it is illustrated how the coding of side information related to spatial prediction can be embedded into the HOA compression processing described patent application <CIT>. For the HOA representation compression, a frame-wise processing with non-overlapping input frames C(k) of HOA coefficient sequences of length L is assumed, where k denotes the frame index. The first step or stage <NUM>/<NUM> in <FIG> is optional and consists of concatenating the non-overlapping k-th and (k - <NUM>) -th frames of HOA coefficient sequences C(k) into a long frame C̃(k) as <MAT> which long frame is <NUM>% overlapped with an adjacent long frame and which long frame is successively used for the estimation of dominant sound source directions. Similar to the notation for C̃(k), the tilde symbol is used in the following description for indicating that the respective quantity refers to long overlapping frames. If step/stage <NUM>/<NUM> is not present, the tilde symbol has no specific meaning.

A parameter in bold means a set of values, e.g. a matrix or a vector.

The long frame C̃(k) is successively used in step or stage <NUM> for the estimation of dominant sound source directions as described in <CIT>. This estimation provides a data set <IMG>(k) ⊆ {<NUM>,. , D} of indices of the related directional signals that have been detected, as well as a data set <IMG>(k) of the corresponding direction estimates of the directional signals. D denotes the maximum number of directional signals that has to be set before starting the HOA compression and that can be handled in the known processing which follows.

In step or stage <NUM>, the current (long) frame C̃(k) of HOA coefficient sequences is decomposed (as proposed in <CIT>) into a number of directional signals XDIR(k - <NUM>) belonging to the directions contained in the set <IMG>(k), and a residual ambient HOA component CAMB(k - <NUM>). The delay of two frames is introduced as a result of overlap-add processing in order to obtain smooth signals. It is assumed that XDIR(k - <NUM>) is containing a total of D channels, of which however only those corresponding to the active directional signals are non-zero. The indices specifying these channels are assumed to be output in the data set <IMG>(k - <NUM>). Additionally, the decomposition in step/stage <NUM> provides some parameters ζ(k - <NUM>) which can be used at decompression side for predicting portions of the original HOA representation from the directional signals (see <CIT> for more details). In order to explain the meaning of the spatial prediction parameters ζ(k - <NUM>), the HOA decomposition is described in more detail in the below section HOA decomposition.

In step or stage <NUM>, the number of coefficients of the ambient HOA component CAMB(k - <NUM>) is reduced to contain only ORED + D - NDIR,ACT(k - <NUM>) non-zero HOA coefficient sequences, where NDIR,ACT(k - <NUM>) = |<IMG>(k - <NUM>)| indicates the cardinality of the data set <IMG>(k - <NUM>), i.e. the number of active directional signals in frame k - <NUM>. Since the ambient HOA component is assumed to be always represented by a minimum number ORED of HOA coefficient sequences, this problem can be actually reduced to the selection of the remaining D - NDIR,ACT(k - <NUM>) HOA coefficient sequences out of the possible O - ORED ones. In order to obtain a smooth reduced ambient HOA representation, this choice is accomplished such that, compared to the choice taken at the previous frame k - <NUM>, as few changes as possible will occur. The final ambient HOA representation with the reduced number of ORED + NDIR,ACT(k - <NUM>) non-zero coefficient sequences is denoted by CAMB,RED(k - <NUM>). The indices of the chosen ambient HOA coefficient sequences are output in the data set <IMG>(k - <NUM>). In step/stage <NUM>, the active directional signals contained in XDIR(k - <NUM>) and the HOA coefficient sequences contained in CAMB,RED(k - <NUM>) are assigned to the frame Y(k - <NUM>) of I channels for individual perceptual encoding as described in <CIT>. Perceptual coding step/stage <NUM> encodes the I channels of frame Y(k - <NUM>) and outputs an encoded frame <IMG>(k - <NUM>).

According to the invention, following the decomposition of the original HOA representation in step/stage <NUM>, the spatial prediction parameters or side information data ζ(k - <NUM>) resulting from the decomposition of the HOA representation are losslessly coded in step or stage <NUM> in order to provide a coded data representation ζCOD(k - <NUM>), using the index set <IMG>(k) delayed by two frames in delay <NUM>.

In <FIG> it is exemplary shown how to embed in step or stage <NUM> the decoding of the received encoded side information data ζCOD(k - <NUM>) related to spatial prediction into the HOA decompression processing described in <FIG> of patent application <CIT>. The decoding of the encoded side information data ζCOD(k - <NUM>) is carried out before entering its decoded version ζ(k - <NUM>) into the composition of the HOA representation in step or stage <NUM>, using the received index set <IMG>(k) delayed by two frames in delay <NUM>.

In step or stage <NUM> a perceptual decoding of the I signals contained in <IMG>(k - <NUM>) is performed in order to obtain the I decoded signals in Ŷ(k - <NUM>).

In signal re-distributing step or stage <NUM>, the perceptually decoded signals in Ŷ(k - <NUM>) are re-distributed in order to recreate the frame X̂DIR(k - <NUM>) of directional signals and the frame ĈAMB,RED(k - <NUM>) of the ambient HOA component. The information about how to re-distribute the signals is obtained by reproducing the assigning operation performed for the HOA compression, using the index data sets <IMG>(k) and <IMG>(k - <NUM>). In composition step or stage <NUM>, a current frame Ĉ(k - <NUM>) of the desired total HOA representation is re-composed (according to the processing described in connection with Fig. 2b and <FIG> of <CIT> using the frame X̂DIR(k - <NUM>) of the directional signals, the set <IMG>(k) of the active directional signal indices together with the set <IMG>(k) of the corresponding directions, the parameters ζ(k - <NUM>) for predicting portions of the HOA representation from the directional signals, and the frame ĈAMB,RED(k - <NUM>) of HOA coefficient sequences of the reduced ambient HOA component. ĈAMB,RED(k - <NUM>) corresponds to component D̂A(k - <NUM>) in <CIT>, and <IMG>(k) and <IMG>(k) correspond to AΩ̂(k) in <CIT>, wherein active directional signal indices can be obtained by taking those indices of rows of AΩ̂(k) which contain valid elements. , directional signals with respect to uniformly distributed directions are predicted from the directional signals X̌DIR(k - <NUM>) using the received parameters ζ(k - <NUM>) for such prediction, and thereafter the current decompressed frame Ĉ(k - <NUM>) is re-composed from the frame of directional signals X̂DIR(k - <NUM>), from <IMG>(k) and <IMG>(k), and from the predicted portions and the reduced ambient HOA component ĈAMB,RED(k - <NUM>).

In connection with <FIG> the HOA decomposition processing is described in detail in order to explain the meaning of the spatial prediction therein. This processing is derived from the processing described in connection with <FIG> of patent application <CIT>.

First, the smoothed dominant directional signals XDIR(k - <NUM>) and their HOA representation CDIR(k - <NUM>) are computed in step or stage <NUM>, using the long frame C̃(k) of the input HOA representation, the set <IMG>(k) of directions and the set <IMG>(k) of corresponding indices of directional signals. It is assumed that XDIR(k - <NUM>) contains a total of D channels, of which however only those corresponding to the active directional signals are non-zero. The indices specifying these channels are assumed to be output in the set <IMG>(k - <NUM>).

In step or stage <NUM> the residual between the original HOA representation C̃(k - <NUM>) and the HOA representation CDIR(k - <NUM>) of the dominant directional signals is represented by a number of O directional signals X̃RES(k - <NUM>), which can be considered as being general plane waves from uniformly distributed directions, which are referred to a uniform grid.

In step or stage <NUM> these directional signals are predicted from the dominant directional signals XDIR(k - <NUM>) in order to provide the predicted signals <MAT> together with the respective prediction parameters ζ(k - <NUM>). For the prediction only the dominant directional signals xDIR,d(k - <NUM>) with indices d, which are contained in the set <IMG>(k - <NUM>), are considered. The prediction is described in more detail in the below section Spatial prediction.

In step or stage <NUM> the smoothed HOA representation ĈRES(k - <NUM>) of the predicted directional signals <MAT> is computed. In step or stage <NUM> the residual CAMB(k - <NUM>) between the original HOA representation C̃(k - <NUM>) and the HOA representation CDIR(k - <NUM>) of the dominant directional signals together with the HOA representation ĈRES(k - <NUM>) of the predicted directional signals from uniformly distributed directions is computed and is output.

The required signal delays in the <FIG> processing are performed by corresponding delays <NUM> to <NUM>.

The goal of the spatial prediction is to predict the O residual signals <MAT> from the extended frame <MAT> <MAT> of smoothed directional signals (see the description in above section HOA decomposition and in patent application <CIT>).

Each residual signal x̃RES,GRID,q(k - <NUM>), q = <NUM>,. , O, represents a spatially dispersed general plane wave impinging from the direction Ωq, whereby it is assumed that all the directions Ωq, q = <NUM>,. , O are nearly uniformly distributed over the unit sphere. The total of all directions is referred to as a 'grid'. Each directional signal x̃DIR,d(k - <NUM>), d = <NUM>,. ,D represents a general plane wave impinging from a trajectory interpolated between the directions ΩACT,d(k - <NUM>), ΩACT,d(k - <NUM>), ΩACT,d(k - <NUM>) and ΩACT,d(k), assuming that the d-th directional signal is active for the respective frames.

To illustrate the meaning of the spatial prediction by means of an example, the decomposition of an HOA representation of order N = <NUM> is considered, where the maximum number of directions to extract is equal to D = <NUM>. For simplicity it is further assumed that only the directional signals with indices '<NUM>' and '<NUM>' are active, while those with indices '<NUM>' and '<NUM>' are non-active. Additionally, for simplicity it is assumed that the directions of the dominant sound sources are constant for the considered frames, i.e. ΩACT,d(k - <NUM>) = ΩACT,d(k - <NUM>) = ΩACT,d(k - <NUM>) = ΩACT,d(k) = ΩACT,d for d = <NUM>,<NUM> (<NUM>) As a consequence of order N = <NUM>, there are O = <NUM> directions Ωq of spatially dispersed general plane waves x̃RES,GRID,q(k - <NUM>), q = <NUM>,. <FIG> shows these directions together with the directions ΩACT,<NUM> and ΩACT,<NUM> of the active dominant sound sources.

One way of describing the spatial prediction is presented in the above-mentioned ISO/IEC document. In this document, the signals x̃RES,GRID,q(k - <NUM>), q = <NUM>,. , O are assumed to be predicted by a weighted sum of a predefined maximum number DPRED of directional signals, or by a low pass filtered version of the weighted sum. The side information related to spatial prediction is described by the parameter set ζ(k - <NUM>) = {pTYPE(k - <NUM>),PIND(k - <NUM>),PQ,F(k - <NUM>)}, which consists of the following three components:.

The following two parameters have to be known at decoding side for enabling the appropriate interpretation of these parameters:.

These two parameters have to either be set to fixed values known to the encoder and decoder, or to be additionally transmitted, but distinctly less frequently than the frame rate. The latter option may be used for adapting the two parameters to the HOA representation to be compressed.

An example for a parameter set may look like the following, assuming O = <NUM>, DPRED = <NUM> and BSC = <NUM>: <MAT> <MAT> <MAT>.

Such parameters would mean that the general plane wave signal x̃RES,GRID,<NUM>(k - <NUM>) from direction Ω<NUM> is predicted from the directional signal x̃DIR,<NUM>(k - <NUM>) from direction ΩACT,<NUM> by a pure multiplication (i.e. full band) with a factor that results from de-quantising the value <NUM>. Further, the general plane wave signal x̃RES,GRID,<NUM>(k - <NUM>) from direction Ω<NUM> is predicted from the directional signals x̃DIR,<NUM>(k - <NUM>) and x̃DIR,<NUM>(k - <NUM>) by a lowpass filtering and multiplication with factors that result from de-quantising the values <NUM> and -<NUM>.

Given this side information, the prediction is assumed to be performed as follows:
First, the quantised prediction factors pQ,F,d,q(k - <NUM>),.

d = <NUM>,. , DPRED , q = <NUM>,. ,O are dequantised to provide the actual prediction factors <MAT>.

As already mentioned, BSC denotes a predefined number of bits to be used for the quantisation of the prediction factors. Additionally, pF,d,q(k - <NUM>) is assumed to be set to zero, if pIND,d,q(k - <NUM>) is equal to zero.

For the previously mentioned example, assuming BSC = <NUM>, the dequantised prediction factor vector would result in <MAT>.

Further, for performing a low pass prediction a predefined low pass FIR filter <MAT> of length Lh = <NUM> is used. The filter delay is given by Dh = <NUM> samples.

Assuming as signals the predicted signals <MAT> and the directional signals <MAT> <MAT> <MAT> the sample values of the predicted signals are given by <MAT> with ỹLP,q(k - <NUM>, l) := <MAT>.

As already mentioned and as now can be seen from equation (<NUM>), the signals x̃RES,GRID,q(k - <NUM>), q = <NUM>,. , O are assumed to be predicted by a weighted sum of a predefined maximum number DPRED of directional signals, or by a low pass filtered versions of the weighted sum.

In the above-mentioned ISO/IEC document the coding of the spatial prediction side information is addressed. It is summarised in Algorithm <NUM> depicted in <FIG> and will be explained in the following. For a clearer presentation the frame index k - <NUM> is neglected in all expressions.

First, a bit array ActivePred consisting of O bits is created, in which the bit ActivePred[q] indicates whether or not for the direction Ωq a prediction is performed. The number of 'ones' in this array is denoted by NumActivePred.

Next, the bit array PredType of length NumActivePred is created where each bit indicates, for the directions where a prediction is to be performed, the kind of the prediction, i.e. full band or low pass. At the same time, the unsigned integer array PredDirSigIds of length NumActivePred · DPRED is created, whose elements denote for each active prediction the DPRED indices of the directional signals to be used. If less than DPRED directional signals are to be used for the prediction, the indices are assumed to be set to zero. Each element of the array PredDirSigIds is assumed to be represented by <MAT> bits. The number of non-zero elements in the array PredDirSigIds is denoted by NumNonZerolds.

Finally, the integer array QuantPredGains of length NumNonZerolds is created, whose elements are assumed to represent the quantised scaling factors PQ,F,d,q(k - <NUM>) to be used in equation (<NUM>). The dequantisation to obtain the corresponding dequantised scaling factors PF,d,q(k - <NUM>) is given in equation (<NUM>). Each element of the array QuantPredGains is assumed to be represented by BSC bits.

In the end, the coded representation of the side information ζCOD consists of the four aforementioned arrays according to ζCOD = [ActivePred PredType PredDirSigIds QuantPredGains]. (<NUM>) For explaining this coding by an example, the coded representation of equations (<NUM>) to (<NUM>) is used: <MAT> <MAT> <MAT> <MAT>.

The number of required bits is equal to <NUM> + <NUM> + <NUM> · <NUM> + <NUM> · <NUM> = <NUM>.

In order to increase the efficiency of the coding of the side information related to spatial prediction, the state-of-the-art processing is advantageously modified.

The above modifications A) to C) for the known side information coding processing result in the example coding processing depicted in <FIG> (not encompassed by the claims).

Consequently, the coded side information consists of the following components: <MAT>.

Remark: in the above-mentioned ISO/IEC document e.g. in section <NUM>. <NUM>, QuantPredGains is called PredGains, which however contains quantised values.

The coded representation for the example in equations (<NUM>) to (<NUM>) would be: <MAT> <MAT> <MAT> <MAT> <MAT> <MAT> <MAT> and the required number of bits is <NUM> + <NUM> + <NUM> + <NUM> · <NUM> + <NUM> + <NUM> · <NUM> + <NUM> · <NUM> = <NUM>. Advantageously, compared to the state of the art coded representation in equations (<NUM>) to (<NUM>), this representation coded according to the invention requires <NUM> bits less.

The decoding of the modified side information related to spatial prediction is summarised in the example decoding processing depicted in <FIG> and <FIG> (the processing depicted in <FIG> is the continuation of the processing depicted in <FIG>) and is explained in the following.

Initially, all elements of vector pTYPE and matrices PIND and PQ,F are initialised by zero. Then the bit PSPredictionActive is read, which indicates if a spatial prediction is to be performed at all. In the case of a spatial prediction (i.e. PSPredictionActive = <NUM>), the bit KindOfCodedPredIds is read, which indicates the kind of coding of the indices of directions for which a prediction is to be performed.

In the case that KindOfCodedPredIds = <NUM>, the bit array ActivePred of length O is read, of which the q-th element indicates if for the direction Ωq a prediction is performed or not. In a next step, from the array ActivePred the number NumActivePred of predictions is computed and the bit array PredType of length NumActivePred is read, of which the elements indicate the kind of prediction to be performed for each of the relevant directions. With the information contained in ActivePred and PredType, the elements of the vector pTYPE are computed.

In case KindOfCodedPredIds = <NUM>, the number NumActivePred of active predictions is read, which is assumed to be coded with <MAT> bits, where MM is the greatest integer number satisfying equation (<NUM>). Then, the data array PredIds consisting of NumActivePred elements is read, where each element is assumed to be coded by <MAT> bits. The elements of this array are the indices of directions, where a prediction has to be performed. Successively, the bit array PredType of length NumActivePred is read, of which the elements indicate the kind of prediction to be performed for each one of the relevant directions. With the knowledge of NumActivePred, PredIds and PredType, the elements of the vector pTYPE are computed.

For both cases (i.e. KindOfCodedPredIds = <NUM> and KindOfCodedPredIds = <NUM>) , in the next step the array PredDirSigIds is read, which consists of NumActivePred · DPRED elements. Each element is assumed to be coded by <MAT> bits. Using the information contained in pTYPE, <IMG> and PredDirSigIds, the elements of matrix PIND are set and the number NumNonZerolds of non-zero elements in PIND is computed.

Finally, the array QuantPredGains is read, which consists of NumNonZerolds elements, each coded by BSC bits. Using the information contained in PIND and QuantPredGains, the elements of the matrix PQ,F are set.

Claim 1:
Method for decoding side information data required for decoding an encoded Higher Order Ambisonics, HOA, representation of a sound field, the encoded HOA representation comprising dominant directional signals as well as a residual ambient HOA component, wherein the side information for a coded frame of HOA coefficients describes a prediction used for said dominant directional signals, wherein the side information can include a bit array (ActivePred) indicating whether or not for a direction a prediction is performed,
said method comprising:
- evaluating a bit value (PSPredictionActive) indicating whether or not said prediction is to be performed;
- if said prediction is to be performed, decoding the side information describing said prediction, including decoding the bit array (ActivePred),
wherein decoding the side information describing said prediction comprises:
- evaluating a bit value (KindOfCodedPredIds) indicating that
said bit array (ActivePred) indicating whether or not for a direction a prediction is to be performed is used in the decoding of said side information data (ζ(k - <NUM>)),
wherein:
- evaluating said bit array (ActivePred) indicating whether or not for a direction a prediction is to be performed wherein its elements indicate if for a corresponding direction a prediction is performed;
- computing, for those elements of the bit array (ActivePred) that indicate that prediction is to be performed for the corresponding direction, the elements of a vector (pTYPE), wherein the elements of the vector (pTYPE) indicate if for a corresponding direction a prediction is performed, and if so, then the elements also indicate which kind of prediction.