Patent Description:
Spatial audio coding tools are well-known in the art and are, for example, standardized in the MPEG-surround standard. Spatial audio coding starts from original input channels such as five or seven channels which are identified by their placement in a reproduction setup, i.e., a left channel, a center channel, a right channel, a left surround channel, a right surround channel and a low frequency enhancement channel. A spatial audio encoder typically derives one or more downmix channels from the original channels and, additionally, derives parametric data relating to spatial cues such as interchannel level differences in the channel coherence values, interchannel phase differences, interchannel time differences, etc. The one or more downmix channels are transmitted together with the parametric side information indicating the spatial cues to a spatial audio decoder which decodes the downmix channel and the associated parametric data in order to finally obtain output channels which are an approximated version of the original input channels. The placement of the channels in the output setup is typically fixed and is, for example, a <NUM> format, a <NUM> format, etc..

Additionally, spatial audio object coding tools are well-known in the art and are standardized in the MPEG SAOC standard (SAOC = spatial audio object coding). In contrast to spatial audio coding starting from original channels, spatial audio object coding starts from audio objects which are not automatically dedicated for a certain rendering reproduction setup. Instead, the placement of the audio objects in the reproduction scene is flexible and can be determined by the user by inputting certain rendering information into a spatial audio object coding decoder. Alternatively or additionally, rendering information, i.e., information at which position in the reproduction setup a certain audio object is to be placed typically over time can be transmitted as additional side information or metadata. In order to obtain a certain data compression, a number of audio objects are encoded by an SAOC encoder which calculates, from the input objects, one or more transport channels by downmixing the objects in accordance with certain downmixing information. Furthermore, the SAOC encoder calculates parametric side information representing inter-object cues such as object level differences (OLD), object coherence values, etc. As in SAC (SAC = Spatial Audio Coding), the inter object parametric data is calculated for individual time/frequency tiles, i.e., for a certain frame of the audio signal comprising, for example, <NUM> or <NUM> samples, <NUM>, <NUM>, or <NUM>, etc., frequency bands are considered so that, in the end, parametric data exists for each frame and each frequency band. As an example, when an audio piece has <NUM> frames and when each frame is subdivided into <NUM> frequency bands, then the number of time/frequency tiles is <NUM>.

Up to now no flexible technology exists combining channel coding on the one hand and object coding on the other hand so that acceptable audio qualities at low bit rates are obtained.

<CIT> discloses an end-to-end solution for creating, encoding, transmitting, decoding and reproducing spatial audio soundtracks. The provided soundtrack encoding format is compatible with legacy surround- sound encoding formats, so that soundtracks encoded in the new format may be decoded and reproduced on legacy playback equipment with no loss of quality compared to legacy formats. Audio objects are included into a base downmix on the encoder-side, and the thus obtained downmix and the explicitly encoded audio objects are transmitted to a decoder-side. On the decoder side, the objects are removed from the transmitted downmix and separately rendered and combined with the residual downmix corresponding to the base downmix.

<CIT> discloses an encoding apparatus for a High Quality Multi-channel Audio Codec (HQMAC) and a decoding apparatus for the HQMAC. The encoding/decoding apparatuses for the HQMAC may perform a High Quality Multi-channel Audio Codec-Channel Based (HQMAC-CB) encoding or an HQMAC-CB decoding in accordance with characteristics of inputted audio signals to provide compatibility with a lower channel.

It is an object of the present invention to provide an improved concept for audio decoding.

This object is achieved by an audio decoder of claim <NUM>, a method of audio decoding of claim <NUM>, or a computer program of claim <NUM>.

The present invention is based on the finding that, for an optimum system being flexible on the one hand and providing a good compression efficiency at a good audio quality on the other hand is achieved by combining spatial audio coding, i.e., channel-based audio coding with spatial audio object coding, i.e., object based coding. In particular, providing a mixer for mixing the objects and the channels already on the encoder-side provides a good flexibility, particularly for low bit rate applications, since any object transmission can then be unnecessary or the number of objects to be transmitted can be reduced. On the other hand, flexibility is required so that the audio encoder can be controlled in two different modes, i.e., in the mode in which the objects are mixed with the channels before being core-encoded, while in the other mode the object data on the one hand and the channel data on the other hand are directly core-encoded without any mixing in between.

This makes sure that the user can either separate the processed objects and channels on the encoder-side so that a full flexibility is available on the decoder side but, at the price of an enhanced bit rate. On the other hand, when bit rate requirements are more stringent, then the present invention already allows to perform a mixing/pre-rendering on the encoder-side, i.e., that some or all audio objects are already mixed with the channels so that the core encoder only encodes channel data and any bits required for transmitting audio object data either in the form of a downmix or in the form of parametric inter object data are not required.

On the decoder-side, the user has again high flexibility due to the fact that the same audio decoder allows the operation in two different modes, i.e., the first mode where individual or separate channel and object coding takes place and the decoder has the full flexibility to rendering the objects and mixing with the channel data. On the other hand, when a mixing/pre-rendering has already taken place on the encoder-side, the decoder is configured to perform a post-processing without any intermediate object processing. On the other hand, the post-processing can also be applied to the data in the other mode, i.e., when the object rendering/mixing takes place on the decoder-side. Thus, the present invention allows a framework of processing tasks which allows a great re-use of resources not only on the encoder side but also on the decoder side. The post-processing may refer to downmixing and binauralizing or any other processing to obtain a final channel scenario such as an intended reproduction layout.

Furthermore, in case of very low bit rate requirements, the present invention provides the user with enough flexibility to react to the low bit rate requirements, i.e., by pre-rendering on the encoder-side so that, for the price of some flexibility, nevertheless very good audio quality on the decoder-side is obtained due to the fact that the bits which have been saved by not providing any object data anymore from the encoder to the decoder can be used for better encoding the channel data such as by finer quantizing the channel data or by other means for improving the quality or for reducing the encoding loss when enough bits are available.

In a preferred embodiment of the present invention, the encoder additionally comprises an SAOC encoder and furthermore allows to not only encode objects input into the encoder but to also SAOC encode channel data in order to obtain a good audio quality at even lower required bit rates. Further embodiments of the present invention allow a post-processing functionality which comprises a binaural renderer and/or a format converter. Furthermore, it is preferred that the whole processing on the decoder side already takes place for a certain high number of loud speakers such as a <NUM> or <NUM> channel loudspeaker setup. However, then the format converter, for example, determines that only a <NUM> output, i.e., an output for a reproduction layout is required which has a lower number than the maximum number of channels, then it is preferred that the format converter controls either the USAC decoder or the SAOC decoder or both devices to restrict the core decoding operation and the SAOC decoding operation so that any channels which are, in the end, nevertheless down mixed into a format conversion are not generated in the decoding. Typically, the generation of upmixed channels requires decorrelation processing and each decorrelation processing introduces some level of artifacts. Therefore, by controlling the core decoder and/or the SAOC decoder by the finally required output format, a great deal of additional decorrelation processing is saved compared to a situation when this interaction does not exist which not only results in an improved audio quality but also results in a reduced complexity of the decoder and, in the end, in a reduced power consumption which is particularly useful for mobile devices housing the encoder or the inventive decoder. The encoders/ inventive decoders, however, cannot only be introduced in mobile devices such as mobile phones, smart phones, notebook computers or navigation devices but can also be used in straightforward desktop computers or any other non-mobile appliances.

The above implementation, i.e. to not generate some channels, may be not optimum, since some information may be lost (such as the level difference between the channels that will be downmixed). This level difference information may not be critical, but may result in a different downmix output signal, if the downmix applies different downmix gains to the upmixed channels. An improved solution only switches off the decorrelation in the upmix, but still generates all upmix channels with correct level differences (as signaled by the parametric SAC). The second solution results in a better audio quality, but the first solution results in greater complexity reduction.

Preferred embodiments are subsequently discussed with respect to the accompanying drawings, in which:.

<FIG> illustrates an encoder in accordance with an example of the present invention. The encoder is configured for encoding audio input data <NUM> to obtain audio output data <NUM>. The encoder comprises an input interface for receiving a plurality of audio channels indicated by CH and a plurality of audio objects indicated by OBJ. Furthermore, as illustrated in <FIG>, the input interface <NUM> additionally receives metadata related to one or more of the plurality of audio objects OBJ. Furthermore, the encoder comprises a mixer <NUM> for mixing the plurality of objects and the plurality of channels to obtain a plurality of pre-mixed channels, wherein each pre-mixed channel comprises audio data of a channel and audio data of at least one object.

Furthermore, the encoder comprises a core encoder <NUM> for core encoding core encoder input data, a metadata compressor <NUM> for compressing the metadata related to the one or more of the plurality of audio objects. Furthermore, the encoder can comprise a mode controller <NUM> for controlling the mixer, the core encoder and/or an output interface <NUM> in one of several operation modes, wherein in the first mode, the core encoder is configured to encode the plurality of audio channels and the plurality of audio objects received by the input interface <NUM> without any interaction by the mixer, i.e., without any mixing by the mixer <NUM>. In a second mode, however, in which the mixer <NUM> was active, the core encoder encodes the plurality of mixed channels, i.e., the output generated by block <NUM>. In this latter case, it is preferred to not encode any object data anymore. Instead, the metadata indicating positions of the audio objects are already used by the mixer <NUM> to render the objects onto the channels as indicated by the metadata. In other words, the mixer <NUM> uses the metadata related to the plurality of audio objects to pre-render the audio objects and then the pre-rendered audio objects are mixed with the channels to obtain mixed channels at the output of the mixer. In this example, any objects may not necessarily be transmitted and this also applies for compressed metadata as output by block <NUM>. However, if not all objects input into the interface <NUM> are mixed but only a certain amount of objects is mixed, then only the remaining non-mixed objects and the associated metadata nevertheless are transmitted to the core encoder <NUM> or the metadata compressor <NUM>, respectively.

<FIG> illustrates a further example of an encoder which, additionally, comprises an SAOC encoder <NUM>. The SAOC encoder <NUM> is configured for generating one or more transport channels and parametric data from spatial audio object encoder input data. As illustrated in <FIG>, the spatial audio object encoder input data are objects which have not been processed by the pre-renderer/mixer. Alternatively, provided that the pre-renderer/mixer has been bypassed as in the mode one where an individual channel/object coding is active, all objects input into the input interface <NUM> are encoded by the SAOC encoder <NUM>.

Furthermore, as illustrated in <FIG>, the core encoder <NUM> is preferably implemented as a USAC encoder, i.e., as an encoder as defined and standardized in the MPEG-USAC standard (USAC = unified speech and audio coding). The output of the whole encoder illustrated in <FIG> is an MPEG <NUM> data stream having the container-like structures for individual data types. Furthermore, the metadata is indicated as "OAM" data and the metadata compressor <NUM> in <FIG> corresponds to the OAM encoder <NUM> to obtain compressed OAM data which are input into the USAC encoder <NUM> which, as can be seen in <FIG>, additionally comprises the output interface to obtain the MP4 output data stream not only having the encoded channel/object data but also having the compressed OAM data.

<FIG> illustrates a further example of the encoder, where in contrast to <FIG>, the SAOC encoder can be configured to either encode, with the SAOC encoding algorithm, the channels provided at the pre-renderer/mixer 200not being active in this mode or, alternatively, to SAOC encode the pre-rendered channels plus objects. Thus, in <FIG>, the SAOC encoder <NUM> can operate on three different kinds of input data, i.e., channels without any pre-rendered objects, channels and pre-rendered objects or objects alone. Furthermore, it is preferred to provide an additional OAM decoder <NUM> in <FIG> so that the SAOC encoder <NUM> uses, for its processing, the same data as on the decoder side, i.e., data obtained by a lossy compression rather than the original OAM data.

The <FIG> encoder can operate in several individual modes.

In addition to the first and the second modes as discussed in the context of <FIG>, the <FIG> encoder can additionally operate in a third mode in which the core encoder generates the one or more transport channels from the individual objects when the pre-renderer/mixer <NUM> was not active. Alternatively or additionally, in this third mode the SAOC encoder <NUM> can generate one or more alternative or additional transport channels from the original channels, i.e., again when the pre-renderer/mixer <NUM> corresponding to the mixer <NUM> of <FIG> was not active.

Finally, the SAOC encoder <NUM> can encode, when the encoder is configured in the fourth mode, the channels plus pre-rendered objects as generated by the pre-renderer/mixer. Thus, in the fourth mode the lowest bit rate applications will provide good quality due to the fact that the channels and objects have completely been transformed into individual SAOC transport channels and associated side information as indicated in <FIG> and <FIG> as "SAOC-SI" and, additionally, any compressed metadata do not have to be transmitted in this fourth mode.

<FIG> illustrates a decoder in accordance with an embodiment of the present invention. The decoder receives, as an input, the encoded audio data, i.e., the data <NUM> of <FIG>. The decoder comprises a metadata decompressor <NUM>, a core decoder <NUM>, an object processor <NUM>, a mode controller <NUM> and a post-processor <NUM>.

Specifically, the audio decoder is configured for decoding encoded audio data and the input interface is configured for receiving the encoded audio data, the encoded audio data comprising a plurality of encoded channels and the plurality of encoded objects and compressed metadata related to the plurality of objects in a certain mode.

Furthermore, the core decoder <NUM> is configured for decoding the plurality of encoded channels and the plurality of encoded objects and, additionally, the metadata decompressor is configured for decompressing the compressed metadata.

Furthermore, the object processor <NUM> is configured for processing the plurality of decoded objects as generated by the core decoder <NUM> using the decompressed metadata to obtain a predetermined number of output channels comprising object data and the decoded channels. These output channels as indicated at <NUM> are then input into a post-processor <NUM>. The post-processor <NUM> is configured for converting the number of output channels <NUM> into a certain output format which can be a binaural output format or a loudspeaker output format such as a <NUM>, <NUM>, etc., output format.

The decoder comprises a mode controller <NUM> which is configured for analyzing the encoded data to detect a mode indication. Therefore, the mode controller <NUM> is connected to the input interface <NUM> in <FIG>. The audio decoder in <FIG> and, controlled by the mode controller <NUM>, is configured to either bypass the object processor and to feed the plurality of decoded channels into the post-processor <NUM>. This is the operation in mode <NUM>, i.e., in which only pre-rendered channels are received, i.e., when mode <NUM> has been applied in the encoder of <FIG>. Alternatively, when mode <NUM> has been applied in the encoder, i.e., when the encoder has performed individual channel/object coding, then the object processor <NUM> is not bypassed, but the plurality of decoded channels and the plurality of decoded objects are fed into the object processor <NUM> together with decompressed metadata generated by the metadata decompressor <NUM>.

Inventively, the indication whether mode <NUM> or mode <NUM> is to be applied is included in the encoded audio data and the mode controller <NUM> analyses the encoded data to detect a mode indication. Mode <NUM> is used when the mode indication indicates that the encoded audio data comprises encoded channels and encoded objects and mode <NUM> is applied when the mode indication indicates that the encoded audio data does not contain any audio objects, i.e., only contain pre-rendered channels obtained by mode <NUM> of the <FIG> encoder.

<FIG> illustrates a preferred embodiment compared to the <FIG> decoder and the embodiment of <FIG> corresponds to the encoder of <FIG>. In addition to the decoder implementation of <FIG>, the decoder in <FIG> comprises an SAOC decoder <NUM>. Furthermore, the object processor <NUM> of <FIG> is implemented as a separate object renderer <NUM> and the mixer <NUM> while, depending on the mode, the functionality of the object renderer <NUM> can also be implemented by the SAOC decoder <NUM>.

Furthermore, the post-processor <NUM> can be implemented as a binaural renderer <NUM> or a format converter <NUM>. Alternatively, a direct output of data <NUM> of <FIG> can also be implemented as illustrated by <NUM>. Therefore, it is preferred to perform the processing in the decoder on the highest number of channels such as <NUM> or <NUM> in order to have flexibility and to then post-process if a smaller format is required. However, when it becomes clear from the very beginning that only small format such as a <NUM> format is required, then it is preferred, as indicated by <FIG> or <FIG> by the shortcut <NUM>, that a certain control over the SAOC decoder and/or the USAC decoder can be applied in order to avoid unnecessary upmixing operations and subsequent downmixing operations.

In a preferred embodiment of the present invention, the object processor <NUM> comprises the SAOC decoder <NUM> and the SAOC decoder is configured for decoding one or more transport channels output by the core decoder and associated parametric data and using decompressed metadata to obtain the plurality of rendered audio objects. To this end, the OAM output is connected to box <NUM>.

Furthermore, the object processor <NUM> is configured to render decoded objects output by the core decoder which are not encoded in SAOC transport channels but which are individually encoded in typically single channeled elements as indicated by the object renderer <NUM>. Furthermore, the decoder comprises an output interface corresponding to the output <NUM> for outputting an output of the mixer to the loudspeakers.

In a further embodiment, the object processor <NUM> comprises a spatial audio object coding decoder <NUM> for decoding one or more transport channels and associated parametric side information representing encoded audio objects or encoded audio channels, wherein the spatial audio object coding decoder is configured to transcode the associated parametric information and the decompressed metadata into transcoded parametric side information usable for directly rendering the output format, as for example defined in an earlier version of SAOC. The post-processor <NUM> is configured for calculating audio channels of the output format using the decoded transport channels and the transcoded parametric side information. The processing performed by the post-processor can be similar to the MPEG Surround processing or can be any other processing such as BCC processing or so.

In a further embodiment, the object processor <NUM> comprises a spatial audio object coding decoder <NUM> configured to directly upmix and render channel signals for the output format using the decoded (by the core decoder) transport channels and the parametric side information
Furthermore, and importantly, the object processor <NUM> of <FIG> additionally comprises the mixer <NUM> which receives, as an input, data output by the USAC decoder <NUM> directly when pre-rendered objects mixed with channels exist, i.e., when the mixer <NUM> of <FIG> was active. Additionally, the mixer <NUM> receives data from the object renderer performing object rendering without SAOC decoding. Furthermore, the mixer receives SAOC decoder output data, i.e., SAOC rendered objects.

The mixer <NUM> is connected to the output interface <NUM>, the binaural renderer <NUM> and the format converter <NUM>. The binaural renderer <NUM> is configured for rendering the output channels into two binaural channels using head related transfer functions or binaural room impulse responses (BRIR). The format converter <NUM> is configured for converting the output channels into an output format having a lower number of channels than the output channels <NUM> of the mixer and the format converter <NUM> requires information on the reproduction layout such as <NUM> speakers or so.

The <FIG> decoder is different from the <FIG> decoder in that the SAOC decoder cannot only generate rendered objects but also rendered channels and this is the case when the <FIG> encoder has been used and the connection <NUM> between the channels/pre-rendered objects and the SAOC encoder <NUM> input interface is active.

Furthermore, a vector base amplitude panning (VBAP) stage <NUM> is configured which receives, from the SAOC decoder, information on the reproduction layout and which outputs a rendering matrix to the SAOC decoder so that the SAOC decoder can, in the end, provide rendered channels without any further operation of the mixer in the high channel format of <NUM>, i.e., <NUM> loudspeakers.

the VBAP block preferably receives the decoded OAM data to derive the rendering matrices. More general, it preferably requires geometric information not only of the reproduction layout but also of the positions where the input signals should be rendered to on the reproduction layout. This geometric input data can be OAM data for objects or channel position information for channels that have been transmitted using SAOC.

However, if only a specific output interface is required then the VBAP state <NUM> can already provide the required rendering matrix for the e.g., <NUM> output. The SAOC decoder <NUM> then performs a direct rendering from the SAOC transport channels, the associated parametric data and decompressed metadata, a direct rendering into the required output format without any interaction of the mixer <NUM>. However, when a certain mix between modes is applied, i.e., where several channels are SAOC encoded but not all channels are SAOC encoded or where several objects are SAOC encoded but not all objects are SAOC encoded or when only a certain amount of pre-rendered objects with channels are SAOC decoded and remaining channels are not SAOC processed then the mixer will put together the data from the individual input portions, i.e., directly from the core decoder <NUM>, from the object renderer <NUM> and from the SAOC decoder <NUM>.

Subsequently, <FIG> is discussed for indicating certain encoder/decoder modes which can be applied by the inventive highly flexible and high quality audio encoder/decoder concept.

In accordance with the first coding mode, the mixer <NUM> in the <FIG> encoder is bypassed and, therefore, the object processor in the <FIG> decoder is not bypassed.

In the second mode, the mixer <NUM> in <FIG> is active and the object processor in <FIG> is bypassed.

Then, in the third coding mode, the SAOC encoder of <FIG> is active but only SAOC encodes the objects rather than channels or channels as output by the mixer. Therefore, mode <NUM> requires that, on the decoder side illustrated in <FIG>, the SAOC decoder is only active for objects and generates rendered objects.

In a fourth coding mode as illustrated in <FIG>, the SAOC encoder is configured for SAOC encoding pre-rendered channels, i.e., the mixer is active as in the second mode. On the decoder side, the SAOC decoding is performed for pre-rendered objects so that the object processor is bypassed as in the second coding mode.

Furthermore, a fifth coding mode exists which can by any mix of modes <NUM> to <NUM>. In particular, a mix coding mode will exist when the mixer <NUM> in <FIG> receives channels directly from the USAC decoder and, additionally, receives channels with pre-rendered objects from the USAC decoder. Furthermore, in this mixed coding mode, objects are encoded directly using, preferably, a single channel element of the USAC decoder. In this context, the object renderer <NUM> will then render these decoded objects and forward them to the mixer <NUM>. Furthermore, several objects are additionally encoded by an SAOC encoder so that the SAOC decoder will output rendered objects to the mixer and/or rendered channels when several channels encoded by SAOC technology exist.

Each input portion of the mixer <NUM> can then, exemplarily, have at least a potential for receiving the number of channels such as <NUM> as indicated at <NUM>. Thus, basically, the mixer could receive <NUM> channels from the USAC decoder and, additionally, <NUM> pre-rendered/mixed channels from the USAC decoder and, additionally, <NUM> "channels" from the object renderer and, additionally, <NUM> "channels" from the SAOC decoder, where each "channel" between blocks <NUM> and <NUM> on the one hand and block <NUM> on the other hand has a contribution of the corresponding objects in a corresponding loudspeaker channel and then the mixer <NUM> mixes, i.e., adds up the individual contributions for each loudspeaker channel.

In a preferred embodiment of the present invention, the encoding/decoding system is based on an MPEG-D USAC codec for coding of channel and object signals. To increase the efficiency for coding a large amount of objects, MPEG SAOC technology has been adapted. Three types of renderers perform the task of rendering objects to channels, rendering channels to headphones or rendering channels to a different loudspeaker setup. When object signals are explicitly transmitted or parametrically encoded using SAOC, the corresponding object metadata information is compressed and multiplexed into the encoded output data.

In an embodiment, the pre-renderer/mixer <NUM> is used to convert a channel plus object input scene into a channel scene before encoding. Functionally, it is identical to the object renderer/mixer combination on the decoder side as illustrated in <FIG> or <FIG> and as indicated by the object processor <NUM> of <FIG>. Pre-rendering of objects ensures a deterministic signal entropy at the encoder input that is basically independent of the number of simultaneously active object signals. With pre-rendering of objects, no object metadata transmission is required. Discrete object signals are rendered to the channel layout that the encoder is configured to use. The weights of the objects for each channel are obtained from the associated object metadata OAM as indicated by arrow <NUM>.

As a core/encoder/decoder for loudspeaker channel signals, discrete object signals, object downmix signals and pre-rendered signals, a USAC technology is preferred. It handles the coding of the multitude of signals by creating channel and object mapping information (the geometric and semantic information of the input channel and object assignment). This mapping information describes how input channels and objects are mapped to USAC channel elements as illustrated in <FIG>, i.e., channel pair elements (CPEs), single channel elements (SCEs), channel quad elements (QCEs) and the corresponding information is transmitted to the core decoder from the core encoder. All additional payloads like SAOC data or object metadata have been passed through extension elements and have been considered in the encoder's rate control.

The coding of objects is possible in different ways, depending on the rate/distortion requirements and the interactivity requirements for the renderer. The following object coding variants are possible:.

The SAOC encoder and decoder for object signals are based on MPEG SAOC technology. The system is capable of recreating, modifying and rendering a number of audio objects based on a smaller number of transmitted channels and additional parametric data (OLDs, lOCs (Inter Object Coherence), DMGs (Down Mix Gains)). The additional parametric data exhibits a significantly lower data rate than required for transmitting all objects individually, making the coding very efficient.

The SAOC encoder takes as input the object/channel signals as monophonic waveforms and outputs the parametric information (which is packed into the 3D-Audio bitstream) and the SAOC transport channels (which are encoded using single channel elements and transmitted).

The SAOC decoder reconstructs the object/channel signals from the decoded SAOC transport channels and parametric information, and generates the output audio scene based on the reproduction layout, the decompressed object metadata information and optionally on the user interaction information.

For each object, the associated metadata that specifies the geometrical position and volume of the object in 3D space is efficiently coded by quantization of the object properties in time and space. The compressed object metadata cOAM is transmitted to the receiver as side information. The volume of the object may comprise information on a spatial extent and/or information of the signal level of the audio signal of this audio object.

The object renderer utilizes the compressed object metadata to generate object waveforms according to the given reproduction format. Each object is rendered to certain output channels according to its metadata. The output of this block results from the sum of the partial results.

If both channel based content as well as discrete/parametric objects are decoded, the channel based waveforms and the rendered object waveforms are mixed before outputting the resulting waveforms (or before feeding them to a post-processor module like the binaural renderer or the loudspeaker renderer module).

The binaural renderer module produces a binaural downmix of the multichannel audio material, such that each input channel is represented by a virtual sound source. The processing is conducted frame-wise in QMF (Quadrature Mirror Filterbank) domain.

The binauralization is based on measured binaural room impulse responses
<FIG> illustrates a preferred embodiment of the format converter <NUM>. The loudspeaker renderer or format converter converts between the transmitter channel configuration and the desired reproduction format. This format converter performs conversions to lower number of output channels, i.e., it creates downmixes. To this end, a downmixer <NUM> which preferably operates in the QMF domain receives mixer output signals <NUM> and outputs loudspeaker signals. Preferably, a controller <NUM> for configuring the downmixer <NUM> is provided which receives, as a control input, a mixer output layout, i.e., the layout for which data <NUM> is determined and a desired reproduction layout is typically been input into the format conversion block <NUM> illustrated in <FIG>. Based on this information, the controller <NUM> preferably automatically generates optimized downmix matrices for the given combination of input and output formats and applies these matrices in the downmixer block <NUM> in the downmix process. The format converter allows for standard loudspeaker configurations as well as for random configurations with non-standard loudspeaker positions.

As illustrated in the context of <FIG>, the SAOC decoder is designed to render to the predefined channel layout such as <NUM> with a subsequent format conversion to the target reproduction layout. Alternatively, however, the SAOC decoder is implemented to support the "low power" mode where the SAOC decoder is configured to decode to the reproduction layout directly without the subsequent format conversion. In this implementation, the SAOC decoder <NUM> directly outputs the loudspeaker signal such a the <NUM> loudspeaker signals and the SAOC decoder <NUM> requires the reproduction layout information and the rendering matrix so that the vector base amplitude panning or any other kind of processor for generating downmix information can operate.

<FIG> illustrates a further embodiment of the binaural renderer <NUM> of <FIG>. Specifically, for mobile devices the binaural rendering is required for headphones attached to such mobile devices or for loudspeakers directly attached to typically small mobile devices. For such mobile devices, constraints may exist to limit the decoder and rendering complexity. In addition to omitting decorrelation in such processing scenarios, it is preferred to firstly downmix using the downmixer <NUM> to an intermediate downmix, i.e., to a lower number of output channels which then results in a lower number of input channel for the binaural converter <NUM>. Exemplarily, <NUM> channel material is downmixed by the downmixer <NUM> to a <NUM> intermediate downmix or, alternatively, the intermediate downmix is directly calculated by the SAOC decoder <NUM> of <FIG> in a kind of a "shortcut" mode. Then, the binaural rendering only has to apply ten HRTFs (Head Related Transfer Functions) or BRIR functions for rendering the five individual channels at different positions in contrast to apply <NUM> HRTF for BRIR functions if the <NUM> input channels would have already been directly rendered. Specifically, the convolution operations necessary for the binaural rendering require a lot of processing power and, therefore, reducing this processing power while still obtaining an acceptable audio quality is particularly useful for mobile devices.

Preferably, the "shortcut" as illustrated by control line <NUM> comprises controlling the decoder <NUM> to decode to a lower number of channels, i.e., skipping the complete OTT processing block in the decoder or a format converting to a lower number of channels and, as illustrated in <FIG>, the binaural rendering is performed for the lower number of channels. The same processing can be applied not only for binaural processing but also for a format conversion as illustrated by line <NUM> in <FIG>.

In a further embodiment, an efficient interfacing between processing blocks is required. Particularly in <FIG>, the audio signal path between the different processing blocks is depicted. The binaural renderer <NUM>, the format converter <NUM>, the SAOC decoder <NUM> and the USAC decoder <NUM>, in case SBR (spectral band replication) is applied, all operate in a QMF or hybrid QMF domain. In accordance with an embodiment, all these processing blocks provide a QMF or a hybrid QMF interface to allow passing audio signals between each other in the QMF domain in an efficient manner. Additionally, it is preferred to implement the mixer module and the object renderer module to work in the QMF or hybrid QMF domain as well. As a consequence, separate QMF or hybrid QMF analysis and synthesis stages can be avoided which results in considerable complexity savings and then only a final QMF synthesis stage is required for generating the loudspeakers indicated at <NUM> or for generating the binaural data at the output of block <NUM> or for generating the reproduction layout speaker signals at the output of block <NUM>.

Subsequently, reference is made to <FIG> in order to explain quad channel elements (QCE). In contrast to a channel pair element as defined in the US AC-MPEG standard, a quad channel element requires four input channels <NUM> and outputs an encoded QCE element <NUM>. In one embodiment, a hierarchy of two MPEG Surround boxes in <NUM>-<NUM>-<NUM> Mode or two TTO boxes (TTO = Two To One) boxes and additional joint stereo coding tools (e.g. MS-Stereo) as defined in MPEG USAC or MPEG surround are provided and the QCE element not only comprises two jointly stereo coded downmix channels and optionally two jointly stereo coded residual channels and, additionally, parametric data derived from the, for example, two TTO boxes. On the decoder side, a structure is applied where the joint stereo decoding of the two downmix channels and optionally of the two residual channels is applied and in a second stage with two OTT boxes the downmix and optional residual channels are upmixed to the four output channels. However, alternative processing operations for one QCE encoder can be applied instead of the hierarchical operation. Thus, in addition to the joint channel coding of a group of two channels, the core encoder/decoder additionally uses a joint channel coding of a group of four channels.

Furthermore, it is preferred to perform an enhanced noise filling procedure to enable uncompromised full-band (<NUM>) coding at <NUM> kbps.

The encoder has been operated in a 'constant rate with bit-reservoir' fashion, using a maximum of <NUM> bits per channel as rate buffer for the dynamic data.

All additional payloads like SAOC data or object metadata have been passed through extension elements and have been considered in the encoder's rate control.

In order to take advantage of the SAOC functionalities also for 3D audio content, the following extensions to MPEG SAOC have been implemented:.

The binaural renderer module produces a binaural downmix of the multichannel audio material, such that each input channel (excluding the LFE channels) is represented by a virtual sound source. The processing is conducted frame-wise in QMF domain.

The binauralization is based on measured binaural room impulse responses. The direct sound and early reflections are imprinted to the audio material via a convolutional approach in a pseudo-FFT domain using a fast convolution on-top of the QMF domain.

A further embodiment of the inventive method is, therefore, a data carrier (or a digital storage medium, or a computer-readable medium) comprising, recorded thereon, the computer program for performing one of the methods described herein.

A further embodiment of the invention method is, therefore, a data stream or a sequence of signals representing the computer program for performing one of the methods described herein.

Claim 1:
Audio decoder for decoding encoded audio data, comprising:
an input interface (<NUM>) configured for receiving the encoded audio data, the encoded audio data comprising a plurality of encoded audio channels or a plurality of encoded audio objects and compressed metadata related to the plurality of audio objects, and a mode indication;
a core decoder (<NUM>) configured for decoding the plurality of encoded audio channels and the plurality of encoded audio objects;
a metadata decompressor (<NUM>) configured for decompressing the compressed metadata;
an object processor (<NUM>) configured for processing the plurality of decoded audio objects using the decompressed metadata to obtain a number of output audio channels (<NUM>) comprising audio data from the audio objects and the decoded audio channels;
a mode controller (<NUM>) connected to the input interface (<NUM>) and configured for analyzing the encoded audio data to detect the mode indication indicating a first mode or a second mode, wherein, in the first mode, the encoded audio data comprise encoded audio channels and encoded audio objects, and wherein, in the second mode, the encoded audio data only comprise the plurality of encoded audio channels; and
a post-processor (<NUM>) configured for converting the number of output audio channels (<NUM>) into an output format,
wherein the audio decoder, controlled by the mode controller (<NUM>), is configured to bypass the object processor (<NUM>) and to feed a plurality of decoded channels into the post-processor (<NUM>), when the second mode has been detected by the mode controller (<NUM>), and to feed the plurality of decoded audio objects and the plurality of decoded audio channels into the object processor (<NUM>), when the first mode has been detected by the mode controller (<NUM>).