Patent Description:
Another embodiment according to the invention is related to an apparatus for providing an upmix signal representation on the basis of the downmix signal representation and the parametric side information.

Another embodiment according to the invention is related to a method for providing one or more adjusted parameters for a provision of an upmix signal representation on the basis of a downmix signal representation and a parametric side information associated with the downmix signal representation.

Another embodiment according to the invention is related to a computer program for performing said method.

Some embodiments according to the invention are related to a parameter limiting scheme for distortion control in MPEG SAOC.

In the art of audio processing, audio transmission and audio storage, there is an increasing desire to handle multi-channel contents in order to improve the hearing impression. Usage of multi-channel audio content brings along significant improvements for the user. For example, a <NUM>-dimensional hearing impression can be obtained, which brings along an improved user satisfaction in entertainment applications. However, multi-channel audio contents are also useful in professional environments, for example in telephone conferencing applications, because the speaker intelligibility can be improved by using a multi-channel audio playback.

However, it is also desirable to have a good tradeoff between audio quality and bitrate requirements in order to avoid an excessive resource load caused by multi-channel applications.

Recently, parametric techniques for the bitrate-efficient transmission and/or storage of audio scenes containing multiple audio objects has been proposed, for example, Binaural Cue Coding (Type I) (see, for example, reference [<NUM>]), Joint Source Coding (see, for example, reference [<NUM>]), and MPEG Spatial Audio Object Coding (SAOC) (see, for example, references [<NUM>], [<NUM>], [<NUM>]).

In combination with user interactivity at the receiving side, such techniques may lead to a low audio quality of the output signals if extreme object rendering is performed (see, for example, reference [<NUM>]).

These techniques aim at perceptually reconstructing the desired output audio scene rather than by a waveform match.

<FIG> shows a system overview of such a system (here: MPEG SAOC). The MPEG SAOC system <NUM> shown in <FIG> comprises an SAOC encoder <NUM> and an SAOC decoder <NUM>. The SAOC encoder <NUM> receives a plurality of object signals x<NUM> to xN, which may be represented, for example, as time-domain signals or as time-frequency-domain signals (for example, in the form of a set of transform coefficients of a Fourier-type transform, or in the form of QMF subband signals). The SAOC encoder <NUM> typically also receives downmix coefficients d<NUM> to dN, which are associated with the object signals x<NUM> to xN. Separate sets of downmix coefficients may be available for each channel of the downmix signal. The SAOC encoder <NUM> is typically configured to obtain a channel of the downmix signal by combining the object signals x<NUM> to xN in accordance with the associated downmix coefficients d<NUM> to dN. Typically, there are less downmix channels than object signals x<NUM> to xN. In order to allow (at least approximately) for a separation (or separate treatment) of the object signals at the side of the SAOC decoder <NUM>, the SAOC encoder <NUM> provides both the one or more downmix signals (designated as downmix channels) <NUM> and a side information <NUM>. The side information <NUM> describes characteristics of the object signals x<NUM> to xN, in order to allow for a decoder-sided object-specific processing.

The SAOC decoder <NUM> is configured to receive both the one or more downmix signals <NUM> and the side information <NUM>. Also, the SAOC decoder <NUM> is typically configured to receive a user interaction information and/or a user control information <NUM>, which describes a desired rendering setup. For example, the user interaction information/user control information <NUM> may describe a speaker setup and the desired spatial placement of the objects which provide the object signals x<NUM> to xN.

The SAOC decoder <NUM> is configured to provide, for example, a plurality of decoded upmix channel signals ŷ<NUM> to ŷM. The upmix channel signals may for example be associated with individual speakers of a multi-speaker rendering arrangement. The SAOC decoder <NUM> may, for example, comprise an object separator 820a, which is configured to reconstruct, at least approximately, the object signals x<NUM> to xN on the basis of the one or more downmix signals <NUM> and the side information <NUM>, thereby obtaining reconstructed object signals 820b. However, the reconstructed object signals 820b may deviate somewhat from the original object signals x<NUM> to xN, for example, because the side information <NUM> is not quite sufficient for a perfect reconstruction due to the bitrate constraints. The SAOC decoder <NUM> may further comprise a mixer 820c, which may be configured to receive the reconstructed object signals 820b and the user interaction information/user control information <NUM>, and to provide, on the basis thereof, the upmix channel signals ŷ<NUM> to ŷM. The mixer 820c may be configured to use the user interaction information /user control information <NUM> to determine the contribution of the individual reconstructed object signals 820b to the upmix channel signals ŷ<NUM> to ŷM. The user interaction information/user control information <NUM> may, for example, comprise rendering parameters (also designated as rendering coefficients), which determine the contribution of the individual reconstructed object signals <NUM> to the upmix channel signals ŷ<NUM> to ŷM.

However, it should be noted that in many embodiments, the object separation, which is indicated by the object separator 820a in <FIG>, and the mixing, which is indicated by the mixer 820c in <FIG>, are performed in one single step. For this purpose, overall parameters may be computed which describe a direct mapping of the one or more downmix signals <NUM> onto the upmix channel signals ŷ<NUM> to ŷM. These parameters may be computed on the basis of the side information and the user interaction information/user control information <NUM>.

Taking reference now to <FIG>, <FIG> and <FIG>, different apparatus for obtaining an upmix signal representation on the basis of a downmix signal representation and object-related side information will be described. It should be noted that the object-related side information is an example of a side information associated with the downmix signal. <FIG> shows a block schematic diagram of an MPEG SAOC system <NUM> comprising an SAOC decoder <NUM>. The SAOC decoder <NUM> comprises, as separate functional blocks, an object decoder <NUM> and a mixer/renderer <NUM>. The object decoder <NUM> provides a plurality of reconstructed object signals <NUM> in dependence on the downmix signal representation (for example, in the form of one or more downmix signals represented in the time domain or in the time-frequency-domain) and object-related side information (for example, in the form of object meta data). The mixer/renderer <NUM> receives the reconstructed object signals <NUM> associated with a plurality of N objects and provides, on the basis thereof and on the rendering information, one or more upmix channel signals <NUM>. In the SAOC decoder <NUM>, the extraction of the object signals <NUM> is performed separately from the mixing/rendering which allows for a separation of the object decoding functionality from the mixing/rendering functionality but brings along a relatively high computational complexity.

Taking reference now to <FIG>, another MPEG SAOC system <NUM> will be briefly discussed, which comprises an SAOC decoder <NUM>. The SAOC decoder <NUM> provides a plurality of upmix channel signals <NUM> in dependence on a downmix signal representation (for example, in the form of one or more downmix signals) and an object-related side information (for example, in the form of object meta data). The SAOC decoder <NUM> comprises a combined object decoder and mixer/renderer, which is configured to obtain the upmix channel signals <NUM> in a joint mixing process without a separation of the object decoding and the mixing/rendering, wherein the parameters for said joint upmix process are dependent both on the object-related side information and the rendering information. The joint upmix process depends also on the downmix information, which is considered to be part of the object-related side information.

To summarize the above, the provision of the upmix channel signals <NUM>, <NUM> can be performed in a one step process or a two step process.

Taking reference now to <FIG>, an MPEG SAOC system <NUM> will be described. The SAOC system <NUM> comprises an SAOC to MPEG Surround transcoder <NUM>, rather than an SAOC decoder.

The SAOC to MPEG Surround transcoder comprises a side information transcoder <NUM>, which is configured to receive the object-related side information (for example, in the form of object meta data) and, optionally, information on the one or more downmix signals and the rendering information. The side information transcoder is also configured to provide an MPEG Surround side information (for example, in the form of an MPEG Surround bitstream) on the basis of a received data. Accordingly, the side information transcoder <NUM> is configured to transform an object-related (parametric) side information, which is received from the object encoder, into a channel-related (parametric) side information, taking into consideration the rendering information and, optionally, the information about the content of the one or more downmix signals.

Optionally, the SAOC to MPEG Surround transcoder <NUM> may be configured to manipulate the one or more downmix signals, described, for example, by the downmix signal representation, to obtain a manipulated downmix signal representation <NUM>. However, the downmix signal manipulator <NUM> may be omitted, such that the output downmix signal representation <NUM> of the SAOC to MPEG Surround transcoder <NUM> is identical to the input downmix signal representation of the SAOC to MPEG Surround transcoder. The downmix signal manipulator <NUM> may, for example, be used if the channel-related MPEG Surround side information <NUM> would not allow to provide a desired hearing impression on the basis of the input downmix signal representation of the SAOC to MPEG Surround transcoder <NUM>, which may be the case in some rendering constellations.

Accordingly, the SAOC to MPEG Surround transcoder <NUM> provides the downmix signal representation <NUM> and the MPEG Surround bitstream <NUM> such that a plurality of upmix channel signals, which represent the audio objects in accordance with the rendering information input to the SAOC to MPEG Surround transcoder <NUM> can be generated using an MPEG Surround decoder which receives the MPEG Surround bitstream <NUM> and the downmix signal representation <NUM>.

To summarize the above, different concepts for decoding SAOC-encoded audio signals can be used. In some cases, an SAOC decoder is used, which provides upmix channel signals (for example, upmix channel signals <NUM>, <NUM>) in dependence on the downmix signal representation and the object-related parametric side information. Examples for this concept can be seen in <FIG> and <FIG>. Alternatively, the SAOC-encoded audio information may be transcoded to obtain a downmix signal representation (for example, a downmix signal representation <NUM>) and a channel-related side information (for example, the channel-related MPEG Surround bitstream <NUM>), which can be used by an MPEG Surround decoder to provide the desired upmix channel signals.

In the MPEG SAOC system <NUM>, a system overview of which is given in <FIG>, the general processing is carried out in a frequency selective way and can be described as follows within each frequency band:.

It has been found that such a scheme is tremendously efficient, both in terms of transmission bitrate (it is only necessary to transmit a few downmix channels plus some side information instead of N discrete object audio signals or a discrete system) and computational complexity (the processing complexity relates mainly to the number of output channels rather than the number of audio objects). Further advantages for the user on the receiving end include the freedom of choosing a rendering setup of his/her choice (mono, stereo, surround, virtualized headphone playback, and so on) and the feature of user interactivity: the rendering matrix, and thus the output scene, can be set and changed interactively by the user according to will, personal preference or other criteria. For example, it is possible to locate the talkers from one group together in one spatial area to maximize discrimination from other remaining talkers. This interactivity is achieved by providing a decoder user interface.

For each transmitted sound object, its relative level and (for non-mono rendering) spatial position of rendering can be adjusted. This may happen in real-time as the user changes the position of the associated graphical user interface (GUI) sliders (for example: object level = +5dB, object position = -30deg).

However, it has been found that the decoder-sided choice of parameters for the provision of the upmix signal representation (e.g. the upmix channel signals ŷ<NUM> to ŷM) brings along audible degradations in some cases. The document "<NPL>, discloses means for limiting the amount of perceptible SAOC system distortion for demanding rendering scenarios to ensure a satisfactory user experience.

In view of this situation, it is the objective of the present invention to create a concept which allows for reducing or even avoiding audible distortion when providing an upmix signal representation (for example, in the form of upmix channel signals ŷ<NUM> to ŷM).

In the following, an apparatus for providing one or more adjusted parameters for a provision of an upmix signal representation on the basis of a downmix signal representation and a parametric side information associated with the downmix signal representation will be described. <FIG> shows a block schematic diagram of such an apparatus <NUM>.

The apparatus <NUM> is configured to receive one or more input parameters <NUM> and to provide, on the basis thereof, one or more adjusted parameters <NUM>. The apparatus <NUM> comprises a parameter adjuster <NUM> which is configured to receive the one or more input parameters <NUM> and to provide, on the basis thereof, the one or more adjusted parameters <NUM>. The parameter adjuster <NUM> is configured to provide the one or more adjusted parameters <NUM> in dependence on an average value <NUM> of a plurality of input parameter values, such that a distortion of an upmix signal representation caused by the use of non-optimal parameters (for example, the one or more input parameters <NUM>) is reduced at least for input parameters (for example, input parameters <NUM>) deviating from optimal parameters by more than a predetermined deviation. For example, the parameter adjuster <NUM> may have the effect that the one or more adjusted parameters <NUM> are "closer" (in the sense of causing smaller distortions) to optimal parameters (which would result in a distortion-free upmix signal representation) than the one or more input parameters <NUM>.

For this purpose, the parameter adjuster <NUM> implements an average value computation, to obtain the average value <NUM> (for example, as a temporal average or an inter-object average) of a set of related input parameters <NUM> (for example, input parameters associated with a common time interval, or input parameters of the same parameter type associated with different time instances). Regarding the operation of the apparatus <NUM>, it should be noted that the provision of the one or more adjusted parameters <NUM> on the basis of the one or more input parameters <NUM> is made in dependence on the average value <NUM>, because it has been found that the average value <NUM> is a meaningful quantity for adjusting the parameters. In particular, it has been found that moderate parameters (with respect to the average value) typically bring along moderate distortions.

Further details will be described subsequently.

In the following, an apparatus for providing an upmix signal representation according to <FIG> will be described. <FIG> shows a block schematic diagram of such an apparatus <NUM>, which can be considered as an audio signal decoder. For example, the apparatus <NUM> may comprise the functionality of an SAOC decoder or an SAOC transcoder.

The apparatus <NUM> is configured to receive a downmix signal representation <NUM> and a parametric side information <NUM>. Also, the apparatus <NUM> is configured to receive user-specified rendering parameters <NUM>. The apparatus is configured to provide an upmix signal representation <NUM>.

The downmix signal representation <NUM> may, for example, be a representation of a one-channel audio signal or of a two-channel audio signal. The downmix signal representation <NUM> may, for example, be a time domain representation or an encoded representation. In some embodiments, the downmix signal representation <NUM> may be a time-frequency-domain representation, in which the one or more channels of the downmix signal representation <NUM> are represented by subsequent sets of spectral values.

The upmix signal representation <NUM> may, for example, be a representation of individual audio channels, for example, in the form of a time domain representation or a time-frequency-domain representation. Alternatively, the upmix signal representation <NUM> may be an encoded representation, comprising both a downmix signal representation and a channel-related side information, for example, an MPEG Surround side information.

The user-specified rendering parameters <NUM> may be provided in the form of rendering matrix entries describing desired contributions of a plurality of audio objects to the one or more channels of the upmix signal representation <NUM>. Alternatively, the user-specified rendering parameters <NUM> may be provided in any other appropriate form, for example, specifying a desired rendering position and rendering volume of the audio objects.

The apparatus <NUM> comprises a signal processor <NUM>, which is configured to provide the upmix signal representation <NUM> on the basis of the downmix signal representation <NUM> and the parametric side information <NUM>. The signal processor <NUM> comprises a remixing functionality <NUM> in order to provide the upmix signal representation <NUM> on the basis of the downmix signal representation <NUM>. For example, the remixing functionality <NUM> may be configured to linearly combine a plurality of channels of the downmix signal representation <NUM> in order to obtain the one or more channels of the upmix signal representation <NUM>. In this remixing, contributions of the channels of the downmix signal representation <NUM> to the channels of the upmix signal representation <NUM> may be determined by mix matrix elements of a mix matrix G, wherein a first dimension (for example, a number of rows) of the mix matrix G may be determined by the number of channels of the upmix signal representation <NUM>, and wherein a second dimension (for example, a number of columns) of the mix matrix G may be determined by a number of channels of the downmix signal representation <NUM>.

For example, the remixing process <NUM> may be used to provide one or more vectors comprising spectral values associated with one or more channels of the upmix signal representation <NUM> by multiplying one or more vectors comprising spectral values of one or more channels of the downmix signal representation <NUM> with the mix matrix G.

The signal processor <NUM> may also comprise a mixing parameter computation <NUM> which provides the mix matrix G (or equivalently, the elements thereof). The mix matrix elements are determined in dependence on the parametric side information <NUM> and modified rendering parameters <NUM> by the mixing parameter computation <NUM>. The mix matrix elements of the mix matrix G are, for example, provided such that the one or more channels of the upmix signal representation <NUM> describe audio objects, which are represented by the one or more channels of the downmix signal representation <NUM>, in accordance with the modified rendering parameters <NUM>. For this purpose, the parametric side information <NUM> is evaluated by the mixing parameter computation <NUM>, wherein the parametric side information <NUM> comprises, for example, an object-level difference information OLD, an inter-object-correlation information IOC, a downmix gain information DMG and (optionally) a downmix-channel-level-difference information DCLD. The object-level difference information may describe, for example, in a frequency-band-wise manner, level differences between a plurality of audio objects. Similarly, the inter-object-correlation information may describe, for example, in a frequency-band-wise manner, correlations between a plurality of audio objects. The downmix-gain information and the (optional) downmix-channel-level-difference information may describe the downmix, which is performed to combine audio object signals from a plurality of audio objects into the one or more channels of the downmix signal representation, wherein there are typically more audio objects than channels of the downmix signal representation <NUM>.

Accordingly, the mixing parameter computation <NUM> may evaluate how the mix matrix elements should be chosen in order to obtain an upmix signal representation <NUM> comprising expected statistic properties on the basis of the parametric side information <NUM> and the modified rendering parameters <NUM>.

The signal processor <NUM> may optionally comprise a side information modification or side information transformation <NUM>, which is configured to receive the parametric side information <NUM> and to provide a modified side information (for example, an MPEG Surround side information), such that the modified side information and the associated remixed downmix signal representation provided by the remixing process <NUM> describe a desired audio scene.

To summarize, the signal processor <NUM> may, for example, fulfill the functionality of the SAOC decoder <NUM>, wherein the downmix signal representation <NUM> takes the role of the one or more downmix signals <NUM>, wherein the parametric side information <NUM> takes the role of the side information <NUM>, and wherein the upmix signal representation <NUM> is equivalent to the output channel signals ŷ<NUM> to ŷM.

Alternatively, the signal processor <NUM> may comprise the functionality of the separate decoder and mixer <NUM>, wherein the downmix signal representation <NUM> may take the role of the one or more downmix signals, wherein the parametric side information <NUM> may take the role of the object meta data, and wherein the upmix signal representation <NUM> may take the role of the one or more output channel signals <NUM>.

Alternatively, the signal processor <NUM> may comprise the functionality of the integrated decoder and mixer <NUM>, wherein the downmix signal representation <NUM> may take the role of the one or more downmix signals, wherein the parametric side information <NUM> may take the role of the object meta data, and wherein the upmix signal representation <NUM> may take the role of the one or more output channel signals <NUM>.

Alternatively, the signal processor <NUM> may comprise the functionality of the SAOC-to-MPEG surround transcoder <NUM>, wherein the downmix signal representation <NUM> may take the role of the one or more downmix signals, wherein the parametric side information <NUM> may take the role of the object meta data, and wherein the upmix signal representation may be equivalent to the one or more downmix signals <NUM> when taken in combination with the MPEG surround bitstream <NUM>.

In any case, the modified rendering parameters <NUM> may take the role of the user interaction/control information <NUM> or of the rendering information.

The apparatus <NUM> also comprises an apparatus <NUM> for providing adjusted rendering parameters. The apparatus <NUM> for providing the adjusted rendering parameters receives the user-specified rendering parameters <NUM> and provides, on the basis thereof, the modified rendering parameters <NUM>. The apparatus <NUM> is typically configured to calculate an average value over a plurality of user-specified rendering parameters associated with different audio objects, to obtain an average value. Also, the apparatus <NUM> is configured to perform a rendering parameter limitation in dependence on the average value, to obtain the modified rendering parameters <NUM> by limiting the user-specified rendering parameters <NUM>. A tolerance interval, to which the modified rendering parameters <NUM> are limited, is typically determined in dependence on the average value, such that strong deviations of the modified rendering parameters <NUM> from the average value are avoided, even if one or more of the user-specified rendering parameters <NUM> comprises such a strong deviation from the average value. In this manner, excessive distortions within the upmix signal representation <NUM> are typically avoided, because the modified rendering parameters <NUM>, which comprise limited inter-object deviation, will result in an upmix signal representation with low-distortions, while a large difference between rendering parameters associated with different audio objects would typically result in audible artifacts.

It should be noted here that the apparatus <NUM> for providing adjusted rendering coefficients may comprise the same overall functionality as apparatus <NUM> for providing one or more adjusted parameters, wherein the user-specified rendering parameters <NUM> may take the role of one or more input parameters <NUM>, and wherein the adjusted rendering parameters <NUM> may take the role of the one or more adjusted parameters <NUM>.

Details regarding the provision of the modified rendering parameters <NUM> will be discussed below, taking reference to <FIG>.

In the following, an apparatus for providing an upmix signal representation according to another embodiment of the invention will be described taking reference to <FIG>, which shows a block schematic diagram of such an apparatus <NUM>.

The apparatus <NUM> typically receives the same type of input signals and provides the same type of output signals as the apparatus <NUM>, such that identical reference numerals are used herein to describe identical or equivalent signals. To summarize, the apparatus <NUM> receives a downmix signal representation <NUM>, parametric side information <NUM> and user-specified rendering parameters <NUM>, and the apparatus <NUM> provides, on the basis thereof, an upmix signal representation <NUM>.

The apparatus <NUM> comprises a signal processor <NUM>, which may be substantially equivalent in the functionality to the signal processor <NUM>. The signal processor <NUM> comprises a remixing functionality <NUM>,which is identical to the remixing functionality <NUM> of the signal processor <NUM> in that it provides remixed audio channel signals on the basis of the downmix signal representation. However, the remixing <NUM> uses an adjusted mix matrix, rather than a mix matrix obtained directly from a mixing parameter computation.

The signal processor <NUM> also comprises a mixing parameter computation <NUM>, which may be identical in function to the mixing parameter computation <NUM> of the signal processor <NUM>. Accordingly, the mixing parameter computation <NUM> receives the parametric side information <NUM> and the user-specified rendering parameters <NUM>, and provides, on the basis thereof, a mix matrix G (or equivalently, mix matrix elements of the mix matrix G, which are also designated with <NUM>).

The signal processor <NUM> optionally also comprises a side information modification <NUM>, the functionality of which is identical to the side information modification <NUM>.

In addition, the apparatus <NUM> comprises an apparatus <NUM> for providing adjusted mix matrix elements. The apparatus <NUM> may or may not be part of the signal processor <NUM>. The apparatus <NUM> is configured to receive the mix matrix <NUM>, G (or, equivalently, the mix matrix elements thereof), which are provided by the mixing parameter computation <NUM>, and to provide, on the basis thereof, an adjusted mix matrix <NUM>' (or, equivalently, adjusted mix matrix elements thereof). For example, one set of mix matrix elements and one set of adjusted mix matrix elements may be provided per frequency band and per audio frame. In other words, the mix matrix G and the modified mix matrix G' may be updated once per audio frame of the downmix signal representation <NUM>, if a frame-wise processing is chosen. However, the update interval may be different in some cases. Also, it is not necessary that there are multiple mix matrices and adjusted mix matrices G, G' for different frequency bands.

However, the apparatus <NUM> is configured to provide adjusted mix matrix elements of the adjusted mix matrix <NUM> on the basis of the mix matrix elements of the mix matrix <NUM> provided by the mixing parameter computation <NUM>. For example, the processing may be performed individually per position of the mix matrix (or adjusted mix matrix), such that a sequence of adjusted mix matrix elements of a given mix matrix position may be dependent on a sequence of mix matrix elements of the mix matrix <NUM> at the same mix matrix position, but independent from mix matrix elements at different mix matrix positions.

The apparatus <NUM> for providing an adjusted mix matrix element is configured to provide the one or more adjusted mix matrix elements of the adjusted mix matrix <NUM> in dependence on one or more average values (for example, one or more matrix-position-individual average values) computed on the basis of the mix matrix <NUM>. The apparatus <NUM> for providing the adjusted mix matrix elements of the adjusted mix matrix <NUM> is preferably configured to calculate an average value of mix matrix elements at a given mix matrix position over time. Thus, for a given mix matrix position, an average value (preferably, but not necessarily, a temporal average value, like, for example, a floating average or a quasi-infinite-impulse-response average value or an average value obtained by a recursive low pass filtering or similar mathematical operations well-known for time averaging) may be computed on the basis of a sequence of mix matrix elements of the given mix matrix position. For example, a sequence of mix matrix elements describing a contribution of a given channel of the downmix signal representation <NUM> onto a given channel of the upmix signal representation <NUM>, which mix matrix elements are associated with a plurality of audio frames, may be used in order to obtain such an average value (also designates as mean value), which average value may be a finite-impulse-response average value or a (quasi) infinite-impulse-response average value (obtained, for example, using a recursive low pass filtering or similar mathematical operations well-known for time averaging). A current adjusted mix matrix element of the given mix matrix position (describing the contribution of the given channel of the downmix signal representation <NUM> onto the given channel of the upmix signal representation <NUM>) may be limited by the apparatus <NUM> to a tolerance interval which is defined in dependence on the average value associated to the given mix matrix position.

Accordingly, excessive temporal fluctuations of mix matrix elements are avoided, because adjusted mix matrix elements are restricted to a tolerance interval which is determined, for example, by an average (finite-impulse-response average or infinite-impulse-response average) of previous mix matrix elements at the same mix matrix position. It has been found that such a restriction of the adjusted mix matrix elements of the adjusted mix matrix <NUM> typically brings along a limitation of the distortions of the upmix signal <NUM> caused by the use of non-optimal parameters (for example non-optimal user-specified rendering parameters) at least if the non-optimal user-specified rendering parameters deviate from optimal user-specified rendering parameters by more than a predetermined deviation.

It should be noted here that the apparatus <NUM> for providing adjusted mix matrix elements may comprise the same overall functionality as apparatus <NUM> for providing one or more adjusted parameters, wherein the mix matrix elements of the mix matrix <NUM> may take the role of one or more input parameters <NUM>, and wherein the adjusted mix matrix elements of the adjusted mix matrix <NUM> may take the role of the one or more adjusted parameters <NUM>.

In the following, parameter limiting schemes according to the invention will be described taking reference to <FIG>, which shows a schematic representation of such parameter limiting schemes.

<FIG> shows the application of parameter limiting schemes in combination with an SAOC decoder <NUM>. However, the parameter limiting schemes may be applied in combination with different types of audio decoders or audio transcoders, like, for example, an SAOC transcoder.

SAOC decoder <NUM> receives a downmix <NUM> and an SAOC bitstream <NUM>. Also, the SAOC decoder provides one or more output channels 430a to <NUM>.

In a first implementation, designated with (a), the parameter limiting scheme <NUM> implements an indirect control. The parameter limiting scheme <NUM> receives an input rendering matrix R, for example, a user specified rendering matrix, and provides, on the basis thereof, an adjusted rendering matrix R̃ to the SAOC decoder. In this case, the SAOC decoder uses the adjusted rendering matrix R̃ for a derivation of the mix matrix G, as described above. The parameter limiting scheme <NUM> may also receive parameters ΛR-, ΛR+, which may determine boundaries of a tolerance interval.

In addition, a second parameter limiting scheme <NUM> may be applied. The second parameter limiting scheme receives transcoding parameters T and provides, on the basis thereof, adjusted transcoding parameters T̃. The transcoding parameters T may be computed in the SAOC decoder <NUM>, and the adjusted transcoding parameters T̃ may be applied by the SAOC decoder <NUM>. For example, the transcoding parameters T may be equivalent to the mix matrix elements of the mix matrix G, as discussed before, and the adjusted transcoding parameters T̃ may be equivalent to the adjusted mix matrix elements of the adjusted mix matrix G'.

The parameter limiting scheme <NUM> may receive one or more parameters ΛT-, ΛT+, which parameters may determine boundaries of tolerance intervals.

In the following, an overview will be given over the parameter limiting scheme for distortion control.

The general SAOC processing is carried out in a time/frequency selective way and will be described in the following.

The SAOC encoder extracts the psychoacoustic characteristics (for example, object power relations and correlations) of several input audio object signals and then downmixes them into a combined mono or stereo channel (which may be designated, for example, as a downmix signal representation). This downmix signal and extracted side information are transmitted (or stored) in compressed format using the well-known perceptual audio coders. On the receiving end, the SAOC decoder conceptually tries to restore the original object signal (i.e., separate downmixed objects) using the transmitted side information (for example, object-level-difference information OLD, inter-object-correlation information IOC, downmix-gain information DMG and downmix-channel-level-difference information DCLD). These approximated object signals are then mixed into a target scene using a rendering matrix (wherein the rendering matrix typically describes contributions of different audio objects to different channels of the upmix signal representation). The rendering matrix is composed of the relative rendering coefficients RCs (or object gains) specified for each transmitted audio object and upmix setup loudspeaker. These object gains determine the spatial position of all separated/rendered objects. Effectively, the separation of the object signals is rarely executed (or even never executed) since the separation and the mixing is performed in a single combined processing step, which results in an enormous reduction of computational complexity. The single combined processing step may, for example, be performed using transcoding coefficients, which describe the combination of the object separation and mixing of the separated objects.

It has been found that this scheme is tremendously efficient, both in terms of transmission bitrate (it is only required to transmit one or two downmix channels plus some side information instead of a number of individual object audio signals) and computational complexity (the processing complexity relates mainly to the number of output channels rather than the number of audio objects).

The SAOC decoder transforms (on a parametric level) the object gains and other side information directly into the transcoding coefficients (TCs) which are applied to the downmix signal to create the corresponding signals for the rendered output audio scene (or a preprocessed downmix signal for a further decoding operation, i.e. typically multi-channel MPEG Surround rendering).

It has been found that the subjectively perceived audio quality of the rendered output scene can be improved by application of distortion control measures or DCMs, as described in non-pre-published <CIT>. This improvement can be achieved for the price of accepting a moderate dynamic modification of the target rendering settings. The modification of the rendering information has time and frequency variant nature which under specific circumstances may result in unnatural sound colorations and temporal fluctuation artifacts.

In an alternative to the distortion control measures (DCMs) described in reference [<NUM>], embodiments according to the present invention use a number of parameter limiting schemes which focus on the reduction of audio artifacts (sound colorations, temporal fluctuations, etc.) and at the same time preserving a natural sound quality.

The proposed parameter limiting scheme concepts described herein do not adjust rendering coefficients (RCs) based on a distortion measure calculated using sophisticated algorithms based on psychoacoustic models. Instead, the proposed parameter limiting scheme concepts show a low computational and structural complexity and are therefore attractive for integration into SAOC technology. Nevertheless, they can also be advantageously combined with the schemes described in reference [<NUM>] in order to achieve better overall output quality by complementing each other.

Within the overall SAOC system, the parameter limiting schemes can be incorporated into the SAOC decoder processing chain in two ways. For example, that parameter limiting scheme can be placed at the front-end for indirect (external) modification of the SAOC output by controlling the rendering coefficients (RCs) R , which is shown as alternative (a) in <FIG>. According to an alternative which is not covered by the claimed invention, the inherent transcoding coefficients (TCs) T are directly (internally) modified at the back-end of the SAOC decoder, before the coefficients are applied to the downmix signal to yield the output upmix channel signals, which is shown as the alternative (b) of <FIG>.

In the following, the concept of indirect control will be discussed in more detail.

The underlying hypothesis of the indirect control method considers a relationship between distortion level and deviations of the RCs from their object-averaged value. This is based on the observation that the more specific attenuation/boosting is applied by the RCs to a particular object with respect to the other objects, the more aggressive modification of the transmitted downmix signal is to be performed by the SAOC decoder/transcoder. In other words: the higher the deviation of the "object gain" values are relative to each other, the higher the chance for unacceptable distortion to occur (assuming identical downmix coefficients). It has been found that this can be tested by examining the deviation of the RCs from the average of the RCs across all objects (e.g. mean rendering value).

Without loss of generality, the subsequent description is based on the configuration considering a mono downmix with unity downmix gains for all objects. For the case of nontrivial downmixes (with different and/or dynamic object gains) the algorithm can be appropriately modified. In addition, the RCs are assumed to be frequency invariant to simplify the notation.

Based on the user specified rendering scenario represented by the coefficients R(i) with object index i, the PLS prevents extreme rendering values by producing modified RC values R̃(i) that are actually used by the SAOC rendering engine. They can be derived as the following function <MAT> where A is a PLS control parameter (i.e. threshold value). The PLS control parameter may be considered as a tolerance parameter.

The deviation Rd (i) of rendering coefficient R(i) from an averaged rendering value R (e.g. the arithmetic mean) can be obtained as <MAT> where <MAT>.

Accordingly, Rd(i) is a ratio between a rendering coefficient R(i) and an averaged rendering value R. The averaged rendering value R is an average value, averaged over the audio objects having audio object indices i, of the rendering coefficients R(i).

The limited deviation R̃d(i) is restricted to a certain tolerance A range as <MAT> <MAT>.

Note that this corresponds to an RC limiting operation which is carried out relative to a reference value, for example R which is computed dynamically from the input RCs rather than a specific pre-defined value.

For the described PLS approach the optimal solution can be formulated as a minimization problem for which the difference between given RC R(i) and modified (limited) R̃(i) value is minimized <MAT>.

In the following, some algorithmic solutions for providing the adjusted rendering coefficients R̃(i) will be described, wherein the adjusted rendering coefficients R̃(i) can be considered as adjusted parameters.

The following two algorithmic solutions are based on the deviation of those rendering values which lie outside the tolerance range, i.e. <MAT>.

A simple and fast one-step solution can be employed to limit all rendering values outside the tolerance range by <MAT> <MAT>.

In contrast, the rendering values inside the tolerance range may be left unaffected, such that <MAT> for such rendering values R̃(i).

Another straightforward method can be employed in which the out-of-range rendering values with associated deviations Rd,out(i) are limited gradually. In each iteration of this algorithm, the maximal rendering deviation Rd,max is defined as <MAT> <MAT>.

The corresponding rendering coefficient is restricted such that <MAT>.

This processing can be performed until all values are inside the tolerance region or with a pre-determined number of iterations.

Accordingly, in each iteration, a rendering coefficient R(imax) is selected for which the deviation Rd,out (imax) (for example, from the average value R) takes the maximum value Rd,max. In other words, the rendering coefficient R(imax) is selected, which comprises a maximum deviation (in terms of the deviation value Rd,out) from the average R over the rendering coefficients in the respective iteration. In addition, the selected rendering coefficient R(imax) is brought closer to the average over the rendering coefficients using the above mentioned linear combination of R(i) and R (which may be applied selectively for i = imax). In each step of the iterative procedure, a new selection of the rendering coefficient having the maximum deviation from the average value may be performed, such that different rendering coefficients may be modified in different steps of the iterative algorithm. In other words, imax is typically updated in every iteration. Also, the average value may optionally be recomputed for every step of the iterative algorithm, considering a previously modified rendering coefficient.

The underlying hypothesis of the direct control method considers a relationship between distortion level and deviations of the TCs from their time-averaged value. This is based on the observation that the more specific attenuation/boosting is applied to a particular object with respect to the other objects, the more aggressive modification of the transmitted downmix signal by the TCs is to be performed by the SAOC decoder/transcoder. In other words: if the value of a TC is unusually large, it can be concluded that the SAOC algorithm attempts to modify an object signal with small power into an output dominated by other object signal(s) with a large power by applying a strong boost. Conversely, if a TC is unusually small, it can be concluded that the SAOC algorithm attempts to modify an object signal with large power into an output dominated by other object signal(s) with a small power by applying a strong attenuation. In both cases, there is a high risk of producing an unacceptably low signal quality at the SAOC output. Thus, the central idea is to prevent large deviations of TCs from an average value.

This PLS can be considered as time and frequency variant, since it includes all dependencies on the SAOC signal parameters (e.g. OLD, IOC) and heuristic elements of the transcoding/decoding process.

Without loss of generality, the subsequent description is based on the configuration considering a mono upmix.

Based on the SAOC output TC T (k) with frequency index k, the PLS prevents extreme values of the TCs by replacing them (e.g., transcoding coefficients outside of a tolerance interval) with modified TC values which are then used by the actual SAOC rendering process. The modified TC values T̃(k) can be derived with the following function <MAT> where A is a PLS control parameter (i.e. threshold value). The PLS control parameter may be considered as a tolerance parameter.

Since the TCs are time-variant, a recursive low pass filter is applied to calculate the mean <MAT>.

The mean T is considered as an average value, wherein a weighting of the individual transcoding values is introduced by the application of the recursive low pass filtering.

Here, n represents the time index of TCs and µ ∈ (<NUM>,<NUM>] is the averaging parameter. The tolerance range for the modified TC value T̃(k) is defined as <MAT>.

Note that this corresponds to a TC limiting operation which is carried out relative to a reference value which is computed dynamically from the TCs rather than a specific pre-defined value.

For the described PLS approach the optimal solution can be formulated as minimization problem for which the difference between given TC T(k) and modified (limited) TC T̃(k) value is minimized <MAT>.

In the following, a possible solution algorithm for this problem will be described.

The modified TC value T̃(k) can be obtained as <MAT> <MAT>.

The above discussed parameter limiting scheme for transcoding coefficients can be applied to different transcoding coefficients which are used, for example, in the SAOC decoders and transcoders discussed above.

For example, the parameter limiting scheme for transcoding coefficients can be applied to limit parameters of the mix matrix G, which is used in the signal processor <NUM> of the apparatus <NUM>. In this case, a mix matrix element at a given matrix position of the matrix G may take the place of a transcoding coefficient T(k), wherein k is a frequency index. A corresponding mix matrix element of the mix matrix G' may correspond to an adjusted transcoding coefficient T̃(k). The transcoding parameter limiting scheme may be applied, for example, individually to the different matrix positions of the mix matrix. For example, if the mix matrix G comprises mix matrix elements g<NUM>, g<NUM>, g<NUM> and g<NUM>, and the adjusted mix matrix G' comprises corresponding matrix elements g<NUM>', g<NUM>', g<NUM>' and g<NUM>', the adjusted mix matrix element g<NUM>'(n<NUM>) may be derived from a sequence g<NUM>(<NUM>) to g<NUM>(n<NUM>). Equivalent derivations may be used for the other mix matrix elements g<NUM>', g<NUM>' and g<NUM>' of the adjusted mix matrix G'.

The table of <FIG> provides a list of transcoding coefficients which can be modified, for example, limited, by the proposed parameter limiting schemes for all SAOC modes of operation. The table of <FIG> shows, in a first column <NUM>, different SAOC modes. The table of <FIG> further shows, in a second column <NUM>, which parameters can be modified (for example, limited) by the proposed parameter limiting scheme. A third column <NUM> shows a reference to the corresponding subclauses of the MPEG SAOC FCD document of reference [<NUM>]. To summarize, the table of <FIG> shows a list of transcoding coefficients which can be modified (for example, limited) by the proposed parameter limiting schemes for all SAOC modes of operation with references to corresponding subclauses of the MPEG SAOC FCD document [<NUM>].

There exists a generalized formulation for the above-discussed PLS. This formulation can be expressed in the form of the following minimization problem for the general parameter variable X̃i as <MAT>.

Here, the value of Xi is initially given and the "reference" value Xi can be estimated as a function of the modified X̃i variable as Xi = F(X̃i).

In the above, the parameter variable Xi may, for example, be identical to R(i) or T(i). Similarly, the adjusted parameter variable X̃i may be identical to the adjusted rendering coefficient R̃(i) or the adjusted transcoding coefficient T̃(i). The variables X, , X̃i may also, for example, be equivalent to mix matrix elements gmn(i) and gmn'(i).

In the following, two solution algorithms will be discussed.

Generally, the analytical approaches for obtaining the exact solution of such minimization problems are computationally demanding. Nevertheless, there exist simple and fast alternative ways providing suboptimal results which are still suitable for the PLS purposes. Two such simple approaches are described here.

The one-step solution based on assumption that Xi ≈ F(X̃i)
limits all values outside the tolerance range to lie inside it by <MAT> <MAT>.

Values which lie inside the tolerance range (which may be considered as a tolerance interval) may, for example, be left unchanged.

The iterative solution modifies in each step one selected out-of-range value Xi* to X̃i* <MAT>.

For instance, the processing index i * can be chosen using the condition: <MAT> or <MAT>.

The number of iterations can be set to a certain value or implicitly derived from the algorithm.

One should note that all these methods can be applied for limiting RCs and TCs as described above.

There exists a generalized linear formulation for the above-discussed PLS. In the previous section the deviation of the general parameter X, is described as a ratio <MAT>. In contrast, it can also be defined as ∥Xi - Xi∥ leading to the following minimization problem for the general parameter variable X̃i as <MAT>.

In the following, two solution algorithms for this problem will be described.

Generally, the analytical approaches for obtaining the exact solution of such minimization problems are generally computationally demanding. Nevertheless, there exist simple and fast alternative ways providing suboptimal results which are still suitable for the PLS purposes. Two such simple approaches are described here:.

The one-step solution based on assumption that Xi ≈ F(Xi) limits all values outside the tolerance range to lie inside it by <MAT>.

The iterative solution modifies in each step one selected value Xi* to Xi* if Xi* is outside a tolerance range: <MAT> <MAT>.

For instance, the processing index i * can be chosen using the condition: ∥Xi* - Xi*∥≥∥Xi - Xi∥ and the modification step size value as S = λ∥Xi* - Xi*∥ with λ∈(<NUM>,<NUM>). The number of iterations can be set to a certain value or implicitly derived from the algorithm.

This algorithm provides a flexible way of using the tolerance range, i.e. it is dynamically changing (depending on Xi*).

One should note that all these methods can be applied for limiting RCs and TCs as described above.

Alternatively, the following algorithm can be used: <MAT> then <MAT> and <MAT> then <MAT>.

This version of the algorithm uses a fix (static) tolerance range ΛX-, A x +.

One should note that all these methods can be applied for limiting rendering coefficients and transcoding coefficients, as described above.

The single TC PLS (e.g. direct control) of a mono downmixlmono upmix scenario extends to a TC matrix considering any combination of downmix/upmix channels. Consequently, the direct control can be applied to each TC individually. The multichannel upmix scenario for the RC PLS (e.g. indirect control) can be realized, for instance, in a simple multiple-mono approach where all individual rendering coefficients are handled independently.

The subjective listening test has been conducted to assess the perceptual performance of the proposed distortion control measure (DCM) concepts and compare it to the regular SAOC reference model (SAOC RM) decoding processing.

The test design includes the cases of individual application of the direct and indirect control approaches of the proposed parameter limiting scheme as well as their combination. The output signal of the regular (unprocessed by the parameter limiting scheme PLS) SAOC decoder is included in the test to demonstrate the baseline performance of the SAOC. In addition, the case of trivial rendering, which corresponds to the downmix signal, is used in the listening test for comparison purposes.

The table of <FIG> describes listening test conditions.

The four items representing typical and most critical artifact types for the extreme rendering conditions have been chosen for the current listening test from the call-for-proposals (CfP) listening test material.

The table of <FIG> describes audio items of the listening test.

The rendering object gains according to the table of <FIG> have been applied for the considered upmix scenarios.

Since the proposed PLS operates using the regular SAOC bitstreams and downmixes (no any PLS related activity on SAOC encoder side is needed) and does not relay on residual information, no core coder has been applied to the corresponding SAOC downmix signals.

For all test items and considered rendering conditions the global settings for the PLS are taken as <MAT>.

The subjective listening tests were conducted in an acoustically isolated listening room that is designed to permit high-quality listening. The playback was done using headphones (STAX SR Lambda Pro with Lake-People D/A-Converter and STAX SRM-Monitor).

The test method followed the procedure used in the spatial audio verification tests, based on the "Multiple Stimulus with Hidden Reference and Anchors" (MUSHRA) method for the subjective assessment of intermediate quality audio [<NUM>]. The test method has been accordantly modified in order to assess the perceptual performance of the proposed DCM concepts. In accordance with the adopted test methodology, the listeners were instructed to compare all test conditions against each other according to the following listening test instructions:
For each audio item please:.

• first read the description of the desired sound mixes that you as a system user would like to achieve:.

• then grade the signals using one common grade to describe both.

A total of <NUM> listeners participated in each of the performed tests. All subjects can be considered as experienced listeners. The test conditions were randomized automatically for each test item and for each listener. The subjective responses were recorded by a computer-based MUSHRA program on a scale ranging from <NUM> to <NUM>. An instantaneous switching between the items under test was allowed.

A short overview in terms of the diagrams demonstrating the obtained listening test results can be found in the appendix. These plots show the average MUSHRA grading per item over all listeners and the statistical mean value over all evaluated items together with the associated <NUM>% confidence intervals.

The following observations can be made based upon the results of the conducted listening tests: For all conducted listening tests the obtained MUSHRA scores prove that the proposed PLS functionality provides better performance in comparison with the regular SAOC RM system in sense of overall statistical mean values. One should note that the quality of all items produced by the regular SAOC decoder (showing strong audio artifacts for the considered extreme rendering conditions) is graded just slightly higher in comparison to the quality of downmix-identical rendering settings which does not fulfill the desired rendering scenario at all. Hence, it can be concluded that the proposed PLS lead to considerable improvement of subjective signal quality for all considered listening test scenarios. It can be also concluded that the most promising limiting system consists of a combination of both RC and TC PLS.

Details regarding the listening test results can be seen in the graphic representation of <FIG>.

The encoded audio signal can be stored on a digital storage medium or can be transmitted on a transmission medium such as a wireless transmission medium or a wired transmission medium such as the Internet.

The implementation can be performed using a digital storage medium, for example a floppy disk, a DVD, a Blue-Ray, a CD, a ROM, a PROM, an EPROM, an EEPROM or a FLASH memory, having electronically readable control signals stored thereon, which cooperate (or are capable of cooperating) with a programmable computer system such that the respective method is performed.

The above described embodiments are merely illustrative for the principles of the present invention. It is understood that modifications and variations of the arrangements and the details described herein will be apparent to others skilled in the art. It is the intent, therefore, to be limited only by the scope of the impending patent claims and not by the specific details presented by way of description and explanation of the embodiments herein.

Embodiments according to the invention create parameter limiting schemes for distortion control in audio decoders. Some embodiments according to the invention are focused on spatial audio object coding (SAOC), which provides means for a user interface for a selection of the desired playback setup (for example, mono, stereo, <NUM>, etc.) and interactive real-time modification of the desired output rendering scene by controlling the rendering matrix according to a personal preference or other criteria. However, it is a straightforward task to adapt the proposed method for parametric techniques in general.

Due to the downmix/separation/mix-based parametric approach, the subjective quality of the rendered audio output depends on the rendering parameter settings. The freedom of selecting rendering settings of the users choice entails the risk of the user selecting inappropriate object rendering options, such as extreme gain manipulations of an object within the overall sound scene.

For a commercial product it is by all means unacceptable to produce bad sound quality and/or audio artifacts for any settings on the user interface. In order to control excessive deterioration of the produced SAOC audio output, several computational measures have been described which are based on the idea of computing a measure of perceptual quality of the rendered scene, and depending on this measure (and other information), modify the actually applied rendering coefficients (see, for example, reference [<NUM>]).

The present invention creates alternative ideas for safeguarding the subjective sound quality of the rendered SAOC scene.

These ideas can thus be implemented in a structurally simple and extremely efficient way within the SAOC decoder/transcoder framework. Since the proposed distortion control mechanisms (DCMs) aim at limiting parameters inherent to the SAOC decoder, namely, the rendering coefficients (RCs) and the transcoding coefficients (TCs), they are called parameter limiting schemes (PLS) throughout the present description.

Claim 1:
An apparatus (<NUM>; <NUM>; <NUM>; <NUM>; <NUM>) for providing one or more adjusted parameters (<NUM>; <NUM>; <NUM>; R̃; T̃) for a provision of an upmix signal representation (<NUM>; 430a-<NUM>) on the basis of a downmix signal representation (<NUM>; <NUM>) and a parametric side information (<NUM>; <NUM>) associated with the downmix signal representation, the apparatus comprising:
a parameter adjuster configured to receive one or more parameters (<NUM>; <NUM>; <NUM>) and to provide, on the basis thereof, one or more adjusted parameters (<NUM>; <NUM>; <NUM>), wherein the parameter adjuster is configured to provide the one or more adjusted parameters in dependence on an average value (<NUM>; (R ; T) of a plurality of parameter values (<NUM>; <NUM>; <NUM>; R; T), such that a distortion of the upmix signal representation caused by the use of non-optimal parameters for the provision of the upmix signal representation is reduced at least for one or more parameters deviating from optimal parameters by more than a predetermined deviation;
characterised in that the apparatus is configured to receive one or more rendering coefficients (<NUM>; R) describing desired contributions of audio objects to one or more channels of the upmix signal representation (<NUM>; 430a-<NUM>), and wherein the apparatus is configured to provide one or more adjusted rendering coefficients (<NUM>; R̃) as the adjusted parameters.