Method for rendering multi-channel audio signals for L1 channels to a different number L2 of loudspeaker channels and apparatus for rendering multi-channel audio signals for L1 channels to a different number L2 of loudspeaker channels

Multi-channel audio content is mixed for a particular loudspeaker setup. However, a consumer's audio setup is very likely to use a different placement of speakers. The present invention provides a method of rendering multi-channel audio that assures replay of the spatial signal components with equal loudness of the signal. A method for obtaining an energy preserving mixing matrix (G) for mixing L1 input audio channels to L2 output channels comprises steps of obtaining (s711) a first mixing matrix G, performing (s712) a singular value decomposition on the first mixing matrix Ĝ to obtain a singularity matrix S, processing (s713) the singularity matrix S to obtain a processed singularity matrix Ŝ, determining (s715) a scaling factor a, and calculating (s716) an improved mixing matrix G according to G=a U Ŝ VT. The perceived sound, loudness, timbre and spatial impression of multi-channel audio replayed on an arbitrary loudspeaker setup practically equals that of the original speaker setup.

This application claims the benefit, under 35 U.S.C. §365 of International Application PCT/EP2014/065517, filed Jul. 18, 2014, which was published in accordance with PCT Article 21(2) on Jan. 22, 2015 in English and which claims the benefit of European patent application No. 13306042.6, filed Jul. 19, 2013.

FIELD OF THE INVENTION

This invention relates to a method for rendering multi-channel audio signals, and an apparatus for rendering multi-channel audio signals. In particular, the invention relates to a method and apparatus for rendering multi-channel audio signals for L1channels to a different number L2of loudspeaker channels.

BACKGROUND

New 3D channel based Audio formats provide audio mixes for loudspeaker channels that not only surround the listening position, but also include channels positioned above (height) and below in respect to the listening position (sweet spot). The mixes are suited for a special positioning of these speakers. Common formats are 22.2 (i.e. 22 channels) or 11.1 (i.e. 11 channels).

FIG. 1shows two examples of ideal speaker positions in different speaker setups: a 22-channel speaker setup (left) and a 12-channel speaker setup (right). Every node shows the virtual position of a loudspeaker. Real speaker positions that differ in distance to the sweet spot are mapped to the virtual positions by gain and delay compensation.

A renderer for channel based audio receives L1digital audio signals w1and processes the output to L2output signals w2.FIG. 2shows, in an embodiment, the integration of a renderer21into a reproduction chain. The renderer output signal w2is converted to an analog signal in a D/A converter22, amplified in an amplifier23and reproduced by loudspeakers24.

The renderer21uses the position information of the input speaker setup and the position information of the output loudspeaker24setup as input to initialize the chain of processing. This is shown inFIG. 3. Two main processing blocks are a Mixing & Filtering block31and a Delay & Gain Compensation block32.

The speaker position information can be given e.g. in Cartesian or spherical coordinates. The position for the output configuration R2may be entered manually, or derived via microphone measurements with special test signals, or by any other method. The positions of the input configuration R1can come with the content by table entry, like an indicator e.g. for 5-channel surround. Ideal standardized loudspeaker positions [9] are assumed. The positions might also be signaled directly using spherical angle positions. A constant radius is assumed for the input configuration. Let R2=[r21, r22, . . . , r2L2] with r2l=[r2l, θ2l, φ2l]T=[r2l, {circumflex over (Ω)}lT]Tbe the positions of the output configuration in spherical coordinates. Origin of the coordinate system is the sweet spot (i.e. listening position). r2lis the distance between the listening position and a speaker l, and θl, φlare the related spherical angles that indicate the spatial direction of the speaker l relative to the listening position.

Delay and Gain Compensation

The distances are used to derive delays and gainslthat are applied to the loudspeaker feeds by amplification/attenuation elements and a delay line with dlunit sample delay steps. First, the maximal distance between a speaker and the sweet spot is determined:
r2max=max([r21, . . . r2L2]).

For each speaker feed the delay is calculated by:
dl=└(r2max−r2l)fs/c+0.5┘  (1)
with sampling fs, speed of sound c (c≅343 m/s at 20° celsius temperature) and └x+0.5┘ indicates rounding to next integer. The loudspeaker gainslare determined by

The task of the Delay and Gain Compensation building block32is to attenuate and delay speakers that are closer to the listener than other speakers, so that these closer speakers do not dominate the sound direction perceived. The speakers are thus arranged on a virtual sphere, as shown inFIG. 1. The Mix & Filter block31now can use virtual speaker positions {circumflex over (R)}2=[1,2, . . . ,L2] withl=[r2max, {circumflex over (Ω)}1T]Twith a constant speaker distance.

Mix & Filter

In an initialization phase, the speaker positions of the input and idealized output configurations R1, {circumflex over (R)}2are used to derive a L2×L1mixing matrix G. During the process of rendering, this mixing matrix is applied to the input signals to derive the speaker output signals. As shown inFIG. 4, two general approaches exist. In the first approach shown inFIG. 4a), the mixing matrix is independent from the audio frequency and the output is derived by:
W2=G W1,  (3)
where W1εL1×τ, W2εL2×τdenote the input and output signals of L1, L2audio channels and τ time samples in matrix notation. The most prominent method is Vector Base Amplitude Panning (VBAP) [1].

In the second approach, the mixing matrix becomes frequency dependent (G(f)), as shown inFIG. 4b). Then, a filter bank of sufficient resolution is needed, and a mixing matrix is applied to every frequency band sample according to eq. (3).

Examples for the latter approach are known [2],[3],[4]. For deriving the mixing matrix, the following approach is used: A virtual microphone array51as depicted inFIG. 5, is placed around the sweet spot. The microphone signals M1of sound received from the input configuration (the original directions, left-hand side) is compared to the microphone signals M2of sound received from the desired speaker configuration (right-hand side). Let1εM×τdenote M microphone signals receiving the sound radiated from the input configuration, and2εM×τbe M microphone signals of the sound from the output configuration. They can be derived by
1=HM,L1W1(4)
and
2=HM,L2W2(5)
with HM,L1εM×L1, HM,L2εM×L2being the complex transfer function of the ideal sound radiation in the free field, assuming spherical wave or plane wave radiation. The transfer functions are frequency dependent. Selecting a mid-frequency fmrelated to a filter bank, eq. (4) and eq. (5) can be equated using eq. (3). For every fmthe following equation needs to be solved to derive G(fm):
HM,L1W1=HM,L2G W1(6)

A solution that is independent of the input signals and that uses the pseudo inverse matrix of HM,L2can be derived as:
G=HM,L2+HM,L1.  (7)

Usually this produces non-satisfying results, and [2] and [5] present more sophisticated approached to solve eq. (6) for G.

Further, there is a completely different way of signal adaptive rendering, where the directional signals of the incoming audio content is extracted and rendered like audio objects. The residual signal is panned and de-correlated to the output speakers. This kind of audio rendering is much more expensive in terms of computational complexity, and often not free from artifacts. Signal adaptive rendering is not used and only mentioned here for completeness.

One problem is that a consumer's home setup is very likely to use a different placement of speakers due to real world constraints of a living room. Also the number of speakers may be different. The task of a renderer is thus to adapt the channel based audio signals to a new setup such that the perceived sound, loudness, timbre and spatial impression comes as close as possible to the original channel based audio as replayed on its original speaker setup, like e.g. in the mixing room.

SUMMARY OF THE INVENTION

The present invention provides a preferably computer-implemented method of rendering multi-channel audio signals that assures replay (i.e. reproduction) of the spatial signal components with correct loudness of the signal (i.e. equal to the original setup). Thus, a directional signal that is perceived in the original mix coming from a direction is also perceived equally loud when rendered to the new loudspeaker setup. In addition, filters are provided that equalize the input signals to reproduce a timbre as close as possible as it would be perceived when listening to the original setup.

In one aspect, the invention relates to a method for rendering L1channel-based input audio signals to L2loudspeaker channels, where L1is different from L2, as disclosed in claim1. In one embodiment, a step of mixing the delay and gain compensated input audio signal for L2audio channels uses a mixing matrix that is generated as disclosed in claim5. A corresponding apparatus according to the invention is disclosed in claim8and claim12, respectively.

In one aspect, the invention relates to a method for generating an energy preserving mixing matrix G for mixing input channel-based audio signals for L1audio channels to L2loudspeaker channels, as disclosed in claim7. A corresponding apparatus for generating an energy preserving mixing matrix G according to the invention is disclosed in claim14. In one aspect, the invention relates to a computer readable medium having stored thereon executable instructions to cause a computer to perform a method according to claim1, or a method according to claim7.

In one embodiment of the invention, a computer-implemented method for generating an energy preserving mixing matrix G for mixing input channel-based audio signals for L1audio channels to L2loudspeaker channels comprises computer-executed steps of obtaining a first mixing matrix Ĝ from virtual source directionsand target speaker directions, performing a singular value decomposition on the first mixing matrix Ĝ to obtain a singularity matrix S, processing the singularity matrix S to obtain a processed singularity matrix Ŝ withnon-zero diagonal elements, determining from the number of non-zero diagonal elements a scaling factor a according to

a=L1⁢(for⁢⁢L⁢⁢2≤L⁢⁢1)⁢⁢or⁢⁢a=L2⁢(for⁢⁢L⁢⁢2>L⁢⁢1),
and calculating a mixing matrix G by using the scaling factor according to G=a U Ŝ VT. As a result, the perceived sound, loudness, timbre and spatial impression of multi-channel audio replayed on an arbitrary loudspeaker setup is improved, and in particular comes as close as possible to the original channel based audio as if replayed on its original speaker setup.

Further objects, features and advantages of the invention will become apparent from a consideration of the following description and the appended claims when taken in connection with the accompanying drawings.

DETAILED DESCRIPTION OF THE INVENTION

FIG. 6a) shows a flow-chart of a method for rendering a first number L1of channel-based input audio signals to a different second number L2of loudspeaker channels according to one embodiment of the invention. The method for rendering L1channel-based input audio signals w11to L2loudspeaker channels, where the number L1of channel-based input audio signals is different from the number L2of loudspeaker channels, comprises steps of determining s60a mix type of the L1input audio signals, performing a first delay and gain compensation s61on the L1input audio signals according to the determined mix type, wherein a delay and gain compensated input audio signal with the first number L1of channels and with a defined mix type is obtained, mixing s624the delay and gain compensated input audio signal for the second number L2of audio channels, wherein a remixed audio signal for the second number L2of audio channels is obtained, clipping s63the remixed audio signal, wherein a clipped remixed audio signal for the second number L2of audio channels is obtained, and performing a second delay and gain compensation s64on the clipped remixed audio signal for the second number L2of audio channels, wherein the second number L2of loudspeaker channels w22are obtained.

Possible mix types include at least one of spherical, cylindrical and rectangular (or, more general, cubic). In one embodiment, the method comprises a further step of filtering s622the delay and gain compensated input audio signal q71having the first number L1of channels in an equalization filter (or equalizer filter), wherein a filtered delay and gain compensated input audio signal is obtained. While the equalization filtering is in principle independent from the usage of, and can be used without, an energy preserving mixing matrix, it is particularly advantageous to use both in combination.

FIG. 6b) shows a flow-chart of a method for generating an energy preserving mixing matrix G according to one embodiment of the invention. The method s710for obtaining an energy preserving mixing matrix G for mixing input channel-based audio signals for a first number L1of audio channels to a second number L2of loudspeaker channels comprises steps of obtaining s711a first mixing matrix Ĝ from virtual source positions/directionsand target speaker positions/directionswherein a panning method is used, performing s712a singular value decomposition on the first mixing matrix Ĝ according to Ĝ=U S VT, wherein UεL2×L2and VεL1×L1are orthogonal matrices and SεL1×L2is a singularity matrix and has s first diagonal elements being the singular values of G in descending order and all other elements of S are zero, processing s713the singularity matrix S, wherein a quantized singularity matrix Ŝ is obtained with diagonal elements that are above a threshold set to one and diagonal elements that are below a threshold set to zero, determining s714a numbermof diagonal elements that are set to one in the quantized singularity matrix Ŝ, determining s715a scaling factor a according to

a=L1⁢(for⁢⁢L⁢⁢2≤L⁢⁢1)⁢⁢or⁢⁢a=L2⁢(for⁢⁢L⁢⁢2>L⁢⁢1),
and calculating s716a mixing matrix G according to G=a U Ŝ VT. The steps of any of the above-mentioned methods can be performed by one or more processing elements, such as microprocessors, threads of a GPU etc.

FIG. 7shows a rendering architecture70according to one embodiment of the invention. In the rendering architecture according to the embodiment shown inFIG. 7a), an additional “Gain and Delay Compensation” block71is used for preprocessing different input setups, such as spherical, cylindrical or rectangular input setups. Further, a modified “Mix & Filter” block72that is capable of preserving the original loudness is used. In one embodiment, the “Mix & Filter” block72comprises an equalization filter722. The “Mix & Filter” block72is described in more detail with respect toFIG. 7b) andFIG. 8. A clipping prevention block73prevents signal overflow, which may occur due to the modified mixing matrix. A determining unit75determines a mix type of the input audio signals.

FIG. 7b) shows the Mix&Filter block72incorporating an equalization filter722and a mixer unit724.FIG. 8shows the structure of the equalization filter722in the Mix&Filter block.

The equalization filter is in principle a filter bank with L1filters EF1, . . . , EFL1, one for each input channel. The design and characteristics of the filters are described below. All blocks mentioned may be implemented by one or more processors or processing elements that may be controlled by software instructions.

The renderer according to the invention solves at least one of the following problems:

First, new 3D audio channel based content can be mixed for at least one of spherical, rectangular or cylindrical speaker setups. The setup information needs to be transmitted alongside e.g. with an index for a table entry signaling the input configuration (which assumes a constant speaker radius) to be able to calculate the real input speaker positions. In an alternative embodiment, full input speaker position coordinates can be transmitted along with the content as metadata. To use mixing matrices independent of the mixing type, a gain and delay compensation is provided for the input configuration.

Second, the invention provides an energy preserving mixing matrix G. Conventionally, the mixing matrix is not energy preserving. Energy preservation assures that the content has the same loudness after rendering, compared to the content loudness in the mixing room when using the same calibration of a replay system [6],[7],[8]. This also assures that e.g. 22-channel input or 10-channel input with equal ‘Loudness, K-weighted, relative to Full Scale’ (LKFS) content loudness appears equally loud after rendering.

One advantage of the invention is that it allows generating energy (and loudness) preserving, frequency independent mixing matrices. It is noted that the same principle can also be used for frequency dependent mixing matrices, which however are not so desirable. A frequency independent mixing matrix is beneficial in terms of computational complexity, but often a drawback can be a in change in timbre after remix. In one embodiment, simple filters are applied to each input loudspeaker channel before mixing, in order to avoid this timbre mismatching after mixing. This is the equalization filter722. A method for designing such filters is disclosed below.

Energy preserving rendering has a drawback that signal overload is possible for peak audio signal components. In one embodiment of the present invention, an additional clipping prevention block73prevents such overload. In a simple realization, this can be a saturation, while in more sophisticated realizations this block is a dynamics processor for peak audio.

In the following, details about the mix type determining unit75and the Input Gain and Delay compensation71are described. If the input configuration is signaled by a table entry plus mix room information, like e.g. rectangular, cylindrical or spherical, the configuration coordinates are read from special prepared tables (e.g. RAM) as spherical coordinates. If the coordinates are transmitted directly, they are converted to spherical coordinates. A determining unit75determines a mix type of the input audio signals. Let R1=[r11, r12, . . . , r1L1] with r1l=[r1l, θ1l, φ1l]T=[r1l, ΩlT]Tbeing the positions of this input configuration.

In a first step the maximum radius is detected: r1max=max([r11, . . . r1L2]. Because only relative differences are of interest for this building block, the radii are r1lscaled by r2maxthat is available from the gain and delay compensation initialization of the output configuration:

The number of delay tabs {hacek over (d)}land the gain valueslfor every speaker are calculated as follows withmax=r2max:
{hacek over (d)}l=└(r2max−l)fs/c+0.5┘  (9)
with sampling rate fs, speed of sound c (c≅343 m/s at 20° celsius temperature) and [x+0.5] indicates rounding to next integer.

The loudspeaker gainsare determined by

The Mix & Filter block now can use virtual speaker positions {circumflex over (R)}1=[1,2, . . . ,L1] withl=[max, ΩlT]Twith a constant speaker distance.

In the following, the Mixing Matrix design is explained.

First, the energy of the speaker signals and perceived loudness are discussed.

FIG. 7a) shows a block diagram defining the descriptive variables. L1loudspeakers signals have to be processed to L2signals (usually, L2≦L1). Replay of the loudspeaker feed signals W2(shown as W22inFIG. 7) should ideally be perceived with the same loudness as if listening to a replay in the mixing room, with the optimal speaker setup. Let W1be a matrix of L1loudspeaker channels (rows) and T samples (columns).

The energy of the signal W1, of the r-time sample block is defined as follows:
Ew1=||W1||fro2=Σi=1τΣl=1L1W1l,i2=Σi=1τw1tTw1t(11)

Here Wl,iare the matrix elements of W1, l denotes the speaker index, i denotes the sample index, || ||frodenotes the Frobenius matrix norm, w1tis the tthcolumn vector of W1and [ ]Tdenotes vector or matrix transposition.

This energy Ewgives a fair estimate of the loudness measure of a channel based audio as defined in [6],[7],[8], where the K-filter suppresses frequencies lower than 200 Hz.

Mixing of the signals W1provides signals W2. The signal energy after mixing becomes:
Ew2=||W2||fro2=Σi=1τΣl=1L2W2l,i2(12)
where L2is the new number of loudspeakers, with L2≦L1.

The process of rendering is assumed to be performed by a mixing matrix G, signals W2are derived from W1as follows:
W2=G W1(13)

In one embodiment, loudness preservation is then obtained as follows.

The loudness of the original signal mix is preserved in the new rendered signal if:
E1=E2(15)
From eq. (14) it becomes apparent that mixing matrix M needs to be orthogonal and
GTG=I  (16)
with I being the L1×L1unit matrix.

An optimal rendering matrix (also called mixing matrix or decode matrix) can be obtained as follows, according to one embodiment of the invention.

Step 1: A conventional mixing matrix Ĝ is derived by using panning methods. A single loudspeaker l1from the set of original loudspeakers is viewed as a sound source to be reproduced by L2speakers of the new speaker setup. Preferred panning methods are VBAP [1] or robust panning [2] for a constant frequency (i.e. a known technology can be used for this step). To determine the mixing matrix Ĝ, the modified speaker positions {circumflex over (R)}2, {circumflex over (R)}1are used, {circumflex over (R)}2for the output configuration and {circumflex over (R)}1for the virtual source directions.

Step 2: Using compact singular value decomposition, the mixing matrix is expressed as a product of three matrices:
Ĝ=U S VT(17)

UεL2×2and VεL1×L1are orthogonal matrices and SεL1×L2has s first diagonal elements (the singular values in descending order), with s≦L2. The other matrix elements are zeros.

Note that this holds for the case of L2≦L1, (remix L2=L1, downmix L2<L1). For the case of upmix (L2>L1), L2needs to be replaced by L1in this section.

Step3: A new matrix Ŝ is formed from S where the diagonal elements are replaced by a value of one, but very low valued singular values<<smaxare replaced by zeros. A threshold in the range of −10 dB . . . −30 dB or less is usually selected (e.g. −20 dB is a typical value). The threshold becomes apparent from actual numbers in realistic examples, since there will occur two groups of diagonal elements: elements with larger value and elements with considerably smaller value. The threshold is for distinguishing among these two groups.

For most speaker settings, the number of non-zero diagonal elementsmism=L2, but for some settings it becomes lower and thenm<L2. This means that L2−mspeakers will not be used to replay content; there is simply no audio information for them, and they remain silent.

Letmdenote the last singular value to be replaced by one. Then the mixing matrix G is determined by:
G=a U Ŝ VT(18)
with the scaling factor

The scaling factor is derived from: GTG=a2VŜ2VT=a2VVT, where VVThasmEigenvalues equal to one. That means that |VVT|fro=√m. Thus, simply down mixing the L1signals tomsignals will reduce the energy, unlessm=L1(in other words: when the number of output speakers matches the number of input speakers). With |IL1|fro=√L1, a scaling factor

a=L1
compensates the loss of energy during down-mixing.

As an example, processing of a singularity matrix is described in the following. E.g., an initial (conventional) mixing matrix for L loudspeakers is decomposed using compact singular value decomposition according to eq. (17): Ĝ=U S VT. The singularity matrix S is square (with L×L elements, L=min{L1,L2} for compact singular value decomposition) and is a diagonal matrix of the form

with s1≧s2≧ . . . ≧sL(i.e., s1=smax) Then the singularity matrix is processed by setting the coefficients s1,s2, . . . ,sLto be either 1 or 0, depending whether each coefficient is above a threshold of e.g. 0.06*smax. This is similar to a relative quantization of the coefficients. The threshold factor is exemplary 0.06, but can be (when expressed in decibel) e.g. in the range of −10 dB or lower.

For a case with e.g. L=5 and e.g. only s1and s2being above the threshold and s3, s4and s5being below the threshold, the resulting processed (or “quantized”) singularity matrix Ŝ is

S^=[1000001000000000000000000].
Thus, the number of its non-zero diagonal coefficientsmis two.

In the following, the Equalization Filter722is described.

When mixing between different 3D setups, and especially when mixing from 3D setups to 2D setups, timbre may change. E.g. for 3D to 2D, a sound originally coming from above is now reproduced using only speakers on the horizontal plane. The task of the equalization filter is to minimize this timbre mismatch and maximize energy preservation. Individual filters Flare applied to each channel of the L1channels of the input configuration before applying the mixing matrix, as shown inFIG. 7b). The following shows the theoretical deviation and describes how the frequency response of the filters is derived. A model according toFIG. 7and eqs. (4) and (5) is used. Both equations are reprinted here for convenience:
1=HM,L1W1(20)
and
2=HM,L2W2(21)
with HM,L1εM×L1, HM,L2εM×L2being the complex transfer function of the ideal sound radiation in the free field assuming spherical wave or plane wave radiation. These matrices are functions of frequency, and they can be calculated using the position information {circumflex over (R)}2, {circumflex over (R)}1. We define W2={tilde over (G)}W1, where {tilde over (G)} is a function of frequency. Instead of equating eqs.(4) and (5), as mentioned in the background section, we will equate the energies. And since we want to equalize for the sound of the speaker directions of the input configuration, we can solve the considerations for each input speaker at a time (loop over L1).

The energy measured at the virtual microphones for the input setup, if only one speaker l is active, is given by
|1,l|fro2=|hM,lw1 l|fro2(22)
with hM,lrepresenting the lth column of HM,L1and w1 lone row of W1, i.e. the time signal of speaker l with τ samples. Rewriting the Frobenius norm analog to eq. (11), we can further evaluate eq. (22) to:
|1,l|fro2=Σi=1τw1 lTw1 lhM,lHhM,l=EwlhM,lHhM,l(23)
where ( )His conjugate complex transposed (Hermitian transposed) and Ewlis the energy of speaker signal l. The vector hM,lis composed out of complex exponentials (see eqs.(31), (32)) and the multiplication of an element with its conjugate complex equals one, thus hM,lHhM,l=L1:
|1,l|fro2=EwlL1(24)

The measures at the virtual microphones after mixing are given by2=HM,L2{tilde over (G)}W1. If only one speaker is active, we can rewrite to:
2,l=HM,L2{tilde over (g)}lw1 l(25)
with {tilde over (g)}lbeing the lth column of {tilde over (G)}. We define {tilde over (G)} to be decomposable into a frequency dependent part related to speaker l and mixing matrix G derived from eq. (24):
{tilde over (G)}(f)=diag(b(f))G(26)
with b as a frequency dependent vector of L1complex elements and (f) denoting frequency dependency, which is neglected in the following for simplicity. With this, eq. (25) becomes:
2,l=HM,L2b1g w1 l(27)
where g is the lthcolumn of G and blthe lthelement of b. Using the same considerations of the Frobenius norm as above, the energy at the virtual microphones becomes:
|2,l|fro2=Ewl(HM,L2blg)H(HM,L2blg)  (28)
which can be evaluated to:
|2,l|fro2=Ewlbl2gTHM,L2HHM,L2g  (29)

We can now equate the energies according to eq.(24) and eq.(29) respectively, and solve for blfor each frequency f:

The blof eq.(30) are frequency-dependent gain factors or scaling factors, and can be used as coefficients of the equalization filter722for each frequency band, since bland HM,L2HHM,L2are frequency-dependent.

In the following, practical filter design for the equalization filter722is described. Virtual microphone array radius and transfer function are taken into account as follows.

To match the perceptual timbre effects of humans best, a microphone radius rMof 0.09 m is selected (the mean diameter of a human head is commonly assumed to be about 0.18 m). M>>L1virtual microphones are placed on a sphere or radius rMaround the origin (sweet spot, listening position). Suitable positions are known [11]. One additional virtual microphone is added at the origin of the coordinate system.

The transfer matrices HM,L2εM×L2are designed using a plane wave or spherical wave model. For the latter, the amplitude attenuation effects can be neglected due to the gain and delay compensation stages. Let hm,lbe an abstract matrix element of the transfer matrices HM,L, for the free field transfer function from speaker l to microphone m (which also indicate column and row indices of the matrices). The plane wave transfer function is given by
hm,l=eikrmcos(γl,m)  (31)
with i the imaginary unit, rmthe radius of the microphone position (ether rMor zero for the origin position) and cos(γl,m)=cos θ1cos θm+sin θ1sin θmcos(φl−φm) the cosine of the spherical angles of the positions of speaker l and microphone m. The frequency dependency is given by

k=2⁢π⁢⁢fc,
with f the frequency and c the speed of sound. The spherical wave transfer function is given by:
hm,l=e−ikrl,m(32)
with rl,mthe distance speaker l to microphone m.

The frequency response BrespεL1×FNof the filter is calculated using a loop over FNdiscrete frequencies and a loop over all input configuration speakers L1:

Calculate G according to the above description (3-step procedure for design of optimal rendering matrices):

The filter responses can be derived from the frequency responses Bresp(l, f) using standard technologies. Typically, it is possible to derive a FIR filter design of order equal or less than 64, or IIR filter designs using cascaded bi-quads with even less computational complexity.FIGS. 9 and 10show design examples.

InFIG. 9, example frequency responses of filters for a remix of 5-channels ITU setup [9] (L,R,C,Ls,Rs) to +/−30° 2-channel stereo, and an exemplary resulting 2×5 mixing matrix G are shown. The mixing matrix was derived as described above, using [2] for 500 Hz. A plane wave model was used for the transfer functions. As shown, two of the filters (upper row, for two of the channels) have in principle low-pass (LP) characteristics, and three of the filters (lower rows, for the remaining three channels) have in principle high-pass (HP) characteristics. It is intended that the filters do not have ideal HP or LP characteristics, because together they form an equalization filter (or equalization filter bank). Generally, not all the filters have substantially same characteristics, so that at least one LP and at least one HP filter is employed for the different channels.

InFIG. 10, example responses of filters for a remix of 22 channels of the 22.2 NHK setup [10] to ITU 5-channel surround [9] are shown. InFIG. 10b), the three filters of the first row ofFIG. 10a) are exemplarily shown. Also a resulting 5×22 mixing matrix G is shown, as obtained by the present invention.

The present invention can be used to adjust audio channel based content with arbitrary defined L1loudspeaker positions to enable replay to L2real-world loudspeaker positions. In one aspect, the invention relates to a method of rendering channel based audio of L1channels to L2channels, wherein a loudness & energy preserving mixing matrix is used. The matrix is derived by singular value decomposition, as described above in the section about design of optimal rendering matrices. In one embodiment, the singular value decomposition is applied to a conventionally derived mixing matrix.

In one embodiment, the matrix is scaled according to eq.(19) or (19′) by a factor of

L1⁢(for⁢⁢L1≥L2),
or by a factor of

Conventional matrices can be derived by using various panning methods, e.g. VBAP or robust panning. Further, conventional matrices use idealized input and output speaker positions (spherical projection, see above). Therefore, in one aspect, the invention relates to a method of filtering the L1input channels before applying the mixing matrix. In one embodiment, input signals that use different speaker positions are mapped to a spherical projection in a Delay & Gain Compensation block71.

In one embodiment, equalization filters are derived from the frequency responses as described above.

In one embodiment, a device for rendering a first number L1of channels of channel-based audio signals (or content) to a second number L2of channels of channel-based audio signals (or content) is assembled out of at least the following building blocks/processing blocks:in put (and output) gain and delay compensation blocks71,74, having the purpose to map the input and output speaker positions to a virtual sphere. Such spherical structure is required for the above-described mixing matrix to be applicable;equalization filters722derived by the method described above for filtering the first number L1of channels after input gain and delay compensation;a mixer unit72for mixing the first number L1of input channels to the second number L2of output channels by applying the energy preserving mixing matrix724as derived by the method described above. The equalization filters722may be part of the mixer unit72, or may be a separate module;a signal overflow detection and clipping prevention block (or clipping unit)73to prevent signal overload to the signals of L2channels; andan output gain and delay correction block74(already mentioned above).

In one embodiment, a method for obtaining or generating an energy preserving mixing matrix G for mixing L1input audio channels to L2output channels comprises steps of obtaining s711a first mixing matrix Ĝ, performing s712a singular value decomposition on the first mixing matrix Ĝ to obtain a singularity matrix S, processing s713the singularity matrix S to obtain a processed singularity matrix Ŝ, determining s715a scaling factor α, and calculating s716an improved mixing matrix G according to G=a U Ŝ VT. One advantage of the improved mixing mode matrix G is that the perceived sound, loudness, timbre and spatial impression of multi-channel audio replayed on an arbitrary loudspeaker setup practically equals that of the original speaker setup. Thus, it is not required any more to locate loudspeakers strictly according to a predefined setup for enjoying a maximum sound quality and optimal perception of directional sound signals.

In one embodiment, an apparatus for rendering L1channel-based input audio signals to L2loudspeaker channels, where L1is different from L2, comprises at least one of each of a determining unit for determining a mix type of the L1input audio signals, wherein possible mix types include at least one of spherical, cylindrical and rectangular; a first delay and gain compensation unit for performing a first delay and gain compensation on the L1input audio signals according to the determined mix type, wherein a delay and gain compensated input audio signal with L1channels and with a defined mix type is obtained;

a mixer unit for mixing the delay and gain compensated input audio signal for L2audio channels, wherein a remixed audio signal for L2audio channels is obtained;

a clipping unit for clipping the remixed audio signal, wherein a clipped remixed audio signal for L2audio channels is obtained; and

a second delay and gain compensation unit for performing a second delay and gain compensation on the clipped remixed audio signal for L2audio channels, wherein L2loudspeaker channels are obtained.

Further, in one embodiment of the invention, an apparatus for obtaining an energy preserving mixing matrix G for mixing input channel-based audio signals for L1audio channels to L2loudspeaker channels comprises at least one processing element and memory for storing software instructions for implementing a first calculation module for obtaining a first mixing matrix Ĝ from virtual source directionsand target speaker directionswherein a panning method is used;

a singular value decomposition module for performing a singular value decomposition on the first mixing matrix Ĝ according to Ĝ=U S VT, wherein UεL2×L2and VεL1×L1are orthogonal matrices and SεL1×L2is a singularity matrix and has s first diagonal elements being the singular values of G in descending order and all other elements of S are zero;

a processing module processing the singularity matrix S, wherein a quantized singularity matrix Ŝ is obtained with diagonal elements that are above a threshold set to one and diagonal elements that are below a threshold set to zero;

a counting module for determining a numbermof diagonal elements that are set to one in the quantized singularity matrix Ŝ;

a second calculation module for determining a scaling factor a according to

a=L1⁢for⁢⁢(L⁢⁢2≤L⁢⁢1)⁢⁢or⁢⁢a=L2⁢for⁢⁢(L⁢⁢2>L⁢⁢1);
and

a third calculation module for calculating a mixing matrix G according to
G=a U Ŝ VT.

Advantageously, the invention is usable for content loudness level calibration. If the replay levels of a mixing facility and of presentation venues are setup in the manner as described, switching between items or programs is possible without further level adjustments. For channel based content, this is simply achieved if the content is tuned to a pleasant loudness level at the mixing site. The reference for such pleasant listening level can either be the loudness of the whole item itself or an anchor signal.

If the reference is the whole item itself, this is useful for ‘short form content’, if the content is stored as a file. Besides adjustment by listening, a measurement of the loudness in Loudness Units Full Scale (LUFS) according to EBU R128 [6] can be used to loudness adjust the content. Another name for LUFS is ‘Loudness, K-weighted, relative to Full Scale’ from ITU-R BS.1770 [7] (1 LUFS=1 LKFS). Unfortunately [6] only supports content for setups up to 5-channel surround. It has not been investigated yet if loudness measures of 22-channel files correlate with perceived loudness if all 22 channels are factored by equal channel weights of one.

If the above-mentioned reference is an anchor signal, such as in a dialog, the level is selected in relation to this signal. This is useful for ‘long form content’ such as film sound, live recordings and broadcasts. An additional requirement, extending the pleasant listening level, is intelligibility of the spoken word here. Again, besides an adjustment by listening, the content may be normalized related a loudness measure, such as defined in ATSC A/85 [8]. First parts of the content are identified as anchor parts. Then a measure as defined in [7] is computed or these signals and a gain factor to reach the target loudness is determined. The gain factor is used to scale the complete item. Unfortunately, again the maximum number of channels supported is restricted to five.

Out of artistic considerations, content should be adjusted by listening at the mixing studio. Loudness measures can be used as a support and to show that a specified loudness is not exceeded. The energy Ewaccording to eq.(11) gives a fair estimate of the perceived loudness of such an anchor signal for frequencies over 200 Hz. Because the K-filter suppresses frequencies lower than 200 Hz [5], Ewis approximately proportional to the loudness measure.

It is noted that when a “speaker” is mentioned herein, a loudspeaker is meant. Generally, a speaker or loudspeaker is a synonym for any sound emitting device. It is noted that usually where speaker directions are mentioned in the specification or the claims, also speaker positions can be equivalently used (and vice versa).

While there has been shown, described, and pointed out fundamental novel features of the present invention as applied to preferred embodiments thereof, it will be understood that various omissions and substitutions and changes in the apparatus and method described, in the form and details of the devices disclosed, and in their operation, may be made by those skilled in the art without departing from the spirit of the present invention. E.g., although in the above embodiments, the number L1of channels of the channel-based input audio signals is usually different from the number L2of loudspeaker channels, it is clear that the invention can also be applied in cases where both numbers are equal (so-called remix). This may be useful in several cases, e.g. if directional sound should be optimized for any irregular loudspeaker setup. Further, it is generally advantageous to use an energy preserving rendering matrix for rendering. It is expressly intended that all combinations of those elements that perform substantially the same function in substantially the same way to achieve the same results are within the scope of the invention.

Substitutions of elements from one described embodiment to another are also fully intended and contemplated. It will be understood that the present invention has been described purely by way of example, and modifications of detail can be made without departing from the scope of the invention.

Each feature disclosed in the description and (where appropriate) the claims and drawings may be provided independently or in any appropriate combination. Features may, where appropriate be implemented in hardware, software, or a combination of the two. Connections may, where applicable, be implemented as wireless connections or wired, not necessarily direct or dedicated, connections.

CITED REFERENCES