Patent ID: 12198705

DETAILED DESCRIPTION OF THE INVENTION

FIG.10aillustrates an embodiment of an apparatus for estimating an inter-channel time difference between a first channel signal such as a left channel and a second channel signal such as a right channel. These channels are input into a time-spectral converter150that is additionally illustrated, with respect toFIG.4eas item451.

Furthermore, the time-domain representations of the left and the right channel signals are input into a calculator1020for calculating a cross-correlation spectrum for a time block from the first channel signal in the time block and the second channel signal in the time block. Furthermore, the apparatus comprises a spectral characteristic estimator1010for estimating a characteristic of a spectrum of the first channel signal or the second channel signal for the time block. The apparatus further comprises a smoothing filter1030for smoothing the cross-correlation spectrum over time using the spectral characteristic to obtain a smoothed cross-correlation spectrum. The apparatus further comprises a processor1040for processing the smoothed correlation spectrum to obtain the inter-channel time difference.

Alternatively, in another embodiment, the element1030is not present, and, therefore, the element1010is not necessary as well, as indicated by the broken line1035. The apparatus further comprises a signal analyzer1037calculating a signal characteristic estimate such as a noise estimate1038. This estimate is forwarded to a weighter1036configured for performing different weighting operations depending on the signal characteristic estimate. The signal characteristic estimate is advantageously also used to control the processor1040for example when the processor1040performs the peak picking operation.FIG.10cfurther illustrates the signal analyzer1037and the controllable weighter1036.

Particularly, an apparatus in accordance with embodiments of the present invention is directed to the estimation of an inter-channel time difference between a first channel signal and a second channel signal. This device comprises the signal analyzer1037ofFIG.10a, a cross-correlation spectrum calculator1020ofFIG.10a, a weighter1036for weighting a smoothed or a non-smoothed cross-correlation spectrum ofFIG.10aand a subsequently connected processor1040for processing the weighted cross-correlation spectrum.

The elements time-spectrum converter150, spectral characteristic estimator1010, smoothing filter1030are not necessary for a basic implementation of the preset invention, but are advantageous for embodiments of the present invention. The signal analyzer1037is configured for estimating a signal characteristic such as a noise level1038of the first channel signal or the second channel signal or both signals or a signal derived from the first channel signal or the second channel signal. Thus, a signal characteristic or signal characteristic estimate such as a noise estimate to be used later on by the weighter1036and, advantageously, also used by the processor1040can be derived only from the left or first channel signal, only from the second or right channel signal, or can be derived from both signals. The derivation of the signal characteristic from both signals could, for example, be a derivation of an individual signal characteristic of the first channel signal, an additional individual signal characteristic from the second or right channel signal and, then, the final signal characteristic1038would be, for example, an average or a weighted average between both channels. Here, for example the weighting can be done in accordance with the amplitude so that different amplitudes in, for example, frames of the channels result in different influences of the corresponding individual noise estimate into the final noise level1038. Furthermore, the signal derived from the first channel signal and the second channel signal could be, for example, a combination signal obtained by adding the left or first channel signal and the second or right channel signal together to obtain a combined signal and, then, the signal characteristic1038is calculated from the combined signal.

In an embodiment, the signal analyzer1036is implemented as a noise estimator or analyzer. However, other ways of signal analysis can be performed as well such as a tonality analysis, a voice activity detection, a transient analysis, a stereo analysis, a speech/music analysis, interfering-talker analysis, a background music analysis, a clean speech analysis or any other signal analysis in order to determine, whether a signal has a first characteristic or a second characteristic so that the matching weighting procedure is selected.

The combination can be a combination with equal weighting factors, i.e., a combination of the left channel without any weighting and the right channel without any weighting which would, correspond to weighting factors of 1.0 or, alternatively, different weighting factors can be applied. Furthermore, the signal derived from the first channel or the signal derived from the second channel can be obtained by performing a high-pass filtering or low-pass filtering or can be derived by performing a processing using an amplitude compression or an amplitude inverse compression function. An amplitude compression function would be a log function or a function with a power value being smaller than 1. An inverse compression function would be an exponential function or a power function with an exponent being greater than 1. Thus, depending on certain implementations, different processing operations can be applied to different left and right channel signals and both channels can be combined or not. In the embodiment, the left and the right channels are added together advantageously even without any specific weighting and the signal characteristic estimate is then calculated from the result of the combination calculation.

The calculator1020for calculating a cross-correlation spectrum for a time block from the first channel signal in the time block and the second channel signal in the time block can be implemented in several ways. One way is that a cross-correlation is calculated from the time domain signals in the time domain frames and the result is then converted from the time domain into the spectral domain. Another implementation is that, for example, by using a DFT or any other time-to-spectral conversion, subsequent frames of the first channel signal and subsequent frames of the second channel signal are converted into a spectral representation where the subsequent frames can be overlapping or non-overlapping. Thus, for each time block of the first channel signal, a spectral representation is obtained and, correspondingly, for each time block of the second channel signal, a spectral representation is obtained. The cross-correlation calculation is performed by multiplying a spectral value of a certain frequency bin k and a certain time block or time sample index s by the conjugate complex value of the spectral value with the same index k and the same index s from the spectral representation of the same time block of the second channel. Other cross-correlation calculation procedures different from the above described can be used as well in order to calculate the cross-correlation spectrum for a time block.

The weighter1036is configured for weighting the cross-correlation spectrum obtained by the calculator. In an implementation, the cross-correlation spectrum is a non-smoothed cross-correlation spectrum, but in other embodiments, the cross-correlation spectrum is smoothed where this smoothing is a smoothing with respect to time. Thus, for the purpose of calculating the smoothed cross-correlation spectrum, the cross-correlation spectrum of the last block can be used together with a (raw) cross-correlation spectrum of the current block and, depending on the implementation, a smoothing control information can be used as is, for example, provided by the spectral characteristic estimator1010ofFIG.10a. However, the smoothing can also be performed using a predetermined, i.e., constant or time-invariant smoothing setting. In accordance with embodiments of the invention, the weighted cross-correlation spectrum is calculated using a first weighting procedure1036aor using a second weighted procedure1036bwhich are, for example, illustrated inFIG.10d. Particularly, the selection, whether the weighted cross-correlation spectrum is derived using the first or the second procedure is done depending on the signal characteristic estimated by the signal analyzer1037. Thus, in accordance with the present invention, a weighting with a first weighting characteristic is used for a certain signal characteristic of the first channel or the second channel or the combined signal while a second weighting procedure is applied depending on another signal characteristic as determined by the signal analyzer1037. The result of the weighter1036is a weighted and smoothed or non-smoothed cross-correlation spectrum that is then further processed by the processor1040to obtain the interchannel time difference between the first channel signal and the second channel signal.

FIG.10dillustrates an implementation of the signal analyzer as a noise estimator and the weighter in connection with the processor1040in accordance with an embodiment of the invention. Particularly, the noise estimator1037comprises a noise estimate calculator1037aand a noise estimate classifier1037b. The noise estimate classifier1037boutputs a control signal1050corresponding to the noise estimate output1038generated by block1037inFIG.10a. This control signal can be applied to a first switch1036cor a second switch1036d. In this implementation, processing kernels1036aimplementing the first weighting procedure and another calculating kernel for implementing the second weighting procedure1036bis provided. Depending on the implementation, only switch1036cis provided and, depending on the control signal1050, only the weighting procedure as determined by the switch1036cis selected, i.e., the cross-correlation spectrum as determined by the calculator1020is input into the switch1036cand depending on the switch setting, forwarding to either the kernel1036aor the kernel1036b. In another implementation, switch1036cis not there by the cross-correlation spectrum as determined by block1020is fed into both processing kernels1036aand1036band, depending on the control of the output switch1036d, either the output of block1036aor the output of block1036bis selected and forwarded to the processor1040. Thus, depending on the implementation, only a single weighted cross-correlation spectrum is calculated where the selection of which one is calculated is done by the control signal1050and the input switch. Alternatively, both weighted cross-correlation spectra are calculated and only the cross-correlation spectrum that is selected by the output switch1036dis forwarded to the processor1040. Furthermore, only a single processing kernel can be there without any input/output switches and depending on the control signal, the correct weighting procedure is set for the corresponding time block. Thus, for each time block, a noise estimate or control signal1050can be calculated and, for each time block, the weighting can be switched from one weighting procedure to the other weighting procedure. In this context, it is to be noted that there can also be implemented three or more different weighting procedures depending on three or more different noise estimates as the case may be. Thus, the present invention not only incurs the selection between two different weighting procedures, but also includes the selection between three or more weighting procedures depending on a control signal derived from the noise characteristic of the first and the second channel signals.

In an implementation, the first weighting procedure comprises a weighting so that an amplitude is normalized and a phase is maintained and the second weighting procedure comprises a weighting factor derived from the smoothed or the non-smoothed cross-correlation spectrum using a power operation having a power being lower than 1 or greater than 0. Preferably, the power is between 0.79 and 0.82. Furthermore, the first weighting procedure can be most identical to the second weighting procedure except that the second weighting procedure uses a power between 0 and 1, i.e., a power being greater than 0 and smaller than 1, while the first weighting procedure does not apply any power or, stated in other words, applies a power of 1. Thus, the normalization performed by the second weighting procedure is compressed, i.e., that normalization factor applied by the first weighing procedure has some value and the normalization factor applied via the second weighting procedure to the same spectral cross-correlation value has a smaller magnitude. This applies for higher spectral values of the cross-correlation spectrum. However, for small values of the cross-correlation spectrum, the normalization value for the second weighting procedure is greater than the normalization value for the first weighting procedure with respect to the same spectral value of the cross-correlation spectrum. This is due to the fact that a power operation with a pow-er lower than 1 such as a square root operation having a power of ½ increases small values but lowers high values. Thus, additional weighting factor calculations for the second weighting procedure can also comprise any compression function such as a log function. In an embodiment, the first weighting procedure operates based on the weighting applied for the phase transform (PHAT), and the second weighting procedure operates based on the calculations applied for modified cross-power spectrum phase procedure (MCSP).

Furthermore, the second weighting procedure is advantageously implemented to comprise a normalization so that an output range of the second normalization procedure is in a range, in which an output range of the first normalization procedure is positioned, or so that the output range of the second normalization procedure is the same as an output range of the first normalization procedure. This can, for example, be implemented by calculating the absolute values of all spectral values of the MCSP-weighted cross-correlation spectrum, by adding together all magnitudes of one spectral representation corresponding to one time block and by then dividing the result by the number of spectral values in a time block.

Generally, the processor1040ofFIG.10ais configured to perform some processing steps with respect to the weighted cross-correlation spectrum where, particularly, a certain peak picking operation is performed in order to finally obtain the inter-channel time difference. Advantageously, this peak picking operation takes place in the time domain, i.e., the weighted and smoothed or non-smoothed cross-correlation spectrum is converted from the spectral representation in a time domain representation and, then, this time domain representation is analyzed and, particularly, a peak or several peaks are picked based on a threshold. Depending on the setting of the noise estimate, either a first peak picking operation or a second peak picking operation is performed, where, advantageously, both peak picking operations are different from each other with respect to the threshold used by the peak picking operation.

FIG.10eillustrates a situation being similar, with respect to input switch1040and output switch1043, to the procedure inFIG.10d. In an implementation illustrated inFIG.10e, both peak picking operations can be applied and the result of the “correct” peak picking operation can be selected by the output switch1043. Alternatively, the input switch is there and depending on the control signal1050, only the correct peak picking procedure is selected, i.e., either1041or1042. Thus, in an implementation, there will not be both switches, but in an implementation there will be either the input switch1040or the output switch1043in analogy of what has been derived before with respect toFIG.10d. In an additional implementation, there only exists a single processing kernel applying the peak picking operation with a variable threshold and the control signal1050is used in order to set the correct threshold within the single processing kernel. In an embodiment, the threshold setting is performed in such a way that the second threshold is higher than the first threshold, where the second threshold, therefore, is used when the second weighting procedure in block1036bhas been applied, and where the first threshold is used, when the first weighting procedure in block1036ahas been applied. Thus, when a high level of background noise is detected, then the second weighting procedure with a power between 0 and 1 or a log operation, i.e., a compression procedure is applied and, then, the threshold for the peak picking should be lower compared to a peak picking threshold to be used when a low level of background noise is detected, i.e., when the first weighting procedure is applied that performs a normalization with a normalization factor that does not rely on a compression function such as a log function or a power function with a power smaller than 1.

Subsequently, an implementation of the signal analyzer as the noise estimator1037is illustrated inFIG.10f. Basically, the noise estimator1037consists of a noise estimate calculator1037aand a noise estimate classifier1037bas illustrated inFIG.10dand also indicated inFIG.10f. The noise estimate calculator1037acomprises a background noise estimator1060and the subsequently connected (time) smoother1061which can, for example, be implemented as an IIR filter.

The input into the noise estimate calculator1037aor, particularly, the background noise estimator1060is a frame of the left or first channel signal, a frame of the second or right channel signal or a signal derived from such channel signal or a combined signal obtained by adding, for example, a time domain representation of the first channel signal and a time domain representation of the second channel signal in the same time block.

With respect to the noise estimate classifier1037b, the input signal is delivered to a signal activity detector1070controlling a selector1071. Based on the result of the signal activity detector1070, the selector1071selects the active frames only. Furthermore, a signal level calculator1072is connected subsequent to the selector1071. The calculated signal level is then forwarded to a (time) smoother1073which is, for example, implemented as an IIR filter. Then, in blocks1074, a signal-to-noise ratio calculation takes place and the result is compared, within a comparator1075to an advantageously predetermined threshold which is, for example, between 45 dB and 25 dB and advantageously is even in a range between 30 and 40 dB and more advantageously, is at 35 dB.

The output of the comparator1075is the detection result indicating either a high noise level or a low noise level or indicating that a threshold setting in a certain way is to be performed by a single weighting procedure processor or, when there are two weighing procedure processors as illustrated inFIG.10d, then the decision result from the comparator1075, i.e., signal1050controls either the input switch1036cor the output switch1036din order to forward the correctly weighted cross-correlation spectrum to the processor1040.

The detection result1050is advantageously calculated for each time block or frame. Thus, when, for example, for a certain frame, the signal activity detector1070indicates that this is a non-active frame, then neither a signal level calculation nor a time smoothing is performed for this frame, since the selector1071only selects an active frame. Thus, for an inactive frame an SNR ratio calculation is not performed in an embodiment and, therefore, in this embodiment, for this inactive frame, a detection result is not provided at all. Thus, in an implementation, the same weighting procedure as has been determined before with respect to the last active frame is used or, alternatively, for an inactive frame, either the first weighting procedure or the second weighting procedure or even a third weighting procedure is applied as fallback solution. Alternatively, the SNR ratio calculator1074can be implemented to use, for an inactive frame, the time-smoothed signal level of the last or most recently occurring active frame. Thus, detection result can either be obtained even for inactive frames or, for inactive frames, a certain (fallback) weighting procedure is used or, for inactive frames, the same weighting procedure as has been determined for the last active frame preceding the inactive frame is continued to be used as the case may be.

In a previous patent application [1], an Inter-channel Time Difference (ITD) estimator was introduced. This estimator is based on the Generalized Cross-Correlation with PHAse Transform (GCC-PHAT), a technique widely used in the TDOA literature (initial paper is [2], another good reference is [3]). The time difference between the two channels is found by peak-picking the output of the GCC. Better robustness can be obtained either by using a large analysis window length or by smoothing the cross-correlation spectrum over time. The main contribution of [1] was to make this smoothing adaptive with a smoothing factor dependent on a spectral flatness measure.

The steps of the ITD estimator of [1] can be described as follows:1. Discrete Fourier Transform: the signal of the left channel xL(n) and the signal of the right channel xR(n) are framed, windowed and transformed to the frequency-domain using a DFT

XL(k,s)=∑n=0ND⁢F⁢T-1xL(n+s⁢N)⁢w⁡(n)⁢e-i⁢2⁢π⁢k⁢nND⁢F⁢T⁢XR(k,s)=∑n=0ND⁢F⁢T-1xR(n+s⁢N)⁢w⁡(n)⁢e-i⁢2⁢π⁢k⁢nND⁢F⁢T⁢XL(k,s)=∑n=0ND⁢F⁢T-1xL(n+s⁢N)⁢w⁡(n)⁢e-i⁢2⁢π⁢k⁢nND⁢F⁢T⁢XR(k,s)=∑n=0ND⁢F⁢T-1xR(n+s⁢N)⁢w⁡(n)⁢e-i⁢2⁢π⁢k⁢nND⁢F⁢Twith n is the time sample index, s is the frame index, k is the frequency index, N is the frame length, NDFTis the DFT length and w(n) is the analysis window.2. Cross-correlation spectrum: the correlation between the two channels is computed in the frequency domain
C(k,s)=XL(k,s)XR*(k,s)3. Smoothing: the cross-correlation spectrum is smoothed over time with a smoothing factor depending on a spectral flatness measure. Stronger smoothing is used when the spectral flatness is low in order to make the ITD estimator more robust on stationary tonal signals. Weaker smoothing is used when the spectral flatness is high in order to make the ITD estimator adapt faster on transient signals i.e. when the signal is quickly changing. The smoothing is performed using

C˜(k,s)=(1-s⁢f⁢m⁡(s))⁢C˜(k,s-1)+s⁢f⁢m⁡(s)⁢C⁡(k,s)⁢with⁢sfm⁡(s)=max⁡(sfm⁢_⁢chan⁢(XL),sfm⁢_⁢chan⁢(XR))⁢and⁢sfm⁡(s)=max⁡(sfm⁢_⁢chan⁢(XL),sfm⁢_⁢chan⁢(XR))⁢sfm⁢_⁢chan⁢(X)=∏k=0Ns⁢f⁢m-1X⁡(k,s)1Ns⁢f⁢m∑k=0Ns⁢f⁢m-1X⁡(k,s)Ns⁢f⁢m⁢sfm⁢_⁢chan⁢(X)=∏k=0Ns⁢f⁢m-1X⁡(k,s)1Ns⁢f⁢m∑k=0Ns⁢f⁢m-1X⁡(k,s)Nsfm4. Weighting: the smoothed cross-correlation spectrum is weighted by the inverse of its magnitude. This weighting normalizes the amplitude and keeps only the phase, this is why it is called the Phase Transform (PHAT).

C˜P⁢H⁢A⁢T(k,s)=C˜(k,s)❘"\[LeftBracketingBar]"C˜(k,s)❘"\[RightBracketingBar]"5. Inverse Transform: the final GCC is obtained by transforming the cross-correlation spectrum {tilde over (C)}PHAT(k,s) back to the time-domain

G⁢C⁢C⁡(n)=1ND⁢F⁢T⁢∑k=0ND⁢F⁢T-1C˜P⁢H⁢A⁢T(k,s)⁢ei⁢2⁢π⁢k⁢nND⁢F⁢T6. Peak-picking: the simplest approach is to search for the global maximum of the absolute value of the GCC found in Step 5. If this maximum has a value above some threshold, an ITD is estimated as the lag n corresponding to this maximum. More advanced approaches use additionally hysteresis- and/or hangover-based mechanisms to obtain a smoother ITD estimation over time.

The GGC-PHAT performs very well in low noise, reverberative environments (see for example [3]). However, when the level of the background noise is high or at presence of other signal components (such as music, transients, complex stereo scenes, frames classified as inactive, interfering talkers), the GCC-PHAT performance drops significantly. The GCC output is then noisy and does not contain one single strong peak. Consequently, a peak-picking often fails to find the correct ITD. This is because the Phase Transform treats all frequencies equally, regardless of the signal-to-noise ratio. The GCC is then polluted by the phase of the bins whose signal-to-noise ratio is low. To avoid this problem, many other GCC weightings were proposed in the literature. One of them was found to be very effective on our problematic test signals. It was first proposed in [4] and was called at that time “modified cross-power spectrum phase” (MCSP). Its good performance in high noise environments was later confirmed in several other papers (see e.g. [5]). The weighting (Step 4. of known technology) is modified as follows:

C~MCSP(k,s)=C~(k,s)❘"\[LeftBracketingBar]"C~(k,s)❘"\[RightBracketingBar]"ρ=C~PHAT(k,s)⁢❘"\[LeftBracketingBar]"C~(k,s)❘"\[RightBracketingBar]"1-ρ
with ρ a parameter between 0 and 1. ρ=0 corresponds to the case of the normal cross-correlation and ρ=1 corresponds to the case of the GCC-PHAT. A value below but close to 1 is usually used, which allows to modify the GCC-PHAT by putting more emphasis to the bins with high correlation, those usually corresponding to the signal while the bins with low correlation corresponding to the noise. More precisely, we have found that a value ρ=0.8 gave the best performance (it was 0.75 in [4] and 0.78 in [5]).

Unfortunately, this new weighting performs better than GCC-PHAT only when a high-level of background noise is present. Alternative scenarios where the new weighting possibly performs better than GCC-PHAT are inactive frames (i.e. voice activity detection detects inactive, which could indicate a low speech level), presence of transients, complex stereo scenarios, music, interfering talkers, presence of background music, speech which is not clean, In clean environments, like speech with no or only a low-level of background noise or music or other signal components which deviate from clean speech, GCC-PHAT still performs better. In order to achieve the best results, it may be useful to switch between the two approaches depending on the signal content.

To detect the presence of high level of background noise in the signal, a noise estimator together with a signal activity detector (SAD) are used. The level of the signal is can be estimated on the frames where the SAD detects a signal, while the level of the noise lNis estimated by the noise estimator. The presence of high level of background noise is then simply detected by comparing the signal-to-noise-ratio snr=lS−lN(in dB) to a threshold, e.g. if snr<35 then high noise level is detected.

Once it is known whether the signal contains a high level of background noise or not, a decision is made to select either the PHAT weighting or the MCSP weighting for computing the GCC (Step 4. in the known technology). The peak-picking (Step 6. in the known technology) can also be modified depending on whether there is high background noise level detected, for exampling by lowering the threshold.

Subsequently, an embodiment is described in a step by step manner.0. High background noise level detection:a. a noise estimator (for example from [6]) is used to estimate the level of background noise lN. An IIR smoothing filter is used to smooth the noise level over time.b. a signal activity detector (for example from [6]) is used to classify a frame as active or inactive. The active frames are then used to compute the signal level lS, simply by computing the signal energy and smoothing it over time using a IIR smoothing filter.c. If the signal-to-noise-ratio snr=lS−lN(in dB) is below a threshold (e.g. dB), then high background noise level is detected.1. Discrete Fourier Transform: same as in any known technology2. Cross-correlation spectrum: same as in any known technology3. Smoothing: same as in any known technology or as described herein based on the spectral characteristic4. Weighting:If low level of background noise is detected, then the same weighting as in the known technology is used (GCC-PHAT).If high level of background noise is detected, then the MCSP weighting is used

C~MCSP(k,s)=C~(k,s)❘"\[LeftBracketingBar]"C~(k,s)❘"\[RightBracketingBar]"ρwith 0<ρ<1 (e.g. ρ=0.8). In order to keep the GCC-MCSP output in the same range as the GCC-PHAT output, an additional normalization step is performed

C~MCSP(k,s)=C~MCSP(k,s)1NDFT⁢∑k=0NDFT-1❘"\[LeftBracketingBar]"C~MCSP(k,s)❘"\[RightBracketingBar]"5. Inverse Transform: same as in any known technology6. Peak-picking: the peak-picking can be adapted in case high level of background noise is detected and the MCSP weighting is used. Particularly, it has been found that a lower threshold is beneficial.

Furthermore,FIG.10aillustrates an implementation that is different from the implementation ofFIG.10c. In the weighter1036ofFIG.10c, the weighter performs either the first or the second weighting procedure. However, in the weighter1036as illustrated inFIG.10a, the weighter only performs the second weighting procedure with respect to the notation inFIG.10dor10c. This implementation is useful, when a smoothing filter as illustrated in block1030is used that already performs the first weighing procedure subsequent to the smoothing or together with the smoothing in e.g. a single mathematical or hardware operation. Thus, in case of performing the first weighting procedure which is the normalization operation without any compression in the smoothing filter, then both, the smoothing filter1030on the one hand and the actual weighter1036on the hand correspond to the actual weighter for weighting the smoothed or non-smoothed or non-smoothed cross-correlation spectrum. Thus, in the implementation ofFIG.10a, the noise estimate1038is only provided to a separate weighter1036and the selection between either the output of the smoothing filter1030which is already weighted in accordance with the weighting procedure and the selection between the output of the actual weighter136inFIG.10ais done by a certain processor setting1040that automatically uses the output from the smoothing filter1030, when the weighter1036does not provide any output signal but automatically prioritizes the output of the weighter1036over the output of the smoothing filter1030, when the weighter1036provides and output. Then, the noise estimate1038or, as discussed in other figures, the control signal1050is then used for either activating or deactivating the weighter1036. Thus, the actual weighter for weighting the smoothed or non-smoothed cross-correlation spectrum using a first order weighting procedure can be implemented in many different ways such as in the specific activation/deactivation mode inFIG.10aor the two-kernel mode inFIG.10dwith an input or an output switch or in accordance with a single weighting procedure kernel that, depending on the control signal selects one or the other weighing procedure or adapts a general weighting processor to perform the first or the second weighting procedure.

Subsequently, an embodiment, where a smoothing is performed before weighting is described. In this context, the functionalities of the spectral characteristic estimator are also reflected byFIG.4e, items453,454in an embodiment.

Furthermore, the functionalities of the cross-correlation spectrum calculator1020are also reflected by item452inFIG.4edescribed later on in an embodiment.

Correspondingly, the functionalities of the smoothing filter1030are also reflected by item453in the context ofFIG.4eto be described later on. Additionally, the functionalities of the processor1040are also described in the context ofFIG.4ein an embodiment as items456to459.

Embodiments of the processor1040are also described inFIG.10c

Advantageously, the spectral characteristic estimation calculates a noisiness or a tonality of the spectrum where an implementation is the calculation of a spectral flatness measure being close to 0 in the case of tonal or non-noisy signals and being close to 1 in the case of noisy or noise-like signals.

Particularly, the smoothing filter is then configured to apply a stronger smoothing with a first smoothing degree over time in case of a first less noisy characteristic or a first more tonal characteristic, or to apply a weaker smoothing with a second smoothing degree over time in case of a second more noisy or second less tonal characteristic.

Particularly, the first smoothing is greater than the second smoothing degree, where the first noisy characteristic is less noisy than the second noisy characteristic or the first tonal characteristic is more tonal than the second tonal characteristic. The implementation is the spectral flatness measure.

Furthermore, as illustrated inFIG.11a, the processor is advantageously implemented to normalize the smoothed cross-correlation spectrum as illustrated at456inFIGS.4eand11abefore performing the calculation of the time-domain representation in step1031corresponding to steps457and458in the embodiment ofFIG.4e. However, as also outlined inFIG.11a, the processor can also operate without the normalization in step456inFIG.4e. Then, the processor is configured to analyze the time-domain representation as illustrated in block1032ofFIG.11ain order to find the inter-channel time difference. This analysis can be performed in any known way and will already result in an improved robustness, since the analysis is performed based on the cross-correlation spectrum being smoothed in accordance with the spectral characteristic.

As illustrated inFIG.11b, an implementation of the time-domain analysis1032is a low-pass filtering of the time-domain representation as illustrated at458inFIG.11bcorresponding to item458ofFIG.4eand a subsequent further processing1033using a peak searching/peak picking operation within the low-pass filtered time-domain representation.

As illustrated inFIG.11c, the implementation of the peak picking or peak searching operation is to perform this operation using a variable threshold. Particularly, the processor is configured to perform the peak searching/peak picking operation within the time-domain representation derived from the smoothed cross-correlation spectrum by determining1034a variable threshold from the time-domain representation and by comparing a peak or several peaks of the time-domain representation (obtained with or without spectral normalization) to the variable threshold, wherein the inter-channel time difference is determined as a time lag associated with a peak being in a predetermined relation to the threshold such as being greater than the variable threshold.

As illustrated inFIG.11d, one embodiment illustrated in the pseudo code related toFIG.4e-bdescribed later on consists in the sorting1034aof values in accordance with their magnitude. Then, as illustrated in item1034binFIG.11d, the highest for example 10 or 5% of the values are determined.

Then, as illustrated in step1034c, a number such as the number3is multiplied to the lowest value of the highest 10 or 5% in order to obtain the variable threshold.

As stated, advantageously, the highest 10 or 5% are determined, but it can also be useful to determine the lowest number of the highest 50% of the values and to use a higher multiplication number such as 10. Naturally, even a smaller amount such as the highest 3% of the values are determined and the lowest value among these highest 3% of the values is then multiplied by a number which is, for example, equal to 2.5 or 2, i.e., lower than 3. Thus, different combinations of numbers and percentages can be used in the embodiment illustrated inFIG.11d. Apart from the percentages, the numbers can also vary, and numbers greater than 1.5 are advantageous.

In a further embodiment illustrated inFIG.11e, the time-domain representation is divided into subblocks as illustrated by block1101, and these subblocks are indicated inFIG.13at1300. Here, about 16 subblocks are used for the valid range so that each subblock has a time lag span of 20. However, the number of subblocks can be greater than this value or lower and advantageously greater than 3 and lower than 50.

In step1102ofFIG.11e, the peak in each subblock is determined, and in step1103, the average peak in all the subblocks is determined. Then, in step1104, a multiplication value a is determined that depends on a signal-to-noise ratio on the one hand and, in a further embodiment, depends on the difference between the threshold and the maximum peak as indicated to the left of block1104. Depending on these input values, one of advantageously three different multiplication values is determined where the multiplication value can be equal to alow, ahighand alowest.

Then, in step1105, the multiplication value a determined in block1104is multiplied by the average threshold in order to obtain the variable threshold that is then used in the comparison operation in block1106. For the comparison operation, once again the time-domain representation input into block1101can be used or the already determined peaks in each subblock as outlined in block1102can be used.

Subsequently, further embodiments regarding the evaluation and detection of a peak within the time-domain cross-correlation function is outlined.

The evaluation and detection of a peak within the time-domain cross correlation function resulting from the generalized cross-correlation (GCC-PHAT) method in order to estimate the Inter-channel Time Difference (ITD) is not always straightforward due to different input scenarios. Clean speech input can result to a low deviation cross-correlation function with a strong peak, while speech in a noisy reverberant environment can produce a vector with high deviation and peaks with lower but still outstanding magnitude indicating the existence of ITD. A peak detection algorithm that is adaptive and flexible to accommodate different input scenarios is described.

Due to delay constraints, the overall system can handle channel time alignment up to a certain limit, namely ITD_MAX. The proposed algorithm is designed to detect whether a valid ITD exists in the following cases:Valid ITD due to outstanding peak. An outstanding peak within the [−ITD_MAX, ITD_MAX] bounds of the cross-correlation function is present.No correlation. When there is no correlation between the two channels, there is no outstanding peak. A threshold should be defined, above which the peak is strong enough to be considered as a valid ITD value. Otherwise, no ITD handling should be signaled, meaning ITD is set to zero and no time alignment is performed.Out of bounds ITD. Strong peaks of the cross-correlation function outside the region [−ITD_MAX, ITD_MAX] should be evaluated in order to determine whether ITDs that lie outside the handling capacity of the system exist. In this case no ITD handling should be signaled and thus no time alignment is performed.

To determine whether the magnitude of a peak is high enough to be considered as a time difference value, a suitable threshold needs to be defined. For different input scenarios, the cross-correlation function output varies depending on different parameters, e.g. the environment (noise, reverberation etc.), the microphone setup (AB, M/S, etc.). Therefore, to adaptively define the threshold is essential.

In the proposed algorithm, the threshold is defined by first calculating the mean of a rough computation of the envelope of the magnitude of the cross-correlation function within the [−ITD_MAX, ITD_MAX] region (FIG.13), the average is then weighted accordingly depending on the SNR estimation.

The step-by-step description of the algorithm is described below.

The output of the inverse DFT of the GCC-PHAT, which represents the time-domain cross-correlation, is rearranged from negative to positive time lags (FIG.12).

The cross-correlation vector is divided in three main areas: the area of interest namely [−ITD_MAX, ITD_MAX] and the area outside the ITD_MAX bounds, namely time lags smaller than −ITD_MAX (max_low) and higher than ITD_MAX (max_high). The maximum peaks of the “out of bound” areas are detected and saved to be compared to the maximum peak detected in the area of interest.

In order to determine whether a valid ITD is present, the sub-vector area [−ITD_MAX, ITD_MAX] of the cross-correlation function is considered. The sub-vector is divided into N sub-blocks (FIG.13).

For each sub-block the maximum peak magnitude peak_sub and the equivalent time lag position index_sub is found and saved.

The maximum of the local maxima peak_max is determined and will be compared to the threshold to determine the existence of a valid ITD value.

The maximum value peak_max is compared to max_low and max_high. If peak_max is lower than either of the two then no itd handling is signaled and no time alignment is performed. Because of the ITD handling limit of the system, the magnitudes of the out of bound peaks do not need to be evaluated.

The mean of the magnitudes of the peaks is calculated:

peakmean=∑Npeak_subN

The threshold thres is then computed by weighting peakmeanwith an SNR depended weighting factor aw:

thres=aw⁢peakmean,where⁢aw={alowSNR≤SNRthresholdahigh,SNR>SNRthreshold

In cases where SNR<<SNRthresholdand |thres−peak_max|<ε, the peak magnitude is also compared to a slightly more relaxed threshold (aw=alowest), in order to avoid rejecting an outstanding peak with high neighboring peaks. The weighting factors could be for example ahigh=3, alow=2.5 and alowest=2, while the SNRthresholdcould be for example 20 dB and the bound ε=0.05.

Advantageous ranges are 2.5 to 5 for ahigh; 1.5 to 4 for alow; 1.0 to 3 for alowest; 10 to 30 dB for SNRthreshold; and 0.01 to 0.5 for ε, where ahighis greater than alowthat is greater than alowest.

If peak_max>thres the equivalent time lag is returned as the estimated ITD, elsewise no itd handling is signaled (ITD=0). Further embodiments are described later on with respect toFIG.4e.

FIG.11fillustrates the implementation of determining a valid ITD (inter-channel time difference) output.

Subblocks of the time domain representation of the weighted and smoothed or non-smoothed cross-correlation spectrum are input into a determination step within the processor1040. This determination step1120determines a valid range and an invalid range within a time-domain representation derived from the weighted and smoothed or non-smoothed cross-correlation spectrum. In step1121, a maximum peak is determined within the invalid range, and in step1122, a maximum peak is determined within the valid range. Particularly, at least one maximum peak is determined within the invalid range and at least one maximum peak is determined within the valid range. In block1123, the maximum peaks of the valid range and the invalid range are compared. In case the valid peak, i.e., the maximum peak in the valid range is greater than the “invalid peak”, the maximum peak in the invalid range, then an ITD determination1124is actually performed and a valid ITD output is provided. When, however, it is detected that an “invalid peak” is greater than the “valid peak” or that the invalid peak has the same size as the valid peak, then a valid output is not provided and, advantageously, an error message or any comparable action is performed in order to bring this to the processor's attention.

Subsequently, an implementation of the present invention within block1050ofFIG.10bfor the purpose of a signal further processor is discussed with respect toFIGS.1to9e, i.e., in the context of a stereo/multi-channel processing/encoding and time alignment of two channels.

However, as stated and as illustrated inFIG.10b, many other fields exist, where a signal further processing using the determined inter-channel time difference can be performed as well.

FIG.1illustrates an apparatus for encoding a multi-channel signal having at least two channels. The multi-channel signal10is input into a parameter determiner100on the one hand and a signal aligner200on the other hand. The parameter determiner100determines, on the one hand, a broadband alignment parameter and, on the other hand, a plurality of narrowband alignment parameters from the multi-channel signal. These parameters are output via a parameter line12. Furthermore, these parameters are also output via a further parameter line14to an output interface500as illustrated. On the parameter line14, additional parameters such as the level parameters are forwarded from the parameter determiner100to the output interface500. The signal aligner200is configured for aligning the at least two channels of the multi-channel signal10using the broadband alignment parameter and the plurality of narrowband alignment parameters received via parameter line10to obtain aligned channels20at the output of the signal aligner200. These aligned channels20are forwarded to a signal processor300which is configured for calculating a mid-signal31and a side signal32from the aligned channels received via line20. The apparatus for encoding further comprises a signal encoder400for encoding the mid-signal from line31and the side signal from line32to obtain an encoded mid-signal on line41and an encoded side signal on line42. Both these signals are forwarded to the output interface500for generating an encoded multi-channel signal at output line50. The encoded signal at output line50comprises the encoded mid-signal from line41, the encoded side signal from line42, the narrowband alignment parameters and the broadband alignment parameters from line14and, optionally, a level parameter from line14and, additionally optionally, a stereo filling parameter generated by the signal encoder400and forwarded to the output interface500via parameter line43.

Advantageously, the signal aligner is configured to align the channels from the multichannel signal using the broadband alignment parameter, before the parameter determiner100actually calculates the narrowband parameters. Therefore, in this embodiment, the signal aligner200sends the broadband aligned channels back to the parameter determiner100via a connection line15. Then, the parameter determiner100determines the plurality of narrowband alignment parameters from an already with respect to the broadband characteristic aligned multi-channel signal. In other embodiments, however, the parameters are determined without this specific sequence of procedures.

FIG.4aillustrates an implementation, where the specific sequence of steps that incurs connection line15is performed. In the step16, the broadband alignment parameter is determined using the two channels and the broadband alignment parameter such as an inter-channel time difference or ITD parameter is obtained. Then, in step21, the two channels are aligned by the signal aligner200ofFIG.1using the broadband alignment parameter. Then, in step17, the narrowband parameters are determined using the aligned channels within the parameter determiner100to determine a plurality of narrowband alignment parameters such as a plurality of inter-channel phase difference parameters for different bands of the multi-channel signal. Then, in step22, the spectral values in each parameter band are aligned using the corresponding narrowband alignment parameter for this specific band. When this procedure in step22is performed for each band, for which a narrowband alignment parameter is available, then aligned first and second or left/right channels are available for further signal processing by the signal processor300ofFIG.1.

FIG.4billustrates a further implementation of the multi-channel encoder ofFIG.1where several procedures are performed in the frequency domain.

Specifically, the multi-channel encoder further comprises a time-spectrum converter150for converting a time domain multi-channel signal into a spectral representation of the at least two channels within the frequency domain.

Furthermore, as illustrated at152, the parameter determiner, the signal aligner and the signal processor illustrated at100,200and300inFIG.1all operate in the frequency domain.

Furthermore, the multi-channel encoder and, specifically, the signal processor further comprises a spectrum-time converter154for generating a time domain representation of the mid-signal at least.

Advantageously, the spectrum time converter additionally converts a spectral representation of the side signal also determined by the procedures represented by block152into a time domain representation, and the signal encoder400ofFIG.1is then configured to further encode the mid-signal and/or the side signal as time domain signals depending on the specific implementation of the signal encoder400ofFIG.1.

Advantageously, the time-spectrum converter150ofFIG.4bis configured to implement steps155,156and157ofFIG.4c. Specifically, step155comprises providing an analysis window with at least one zero padding portion at one end thereof and, specifically, a zero padding portion at the initial window portion and a zero padding portion at the terminating window portion as illustrated, for example, inFIG.7later on. Furthermore, the analysis window additionally has overlap ranges or overlap portions at a first half of the window and at a second half of the window and, additionally, advantageously a middle part being a non-overlap range as the case may be.

In step156, each channel is windowed using the analysis window with overlap ranges. Specifically, each channel is windowed using the analysis window in such a way that a first block of the channel is obtained. Subsequently, a second block of the same channel is obtained that has a certain overlap range with the first block and so on, such that subsequent to, for example, five windowing operations, five blocks of windowed samples of each channel are available that are then individually transformed into a spectral representation as illustrated at157inFIG.4c. The same procedure is performed for the other channel as well so that, at the end of step157, a sequence of blocks of spectral values and, specifically, complex spectral values such as DFT spectral values or complex subband samples is available.

In step158, which is performed by the parameter determiner100ofFIG.1, a broadband alignment parameter is determined and in step159, which is performed by the signal alignment200ofFIG.1, a circular shift is performed using the broadband alignment parameter. In step160, again performed by the parameter determiner100ofFIG.1, narrowband alignment parameters are determined for individual bands/subbands and in step161, aligned spectral values are rotated for each band using corresponding narrowband alignment parameters determined for the specific bands.

FIG.4dillustrates further procedures performed by the signal processor300. Specifically, the signal processor300is configured to calculate a mid-signal and a side signal as illustrated at step301. In step302, some kind of further processing of the side signal can be performed and then, in step303, each block of the mid-signal and the side signal is transformed back into the time domain and, in step304, a synthesis window is applied to each block obtained by step303and, in step305, an overlap add operation for the mid-signal on the one hand and an overlap add operation for the side signal on the other hand is performed to finally obtain the time domain mid/side signals.

Specifically, the operations of the steps304and305result in a kind of cross fading from one block of the mid-signal or the side signal in the next block of the mid signal and the side signal is performed so that, even when any parameter changes occur such as the inter-channel time difference parameter or the inter-channel phase difference parameter occur, this will nevertheless be not audible in the time domain mid/side signals obtained by step305inFIG.4d.

The new low-delay stereo coding is a joint Mid/Side (M/S) stereo coding exploiting some spatial cues, where the Mid-channel is coded by a primary mono core coder, and the Side-channel is coded in a secondary core coder. The encoder and decoder principles are depicted inFIGS.6a,6b.

The stereo processing is performed mainly in Frequency Domain (FD). Optionally some stereo processing can be performed in Time Domain (TD) before the frequency analysis. It is the case for the ITD computation, which can be computed and applied before the frequency analysis for aligning the channels in time before pursuing the stereo analysis and processing. Alternatively, ITD processing can be done directly in frequency domain. Since usual speech coders like ACELP do not contain any internal time-frequency decomposition, the stereo coding adds an extra complex modulated filter-bank by means of an analysis and synthesis filter-bank before the core encoder and another stage of analysis-synthesis filter-bank after the core decoder. In the embodiment, an oversampled DFT with a low overlapping region is employed. However, in other embodiments, any complex valued time-frequency decomposition with similar temporal resolution can be used.

The stereo processing consists of computing the spatial cues: inter-channel Time Difference (ITD), the inter-channel Phase Differences (IPDs) and inter-channel Level Differences (ILDs). ITD and IPDs are used on the input stereo signal for aligning the two channels L and R in time and in phase. ITD is computed in broadband or in time domain while IPDs and ILDs are computed for each or a part of the parameter bands, corresponding to a non-uniform decomposition of the frequency space. Once the two channels are aligned a joint M/S stereo is applied, where the Side signal is then further predicted from the Mid signal. The prediction gain is derived from the ILDs.

The Mid signal is further coded by a primary core coder. In the embodiment, the primary core coder is the 3GPP EVS standard, or a coding derived from it which can switch between a speech coding mode, ACELP, and a music mode based on a MDCT transformation. Advantageously, ACELP and the MDCT-based coder are supported by a Time Domain BandWidth Extension (TD-BWE) and or Intelligent Gap Filling (IGF) modules respectively.

The Side signal is first predicted by the Mid channel using prediction gains derived from ILDs. The residual can be further predicted by a delayed version of the Mid signal or directly coded by a secondary core coder, performed in the embodiment in MDCT domain. The stereo processing at encoder can be summarized byFIG.5as will be explained later on.

FIG.2illustrates a block diagram of an embodiment of an apparatus for decoding an encoded multi-channel signal received at input line50.

In particular, the signal is received by an input interface600. Connected to the input interface600are a signal decoder700, and a signal de-aligner900. Furthermore, a signal processor800is connected to a signal decoder700on the one hand and is connected to the signal de-aligner on the other hand.

In particular, the encoded multi-channel signal comprises an encoded mid-signal, an encoded side signal, information on the broadband alignment parameter and information on the plurality of narrowband parameters. Thus, the encoded multi-channel signal on line50can be exactly the same signal as output by the output interface of500ofFIG.1.

However, importantly, it is to be noted here that, in contrast to what is illustrated inFIG.1, the broadband alignment parameter and the plurality of narrowband alignment parameters included in the encoded signal in a certain form can be exactly the alignment parameters as used by the signal aligner200inFIG.1but can, alternatively, also be the inverse values thereof, i.e., parameters that can be used by exactly the same operations performed by the signal aligner200but with inverse values so that the de-alignment is obtained.

Thus, the information on the alignment parameters can be the alignment parameters as used by the signal aligner200inFIG.1or can be inverse values, i.e., actual “de-alignment parameters”. Additionally, these parameters will typically be quantized in a certain form as will be discussed later on with respect toFIG.8.

The input interface600ofFIG.2separates the information on the broadband alignment parameter and the plurality of narrowband alignment parameters from the encoded mid/side signals and forwards this information via parameter line610to the signal de-aligner900. On the other hand, the encoded mid-signal is forwarded to the signal decoder700via line601and the encoded side signal is forwarded to the signal decoder700via signal line602.

The signal decoder is configured for decoding the encoded mid-signal and for decoding the encoded side signal to obtain a decoded mid-signal on line701and a decoded side signal on line702. These signals are used by the signal processor800for calculating a decoded first channel signal or decoded left signal and for calculating a decoded second channel or a decoded right channel signal from the decoded mid signal and the decoded side signal, and the decoded first channel and the decoded second channel are output on lines801,802, respectively. The signal de-aligner900is configured for de-aligning the decoded first channel on line801and the decoded right channel802using the information on the broadband alignment parameter and additionally using the information on the plurality of narrowband alignment parameters to obtain a decoded multi-channel signal, i.e., a decoded signal having at least two decoded and de-aligned channels on lines901and902.

FIG.9aillustrates a sequence of steps performed by the signal de-aligner900fromFIG.2. Specifically, step910receives aligned left and right channels as available on lines801,802fromFIG.2. In step910, the signal de-aligner900de-aligns individual subbands using the information on the narrowband alignment parameters in order to obtain phase-de-aligned decoded first and second or left and right channels at911aand911b. In step912, the channels are de-aligned using the broadband alignment parameter so that, at913aand913b, phase and time-de-aligned channels are obtained.

In step914, any further processing is performed that comprises using a windowing or any overlap-add operation or, generally, any cross-fade operation in order to obtain, at915aor915b, an artifact-reduced or artifact-free decoded signal, i.e., to decoded channels that do not have any artifacts although there have been, typically, time-varying de-alignment parameters for the broadband on the one hand and for the plurality of narrowbands on the other hand.

FIG.9billustrates an implementation of the multi-channel decoder illustrated inFIG.2.

In particular, the signal processor800fromFIG.2comprises a time-spectrum converter810.

The signal processor furthermore comprises a mid/side to left/right converter820in order to calculate from a mid-signal M and a side signal S a left signal L and a right signal R.

However, importantly, in order to calculate L and R by the mid/side-left/right conversion in block820, the side signal S is not necessarily to be used. Instead, as discussed later on, the left/right signals are initially calculated only using a gain parameter derived from an inter-channel level difference parameter ILD. Generally, the prediction gain can also be considered to be a form of an ILD. The gain can be derived from ILD but can also be directly computed. It is advantageous to not compute ILD anymore, but to compute the prediction gain directly and to transmit and use the prediction gain in the decoder rather than the ILD parameter.

Therefore, in this implementation, the side signal S is only used in the channel updater830that operates in order to provide a better left/right signal using the transmitted side signal S as illustrated by bypass line821.

Therefore, the converter820operates using a level parameter obtained via a level parameter input822and without actually using the side signal S but the channel updater830then operates using the side821and, depending on the specific implementation, using a stereo filling parameter received via line831. The signal aligner900then comprises a phased-de-aligner and energy scaler910. The energy scaling is controlled by a scaling factor derived by a scaling factor calculator940. The scaling factor calculator940is fed by the output of the channel updater830. Based on the narrowband alignment parameters received via input911, the phase de-alignment is performed and, in block920, based on the broadband alignment parameter received via line921, the time-de-alignment is performed. Finally, a spectrum-time conversion930is performed in order to finally obtain the decoded signal.

FIG.9cillustrates a further sequence of steps typically performed within blocks920and930ofFIG.9bin an embodiment.

Specifically, the narrowband de-aligned channels are input into the broadband de-alignment functionality corresponding to block920ofFIG.9b. A DFT or any other transform is performed in block931. Subsequent to the actual calculation of the time domain samples, an optional synthesis windowing using a synthesis window is performed. The synthesis window is advantageously exactly the same as the analysis window or is derived from the analysis window, for example interpolation or decimation but depends in a certain way from the analysis window. This dependence advantageously is such that multiplication factors defined by two overlapping windows add up to one for each point in the overlap range. Thus, subsequent to the synthesis window in block932, an overlap operation and a subsequent add operation is performed. Alternatively, instead of synthesis windowing and overlap/add operation, any cross fade between subsequent blocks for each channel is performed in order to obtain, as already discussed in the context ofFIG.9a, an artifact reduced decoded signal.

WhenFIG.6bis considered, it becomes clear that the actual decoding operations for the mid-signal, i.e., the “EVS decoder” on the one hand and, for the side signal, the inverse vector quantization VQ−1and the inverse MDCT operation (IMDCT) correspond to the signal decoder700ofFIG.2.

Furthermore, the DFT operations in blocks810correspond to element810inFIG.9band functionalities of the inverse stereo processing and the inverse time shift correspond to blocks800,900ofFIG.2and the inverse DFT operations930inFIG.6bcorrespond to the corresponding operation in block930inFIG.9b.

Subsequently,FIG.3is discussed in more detail. In particular,FIG.3illustrates a DFT spectrum having individual spectral lines. Advantageously, the DFT spectrum or any other spectrum illustrated inFIG.3is a complex spectrum and each line is a complex spectral line having magnitude and phase or having a real part and an imaginary part.

Additionally, the spectrum is also divided into different parameter bands. Each parameter band has at least one and advantageously more than one spectral lines. Additionally, the parameter bands increase from lower to higher frequencies. Typically, the broadband alignment parameter is a single broadband alignment parameter for the whole spectrum, i.e., for a spectrum comprising all the bands1to6in the exemplary embodiment inFIG.3.

Furthermore, the plurality of narrowband alignment parameters are provided so that there is a single alignment parameter for each parameter band. This means that the alignment parameter for a band applies to all the spectral values within the corresponding band.

Furthermore, in addition to the narrowband alignment parameters, level parameters are also provided for each parameter band.

In contrast to the level parameters that are provided for each and every parameter band from band1to band6, it is advantageous to provide the plurality of narrowband alignment parameters only for a limited number of lower bands such as bands1,2,3and4.

Additionally, stereo filling parameters are provided for a certain number of bands excluding the lower bands such as, in the exemplary embodiment, for bands4,5and6, while there are side signal spectral values for the lower parameter bands1,2and3and, consequently, no stereo filling parameters exist for these lower bands where wave form matching is obtained using either the side signal itself or a prediction residual signal representing the side signal.

As already stated, there exist more spectral lines in higher bands such as, in the embodiment inFIG.3, seven spectral lines in parameter band6versus only three spectral lines in parameter band2. Naturally, however, the number of parameter bands, the number of spectral lines and the number of spectral lines within a parameter band and also the different limits for certain parameters will be different.

Nevertheless,FIG.8illustrates a distribution of the parameters and the number of bands for which parameters are provided in a certain embodiment where there are, in contrast toFIG.3, actually 12 bands.

As illustrated, the level parameter ILD is provided for each of 12 bands and is quantized to a quantization accuracy represented by five bits per band.

Furthermore, the narrowband alignment parameters IPD are only provided for the lower bands up to a border frequency of 2.5 kHz. Additionally, the inter-channel time difference or broadband alignment parameter is only provided as a single parameter for the whole spectrum but with a very high quantization accuracy represented by eight bits for the whole band.

Furthermore, quite roughly quantized stereo filling parameters are provided represented by three bits per band and not for the lower bands below 1 kHz since, for the lower bands, actually encoded side signal or side signal residual spectral values are included.

Subsequently, a processing on the encoder side is summarized with respect toFIG.5. In a first step, a DFT analysis of the left and the right channel is performed. This procedure corresponds to steps155to157ofFIG.4c. In step158, the broadband alignment parameter is calculated and, particularly, the broadband alignment parameter inter-channel time difference (ITD). As illustrated in170, a time shift of L and R in the frequency domain is performed. Alternatively, this time shift can also be performed in the time domain. An inverse DFT is then performed, the time shift is performed in the time domain and an additional forward DFT is performed in order to once again have spectral representations subsequent to the alignment using the broadband alignment parameter.

ILD parameters, i.e., level parameters and phase parameters (IPD parameters), are calculated for each parameter band on the shifted L and R representations as illustrated at step171. This step corresponds to step160ofFIG.4c, for example. Time shifted L and R representations are rotated as a function of the inter-channel phase difference parameters as illustrated in step161ofFIG.4corFIG.5. Subsequently, the mid and side signals are computed as illustrated in step301and, advantageously, additionally with an energy conversation operation as discussed later on. In a subsequent step174, a prediction of S with M as a function of ILD and optionally with a past M signal, i.e., a mid-signal of an earlier frame is performed. Subsequently, inverse DFT of the mid-signal and the side signal is performed that corresponds to steps303,304,305ofFIG.4din the embodiment.

In the final step175, the time domain mid-signal m and, optionally, the residual signal are coded as illustrated in step175. This procedure corresponds to what is performed by the signal encoder400inFIG.1.

At the decoder in the inverse stereo processing, the Side signal is generated in the DFT domain and is first predicted from the Mid signal as:
=g·Mid
where g is a gain computed for each parameter band and is function of the transmitted Inter-channel Level Difference (ILDs).

The residual of the prediction Side−g·Mid can be then refined in two different ways:By a secondary coding of the residual signal:
=g·Mid+gcod·(SideMιd)
where gcodis a global gain transmitted for the whole spectrumBy a residual prediction, known as stereo filling, predicting the residual side spectrum with the previous decoded Mid signal spectrum from the previous DFT frame:
=g·Mid+gpred·Mid·z−1
where gpredis a predictive gain transmitted per parameter band.

The two types of coding refinement can be mixed within the same DFT spectrum. In the embodiment, the residual coding is applied on the lower parameter bands, while residual prediction is applied on the remaining bands. The residual coding is in the embodiment as depict inFIG.1performs in MDCT domain after synthesizing the residual Side signal in Time Domain and transforming it by a MDCT. Unlike DFT, MDCT is critical sampled and is more suitable for audio coding. The MDCT coefficients are directly vector quantized by a Lattice Vector Quantization but can be alternatively coded by a Scalar Quantizer followed by an entropy coder. Alternatively, the residual side signal can be also coded in Time Domain by a speech coding technique or directly in DFT domain.

1. Time-Frequency Analysis: DFT

It is important that the extra time-frequency decomposition from the stereo processing done by DFTs allows a good auditory scene analysis while not increasing significantly the overall delay of the coding system. By default, a time resolution of 10 ms (twice the ms framing of the core coder) is used. The analysis and synthesis windows are the same and are symmetric. The window is represented at 16 kHz of sampling rate inFIG.7. It can be observed that the overlapping region is limited for reducing the engendered delay and that zero padding is also added to counter balance the circular shift when applying ITD in frequency domain as it will be explained hereafter.

2. Stereo Parameters

Stereo parameters can be transmitted at maximum at the time resolution of the stereo DFT. At minimum it can be reduced to the framing resolution of the core coder, i.e. 20 ms. By default, when no transients is detected, parameters are computed every 20 ms over 2 DFT windows. The parameter bands constitute a non-uniform and nonoverlapping decomposition of the spectrum following roughly 2 times or 4 times the Equivalent Rectangular Bandwidths (ERB). By default, a 4 times ERB scale is used for a total of 12 bands for a frequency bandwidth of 16 kHz (32 kbps sampling-rate, Super Wideband stereo).FIG.8summarized an example of configuration, for which the stereo side information is transmitted with about 5 kbps.

3. Computation of ITD and Channel Time Alignment

The ITD are computed by estimating the Time Delay of Arrival (TDOA) using the Generalized Cross Correlation with Phase Transform (GCC-PHAT):

ITD=arg⁢max⁡(IDFT⁡(Li(f)⁢Ri*(k)❘"\[LeftBracketingBar]"Li(f)⁢Ri*(k)❘"\[RightBracketingBar]"))
where L and R are the frequency spectra of the of the left and right channels respectively. The frequency analysis can be performed independently of the DFT used for the subsequent stereo processing or can be shared. The pseudo-code for computing the ITD is the following:

L =fft(window(l));R =fft(window(r));tmp = L . * conj( R );sfm_L = prod(abs(L).{circumflex over ( )}(1/length(L)))/(mean(abs(L))+eps);sfm_R = prod(abs(R).{circumflex over ( )}(1/length(R)))/(mean(abs(R))+eps);sfm = max(sfm_L,sfm_R);h.cross_corr_smooth = (1−sfm)*h.cross_corr_smooth+sfm*tmp;tmp = h.cross_corr_smooth ./ abs( h.cross_corr_smooth+eps );tmp = ifft( tmp );tmp = tmp([length(tmp)/2+1:length(tmp) 1:length(tmp)/2+1]);tmp_sort = sort( abs(tmp) );thresh = 3 * tmp_sort( round(0.95*length(tmp_sort)) );xcorr_time=abs(tmp(− ( h.stereo_itd_q_max − (length(tmp)−1)/2 − 1 ):− ( h.ste−reo_itd_q_min − (length(tmp)−1)/2 − 1 )));%smooth output for better detectionxcorr_time=[xcorr_time 0];xcorr_time2=filter([0.25 0.5 0.25], 1,xcorr_time);[m,i] = max(xcorr_time2(2:end));if m > threshitd = h.stereo_itd_q_max − i + 1;elseitd = 0;end

FIG.4eillustrates a flow chart for implementing the earlier illustrated pseudo code in order to obtain a robust and efficient calculation of an inter-channel time difference as an example for the broadband alignment parameter.

In block451, a DFT analysis of the time domain signals for a first channel (I) and a second channel (r) is performed. This DFT analysis will typically be the same DFT analysis as has been discussed in the context of steps155to157inFIG.5orFIG.4c, for example.

A cross-correlation is then performed for each frequency bin as illustrated in block452.

Thus, a cross-correlation spectrum is obtained for the whole spectral range of the left and the right channels.

In step453, a spectral flatness measure is then calculated from the magnitude spectra of L and R and, in step454, the larger spectral flatness measure is selected. However, the selection in step454does not necessarily have to be the selection of the larger one but this determination of a single SFM from both channels can also be the selection and calculation of only the left channel or only the right channel or can be the calculation of weighted average of both SFM values.

In step455, the cross-correlation spectrum is then smoothed over time depending on the spectral flatness measure.

Advantageously, the spectral flatness measure is calculated by dividing the geometric mean of the magnitude spectrum by the arithmetic mean of the magnitude spectrum. Thus, the values for SFM are bounded between zero and one.

In step456, the smoothed cross-correlation spectrum is then normalized by its magnitude and in step457an inverse DFT of the normalized and smoothed cross-correlation spectrum is calculated. In step458, a certain time domain filter is advantageously performed but this time domain filtering can also be left aside depending on the implementation but is advantageous as will be outlined later on.

In step459, an ITD estimation is performed by peak-picking of the filter generalized cross-correlation function and by performing a certain thresholding operation.

If no peak above the threshold is obtained, then ITD is set to zero and no time alignment is performed for this corresponding block.

The ITD computation can also be summarized as follows. The cross-correlation is computed in frequency domain before being smoothed depending of the Spectral Flatness Measurement. SFM is bounded between 0 and 1. In case of noise-like signals, the SFM will be high (i.e. around 1) and the smoothing will be weak. In case of tone-like signal, SFM will be low and the smoothing will become stronger. The smoothed cross-correlation is then normalized by its amplitude before being transformed back to time domain. The normalization corresponds to the Phase-transform of the cross-correlation, and is known to show better performance than the normal cross-correlation in low noise and relatively high reverberation environments. The so-obtained time domain function is first filtered for achieving a more robust peak peaking. The index corresponding to the maximum amplitude corresponds to an estimate of the time difference between the Left and Right Channel (ITD). If the amplitude of the maximum is lower than a given threshold, then the estimated of ITD is not considered as reliable and is set to zero.

If the time alignment is applied in Time Domain, the ITD is computed in a separate DFT analysis. The shift is done as follows:

{r⁡(n)=r⁡(n+ITD)⁢if⁢ITD>0l⁡(n)=l⁡(n-ITD)⁢if⁢ITD<0

An extra delay at encoder may be useful, which is equal at maximum to the maximum absolute ITD which can be handled. The variation of ITD over time is smoothed by the analysis windowing of DFT.

Alternatively the time alignment can be performed in frequency domain. In this case, the ITD computation and the circular shift are in the same DFT domain, domain shared with this other stereo processing. The circular shift is given by:

{L⁡(f)=L⁡(f)⁢e-j⁢2⁢π⁢f⁢ITD2R⁡(f)=R⁡(f)⁢e+j⁢2⁢π⁢f⁢ITD2

Zero padding of the DFT windows is needed for simulating a time shift with a circular shift. The size of the zero padding corresponds to the maximum absolute ITD which can be handled. In the embodiment, the zero padding is split uniformly on the both sides of the analysis windows, by adding 3.125 ms of zeros on both ends. The maximum absolute possible ITD is then 6.25 ms. In A-B microphones setup, it corresponds for the worst case to a maximum distance of about 2.15 meters between the two microphones. The variation in ITD over time is smoothed by synthesis windowing and overlap-add of the DFT.

It is important that the time shift is followed by a windowing of the shifted signal. It is a main distinction with the conventional Binaural Cue Coding (BCC), where the time shift is applied on a windowed signal but is not windowed further at the synthesis stage. As a consequence, any change in ITD overtime produces an artificial transient/click in the decoded signal.

4. Computation of IPDs and Channel Rotation

The IPDs are computed after time aligning the two channels and this for each parameter band or at least up to a given ipd_max_band, dependent of the stereo configuration.

IPD[b]=angle(∑k=bandlimits[b]bandlimits[b+1]L[k]⁢R*[k])

IPDs is then applied to the two channels for aligning their phases:

{L′(k)=L⁡(k)⁢e-j⁢βR′(k)=R⁡(k)⁢ej⁡(IPD[b]-β)

Where β=atan 2(sin(IPDi[b]), cos(IPDi[b])+c), c=10ILDi[b]/20and b is the parameter band index to which belongs the frequency index k. The parameter β is responsible of distributing the amount of phase rotation between the two channels while making their phase aligned. β is dependent of IPD but also the relative amplitude level of the channels, ILD. If a channel has higher amplitude, it will be considered as leading channel and will be less affected by the phase rotation than the channel with lower amplitude.

5. Sum-Difference and Side Signal Coding

The sum difference transformation is performed on the time and phase aligned spectra of the two channels in a way that the energy is conserved in the Mid signal.

{M⁡(f)=(L′(f)+R′(f))·a·12S⁡(f)=(L′(f)-R′(f))·a·12
where

a=L′⁢2+R′⁢2(L′+R′)2
is bounded between 1/1.2 and 1.2, i.e. −1.58 and +1.58 dB. The limitation avoids artefact when adjusting the energy of M and S. It is worth noting that this energy conservation is less important when time and phase were beforehand aligned. Alternatively the bounds can be increased or decreased.

The side signal S is further predicted with M:
S′(f)=S(f)−g(ILD)M(f)
where

g⁡(ILD)=c-1c+1,
where c=10ILDi[b]/20. Alternatively the optimal prediction gain g can be found by minimizing the Mean Square Error (MSE) of the residual and ILDs deduced by the previous equation.

The residual signal S′(f) can be modeled by two means: either by predicting it with the delayed spectrum of M or by coding it directly in the MDCT domain in the MDCT domain.

6. Stereo Decoding

The Mid signal X and Side signal S are first converted to the left and right channels L and R as follows:
Li[k]=Mi[k]+gMi[k], for band_limits[b]≤k<band_limits[b+1],
Ri[k]=Mi[k]−gMi[k], for band_limits[b]≤k<band_limits[b+1],
where the gain g per parameter band is derived from the ILD parameter:

g=c-1c+1,
where c=10ILDi[b]/20.

For parameter bands below cod_max_band, the two channels are updated with the decoded Side signal:
Li[k]=Li[k]+cod_gaini·Si[k], for 0≤k<band_limits[cod_max_band],
Ri[k]=Ri[k]−cod_gaini·Si[k], for 0≤k<band_limits[cod_max_band],

For higher parameter bands, the side signal is predicted and the channels updated as:
Li[k]=Li[k]+cod_predi[b]·Mi-1[k], for band_limits[b]≤k<band_limits[b+1],
Ri[k]=Ri[k]−cod_predi[b]·Mi-1[k], for band_limits[b]≤k<band_limits[b+1],

Finally, the channels are multiplied by a complex value aiming to restore the original energy and the inter-channel phase of the stereo signal:

Li[k]=a·ej⁢2⁢π⁢β·Li[k]⁢Ri[k]=a·ej⁢2⁢π⁢β-IPDi[b]·Ri[k]⁢where⁢a=2·∑k=band⁢_⁢limits[b]band⁢_⁢limits[b+1]Mi2[k]∑k=band⁢_⁢limits[b]band⁢_⁢limits[b+1]-1Li2[k]+∑k=band⁢_⁢limits[b]band⁢_⁢limits[b+1]-1Ri2[k]
where a is defined and bounded as defined previously, and where β=atan 2(sin(IPDi[b]), cos(IPDi[b])+c), and where atan 2(x,y) is the four-quadrant inverse tangent of x over y.

Finally, the channels are time shifted either in time or in frequency domain depending of the transmitted ITDs. The time domain channels are synthesized by inverse DFTs and overlap-adding.

Specific features of the invention relate to the combination of spatial cues and sum-difference joint stereo coding. Specifically, the spatial cues IDT and IPD are computed and applied on the stereo channels (left and right). Furthermore, sum-difference (M/S signals) are calculated and advantageously a prediction is applied of S with M.

On the decoder-side, the broadband and narrowband spatial cues are combined together with sum-different joint stereo coding. In particular, the side signal is predicted with the mid-signal using at least one spatial cue such as ILD and an inverse sum-difference is calculated for getting the left and right channels and, additionally, the broadband and the narrowband spatial cues are applied on the left and right channels.

Advantageously, the encoder has a window and overlap-add with respect to the time aligned channels after processing using the ITD. Furthermore, the decoder additionally has a windowing and overlap-add operation of the shifted or de-aligned versions of the channels after applying the inter-channel time difference.

The computation of the inter-channel time difference with the GCC-Phat method is a specifically robust method.

The new procedure is advantageous known technology since is achieves bit-rate coding of stereo audio or multi-channel audio at low delay. It is specifically designed for being robust to different natures of input signals and different setups of the multichannel or stereo recording. In particular, the present invention provides a good quality for low bit rate stereo speech coding.

The procedures find use in the distribution of broadcasting of all types of stereo or multichannel audio content such as speech and music alike with constant perceptual quality at a given low bit rate. Such application areas are a digital radio, internet streaming or audio communication applications.

While this invention has been described in terms of several embodiments, there are alterations, permutations, and equivalents which fall within the scope of this invention. It should also be noted that there are many alternative ways of implementing the methods and compositions of the present invention. It is therefore intended that the following appended claims be interpreted as including all such alterations, permutations and equivalents as fall within the true spirit and scope of the present invention.

Although some aspects have been described in the context of an apparatus, it is clear that these aspects also represent a description of the corresponding method, where a block or device corresponds to a method step or a feature of a method step. Analogously, aspects described in the context of a method step also represent a description of a corresponding block or item or feature of a corresponding apparatus. Some or all of the method steps may be executed by (or using) a hardware apparatus, like for example, a microprocessor, a programmable computer or an electronic circuit. In some embodiments, some one or more of the most important method steps may be executed by such an apparatus.

The inventive encoded image signal can be stored on a digital storage medium or can be transmitted on a transmission medium such as a wireless transmission medium or a wired transmission medium such as the Internet.

Depending on certain implementation requirements, embodiments of the invention can be implemented in hardware or in software. The implementation can be performed using a digital storage medium, for example a floppy disk, a DVD, a Blu-Ray, a CD, a ROM, a PROM, an EPROM, an EEPROM or a FLASH memory, having electronically readable control signals stored thereon, which cooperate (or are capable of cooperating) with a programmable computer system such that the respective method is performed. Therefore, the digital storage medium may be computer readable.

Some embodiments according to the invention comprise a data carrier having electronically readable control signals, which are capable of cooperating with a programmable computer system, such that one of the methods described herein is performed.

Generally, embodiments of the present invention can be implemented as a computer program product with a program code, the program code being operative for performing one of the methods when the computer program product runs on a computer. The program code may for example be stored on a machine readable carrier.

Other embodiments comprise the computer program for performing one of the methods described herein, stored on a machine readable carrier.

In other words, an embodiment of the inventive method is, therefore, a computer program having a program code for performing one of the methods described herein, when the computer program runs on a computer.

A further embodiment of the inventive methods is, therefore, a data carrier (or a digital storage medium, or a computer-readable medium) comprising, recorded thereon, the computer program for performing one of the methods described herein. The data carrier, the digital storage medium or the recorded medium are typically tangible and/or non-transitionary.

A further embodiment of the inventive method is, therefore, a data stream or a sequence of signals representing the computer program for performing one of the methods described herein. The data stream or the sequence of signals may for example be configured to be transferred via a data communication connection, for example via the Internet.

A further embodiment comprises a processing means, for example a computer, or a programmable logic device, configured to or adapted to perform one of the methods described herein.

A further embodiment comprises a computer having installed thereon the computer program for performing one of the methods described herein.

A further embodiment according to the invention comprises an apparatus or a system configured to transfer (for example, electronically or optically) a computer program for performing one of the methods described herein to a receiver. The receiver may, for example, be a computer, a mobile device, a memory device or the like. The apparatus or system may, for example, comprise a file server for transferring the computer program to the receiver.

In some embodiments, a programmable logic device (for example a field programmable gate array) may be used to perform some or all of the functionalities of the methods described herein. In some embodiments, a field programmable gate array may cooperate with a microprocessor in order to perform one of the methods described herein. Generally, the methods are advantageously performed by any hardware apparatus.

The apparatus described herein may be implemented using a hardware apparatus, or using a computer, or using a combination of a hardware apparatus and a computer.

The methods described herein may be performed using a hardware apparatus, or using a computer, or using a combination of a hardware apparatus and a computer.

While this invention has been described in terms of several embodiments, there are alterations, permutations, and equivalents which fall within the scope of this invention. It should also be noted that there are many alternative ways of implementing the methods and compositions of the present invention. It is therefore intended that the following appended claims be interpreted as including all such alterations, permutations and equivalents as fall within the true spirit and scope of the present invention.

REFERENCES

[1] Patent application. “Apparatus and Method for Estimating an Inter-Channel Time Difference.” International Application Number PCT/EP2017/051214.[2] Knapp, Charles, and Glifford Carter. “The generalized correlation method for estimation of time delay.” IEEE Transactions on Acoustics, Speech, and Signal Processing 24.4 (1976): 320-327.[3] Zhang, Cha, Dinei Flore̊ncio, and Zhengyou Zhang. “Why does PHAT work well in low-noise, reverberative environments?” Acoustics, Speech and Signal Processing, 2008. ICASSP 2008. IEEE International Conference on. IEEE, 2008.[4] Rabinkin, Daniel V., et al. “DSP implementation of source location using microphone arrays.” Advanced signal processing algorithms, architectures, and implementations VI. Vol. 2846. International Society for Optics and Photonics, 1996.[5] Shen, Miao, and Hong Liu. “A modified cross power-spectrum phase method based on microphone array for acoustic source localization.” Systems, Man and Cybernetics, 2009. SMC 2009. IEEE International Conference on. IEEE, 2009.[6] 3GPP TS 26.445; Codec for Enhanced Voice Services (EVS); Detailed algorithmic description.