Patent ID: 12238509

DETAILED DESCRIPTION OF THE INVENTION

Embodiments of the inventive approach for processing an audio signal in accordance with a room impulse response and for determining in a room impulse response a transition from early reflections to late reverberation will be described. The following description will start with a system overview of a 3D audio codec system in which the inventive approach may be implemented.

FIGS.1and2show the algorithmic blocks of a 3D audio system in accordance with embodiments. More specifically,FIG.1shows an overview of a 3D audio encoder100. The audio encoder100receives at a pre-renderer/mixer circuit102, which may be optionally provided, input signals, more specifically a plurality of input channels providing to the audio encoder100a plurality of channel signals104, a plurality of object signals106and corresponding object metadata108. The object signals106processed by the pre-renderer/mixer102(see signals110) may be provided to a SAOC encoder112(SAOC=Spatial Audio Object Coding). The SAOC encoder112generates the SAOC transport channels114provided to an USAC encoder116(USAC=Unified Speech and Audio Coding). In addition, the signal SAOC-SI118(SAOC-SI=SAOC side information) is also provided to the USAC encoder116. The USAC encoder116further receives object signals120directly from the pre-renderer/mixer as well as the channel signals and pre-rendered object signals122. The object metadata information108is applied to a OAM encoder124(OAM=object metadata) providing the compressed object metadata information126to the USAC encoder. The USAC encoder116, on the basis of the above mentioned input signals, generates a compressed output signal mp4, as is shown at128.

FIG.2shows an overview of a 3D audio decoder200of the 3D audio system. The encoded signal128(mp4) generated by the audio encoder100ofFIG.1is received at the audio decoder200, more specifically at an USAC decoder202. The USAC decoder202decodes the received signal128into the channel signals204, the pre-rendered object signals206, the object signals208, and the SAOC transport channel signals210. Further, the compressed object metadata information212and the signal SAOC-SI214is output by the USAC decoder202. The object signals208are provided to an object renderer216outputting the rendered object signals218. The SAOC transport channel signals210are supplied to the SAOC decoder220outputting the rendered object signals222. The compressed object meta information212is supplied to the OAM decoder224outputting respective control signals to the object renderer216and the SAOC decoder220for generating the rendered object signals218and the rendered object signals222. The decoder further comprises a mixer226receiving, as shown inFIG.2, the input signals204,206,218and222for outputting the channel signals228. The channel signals can be directly output to a loudspeaker, e.g., a 32 channel loudspeaker, as is indicated at230. The signals228may be provided to a format conversion circuit232receiving as a control input a reproduction layout signal indicating the way the channel signals228are to be converted. In the embodiment depicted inFIG.2, it is assumed that the conversion is to be done in such a way that the signals can be provided to a 5.1 speaker system as is indicated at234. Also, the channel signals228may be provided to a binaural renderer236generating two output signals, for example for a headphone, as is indicated at238.

In an embodiment of the present invention, the encoding/decoding system depicted inFIGS.1and2is based on the MPEG-D USAC codec for coding of channel and object signals (see signals104and106). To increase the efficiency for coding a large amount of objects, the MPEG SAOC technology may be used. Three types of renderers may perform the tasks of rendering objects to channels, rendering channels to headphones or rendering channels to a different loudspeaker setup (seeFIG.2, reference signs230,234and238). When object signals are explicitly transmitted or parametrically encoded using SAOC, the corresponding object metadata information108is compressed (see signal126) and multiplexed into the 3D audio bitstream128.

The algorithm blocks of the overall 3D audio system shown inFIGS.1and2will be described in further detail below.

The pre-renderer/mixer102may be optionally provided to convert a channel plus object input scene into a channel scene before encoding. Functionally, it is identical to the object renderer/mixer that will be described below. Pre-rendering of objects may be desired to ensure a deterministic signal entropy at the encoder input that is basically independent of the number of simultaneously active object signals. With pre-rendering of objects, no object metadata transmission is required. Discrete object signals are rendered to the channel layout that the encoder is configured to use. The weights of the objects for each channel are obtained from the associated object metadata (OAM).

The USAC encoder116is the core codec for loudspeaker-channel signals, discrete object signals, object downmix signals and pre-rendered signals. It is based on the MPEG-D USAC technology. It handles the coding of the above signals by creating channel—and object mapping information based on the geometric and semantic information of the input channel and object assignment. This mapping information describes how input channels and objects are mapped to USAC-channel elements, like channel pair elements (CPEs), single channel elements (SCEs), low frequency effects (LFEs) and quad channel elements (QCEs) and CPEs, SCEs and LFEs, and the corresponding information is transmitted to the decoder. All additional payloads like SAOC data114,118or object metadata126are considered in the encoder's rate control. The coding of objects is possible in different ways, depending on the rate/distortion requirements and the interactivity requirements for the renderer. In accordance with embodiments, the following object coding variants are possible:Pre-rendered objects: Object signals are pre-rendered and mixed to the 22.2 channel signals before encoding. The subsequent coding chain sees 22.2 channel signals.Discrete object waveforms: Objects are supplied as monophonic waveforms to the encoder. The encoder uses single channel elements (SCEs) to transmit the objects in addition to the channel signals. The decoded objects are rendered and mixed at the receiver side. Compressed object metadata information is transmitted to the receiver/renderer.Parametric object waveforms: Object properties and their relation to each other are described by means of SAOC parameters. The downmix of the object signals is coded with the USAC. The parametric information is transmitted alongside. The number of downmix channels is chosen depending on the number of objects and the overall data rate. Compressed object metadata information is transmitted to the SAOC renderer.

The SAOC encoder112and the SAOC decoder220for object signals may be based on the MPEG SAOC technology. The system is capable of recreating, modifying and rendering a number of audio objects based on a smaller number of transmitted channels and additional parametric data, such as OLDs, IOCs (Inter Object Coherence), DMGs (DownMix Gains). The additional parametric data exhibits a significantly lower data rate than may be used for transmitting all objects individually, making the coding very efficient. The SAOC encoder112takes as input the object/channel signals as monophonic waveforms and outputs the parametric information (which is packed into the 3D-Audio bitstream128) and the SAOC transport channels (which are encoded using single channel elements and are transmitted). The SAOC decoder220reconstructs the object/channel signals from the decoded SAOC transport channels210and the parametric information214, and generates the output audio scene based on the reproduction layout, the decompressed object metadata information and optionally on the basis of the user interaction information.

The object metadata codec (see OAM encoder124and OAM decoder224) is provided so that, for each object, the associated metadata that specifies the geometrical position and volume of the objects in the 3D space is efficiently coded by quantization of the object properties in time and space. The compressed object metadata cOAM126is transmitted to the receiver200as side information.

The object renderer216utilizes the compressed object metadata to generate object waveforms according to the given reproduction format. Each object is rendered to a certain output channel according to its metadata. The output of this block results from the sum of the partial results. If both channel based content as well as discrete/parametric objects are decoded, the channel based waveforms and the rendered object waveforms are mixed by the mixer226before outputting the resulting waveforms228or before feeding them to a postprocessor module like the binaural renderer236or the loudspeaker renderer module232.

The binaural renderer module236produces a binaural downmix of the multichannel audio material such that each input channel is represented by a virtual sound source. The processing is conducted frame-wise in the QMF (Quadrature Mirror Filterbank) domain, and the binauralization is based on measured binaural room impulse responses.

The loudspeaker renderer232converts between the transmitted channel configuration228and the desired reproduction format. It may also be called “format converter”. The format converter performs conversions to lower numbers of output channels, i.e., it creates downmixes.

FIG.3shows an example for implementing a format converter232. The format converter232, also referred to as loudspeaker renderer, converts between the transmitter channel configuration and the desired reproduction format. The format converter232performs conversions to a lower number of output channels, i.e., it performs a downmix (DMX) process240. The downmixer240, which advantageously operates in the QMF domain, receives the mixer output signals228and outputs the loudspeaker signals234. A configurator242, also referred to as controller, may be provided which receives, as a control input, a signal246indicative of the mixer output layout, i.e., the layout for which data represented by the mixer output signal228is determined, and the signal248indicative of the desired reproduction layout. Based on this information, the controller242, advantageously automatically, generates optimized downmix matrices for the given combination of input and output formats and applies these matrices to the downmixer240. The format converter232allows for standard loudspeaker configurations as well as for random configurations with non-standard loudspeaker positions.

FIG.4illustrates an embodiment of the binaural renderer236ofFIG.2. The binaural renderer module may provide a binaural downmix of the multichannel audio material. The binauralization may be based on a measured binaural room impulse response. The room impulse response may be considered a “fingerprint” of the acoustic properties of a real room. The room impulse response is measured and stored, and arbitrary acoustical signals can be provided with this “fingerprint”, thereby allowing at the listener a simulation of the acoustic properties of the room associated with the room impulse response. The binaural renderer236may be programmed or configured for rendering the output channels into two binaural channels using head related transfer functions or binaural room impulse responses (BRIR). For example, for mobile devices binaural rendering is desired for headphones or loudspeakers attached to such mobile devices. In such mobile devices, due to constraints it may be useful to limit the decoder and rendering complexity. In addition to omitting decorrelation in such processing scenarios, it may be advantageous to first perform a downmix using a downmixer250to an intermediate downmix signal252, i.e., to a lower number of output channels which results in a lower number of input channel for the actual binaural converter254. For example, a 22.2 channel material may be downmixed by the downmixer250to a 5.1 intermediate downmix or, alternatively, the intermediate downmix may be directly calculated by the SAOC decoder220inFIG.2in a kind of a “shortcut” mode. The binaural rendering then only has to apply ten HRTFs (Head Related Transfer Functions) or BRIR functions for rendering the five individual channels at different positions in contrast to applying 44 HRTF or BRIR functions if the 22.2 input channels were to be directly rendered. The convolution operations that may be used for the binaural rendering involve a lot of processing power and, therefore, reducing this processing power while still obtaining an acceptable audio quality is particularly useful for mobile devices. The binaural renderer236produces a binaural downmix238of the multichannel audio material228, such that each input channel (excluding the LFE channels) is represented by a virtual sound source. The processing may be conducted frame-wise in QMF domain. The binauralization is based on measured binaural room impulse responses, and the direct sound and early reflections may be imprinted to the audio material via a convolutional approach in a pseudo-FFT domain using a fast convolution on-top of the QMF domain, while late reverberation may be processed separately.

FIG.5shows an example of a room impulse response h(t)300. The room impulse response comprises three components, the direct sound301, early reflections302and late reverberation304. Thus, the room impulse response describes the reflection behavior of an enclosed reverberant acoustic space when an impulse is played. The early reflections302are discrete reflections with increasing density, and the part of the impulse response where the individual reflections can no longer be discriminated is called late reverberation304. The direct sound301can be easily identified in the room impulse response and can be separated from early reflections, however, the transition from the early reflection302to late reverberation304is less obvious.

In the following embodiments of the inventive approach will be described in further detail. In accordance with embodiments of the invention, an audio signal is separately processed with an early part and a late reverberation of a room impulse response. The audio signal processed with the early part of the room impulse response and the reverberated signal are combined and output as the output audio signal. For the separate processing the transition in the room impulse response from the early part to the late reverberation needs to be known. The transition is determined by a correlation measure that reaches a threshold, wherein the threshold is set dependent on the correlation measure for a selected one of the early reflections in the early part of the room impulse response. The correlation measure may describe with regard to the room impulse response the similarity of the decay in acoustic energy including the initial state and the decay in acoustic energy starting at any time following the initial state over a predefined frequency range.

In accordance with embodiments, the separate processing of the audio signal comprises processing the audio signal with the early reflection part301,302of the room impulse response during a first process, and processing the audio signal with the diffuse reverberation304of the room impulse response during a second process that is different and separate from the first process. Changing from the first process to the second process occurs at the transition time. In accordance with further embodiments, in the second process the diffuse (late) reverberation304may be replaced by a synthetic reverberation. In this case the room impulse response provided may contain only the early reflection part301,302(seeFIG.5) and the late diffuse reverberation304is not included.

FIG.6(A)shows a block diagram illustrating a first exemplary signal processing unit for separately processing an audio signal with an early part and a late reverberation of the room impulse in accordance with an embodiment of the invention. The processing of the audio signal in accordance with different parts of the room impulse response may be carried out in a binaural renderer236that has been described above. The audio input signal400may be a non-reverberant audio material, e.g. a multichannel audio input signal, that is convolved with the room impulse response, for example a room impulse response measured using an artificial head or in-ear microphones. This convolution allows to gain a spatial impression of the original non-reverberant audio material as if the audio material is listened to in the room associated with room impulse response. For example, in the above mentioned binaural renderer236, it may be desired to process the audio signal with the direct sound301and the early reflection302in the room impulse response and to process the audio signal with the late reverberation304separately. For processing the audio input signal400, a block402for direct sound processing, a block404for early reflections processing and a block406for late reverberation processing are provided. The output signals408and410of the respective blocks402to406are combined by a first adder412for generating an early processed signal414. The early processed signal414and the reverberated signal416provided by processor406are combined by a second adder418for generating the audio output signal420which provides to a listener the impression as if the audio signal is listened to in the room associated with the room impulse responses.

Processing the late reverberation302separate from the direct sound and early reflections is advantageous due to the reduced computational complexity. More specifically, using a convolution for the entire impulse response is computationally very costly. Therefore, reverberation algorithms with lower complexity are typically used to process audio signals in order to simulate late reverberation. The direct sound and early reflections part of the impulse response are computed more accurately, for example by a convolution. A further advantage is the possibility of reverberation control. This allows the late reverberation to be modified dependent, for example, on a user input, a measured room parameter or dependent on the contents of the audio signal. To achieve the above advantages the transition (e.g., the point in time) where the early reflections302end and where the late reverberation304starts needs to be known. When the late reverberation processing starts too early, the audio signal may be of lower quality as the human hearing can detect the missing distinct early reflections. On the other hand, if the transition time is detected too late, the computational efficiency will not be exploited, as the early reflections processing is typically more costly than the late reverberation processing. The transition, e.g., in time domain samples, may be fed to the binaural renderer as an input parameter which will then, dependent on the received transition, control the processors402to406for separately processing the audio signal.

FIG.6(B)illustrates a block diagram of another exemplary signal processing unit for separately processing an audio signal with an early part and a late reverberation of the room impulse in accordance with another embodiment of the invention. The input signal400, for example a multichannel audio input signal, is received and applied to a first processor422for processing the early part, namely for processing the audio signal in accordance with the direct sound301and the early reflections302in the room impulse response300shown inFIG.5. The multichannel audio input signal400is also applied to a second processor424for processing the audio signal in accordance with the late reverberation304of the room impulse response. In a binaural renderer, as mentioned above, it may be desired to process the direct sound and early reflections separate from the late reverberation, mainly because of the reduced computational complexity. The processing of the direct sound and early reflections may, for example, be imprinted to the audio signal by a convolutional approach carried out by the first processor422, while the late reverberation may be replaced by a synthetic reverberation provided by the second processor424. The overall binaural output signal420is then a combination of the convolutional result428provided by the processor422and the synthetic reverberated signal430provided by the processor424. In accordance with embodiments the signals428and430are combined by an adder432outputting the overall binaural output signal420.

As mentioned, the first processor422may cause a convolution of the audio input signal400with a direct sound and early reflections of the room impulse response that may be provided to the first processor422from an external database434holding a plurality of recorded binaural room impulse responses. The second processor or reverberator424may operate on the basis of reverberator parameters, like the reverberation RT60 and the reverberation energy, that may be obtained from the stored binaural room impulse responses by an analysis436. It is noted that the analysis436is not necessarily part of the renderer, rather this is to indicate that from the respective responses stored in database434the respective reverberation parameters may be derived; this may be done externally. The reverberator parameters may be determined, for example, by calculating the energy and the RT60 reverberation time in an octave or one-third octave filterbank analysis, or may be mean values of the results of multiple impulse response analyses.

In addition, both processors422and424receive from the database434—directly or via the analysis436—as input parameter also information about the transition in the room impulse response from the early part to the late reverberation. The transition may be determined in a way as will be described in further detail below.

In accordance with embodiments, the transition analysis may be used to separate the early reflections and the late reverberation. It may be fed to the binaural renderer as an input parameter (e.g., it may be read from a dedicated file/interface along with RT60-values and energy values that are used to configure the reverberator). The analysis may be based on one set of binaural room impulse responses (a set of BRIR pairs for a multitude of azimuth and elevation angles). The analysis may be a preprocessing step that is carried out separately for every impulse response and then the median of all transition values is taken as an overall transition value of the one BRIR set. This overall transition value may then be used to separate the early reflections from the late reverberation in the calculation of the binaural output signal.

Several approaches for determining the transition are known, however, these approaches are disadvantages as will be described now. In conventional-technology reference [1] a method is described which uses the energy decay relief (EDR) and a correlation measure to determine the transition time from early reflections to late reverberation. However, the approach described in conventional-technology reference [1] is disadvantageous.1. The approach is strongly dependent on the azimuthal angle of the binaural impulse response and the relation between the amplitudes of direct sound and first impinging reflection.2. The transition time is calculated in arbitrary frequency bands. There is no general knowledge about which of the frequency bands gives the right transition time to be used for the overall impulse response.3. There is no information about the essential correlation step of the approach.

Another known approach is to describe early reflections by the dispersion of echoes in a space, for example by the average number of reflections per second, and to determine the beginning of the late reverberation when this number exceeds a predefined threshold (see conventional-technology reference [2]). This approach relies on the room characteristic, namely the room volume, which is often unknown. The room volume cannot be easily extracted from a measured impulse response. Therefore, this method is not applicable for the calculation of the transition from measured impulse responses. Also, there is no common knowledge how dense the reflections have to be to be called late reverberation.

Another possibility, described in conventional-technology reference [3], is to compare the actual distribution at a time in an impulse response window to a Gaussian distribution in the time domain. The late reverberation is assumed to have a normal distribution. In a normal distribution approximately one third (exactly 1/e) of the samples lie outside one standard deviation of the mean and two thirds of the samples are within one standard deviation of the mean. Distinct early reflections have more samples within one standard deviation and fewer outside. The ratio of samples outside one standard deviation versus the samples inside one standard deviation may be used to define the transition time. However, the disadvantage of this approach is that the transition is difficult to define with this measure, because the ratio sometimes fluctuates around the threshold. The measure is also strongly dependent on the size and the type of the sliding window in which the ratio is calculated.

Besides the above mentioned approaches, also the Kurtosis (the higher order cumulant of a stochastic signal) may be used to determine the transition time. It rapidly decreases when approaching towards the late part of the impulse response, as is outlined in conventional-technology reference [4]. However, the definition of the threshold for the transition (either use of a rapid decrease or the time when it first reaches zero) is not clear.

There is yet another approach that does not rely on the analysis of a measured impulse response, but on the room volume, as is described in [2]. This approach assumes that the transition time is only dependent on the volume, but it does not take into account the diffusing properties of the boundaries. Therefore, the result can only be an approximation of the transition time and is not as accurate as needed for avoiding the above mentioned disadvantages when not precisely determining the transition time. Further, the volume of a room is often not known and cannot be easily extracted from a measured impulse response.

Other known approaches completely disregard the environment and define the transition time to be simply 80 ms, see for example in conventional-technology reference [5]. This number, however, is totally detached from the room characteristics or a measured impulse response and, therefore, is much too inaccurate for the purpose of separating late reverberation from the reminder of the impulse response.

The present invention, in accordance with embodiments, provides in addition to the improved audio signal processing also an improved approach for determining the transition time between early reflections and late reverberation in a room impulse response yielding a more accurate determination of the transition time. Embodiments, as will be described below, provide a simple and effective possibility to calculate the transition time from a measured impulse response using an FFT analysis.

FIG.7shows a flow diagram of an approach for determining a transition time between early reflections and late reverberation in a room impulse response in accordance with an embodiment of the invention. To determine the transition time from early reflections to late reverberation, in a first step500a time-frequency distribution of the acoustic energy is determined. For example, in accordance with embodiments the energy decay relief (E(t,f), EDR) may be calculated in step500. The EDR can be directly calculated from a measured (e.g., binaural) room impulse response and may be interpreted as a frequency-domain expansion of the commonly used energy decay curve (Schroeder integration, EDC (d)) that shows the remaining energy in the impulse response after a time t. Instead of using the broadband impulse response, the EDR is derived from a time-frequency representation and many different time-frequency representations may be used for this purpose. Once the time-frequency distribution of the acoustic energy has been determined in step500, in step502a correlation measure between the acoustic energy at a time block of the time-frequency distribution and the overall acoustic energy at an initial state is determined. In step504it is determined as to whether the correlation measure reaches a defined threshold (e.g., falls below the defined threshold) or not. If it does not reach the threshold, the method proceeds to step506where the next time block and the distribution following the current time block is selected and steps502and504are repeated for the next time block. Thus, in accordance with steps502to506a correlation measure is used to calculate the correlation value between each time block of the EDR determined in step500with the overall energy at the initial state. The transition time is reached when the correlation measure reaches the defined threshold (e.g., falls below the defined threshold). In other words, when it is determined in step504that for a current time block the correlation measure is lower than the threshold, the method proceeds to step508where the time of the current time block is output as the transition time.

In the following, an embodiment of the inventive approach will be described in further detail. Initially, a measured binaural impulse response may be taken as an input for the calculation of the transition time. Then, a Page or Levin distribution is employed for the calculation of the energy decay relief (EDR). The Page distribution refers to the derivative of the past running spectrum and the Page distribution of the time-reverse signal is called the Levin distribution (see also conventional-technology reference [2]). This distribution describes an instantaneous power spectrum, and the EDR of the impulse response h(t) (see, for example,FIG.5) is calculated as follows:
E(t,ω)=|∫τ∞h(τ)e−jωτdτ|whereE(t,ω)=energy decay relief,h(τ)=room impulse response,ω=2πf.

The calculation in accordance with the above equation starts at the direct sound301(seeFIG.5), and with increasing time the energy decay relief contains less distinct reflections and more stochastic reverberation. In accordance with the described embodiment, the energy decay relief is calculated for time blocks having a length of 1 ms for ease of computation. By means of the above described functionality, the time-frequency distribution of the acoustic energy is determined as has been described with regard to step500inFIG.7.

Following this, as has been described with regard to steps502to506inFIG.7, the correlation measure ρ(t) that is based on the Pearson's Product-Moment Correlation (also known as correlation coefficient) is determined. More specifically, the correlation of the acoustic energy for each time block with the overall energy at the initial state is determined, in accordance with embodiments, as follows:

ρ⁡(t)=∑ω⁢(E⁡(1,ω)-E¯(1,ω))·∑ω⁢(E⁡(t,ω)-E¯(t,ω))∑ω⁢(E⁡(1,ω)-E¯(1,ω))2·∑ω⁢(E⁡(t,ω)-E¯(t,ω))2whereE(1,ω)=full frequency range energy decay relief at frequency f,Ē(1,ω)=mean value over all frequencies of the initial full range energy decay relief,E(t,ω)=energy decay relief at frequency f starting a time t,Ē(t,ω)=mean value over all frequencies of the full range energy decay relief starting at time t,ω=2πf.

The above correlation describes the similarity of the decay including the initial state and the decay starting at any time t. It is calculated from the broadband EDR, using the full frequency range of the EDR for the calculation, thereby comparing the complete initial energetic situation with the situation at the time t.

The present invention is not limited to the calculation of the correlation over all frequencies. Rather, the correlation may also be calculated over a predefined frequency range. The frequency range may be determined from the audio signal to be processed. For example, for specific audio signals the frequency range may be limited to a predefined range, e.g., the range of audible frequencies. In accordance with embodiments, the frequency range may be 20 Hz to 20 kHz. It is noted that other ranges may also be selected, e.g. by empirical studies.

In accordance with an embodiment, an effective FFT-based implementation of the EDR may be used. A window having an effective length of the measured impulse response is applied, and it is assumed that a measured impulse response has an effective length of 213which is equal to 8192 frequency bins. During the calculation, this window is shifted by the discrete length of a single time block, and the end of the window is zero-padded. In accordance with embodiments a time block length of 1 ms is used, and for a simple and effective calculation of the EDR the following approach is applied:(1) The whole effective length of the measured impulse response is taken to calculate the FFT spectrum, and the absolute values are squared yielding E(1,ω).(2) Until the end of the impulse response is reached, the window is moved by the discrete time-block length of 1 ms towards the end of the impulse response, the windowed samples are zero-padded to the effective length (i.e., those samples beyond the effective length are made zero), and then the FFT spectrum is calculated which yields E(t,ω).

The above approach is advantageous as no additional filter bank or the like is required for the narrow band calculation of the EDR; only a shifting of the window may be used.FIG.8shows an example for an energy decay relief achieved for an impulse response in accordance with the above described FFT-based approach.

As has been described inFIG.7with regard to steps504and508, the correlation determined in the above described way will then be compared to a predefined threshold. The smaller the threshold is, the more the transition time moves towards the end of the impulse response. For example, for binaural impulse responses, if the threshold is chosen to be 1/e≈0.3679 (see also conventional-technology reference [2]), the transition is too early at some azimuthal angles, because the correlation falls below the threshold already before the first reflection occurred or impinged. However, since it is known that the transition time is later than the arrival time of the first reflection, because the first reflection is clearly distinct and can for sure not be the late diffuse reverberation, in accordance with embodiments, the threshold is not defined as a fixed threshold. Rather, in accordance with the inventive approach the threshold is defined such that it is dependent on the correlation at the impinging time of the first reflection. With this definition, it is assured that the first reflection is located before the transition time. In accordance with embodiments, the transition time, as shown in step508, is considered to be reached when the following applies:
ρ(t)=c·ρ(tF)whereρ(tF)=correlation measure for the selected one of the early reflections,tF=time index where the selected one of the early reflections after the direct sound impinges,c=the constant value that is based on 1/e, e being the Euler number.

In accordance with embodiments, the constant value may be 1/e, however, the present invention is not limited to this value. In accordance with embodiments the constant value may be approximated by 1/e, e.g. by rounding or truncating 1/e with respect to a predefined decimal place (see below).

In the described embodiment, tFis the time block index where the first reflection after the direct sound impinges.

FIG.9depicts the transition time determination in accordance with the inventive approach where the threshold is calculated dependent on the impulse response by multiplication of the correlation at the impinging point of the first reflection and a fixed or constant value of 1/e. The amplitude of the room impulse response600is shown over the number of samples, and a first reflection602is also indicated. The waveform604indicates the correlation values obtained by applying equation (2). At606the correlation value at the first reflection is shown which, in the example depicted has a value of 0.58. Also, the conventionally used fixed threshold of 1/e is shown at608. The correlation value606for the first reflection and the original fixed value 1/e are applied to a multiplier610which generates the new threshold that is dependent on the correlation value at the first reflection and, in the described embodiment has a value of 0.21 as is shown at612. Thus, when compared to conventional approaches, the transition point614is moved further towards the right so that all samples following the transition point614are now considered late reverberation304and all samples before are considered early reflection302. It can be seen that the resulting decision time614is more robust. For example, in a binaural room impulse response this means that the calculated transition time is much more stable over the azimuthal angle. This can be seen from a comparison ofFIGS.10and11.FIG.10shows the transition times when applying the approach described in conventional-technology reference [1] for the left channel700and the right channel702for a measured binaural room impulse response using the above described EDC implementation but with a fixed threshold of 1/e. A dependency on the ear and the azimuthal angle is clearly visible as well as the deep dips in the transition time down to less than 10 ms that are due to the fact that the correlation ρ(t) falls below the threshold before the first reflection impinges.FIG.11shows the transition time for the left channel700and the right channel702when calculated in accordance with the inventive approach. It can be seen that the resulting transition time is much less dependent on the ear and the azimuthal angle when compared to the conventional approach explained with regard toFIG.10.

In accordance with embodiments, the transition time is considered to be reached when the correlation falls below or is equal to the threshold value for the first time and does not increase again over the threshold afterwards. The time value that is associated with this sample in the calculated correlation function is the time where the late reverberation of the impulse response is considered to start. In accordance with the inventive approach, the impinging time of the first reflection may be determined by a running kurtosis operator, as is described in conventional-technology reference [6]. Alternatively, the first reflection may be detected by other methods, for example, by a threshold detection or by an attack detection as it is, for example, described in conventional-technology reference [7].

In accordance with embodiments, e−1=0.3679 is used as a value to indicate a low correlation in stochastic processes as is, for example, indicated also in conventional-technology reference [1]. In accordance with embodiments, this value is used with four decimal digits such that e−1is approximated as 0.3679. In accordance with other embodiments also more or less decimal digits may be used and it has been observed that the detected transition time changes accordingly with the deviation from the exact number of e−1. For example, when using value of 0.368 this results only in minimal changes in the transition time of below 1 ms.

In accordance with further embodiments, the impulse response may be band-limited, and in this case, the EDR may be calculated over a limited frequency range and also the correlation may be calculated over the limited frequency range of the EDR. Alternative frequency transforms or filter banks may also be used, for example, approaches operating completely in the FFT domain, thereby saving additional transforms, for example when using FFT based filtering/convolution.

It is noted that in the above description of the embodiments reference has been made to a value of the correlation value for the first reflection. However, other embodiments may use a correlation value calculated for another one of the early reflections.

As mentioned above, the inventive approach, in accordance with embodiments may be used in a binaural processor for binaural processing of audio signals. In the following an embodiment of binaural processing of audio signals will be described. The binaural processing may be carried out as a decoder process converting the decoded signal into a binaural downmix signal that provides a surround sound experience when listened to over headphones.

FIG.12shows a schematic representation of a binaural renderer800for binaural processing of audio signals in accordance with an embodiment of the present invention.FIG.12also provides an overview of the QMF domain processing in the binaural renderer. At an input802the binaural renderer800receives the audio signal to be processed, e.g., an input signal including N channels and 64 QMF bands. In addition the binaural renderer800receives a number of input parameters for controlling the processing of the audio signal. The input parameters include the binaural room impulse response (BRIR)804for 2×N channels and 64 QMF bands, an indication Kmax806of the maximum band that is used for the convolution of the audio input signal with the early reflection part of the BRIRs804, and the reverberator parameters808and810mentioned above (RT60 and the reverberation energy). The binaural renderer800comprises a fast convolution processor812for processing the input audio signal802with the early part of the received BRIRs804. The processor812generates at an output the early processed signal814including two channels and K max QMF bands. The binaural renderer800comprises, besides the early processing branch having the fast convolution processor812, also a reverberation branch including two reverberators816aand816beach receiving as input parameter the RT60 information808and the reverberation energy information810. The reverberation branch further includes a stereo downmix processor818and a correlation analysis processor820both also receiving the input audio signal802. In addition, two gain stages821aand821bare provided between the stereo downmix processor818and the respective reverberators816aand816bfor controlling the gain of a downmixed signal822provided by the stereo downmix processor818. The stereo downmix processor818provides on the basis of the input signal802the downmixed signal822having two bands and 64 QMF bands. The gain of the gain stages821aand821bis controlled by a respective control signals824aand824bprovided by the correlation analysis processor820. The gain controlled downmixed signal is input into the respective reverberators816aand816bgenerating respective reverberated signals826a,826b. The early processed signal814and the reverberated signals826a,826bare received by a mixer828that combines the received signals into the output audio signal830having two channels and 64 QMF bands. In addition, in accordance with the present invention, the fast convolution processor812and the reverberators816aand816breceive an additional input parameter832indicating the transition in the room impulse response804from the early part to the late reverberation determined as discussed above.

The binaural renderer module800(e.g., the binaural renderer236ofFIG.2orFIG.4) has as input802the decoded data stream. The signal is processed by a QMF analysis filterbank as outlined in ISO/IEC 14496-3:2009, subclause 4.6.18.2 with the modifications stated in ISO/IEC 14496-3:2009, subclause 8.6.4.2. The renderer module800may also process QMF domain input data; in this case the analysis filterbank may be omitted. The binaural room impulse responses (BRIRs)804are represented as complex QMF domain filters. The conversion from time domain binaural room impulse responses to the complex QMF filter representation is outlined in ISO/IEC FDIS 23003-1:2006, Annex B. The BRIRs804are limited to a certain number of time slots in the complex QMF domain, such that they contain only the early reflection part301,302(seeFIG.5) and the late diffuse reverberation304is not included. The transition point832from early reflections to late reverberation is determined as described above, e.g., by an analysis of the BRIRs804in a preprocessing step of the binaural processing. The QMF domain audio signals802and the QMF domain BRIRs804are then processed by a bandwise fast convolution812to perform the binaural processing. A QMF domain reverberator816a,816bis used to generate a 2-channel QMF domain late reverberation826a,826b. The reverberation module816a,816buses a set of frequency-dependent reverberation times808and energy values810to adapt the characteristics of the reverberation. The waveform of the reverberation is based on a stereo downmix818of the audio input signal802and it is adaptively scaled821a,821bin amplitude depending on a correlational analysis820of the multi-channel audio signal802. The 2-channel QMF domain convolutional result814and the 2-channel QMF domain reverberation816a,816bare then combined828and finally, two QMF synthesis filter banks compute the binaural time domain output signals830as outlined in ISO/IEC 14496-3:2009, subclause 4.6.18.4.2. The renderer can also produce QMF domain output data; the synthesis filterbank is then omitted.

Definitions

Audio signals802that are fed into the binaural renderer module800are referred to as input signals in the following. Audio signals830that are the result of the binaural processing are referred to as output signals. The input signals802of the binaural renderer module800are audio output signals of the core decoder (see for example signals228inFIG.2). The following variable definitions are used:

NinNumber of input channelsNoutNumber of output channels, Nout= 2MDMXDownmix matrix containing real-valued non-negativedownmixcoefficients (downmix gains). MDMXis ofdimension Nout× NinLFrame length measured in time domain audio samples.vTime domain sample indexnQMF time slot index (subband sample index)LnFrame length measured in QMF time slotsFFrame index (frame number)KNumber of QMF frequency bands, K = 64kQMF band index (1..64)A, B, chChannel indices (channel numbers of channelconfigurations)LtransLength of the BRIR's early reflection part in time domainsamplesLtrans,nLength of the BRIR's early reflection part in QMF time slotsNBRIRNumber of BRIR pairs in a BRIR data setLFFTLength of FFT transform(·)Real part of a complex-valued signal(·)Imaginary part of a complex-valued signalmconvVector that signals which input signal channel belongs towhich BRIR pair in the BRIR data setƒmaxMaximum frequency used for the binaural processingƒmax,decoderMaximum signal frequency that is present in the audiooutput signal of the decoderKmaxMaximum band that is used for the convolution of the audioinput signal with the early reflection part of the BRIRsaDownmix matrix coefficientceq,kBandwise energy equalization factorεNumerical constant, ε = 10−20dDelay in QMF domain time slotsy̆chn',kPseudo-FFT domain signal representation infrequency band kn'Pseudo-FFT frequency indexh̆n',kPseudo-FFT domain representation of BRIR in frequencyband kz̆ch,convn',kPseudo-FFT domain convolution result in frequency band k{circumflex over (z)}ch,convn,kIntermediate signal: 2-channel convolutional result in QMFdomain{circumflex over (z)}ch,revn,kIntermediate signal: 2-channel reverberation in QMFdomainKanaNumber of analysis frequency bands (used for thereverberator)ƒc,anaCenter frequencies of analysis frequency bandsNDMX,actNumber of channels that are downmixed to one channel ofthe stereo downmix and are active in the actual signal frameccorrOverall correlation coefficient for one signal frameccorrA,BCorrelation coefficient for the combination of channels A, BStandard deviation for timeslot n ofcscaleVector of two scaling factor{tilde over (c)}scaleVector of two scaling factor, smoothed over time
Processing

The processing of the input signal is now described. The binaural renderer module operates on contiguous, non-overlapping frames of length L=2048 time domain samples of the input audio signals and outputs one frame of L samples per processed input frame of length L.

(1) Initialization and Preprocessing

The initialization of the binaural processing block is carried out before the processing of the audio samples delivered by the core decoder (see for example the decoder of200inFIG.2) takes place. The initialization consists of several processing steps.

(a) Reading of Analysis Values

The reverberator module816a,816btakes a frequency-dependent set of reverberation times808and energy values810as input parameters. These values are read from an interface at the initialization of the binaural processing module800. In addition the transition time832from early reflections to late reverberation in time domain samples is read. The values may be stored in a binary file written with 32 bit per sample, float values, little-endian ordering. The read values that are needed for the processing are stated in the table below:

Value descriptionNumberDatatypetransition length Ltrans1IntegerNumber of frequency bands Kana1IntegerCenter frequencies fc,anaof frequencyKanaFloatbandsReverberation times RT60 in secondsKanaFloatEnergy values that represent theKanaFloatenergy (amplitude to the power oftwo) of the late reverberation part ofone BRIR
(b) Reading and Preprocessing of BRIRs

The binaural room impulse responses804are read from two dedicated files that store individually the left and right ear BRIRs. The time domain samples of the BRIRs are stored in integer wave-files with a resolution of 24 bit per sample and 32 channels. The ordering of BRIRs in the file is as stated in the following table:

ChannelSpeakernumberlabel1CH_M_L0452CH_M_R0453CH_M_0004CH_LFE15CH_M_L1356CH_M_R1357CH_M_L0308CH_M_R0309CH_M_18010CH_LFE211CH_M_L09012CH_M_R09013CH_U_L04514CH_U_R04515CH_U_00016CH_T_00017CH_U_L13518CH_U_R13519CH_U_L09020CH_U_R09021CH_U_18022CH_L_00023CH_L_L04524CH_L_R04525CH_M_L06026CH_M_R06027CH_M_L11028CH_M_R11029CH_U_L03030CH_U_R03031CH_U_L11032CH_U_R110

If there is no BRIR measured at one of the loudspeaker positions, the corresponding channel in the wave file contains zero-values. The LFE channels are not used for the binaural processing.

As a preprocessing step, the given set of binaural room impulse responses (BRIRs) is transformed from time domain filters to complex-valued QMF domain filters. The implementation of the given time domain filters in the complex-valued QMF domain is carried out according to ISO/IEC FDIS 23003-1:2006, Annex B. The prototype filter coefficients for the filter conversion are used according to ISO/IEC FDIS 23003-1:2006, Annex B, Table B.1. The time domain representation {tilde over (h)}chv[{tilde over (h)}1v. . . {tilde over (h)}NBRIRv] with 1≤v≤Ltransis processed to gain a complex valued QMF domain filter ĥchn,k=[ĥ1n,k. . . ĥNBRIRn,k] with 1≤n≤Ltrans,n.

(2) Audio Signal Processing

The audio processing block of the binaural renderer module800obtains time domain audio samples802for Nininput channels from the core decoder and generates a binaural output signal830consisting of Nout=2 channels.

The processing takes as inputthe decoded audio data802from the core decoder,the complex QMF domain representation of the early reflection part of the BRIR set804, andthe frequency-dependent parameter set808,810,832that is used by the QMF domain reverberator816a,816bto generate the late reverberation826a,826b.
(a) QMF Analysis of the Audio Signal

As the first processing step, the binaural renderer module transforms L=2048 time domain samples of the Nin-channel time domain input signal (coming from the core decoder) [{tilde over (y)}ch,1v. . . {tilde over (y)}ch,Ninv]={tilde over (y)}chvto an Nin-channel QMF domain signal representation802of dimension Ln=32 QMF time slots (slot index n) and K=64 frequency bands (band index k).

A QMF analysis as outlined in ISO/IEC 14496-3:2009, subclause 4.6.18.2 with the modifications stated in ISO/IEC 14496-3:2009, subclause 8.6.4.2. is performed on a frame of the time domain signal {tilde over (y)}chvto gain a frame of the QMF domain signal [ŷch,1n,k. . . ŷch,Ninn,k]={tilde over (y)}chn,kwith 1≤v≤L and 1≤n≤Ln.

(b) Fast Convolution of the QMF Domain Audio Signal and the QMF Domain BRIRs

Next, a bandwise fast convolution812is carried out to process the QMF domain audio signal802and the QMF domain BRIRs804. A FFT analysis may be carried out for each QMF frequency band k for each channel of the input signal802and each BRIR804.

Due to the complex values in the QMF domain one FFT analysis is carried out on the real part of the QMF domain signal representation and one FFT analysis on the imaginary parts of the QMF domain signal representation. The results are then combined to form the final bandwise complex-valued pseudo-FFT domain signal
y̆chn′,kFFT(ŷchn′,k)=FFT((ŷchn′,k))+j·FFT((ŷchn′,k))
and the bandwise complex-valued BRIRs
h̆1n′,kFFT(ĥ1n′,k)=FFT((ĥ1n′,k))+j·FFT((ĥ1n′,k))for the leftear
h̆2n′,k=FFT(ĥ2n′,k)=FFT((ĥ2n′,k))+j·FFT((ĥ2n′,k))for the rightear.

The length of the FFT transform is determined according to the length of the complex valued QMF domain BRIR filters Ltrans,nand the frame length in QMF domain time slots Lnsuch that
LFFT=Ltrans,n+Ln−1.

The complex-valued pseudo-FFT domain signals are then multiplied with the complex-valued pseudo-FFT domain BRIR filters to form the fast convolution results. A vector mconvis used to signal which channel of the input signal corresponds to which BRIR pair in the BRIR data set.

This multiplication is done bandwise for all QMF frequency bands k with 1≤k≤Kmax. The maximum band Kmaxis determined by the QMF band representing a frequency of either 18 kHz or the maximal signal frequency that is present in the audio signal from the core decoder
fmax=min(fmax,decoder,18 kHz).

The multiplication results from each audio input channel with each BRIR pair are summed up in each QMF frequency band k with 1≤k≤Kmaxresulting in an intermediate 2-channel Kmax-band pseudo-FFT domain signal.

z⌣ch,1,convn′,k=∑ch=1ch=Niny⌣ch,chn′,k·h⌣1,mconv[ch]n′,k⁢andz⌣ch,2,convn′,k=∑ch=1ch=Niny⌣ch,chn′,k·h⌣2,mconv[ch]n′,k
are the pseudo-FFT convolution result z̆ch,convn′,k=[z̆ch,1,convn′,k, z̆ch,2,convn′,k] in the QMF domain frequency band k.

Next, a bandwise FFT synthesis is carried out to transform the convolution result back to the QMF domain resulting in an intermediate 2-channel Kmax-band QMF domain signal with LFFTtime slots {circumflex over (z)}ch,convn,k=[{circumflex over (z)}ch,1,convn,k, {circumflex over (z)}ch,2,convn,k] with 1≤n≤LFFTand 1≤k≤Kmax.

For each QMF domain input signal frame with L=32 timeslots a convolution result signal frame with L=32 timeslots is returned. The remaining LFFT−32 timeslots are stored and an overlap-add processing is carried out in the following frame(s).

(c) Generation of Late Reverberation

As a second intermediate signal826a,826ba reverberation signal called {circumflex over (z)}ch,revn,k=[{circumflex over (z)}ch,1,revn,k,{circumflex over (z)}ch,2,revn,k] is generated by a frequency domain reverberator module816a,816b. The frequency domain reverberator816a,816btakes as inputa QMF domain stereo downmix822of one frame of the input signal,a parameter set that contains frequency-dependent reverberation times808and energy values810.

The frequency domain reverberator816a,816breturns a 2-channel QMF domain late reverberation tail.

The maximum used band number of the frequency-dependent parameter set is calculated depending on the maximum frequency.

First, a QMF domain stereo downmix818of one frame of the input signal ŷchn,kis carried out to form the input of the reverberator by a weighted summation of the input signal channels. The weighting gains are contained in the downmix matrix MDMX. They are real-valued and non-negative and the downmix matrix is of dimension Nout×Nin. It contains a non-zero value where a channel of the input signal is mapped to one of the two output channels.

The channels that represent loudspeaker positions on the left hemisphere are mapped to the left output channel and the channels that represent loudspeakers located on the right hemisphere are mapped to the right output channel. The signals of these channels are weighted by a coefficient of 1. The channels that represent loudspeakers in the median plane are mapped to both output channels of the binaural signal. The input signals of these channels are weighted by a coefficient

a=0.7⁢0⁢7⁢1≈12.

In addition, an energy equalization step is performed in the downmix. It adapts the bandwise energy of one downmix channel to be equal to the sum of the bandwise energy of the input signal channels that are contained in this downmix channel. This energy equalization is conducted by a bandwise multiplication with a real-valued coefficient ceq,k=√{square root over (Pink/Poutk+ε)}.

The factor ceq,kis limited to an interval of [0.5, 2]. The numerical constant ε is introduced to avoid a division by zero. The downmix is also bandlimited to the frequency fmax; the values in all higher frequency bands are set to zero.

FIG.13schematically represents the processing in the frequency domain reverberator816a,816bof the binaural renderer800in accordance with an embodiment of the present invention.

In the frequency domain reverberator a mono downmix of the stereo input is calculated using an input mixer900. This is done incoherently applying a 90° phase shift on the second input channel.

This mono signal is then fed to a feedback delay loop902in each frequency band k, which creates a decaying sequence of impulses. It is followed by parallel FIR decorrelators that distribute the signal energy in a decaying manner into the intervals between the impulses and create incoherence between the output channels. A decaying filter tap density is applied to create the energy decay. The filter tap phase operations are restricted to four options to implement a sparse and multiplier-free decorrelator.

After the calculation of the reverberation an inter-channel coherence (ICC) correction904is included in the reverberator module for every QMF frequency band. In the ICC correction step frequency-dependent direct gains gdirectand crossmix gains gcrossare used to adapt the ICC.

The amount of energy and the reverberation times for the different frequency bands are contained in the input parameter set. The values are given at a number of frequency points which are internally mapped to the K=64 QMF frequency bands.

Two instances of the frequency domain reverberator are used to calculate the final intermediate signal {circumflex over (z)}ch,revn,k=[{circumflex over (z)}ch,1,revn,k, {circumflex over (z)}ch,2,revn,k]. The signal {circumflex over (z)}ch,1,revn,kis the first output channel of the first instance of the reverberator, and {circumflex over (z)}ch,2,revn,kis the second output channel of the second instance of the reverberator. They are combined to the final reverberation signal frame that has the dimension of 2 channels, 64 bands and 32 time slots.

The stereo downmix822is both times scaled821a,baccording to a correlation measure820of the input signal frame to ensure the right scaling of the reverberator output. The scaling factor is defined as a value in the interval of [√{square root over (NDMX,act)}, NDMX,act] linearly depending on a correlation coefficient ccorrbetween 0 and 1 with

ccorr=1Nin2·∑A=1A=NDMX.act∑B=1B=NDMX.actccorrA,B⁢andccorrA,B=❘"\[LeftBracketingBar]"1K-1·∑k∑ny^^ch,An,k·y^^ch,Bn,k*∑nσy^^ch,An·σy^^ch,Bn❘"\[RightBracketingBar]"
where σŷchAnmeans the standard deviation across one time slot n of channel A, the operator {*} denotes the complex conjugate and {circumflex over (ŷ)} is the zero-mean version of the QMF domain signal ŷ in the actual signal frame.

ccorris calculated twice: once for all channels A, B that are active at the actual signal frame F and are included in the left channel of the stereo downmix and once for all channels A, B that are active at the actual signal frame F and that are included in the right channel of the stereo downmix.

NDMX,actis the number of input channels that are downmixed to one downmix channel A (number of matrix element in the Ath row of the downmix matrix MDMXthat are unequal to zero) and that are active in the current frame.

The scaling factors then are

cscale=[cscale,1,cscale,2]=[NDMX,act,1+ccorr·(NDMX,act,1-NDMX,act,1),NDMX,act,2+ccorr·(NDMX,act,2-NDMX,act,2)].

The scaling factors are smoothed over audio signal frames by a 1storder low pass filter resulting in smoothed scaling factors {tilde over (c)}scale=[{tilde over (c)}scale,1,{tilde over (c)}scale,2].

The scaling factors are initialized in the first audio input data frame by a time-domain correlation analysis with the same means.

The input of the first reverberator instance is scaled with the scaling factor {tilde over (c)}scale,1and the input of the second reverberator instance is scaled with the scaling factor {tilde over (c)}scale,2.

(d) Combination of Convolutional Results and Late Reverberation

Next, the convolutional result814, {circumflex over (z)}ch,convn,k=[{circumflex over (z)}ch,1,convn,k, {tilde over (z)}ch,2,convn,k], and the reverberator output826a,826b, {circumflex over (z)}ch,revn,k=[{circumflex over (z)}ch,1,revn,k,{circumflex over (z)}ch,2,revn,k], for one QMF domain audio input frame are combined by a mixing process828that bandwise adds up the two signals. Note that the upper bands higher than Kmaxare zero in {circumflex over (z)}ch,convn,kbecause the convolution is only conducted in the bands up to Kmax.

The late reverberation output is delayed by an amount of d=((Ltrans−20·64 +1)/64+0.5)+1 time slots in the mixing process.

The delay d takes into account the transition time from early reflections to late reflections in the BRIRs and an initial delay of the reverberator of 20 QMF time slots, as well as an analysis delay of 0.5 QMF time slots for the QMF analysis of the BRIRs to ensure the insertion of the late reverberation at a reasonable time slot. The combined signal {circumflex over (z)}chn,kat one time slot n calculated by {circumflex over (z)}ch,convn,k+{circumflex over (z)}ch,revn−d,k.

(e) QMF Synthesis of Binaural QMF Domain Signal

One 2-channel frame of 32 time slots of the QMF domain output signal {circumflex over (z)}chn,kis transformed to a 2-channel time domain signal frame with length L by the QMF synthesis according to ISO/IEC 14496-3:2009, subclause 4.6.18.4.2. yielding the final time domain output signal830, {tilde over (z)}chv=[{tilde over (z)}ch,1v. . . {tilde over (z)}ch,2v].

Although some aspects have been described in the context of an apparatus, it is clear that these aspects also represent a description of the corresponding method, where a block or device corresponds to a method step or a feature of a method step. Analogously, aspects described in the context of a method step also represent a description of a corresponding block or item or feature of a corresponding apparatus. Some or all of the method steps may be executed by (or using) a hardware apparatus, like for example, a microprocessor, a programmable computer or an electronic circuit. In some embodiments, some one or more of the most important method steps may be executed by such an apparatus.

Depending on certain implementation requirements, embodiments of the invention can be implemented in hardware or in software. The implementation can be performed using a non-transitory storage medium such as a digital storage medium, for example a floppy disc, a DVD, a Blu-Ray, a CD, a ROM, a PROM, and EPROM, an EEPROM or a FLASH memory, having electronically readable control signals stored thereon, which cooperate (or are capable of cooperating) with a programmable computer system such that the respective method is performed. Therefore, the digital storage medium may be computer readable.

Some embodiments according to the invention comprise a data carrier having electronically readable control signals, which are capable of cooperating with a programmable computer system, such that one of the methods described herein is performed.

Generally, embodiments of the present invention can be implemented as a computer program product with a program code, the program code being operative for performing one of the methods when the computer program product runs on a computer. The program code may, for example, be stored on a machine readable carrier.

Other embodiments comprise the computer program for performing one of the methods described herein, stored on a machine readable carrier.

In other words, an embodiment of the inventive method is, therefore, a computer program having a program code for performing one of the methods described herein, when the computer program runs on a computer.

A further embodiment of the inventive method is, therefore, a data carrier (or a digital storage medium, or a computer-readable medium) comprising, recorded thereon, the computer program for performing one of the methods described herein. The data carrier, the digital storage medium or the recorded medium are typically tangible and/or non-transitionary.

A further embodiment of the invention method is, therefore, a data stream or a sequence of signals representing the computer program for performing one of the methods described herein. The data stream or the sequence of signals may, for example, be configured to be transferred via a data communication connection, for example, via the internet.

A further embodiment comprises a processing means, for example, a computer or a programmable logic device, configured to, or programmed to, perform one of the methods described herein.

A further embodiment comprises a computer having installed thereon the computer program for performing one of the methods described herein.

A further embodiment according to the invention comprises an apparatus or a system configured to transfer (for example, electronically or optically) a computer program for performing one of the methods described herein to a receiver. The receiver may, for example, be a computer, a mobile device, a memory device or the like. The apparatus or system may, for example, comprise a file server for transferring the computer program to the receiver.

In some embodiments, a programmable logic device (for example, a field programmable gate array) may be used to perform some or all of the functionalities of the methods described herein. In some embodiments, a field programmable gate array may cooperate with a microprocessor in order to perform one of the methods described herein. Generally, the methods are advantageously performed by any hardware apparatus.

A further embodiment according to the invention comprises a method for processing an audio signal in accordance with a room impulse response, the method comprising: separately processing the audio signal with an early part and a late reverberation of the room impulse response; and combining the audio signal processed with the early part of the room impulse response and the reverberated signal, wherein a transition from the early part to the late reverberation in the room impulse response is determined by a correlation measure that reaches a threshold, the threshold being set dependent on the correlation measure for a selected one of the early reflections in the early part of the room impulse response.

A further embodiment according to the invention comprises the method of the immediately preceding embodiment, wherein the correlation measure describes with regard to the room impulse response the similarity of the decay in acoustic energy including the initial state and of the decay in acoustic energy starting at any time following the initial state over a predefined frequency range.

A further embodiment according to the invention comprises the method of either of the two immediately preceding embodiments, wherein determining the transition comprises: determining a distribution of acoustic energy based on the room impulse response; and; determining a plurality of correlation measures indicating for a plurality of portions of the determined distribution a correlation between the acoustic energy in the respective portion of the determined distribution and the acoustic energy at an initial state.

A further embodiment according to the invention comprises the method of the immediately preceding embodiment, wherein determining the distribution comprises determining a time-frequency distribution of the acoustic energy, and a portion of the distribution comprises a time block of a predefined length, the initial state being defined by the first one of the plurality of time blocks of the time-frequency distribution.

A further embodiment according to the invention comprises the method of either of the two immediately preceding embodiments, wherein determining the distribution comprises calculating the energy decay relief (EDR) from the room impulse response (300,804).

A further embodiment according to the invention comprises the method of the immediately preceding embodiment, wherein the EDR is calculated as follows:
E(t,ω)=|∫t∞h(τ)e−jωτdτ|2whereE(ζω) energy decay relief,h) room impulse response,ω2% f.

A further embodiment according to the invention comprises the method of any of the three immediately preceding embodiments, wherein the room impulse response has a predefined effective length, and wherein determining the time-frequency distribution comprises calculating the FFT spectrum of the room impulse response using a window having a length corresponding to the effective length of the room impulse response.

A further embodiment according to the invention comprises the method of the immediately preceding embodiment, wherein the acoustic energy at the initial state is determined by taking the whole effective length of the room impulse response (300,804) (300,804), calculating the FFT spectrum and taking the square of the absolute values, and the acoustic energy of a time block is determined by shifting the window by the time associated with the time block, zero-padding the windowed samples to the effective length, calculating the FFT and taking the square of the absolute values.

A further embodiment according to the invention comprises the method of any of the eight immediately preceding embodiments, wherein the correlation measure is calculated as follows:
Σω(E(1,ω)−E(l,ω))▪Σω{E(í,ω)−E{t,ω))
p(=Σω{{1,ω)−E(1,ω))2▪{circumflex over ( )}Σω(E(í,{acute over (ω)})−E(t,ω))′wherep(t)=correlation measure,E(l,ω)=full frequency range energy decay relief at frequency f,E(l,ω)=mean value over all frequencies of the initial full range energy decay relief,E(t,ω)=energy decay relief at frequency f starting a time t,E(t,ω)=mean value over all frequencies of the full range energy decay relief starting at time t,ω=2πí.

A further embodiment according to the invention comprises the method of any of the nine immediately preceding embodiments, wherein the threshold is determined based on a constant value and the correlation measure for the selected one of the early reflections.

A further embodiment according to the invention comprises the method of the immediately preceding embodiment, wherein the constant is 1/e, and wherein the threshold is defined as follows:
p(t)=c▪p(tF)wherep(tF)=correlation measure for the selected one of the early reflections,tF =time index where the selected one of the early reflections after the direct sound impinges,c=the constant value that is based on —, e being the Euler number.

A further embodiment according to the invention comprises the method of any of the eleven immediately preceding embodiments, wherein determining the transition comprises: determining the time of the selected one of the early reflections.

A further embodiment according to the invention comprises the method of the immediately preceding embodiment, wherein the time of the selected one of the early reflections is determined by a running kurtosis operator, by a threshold detection or by an attack detection.

A further embodiment according to the invention comprises the method of any of the thirteen immediately preceding embodiments, wherein the selected one of the early reflections is the first reflection.

A further embodiment according to the invention comprises a non-tangible computer program product comprising a computer readable medium storing instructions which, when executed on a computer, carry out the method of any one of the fourteen preceding embodiments.

A further embodiment according to the invention comprises a signal processing unit, comprising an input for receiving an audio signal; a processor configured to process the received audio signal in accordance with a room impulse response according to the method of any of the fourteen embodiments of the invention that precede the immediately preceding embodiment; and an output for combining the processed early part of the received audio signal and the reverberated signal into an output audio signal.

A further embodiment according to the invention comprises a signal processing unit of the immediately preceding embodiment, comprising: an early part processor for processing the received audio signal in accordance with the early part of the room impulse response; and a late reverberation processor for processing the received audio signal in accordance with the late reverberation of the room impulse response.

A further embodiment according to the invention comprises an audio encoder for encoding an audio signal, wherein the audio encoder is configured to process an audio signal to be encoded in accordance with a room impulse response in accordance with the method of any of the thirteen embodiments that precede the immediately preceding five embodiments of the invention.

A further embodiment according to the invention comprises an audio encoder of the immediately preceding embodiment, wherein the audio encoder comprises a signal processing unit in accordance with any of the two embodiments that precede the immediately preceding embodiment of the invention.

A further embodiment according to the invention comprises an audio decoder for decoding an encoded audio signal, wherein the audio decoder is configured to process a decoded audio signal in accordance with a room impulse response) in accordance with a method of any of claims of the fourteen embodiments immediately preceding the six immediately preceding embodiments.

A further embodiment according to the invention comprises the audio decoder of the immediately preceding embodiment, wherein the audio decoder comprises a signal processing unit of the either of two embodiments that immediately proceed the two immediately preceding embodiments.

A further embodiment according to the invention comprises the audio decoder of either of the two immediately preceding embodiments, comprising a renderer configured to receive the decoded audio signal and to render output signals based on the room impulse response.

A further embodiment according to the invention comprises the audio decoder of the immediately preceding embodiment, wherein the renderer comprises a binaural renderer.

A further embodiment according to the invention comprises a binaural renderer, comprising a signal processing unit either of the two embodiments that immediately precede the six immediately preceding embodiments.

While this invention has been described in terms of several embodiments, there are alterations, permutations, and equivalents which fall within the scope of this invention. It should also be noted that there are many alternative ways of implementing the methods and compositions of the present invention. It is therefore intended that the following appended claims be interpreted as including all such alterations, permutations and equivalents as fall within the true spirit and scope of the present invention.

LITERATURE

[1] T. Hidaka et al: “A new definition of boundary point between early reflections and late reverberation in room impulse responses”. Forum Acusticum, 2005.[2] Jot et al: “Analysis and synthesis of room reverberation based on a statistical time frequency model”.[3] J. S. Abel, P. Huang: “A Simple, Robust Measure of Reverberation Echo Density”. AES Convention, San Francisco, 2006.[4] R. Stewart, M. Sandler: “Statistical Measures of Early Reflections of Room Impulse Responses”. DAFx, 2007.[5] Reilly et al: “Using Auralisation for Creating Animated 3-D Sound Fields Across Multiple Speakers”. AES Convention, New York, 1995.[6] Usher, J.: “An improved method to determine the onset timings of reflections in an acoustic impulse response”. Journal of the Acoustical Society of America, (2010, volume 127) band 4, p. 172-177.[7] Masri, P.: “Computer Modelling of Sound for Transformation and Synthesis of Musical Signals”. PhD thesis, University of Bristol, 1996.