Patent Description:
Parametric spatial audio capture and processing is a field of audio signal capture and processing where the spatial aspect of the sound is described using a set of parameters. For example, in parametric spatial audio capture from microphone arrays, it is a typical and an effective choice to estimate from the microphone array signals a set of parameters such as directions of the sound in frequency bands, and the ratio parameters expressing relative energies of the directional and non-directional parts of the captured sound in frequency bands. These parameters are known to well describe the perceptual spatial properties of the captured sound at the position of the microphone array. These parameters can be utilized in synthesis of the spatial sound accordingly, for headphones binaurally, for loudspeakers, or to otherformats, such as Ambisonics.

The directions and direct-to-total energy ratios in frequency bands are thus a parameterization that is particularly effective for spatial audio capture.

Traditional linear spatial audio capture methods can be applied to high-end arrays (e.g. multi-microphone spheres, or a set of directional microphones). Parametric spatial audio capture methods can be applied to the high-end arrays, and also to more modest arrays, such as those on mobile phones, or small VR cameras, etc..

<CIT>, discusses a sound signal processing method and apparatus. In particular it relates to the audio signal processing field, and can collect and process signals in a three-dimensional sound field surrounding a terminal. The method includes: acquiring, by a mobile terminal, sound signals from a three-dimensional sound field, whereat least three microphones are disposed on the mobile terminal and one microphone is configured to receive a sound signal in at least one direction; acquiring, according to the acquired sound signals, a direction of a sound source relative to the mobile terminal; and obtaining spatial audio signals according to the direction of the sound source relative to the mobile terminal and the acquired sound signals, where the spatial audio signals are used for simulating the three-dimensional sound field.

<NPL>is a chapterwhich discusses methods for factorizing the spectrogram of multichannel audio into repetitive spectral objects and apply the introduced models to the analysis of spatial audio and modification of spatial sound through source separation. The purpose of decomposing an audio spectrogram using spectral templates is to leam the underlying structures (audio objects) from the observed data. The chapter discusses two main scenarios such as parameterization of multichannel surround sound and parameterization of microphone array signals. It explains the principles of source separation by time-frequency filtering using separation masks constructed from the spectrogram models.

<CIT> discloses an apparatus and method to localize multiple sound sources. Virtual microphone signals are generated based on actual microphone signals from a microphone array including a plurality of microphones, which are arranged at intervals that may minimize space aliasing ata given sampling frequency, and sound source directions are tracked using the actual microphone signals and the virtual microphone signals. Thus, without increasing the aperture length of the microphone array, it is possible to achieve almost the same resolution as when a microphone array having a relatively long length is used.

<CIT> discloses apparatus comprising:an audio capture application configured to determine separate microphones from a plurality of microphones and identify a sound source direction of at least one audio source within an audio scene by analysing respective two or more audio signals from the separate microphones. The audio capture application is further configured to adaptively select, from the two or more respective audio signals, a reference audio signal also based on the determined direction; and a signal generator configured to generate a mid signal representing the at least one audio source based on a combination of the selected two or more respective audio signals and with reference to the reference audio signal.

The following describes in further detail suitable apparatus and possible mechanisms for the provision of acoustic capture.

As described above parametric spatial audio capture and processing methods can be used to enable a perceptually accurate spatial sound reproduction based on practical microphone arrangements. Parametric spatial audio capture refers to adaptive DSP-driven audio capture methods covering <NUM>) analysing perceptually relevant parameters in frequency bands, for example, the directionality of the propagating sound at the recording position, and <NUM>) reproducing spatial sound in a perceptual sense at the rendering side according to the estimated spatial parameters. The reproduction can be, for example, for headphones or multichannel loudspeaker setups. By estimating and reproducing the perceptually relevant spatial properties (parameters) of the sound field, a spatial perception similar to that which would occur in the original sound field can be reproduced. As the result, the listener can perceive the multitude of sources, their directions and distances, as well as properties of the surrounding physical space, among the other spatial sound features, as if the listener was in the position of the capture device.

An example video capture device is shown in <FIG>. The device is shown with two cameras <NUM> and <NUM> located on the sides of the cube like apparatus shape. There may furthermore be additional cameras unseen on <FIG> located on the other sides to enable the device <NUM> to capture <NUM>-degree video. In some embodiments the device may be configured to also capture surround or 3D audio, so that the user of the virtual reality (VR) media content may utilize the auditory cues to orient the view towards the direction of interest, even if that direction is not in the field of view. Furthermore a user may require a surround or3D audio reproduced output to produce an overall realistic spatial auditory perception which together with the visual VR reproduction produces an overall immersive experience.

The device <NUM> may be other than a VR camera. For example, the device may be a teleconferencing device, mobile phone, tablet computer. The device in some embodiments may comprise a single camera (or even no cameras at all). In some embodiments the device may comprise an integrated loudspeaker and the device could be of a similar shape as the device in <FIG>.

In order to capture the spatial audio, microphones are located or mounted on the device. For a device such as shown in <FIG> a minimum number of microphones to generate a suitable 3D (azimuth + elevation) audio capture is four. This is because at least some spacing between the microphones of the microphone array is required on all axes. A 2D (azimuth or elevation) audio capture device may employ a three microphone device.

The placement of the microphones may be defined in an attempt to generate the best foundation for the spatial audio capture. To aim for the best performance at low frequencies (because of audio wavelengths with respect to the device size) the microphones should be placed as far apart as possible from each other.

With respect to <FIG> the example device <NUM> is shown with an example placement of four microphones <NUM>, <NUM>, <NUM>, <NUM> in a tetrahedral arrangement. For example a first microphone <NUM> located at a corner of the top, rear and left faces of the device <NUM>, a second microphone <NUM> located at a corner of the top, front and right faces of the device <NUM>, a third microphone <NUM> located at a corner of the bottom, front and left faces of the device <NUM> and a fourth microphone <NUM> located at a corner of the bottom, rear and right faces of the device <NUM>. This arrangement maximizes the size of the microphone array for the device size and therefore performance at low frequencies.

In some embodiments the microphones may not be located at the corners as shown in <FIG> but may be located towards the centre of the device sides on any axis if other aspects of the hardware design have an impact. However, in such examples the array separation is smaller.

In some examples the output signal of a capture device is in an Ambisonics format (e.g. first-order Ambisonics (FOA)). Some VR formats use FOA as the audio format to convey spatial audio. In such examples FOA signals may be synthesized such that.

In other words in some examples a linear reproduction of FOA signals needs also be considered in terms of the microphone placement to avoid biasing of the FOA pattern shapes at some frequencies.

Therefore the device with an example diameter of <NUM>-<NUM> may be designed to have.

An example <NUM>-degree horizontal surround capture device <NUM> and placement of microphones on it is shown in <FIG>. The device is cylindrical in shape with cameras <NUM>, <NUM>, <NUM>, <NUM>, <NUM> located on the side and three microphones <NUM>, <NUM>, <NUM> are placed, in a balanced distribution and as far as possible from each other with angle <NUM>, on the top base of the device. In some examples the microphones <NUM>, <NUM>, <NUM> may be moved towards the centre, which may cause some reduced performance in the low-frequencies, depending on how small the microphone array becomes because of other design requirements. It is understood that the example layout formed by the three microphones provides a two dimensional geometry in <FIG> and any suitable geometry may be formed in other alternative examples where the two dimensional geometry can have a non-uniform shape and the two dimensional geometry does not have to be perfectly aligned with the horizontal plane.

The following examples are described with respect to the devices shown in <FIG>. However, the methods described herein can be applied also to the horizontal surround capture for devices such as that in <FIG> and other shaped 3D and horizontal surround capture arrangements.

The concept is one of optimising capture with the microphone array such as shown in <FIG> and enabling determination of the direction-of-arrival at the frequencies below the spatial aliasing frequency, which could for a typical device size be approximately <NUM>, and also above the spatial aliasing frequency. At these frequencies, the audio wavelength is too short to generate FOA signals or to utilize other analysis techniques that are based on an assumption that the inter-microphone phase may unambiguously contain information about the direction of the arriving sound.

Furthermore the examples may be employed to determine spatial parameters even for product design shapes when the shape does not generate substantial acoustic shadowing at some coordinate axes in respect of microphone layout, such as the horizontal plane as shown in the device in <FIG>. Acoustic shadowing enables the employment of an energy-based method to estimate a direction of arrival based on the microphone signal energies (or magnitudes).

A block diagram of an example system for implementing some examples is shown in <FIG> shows an example system within which the examples may be implemented. The microphone audio signals <NUM> captured by the microphones <NUM>, <NUM>, <NUM>, <NUM> may be stored and later processed, or directly processed.

An analysis processor <NUM> may receive the microphone audio signals <NUM> from the capture device <NUM>. The analysis processor <NUM> can, for example, be a computer, VR camera or a mobile phone (running suitable software), or alternatively a specific device utilizing, for example, field programmable gate arrays (FPGAs) or application specific integrated circuits (ASICs). In some examples the capture device <NUM> and the analysis processor <NUM> are implemented on the same apparatus or device.

Based on the microphone-array signals, the analysis processor creates a data stream comprising spatial metadata <NUM> (e.g., directions <NUM> and energy ratios <NUM> in frequency bands). For example spatial metadata parameters determined include (but are not limited to): Direction and direct-to-total energy ratio; Direction and diffuseness; Inter-channel level difference, inter-channel phase difference, and inter-channel coherence. In some examples these parameters are determined in time-frequency domain. It should be noted that also other parametrizations may be used than those presented above. In general, typically the spatial audio parametrizations describe how the sound is distributed in space, either generally (e.g., using directions) or relatively (e.g., as level differences between certain channels). In the example shown in <FIG> the metadata <NUM> comprises directions <NUM> and energy ratios <NUM>.

Furthermore the microphone array signals <NUM> may be passed to a pre-processor <NUM> (which may be optional). The pre-processor <NUM> may be configured to generate suitable transport audio signals <NUM>. The pre-processor <NUM> may be configured to generate the transport audio signals by selecting an audio signal from the microphone array signals. In some examples the microphone array audio signals may be combined, for example by beamforming methods, to generate the transport audio signals. In some examples the transport audio signals may be obtained by otherwise processing the microphone array signals.

The pre-processor <NUM> may be configured to generate any suitable number of transport audio signals (or channels), for example in some examples the pre-processor <NUM> is configured to generate two transport audio signals. In some examples the pre-processor is further configured to encode the audio signals. For example in some examples the audio signals may be encoded using an advanced audio coding (AAC) or enhanced voice services (EVS) coding. In some examples the pre-processor <NUM> is configured to equalize the audio signals, apply automatic noise control, dynamic processing, or any other suitable processing.

The spatial metadata <NUM> and the transport audio signals <NUM> may be transmitted or stored for example within some storage <NUM> such as memory, or alternatively directly processed in the same device. In some examples the spatial metadata <NUM> and the transport audio signals <NUM> may be encoded or quantized or combined or multiplexed into a single data stream by a suitable encoding and/or multiplexing operation. In some examples the coded audio signal is bundled with a video stream (e.g., <NUM>-degree video) in a media containersuch as an mp4 container, to be transmitted to a receiver.

A synthesis processor <NUM> may receive the spatial metadata <NUM> and the transport audio signals <NUM> (or in some examples the combined data stream). In some examples where a single data stream is received there may be a decoder and/or demultiplexer for decoding the received signal and/or demultiplexing the combined data stream into separate spatial metadata <NUM> and transport audio signals <NUM>. Furthermore where either of the spatial metadata <NUM> and/or the transport audio signals <NUM> are encoded or quantized then the synthesis processor <NUM> or pre-processor may further decode or dequantize the signals.

The synthesis processor <NUM> can, for example, be a computer, a VR playback device or VR camera with playback capability, or a mobile phone (running suitable software), or alternatively a specific device utilizing, for example, FPGAs or ASICs. Based on the data stream (the transport audio signals and the metadata), the synthesis processor <NUM> can be configured to produce output audio signals. For headphone listening, the outputsignals can be binaural signals <NUM>. For loudspeaker rendering, the output signals can be multi-channel loudspeaker signals <NUM>. The outputcan also be Ambisonic signals <NUM>. Ambisonicsignalsarea spatial audiosignal representation that are employed to be decoded (to e.g. binaural)for listening. This typically means processing (using filters and/or mixing) to obtain the loudspeaker or binaural output from the Ambisonic signals.

A typical use case for the example capture devices using the embodiments described herein and where the output is Ambisonics is the following:.

Then, the output Ambisonic signals are used as a spatial audio signal, to be combined with a suitable video data stream (for example a <NUM> video) and uploaded/streamed to a suitable VR service. The Ambisonic signals could in some embodiments be encoded with AAC at a sufficient bit rate, or similar codecs may be used.

The analysis and synthesis processors may be also within the same device, and may be also a part of the same software.

A suitable output form may be passed to headphones or other playback apparatus may be configured to receive the output of the synthesis processor <NUM> and output the audio signals in a format suitable for listening.

With respect to <FIG> is shown an example summary of the operations of the apparatus shown in <FIG>.

The initial operation is receiving (from the capture device directly or otherwise) of the microphone array audio signals as shown in <FIG> by step <NUM>.

The received microphone array audio signals may optionally be processed to generate transport audio signals as shown in <FIG> by step <NUM>.

Also the microphone array audio signals may be analysed to generate the metadata (for example the directions and/or energy ratios) as shown in <FIG> by step <NUM>.

The transport audio signals and metadata may then be optionally combined to form a data stream as shown in <FIG> by step <NUM>.

The transport audio signals and metadata (or the combined data stream) may (optionally) then be transmitted and received (or stored and retrieved) as shown in <FIG> by step <NUM>.

Having received or retrieved the transport audio signals and metadata (or data stream), the output audio signals may be synthesized based at least on the transport audio signals and metadata as shown in <FIG> by step <NUM>.

The synthesized audio signal output signals may then be output to a suitable output.

With respect to <FIG> an example analysis processor <NUM>, such as shown in <FIG>, is presented. The input to the analysis processor <NUM> are the microphone array signals <NUM>.

The analysis processor <NUM> may comprise a time-frequency domain transformer (T-F domain transformer) <NUM>. The forward filter bank <NUM> is configured to transform the wide-band (time domain) signals into time-frequency signals. The output of the filter bank can then be grouped into frequency bands and the signal components for the lower and higher bands can be selected after the filter bank. A lower frequency analyser <NUM> is configured to receive the audio signal frequency bands below the spatial aliasing frequency <NUM> and determine metadata based on the audio signals. The lower frequency analyser <NUM> is configured to determine any suitablespatial metadata parameter. Spatial aliasing frequency is the frequency above which an audio wavelength becomes too small with respect to the array microphone spacing for many traditional methods. Above the spatial aliasing frequency, many traditional methods like linear beamforming fail to produce accurate results.

Any suitable method may be used to determine a direction parameter at the frequencies below the spatial aliasing frequency. For example, a method known as Directional Audio Coding (DirAC) may be implemented which operates on the first order Ambisonics (FOA) signal, or any of its variants.

The FOA signal can be generated by designing and applying appropriate matrix of filters (or matrix of complex gains in frequency bands) for the microphone array signals. Such gains in frequency bands can be generated for example by measuring or simulating the impulse response of the device from (approximately) equally distributed points, and using least-squares optimization methods to derive a set of mixing gains to obtain a matrix that enables obtaining frequency band FDA signal from the frequency band microphone array signals, for each frequency band below the spatial aliasing frequency.

The FOA signals have an omnidirectional component w(k,n) and the three orthogonal dipole components x(k,n), y(k,n) andz(k,n), where k is the frequency index and n is the time index. As in DirAC, the direction can be formulated from the sound field intensity shown as I in the expression below: <MAT> where Re denotes the real-part and * denotes the complex conjugate. The direction-of-arrival is then the opposite direction of the intensity vector. Note that the intensity vector may be averaged over several frequency and/or time indices (k,n).

In some examples the implementation of the DirAC method is configured to also estimate the diffuseness which is a ratio factor between <NUM> and <NUM> that determines how much of the total sound energy is non-directional. In other words the ratio of the energy of the non-directional parts of the received audio signals with respect to the overall energy of the received audio signals.

In some other examples the lower frequency analyser <NUM> may be configured to apply methods other than DirAC to the Ambisonics signals. Forexample a Harmonic planewave expansion (Harpex) and its variants may be employed to estimate two simultaneous directions-of-arrival from the FOA signal. Another example method which may be employed by the lowerfrequencyanalyserisa Higher-orderDirAC (HO-DirAC). The HO-DirAC is configured to estimate more than one direction of arrival in sectors from the second or higherorder Ambisonic signals. Such Ambisonic signals require more microphones from the device, for example at least <NUM> microphones for second order Ambisonics in 3D.

A further method may be the application of a delay-search algorithm to determine a delay that produces the maximum coherence between microphone array audio signals.

For example, for a capture device in the form of a mobile phone in a landscape mode a delay parameter may be determined between a 'left' and 'right' microphone audio signal. The delay may be normalized so that the maximum delay (the sound arriving from the axis of the microphones) is -<NUM> and <NUM>. The angle parameter may then be determined as acos(d), where d is the estimated and normalized delay parameter. The delay and angle parameter from a pair of microphone audio signals is ambiguous, in other words the same delay can be observed for example if the sound comes <NUM> degrees (at the front side) and <NUM> degrees (at the rear side). This ambiguity may be overcome by determining a binary front-back audio signal direction to determine if the sound comes from the front or the back by performing a delay-analysis at a further front-back microphone pair. This method may produce accurate results for capture devices such as mobile phones but may not produce accurate results for general capture device shapes such as shown in <FIG>. The reason for this is that the method applies a binary front-back determination to account for the thin axis of the device and estimates the remaining (majority) of the spatial information using the microphones at the two wider axes of the mobile-phone like device. While the approach is suitable for mobile phones and other similarly thin devices, for device in <FIG> there is more information available to utilize in all axes of the device. Therefore having only a binary front-back choice at one axis with such a device inevitably may lead to not effectively utilizing the information available at the microphone signals for the directional estimation. This may lead to suboptimal performance in directional estimation especially in more demanding acoustic conditions such as in presence of background noise. Additionally, having only a front-back determination for one axis (and determination of the majority of directional information from other axes) restricts the microphone placement to certain axes mutually dependent from each other, which may make it more difficultto optimize the microphone placementto account for other factors, such as to optimize the linear FOA capture at low frequencies. This method and the methods described herein are applicable for both lower and higher frequencies. The methods described herein utilize better the information conveyed by the inter-microphone properties for devices such as in <FIG>, and provide a flexible positioning of the microphones to enable optimization of other factors, such as linear FOA capture at low frequencies.

The higherfrequencyanalyser405 is configured to receive the audio signals within bands higherthan the spatial aliasing frequency. In some examples the higher frequency analyser is further configured to receive additional information from the lower frequencies/lower frequency analyser. For example there may be spatial weighting of the higher frequency directional analysis based on lower frequency directional analysis results.

In the higher frequency analyser <NUM> the audio signals are analysed to determine the spatial metadata parameters. This analysis may be used to produce accurate direction analysis because of the more regular microphone spacing found in some capture apparatus such as shown in <FIG>. The term "more regular" indicates that the dimensions of the capture device are more regularthan other capture device types such as a mobile phone which would have a long axis (the mobile phone length which may be <NUM>-<NUM>), a short axis (the mobile phone width which may be <NUM>-<NUM>) and very short axis (the mobile phone thickness which is typically less than <NUM>). Furthermore this analysis is less susceptible to hardware integration and/or design compromised microphone placement. Any capture devices which implement optional mobile-phone type algorithms may require the locating or positioning of microphones on top of each other and adjacent to each other, which may lead to a non-uniform microphone positioning distribution (more microphones located on one side compared to the other side of the device), if the number of microphones is desired to be minimized. For example the capture device such as shown in <FIG> implementing mobile-phone type algorithms may require the microphone <NUM> to be moved to the lower right corner to have it exactly below the microphone <NUM> and on the same plane as the microphones <NUM> and <NUM>. This positioning example would result in having more microphones on one side of the device.

The embodiments as discussed herein allow microphones to be positioned flexibly (e.g. no requirement for positioning the microphones on top of each other). Thus for example with respect to the capture device shown in <FIG>, the microphones can be placed relatively evenly around device. Furthermore in such embodiments a more regular or even microphone placement may also improve low-frequency analysis performance. For example, generating Ambisonic output signals using linear methods at low frequencies produces more accurate Ambisonic output signals when the microphones are placed around the device more evenly than when they are placed more on the same side.

The higherfrequencyanalyser405 as implemented in some examples employs direction analysis methods which utilize a modified delay analysis approach suitable for non-flat (i.e. non-mobile-phone-like) device shapes, such as the ones shown in <FIG> and <FIG>. Thus these examples produce more accurate direction analysis for a less constrained microphone placement where the sound is able to be captured more evenly from all directions.

In these examples the higher frequency analyser <NUM> (<NUM> in <FIG>) is configured to analyse the higherfrequency audio signals. In some examples the same methods may also be applied in the lower frequency analyser.

In other words in some examples both frequency bands above the spatial aliasing frequency <NUM> and frequency bands below the spatial aliasing frequency <NUM> are analysed by the higher frequency analyser <NUM>.

In such examples the direction-analysis is provided that can be used at all frequencies (including above the spatial aliasing frequency) and with devices with no shadowing, partial shadowing, and/or prominent shadowing (at any or all axes).

The direction analysis in the higher frequency analyser <NUM> may be summarised as:.

The direction analysis therefore may be employed from audio signals captured from microphones of the microphone array having displacement in x, y, and z directions. However these placements do not require the microphones to be placed along the x, y, or z axes (e.g., having one microphone on top of the other one).

Another possible analyser method may be for a vector to be determined to express each microphone of the array, where a length of the vector is the estimated energy (or magnitude)of that microphone signal in the frequency bands. The direction-of-arrival may then be determined as the direction of the sum of these vectors. This method, however, requires prominent acoustic shadowing at all axes, which the embodiments described herein do not.

With respect to <FIG> the operation of the analysis processor <NUM> is shown.

The initial operation is receiving the microphone array audio signals as shown in <FIG> by step <NUM>.

Having received the microphone audio signals they are converted into a time-frequency domain components and frequency bands generated (which include bands above and below the spatial aliasing frequencies) as shown in <FIG> by step <NUM>.

Furthermore the higher frequency microphone audio signals are spatially analysedto generate the metadata associated with the audio signals above the spatial aliasing frequencies, for example the directions and energy ratios as shown in <FIG> by step <NUM>.

Furthermore the lower frequency microphone audio signals are spatially analysed to generate the metadata associated with the audio signals below the spatial aliasing frequencies, for example the directions and energy ratios as shown in <FIG> by step <NUM>.

With respect to <FIG> an example higherfrequencyanalyseris shown. The inputs to the higher frequency analyser <NUM> are the audio signals of frequency bands above the spatial aliasing frequency from the forward filter bank.

The higher frequency analyser <NUM> in some examples comprises delay analysers shown in <FIG> as delay analyser (microphone pair <NUM>) <NUM>, delay analyser (microphone pair <NUM>) <NUM>, and delay analyser (microphone pair <NUM>) <NUM>. Each delay analyserreceives a selected pairof microphone audio signals and determines a delay parameter in frequency bands. In some examples there is at least two microphone pairs for horizontal surround capture, and at least three microphone pairs for 3D capture.

The selected microphone pairs are determined to span the desired dimensions and may be determined to be as orthogonal as possible (for example: for 3D capture the microphone pairs may not be only in the same plane).

In some examples more than one delay analyser is configured to receive the same microphone audio signals. In other words a microphone may be part of more than one microphone pair.

With respect to <FIG> is shown the example delay analyser selected pairs of microphonesfrom the capture device shown in <FIG>. The first pairof microphones may for example be microphones <NUM> and <NUM> defining a first displacement <NUM> diagonally over the 'top' surface of the device <NUM>. The second pair of microphones may for example be microphones <NUM> and <NUM> defining a second displacement <NUM> diagonally over a 'front' surface of the device. The third pair of microphones may for example be microphones <NUM> and <NUM> defining a third displacement <NUM> diagonally over a 'bottom' surface of the device. <FIG> furthermore shows the rear cameras <NUM> and <NUM> not shown in <FIG>.

For a microphone pair the delay analyser <NUM>, <NUM>, <NUM> is configured to estimate the delay in frequency bands between the two microphones audio signals. For example, the delay in frequency bands between two microphones can be estimated by finding a delay parameter that maximizes the cross-correlation of the microphone audio signals.

For complex-valued time-frequency signals A(k,n) and B(k,n), where k is the frequency bin index (for example the short time Fourier transform (STFT) bin index), and n is the time index of the time-frequency signal. The spatial analysis (determination of the spatial metadata) may take place in frequency bands which may involve one or more frequency bins of the time-frequency transform. The frequency bands are often designed to approximate a frequency resolution that is relevant for human spatial hearing, such as the Bark bands. One frequency band b, for example, may involve the frequency bins ranging from kb,bottom to kb,top. The delay d (b,n) for band b is found by finding a delay that maximizes <MAT> where Re is the real-part operator, * is the complex conjugate and K is the FFT size used in the STFT. In some examples similar but different delay analyses formulas can be applied if the filter-bank is something else than STFT.

The delay d(b,n) providing the maximum correlation can be found by determining the correlation with a set of delays ξ ranging between -Dmax to Dmax, where this maximum delay value Dmax is determined according to the microphone distance. For example, Dmax could be exactly the delay value corresponding to the time (in samples) that the sound travels the distance between the microphone pair. Dmax could also be a somewhatlongerthan the delay value corresponding to the time (in samples) that the sound travels the distance between the microphone pair, thus accounting for some measurement noise at the delay search edges. The set of delays ξ may be determined with the accuracy of a sample, but also fractional-sample delays may be used (or any other accuracy).

In some examples the maximum correlation parameter can be normalized to provide the ratio parameter such as defined below: <MAT>.

The direction vector formulator607 may be configured to receive the outputof the delay analysers <NUM>, <NUM>, <NUM> and determine a direction vector <NUM> based on the estimated delay values <NUM> of the microphone pairs.

The direction vector formulator <NUM> may in some examples be configured to formulate a direction vector by the following matrix operation <MAT> where M is a pre-formulated matrix and the subscripts of d denote the microphone pair. The numberof delay values d, i.e., the numberof microphone pairs can be two or more, and the matrix M size is accordingly determined. For example, for 3D direction determination and five delay values d the matrix M size is 3x5.

The estimated direction parameter is then the direction of vector s.

In some examples the matrix M can be formulated by the following method.

First, vectors are formulated for each microphone pair. For example, if microphone <NUM> is at position (x<NUM>, y<NUM>, z<NUM>) and a microphone <NUM> is at position (x<NUM>, y<NUM>, z<NUM>), then a vector between them is <MAT>.

Furthermore the direction vector formulator may be configured to determine a matrix V <MAT>.

The direction vector formulator may be configured to determine a vector (any vector of any length) indicating the direction of an assumed source, and denote this 3x1 vector as s. Then the direction vector formulator may define a vector: <MAT>.

The vector d contains relative delay values for the microphone pairs as the result of an assumed sound arriving from the direction of vector s. The overall scaling of vector d may be any suitable scale. Furthermore the scaling of vector s or vectors v may be any suitable scaling provided the vectors v have the same mutual scaling.

In some examples, for example a real-time example, at a certain time-frequency interval it may be possible to obtain for the set of delay values (in any units orscaling), stored in vector d. The direction vector formulator <NUM> may estimate a direction vector s corresponding to the estimated delays d. In other words, the direction vector formulator <NUM> determines matrix M, for which it is able to obtain s from d: <MAT> where the result of the formula is M = V-<NUM>, which denotes the pseudo-inverse of a matrix containing vectors corresponding to the microphone pairs (from one microphone to the next). In some example examples it would be understood that any suitable application of matrix arithmetic can be employed.

As the matrix M is the pseudo-inverse of V, then the vectors within V should represent all axes sufficiently. For example, the vectors in V should span a reasonable volume for 3D capture, or a reasonable area in 2D capture. Considering the 3D case, more "flat" the vector base is, the more unstable the matrix inverse of V is in terms of the effect of the measurement noise to the directional estimates. A practical example of having such flatvector base would be a mobile phone,whereone dimension isvery small. On the contrary, for 3D devices (or disc-type devices in horizontal capture), the present method is robustin terms of spatial analysis accuracy and stability at all axes.

In some examples there may be determined different matrices M that can be applied to estimate the direction vector from the delay values which take account of the situations when it is determined that one or more delay estimates are unreliable. For example a determination of reliability may be made with respect to a coherence value, where the delay estimate is unreliable where the coherence value is smaller than a threshold value. A further determination of reliability may be made with respect to the energy for the corresponding microphone pair in relation to the energies of the other microphone pairs. In some examples these unreliable delay values can be omitted or discarded, and another M matrix is applied to the remaining delay values. The replacement matrix M may be the pseudo-inverse of a modified matrix V which is otherwise the same matrix V as above, without the rows corresponding to the omitted microphone pairs.

The direction formulator <NUM> may then be configured to receive the direction vector <NUM> and output the direction parameter <NUM> component of the metadata. The direction-of-arrival is the direction of this resulting vector s in <MAT>.

In some examples an estimate of the direct-to-total ratio parameter <NUM> at the higherfrequencies may be generated by selecting the microphone pair that has the largest displacement, and to set as the ratio parameter as a normalized (between <NUM>. <NUM>) delay-compensated correlation <NUM> between the microphone signals such as found from the delay analysers <NUM>, <NUM>, <NUM>.

With respect to <FIG> an example operation of the 'higher' frequency analyser is shown.

The receipt of the filter bank output audio signals is shown in <FIG> by step <NUM>.

The selection of suitable microphone pairs to analyse the audio signals is shown in <FIG> by step <NUM>.

The analysing of the delay between the microphone pair <NUM> audio signals is shown in <FIG> by step <NUM>.

The formulation of the direction vector by the application of a determined microphone matrix to the delays is shown in <FIG> by step <NUM>.

The determination of the direction parameter based on the direction vector is shown in <FIG> by step <NUM>.

The outputting of the direction parameter is shown in <FIG> by step <NUM>.

The selection of the microphone pair with the largest distance isshown in <FIG> by step <NUM>.

The normalisation of the delay compensated correlation is shown in <FIG> by step <NUM>. The determination of the delay where maximum correlation is received is a product of steps <NUM>, <NUM>, <NUM> and <NUM> and whichever has the largest distance. These steps generate and search different delays and produce the delay that has the maximum correlation which may be normalised by step <NUM>.

The output of the normalised delay compensated correlation value as the energy ratio parameter is shown in <FIG> by step <NUM>.

With respect to <FIG> an example synthesis processor <NUM> (as shown in <FIG>) according to some examples is shown.

In some examples a demultiplexer may be configured to receive a data stream and demultiplex the data stream into transport audio signals <NUM> and metadata <NUM> such as the ratio or other diffuseness parameter <NUM> and direction parameters <NUM>. In some examples, where the transport audio signals were encoded within the analysis processor, the demultiplexer is furthermore caused to decode the audio signals. In some examples the metadata is decoded if it was encoded in the analysis processor.

In other examples the metadata and transport signals are received or inputto the synthesis processor separately.

A T-F domain transformer <NUM> may receive the transport audio signals <NUM> and divide the signals into a time-frequency representation. In some examples the transport audio signals <NUM> are transformed to the time-frequency domain using a suitable transformer. For example a short-time Fourier transformer (STFT) may apply a short-time Fourier transform to the transport audio signals to generate suitable time-frequency domain audio signals. In some examples any suitable time-frequency transformer may be used, for example a complex-modulated quadrature mirror filterbank (QMF).

A divider <NUM> may receive the output of the T-F domain transformer <NUM> and divide the signals into direct and ambient parts based on the ratio metadata parameters <NUM>. Thus the divider <NUM> may receive the time-frequency domain audio signals and the energy ratios and divide the time-frequency domain audio signals to ambient and direct parts using the energy ratio r(k, n). Note that here ratio r(k, n) is determined for each frequency bin index k instead of ratio(b, n) for each band b. If ratio(b, n) is obtained as an input, it can be mapped to several bins within that band, thus the ratio r(k, n) is obtained. For example in some examples the direct part could be obtained by multiplying the inputfrequency band signals with a factor sqrt(r(k, n)). The ambient part could be obtained by multiplying the input frequency band signals with a factor sqrt(<NUM>- r(k, n)).

A decorrelator <NUM> may be configured to receive the ambient audio signal part and process it to make it perceived as being surrounding, for example by decorrelating and spreading the ambient audio signal part across the audio scene.

A spatial processor <NUM> may be configured to receive the direct audio signal part and directions <NUM> and position the direct audio signal part based on the directions. For example in some examples the directions <NUM> are employed to determine panning gains using vector-base amplitude panning (VBAP) and the direct audio signal part is positioned by applying the panning gains to the direct part signal. In some examples the gains might be temporally smoothed before they are applied on the direct audio signal.

In examples where the output is an Ambisonicsignal the VBAP processing may be replaced with spherical-harmonic gain formulation as a function of the direction parameter. The outputof the decorrelation in such examples may also be processed with determined gains to fit the selected Ambisonic normalization scheme.

In some examples the output audio signal is a binaural signal. In that case, the VBAP processing may be replaced with head-related transfer function (HRTF) processing as a function of the direction parameter and the band frequency. In such examples the decorrelated signals may be processed to have a frequency-dependent binaural inter-channel coherence instead of full incoherence at all frequencies.

A combiner <NUM> may be configured to receive the spatially spread ambient signal part from the decorrelator <NUM> and the positioned direct audio signals part from the spatial processor <NUM> and combine or merge these resulting audio signals.

An inverse T-F domain transformer which may be a suitable inverse short-time Fourier transformer (Inverse STFT) is configured to receive the combined audio signals and apply an inverse transform to generate the multi-channel audio signals <NUM> (or binaural signals <NUM> or ambisonic signals <NUM>) which may be passed to a suitable output device such as the headphones or multi-channel loudspeaker setup.

With respect to <FIG> the operations of the spatial synthesizer shown in <FIG> according to some examples are described in further detail.

The spatial synthesizer in some examples is configured to receive the transport audio signals and metadata (the energy ratios/directions) as shown in <FIG> by step <NUM>.

The received transport audio signals are in some examples converted into a time-frequency domain form (for example by applying a suitable time-frequency domain transform) as shown in <FIG> by step <NUM>.

The time-frequency domain audio signals may then in some examples be divided into ambient and direct parts (based on the energy ratios) as shown in <FIG> by step <NUM>.

The ambient audio signal part may be decorrelated as shown in <FIG> by step <NUM>.

The direct part may be spatially processed, for example the determination and application of panning gains to the direct audio signal part as shown in <FIG> by step <NUM>.

The positional component of the audio signals and the decorrelated ambient audio signal may then be combined or merged as shown in <FIG> step <NUM>.

Furthermore the combined audio signals may then be inverse time-frequency domain transformed to generate the multichannel audio signals/binaural audio signals/ambisonic audio signals as shown in <FIG> by step <NUM>.

These multichannel audio signals/binaural audio signals/ambisonic audio signals may be output as shown in <FIG> by step <NUM>.

The synthesis of the audio signal may be any suitable synthesis as the output from both the lower frequency analyser and the higher frequency analyser may be generated in a similar format.

A synthesis processor in some examples may be configured to process the spatial sound in terms of the covariance matrices. In such examples the inputsignal has a covariance matrix, and the outputsignal has another "target" covariance matrix that is determined by the spatial metadata. Such methods are configured to formulate a mixing solution in frequency bands that, when applied to the inputsignal, generates the target covariance matrix for the output signal, and therefore the intended perceptual spatial characteristics.

Some of the advantages of the proposed embodiments is that capture devices (having microphone arrays) that do not exhibit susubstantial acoustic shadowing atsome axes can be employed with the proposed direction-of-arrival analysis at frequencies above the spatial aliasing frequency. The proposed embodiments can be used to enable robust direction-of-arrival at all frequencies also in the case of no shadowing, partial shadowing or prominent shadowing at all or some axes.

With respect to <FIG> an example electronic device which may be used as the analysis or synthesis processor is shown. The device may be any suitable electronics device or apparatus. For example in some embodiments the device <NUM> is a VR camera or other camera device, mobile device, user equipment, tablet computer, computer, audio playback apparatus, etc..

In some embodiments the device <NUM> comprises a memory <NUM>. In some embodiments the at least one processor <NUM> is coupled to the memory <NUM>. The memory <NUM> can be any suitable storage means. In some embodiments the memory <NUM> comprises a program code section for storing program codes implementable upon the processor <NUM>. Furthermore in some embodiments the memory <NUM> can further comprise a stored data section for storing data, for example data that has been processed or to be processed in accordance with the embodiments as described herein. The implemented program code stored within the program code section and the data stored within the stored data section can be retrieved by the processor <NUM> whenever needed via the memory-processor coupling.

In some embodiments the device <NUM> comprises a user interface <NUM>. The user interface <NUM> can be coupled in some embodiments to the processor <NUM>. In some embodiments the processor <NUM> can control the operation of the user interface <NUM> and receive inputs from the user interface <NUM>. In some embodiments the user interface <NUM> can enable a user to input commands to the device <NUM>, for example via a keypad. In some embodiments the user interface <NUM> can enable the user to obtain information from the device <NUM>. For example the user interface <NUM> may comprise a display configured to display information from the device <NUM> to the user. The user interface <NUM> can in some embodiments comprise a touch screen or touch interface capable of both enabling information to be entered to the device <NUM> and further displaying information to the user of the device <NUM>.

In some embodiments the device <NUM> comprises an input/output port <NUM>. The input/output port <NUM> in some embodiments comprises a transceiver. The transceiver in such embodiments can be coupled to the processor <NUM> and configured to enable a communication with other apparatus or electronic devices, for example via a wireless communications network. The transceiver or any suitable transceiver or transmitter and/or receiver means can in some embodiments be configured to communicate with other electronic devices or apparatus via a wire or wired coupling.

The transceiver can communicate with further apparatus by any suitable known communications protocol. Forexample in some embodiments the transceiver can use a suitable universal mobile telecommunications system (UMTS) protocol, a wireless local area network (WLAN) protocol such as for example IEEE <NUM>. X, a suitable short-range radio frequency communication protocol such as Bluetooth, or infrared data communication pathway (IRDA).

The transceiver in put/outputport <NUM> may be configured to receive the signals and in some embodiments determine the parameters as described herein by using the processor <NUM> executing suitable code. Furthermore the device may generate a suitable transport signal and parameter output to be transmitted to the synthesis device.

In some embodiments the device <NUM> may be employed as at least part of the synthesis device. As such the input/output port <NUM> may be configured to receive the transport signals and in some embodiments the parameters determined at the capture device or processing device as described herein, and generate a suitable audio signal format output by using the processor <NUM> executing suitable code. The input/output port <NUM> may be coupled to any suitable audio output for example to a multichannel speaker system and/or headphones or similar.

Further in this regard it should be noted that any blocks of the logic flow as in the Figures may represent program steps, or interconnected logic circuits, blocks and functions, ora combination of program steps and logic circuits, blocks and functions.

The memory may be of any type suitableto the local technical environment and may be implemented using any suitable data storage technology, such as semiconductor-based memory devices, magnetic memory devices and systems, optical memory devices and systems, fixed memory and removable memory.

Embodiments of the inventions may be practiced in various components such as integrated circuitmodules.

of Mountain View, California and Cadence Design, of San Jose, California automatically route conductors and locate components on a semiconductorchip usingwell established rules of design as well as libraries of pre-stored design modules. Once the design fora semiconductor circuit has been completed, the resultant design, in a standardized electronic format (e.g., Opus, GDSII, or the like) may be transmitted to a semiconductor fabrication facility or "fab" for fabrication.

Claim 1:
An apparatus for spatial audio capture, the apparatus comprising means for:
receiving audio signals (<NUM>) from a microphone array, the microphone array comprising three or more microphones forming a geometry with defined displacements between pairs of the three or more microphones;
selecting (<NUM>) from the audio signals a first part comprising audio signals above a spatial aliasing frequency associated with the geometry with defined displacements between the pairs of the three or more microphones;
determining (<NUM>) delay information between the first part comprising audio signals above the spatial aliasing frequency associated with the geometry with defined displacements between the pairs of the three or more microphones;
selecting (<NUM>) from the audio signals a second part comprising audio signals below the spatial aliasing frequency associated with the geometry with defined displacements between the pairs of the three or more microphones;
analysing (<NUM>) the second part comprising audio signals below the spatial aliasing frequency to determine further spatial parameters;
determining (<NUM>) an operator based on the geometry with the defined displacements between the pairs of the three or more microphones; and
applying (<NUM>) the operator to the determined delay information to generate at least one direction parameter associated with the audio signals.