Patent Description:
VR sound recording typically requires Ambisonics B-format to be captured with four first-order microphone capsules. To this end, professional audio microphones may either record A-format - to be then encoded into B-format by applying a four by four conversion matrix - or may record directly the Ambisonics B-format - for instance by using soundfield, like microphones.

However, in many consumer products, first-order microphones (or other directive microphones) are not suitable, since they have to lay in free-field to be operational. Instead, omnidirectional microphones are used in such products, and their signals are first mutually pre-processed to obtain at least four virtual first-order microphone signals to be then transformed into FOA.

In an exemplary method, a pair of two omnidirectional microphone signals can be converted into a first-order differential signal, yielding a virtual cardioid signal. Then, using a distribution of omnidirectional microphones, the resulting four differential signals can be encoded into B-format. However there are two main limitations with this method. A first limitation is related to the spectral defects at higher frequencies (given the spatial aliasing resulting from the microphones spacing), and a second limitation relates to the microphone placement constraints, due to design and hardware specifications, which prevent them looking in all directions.

The first limitation results from the spatial aliasing, which, by design, reduces the bandwidth to frequencies f in the range of: <MAT>.

In the above equation (<NUM>), c stands for the sound celerity, and dmic stands for the distance between a pair of two omnidirectional microphones.

Another exemplary method for generating FOA signals from omnidirectional microphones samples the soundfield using a dense enough distribution of microphones (e.g. the Eingenmike with <NUM> capsules). The sampled sound pressure signals are then converted to spherical harmonics, and then linearly combined to eventually generate FOA signals. The main limitation of this method is the required number of microphones. For consumer applications, with only few microphones available (commonly only up to <NUM>), linear processing is too limited. This limitation leads to signal to noise ratio (SNR) issues at low frequencies, and again, to aliasing at high frequencies.

In summary, it is a challenging task to provide suitable audio recordings, in particular for VR applications, when using small and/or mobile devices such as phones, tablets, or on-board cameras. The non-consistent dimensions of many mobile devices (large screen/minimum thinness) restrict the possibility to record relevant sound in all directions and over all of the frequency bandwidth. Many constraints result directly from the device design: E. often only omnidirectional microphones can be used, while directive microphones are not suitable because they have to lie in free field. Further, microphone placement is often restricted to a limited number of possible positions on the device.

<CIT> relates to coding of a soundfield representation.

In view of the above-mentioned challenges and limitations, embodiments of the present invention aim to improve the current methods. An objective is to provide a device and method that enable improved 3D audio recordings, which are suitable for VR applications, and can be performed with small and/or mobile devices. The device and method should provide a FOA signal from multiple microphone signals. The use of directive microphones should be possible. Further, the encoding of the multiple microphone sound signals into the FOA signal should be more robust, in particular over a larger frequency bandwidth and over a larger set of directions.

The objective is achieved by embodiments of the invention as described in the enclosed independent claims. Advantageous implementations of embodiments are further defined in the dependent claims.

In particular, considering a system of M≥<NUM> (possibly virtual) directive microphone signals, embodiments of the invention can generate a corresponding FOA signals successively by: deriving the look direction angles of the M directive microphones producing the microphone signals, and then computing a matrix representing how these directive microphones would be obtained for the FOA channels (W, X, Y, Z). This matrix is then inverted, e.g. using a pseudo-inverse algorithm, to obtain an inverted matrix, and the inverted matrix can be applied to the M microphone signals to generate the FOA channels.

A first aspect of the invention provides a device for obtaining a FOA signal from signals of at least five first-order directive microphones, the device being configured to: determine a look direction of each microphone, calculate a decoding matrix based on the determined look directions, wherein the decoding matrix is suitable for decoding a FOA signal into the signals of the microphones, invert the decoding matrix to obtain an encoding matrix, and encode the signals of the microphones based on the encoding matrix to obtain the FOA signal.

Thus, the device of the first aspect allows obtaining the FOA signal from multiple microphone signals, wherein the use of directive microphones is possible. The device size can be reduced compared to the exemplary methods described above. Due to the calculation and use of the encoding matrix, the encoding of the multiple microphone sound signals into the FOA signal is also more robust, in particular over a larger frequency bandwidth and over a larger set of directions. Thus, the device of the first aspect enables improved recording of 3D audio suitable for VR applications and/or surround sound.

In this implementation form, the device of the first aspect and the microphones provide an overdetermined system of M><NUM> directive microphone signals. This leads to even more accurate directional responses, and thus a more accurate FOA signal.

In an implementation form of the first aspect, the device comprises the at least five first-order directive microphones.

Thus, limitations of the exemplary methods mentioned above are overcome, and directive microphones can be used in the device. The device can be reduced in size.

In an implementation form of the first aspect, at least one of the microphones is a virtual directive microphone, in particular based on at least two omnidirectional microphones.

In an implementation form of the first aspect, the device is further configured to determine the look direction of the virtual directive microphone based on an orientation of the at least two omnidirectional microphones.

Thus, an alternative to the used of directive microphones is provided. It is also possible to have directive microphones and omnidirectional microphones, of which the device receives signals, or which are part of the device.

In an implementation form of the first aspect, the look direction of a microphone is based on an azimuth angle and an elevation angle of that microphone.

In an implementation form of the first aspect, the decoding matrix is a B-format decoding matrix.

In an implementation form of the first aspect, the device is further configured to invert the decoding matrix using a pseudo-inverse algorithm.

In an implementation form of the first aspect, the device is further configured to perform a Direction of Arrival (DOA) estimation based on the FOA signal.

In an implementation form of the first aspect, the FOA signal comprises four FOA channels.

In an implementation form of the first aspect, the device is a mobile device.

For instance, the device may be a mobile phone, smartphone, laptop, tablet, camera, on-board camera or similar device. The device can have a larger screen and/or can be fabricated thinner than a device working with an exemplary method described above.

A second aspect of the invention provides a mobile device, particularly a smartphone, tablet or camera, including the device according to the first aspect or any of its implementation forms.

The mobile device enjoys all advantages and technical effects described above for the device of the first aspect.

A third aspect of the invention provides a method for obtaining a FOA signal from signals of at least five first-order directive microphones, the method comprising: determining a look direction of each microphone, calculating a decoding matrix based on the determined look directions, wherein the decoding matrix is suitable for decoding a FOA signal into the signals of the microphones, inverting the decoding matrix to obtain an encoding matrix, and encoding the signals of the microphones based on the encoding matrix to obtain the FOA signal.

In an implementation form of the third aspect, the method is performed by or in a mobile device.

In an implementation form of the third aspect, at least one of the microphones is a virtual directive microphone, in particular based on at least two omnidirectional microphones.

In an implementation form of the third aspect, the method further comprises: determining the look direction of the virtual directive microphone based on an orientation of the at least two omnidirectional microphones.

In an implementation form of the third aspect, the look direction of a microphone is based on an azimuth angle and an elevation angle of that microphone.

In an implementation form of the third aspect, the decoding matrix is a B-format decoding matrix.

In an implementation form of the third aspect, the method further comprises: inverting the decoding matrix using a pseudo-inverse algorithm.

In an implementation form of the third aspect, the method further comprises: performing a DOA estimation based on the FOA signal.

In an implementation form of the third aspect, the FOA signal comprises four FOA channels.

Accordingly, the method of the third aspect and its implementation forms achieve the same advantages and technical effects as described above for the device of the first aspect and its respective implementation forms, in particular because the method can be performed by the device of the first aspect.

A fourth aspect of the invention provides a computer program product comprising a program code for controlling a device according to the first aspect or any of its implementation forms, or for carrying out, when implemented on a processor, the method according to the third aspect or any of its implementation forms.

Thus, all advantages and technical effects described above for the device of the first aspect and method of the third aspect can be achieved.

<FIG> shows a device <NUM> for illustrative purposes. The device <NUM> may comprise processing circuitry (not shown) configured to perform, conduct or initiate the various operations of the device <NUM> described herein. The processing circuitry may comprise hardware and software. The hardware may comprise analog circuitry or digital circuitry, or both analog and digital circuitry. The digital circuitry may comprise components such as application-specific integrated circuits (ASICs), field-programmable arrays (FPGAs), digital signal processors (DSPs), or multi-purpose processors. In one embodiment, the processing circuitry comprises one or more processors and a non-transitory memory connected to the one or more processors. The non-transitory memory may carry executable program code which, when executed by the one or more processors, causes the device <NUM> to perform, conduct or initiate the operations or methods described herein.

The device <NUM> is configured to obtain a FOA signal <NUM> from signals <NUM> of at least four directive microphones <NUM>. <FIG> exemplarily illustrates a scenario with four directive microphones, which may also be four virtual directive microphones (i.e. the sound may actually be captured by omnidirectional microphones). The device <NUM> may be a small and/or mobile device, or may be included in such a mobile device. The mobile device may, for example, be a smartphone, tablet, or camera.

The device <NUM> is configured to determine a look direction <NUM> of each directive microphone <NUM>, e.g. based on the respective microphone signals <NUM>. The look direction <NUM> of a directive microphone <NUM> may be derived based on an azimuth angle and an elevation angle of that microphone or based on an orientation of at least two omnidirectional microphones (in case of a virtual directive microphone <NUM>).

The device <NUM> is further configured to calculate a decoding matrix <NUM> based on the determined look directions <NUM> of the microphones <NUM>, wherein the decoding matrix <NUM> is a matrix that is suitable for decoding a FOA signal into the microphone signals <NUM> of the microphones <NUM>. That is, the decoding matrix <NUM> is such that it could be used to generate/recover the microphone signals <NUM> from a FOA signal.

The device <NUM> is further configured to invert the decoding matrix <NUM> to obtain an encoding matrix <NUM>, and to then encode the signals <NUM> of the microphones <NUM> based on the obtained encoding matrix <NUM> to generate the FOA signal <NUM>. The FOA signal <NUM> may then be output, or may be used to obtain a DOA estimate for the microphone signals <NUM>.

<FIG> shows a device <NUM> according to an embodiment of the invention, which builds on the device <NUM> shown in <FIG>. Same elements in <FIG> and <FIG> are labelled with the same reference signs and function likewise.

The device <NUM> shown in <FIG> receives signals <NUM> from more than four (e.g. M=<NUM>, M=<NUM>, M=<NUM>-<NUM>, M><NUM>, or even M><NUM>) directive (potentially virtual) first-order directive microphones <NUM>. In <FIG>, the device <NUM> is further shown to include the multiple directive microphones <NUM>. As shown further in <FIG>, the look direction <NUM> of a microphone <NUM> may be based on an azimuth angle and an elevation angle of that microphone <NUM>. Further, the decoding matrix <NUM> may specifically be a B-format decoding matrix (e.g. an Mx4 matrix). The encoding matrix <NUM> may be a pseudo-inverse encoding matrix (e.g. a 4xM matrix). The encoding of the signals <NUM> may be performed by matrixing the signals <NUM> with the encoding matrix <NUM>, in order to obtain the FOA signal <NUM>. The FOA signal <NUM> may comprises four FOA channels (W, X, Y, Z).

The functions carried out by the device <NUM> shown in <FIG> are now further explained. Considered are generally M first-order microphones <NUM>, which are distributed in the XYZ-space with their coordinates: <MAT>.

Their look directions <NUM> may be defined by their azimuth (Θ) and elevation (ϕ) angles. The look direction <NUM> may in particular be retrieved by using:.

Given the look directions <NUM> of the (potentially virtual) directive microphones <NUM>, a corresponding M×<NUM> matrix Γ (the decoding matrix <NUM>) may be obtained, wherein the matrix would enable to retrieve the M microphone signals <NUM> from the FOA channels (W, X, Y, Z) by: <MAT>.

Thereby, u is the first-order microphone directional response characteristic, i.e.:.

The decoding matrix Γ is then inverted, for example, by using a pseudo-inverse algorithm. The resulting <NUM>×M matrix _1 (the encoding matrix <NUM>): <MAT>.

The pseudo-inverse is the generalized inverse of a matrix. It corresponds to solving the overdetermined linear system of the equations (<NUM>). It has <NUM>, <NUM>, or infinitely many solutions. The equation (<NUM>) is the closest solution when none exists in the norm <NUM> sense, i.e. minimizing |b s |<NUM>. It gives the single answer when one solution exists. And when many exist, it is the smallest solution in the sense that |b|<NUM> is smallest.

The encoding matrix <NUM> can then be directly used to encode the directive microphone signals <NUM> (S<NUM>,S<NUM>,. ,SM) into the FOA signal <NUM>. It is also possible to capture/receive microphone signals <NUM> over time and obtain multiple successive FOA signals.

Given the four encoded FOA channels of the FOA signal <NUM>, a DOA estimation can be performed based on the FOA signal <NUM> by: <MAT> and <MAT>.

The proposed device <NUM> according to an embodiment of the invention, e.g. as shown in <FIG> or <FIG>, can achieve an improved 3D audio recording, and particular the following advantages:.

As shown in <FIG>, the resulting directional responses of the FOA channels (W, X, Y, Z) have been measured using a phone prototype (including/being a device <NUM> according to an embodiment of the invention) with <NUM> omnidirectional microphone capsules. Using these <NUM> microphones, up to <NUM> pairs can be formed leading to M=<NUM> virtual cardioid signals composing the A format (S<NUM>,S<NUM>,···,,S<NUM>), and thus yielding an overdetermined system. <FIG> shows these directional responses for various octave bands.

<FIG> shows the directional responses using M=<NUM> microphone pairs in a device <NUM> for illustrative purposes. The results shown in <FIG> are thus not from an overdetermined system. This leads to somewhat less accurate directional responses compared to <FIG>.

<FIG> shows a method <NUM> according to an embodiment of the invention. The method <NUM> is suitable for obtaining a FOA signal <NUM> from signals <NUM> of at least five directive microphones <NUM>. The method <NUM> may be carried out by the device <NUM> shown in <FIG>, or may be carried out by a mobile device including such a device <NUM>.

The method <NUM> comprises: a step <NUM> of determining <NUM> a look direction <NUM> of each microphone <NUM>; a step <NUM> of calculating a decoding matrix <NUM> based on the determined look directions <NUM>, wherein the decoding matrix <NUM> is suitable for decoding a FOA signal into the signals <NUM> of the microphones <NUM>; a step <NUM> of inverting the decoding matrix <NUM> to obtain an encoding matrix <NUM>; and a step <NUM> of encoding <NUM> the signals <NUM> of the microphones <NUM> based on the encoding matrix <NUM> to obtain the FOA signal <NUM>.

Claim 1:
A device (<NUM>) for obtaining a First Order Ambisonic, FOA, signal (<NUM>) from signals (<NUM>) of at least five first-order directive microphones (<NUM>) comprised by the device, the device (<NUM>) being configured to:
determine a look direction (<NUM>) of each microphone (<NUM>),
calculate a decoding matrix (<NUM>) based on the determined look directions (<NUM>), wherein the decoding matrix (<NUM>) is suitable for decoding a FOA signal into the signals (<NUM>) of the microphones (<NUM>),
invert the decoding matrix (<NUM>) to obtain an encoding matrix (<NUM>), and
encode the signals (<NUM>) of the microphones (<NUM>) based on the encoding matrix (<NUM>) to obtain the FOA signal (<NUM>).