Patent Description:
Two distinct classes of spatial sound capture and reproduction exist that are relevant to the following disclosure:.

The Ambisonic audio format (or spherical harmonic signals) is a classical spatial audio signal representation. Recently, this signal representation (or format) has also become a commonly implemented choice for spatial audio transmission. It consists of different orders of spherical harmonics. A zeroth order harmonic (= zero spatial frequency) is represented by an omnidirectional signal. A first order harmonic is represented by dipole patterns, and the higher orders have quadrupoles, etc. The term higher-order Ambisonics (HOA) in the following disclosure refers to techniques using the zeroth to second (or to higher) order spherical harmonic signals. There are many variants or configurations for spherical harmonic signals. For example, the relative amplitudes or the ordering of the spherical harmonics may vary in different definitions. The conversions between any such variants is typically straightforward using linear (matrix) operations.

The Ambisonic audio format (or spherical harmonic signals) can also be used as a format to transmit spatial audio. For example, YouTube 3D audio/video services have started to stream spatial audio using the first order Ambisonic format (spherical harmonic signals), consisting of one omnidirectional signal (zeroth order) and three dipole signals (first order). Although the approach is not optimum for quality nor the bit rate, the existing streaming service shows that the approach in practice produces a satisfactory experience for the end user. Furthermore, the Ambisonic audio format is a straightforward and a fully defined format. As such, it is a useful audio format for services such as YouTube and alike to use. The Ambisonic audio format signals can be linearly decoded at the receiver end and rendered to headphones (binaural) or to loudspeakers, using known methods.

The generation of spherical harmonic signals is problematic. To generate the spherical harmonic signals specialist apparatus in the form of specialist microphone arrays may be required to capture the signals using linear means. Other ways to generate spherical harmonic signals using conventional or general microphone arrangements and then processing the microphone signals using linear combination processing may produce spherical harmonic signals which produce poor quality results.

<CIT> discusses an apparatus for determining a converted spatial audio signal, the converted spatial audio signal having an omnidirectional audio component and at least one directional audio component, from an input spatial audio signal, the input spatial audio signal having an input audio representation and an input direction of arrival. The apparatus comprises an estimator for estimating a wave representation comprising a wave field measure and a wave direction of arrival measure based on the input audio representation and the input direction of arrival. The apparatus further comprises a processor for processing the wave field measure and the wave direction of arrival measure to obtain the omnidirectional audio component and the at least one directional component.

There is provided according to a first aspect an apparatus as defined in claim <NUM>.

According to a second aspect there is provided a method as defined in claim <NUM>.

The following describes in further detail suitable apparatus and possible mechanisms for the provision of effective spherical harmonic signal generation from a microphone array. In the following examples, audio signals and audio capture signals are described. However it would be appreciated that in some embodiments the apparatus may be part of any suitable electronic device or apparatus configured to capture an audio signal or receive the audio signals and other information signals. In the following the term spherical harmonics denote harmonics over space. Furthermore as explained in further detail hereafter adaptive means denote that the processing is adaptive with respect to the properties of the signal that is processed. Thus as described hereafter features may be extracted from the audio signals, and the signals processed differently depending on these features. The embodiments described herein describe the adaptive processing in terms of at least at some frequency bands and spherical harmonic orders, and optionally spatial dimensions. Thus in contrast to conventional ambisonics there is no linear correspondence between output and input.

The following disclosure specifically describes adaptive SPAC techniques which represent methods for spatial audio capture from microphone arrays previously to loudspeakers or headphones. The embodiments as described hereafter are concerned with enabling the compatibility of SPAC capture methodology with spherical harmonic signal representation. In other words, to enable the output of the systems utilizing the dynamic SPAC analysis to be compatible with existing Ambisonic decoders. Spatial audio capture (SPAC) refers here to techniques that use adaptive time-frequency analysis and processing to provide high perceptual quality spatial audio reproduction from any device equipped with a microphone array, for example, Nokia OZO or a mobile phone. At least <NUM> microphones are required for SPAC capture in horizontal plane, and at least <NUM> microphones are required for 3D capture. The SPAC methods are adaptive, in other words they use non-linear approaches to improve on spatial accuracy from the state-of-the art traditional linear capture techniques.

The problem of traditional linear operations and methods (to capture the spherical harmonic signals from a microphone array) is that the requirements for a microphone array in order to accurately capture the audio signals are strict. For example, a first order spherical harmonic audio signal capture would require a B-format microphone with directional sensors.

Alternatively, for rigid devices such as a Nokia OZO or a smart phone, omnidirectional microphones (sensors) could be mounted on the surface of a device. In principle, based on the microphone signals the spherical harmonic signals could be retrieved using linear methods. In practice, as will be further discussed in detail below, the linear methods pose excessively strict requirements for many relevant practical use cases.

A first linear approach is to apply a matrix of designed linear filters to the microphone signals to obtain the spherical harmonic components. An equivalent alternative linear approach is to transform the microphone signals to the time-frequency domain, and for each frequency band use a designed mixing matrix to obtain the spherical harmonic signals in the time-frequency domain. The resultant spherical harmonic signals in the time-frequency domain are then inverse-transformed back to time-domain PCM signals.

However, due to fundamental constraints of linear spatial audio capture (discussed in further detail below), the device must firstly be sufficiently large for low-frequency capture (e.g. size of OZO which is approximately 260x170x160mm), and the microphone spacing must be sufficiently dense for high-frequency capture (e.g. <NUM> apart). This produces a requirement for a large number of microphones. An example of a device fulfilling both these properties satisfactorily simultaneously is a <NUM>-microphone Eigenmike, which is an audio-only solution.

The issue with the application of linear methods to OZO-sized devices with for example <NUM> microphones is that the medium to high auditory frequencies (for example, above <NUM>) have a wavelength that is too small in comparison to the microphone spacing. At these frequencies the well-known effect of spatial aliasing occurs. This means that spherical harmonic audio signals no longer retain their intended spatial capture patterns, and as the result, any decoding of such signals to loudspeakers or headphones will be spatially wrong at these frequencies. For example, it may occur that the reproduced sound is perceived arriving from a wrong direction, or that the directional perception is vague. In other words, traditional linear methods do not enable the capture of spherical harmonic audio signals in a satisfactory auditory bandwidth using an OZO or any similar device.

The issue with small devices is the large wavelength at low frequencies with respect to the array size. At low frequencies (for example <NUM>) the audio wavelength is <NUM> meters. A small device, which may be a smartphone, may have microphones located <NUM> apart. Since the audio wavelength is long, the sound arriving to the different microphones is very similar. The 1st and higher order spherical harmonics are formulated from the differences between the microphone signals, and this difference signal can with small devices be very small in amplitude with respect to the microphone self-noise or other interferences. For example, at <NUM> the assumed small device can suffer from approximately <NUM> dB reduced signal-to-noise ratio at the 1st order spherical harmonics. The effect is larger for higher orders of spherical harmonics. The higher order linear capture also requires many microphones (for example <NUM> or more), which is not practical for small devices. In other words, traditional linear methods do not enable the capture of spherical harmonic audio signals in a satisfactory auditory bandwidth using a mobile phone or any similar device.

As a summary of the above, with the OZO device the microphones are too sparse for higher frequencies, and for small devices such as a mobile phone the array size is too small for the low frequencies.

In other words, for devices other than the high-end arrays such as the <NUM>-microphone Eigenmike, a large proportion of the auditory frequency range is not well captured with traditional linear methods. This issue is equivalent in all linear, i.e., non-adaptive spatial capture techniques, and not only when the spherical harmonic representation is employed. Hence, for a large portion of the practical device categories it is a requirement to employ adaptive SPAC methods for spatial audio capture, also in context of generating spherical harmonics.

Although to overcome this problem and linearly obtain the spherical harmonic signals at a satisfactory bandwidth an approach may be to equip the OZO-type camera with many high-quality microphones, such as <NUM> or more, this produces a complex and significantly more expensive device. The concept in these embodiments is to build the device with fewer microphones, such as <NUM>, which is simpler and more cost-efficient. For small devices such as a hand-held spherical camera or a smart phone, there is no such a prior art linear capture option available.

Similarly although for audio/video capture it is possible to use an external high-quality microphone array enabling linear spherical harmonic capture additionally to the video capture means, it is more convenient to use directly the microphones mounted on a video device itself.

There exist many high-quality methods for adaptive perceptually motivated spatial audio capture. The concept as described in further detail herein is use a SPAC method in generation of spherical harmonic audio signals from a microphone array. Specifically in some embodiments to use the SPAC methods to enable spherical harmonic signal generation with a microphone array for which at least at some frequencies it is not possible to satisfactorily linearly retrieve the spherical harmonic signals.

The term SPAC is used in this document as a generalized term covering any adaptive array signal processing technique providing spatial audio capture. The methods in scope apply the analysis and processing in frequency band signals, since it is a domain that is meaningful for spatial auditory perception. Spatial metadata such as directions of the arriving sounds, and/or ratio or energy parameters determining the directionality or non-directionality of the recorded sound, are dynamically analyzed in frequency bands. The metadata is applied at the reproduction stage to dynamically synthesize spatial sound to headphones or loudspeakers with a spatial accuracy beyond that obtainable with Ambisonics using an equivalent microphone array. For example, a plane wave arriving to the array can be reproduced as a point source at the receiver end, which is comparable to the performance of very high order Ambisonic reproduction.

One method of spatial audio capture (SPAC) reproduction is Directional Audio Coding (DirAC), which is a method using sound field intensity and energy analysis to provide spatial metadata that enables the high-quality adaptive spatial audio synthesis for loudspeakers or headphones. Another example is harmonic planewave expansion (Harpex), which is a method that can analyze two plane waves simultaneously, which may further improve the spatial precision in certain sound field conditions. A further method is a method intended primarily for mobile phone spatial audio capture, which uses delay and coherence analysis between the microphones to obtain the spatial metadata, and its variant for devices containing more microphones and a shadowing body, such as OZO. Although two variants are described in the following examples, any suitable method applied to obtain the spatial metadata can be used. The concept as such is one where from the microphone signals a set of spatial metadata (such as in frequency bands the directions of the sound, and the relative amount of non-directional sound such as reverberation) is analysed from microphone audio signals, and which enable the adaptive accurate synthesis of the spatial sound.

The use of SPAC methods are also robust for small devices for two reasons: Firstly, they typically use short-time stochastic analysis, which means that the effect of noise is reduced at the estimates. Secondly, they typically are designed for analysing perceptually relevant properties of the sound field, which is the primary interest in spatial audio reproduction. The relevant properties are typically direction(s) of arriving sounds and their energies, and the amount of non-directional ambient energy. The energetic parameters can be expressed in many ways, such as in terms of a direct-to-total ratio parameter, ambience-to-total ratio parameter, or other. The parameters are estimated in frequency bands, because in such a form these parameters are particularly relevant for human spatial hearing. The frequency bands could be Bark bands, equivalent rectangular bands (ERBs), or any other perceptually motivated non-linear scale. Also linear frequency scales are applicable, although in this case it is desirable that the resolution is sufficiently fine to cover also the low frequencies at which the human hearing is most frequency selective.

The use of SPAC analysis thus provides the perceptually relevant dynamic spatial metadata, e.g. the direction(s) and energy ratio(s) in frequency bands. The SPAC synthesis refers to processing of the audio signals to obtain for the reproduced sound the perceptual spatial characteristics according to the analysed spatial metadata. For example, if the SPAC analysis provides an information that the sound in a frequency band arrives to the microphone array from a particular direction, the SPAC synthesis stage could for example apply to the signals the head-related transfer function (HRTFs) corresponding to that direction. As the result, the reproduced sound over headphones at that frequency is perceptually similar as if an actual sound would arrive at the analysed direction. The same procedure may be applied to all other frequency bands as well (usually independently), and adaptively over time.

Similarly, many SPAC analysis and synthesis methods also account for ambience signals such as reverberation, which are typically reproduced spatially spread at the synthesis stage, adaptively in time and in frequency according to the spatial metadata.

The examples thus as described with respect to <FIG>, <FIG>, <FIG> show embodiments where a SPAC method is applied to adaptively synthesize any-order spherical harmonic signals from a microphone array with which at least for some frequencies it is not possible to obtain a first order spherical harmonic representation.

For example, as described previously spatial aliasing may prevent generation of first-order spherical harmonic audio signals, or the device shape (e.g. smart phone) may prevent generation of a practically usable spherical harmonic component (due to SNR) at the axis of the narrow direction of the device.

By implementing such embodiments it may be possible to enable spatial sound reproduction through channels such as YouTube for a broad range of devices, such as OZO, mobile phones, Ricoh Theta type devices, or any other, where the prior techniques fail at least at some frequencies.

As is shown in further detail later a hybrid approach is employed according to the invention for spatial sound reproduction wherein for some frequencies and a, predetermined, spherical harmonic order the microphone audio signals are processed using linear methods while for other some frequencies and the, predetermined, spherical harmonic order the microphone audio signals are processed with dynamic (i.e. adaptive) processes. The hybrid approach can be beneficial for such configurations where for example linear methods can produce very high quality spherical harmonic components only for certain frequencies and for at least a certain spherical harmonic order.

With respect to <FIG> is shown an example audio capture and processing system <NUM> suitable for implementing some embodiments.

The system <NUM> may further comprise a spatial audio capture (SPAC) device <NUM>. The spatial audio capture device <NUM> may in some embodiments comprise a directional or omnidirectional microphone array <NUM> configured to capture an audio signal associated with a sound field represented for example by the sound source(s) and ambient sound. The spatial audio capture device <NUM> may be configured to output the captured audio signals to the processor and synthesizer <NUM>.

In some embodiments the spatial audio capture device <NUM> is implemented within a mobile device/OZO, or any other device with or without cameras. The spatial audio capture device is thus configured to capture spatial audio, which, when rendered to a listener, enables the listener to experience the spatial sound similar to that if they were present in the location of the spatial audio capture device.

The system <NUM> furthermore may comprise a processor and synthesizer <NUM> configured to receive the outputs of the microphone array <NUM> of the spatial audio capture device <NUM>.

The processor and synthesizer <NUM> may be configured to process (for example adaptively mix) the outputs of the spatial audio capture device <NUM> and output these processed signals as spherical harmonic audio signals to be stored internally or transmitted to other devices (for example to be decoded and rendered to a user). Typically, the processing is adaptive and takes place in frequency bands.

<FIG> shows an example processor and synthesizer <NUM> in further detail. The processor and synthesizer <NUM> is configured to receive audio signals/streams. For example the processor and synthesizer <NUM> may be configured to receive audio signals from the microphone array <NUM> (within the spatial audio capture device <NUM>). The input may in some embodiments be 'recorded' or stored audio signals. In some embodiments the audio input may comprise sampled audio signals and metadata describing audio source or object directions or locations, or other directional parameters such as analysed SPAC metadata, including for example directional parameters and energy ratio parameters in frequency bands. In some embodiments the audio input signal (which includes the audio input signals associated with the microphones) may comprise other optional parameters such as gain values, or equalisation filters to be applied to the audio signals.

If the input signal contains also loudspeaker signals or audio-object signals, such can be processed into the spherical harmonic signals using conventional methods, in other words, by applying the spherical harmonic transform weights according to the spatial direction(s) to the input channel signals. Such processing is straightforward and different than the SPAC processing which relies on the perceptually motivated spatial metadata analysis in frequency bands.

The processor and synthesizer <NUM> in some embodiments comprises a filter-bank <NUM>. The filter-bank <NUM> enables the time domain microphone audio signals to be transformed into frequency band signals. As such any suitable time to frequency domain transform may be applied to the microphone signals. A typical filter-bank which may be implemented in some embodiments is a short-time Fourier transform (STFT), involving an analysis window and FFT. Other suitable transforms in place of the STFT may be a complex-modulated quadrature mirror filter (QMF) bank. The filter-bank may produce complex-valued frequency band signals, indicating the phase and the amplitude of the input signals as a function of time and frequency. The filter bank may be uniform in its frequency resolution which enables highly efficient signal processing structures. However uniform frequency bands may be grouped into a non-linear frequency resolution approximating a spectral resolution of human spatial hearing.

For example where the microphone array <NUM> of the spatial audio capture device <NUM> comprises M microphones. The filter-bank <NUM> may receive microphone signals x(m,n'), where m and n' are indices for microphone and time respectively and transform the input signals into the frequency band signals by means of a short time Fourier transform <MAT> where X denotes the transformed frequency band signals, and k denotes the frequency band index, and n denotes the time index.

These signals may then be output to the synthesizer <NUM> and to the analyzer <NUM>.

The processor and synthesizer <NUM> in some embodiments comprises the analyser <NUM> which is configured to analyse the audio signals from the filter-bank <NUM> and determine spatial metadata associated with the sound field at the recording position.

The SPAC analysis (any such technique) may be applied on the frequency band signals (or groups of them) to obtain the spatial metadata. A typical example of the spatial metadata is direction(s) and direct-to-total energy ratio(s) at each frequency interval and at each time frame. For example, it is an option to retrieve the directional parameter based on inter-microphone delay-analysis, which in turn can be performed for example by formulating the cross-correlation of the signals with different delays and finding the maximum correlation. Another method to retrieve the directional parameter is to use the sound field intensity vector analysis, which is the procedure applied in Directional Audio Coding (DirAC).

At the higher frequencies (above spatial aliasing frequency) it is an option to use the device acoustic shadowing for some devices such as OZO to obtain the directional information. The microphone signal energies are typically higher at that side of the device where most of the sound arrives, and thus the energy information can provide an estimate for the directional parameter.

There are many further methods in the field of array signal processing to estimate the direction-of-arrival.

It is also an option to use inter-microphone coherence analysis to estimate the amount of the non-directional ambience at each time-frequency interval (in other words, the energy ratio parameter). The ratio parameter can be estimated also with other methods, such as using a stability measure of the directional parameter, or similar. The specific method applied to obtain the spatial metadata is not of main interest in the present scope.

In this section, one method using delay estimation based on correlation between audio input signal channels is described. In this method the direction of arriving sound is estimated independently for B frequency domain subbands. The idea is to find at least one direction parameter for every subband which may be a direction of an actual sound source, or a direction parameter approximating the combined directionality of multiple sound sources. For example, in some cases the direction parameter may point directly towards a single active source, while in other cases, the direction parameter may, for example, fluctuate approximately in an arc between two active sound sources. In presence of room reflections and reverberation, the direction parameter may fluctuate more. Thus, the direction parameter can be considered a perceptually motivated parameter: Although for example one direction parameter at a time-frequency interval with several active sources may not point towards any of these active sources, it approximates the main directionality of the spatial sound at the recording position. Along with the ratio parameter, this directional information roughly captures the combined perceptual spatial information of the multiple simultaneous active sources. Such analysis is performed each time-frequency interval, and as the result the spatial aspect of the sound is captured in a perceptual sense. The directional parameters fluctuate very rapidly, and express how the sound energy fluctuates through the recording position. This is reproduced for the listener, and the listener's hearing system then gets the spatial perception. In some time-frequency occurrences one source may be very dominant, and the directional estimate points exactly to that direction, but this is not a general case.

The frequency band signal representation is denoted as X(k,m,n) where m is the microphone index, k the frequency band index {k = <NUM>,. , N-<NUM>} and where N is the number of frequency bands of the time-frequency transformed signals. The frequency band signal representation is grouped into B subbands, each of which has a lower frequency band index <MAT> and an upper frequency band index <MAT>. The widths of the subbands ( <MAT>) can approximate, for example, the ERB (equivalent rectangular bandwidth) scale or the Bark scale.

The directional analysis may feature the following operations. In this case, we assume a flat mobile device with three microphones. This configuration can provide the analysis of the directional parameter in the horizontal plane, and a ratio parameter, or similar.

First the horizontal direction is estimated with two microphone signals (in this example microphones <NUM> and <NUM> being located in the horizontal plane of the capture device at the opposing edges of the device). For the two input microphone audio signals, the time difference between the frequency-band signals in those channels is estimated. The task is to find delay τb that maximizes the correlation between two channels for subband b.

The frequency band signals X(k,m,n) can be shifted τb time domain samples using <MAT>.

Where fk is the center frequency of band k, and fs is the sampling rate. The optimal delay for subband b and time index n is then obtained from <MAT> where Re indicates the real part of the result and * denotes complex conjugate, and Dmax is the maximum delay in samples, which can be a fractional number, and occurs when the sound arrives exactly at the axis determined by the microphone pair. Although an example of delay estimation over one time index n is exemplified above, in some embodiments the estimation of the delay parameter may be performed over several indices n by averaging or adding the estimates also in that axis. For τb the resolution of approximately one sample is for many smart phones satisfactory for the search of the delay. Also other perceptually motivated similarity measures than correlation can be used.

A 'sound source', which is a representation of the audio energy captured by the microphones, thus may be considered to create an event described by an exemplary time-domain function which is received at a microphone for example a second microphone in the array and the same event received by a third microphone. In an ideal scenario, the exemplary time-domain function which is received at the second microphone in the array is simply a time shifted version of the function received at the third microphone. This situation is described as ideal because in reality the two microphones will likely experience different environments for example where their recording of the event could be influenced by constructive or destructive interference or elements that block or enhance sound from the event, etc..

The shift τb indicates how much closer the sound source is to the second microphone than the third microphone (when τb is positive, the sound source is closer to the second microphone than the third microphone). The between -<NUM> and <NUM> normalized delay can be formulated as <MAT>.

Utilizing basic geometry, and assuming that the sound is a plane wave arriving at the horizontal plane, it can be determined that the horizontal angle of the arriving sound is equal to <MAT>,.

Notice that there are two alternatives for the direction of the arriving sound as the exact direction cannot be determined with only two microphones. For example, a source at a mirror-symmetric angle at the front or rear of the device may produce the same inter-microphone delay estimate.

A further microphone, for example a first microphone in an array of three microphones, can then be utilized to define which of the signs (the + or -) is correct. This information can be obtained in some configurations by estimating the delay parameter between a microphone pair having one (e.g. the first microphone) at the rear side of the smart phone, and another (e.g. the second microphone) at the front side of the smart phone. The analysis at this thin axis of the device may be noisy to produce reliable delay estimates. However, the general tendency if the maximum correlation is found at the front side or the rear side of the device may be robust. With this information the ambiguity of the two possible directions can be resolved. Also other methods may be applied for resolving the ambiguity.

The same estimation is repeated for each subband.

An equivalent method can be applied to microphone arrays where there is both 'horizontal' and 'vertical' displacement in order that the azimuth and elevation can be determined. For devices or smartphones with four or more microphones (which are displaced from each other in a plane perpendicular to the directions described above) it may be also possible to perform elevation analysis. In that case, for example, the delay analysis can be formulated first in the horizontal plane and then in the vertical plane. Then, based on the two delay estimates one can find an estimated direction of arrival. For example, one may perform a delay-to-position analysis similar to that in GPS positioning systems. In this case also, there is a directional front-back ambiguity, which is solved for example as described above.

In some embodiments the ratio metadata expressing the relative proportions of non-directional and directional sound may be generated according to the following method:.

The resulting ratio parameter is then truncated between <NUM> and <NUM>. With such an estimate method:.

The above simple formulation provides an approximation of the ratio parameter. At the extremes (the fully directional and fully non-directional sound field conditions) the estimate is true. The ratio estimate between extremes may have some bias depending on the sound arrival angle. Nevertheless, the above formulation can be demonstrated to be satisfactorily accurate in practice also in these conditions. Other methods to generate the directional and ratio parameters (or other spatial metadata depending on the applied analysis technique) are also applicable.

The aforementioned method in the class of SPAC analysis methods is intended for primarily flat devices such as smart phones: The thin axis of the device is determined suitable only for the binary front-back choice, because more accurate spatial analysis may not be robust at that axis. The spatial metadata is analysed primarily at the longer axes of the device, using the aforementioned delay/correlation analysis, and directional estimation accordingly.

A further method to estimate the spatial metadata is described in the following, providing an example of the practical minimum of two microphone channels. Two directional microphones having different directional patterns may be placed, for example <NUM> apart. Equivalently to the previous method, two possible horizontal directions of arrival can be estimated using the microphone-pair delay analysis. The front-back ambiguity can then be resolved using the microphone directivity: If one of the microphones has more attenuation towards the front, and the other microphone has more attenuation towards the back, the front-back ambiguity can be resolved for example by measuring the maximum energy of the microphone frequency band signals. The ratio parameter can be estimated using correlation analysis between the microphone pair, for example, using a similar method than as described previously.

Clearly, other spatial audio capture methods can also be suitable for obtaining the spatial metadata. In particular, for non-flat devices such as spherical devices, other methods may be more suitable, for example, by enabling higher robustness for the parameter estimation. A well-known example in the literature is Directional Audio Coding (DirAC), which in its typical form comprises of the following steps:.

Thus, in one embodiment the spatial analysis according to the DirAC paradigm can be applied to produce the spatial metadata, thus ultimately enabling the synthesis of the spherical harmonic signals. In other words, a directional parameter and a ratio parameter can be estimated by several different methods.

For further clarification of the aforementioned processing steps in DirAC analysis, let us specify the difference of the input B-format (i.e. spherical harmonic or Ambisonic format) signal and the reproduced output spherical harmonic signal of an overall embodiment. The input B-format signal may have excessive noise at low frequencies for the X,Y,Z components, for example, if the signals have been retrieved from a compact microphone array. The noise, however, has only a minor impact to the DirAC spatial metadata analysis, since the metadata is analysed from the short-time stochastic estimates. In specific, the stochastic analysis reduces the effect of the noise at the estimates. Therefore, an embodiment using the DirAC analysis technique could <NUM>) robustly estimate the directional parameters, and <NUM>) using the available high-SNR W-signal (the zeroth order signal) synthesize the spherical harmonic output signals. Thus, the output spherical harmonic signals may have a higher perceived fidelity than the input spherical harmonic signals.

The processor and synthesizer <NUM> in some embodiments comprises a synthesizer <NUM>. The synthesizer <NUM> may be configured to receive the frequency band signal representations and the spatial metadata and be configured to generate spherical harmonic signals. The synthesizer <NUM> is described in further detail with respect to the examples shown in <FIG>. In some embodiments the spherical harmonic frequency band signals are output to an inverse filter bank <NUM>. Although the synthesizer <NUM> may operate fully in the frequency domain such as shown in <FIG> it may in some embodiments, such as shown in the example shown in <FIG> below, operate partially in the frequency band domain and partially in the time domain.

For example the synthesizer <NUM> may comprise a first or frequency band domain part which outputs a frequency band domain signal to the inverse filter bank <NUM> and a second or time domain part which receives a time domain signal from the inverse filter bank <NUM> and outputs suitable time domain spherical harmonic signals.

The processor and synthesizer <NUM> in some embodiments comprises an inverse filter-bank <NUM>. The inverse filter-bank <NUM> may receive the generated spherical harmonic frequency band signals and perform a frequency to time domain transform on them in order to generate time domain representations of the spherical harmonic signals.

With respect to <FIG> a first example of a synthesizer <NUM> is shown. This synthesizer example is configured such that having the spatial metadata available from the SPAC analysis, the synthesizer first synthesizes an intermediate virtual multichannel loudspeaker signal, for example, <NUM> virtual loudspeaker channels covering a sphere in 3D and to this signal apply a spherical harmonic transform.

The synthesizer <NUM> may thus comprise a directional divider <NUM>. The directional divider <NUM> may be configured to receive the frequency band representations and the ratio values associated with the directional components of the audio signals. The directional divider <NUM> may then apply the ratio values to each band in order to generate a directional and non-directional (or ambient) part of the audio signals. For example, multipliers as a function of the ratio parameters may be formulated and applied to the input frequency band signals to generate the directional and non-directional parts. The directional part may be passed to an amplitude panning synthesizer <NUM> and the non-directional part may be passed to a decorrelation synthesizer <NUM>.

The synthesizer <NUM> may further comprise an amplitude panning synthesizer <NUM>. The amplitude panning synthesizer <NUM> is configured to receive the directional part of the audio signals and furthermore the directional information part of the spatial metadata and from these generate or synthesize 'virtual' loudspeaker signals. In some embodiments there are <NUM> 'virtual' loudspeaker channels arranged in a 3D space. The <NUM> channels may for example be located such that there are <NUM> channels arranged in a horizontal plane, <NUM> channels located above the plane and <NUM> channels located below). However, this is only an example and there may be implemented any other number or arrangement of virtual loudspeaker channels.

The amplitude panning synthesizer may, for example, apply vector-base amplitude panning (VBAP) to reproduce the direct part of the sound at the direction determined by the spatial metadata, at each frequency band. The virtual loudspeaker signals may then be output to a combiner <NUM>. Although the virtual loudspeaker signals may be generated by VBAP any other suitable virtual channel signal generation method may be employed. The term 'virtual' refers to that the loudspeaker signals are an intermediate representation.

The synthesizer <NUM> may further comprise a decorrelation synthesizer <NUM>. The decorrelation synthesizer <NUM> may be configured to receive the non-directional part of the audio signal and generate an ambient or non-directional component for combining within the virtual loudspeaker signals. For example the ambient part can be synthesized for example using decorrelators to spread the sound energy to all or many of the virtual loudspeakers. The ambient part may be output to the combiner <NUM>.

The synthesizer <NUM> may further comprise a combiner <NUM>. The combiner <NUM> may be configured to receive the virtual loudspeaker signals and the ambient part and generate a combined directional and ambient representation using the virtual loudspeaker arrangement. This combined virtual loudspeaker frequency band representation may be passed to the inverse filter-bank <NUM>.

The inverse filter-bank <NUM> may in this arrangement pass the time domain signals associated with the virtual loudspeaker representation to a spherical harmonic transformer <NUM>.

The synthesizer <NUM> may further comprise a spherical harmonic transformer <NUM>. The spherical harmonic transformer <NUM> may be configured to receive the time domain signals associated with the virtual loudspeaker representation and transform the virtual loudspeaker signals into spherical harmonic components by any known method. For example each virtual loudspeaker signal is weighted (with a specific weighting) and output to each of the spherical harmonic outputs. The weights can be applied for wide-band signals. The weights are formulated as a function of the azimuths and elevations of the virtual loudspeakers.

Although the example shown in <FIG> shows the generation of the spherical harmonic transform in the time domain it is understood that in some embodiments the spherical harmonic transform is applied in the frequency domain (or frequency band domain). In other words the spherical harmonic transformer <NUM> is a frequency band signal transformer and is located before the inverse filter bank <NUM> and after the combiner <NUM>. The weights can be applied in this example to the frequency band signals.

With respect to <FIG> a second example synthesizer <NUM> is shown. In this example the spherical harmonic signals could be synthesized (using the spatial metadata) directly, i.e., without an intermediate virtual loudspeaker layout representation.

The synthesizer <NUM> may thus comprise a directional divider <NUM>. The directional divider <NUM> may be configured to receive the frequency band representations and the ratio values associated with the directional components of the audio signals. The directional divider <NUM> may then apply the ratio values to each band in order to generate a directional and non-directional (or ambient) part of the audio signals. The directional part may be passed to a moving source synthesizer <NUM> and the non-directional part may be passed to a decorrelation synthesizer <NUM>.

The synthesizer <NUM> may further comprise a moving source synthesizer <NUM>. The moving source synthesizer <NUM> is configured to receive the directional part of the audio signals and furthermore the directional information part of the spatial metadata and from these generate spherical harmonic transform weights associated with the moving source being modelled based on the directional analysis. For example, the directional part(s) of the audio signals can be considered as virtual moving source(s). The directional metadata may determine the direction of the moving source, and the energetic metadata (e.g. ratio parameter) determines the amount of the energy that is reproduced at that direction. In some embodiments the directional estimates are smoothed (for example low-pass filtered over time or over frequency bands) in order to reduce sudden audible fluctuations in the output. The location of the virtual source may therefore potentially change at every time instant of each frequency band signal. Since the direction of the virtual moving source can potentially vary as a function of frequency, the spherical harmonic transform is performed for each frequency band independently and the spherical harmonic weights, which this time are adaptive in time and in frequency can be generated and passed to a spherical harmonic transformer <NUM> together with the audio signals.

The synthesizer <NUM> in some embodiments comprises a spherical harmonic transformer <NUM> configured to receive the determined weights and audio signals and generate the directional part of the frequency band spherical harmonic signals. The directional part of the frequency band spherical harmonic signals may then be passed to a combiner <NUM>. In some embodiments the operations of the moving source synthesizer <NUM> and the spherical harmonic transformer <NUM> can be performed in a single operation or module.

The synthesizer <NUM> may further comprise a decorrelation synthesizer <NUM>. The decorrelation synthesizer <NUM> may be configured to synthesize the ambient parts of the signal energy directly. This can be performed because according to the definition of spherical harmonic signals they are mutually incoherent in ideal ambience or diffuse sound fields, e.g. in reverberation. Thus, it is possible to synthesize the ambience portion by decorrelating the input microphone frequency band signals to obtain the incoherent spherical harmonic frequency band signals. These signals may be weighted weights for each of the spherical harmonic coefficients. These spherical harmonic coefficient based weights are scalars as a function of the spherical harmonic order, and depend of the applied normalization scheme. An example normalization scheme is such that for the ambience each of the spherical harmonic (SH) orders have in total the same signal energy. Thus if the zeroth order has <NUM> unit of energy, the three first order SH signals would have <NUM>/<NUM> units of energy each, the five second order SH signals would have <NUM>/<NUM> units of energy, and so forth. The ambient part may furthermore be output to the combiner <NUM>. It is understood that the normalization scheme does not apply only for the ambience part, but the same weighting is incorporated as part of the formulation of the spherical transform coefficients for the direct signal part.

The synthesizer <NUM> may further comprise a combiner <NUM>. The combiner <NUM> may be configured to receive the ambience and directional parts of the directly determined spherical harmonic signals and combine these to generate a combined frequency domain spherical harmonic signal. This combined spherical harmonic frequency band representation may be passed to the inverse filter-bank <NUM>.

The inverse filter-bank <NUM> may in this arrangement output the time domain spherical harmonic representation.

With respect to <FIG> a third example synthesizer <NUM> is shown. In this example an optimized mixing technique, such as a least-squares optimized solution, is used to generate the spherical harmonic signals based on the spatial metadata and the microphone signals in frequency bands. This approach differs from the previous examples, since it.

The synthesizer <NUM> may comprise a short time stochastic analyser <NUM>. The short time stochastic analyser <NUM> is configured to receive the frequency domain representations and perform the short-time stochastic analysis in order to determine the covariance matrix for the frequency band microphone signals. The covariance matrix may be passed to the least squares optimized matrix generator <NUM>.

The synthesizer <NUM> may comprise a target stochastic property determiner <NUM>. The target stochastic property determiner <NUM> may be configured to determine the intended covariance matrix for the spherical harmonic signals based on the spatial metadata and overall frequency band energy information obtained from the short-time stochastic analysis. The intended target covariance matrix for the spherical harmonic signals can be obtained by first formulating the covariance matrix for the direct energy portion corresponding to the direction determined by the spatial metadata, second by formulating the covariance matrix for the ambience (or non-directional) energy portion, and combining these matrices to form the intended target covariance matrix. The ambience portion covariance matrix is a diagonal matrix, which expresses that the spherical harmonic signals for ambience are mutually incoherent. The relative energies of the diagonal coefficients are according to the normalization scheme as described previously. Similarly, the direct part covariance matrix is formulated using the spherical harmonic weights (being affected by normalization scheme) according to the analysed spatial metadata.

This target property may then be passed to the least squares optimized matrix generator <NUM>.

The least squares optimized matrix generator <NUM> may take the stochastic estimates from the short time stochastic analyser <NUM> and the target property from the property determiner <NUM> and apply a least squares (or other suitable optimization) method to determine suitable mixing coefficients which may be passed to a signal mixer and decorrelator <NUM>. An example implementation would in other words perform the short-time stochastic (covariance matrix) analysis for the frequency band microphone signals, formulate the intended target covariance matrix for the spherical harmonic output signals, and obtain processing gains based on at least these two matrices using the least squares optimized matrix generator <NUM> (for example using a method as described in, or similar to the method described in, <CIT>). The resulting processing gains are used as weighting values to be applied by the signal mixer and decorrelator <NUM>.

These embodiments can thus be applied in order to synthesize the spherical harmonic signals from the microphone signals. The output of the signal mixer and decorrelator <NUM> is passed to the inverse filter-bank <NUM>.

According to the invention, a hybrid approach is implemented where for some frequencies the apparatus would use traditional linear methods, and at other frequencies the SPAC methods as described above would be used, to obtain the spherical harmonic components. For example, for a Nokia OZO device linear methods could be used to obtain up to first order spherical harmonics approximately at frequencies <NUM>-<NUM>, and SPAC methods at the other frequencies.

A block diagram of a hybrid configuration is shown in <FIG>.

The system comprises a frequency band router configured to direct some of the frequency band representations to an adaptive spherical harmonic signal generator or synthesizer <NUM> which may be any of the example adaptive harmonic signal synthesizers <NUM> as shown in <FIG>, and some of the frequency band representations to a linear spherical harmonic signal generator <NUM>.

The outputs of the adaptive spherical harmonic signal generator or synthesizer <NUM> and linear spherical harmonic signal generator <NUM> are then passed to a combiner <NUM> which then outputs the combined spherical harmonic audio signal representation to the inverse filter-bank <NUM>. The combination may require temporal alignment of the signals if the adaptive and linear processing have different latencies.

In other words part of the frequency bands are processed with adaptive methods and other frequency bands are processed with linear methods.

In some embodiments the hybrid approach such as shown in <FIG> may be applied to a spatial division as well as frequency division of the audio signals. Thus linear methods in embodiments not forming part of the claimed invention may be used to obtain some lower orders of the spherical harmonics, and to use the adaptive SPAC-type methods such as described to synthesize the higher orders of spherical harmonics. For example, for a Nokia OZO device, at approximately <NUM>-<NUM>, linear approach may be used to obtain the 0th and the 1st order spherical harmonics, and the SPAC approach to synthesize the 2nd order spherical harmonics, or also higher orders.

In some embodiments not forming part of the claimed invention both the adaptive synthesizer and linear method synthesizer may be implemented to function sequentially. For example, at <NUM>-<NUM> the apparatus may first generate the 1st order spherical harmonic signals and, based on the 1st order spherical signals synthesize the higher orders using adaptive methods known in the art, or, above the spatial aliasing frequency (~<NUM> for OZO), apply the adaptive methods described herein. Generating an intermediate 1st order signal representation at some frequencies (and thus utilizing the prior art) may be an optional step.

According to the invention, the produced spherical harmonic signal is of a pre-determined order. First, second, third or higher order harmonics are possible. Furthermore, it is understood that a mixed-order output can also be provided. For example, in some cases, not all spherical harmonic output signals for some of the orders are processed. By way of example, in some use cases it may be desirable to have a higher order spherical harmonic representation at the horizontal directions than at the vertical directions. One such a use case is when the spherical harmonic signals are known to be decoded for a loudspeaker setup with mostly horizontal loudspeakers.

In some embodiments the hybrid approach could be applied based on the spatial axis of the device. For example, a mobile phone having an irregular array may therefore have different dimensions at different axes. Therefore, at different axes the hybrid approach could be applied differently, or used only for some of the axes. For example, at the width axis of a smart phone, one could use a linear method at some frequencies to obtain the first order spherical harmonic signals, while in the thin axis of a smart phone, the SPAC methods are applied to form all orders of spherical harmonic signals above the zeroth order.

The general motivation for implementing a hybrid approach is primarily because of the simplicity of the linear methods: Although linear methods are not applicable for typical microphone arrays for a wide bandwidth, nor to produce high orders of SH coefficients, at their typical operational range they may be robust and computationally light. Thus, the hybrid approach may be a preferable configuration for some devices.

The hybrid approach may require an alignment between the linear and non-linear signal components in terms of time and/or phase, to avoid any temporal or spectral artefacts. This is since the linear methods may have a different and typically smaller latency than the adaptive methods.

In some embodiments the spatial metadata may be analysed based on at least two microphone signals of a microphone array, and the spatial synthesis of the spherical harmonic signals may be performed based on the metadata and at least two microphone signals in the same array. For example, with a smartphone, all or some of the microphones could be used for the metadata analysis, and for example only the front microphone could be used for the synthesis of the spherical harmonic signals in an embodiment not forming part of the claimed invention.

However, it is understood that the microphones being used for the analysis may in some embodiments be different than the microphones being used for the synthesis in embodiments not forming part of the claimed invention.

The microphones could also be a part of a different device. For example, it could be that the spatial metadata analysis is performed based on the microphone signals of a presence capture device with a cooling fan. Although the metadata is obtained, these microphone signals could be of low fidelity due to, by way of example, fan noise. In such a case, one or more microphones could be placed externally to the presence capture device. The signals from these external microphones could be processed according to the spatial metadata obtained using the microphone signals from the presence capture device in embodiments not forming part of the claimed invention.

There are various configurations that may be used to obtain the microphone signals.

It is also understood that any of the microphone signals discussed herein may be pre-processed microphone signals. For example, a microphone signal could be an adaptive or non-adaptive combination of actual microphone signals of a device. For example, there could be several microphone capsules nearby each other that are combined to provide a signal with an improved SNR.

The microphone signals could also be pre-processed, such as adaptively or non-adaptively equalized, or processed with noise-removal processes. Furthermore, the microphone signals may in some embodiments be beamform signals, in other words, spatial capture pattern signals that are obtained by combining two or more microphone signals.

It is thus understood that there are many configurations, devices, and approaches to obtain the microphone signals for the processing according to the methods provided herein.

In some embodiments not forming part of the claimed invention, there may be only one microphone or audio signal, and the associated spatial metadata has been analysed previously. For example, it may be that after the analysis of the spatial metadata using at least two microphones the number of microphone signals has been reduced for transmission or storage, for example to only one channel. After the transmission, in such an example configuration, the decoder receives only one audio channel and the spatial metadata, and then performs the spatial synthesis of the spherical harmonic signals using the methods provided herein. Clearly, there could be also two or more transmitted audio signals, and the previously analysed metadata can also in such cases be applied at the adaptive synthesis of the spherical harmonic signals.

In some embodiments not forming part of the claimed invention the spatial metadata is analyzed from at least two microphone signals, and the metadata along with at least one audio signal are transmitted to a remote receiver, or stored. In other words, the audio signals and the spatial metadata may be stored or transmitted in an intermediate format that is different than the spherical harmonic signal format. The format, for example, may feature lower bit rate than the spherical harmonic signal format. The at least one transmitted or stored audio signal can be based on the same microphone signals using which the spatial metadata was also obtained, or based on signals from other microphones in the sound field. At a decoder, the intermediate format may be transcoded into a spherical harmonic signal format, thus enabling the compatibility with services such as YouTube. In other words, at a receiver or a decoder, the transmitted or stored at least one audio channel is processed to a spherical harmonic audio signal representation utilizing the associated spatial metadata and using the methods described herein. While transmitted or stored, in some embodiments the audio signal(s) may be encoded, for example, using AAC. In some embodiments the spatial metadata may be quantized, encoded and/or embedded to the AAC bit stream. In some embodiments the AAC or otherwise encoded audio signals and the spatial metadata may be embedded into a container such as the MP4 media container. In some embodiments the media container, being for example MP4, may include a video stream, such as an encoded spherical panoramic video stream. Many other configurations to transmit or store the audio signals and the associated spatial metadata exist.

Regardless of the applied methods to transmit or store the audio signals and the spatial metadata, at the receiver (or decoder or processor) the methods described herein provide the means to generate the spherical harmonic signals adaptively based on the spatial metadata and at least one audio signal. In other words, for the methods presented herein, it is in practice not relevant if the audio signals and/or the spatial metadata are obtained from the microphone signals directly, or indirectly, for example, through encoding, transmission/storing and decoding. With respect to <FIG> an example electronic device <NUM> which may be used as at least part of the processor and synthesizer <NUM> or as part of the system <NUM> is shown. The device may be any suitable electronics device or apparatus. For example in some embodiments the device <NUM> is a virtual or augmented reality capture device, a mobile device, user equipment, tablet computer, computer, audio playback apparatus, etc..

The device <NUM> may comprise a microphone array <NUM>. The microphone array <NUM> may comprise a plurality (for example a number M) of microphones. However it is understood that there may be any suitable configuration of microphones and any suitable number of microphones. In some embodiments the microphone array <NUM> is separate from the apparatus and the audio signals transmitted to the apparatus by a wired or wireless coupling. The microphone array <NUM> may in some embodiments be the SPAC microphone array <NUM> as shown in <FIG>.

The microphones may be transducers configured to convert acoustic waves into suitable electrical audio signals. In some embodiments the microphones can be solid state microphones. In other words the microphones may be capable of capturing audio signals and outputting a suitable digital format signal. In some other embodiments the microphones or microphone array <NUM> can comprise any suitable microphone or audio capture means, for example a condenser microphone, capacitor microphone, electrostatic microphone, Electret condenser microphone, dynamic microphone, ribbon microphone, carbon microphone, piezoelectric microphone, or microelectrical-mechanical system (MEMS) microphone. The microphones can in some embodiments output the audio captured signal to an analogue-to-digital converter (ADC) <NUM>.

The device <NUM> may further comprise an analogue-to-digital converter <NUM>. The analogue-to-digital converter <NUM> may be configured to receive the audio signals from each of the microphones in the microphone array <NUM> and convert them into a format suitable for processing. In some embodiments where the microphones are integrated microphones the analogue-to-digital converter is not required. The analogue-to-digital converter <NUM> can be any suitable analogue-to-digital conversion or processing means. The analogue-to-digital converter <NUM> may be configured to output the digital representations of the audio signals to a processor <NUM> or to a memory <NUM>.

The processor <NUM> can be configured to execute various program codes. The implemented program codes can comprise, for example, SPAC analysis, and synthesizing such as described herein.

In some embodiments the device <NUM> comprises a memory <NUM>. In some embodiments the at least one processor <NUM> is coupled to the memory <NUM>. The memory <NUM> can be any suitable storage means. In some embodiments the memory <NUM> comprises a program code section for storing program codes implementable upon the processor <NUM>. Furthermore in some embodiments the memory <NUM> can further comprise a stored data section for storing data, for example data that has been processed or to be processed in accordance with the embodiments as described herein. The implemented program code stored within the program code section and the data stored within the stored data section can be retrieved by the processor <NUM> whenever needed via the memory-processor coupling.

In some embodiments the device <NUM> comprises a user interface <NUM>. The user interface <NUM> can be coupled in some embodiments to the processor <NUM>. In some embodiments the processor <NUM> can control the operation of the user interface <NUM> and receive inputs from the user interface <NUM>. In some embodiments the user interface <NUM> can enable a user to input commands to the device <NUM>, for example via a keypad. In some embodiments the user interface <NUM> can enable the user to obtain information from the device <NUM>. For example the user interface <NUM> may comprise a display configured to display information from the device <NUM> to the user. The user interface <NUM> can in some embodiments comprise a touch screen or touch interface capable of both enabling information to be entered to the device <NUM> and further displaying information to the user of the device <NUM>.

In some implementations the device <NUM> comprises a transceiver <NUM>. The transceiver <NUM> in such embodiments can be coupled to the processor <NUM> and configured to enable a communication with other apparatus or electronic devices, for example via a wireless communications network. The transceiver <NUM> or any suitable transceiver or transmitter and/or receiver means can in some embodiments be configured to communicate with other electronic devices or apparatus via a wire or wired coupling.

The transceiver <NUM> can communicate with further apparatus by any suitable known communications protocol. For example in some embodiments the transceiver <NUM> or transceiver means can use a suitable universal mobile telecommunications system (UMTS) protocol, a wireless local area network (WLAN) protocol such as for example IEEE <NUM>. X, a suitable short-range radio frequency communication protocol such as Bluetooth, or infrared data communication pathway (IRDA).

In some embodiments the device <NUM> may be employed as a synthesizer apparatus. As such the transceiver <NUM> may be configured to receive the audio signals and determine the spatial metadata such as position information and ratios, and generate a suitable audio signal rendering by using the processor <NUM> executing suitable code. The device <NUM> may comprise a digital-to-analogue converter <NUM>. The digital-to-analogue converter <NUM> may be coupled to the processor <NUM> and/or memory <NUM> and be configured to convert digital representations of audio signals (such as from the processor <NUM> following an audio rendering of the audio signals as described herein) to a suitable analogue format suitable for presentation via an audio subsystem output. The digital-to-analogue converter (DAC) <NUM> or signal processing means can in some embodiments be any suitable DAC technology.

Furthermore the device <NUM> can comprise in some embodiments an audio subsystem output <NUM>. An example, such as shown in <FIG>, may be where the audio subsystem output <NUM> is an output socket configured to enabling a coupling with headphones <NUM>. However the audio subsystem output <NUM> may be any suitable audio output or a connection to an audio output. For example the audio subsystem output <NUM> may be a connection to a multichannel speaker system. In order to be reproduced over loudspeaker or headphones, the spherical audio signals described earlier are first decoded using a spherical harmonic decoder (or Ambisonics decoder). There are Ambisonics decoders for both loudspeaker playback as well as binaural headphone playback.

In some embodiments the digital to analogue converter <NUM> and audio subsystem <NUM> may be implemented within a physically separate output device. For example the DAC <NUM> and audio subsystem <NUM> may be implemented as cordless earphones communicating with the device <NUM> via the transceiver <NUM>.

Although the device <NUM> is shown having both audio capture and audio rendering components, it would be understood that in some embodiments the device <NUM> can comprise just the audio capture or audio render apparatus elements.

The embodiments of this invention may be implemented by computer software executable by a data processor of the electronic device, such as in the processor entity, or by hardware, or by a combination of software and hardware.

Claim 1:
Apparatus comprising means configured to:
receive at least two microphone audio signals;
obtain spatial metadata, the spatial metadata being perceptually relevant spatial information from dynamic analysis of one or more frequency bands of the at least two microphone audio signals; and
synthesize adaptively (<NUM>, <NUM>), for at least one, predetermined, order of spherical harmonic audio signals, spherical harmonic audio signals based on a first frequency band part of the at least two microphone audio signals and the spatial metadata;
synthesize (<NUM>), for the least one order of spherical harmonic audio signals, spherical harmonic audio signals using linear operations for a second frequency band part of the at least two microphone audio signals;
combine (<NUM>) the adaptively synthesized and linear operations synthesized spherical harmonic audio signals to output a combined at least one, predetermined, order of spherical harmonic audio signals.