Patent Description:
The capture of audio signals from multiple sources and mixing of those audio signals when these sources are moving in the spatial field requires significant manual effort. For example the capture and mixing of an audio signal source such as a speaker or artist within an audio environment such as a theatre or lecture hall to be presented to a listener and produce an effective audio atmosphere requires significant investment in equipment and training.

A commonly implemented system is one where one or more close or external microphones (for example Lavalier microphones worn by users or attached to a boom pole to capture audio signals) generate audio signals which are to be mixed with a spatial (or environmental or audio field) audio signal such that the produced source appears to come from an intended direction. As would be expected positioning a sound source within the spatial audio field requires significant time and effort to do manually.

A problem with mixing or processing of the close microphone signals is one of identifying at the mixer or processor an association between a received close microphone audio signal and the received or determined position of a close microphone. In other words how to identify or verify that a received audio signal is associated with a determined or received position. A proposed system is one where each close microphone is associated with a position tag. The position tag has a unique identifier and is configured to broadcast a radio frequency signal which may be received by a tag receiver and the position of the close microphone may be determined from the received signal. An example of a tag based system is the Nokia high accuracy indoor positioning (HAIP) system.

A tag system however has one problem in that although it is possible to estimate the position of each tag accurately it still requires significant manual effort to determine which audio signal or audio track is associated with the direction. In other words a system operator needs to manually assign tags to each audio channel, which is laborious and error prone. Since the microphones and tags are not physically attached, it is very easy to get them mixed up. This is especially true for concert type events where there are several musical acts performing one after the other. One act leaving the stage and another entering quickly can easily lead to the wrong tag being associated with a close microphone.

An automatic association method for linking close microphones/audio sources and tags would solve these problems.

<CIT> discusses a system which tracks a social interaction between a plurality of participants, includes a fixed beamformer that is adapted to output a first spatially filtered output and configured to receive a plurality of second spatially filtered outputs from a plurality of steerable beamformers. Each steerable beamformer outputs a respective one of the second spatially filtered outputs associated with a different one of the participants. The system also includes a processor capable of determining a similarity between the first spatially filtered output and each of the second spatially filtered outputs. The processor determines the social interaction between the participants based on the similarity between the first spatially filtered output and each of the second spatially filtered outputs.

"<NPL>) discusses a multimodal system to track the participants and identify the active speaker in the smart meeting room. The system, Cicada, is used to obtain the location and identity information of the participants. The system comprises four eight-element microphone arrays are used to estimate TDOA (Time Difference of Arrival) of the audio signals, which are indicators of the location of the speaker. Integrating these two cues from Cicada and microphone arrays into a statistical framework, results in a more robust solution to speaker localization and identification.

There is provided according to a first aspect an apparatus according to claim <NUM>.

According to a second aspect there is provided a method as disclosed in claim <NUM>.

A computer program product stored on a medium may cause an apparatus to perform the method as described herein.

An electronic device may comprise apparatus as described herein.

A chipset may comprise apparatus as described herein.

The following describes in further detail suitable apparatus and possible mechanisms for the provision of effective automatic tag assignment to close microphone/audio source audio signals. In the following examples, audio signals and audio capture signals are described. However it would be appreciated that in some embodiments the apparatus may be part of any suitable electronic device or apparatus configured to capture an audio signal or receive the audio signals and other information signals.

This problem is shown with respect to <FIG>. In <FIG> is shown a series of close microphones (with associated high accuracy indoor positioning HAIP tags). For example <FIG> shows a first close microphone (close microphone <NUM>) and an associated tag (HAIP tag <NUM>) which are referenced as <NUM>. Further shown are a second close microphone (close microphone <NUM>) and an associated tag (HAIP tag <NUM>) which are referenced as <NUM>. Also shown are a third close microphone (close microphone <NUM>) and an associated tag (HAIP tag <NUM>) which are referenced as <NUM>. Also shown is the microphone array with tag receiver (HAIP receiver <NUM>). In <FIG> an operator of the system may be configured to assign the tag identifier values to the received audio signals. Thus for example the operator may make a first assignment between the audio signal of the first close microphone and the tag identified as HAIP tag <NUM> shown by reference <NUM>. The operator may further make additional assignments between the audio signal of the second close microphone and the tag identified as HAIP tag <NUM> shown by reference <NUM> and also the audio signal of the third close microphone and the tag identified as HAIP tag <NUM> shown by reference <NUM>.

However as discussed above in some situations assignment mistakes may be made. This for example is shown in <FIG>. In <FIG> the same series of close microphones (with associated high accuracy indoor positioning HAIP tags) are shown. For example the first close microphone (close microphone <NUM>) and associated tag referenced as <NUM>, the second close microphone (close microphone <NUM>) and associated tag referenced as <NUM>, and third close microphone (close microphone <NUM>) and associated tag referenced as <NUM>. In <FIG> an operator of the system correctly assigns the audio signal of the first close microphone and the tag identified as HAIP tag <NUM> (in other words the tag associated with the first microphone) shown by reference <NUM>. However the operator in the example shown in <FIG> incorrectly assigns the tag identifier values associated with the second close microphone to the audio signal of the third close microphone and the tag identifier value associated with the third close microphone to the audio signal of the second close microphone. Thus for example the operator makes the assignment close mic <NUM>/HAIP tag <NUM> shown by reference <NUM> and close microphone <NUM>/HAIP tag <NUM> shown by reference <NUM>.

The concept which is described in embodiments in further detail hereafter is the implementation of a system and method for the association of positioning (HAIP) tags and close microphones (which may also be referred to as audio or sound sources). The embodiments as described hereafter comprise a system configured to capture data on:.

Having obtained this information for each positioning (HAIP) tag the system is configured to determine angle-of-arrival data for the positioning tag relative to the microphone array. The system may then perform audio beamforming in the directions of the angles-of-arrival using the audio data from the microphone array and store the resulting beamformed audio signal. The system may then be configured for each close microphone audio signal compare the close microphone audio signal to the beamformed audio signal (following an audio alignment process). The system may then be configured to determine the close microphone audio signal which best matches the beamformed audio signal. The system may then associate the close microphone audio signal which provided the matching close microphone audio signal with the current positioning (HAIP) tag.

In configuring the system to implement the above processes it is possible to associate all positioning (HAIP) tags with all close microphones (and their audio signals). As a result, any Spatial Audio Mixing system or other automatic panning system is able to create high quality spatial audio scenes where sound objects are associated to the correct positioning tags.

In <FIG> the 'close microphone recorder' and 'close microphone position determiner' parts of the system according to some embodiments are shown. In this example there are three example close microphones of which their tags are currently unknown or unassigned. Furthermore the system comprises a microphone array <NUM>. In this example the microphone array <NUM> is part of an OZO camera comprising a positioning tag (HAIP) receiver array however in some embodiments the positioning tag (HAIP) receiver array is separate from the microphone array and the microphone array may be equipped with a positioning tag of its own enabling the positioning tag to determine the position of the close microphone tags relative to the microphone array tags.

In the example shown in <FIG> there is shown a recorder/memory <NUM> configured to store the audio signals a<NUM>, a<NUM>, a<NUM> from close microphones m<NUM>, m<NUM>, m<NUM> respectively. Each of the close microphones m<NUM>, m<NUM>, m<NUM> has positioning (HAIP) tags which generates a positioning signal received by the positioning tag receiver array. However the positioning tag receiver array/system does not know which close microphone and also which close microphone audio signal the positioning tag is associated with.

<FIG> shows a close microphone position determiner <NUM>. The close microphone position determiner <NUM> may in some embodiments be configured to receive the output of the positioning tag receiver array and be configured to determine position data associated with position tags. The close microphone position determiner may be configured to output positions of an 'unknown' or 'unassigned' tag over time. This is shown in <FIG> by the position of unknown tag x at times i shown as the output Px,i. <FIG> furthermore shows the position of unknown or unassigned tag <NUM> and the positions of the unknown or unassigned tag <NUM> p<NUM> at time t=<NUM>, p<NUM> at time t=<NUM>, p<NUM> at time t=<NUM>, p<NUM> at time t=<NUM> values.

In some embodiments the position determiner is further configured to determine an angle of arrival or orientation relative to the microphone array of the unknown or unassigned tag. For example as shown in <FIG> the example unknown tag <NUM> has an orientation α<NUM> when at position p<NUM>, α<NUM> when at position p<NUM>, α<NUM> when at position p<NUM>, and α<NUM> when at position p<NUM>. In some embodiments the orientation is output by the position determiner <NUM>.

<FIG> shows schematically a further 'beamformer' part of the system according to some embodiments. The beamformer <NUM> is configured to generate a beamformed audio signal ab based on the output of the microphone array where the beam is directed towards the determined tag orientation (or position). Thus as shown in <FIG> the beamformer <NUM> is configured at t<NUM> to direct the beam on orientation α<NUM>, at t<NUM> to direct the beam on orientation α<NUM>, at t<NUM> to direct the beam on orientation α<NUM>, and at t<NUM> to direct the beam on orientation α<NUM>.

<FIG> shows schematically an aligner <NUM>, a correlator <NUM> and a tag assigner <NUM> parts of the system according to some embodiments. The parts shown in <FIG> show that once the close microphone audio signals and the beamformed audio signal are determined/recorded, these audio signals can be compared to determine which close microphone audio signal corresponds to the beamformed audio signal. Since the beamformed audio was obtained based on the positions of a tag, which close microphone (audio signal) to be associated with this tag can be found. In some embodiments the comparison is a two stage process of alignement and cross-correlation performed by an aligner <NUM> and correlator <NUM> respectively.

In some embodiments the aligner <NUM> is configured to receive the output of the beamformer <NUM> and the close microphone recorder <NUM> and attempt to align the beamformed audio signal ab with each of the recorded close microphone audio signals a<NUM>, a<NUM>, a<NUM>.

In some embodiments the correlator <NUM> is configured to receive the time aligned audio signals and perform a cross-correlation of pairs selected from the beamformed audio signal and one of the close microphone audio signals. For example cross correlations of the pairs (ab,a<NUM>) (ab,a<NUM>) and (ab,a<NUM>) are performed and their cross-correlation values stored.

In some embodiments the alignment and correlation operations may be performed using a generalized cross-correlation with phase transform (GCC-PHAT) operation. In such embodiments a GCC-PHAT value is calculated for each close microphone audio signal and the beamformed audio signal.

The GCC-PHAT cross-correlation between two signals xi(n) and xj(n) may be calculated as follows. <MAT> where Xi(f) is the Fourier transform of xi(n).

From this we may calculate the maximum correlation value CPHAT by using <MAT> where RPHAT(d) is the inverse Fourier transform of GPHAT(f).

In this example the cross correlation value CPHAT(b,i) is calculated for the beamformed signal xb(n) and all close microphone audio signals xi(n).

In some embodiments a tag assigner <NUM> is further configured to then assign the tag to the close microphone which has the highest match between the beamformed audio signal and the close microphone audio signal. This match may be the greatest GCC-PHAT value or cross-correlation value.

In some embodiments a threshold on the correlation value may be imposed such that the association of the close microphone (and the close microphone audio signal) and the tag is dependent on whether or not the correlation value is higher than the threshold. For example, in the case the system or the method operates when no one is speaking into the microphones, the correlation value would be low for each comparison, so the system may decide to continue monitoring and analyzing the data. In some embodiments the system may furthermore comprise a voice/instrument detector (for example a voice activity detector (VAD) or other thresholding mechanism) for selecting suitable times for performing the time-alignment.

In some embodiments where only one speaker or close microphone or audio source or sound source is producing audio at a time, instead of using correlations we may use a simple audio activity detector on the close microphone and on the beamformed audio. In these embodiments each microphone array (OZO) audio beam in the direction of each position (HAIP) tag is analyzed, and if there is a detected audio activity at the beam this close microphone for which we have previously detected audio activity is associated with the corresponding tag.

With respect to <FIG> an example flow diagram of the operation of the system is shown in further detail.

In some embodiments the close microphone audio signals are received/recorded.

The operation of receiving/recording the close microphone audio signals is shown in <FIG> by step <NUM>.

While the close microphone signals are being received/recorded the unknown or unassigned tag positions (orientations) are determined.

The operation of determining the unknown or unassigned tag positions (orientations) is shown in <FIG> by step <NUM>.

Furthermore while the close microphone signals are being received/recorded the microphone array audio signals are received/recorded.

The operation of receiving/recording the microphone array audio signals is shown in <FIG> by step <NUM>.

The system, for example the beamformer, is configured to generate a beamformed audio signal based on the microphone array audio signals, where the beam is based on the determined position/orientation of the tag.

The operation of generating the beamformed audio signal is shown in <FIG> by step <NUM>.

The aligner may then align the beamformed audio signal with the close microphone audio signals.

The operation of aligning the beamformed audio signal with the close microphone audio signals is shown in <FIG> by step <NUM>.

The correlator may then be configured to generate cross-correlation values associated with the aligned audio signals.

The operation of generating cross-correlation values between the aligned beamformed audio signal and the close microphone audio signals is shown in <FIG> by step <NUM>.

The system may then select the close microphone (or close microphone audio signal) which has the largest cross-correlation value with the aligned beamformed audio signal.

The operation of selecting the close microphone with the largest cross correlation value is shown in <FIG> by step <NUM>.

The system may then assign the selected close microphone (and therefore the selected close microphone audio signal) the tracked tag identifier.

The operation of associating or assigning the 'unknown' tag value with the selected close microphone is shown in <FIG> by step <NUM>.

The system and methods described herein describe only part of the whole mixing and processing system and specifically the assignment of association of close microphones with position (HAIP) tags in such a way that subsequent processing, mixing and rendering operations which are not described in further detail are able to correctly associate a microphone audio signal with a position determined from the associated tag and thus process/mix/render the correct microphone signal or signal derived from the correct microphone.

The close microphones and/or microphone array microphones may be transducers configured to convert acoustic waves into suitable electrical audio signals. In some embodiments the microphones can be solid state microphones. In other words the microphones may be capable of capturing audio signals and outputting a suitable digital format signal. In some other embodiments the microphones or microphone array can comprise any suitable microphone or audio capture means, for example a condenser microphone, capacitor microphone, electrostatic microphone, Electret condenser microphone, dynamic microphone, ribbon microphone, carbon microphone, piezoelectric microphone, or microelectrical-mechanical system (MEMS) microphone.

In some embodiments the functional elements described herein such as the position determiner <NUM>, beamformer <NUM>, recorder <NUM>, aligner <NUM>, correlator <NUM> and tag assigner <NUM> may be implemented by a processor configured to execute various program codes.

In some embodiments the device comprises a memory. In some embodiments the at least one processor is coupled to the memory. The memory can be any suitable storage means. In some embodiments the memory comprises a program code section for storing program codes implementable upon the processor. Furthermore in some embodiments the memory can further comprise a stored data section for storing data, for example the recorded close microphone audio signals, the microphone array audio signals or other data that has been processed or to be processed in accordance with the embodiments as described herein. The implemented program code stored within the program code section and the data stored within the stored data section can be retrieved by the processor whenever needed via the memory-processor coupling. The device may comprise a transceiver coupled to the processor and configured to enable a communication with other apparatus or electronic devices, for example via a wireless communications network. The transceiver or any suitable transceiver or transmitter and/or receiver means can in some embodiments be configured to communicate with other electronic devices or apparatus via a wire or wired coupling.

The transceiver can communicate with further apparatus by any suitable known communications protocol. For example in some embodiments the transceiver or transceiver means can use a suitable universal mobile telecommunications system (UMTS) protocol, a wireless local area network (WLAN) protocol such as for example IEEE <NUM>. X, a suitable short-range radio frequency communication protocol such as Bluetooth, or infrared data communication pathway (IRDA).

The embodiments of this invention may be implemented by computer software executable by a data processor of the electronic device, such as in the processor entity, or by hardware, or by a combination of software and hardware.

Claim 1:
An apparatus for identifying which sound sources are associated with which close microphone audio signals, the apparatus comprising a processor configured to:
determine/receive (<NUM>) a position/orientation of at least one sound source of the sound sources relative to a microphone array; and
obtain (<NUM>) a beam-formed audio signal from the microphone array, wherein the beam-formed audio signal is directed from the microphone array towards the position/orientation of one of the at least one sound source so as to enhance the beam-formed audio signal; characterized by the processor being configured to:
receive (<NUM>) at least one close microphone audio signal of the close microphone audio signals, wherein each close microphone audio signal is received from a respective close microphone;
compare (<NUM>, <NUM>) the beam-formed audio signal against each close microphone audio signal to identify a match between one of the at least one close microphone audio signal and the beam-formed audio signal; and
associate (<NUM>) the one of the at least one close microphone audio signal with the at least one sound source, based on the identified match.