Patent Description:
A sound object as recorded is a recorded sound object. A sound object as rendered is a rendered sound object.

The recorded sound objects in the recorded sound scene have positions (as recorded) within the recorded sound scene. The rendered sound objects in the rendered sound scene have positions (as rendered) within the rendered sound scene.

Spatial audio renders a recorded sound object (sound source) as a rendered sound object (sound source) at a controlled position within the rendered sound scene.

If a rendered sound scene is to accurately reproduce a recorded sound scene then the positions (as rendered) need to be the same as the positions (as recorded).

It is possible to use a source microphone which moves with a sound source to create a recorded sound object (sound source). One example of a source microphone is a Lavalier microphone. Another example of a source microphone is a boom microphone.

The position of the sound source (microphone) in the recorded sound scene can be tracked. The position (as recorded) of the recorded sound source is therefore known and can be re-used as the position (as rendered) of the rendered sound source. It is therefore important for the position (as rendered) to track the position (as recorded) as the position (as recorded) changes.

However, any measurements of position are subject to noise which introduces (positional) noise to the rendered sound scene.

It would be desirable to reduce or remove such noise.

<CIT> describes a method for applying panning behaviours to audio content. Panning presets may be stored.

<CIT> describes a graphical user interface for audio processing, comprising a positioning area. Audio objects are movable by the user to different locations in the position area, to control playback position in a listening environment. Presets may be stored.

<CIT> describes spatial audio filter profiles. One profile partially damps sounds from outside a visible angle of view scene. Another profile does not. In a use case, audience noise can be reduced during an audio-visual recording of a performance.

<CIT> describes an audio decoder smoother for smoothing a quantized audio reconstruction parameter (e.g. IID, ICLD), which adapts its time constant to the speed of a spatial movement of a point source (e.g. speed of panning), to reduce lag of the reproduced position compared to the originally intended position.

<CIT> describes stabilizing spatial audio signals to compensate for motion (e.g. shake) of a recording device. Specifically, the stabilization is provided to direction estimates of audio sources. The direction is estimated by comparing the relative delays between pairs of microphones receiving the audio.

The invention is as claimed in the appended claims.

<FIG> illustrates an example of an apparatus <NUM> comprising a controller <NUM> for at least controlling spatial audio processing via a man machine interface <NUM>. The controller <NUM> is configured to control input/output circuitry <NUM> to provide a man machine user interface <NUM> to a user of the apparatus <NUM>. An example of the MMI <NUM> is illustrated in <FIG>.

Implementation of the controller <NUM> may be as controller circuitry. The controller <NUM> may be implemented in hardware alone, have certain aspects in software including firmware alone or can be a combination of hardware and software (including firmware).

As illustrated in <FIG> the controller <NUM> may be implemented using instructions that enable hardware functionality, for example, by using executable instructions of a computer program <NUM> in a general-purpose or special-purpose processor <NUM> that may be stored on a computer readable storage medium (disk, memory etc.) to be executed by such a processor <NUM>.

The memory <NUM> stores a computer program <NUM> comprising computer program instructions (computer program code) that controls the operation of the apparatus <NUM> when loaded into the processor <NUM>. The computer program instructions, of the computer program <NUM>, provide the logic and routines that enables the apparatus to perform the methods illustrated in <FIG>. The processor <NUM> by reading the memory <NUM> is able to load and execute the computer program <NUM>.

In this example, the memory <NUM> is a non-volatile memory storing, in a database <NUM>, multiple sets <NUM> of predetermined spatial audio processing parameters P for differently moving sound sources <NUM>.

As illustrated in the example in <FIG>, the man machine interface <NUM> presents a user-selectable option <NUM> that enables the user to select one of the stored sets <NUM> of predetermined spatial audio processing parameters P for differently moving sound sources <NUM>.

The controller <NUM>, in response to the user selecting one of the stored sets <NUM> of predetermined spatial audio processing parameters P for differently moving sound sources <NUM>, uses the selected one of the stored multiple sets <NUM> of predetermined spatial audio processing parameters P to spatially process audio from one or more sound sources <NUM>.

The controller <NUM> may itself perform the spatial audio processing or it may instruct another processor to perform the spatial audio processing.

In some examples, selection of an option <NUM> by the user may cause the selected spatial audio processing parameters P to be used to spatially process audio from one sound source or from a group of sound sources. The option may visually indicate that sound source of that group of sound sources.

In other examples, a different user selectable option <NUM> may be provided for each different sound source or each different group of sound sources. Selection of an option causes the selected spatial audio processing parameters P to be used to spatially process audio from the one sound source or from the group of sound sources associated with the selected option <NUM>. The option <NUM> may visually indicate that sound source of that group of sound sources associated with that option <NUM>.

In other examples, the user may be able to select which sound source or which group of sound sources, the selected spatial audio processing parameters P are used to spatially process audio from. The option <NUM> may then visually indicate the selected sound source or selected group of sound sources associated with that option.

In this particular example, the non-volatile memory <NUM> stores at least a first set <NUM>, of predetermined spatial audio processing parameters P for slowly moving sound sources <NUM>; and a second set <NUM><NUM> of predetermined spatial audio processing parameters P for quickly moving sound sources <NUM>.

An option <NUM> presented in the user interface may present two or more independently user selectable options, for example, a first one for the first set <NUM>, of predetermined spatial audio processing parameters P for slowly moving sound sources <NUM> and a second one for the second set <NUM><NUM> of predetermined spatial audio processing parameters P for fast moving sound sources <NUM>. The first option may visually indicate to a user that selection of this option by a user should be made for slowly moving sound sources. The second option may visually indicate to a user that selection of this option by a user should be made for fast moving sound sources.

Instead of presenting both the first option and the second option prompting manual selection, the system may perform semi-automatic selection and present only the first option if the associated sound source or group of sound sources is slow moving and present only the second option if the if the associated sound source or group of sound sources is fast moving.

The man machine interface <NUM> may have user input controls <NUM> configured to adapt one or more of the spatial audio processing parameters P of the selected one of the stored multiple sets <NUM> of predetermined spatial audio processing parameters P. In some but not necessarily all examples, the adaptation changes the spatial audio processing parameters P in use for spatially processing audio. However, the stored sets <NUM> of predetermined spatial audio processing parameters P for differently moving sound sources <NUM> are not varied, they are read-only.

The above mentioned group or groups of sound sources may be a sub-set or sub-sets of active sound sources. The sub-sets may be user selected or automatically selected.

<FIG> illustrates an example of a system for spatial audio processing audio from multiple sound sources <NUM> that may move <NUM>.

Each of the microphones <NUM> represents a sound source (a recorded sound object). At least some of the microphones <NUM> are capable of independent movement <NUM>. A movable microphone may, for example, be a Lavalier microphone or a boom microphone.

The processor <NUM> is configured to process the audio <NUM> recorded by the movable microphones <NUM> to produce spatial audio <NUM> which when rendered produces one or more rendered sound objects at specific controlled positions within a rendered sound scene.

The recorded sound objects in the recorded sound scene have positions <NUM> within the recorded sound scene. The position module <NUM> determines the positions <NUM> and provides them to the processor <NUM>.

If a rendered sound scene is to accurately reproduce a recorded sound scene then the positions (as rendered) of sound sources need to be the same as the positions (as recorded).

The positions <NUM> are subject to noise which introduces (positional) noise to the rendered sound scene. It would be desirable to reduce or remove such noise.

The controller <NUM> provides a set <NUM> of predetermined spatial audio processing parameters P to the processor <NUM>.

The set <NUM> of predetermined spatial audio processing parameters P are used by the processor <NUM> to control production of the spatial audio <NUM>. In particular, to control rendering of one or more sound sources in the rendered sound scene.

In some but not necessarily all examples, at least some of the stored sets <NUM> of predetermined spatial audio processing parameters P for differently moving sound sources <NUM>, when used for the same sound source (or group of sound sources), cause one or more of the following relative differences during spatial audio processing: different location-based processing such as, for example, different orientation or distance; different sound intensity; different frequency spectrum; different reverberation, different sound source size.

The first set <NUM>, of predetermined spatial audio processing parameters P may be used to control spatial audio processing by processor <NUM> for a slowly moving sound source <NUM> or for a group of slowly moving sound sources <NUM>. The resultant spatial audio <NUM> is compensated for the movement or change in movement of the slowly moving sound source(s) <NUM>.

The second set <NUM><NUM> of predetermined spatial audio processing parameters P may be used to control spatial audio processing by processor <NUM> for a fast moving sound source <NUM> or for a group of fast moving sound sources <NUM>. The resultant spatial audio <NUM> is compensated for the movement or change in movement of the fast moving sound source(s) <NUM>.

Using a particular set <NUM>n of predetermined spatial audio processing parameters P to control spatial audio processing by processor <NUM> for multiple sound sources may therefore cause the same relative variation of audio processing parameters for those multiple sound sources <NUM>.

It will be appreciated that different sets <NUM>n of predetermined spatial audio processing parameters P may be used in different combinations for different sound sources <NUM> having different movements.

It will be appreciated that a set <NUM> of predetermined spatial audio processing parameters P used for a particular sound source <NUM> may change (or an option <NUM> may be provided to change the set <NUM>) when the movement of that sound source changes.

In the example illustrated in <FIG>, the set <NUM> of predetermined spatial audio processing parameters P are used by the processor <NUM> to control at least a characteristic of a filter <NUM>. The set <NUM> of predetermined spatial audio processing parameters P comprises a filter parameter p for the filter <NUM>. The filter <NUM> controls a position at which one or more sound sources are rendered in the rendered sound scene.

The filter <NUM> comprises a noise reduction filter used to more accurately position a rendered sound source in the rendered sound scene by removing or reducing noise in the position <NUM> of the sound source.

A first set <NUM>, of predetermined spatial audio processing parameters P for slowly moving sound sources <NUM> has a first filter parameter p<NUM> for the noise reduction filter <NUM> suitable for filtering slowly varying positions <NUM> and a second set <NUM><NUM> of predetermined spatial audio processing parameters P for fast moving sound sources <NUM> has a second filter parameter p<NUM> for the noise reduction filter <NUM> suitable for filtering quickly varying positions <NUM>. The first filter parameter and the second filter parameter are different.

The first filter parameter p<NUM> and second filter parameter p<NUM> may define different durations of a filter window used for time averaging. The filter parameter p depends upon the actual or expected speed (rate of change of position <NUM>) of the sound source(s) affected by the filter parameter p. The first filter parameter is longer than the second filter parameter.

Each of the first filter parameter p<NUM> and the second filter parameter p<NUM> may define a variance parameter in a Kalman filter, where the second filter parameter pz allows for greater change in position <NUM> than the first filter parameter p<NUM>. In some examples, a random walk model may be used with the Kalman filter.

It should be noted that if an incorrect filter parameter is applied then noise or lag increases and that if a correct filter parameter is applied then noise and lag is reduced. The storage and use of multiple sets <NUM> of predetermined spatial audio processing parameters P for differently moving sound sources <NUM> in the non-volatile memory <NUM>, makes it much easier for a user of the man machine interface <NUM> to use correct filter parameters.

In the example of <FIG>, the processor <NUM> performs spatial audio processing by controlling an orientation of a rendered sound source using orientation module <NUM> to process the audio signals <NUM> from the sound source <NUM> and rotate the sound source within the rendered sound scene using a transfer function. The extent of rotation is controlled by a bearing of the position <NUM> after it has been filtered by the filter <NUM> using a provided filter parameter <NUM>.

The processor <NUM> performs spatial audio processing by controlling a distance of a rendered sound source using distance module <NUM> to process the audio signals <NUM> from the sound source <NUM>. The distance module may simulate a direct audio path and an indirect audio path. Controlling the relative and absolute gain between the direct and indirect paths can be used to control the perception of distance of a sound source. The distance control is based upon a distance to the position <NUM> after it has been filtered by the filter <NUM> using a provided filter parameter <NUM>.

The remaining description will refer to filter parameters p as an example of a set <NUM> of spatial audio processing parameters P.

<FIG> illustrates an example of a method <NUM> for enabling adaptation of the current filter parameter p for the one or more sound sources <NUM>.

The method at block <NUM> comprises determining an actual or expected change in movement for one or more sound sources <NUM> rendered as spatial audio.

The method at block <NUM> comprises, in dependence upon determining an actual or expected change in movement for one or more sound sources <NUM> rendered as spatial audio, determining that current filter parameter p for the one or more sound sources <NUM> is to be changed.

The method at block <NUM> comprises, in dependence upon determining that a current filter parameter p for the one or more sound sources <NUM> is to be changed, enabling adaptation of the current filter parameter p for the one or more sound sources <NUM> to render the one or more sound sources <NUM> as spatial audio, compensated for the determined actual or expected change in movement.

The actual movement of a sound source may be determined from the position <NUM> of the sound source. The position <NUM> of the sound source may be determined by using a positioning system to locate and position the sound source <NUM> as it moves. Such a positioning system may use one or more of: one or more accelerometers at the microphone <NUM> or that move with the microphone <NUM> and then using dead reckoning for positioning, a trilateration or triangulation system based on radio communication between a transmitter/receiver at the microphone <NUM> or that moves with the microphone, an alternative positioning system such as one that relies on computer vision processing and/or depth mapping.

An expected movement of a sound source may be determined based upon predictive analysis based on patterns of past movement of the sound source.

An expected movement of a sound source may be determined based upon knowledge of future activities or likely future activities of the sound source. This may for example include knowledge of a future increase or decrease in music tempo where the sound source is attached to someone whose movement typically depends upon the tempo of the music.

<FIG> illustrates an example of the method <NUM> illustrated in <FIG> in more detail. In this example, the method at block <NUM> comprises, in dependence upon determining that a current filter parameter p for the one or more sound sources <NUM> are to be changed, enabling adaptation of the current filter parameter p for the one or more sound sources <NUM>:.

In some examples, the set <NUM> of predetermined spatial audio processing parameters P (e.g. filter parameter p) used for spatial processing is based on an algorithm in dependence upon the actual or expected change in movement for one or more sound sources <NUM> rendered as spatial audio. New filter parameters pnew used for spatial audio processing the one or more sound sources <NUM> may be generated by adapting the current filter parameters pcurrent used for spatial audio processing the one or more sound sources <NUM> now, in dependence upon the algorithm pnew = λ pcurrent, where λ is determined based upon the actual or expected change in movement for the one or more sound sources <NUM> rendered as spatial audio. For example, if there is less movement the filter window length of an average filter may be lengthened and if there is more movement the filter window length can be shortened. The exact value of λ may depend on additional inputs for example λ may have a linear or non-linear relationship to a speed of a sound source.

The predetermined spatial audio processing parameters P may be a value of λ.

Other approaches may be used to determine the sets <NUM> of predetermined spatial audio processing parameters P used for spatial processing.

<FIG> illustrates an example of block <NUM> and <NUM> of the method <NUM>.

The database <NUM> in the non-volatile memory <NUM> stores sets <NUM> of predetermined spatial audio processing parameters P in association <NUM> with different movement classifications <NUM>.

At sub-block <NUM>, of block <NUM>, in dependence upon determining an actual or expected change in movement for one or more sound sources <NUM> rendered as spatial audio, the method <NUM> automatically determines a movement classification for the actual or expected change in movement for one or more sound sources <NUM> rendered as spatial audio. If the movement can be classified, the method moves to the next sub-block.

Then at sub-block <NUM>, the determined movement classification is used to access, in the database <NUM>, the set of predetermined spatial audio processing parameters P associated with the determined movement classification.

The method <NUM> then proceeds, for example, as illustrated in <FIG>, <FIG>, to automatically provide the option <NUM> to a user to select the accessed set of predetermined spatial audio processing parameters P for differently moving sound sources <NUM> and use the selected set of predetermined spatial audio processing parameters P to spatially process audio from one or more sound sources <NUM>.

<FIG> illustrates another example of block <NUM> and <NUM> of the method <NUM>.

This figure illustrates an example of a method that enables adaptation of the current filter parameters p for the one or more sound sources <NUM> by adapting the current filter parameters p for the one or more sound sources <NUM> based on a search for better filter parameters p for the one or more sound sources <NUM>.

At sub-block <NUM>, a reference value is determined. The current filter parameters p for the one or more sound sources <NUM> are used to filter expected positions representing an expected movement of the sound source(s).

An error value can be determined by measuring a fit between the filtered expected positions and the unfiltered expected positions. The error value is stored as a reference value. It is a figured of merit for the current filter parameters p.

At sub-block <NUM> the filter parameters p for the one or more sound sources <NUM> are varied. The variation may be based upon the expected positions of the one or more sound sources. For example, if the filter parameter is a filter window length, it may be lengthened if the expected positions indicate that the one or more sound sources are slowing down or may be shortened if the expected positions indicate that the one or more sound sources are speeding up.

At sub-block <NUM> the varied filter parameters Δp for the one or more sound sources <NUM> are used to filter expected positions representing an expected movement of the sound source(s).

An error value can be determined by measuring a fit between the newly filtered expected positions and the unfiltered positions. The error value is stored as a test value. It is a figure of merit for the new filter parameters Δp.

At sub-block <NUM> the test value is compared to the reference value. If the difference between the test value and the reference value is less than a threshold, the new filter parameters Δp is selected for use.

If the difference between the test value and the reference value is not less than a threshold, the method returns <NUM> to sub-block <NUM> and varies the new filter parameters Δp. The method then proceeds from sub-block <NUM>. In this way, the method searches the filter parameter space for a suitable filter parameter value.

A constraint may be placed as to which portions of the parameter space can and cannot be searched. For example, a filter window length may be forced to be greater than or equal to a minimum value.

The determination of expected positions may, for example, be determined by applying a gain value to the current movement, adding noise, such as white Gaussian distributed noise with a variance dependent upon movement, predicting future movement based on past movement and the expectation that prior patterns of movement will be repeated, or by seeking input from the user via the MMI <NUM> concerning expected movement e.g. horizontal- left, horizontal-right, dancing, etc..

It will therefore be appreciated from the foregoing that the apparatus <NUM> therefore comprises:.

As illustrated in <FIG>, the computer program <NUM> may arrive at the apparatus <NUM> via any suitable delivery mechanism <NUM>. The delivery mechanism <NUM> may be, for example, a non-transitory computer-readable storage medium, a computer program product, a memory device, a record medium such as a compact disc read-only memory (CD-ROM) or digital versatile disc (DVD), an article of manufacture that tangibly embodies the computer program <NUM>. The delivery mechanism may be a signal configured to reliably transfer the computer program <NUM>. The apparatus <NUM> may propagate or transmit the computer program <NUM> as a computer data signal.

Although the memory <NUM> is illustrated in <FIG> as a single component/circuitry it may be implemented as one or more separate components/circuitry some or all of which may be integrated/removable and/or may provide permanent/semi-permanent/ dynamic/cached storage.

Although the processor <NUM> is illustrated in <FIG> as a single component/circuitry it may be implemented as one or more separate components/circuitry some or all of which may be integrated/removable.

References to `computer-readable storage medium', 'computer program product', 'tangibly embodied computer program' etc. or a 'controller', 'computer', 'processor' etc. should be understood to encompass not only computers having different architectures such as single /multi- processor architectures and sequential (Von Neumann)/parallel architectures but also specialized circuits such as field-programmable gate arrays (FPGA), application specific circuits (ASIC), signal processing devices and other processing circuitry.

As used in this application, the term 'circuitry' refers to all of the following:.

This definition of 'circuitry' applies to all uses of this term in this application, including in any claims. The term "circuitry" would also cover, for example and if applicable to the particular claim element, a baseband integrated circuit or applications processor integrated circuit for a mobile phone or a similar integrated circuit in a server, a cellular network device, or other network device.

The blocks illustrated in <FIG> may represent steps in a method and/or sections of code in the computer program <NUM>. The illustration of a particular order to the blocks does not necessarily imply that there is a required or preferred order for the blocks and the order and arrangement of the block may be varied. Furthermore, it may be possible for some blocks to be omitted.

If it is intended to use 'comprise' with an exclusive meaning then it will be made clear in the context by referring to "comprising only one" or by using "consisting".

In this brief description, reference has been made to various examples. The use of the term 'example' or 'for example' or 'may' in the text denotes, whether explicitly stated or not, that such features or functions are present in at least the described example, whether described as an example or not, and that they can be, but are not necessarily, present in some of or all other examples. Thus 'example', 'for example' or 'may' refers to a particular instance in a class of examples. It is therefore implicitly disclosed that a features described with reference to one example but not with reference to another example, can where possible be used in that other example but does not necessarily have to be used in that other example.

Claim 1:
A movement compensation method for spatial audio rendering, comprising:
storing in a non-volatile memory (<NUM>) multiple sets (<NUM>) of predetermined spatial audio processing parameters (P) for differently moving sound sources (<NUM>), wherein storing in a non-volatile memory multiple sets of predetermined spatial audio processing parameters for differently moving sound sources comprises:
storing in the non-volatile memory a first set (<NUM><NUM>) of predetermined spatial audio processing parameters for slowly moving sound sources having a filter parameter for a noise reduction filter for filtering slowly varying positions; and
storing in the non-volatile memory a second set (<NUM><NUM>) of predetermined spatial audio processing parameters for quickly moving sound sources having a different filter parameter for the noise reduction filter for filtering quickly varying positions;
providing in a man machine interface (<NUM>) an option (<NUM>) for a user to select one of the stored multiple sets of predetermined spatial audio processing parameters for differently moving sound sources, the option (<NUM>) comprising a first option for the first set (<NUM><NUM>) of predetermined spatial audio processing parameters and a second option for the second set (<NUM><NUM>) of predetermined spatial audio processing parameters, wherein the first option visually indicates to a user that selection of the first option should be made for slowly moving sound sources and the second option visually indicates to a user that selection of the second option should be made for fast moving sound sources;
receiving a recorded position of one or more sound sources rendered as spatial audio; and
in response to the user selecting one of the stored multiple sets of predetermined spatial audio processing parameters for differently moving sound sources, using the selected one of the stored multiple sets of predetermined spatial audio processing parameters to spatially process audio (<NUM>) from the one or more sound sources, comprising the noise reduction filter filtering the position based on the filter parameter of the selected one of the stored multiple sets of predetermined spatial audio processing parameters.