Recording and rendering sound spaces

A method, apparatus and computer program, the method including enabling an output of an audio mixer to be rendered for a user where the user is located within a sound space, wherein at least one input channel is provided to the audio mixer and the at least one input channel receives a plurality of microphone output signals obtained by a plurality of microphones recording the sound space; determining that a first microphone records one or more sound objects within the sound space; and in response to the determining, enabling one or more of the plurality of microphone output signals to be, at least partially, removed from the at least one input channel to the audio mixer.

CROSS REFERENCE TO RELATED APPLICATION

This patent application is a U.S. National Stage application of International Patent Application Number PCT/FI2018/050487 filed Jun. 21, 2018, which is hereby incorporated by reference in its entirety, and claims priority to GB 1710236.9 filed Jun. 27, 2017.

TECHNOLOGICAL FIELD

Embodiments of the invention relate to recording and rendering sound spaces. In particular they relate to recording and rendering sound spaces where a user may be located within the sound space and may be free to move within the sound space.

BACKGROUND

Sound spaces may be recorded and rendered in any applications where spatial audio is used. For example the sound spaces may be recorded for use in mediated reality content applications such as virtual reality or augmented reality applications.

To enable sound spaces to be accurately reproduced it is useful to use a plurality of microphones. However increasing the number of microphones used increases the amount of data that has to be provided to an audio mixer. If the user's rendering device is located separately to the audio mixer then the signal comprising the audio output may be transmitted via a wireless communication link. The amount of data that can be transmitted may be limited by the bandwidth of the communication link. This may limit the quality of the audio output that can be recorded and subsequently rendered for the user via the audio mixer.

BRIEF SUMMARY

According to various, but not necessarily all, examples of the disclosure there is provided a method comprising: enabling an output of an audio mixer to be rendered for a user where the user is located within a sound space, wherein at least one input channel is provided to the audio mixer and the at least one input channel receives a plurality of microphone output signals obtained by a plurality of microphones recording the sound space; determining that a first microphone records one or more sound objects within the sound space; and in response to the determining, enabling one or more of the plurality of microphone output signals to be, at least partially, removed from the at least one input channel to the audio mixer.

The method may comprise replacing the removed one or more microphone output signals in the output provided to the user with a signal recorded by the first microphone.

The first microphone may be a microphone associated with the user. The microphone associated with the user may be worn by the user. The microphone associated with the user may be located in a head set worn by the user.

Determining that a first microphone can be used to record one or more sound objects within the sound space may comprise determining that a signal captured by the first microphone has at least one parameter within a threshold range.

Determining that a first microphone can be used to record one or more sound objects within the sound space may comprise determining that the user is located within a threshold distance of the one or more sound objects.

The method may comprise identifying one or more microphone output signals that correspond to the sound object that can be recorded by the microphone associated with the user.

The plurality of microphones may enable a sound object within the sound space to be isolated.

Enabling one or more of the microphone output signals to be, at least partially, removed from the input channel to the audio mixer may occur automatically when it is determined that the microphone associated with the user can be used to record the sound object.

Enabling one or more of the microphone output signals to be, at least partially, removed from the input channel to the audio mixer may comprise sending a signal to an audio mixing device indicating that one or more of the microphone output signals can be, at least partially, removed. The signal sent to the audio mixing device may comprise information that enables a controller to identify the microphone output signals that can be, at least partially, removed. The signal sent to the audio mixing device may identify the microphone output signals that can be, at least partially, removed.

The signal recorded by the first microphone might not be provided to the audio mixer.

The signals provided by the first microphone may provide a higher quality output than the microphone output signals that are, at least partially, removed from the input channel to the audio mixer.

At least partially removing one or more of the plurality of output signals from the input channel to the audio mixer may increase the efficacy of the available bandwidth between the audio mixer and a user device.

According to various, but not necessarily all, examples of the disclosure there is provided an apparatus comprising: processing circuitry; and memory circuitry including computer program code, the memory circuitry and the computer program code configured to, with the processing circuitry, enable the apparatus to: enable an output of an audio mixer to be rendered for a user where the user is located within a sound space, wherein at least one input channel is provided to the audio mixer and the at least one input channel receives a plurality of microphone output signals obtained by a plurality of microphones recording the sound space; determine that a first microphone records one or more sound objects within the sound space; and in response to the determining, enable one or more of the plurality of microphone output signals to be, at least partially, removed from the at least one input channel to the audio mixer.

The memory circuitry and the computer program code may be configured to, with the processing circuitry, enable the apparatus to replace the, at least partially, removed one or more microphone output signals in the output provided to the user with a signal recorded by the first microphone.

The first microphone may be a microphone associated with the user. The microphone associated with the user may be worn by the user. The microphone associated with the user may be located in a head set worn by the user.

Determining that a first microphone can be used to record one or more sound objects within the sound space may comprise determining that a signal captured by the first microphone has at least one parameter within a threshold range.

Determining that a first microphone can be used to record one or more sound objects within the sound space may comprise determining that the user is located within a threshold distance of the one or more sound objects.

The memory circuitry and the computer program code may be configured to, with the processing circuitry, enable the apparatus to identify one or more microphone output signals that correspond to the sound object that can be recorded by the microphone associated with the user.

The plurality of microphones may enable a sound object within the sound space to be isolated.

Enabling one or more of the microphone output signals to be, at least partially, removed from the input channel to the audio mixer may occur automatically when it is determined that the microphone associated with the user can be used to record the sound object.

Enabling one or more microphone output channels to be, at least partially, removed from the input channel to the audio mixer may comprise sending a signal to an audio mixing device indicating that one or more of the microphone output signals can be, at least partially, removed.

The signal sent to the audio mixing device may comprise information that enables a controller to identify the microphone output signals that can be, at least partially, removed.

The signal sent to the audio mixing device may identify the microphone output signals that can be, at least partially, removed.

The signal recorded by the first microphone might not be provided to the audio mixer.

The signals provided by the first microphone may provide a higher quality output than the microphone output signals that are removed from the input channel to the audio mixer.

At least partially removing one or more of the plurality of output signals from the input channel to the audio mixer may increase the efficacy of the available bandwidth between the audio mixer and a user device.

According to various, but not necessarily all, examples of the disclosure there is provided an apparatus comprising: means for enabling an output of an audio mixer to be rendered for a user where the user is located within a sound space, wherein at least one input channel is provided to the audio mixer and the at least one input channel receives a plurality of microphone output signals obtained by a plurality of microphones recording the sound space; means for determining that a first microphone records one or more sound objects within the sound space; and means for enabling, in response to the determining, one or more of the plurality of microphone output signals to be, at least partially, removed from the at least one input channel to the audio mixer.

According to various, but not necessarily all, examples of the disclosure there is provided an electronic device comprising an apparatus as described above.

The electronic device may be arranged to be worn by a user.

According to various, but not necessarily all, examples of the disclosure there is provided a computer program comprising computer program instructions that, when executed by processing circuitry, enable: enabling an output of an audio mixer to be rendered for a user where the user is located within a sound space, wherein at least one input channel is provided to the audio mixer and the at least one input channel receives a plurality of microphone output signals obtained by a plurality of microphones recording the sound space; determining that a first microphone records one or more sound objects within the sound space; and in response to the determining, enabling one or more of the plurality of microphone output signals to be, at least partially, removed from the at least one input channel to the audio mixer.

According to various, but not necessarily all, examples of the disclosure there is provided a computer program comprising program instructions for causing a computer to perform any of the methods described above.

According to various, but not necessarily all, examples of the disclosure there is provided a physical entity embodying the computer programs as described above.

According to various, but not necessarily all, examples of the disclosure there is provided an electromagnetic carrier signal carrying the computer programs as described above.

DEFINITIONS

“Artificial environment” may be something that has been recorded or generated. “Visual space” refers to fully or partially artificial environment that may be viewed that may be three dimensional.

“Visual scene” refers to a representation of the visual space viewed from a particular point of view within the visual space.

“Visual object” is a visible object within a virtual visual scene.

“Sound space” refers to an arrangement of sound sources in a three-dimensional space. A sound space may be defined in relation to recording sounds (a recorded sound space) and in relation to rendering sounds (a rendered sound space).

“Sound scene” refers to a representation of the sound space listened to from a particular point of view within the sound space.

“Sound object” refers to a sound source that may be located within the sound space. A source sound object represents a sound source within the sound space. A recorded sound object represents sounds recorded at a particular microphone or position. A rendered sound object represents sounds rendered from a particular position.

“Virtual space” may mean a visual space, a sound space or a combination of a visual space and corresponding sound space. In some examples, the virtual space may extend horizontally up to 360° and may extend vertically up to 180°.

“Virtual scene” may mean a visual scene, a sound scene or a combination of a visual scene and a corresponding sound scene.

“Virtual object” is an object within a virtual scene, it may be an artificial virtual object (such as a computer generated virtual object) or it may be an image of a real object that is live or recorded. It may be a sound object and/or a visual object.

“Correspondence” or “corresponding” when used in relation to a sound space and a virtual visual space means that the sound space and virtual visual space are time and space aligned, that is they are the same space at the same time.

“Correspondence” or “corresponding” when used in relation to a sound scene and a visual scene means that the sound scene and visual scene are corresponding and a notional listener whose point of view defines the sound scene and a notional viewer whose point of view defines the visual scene are at the same position and orientation, that is they have the same point of view.

“Real space” refers to a real environment, which may be three dimensional.

“Real visual scene” refers to a representation of the real space viewed from a particular point of view within the real space.

“Real visual object” is a visible object within a real visual scene.

The “visual space”, “visual scene” and “visual object” may also be referred to as the “virtual visual space”, “virtual visual scene” and “virtual visual object” to clearly differentiate them from “real visual space”, “real visual scene” and “real visual object”.

“Mediated reality” in this document refers to a user visually experiencing a fully or partially artificial environment (a virtual space) as a virtual scene at least partially rendered by an apparatus to a user. The virtual scene is determined by a point of view within the virtual space. Displaying the virtual scene means providing it in a form that can be perceived by the user.

“Mediated reality content” is content which enables a user to visually experience a fully or partially artificial environment (a virtual space) as a virtual visual scene. Mediated reality content could include interactive content such as a video game or non-interactive content such as motion video or an audio recording.

“Augmented reality” in this document refers to a form of mediated reality in which a user experiences a partially artificial environment (a virtual space) as a virtual scene comprising a real scene of a physical real world environment (real space) supplemented by one or more visual or audio elements rendered by an apparatus to a user.

“Augmented reality content” is a form of mediated reality content which enables a user to visually experience a partially artificial environment (a virtual space) as a virtual visual scene.

Augmented reality content could include interactive content such as a video game or non-interactive content such as motion video or an audio recording.

“Virtual reality” in this document refers to a form of mediated reality in which a user experiences a fully artificial environment (a virtual visual space) as a virtual scene displayed by an apparatus to a user.

“Virtual reality content” is a form of mediated reality content which enables a user to visually experience a fully artificial environment (a virtual space) as a virtual visual scene. Virtual reality content could include interactive content such as a video game or non-interactive content such as motion video or an audio recording.

“Perspective-mediated” as applied to mediated reality, augmented reality or virtual reality means that user actions determine the point of view within the virtual space, changing the virtual scene.

“First person perspective-mediated” as applied to mediated reality, augmented reality or virtual reality means perspective mediated with the additional constraint that the user's real point of view determines the point of view within the virtual space;

“Third person perspective-mediated” as applied to mediated reality, augmented reality or virtual reality means perspective mediated with the additional constraint that the user's real point of view does not determine the point of view within the virtual space;

“User interactive” as applied to mediated reality, augmented reality or virtual reality means that user actions at least partially determine what happens within the virtual space;

“Displaying” means providing in a form that is perceived visually (viewed) by the user.

“Rendering” means providing in a form that is perceived by the user

DETAILED DESCRIPTION

The following description describes methods, apparatus and computer programs that control how audio content is recorded and rendered to a user. In particular they control how the audio content is recorded and rendered as a user moves within a sound space.

FIG. 1Aillustrates an example of a sound space10comprising a sound object12within the sound space10. The sound object12may be a sound object as recorded or it may be a sound object as rendered. It is possible, for example using spatial audio processing, to modify a sound object12, for example to change its sound or positional characteristics. For example, a sound object can be modified to have a greater volume, to change its position within the sound space10(FIGS. 1B & 1C) and/or to change its spatial extent within the sound space10(FIG. 1D)

FIG. 1Billustrates the sound space10before movement of the sound object12in the sound space10.FIG. 1Cillustrates the same sound space10after movement of the sound object12.

The sound object12may be a sound object as recorded and be positioned at the same position as a sound source of the sound object or it may be positioned independently of the sound source.

The position of a sound source may be tracked to render the sound object at the position of the sound source. This may be achieved, for example, when recording by placing a positioning tag on the sound source. The position and any changes in the position of the sound source can then be recorded. The positions of the sound source may then be used to control a position of the sound object12. This may be particularly suitable where a close-up microphone is used to record the sound source. In the example ofFIG. 1Cthe sound source has moved. It is to be appreciated that the user could move within the sound space10as well as, or instead of, the sound object12.

In other examples, the position of the sound source within the visual scene may be determined during recording of the sound source by using spatially diverse sound recording. An example of spatially diverse sound recording is using a microphone array. The phase differences between the sound recorded at the different, spatially diverse microphones, provides information that may be used to position the sound source using a beam forming equation. For example, time-difference-of-arrival (TDOA) based methods for sound source localization may be used.

The positions of the sound source may also be determined by post-production annotation. As another example, positions of sound sources may be determined using Bluetooth-based indoor positioning techniques, or visual analysis techniques, a radar, or any suitable automatic position tracking mechanism.

FIG. 1Dillustrates a sound space10after extension of the sound object12in the sound space10. The sound space10ofFIG. 1Ddiffers from the sound space10ofFIG. 10in that the spatial extent of the sound object12has been increased so that the sound object has a greater breadth (greater width).

In some examples, a visual scene20may be rendered to a user that corresponds with the rendered sound space10. The visual scene20may be the scene recorded at the same time the sound source that creates the sound object12is recorded.

FIG. 2Aillustrates an example of a visual scene20that corresponds with the sound space10. Correspondence in this sense means that there is a one-to-one mapping between the sound space10and the visual scene20such that a position in the sound space10has a corresponding position in the visual scene20and a position in the visual scene20has a corresponding position in the sound space10. Corresponding also means that the coordinate system of the sound space10and the coordinate system of the visual scene20are aligned such that an object is positioned as a sound object12in the sound space10and as a visual object22in the visual scene20at the same common position from the perspective of a user.

The sound space10and the visual scene20may be three-dimensional.

A portion of the visual scene20is associated with a position of visual object22representing a sound source within the visual scene20. The position of the visual object22representing the sound source in the visual scene20corresponds with a position of the sound object12within the sound space10.

In this example, but not necessarily all examples, the sound source is an active sound source producing sound that is or can be heard by a user depending on the position of the user within the sound space10, for example via rendering or live, while the user is viewing the visual scene via the display200.

In some examples, parts of the visual scene20are viewed through the display200(which would then need to be a see-through display). In other examples, the visual scene20is rendered by the display200.

In an augmented reality application, the display200is a see-through display and at least parts of the visual scene20is a real, live scene viewed through the see-through display200. The sound source may be a live sound source or it may be a sound source that is rendered to the user. This augmented reality implementation may, for example, be used for capturing an image or images of the visual scene20as a photograph or a video.

In another application, the visual scene20may be rendered to a user via the display200, for example, at a location remote from where the visual scene20was recorded. This situation is similar to the situation commonly experienced when reviewing images via a television screen, a computer screen or a mediated/virtual/augmented reality headset. In these examples, the visual scene20is a rendered visual scene. The active sound source produces rendered sound, unless it has been muted. This implementation may be particularly useful for editing a sound space by, for example, modifying characteristics of sound sources and/or moving sound sources within the visual scene20.

FIG. 2Billustrates a visual scene20corresponding to the sound space10ofFIG. 1B, before movement of the sound source in the visual scene20.FIG. 2Cillustrates the same visual scene20corresponding to the sound space10ofFIG. 10, after movement of the sound source.

FIG. 2Dillustrates the visual scene20after extension of the sound object12in the corresponding sound space10. While the sound space10ofFIG. 1Ddiffers from the sound space10ofFIG. 10in that the spatial extent of the sound object12has been increased so that the sound object has a greater breadth, the visual scene20is not necessarily changed.

The above described methods may be performed using an apparatus30such as a controller300. An example of a controller300is illustrated inFIG. 3A.

Implementation of the controller300may be as controller circuitry. The controller300may be implemented in hardware alone, have certain aspects in software including firmware alone or can be a combination of hardware and software (including firmware).

As illustrated inFIG. 3Athe controller300may be implemented using instructions that enable hardware functionality, for example, by using executable instructions of a computer program306in a general-purpose or special-purpose processor302that may be stored on a computer readable storage medium (disk, memory etc.) to be executed by such a processor302.

The processor302is configured to read from and write to the memory304. The processor302may also comprise an output interface via which data and/or commands are output by the processor302and an input interface via which data and/or commands are input to the processor302.

The memory304stores a computer program306comprising computer program instructions (computer program code) that controls the operation of the apparatus30when loaded into the processor302. The computer program instructions, of the computer program306, provide the logic and routines that enables the apparatus to perform the methods illustrated in the figures. The processor302by reading the memory304is able to load and execute the computer program306.

The controller300may be part of an apparatus30or system320. The apparatus30or system320may comprise one or more peripheral components312. The display200is a peripheral component. Other examples of peripheral components312may include: an audio output device or interface for rendering or enabling rendering of the sound space10to the user; a user input device for enabling a user to control one or more parameters of the method; a positioning system for positioning a sound object12and/or the user; an audio input device such as a microphone or microphone array for recording a sound object12; an image input device such as a camera or plurality of cameras.

The apparatus30or system320may be comprised in a headset for providing mediated reality.

The controller300may be configured as a sound rendering engine that is configured to control characteristics of a sound object12defined by sound content. For example, the rendering engine may be configured to control the volume of the sound content, a position of the sound object12for the sound content within the sound space10, a spatial extent of new sound object12for the sound content within the sound space10, and other characteristics of the sound content such as, for example, tone or pitch or spectrum or reverberation etc. The sound object12may, for example, be rendered via an audio output device or interface. The sound content may be received by the controller300.

The sound rendering engine may, for example comprise a spatial audio processing system that is configured to control the position and/or extent of a sound object12within a sound space10. The sound rendering engine may enable any properties of the sound object12to be controlled. For instance, the sound rendering engine may enable reverberation, gain or any other properties to be controlled.

FIG. 4illustrates a method according to examples of the disclosure. The method may be implemented using an apparatus30, controller300or system312as described above.

The method comprises, at block400, enabling an output of an audio mixer700to be rendered for a user500where the user500is located within a sound space10. The sound space10may comprise one or more sound objects12.

The audio mixer700may be arranged to receive a plurality of input channels and combine these to provide an output to the user500. In other examples the audio mixer700may be arranged to receive a single input channel. The single input channel could comprise a plurality of combined signals.

The one or more input channels comprises a plurality of microphone output signals obtained by a plurality of microphones504which are arranged to record the sound space10. In some examples one input channel could comprise a plurality of microphone output signals. In other examples a plurality of input channels could comprise a plurality of microphone output signals.

In some of these examples each of the plurality of input channels could comprise a single microphone output signal or alternatively, some of the plurality of input channels could comprise two or more microphone output signals.

The plurality of microphones504may comprise any arrangement of microphones which enables spatially diverse sound recording. The plurality of microphones504may comprise one or more microphone arrays502, and one or more close up microphones506or any other suitable types of microphones and microphone arrangements.

The plurality of microphones504may be arranged to enable a sound object12within the sound space10to be isolated. The sound object12may be isolated in that it can be separated from other sound objects within the sound space10. This may enable the microphone output signals associated with the sound object12to be identified and removed from the input channels provided to the mixer. The plurality of microphones504may comprise any suitable means which enable the sound object12to be isolated. In some examples the plurality of microphones504may comprise one or more directional microphones or microphone arrays which may be focused on the sound object12. In some examples the plurality of microphones504may comprise one or more microphones positioned close to the sound object12so that they mainly record the sound object. In some examples processing means may be used to analyse the input channels and/or the microphone output signals and identify the microphone output signals corresponding to the sound object12.

The output of the audio mixer700may be rendered using any suitable rendering device. In some examples the output may be rendered using an audio output device312positioned within a head set. The head set could be used for mediated reality applications or any other suitable applications.

The rendering device may be located separately to the audio mixer700. For example the rendering device may be worn by the user500while the device which comprises the audio mixer700may be in a device which is separate from the user. The output of the audio mixer700may be provided to the rendering device via a wireless communication link so that the user can move within the sound space10. The quality of the signal that can be transmitted via the wireless communication link may be limited by the bandwidth of the communication link. This may limit the quality of the audio output that can be rendered for the user via the audio mixer700and the headset.

At block401it is determined that a first microphone508can be used to record one or more sound objects12within the sound space10. The first microphone508may be a microphone508associated with the user500. In other examples the first microphone508could be one of the plurality microphones504.

The microphone508that is associated with the user500may be worn by, or positioned close to the user500. The microphone508that is associated with the user500may move with the user500so that as the user500moves through the sound space10the microphone508also moves. In some examples the microphone508may be positioned within the rendering device. For example, a mediated reality headset may also comprise one or more microphones.

Determining that a first microphone508can be used to record one or more sound objects12within the sound space10may comprise determining that the microphone508can obtain high quality audio signals. This may enable a high quality output, representing the sound object12, to be provided to the user500. The high quality output may enable the sound object12to be recreated more faithfully than the output of the audio mixer700. It may be determined that the audio signal has a high quality by determining that at least one parameter of the signal is within a threshold range. The parameters could be any suitable parameter such as, but not limited to, frequency range or clarity.

In some examples determining that a first microphone508can be used to record one or more sound objects12within the sound space10may comprise determining that the user500is located within a threshold distance of the one or more sound objects12. For example if the user500is located close enough to a sound object12it may be determined that the microphone508associated with the user500should be able to obtain a high quality signal. In some examples the direction of the user500relative to the sound object12may also be taken into account when determining whether or not a high quality signal could be obtained. The positioning device312of the apparatus30could be used to determine the relative positions of the user500and the sound object12.

The sound object may be an object that is positioned close to the first microphone508. In other examples the sound object could be located far away from the first microphone508.

At block402the method comprises enabling one or more of the microphone output signals to be, at least partially, removed from the input channel to the audio mixer700. This enables the controller300to switch into an improved bandwidth mode of operation.

In some examples enabling the microphone output signals to be, at least partially, removed may comprise sending a signal to the audio mixer700to cause the microphone output signals to be, at least partially, removed. In some examples the signal sent to the audio mixer700identifies the microphone output signals that can be, at least partially, removed. In other examples the signal sent to the audio mixer700may comprise information which enables the audio mixer700to identify the microphone output signals that can be, at least partially, removed.

Any suitable means may be used to identify the microphone output signals that can be, at least partially, removed from the input to the audio mixer700. In some examples the microphone output signals may be identified as the microphone output signals that correspond to the sound object12that can be recorded by the first microphone508. The microphone output signals that can be removed may be identified by isolating the sound object12and identifying the input channels associated with the isolated sound object12.

In some examples removing the microphone output signals from the input to the audio mixer700may comprise completely removing one or more microphone output signals so that the removed microphone output signals are no longer provided to the audio output mixer. In some examples one or more of the microphone output signals may be partially removed. In such cases part of at least one microphone output signal may be removed so that some of the microphone output signal is provided to the audio mixer700and some of the same microphone output signal is not provided to the audio mixer700.

Removing, at least part of, the one or more microphone output signals changes the output provided by the audio mixer700so that the sound object12may be removed, or partially removed, from the output. It is to be appreciated that in some examples a subset of microphone output signals would be removed so that at least some microphone output signals are still provided in the input channel to the audio mixer700. In other examples all of the microphone output signals could be removed. The number of microphone output signals that are, at least partially, removed and the identity of the microphone output signals that are, at least partially, removed would be dependent on the position of the user500relative to the sound objects12and the clarity with which the microphone508associated with the user500can record the sound objects. Therefore there may be a plurality of different improved bandwidth modes of operation available where different modes have different microphone output signals removed. The mode that is selected is dependent upon the user's position within the sound space10.

In examples of the disclosure the enabling the one or more of the microphone output signals to be, at least partially, removed from the input to the audio mixer700occurs automatically. The removal of at least part of the microphone output signals may occur without any specific input by the user500. For example, the removal may occur when it is determined that the microphone508associated with the user500can be used to record the sound object12.

In some, but not all examples, the method also comprises, at block403, replacing the removed one or more microphone output signals in the output provided to the user500with a signal recorded by the first microphone508. The signal recorded by the first microphone508is routed differently to the signals recorded by the plurality of microphones504. The signal recorded by the first microphone508is not provided to the audio mixer700. As the signals representing the sound object12are not routed through the audio mixer700and do not need to be transmitted to the user via the communication link. This means that they are not limited by the bandwidth of the communication link and so may enable a higher quality signal to be provided to the user500when the controller is operating in an improved bandwidth mode of operation. This may increase the efficacy of the available bandwidth between the audio mixer700and a user device710as it allows for a more efficient use of the bandwidth. In some examples this may optimize the available bandwidth between the audio mixer700and a user device710.

The higher quality of the signal provided to the user500may comprise one or more parameters of the audio output that has a higher threshold value in the signal provided by the microphone508associated with the user500compared to the signal routed via the audio mixer700. The parameters could be any suitable parameter such as, but not limited to, frequency range or clarity. The higher quality could be achieved using any suitable means. For example the first microphone508could have a higher sampling rate. This may enable more information to be obtained and enable the signal recorded by the first microphone508to be as faithful a reproduction of the sound object12as possible.

In some examples the higher quality may be achieved by reducing the data that needs to be routed via the audio mixer700. As one or more microphone output signals are removed from the input channel to the audio mixer this reduces the data that needs to be processed and transmitted by the audio mixer700. This may reduce the processing time and any latency in the output provided to the user. This may also reduce the amount of compression needed to transmit the signal and may enable a higher quality audio output to be provided.

FIG. 5illustrates an example of a sound space comprising a plurality of sound objects12A to12J. The sound objects12A to12J are distributed throughout the sound space. The example sound space ofFIG. 5could represent the recording of a band or orchestra or other situation comprising a plurality of sound objects12A to12J.

The sound space is three-dimensional, so that the location of the user500within the sound space has three degrees of freedom, up/down, forward/back, left/right and the direction that the user500faces within the sound space has three degrees of freedom, roll, pitch, yaw. The position of the user500may be continuously variable in location and direction. This gives the user500six degrees of freedom within the sound space.

A plurality of microphones504are arranged to enable the sound space to be recorded. The plurality of microphones504may comprise any means which enables spatially diverse sound recording. In the example ofFIG. 5the plurality of microphones504comprises a plurality of microphone arrays502A to502C. The microphone arrays502A to502C are positioned around the plurality of sound objects12A to12J. The plurality of microphones504also comprises a plurality of close up microphones506. In the example ofFIG. 5the close up microphones506A to506J are arranged close to the sound objects12A to12J so that the close up microphones506A to506J can record the sound objects12A to12J.

The user500is located within the sound space. The user500may be wearing an electronic device such as a headset which enables the user to listen to the sound space. In some examples the user500could be located within the sound space while the sound space is being recorded. This may enable the user500to check that the sound space is being recorded accurately. In some examples the user500could be using augmented reality applications, or other mediated reality applications, in which the user500is provided with audio outputs corresponding to the user's500position within the sound space.

The output signals of the plurality of microphones504may be provided to an audio mixer700. As a large number of microphones504are used to record the sound space this generates a large amount of data that is provided to the audio mixer700. However the amount of data that can be transmitted from the audio mixer700to the user's device may be limited by the bandwidth of the communication link between the user's device and the audio mixer700. In examples of the disclosure the user's device may be switched to an improved bandwidth mode of operation, as described above, so that some of the signals do not need to be routed via the audio mixer700.

FIG. 6illustrates the user500moving through the sound space as illustrated inFIG. 5. As the user500moves through the sound space the user's device may be switched between improved bandwidth modes of operation and normal modes of operation. In the normal mode of operation all of the signals obtained by the plurality of microphones504are routed via the audio mixer700while in an improved bandwidth mode of operation only some of the signals obtained by the plurality of microphones504are routed via the audio mixer700.

InFIG. 6the user500follows a trajectory indicted by the dashed line600. The user500moves from location I to location V via locations II, III and IV. The user500is wearing a headset or other suitable device which enables the output of an audio mixer700to be rendered to the user500. The output of the audio mixer700may provide a recording of the sound space to the user500.

The user500may also be wearing a microphone508. The microphone508may be provided within the headset or in any other suitable device. The user500may be wearing the microphone508so that as the user500moves through the sound space the microphone508also moves with them.

When the user500is located at location I the audio output that is provided to the user500comprises the output of the audio mixer700. This corresponds to the sound space as captured by the microphone arrays502A to502C and the close up microphones506A to506C. As a large number of microphones504are used to capture the sound scene the data may be compressed before being transmitted to the user500. This may limit the quality of the audio output.

In the example ofFIG. 6only sound objects12within a threshold area may be included in the output. The threshold area is indicated by the dashed line602. The sound objects12D,12G,12F and12J are located outside of the threshold area and so are excluded from the audio output. The signals captured by a close up microphones506D,506G,506F,506J would not be provided to the audio mixer700.

When the user500is located in the first location I the output of the audio mixer700is rendered via the user's headset or other suitable device. The output comprises the output of the microphone arrays502A to502C mixed with the outputs of the close up microphones506E,506A,506H,506I,506C,506B. At location I the user500is located above a threshold distance from the sound objects12E,12A,12H,12I,12C and12B. At this location it may be determined that a microphone508associated with the user500should not be used to capture these sound objects. This determination may be made based on the relative positions of the user500and the sound objects12E,12A,12H,12I,12C and12B and/or an analysis of the signal recorded by the microphone associated with the user500. In response to this determination the controller300remains in the normal mode of operation where all of the signals provided to the user500are routed via the audio mixer700.

The user500moves though the sound space from location I to location II. At location II the user500is close to the sound object12E but is still located above a threshold distance from the other sound objects12A,12H,12I,12C and12B. It may be determined that the microphone associated with the user500can capture the sound object12E with sufficient quality but not the other sound objects12A,12H,12I,12C and12B. In response to this determination the controller300switches into an improved bandwidth mode. The microphone output signals corresponding to the sound object12E are identified and removed from the input channels to the audio mixer700. These may be replaced in the output with a signal obtained by the microphone508associated with the user500. The signal from the microphone508associated with the user500is not provided to the audio mixer700. This signal from the microphone508associated with the user500is not restricted by the bandwidth of the communication link between the audio mixer700and the user's device. This may enable a higher quality signal to be provided to the user500.

The user500then moves though the sound space from location II to location III. At location III the user500is close to the sound objects12E,12A,12H,12I,12C and12B. It may be determined that the microphone508associated with the user500can capture the sound objects12E,12A,12H,12I,12C and12B. In response to this determination the controller300switches to a different improved bandwidth mode of operation in which the microphone output signals corresponding to the sound objects12E,12A,12H,12I,12C and12B are identified and removed from the input channels to the audio mixer700. These may be replaced in the output with a signal obtained by the microphone associated with the user500. In this location none of the close up microphones are used to provide a signal to the audio mixer700. The output provided to the user500may be a combination of the signal recorded by the microphone508associated with the user500and the signals recorded by the microphone arrays502A to502C.

The user500continues along the trajectory to location IV. At location IV the user500is still located close to the sound object12B but is now located above a threshold distance from the other sound objects12E,12A,12H,12I, and12C. It may be determined that the microphone associated with the user500can still capture the sound object12B with sufficient quality but not the other sound objects12E,12A,12H,12I and12C. In response to this determination the controller300switches to another improved bandwidth mode of operation in which the input channels to the audio mixer corresponding to the sound objects12E,12A,12H,12I, and12C are identified and reinstated in the inputs to the audio mixer700.

The user then continues to location V. At location V the user500is located above a threshold distance from the sound objects12E,12A,12H,12I,12C and12B. It is determined that the microphone508associated with the user can no longer record any of the sound objects12E,12A,12H,12I,12C and12B with sufficient quality and so the controller300switches back to the normal mode of operation. In the normal mode of operation all of the microphone output signals are reinstated in the inputs to the audio mixer700and the signal captured by the microphone508associated with the user500is no longer rendered for the user500.

As the system switches between the different modes of operation temporal latency information from the respective signals may be used to prevent transition artefacts from appearing. The temporal latency information is used to ensure that the signals that are routed through the audio mixer700are synchronized with the signals that are not routed through the audio mixer700.

FIGS. 7A and 7Bschematically illustrate the routing of signals captured by the plurality of microphones504in different modes of operation according to examples of the disclosure.

FIGS. 7A and 7Billustrates a system320comprising an audio mixer700, a user device710and a plurality of microphones504. The plurality of microphones504comprises a plurality of microphone arrays502A,502B and502C and also a plurality of close up microphones506A to506D. The plurality of microphones504may be arranged within a sound space10to enable a plurality of sound objects12to be recorded.

The audio mixer700comprises any means which may be arranged to receive the inputs channels704comprising the microphone output signals from the plurality of microphones504and combine these into an output signal for rendering by the user device710. The output of the audio mixer700is provided to the user device710via the communication link706. The communication link706may be a wireless communication link.

The user device710may be any suitable device which may be arranged to render an audio output for the user500. The user device710may be a head set which may be arranged to render mediated reality applications such as augmented reality or virtual reality. The user device710may comprise one or more microphones which may be arranged to record sound objects12that are positioned close to the user500.

When the system320is operating in a normal mode of operation all of the signals from the close up microphones506A to506D are provided to the audio mixer700and included in the output provided to the user device710as indicated by arrow712. The system320may operate within the normal mode of operation when the microphone within the user device710is determined not to be able to record sound objects within the sound space10with high enough quality. For example it may be determined that the distance between the user500and the sound object12exceeds a threshold.

When the system320switches from normal mode to the improved bandwidth mode the sound objects12may be recorded by the microphone508within the user device712. This enables the sound object12to be provided direct to the user500, as indicated by arrow702, without having to be routed via the audio mixer700.

FIG. 8schematically illustrates another system320that may be used to implement examples of the disclosure. In the example ofFIG. 8the determination of whether to use a normal mode or an improved bandwidth mode is made by the user device712.

The system320ofFIG. 8comprises a plurality of microphones504, an audio mixer700and a user device710which may be as described above. The system320also comprises an audio network806which is arranged to collect the signals from the plurality of microphones504and provide them in the input channels to the audio mixer700. In the example ofFIG. 4the audio mixer700has34input channels. Other numbers of input channels may be used in other examples of the disclosure.

The output of the audio mixer700is transmitted to the user device710as a coded stream802. The coded stream802may be transmitted via the wireless communication link.

In the example ofFIG. 8the user device710comprises a monitoring module804. The monitoring module804enables a monitoring application to be implemented. The monitoring application804may be used to determine whether or not a microphone508within the user device710can be used to record a sound object12. The monitoring application804may use any suitable methods to make such a determination. For example the monitoring application may monitor the quality of signals recorded by a microphone508within the user device710and/or may use positioning systems to monitor the position of the user500relative to the sound objects12.

If the monitoring application804may cause a signal808to be sent to the audio mixer700indicating which mode of operation the system320should operate in. If it is determined that the microphone508can be used to record the sound object12then the signal808indicates that the system320should operate in a reduced bandwidth mode of operation. If it is determined that the microphone508cannot be used to record the sound object12then the signal808indicates that the system320should operate in a normal mode of operation. Once the audio mixer700has received the signal808the audio mixer may remove and/or reinstate microphone output signals as indicated by the signal808.

FIG. 9schematically illustrates another system320that may be used to implement examples of the disclosure. In the example ofFIG. 9the determination of whether to use a normal mode or an improved bandwidth mode is made by a controller associated with the mixer700. The system ofFIG. 9comprises a plurality of microphones504, an audio mixer700and a user device710which may be as described above.

In the example ofFIG. 9the audio mixer700receives the microphone output signals from the plurality of microphones504. The audio mixer700also receives an input900comprising information on the sound space and the position of the user500within the sound space. The information relating to the sound space may comprise information indicating the locations of the sound objects12within the sound space and the user's position relative to the sound objects12. The input900may be obtained from a position system or any other suitable means.

The input signal900may be provided to a monitoring module804which may comprise a monitoring application. The monitoring application804may use the information received in the input signal900to determine whether or not a microphone508within the user device710can be used to record a sound object12and cause the system320to be switched between the normal modes of operation and the improved bandwidth modes of operation as necessary.

In the example ofFIG. 9the audio mixer700comprises a channel selection module902which is arranged to remove and reinstate the microphone output signals from the input channel of the audio mixer700as indicated by the monitoring module804. This enables the system320to be switched between the different modes of operation. Once the microphone output signals have been removed or reinstated as needed the signal906is transmitted to the user device710via a wireless network904. The audio mixer700may also send a signal908indicating that the signal recorded by a microphone508in the user device710is to be provided to the user500.

The user device710may also provide a feedback signal910to the audio mixer700. The feedback signal910could be used to enable the position of the user500to be determined. In some examples the feedback signal910could be used to reduce artifacts from appearing as the system320switches between different modes of operation.

FIG. 10schematically illustrates another method according to examples of the disclosure. The example method ofFIG. 10could be implemented using the systems320as described above.

At block1000the microphone508of the user device710records the audio scene at the location of the user500and provides a coded bitstream of the captured audio scene to the audio mixer700. In some examples the coded bitstream may comprise a representation of the audio scene. The representation may comprise spectrograms, information indicating the direction of arrival of dominant sound sources in the location of the user500and any other suitable information.

In some examples the user device710may also provide information relating to user preferences to the audio mixer700. For example the user of the user device710may have selected audio preferences which can then be provided to the audio mixer700.

At block1001the audio mixer700selects the content for the output to be provided to the user500. This selection may comprise selecting which microphone output signals to be removed and reinstated.

At block1002the audio mixer700identifies the sound objects12that are close to the user. The audio mixer700may identify the sound objects12by comparing the spectral information obtained from the microphone508in the user device710with the audio data obtained by the plurality of microphones504. This may enable sound objects12that could be recorded by the microphone508in the user device710to be identified.

Any suitable methods may be used to compare the spectral information obtained from the microphone508in the user device710with the audio data obtained by the plurality of microphones504. In some examples the method may comprise matching spectral properties and/or waveform matching for a given set of spatiotemporal coordinates.

At block1003the clarity of any identified sound objects12is analyzed. This analysis may be used to determine whether or not the microphone508in the user device710can be used to capture the sound object12with sufficient quality.

The analysis of the clarity of the identified sound objects12comprises comparing the audio signals from the microphone508in the user device710with the signals from the plurality of microphones504. Any suitable methods may be used to compare the signals. In some examples the analysis may combine time-domain and frequency-domain methods. In such examples several separate metrics may be derived from the different captured signals and compared.

At block1004the analysis of the sound objects12is used to determine whether or not the microphone508in the user device710can be used to record the sound object12and identify which microphone output signals should be included in the output of the audio mixer700and which should be replaced with the output of the microphone508in the user device710. This information is provided to the audio mixer700to enable the audio mixer700to control the mixing of the input channels as required.

Once the audio mixer700has received the information indicating the selection of the input channels to be transmitted the audio mixer700controls the mixing of the input channels as needed and provides, at block1005, the modified output to the user device710.

The methods as described with reference to the Figures may be performed by any suitable apparatus (e.g. apparatus30), computer program (e.g. computer program306) or system (e.g. system320) such as those previously described or similar.

In the foregoing examples, reference has been made to a computer program or computer programs. A computer program, for example either of the computer programs306or a combination of the computer programs306may be configured to perform the methods.

Also as an example, an apparatus30may comprise: at least one processor302; and at least one memory304including computer program code the at least one memory304and the computer program code306configured to, with the at least one processor302, cause the apparatus30at least to perform: enabling400an output of an audio mixer700to be rendered for a user500where the user500is located within a sound space10, wherein at least one input channel is provided to the audio mixer700and the at least one input channel receives a plurality of microphone output signals obtained by a plurality of microphones504recording the sound space10; determining that a microphone508associated with the user500can be used to record one or more sound objects12within the sound space10; and enabling one or more of the plurality of microphone output signals to be removed from the at least one input channel to the audio mixer700.

The computer program306may arrive at the apparatus30via any suitable delivery mechanism. The delivery mechanism may be, for example, a non-transitory computer-readable storage medium, a computer program product, a memory device, a record medium such as a compact disc read-only memory (CD-ROM) or digital versatile disc (DVD), an article of manufacture that tangibly embodies the computer program306. The delivery mechanism may be a signal configured to reliably transfer the computer program306. The apparatus30may propagate or transmit the computer program306as a computer data signal.

It will be appreciated from the foregoing that the various methods described may be performed by an apparatus30, for example an electronic apparatus30.

The electronic apparatus30, may in some examples be a part of an audio output device such as a head-mounted audio output device or a module for such an audio output device. The electronic apparatus30, may in some examples additionally or alternatively be a part of a head-mounted apparatus comprising the rendering device(s) that renders information to a user visually and/or aurally and/or haptically.

References to “computer-readable storage medium”, “computer program product”, “tangibly embodied computer program” etc. or a “controller”, “computer”, “processor” etc. should be understood to encompass not only computers having different architectures such as single /multi-processor architectures and sequential (Von Neumann)/parallel architectures but also specialized circuits such as field-programmable gate arrays (FPGA), application specific circuits (ASIC), signal processing devices and other processing circuitry. References to computer program, instructions, code etc. should be understood to encompass software for a programmable processor or firmware such as, for example, the programmable content of a hardware device whether instructions for a processor, or configuration settings for a fixed-function device, gate array or programmable logic device etc.

As used in this application, the term “circuitry” refers to all of the following:

(b) to combinations of circuits and software (and/or firmware), such as (as applicable): (i) to a combination of processor(s) or (ii) to portions of processor(s)/software (including digital signal processor(s)), software, and memory(ies) that work together to cause an apparatus, such as a mobile phone or server, to perform various functions and

(c) to circuits, such as a microprocessor(s) or a portion of a microprocessor(s), that require software or firmware for operation, even if the software or firmware is not physically present.

For instance in some examples the microphone output signals that are removed from the output of the audio mixer700are replaced with a signal recorded by the microphone508associated with the user500. In other examples the signal recorded by the microphone508associated with the user500might not be used and the user could the sound objects12directly. This could be useful in implementations where there is very little delay in the outputs provided by the audio mixer700.

As used here “module” refers to a unit or apparatus that excludes certain parts/components that would be added by an end manufacturer or a user. The controller300may, for example be a module. The apparatus may be a module. The rendering devices312may be a module or separate modules.

The term “comprise” is used in this document with an inclusive not an exclusive meaning. That is any reference to X comprising Y indicates that X may comprise only one Y or may comprise more than one Y. If it is intended to use “comprise” with an exclusive meaning then it will be made clear in the context by referring to “comprising only one” or by using “consisting”.