Changing spatial audio fields

Embodiments herein relate generally to changing spatial audio fields that are defined for audio sources. In the embodiments, the spatial audio fields are indicated to a user performing audio mixing, for instance by displaying them as polygons on a touch screen. The spatial audio fields move as the related audio sources move, and/or as the position of a notional consumer changes. Apparatus of the embodiments is configured to detect whether at any time (initially, or after movement) there is overlapping of two spatial audio fields. If an overlap is detected, this is indicated to a user performing audio mixing The apparatus then responds to a user input (e.g. a gesture on the touch screen) by detecting the nature of the user input and then moving or sizing one or both of overlapping spatial audio fields and such that overlapping is avoided or reduced.

FIELD

This specification relates generally to methods and apparatuses for changing spatial audio fields.

BACKGROUND

Spatial audio refers to playable audio data that exploits sound localisation. In a real world space, for example in a concert hall, there will be multiple audio sources, for example the different members of an orchestra or band, located at different locations on the stage. The location and movement of the sound sources is a parameter of the captured audio. In rendering the audio as spatial audio for playback, such parameters are incorporated in the data using processing algorithms so that the listener is provided with an immersive and spatially oriented experience.

It is known to process audio captured via a microphone array into spatial audio; that is audio with a spatial percept. The intention is to capture audio so that when it is rendered to a user the user will experience the sound field as if they are present at the location of the capture device.

An example application of spatial audio is in virtual reality (VR) whereby both video and audio data is captured within a real world space. In the rendered version of the space, i.e. the virtual space, the user, through a VR headset, may view and listen to the captured video and audio which has a spatial percept.

SUMMARY

According to a first aspect of the present invention, there is provided a method comprising:receiving at least first and second audio signals representing audio respectively from first and second audio sources in a space;defining for the first and second audio sources first and second spatial audio fields, respectively, each being indicative of the propagation of the respective audio signals within the space;detecting at least partial overlapping of the first spatial audio field at least with the second spatial audio field in an overlap region;causing an indication of the at least partial overlapping to a user; andresponding to a user input by moving or re-sizing one or both of the first spatial audio field and the second spatial audio field.

The method may comprise responding to the user input by reducing a size of one or both of the first spatial audio field and the second spatial audio field.

The method may comprise responding to the user input by moving one or both of the first spatial audio field and the second spatial audio field in a horizontal direction. Alternatively, the method may comprise responding to the user input by moving one or both of the first spatial audio field and the second spatial audio field in a vertical direction.

The method may comprise responding to the user input by placing the first spatial audio field and the second spatial audio field adjacent to one another in a horizontal direction. Alternatively, the method may comprise responding to the user input by placing the first spatial audio field and the second spatial audio field adjacent to one another in a vertical direction.

The method may comprise responding to the user input by splitting the first spatial audio field into two portions and placing them either side of the second spatial audio field.

The method may comprise responding to the user input by moving or re-sizing the first spatial audio field but not the second spatial audio field.

The method may comprise responding to the user input by moving a rearmost one of the first and second audio fields to a frontmost position.

The method may comprise causing an indication of the at least partial overlapping to a user by changing a visual representation of the overlap region.

The method may comprise causing an indication of the at least partial overlapping to a user by causing a visual representation of the overlap region to change in an alternating manner.

The at least first and second audio signals may represent live audio and the detecting and the causing an indication may be performed in real time.

Another aspect provides apparatus configured to perform any of the above methods.

A further aspect provides a computer program comprising instructions that when executed by computing apparatus causes it to perform any of the above methods.

Apparatus comprising:at least one processor;at least one memory having computer-readable instructions stored thereon, the computer-readable instructions when executed by the at least one processor causing the apparatus to perform:receiving at least first and second audio signals representing audio respectively from first and second audio sources in a space;defining for the first and second audio sources first and second spatial audio fields, respectively, each being indicative of the propagation of the respective audio signals within the space;detecting at least partial overlapping of the first spatial audio field at least with the second spatial audio field in an overlap region;causing an indication of the at least partial overlapping to a user; andresponding to a user input by moving or re-sizing one or both of the first spatial audio field and the second spatial audio field.

The computer-readable instructions when executed by the at least one processor may cause the apparatus to perform reducing a size of one or both of the first spatial audio field and the second spatial audio field.

The computer-readable instructions when executed by the at least one processor may cause the apparatus to perform responding to the user input by moving one or both of the first spatial audio field and the second spatial audio field in a horizontal direction.

The computer-readable instructions when executed by the at least one processor may cause the apparatus to perform responding to the user input by moving one or both of the first spatial audio field and the second spatial audio field in a vertical direction.

The computer-readable instructions when executed by the at least one processor may cause the apparatus to perform responding to the user input by placing the first spatial audio field and the second spatial audio field adjacent to one another in a horizontal direction.

The computer-readable instructions when executed by the at least one processor may cause the apparatus to perform responding to the user input by placing the first spatial audio field and the second spatial audio field adjacent to one another in a vertical direction.

The computer-readable instructions when executed by the at least one processor may cause the apparatus to perform responding to the user input by splitting the first spatial audio field into two portions and placing them either side of the second spatial audio field.

The computer-readable instructions when executed by the at least one processor may cause the apparatus to perform responding to the user input by moving or re-sizing the first spatial audio field but not the second spatial audio field.

The computer-readable instructions when executed by the at least one processor may cause the apparatus to perform responding to the user input by moving a rearmost one of the first and second audio fields to a frontmost position.

The computer-readable instructions when executed by the at least one processor may cause the apparatus to perform causing an indication of the at least partial overlapping to a user by changing a visual representation of the overlap region.

The computer-readable instructions when executed by the at least one processor may cause the apparatus to perform causing an indication of the at least partial overlapping to a user by causing a visual representation of the overlap region to change in an alternating manner.

The at least first and second audio signals may represent live audio and the computer-readable instructions when executed by the at least one processor may cause the apparatus to perform the detecting and the causing an indication in real time.

Another aspect provides a computer-readable medium having computer-readable code stored thereon, the computer-readable code, when executed by at least one processor, cause performance of:receiving at least first and second audio signals representing audio respectively from first and second audio sources in a space;defining for the first and second audio sources first and second spatial audio fields, respectively, each being indicative of the propagation of the respective audio signals within the space;detecting at least partial overlapping of the first spatial audio field at least with the second spatial audio field in an overlap region;causing an indication of the at least partial overlapping to a user; and responding to a user input by moving or re-sizing one or both of the first spatial audio field and the second spatial audio field.

DETAILED DESCRIPTION OF EMBODIMENTS

Embodiments herein relate generally to changing spatial audio fields that are defined for audio sources. In the embodiments, the spatial audio fields are indicated to a user performing audio mixing, for instance by displaying them as polygons on a touch-screen.

The spatial audio fields move as the related audio sources move, and/or as the position of a notional consumer changes. Apparatus of the embodiments is configured to detect whether at any time (initially, or after movement) there is overlapping of two spatial audio fields. If an overlap is detected, this is indicated to a user performing audio mixing. indicating can be done through changing display parameters of the polygon representations. The apparatus then responds to a user input (e.g. a gesture on the touch screen) by detecting the nature of the user input and then moving or sizing one or both of overlapping spatial audio fields and such that overlapping is avoided or reduced. The embodiments avoid or ameliorate the situation where sound from overlapping spatial audio fields may appear distorted from the perspective of the user or one sound source may be masked entirely.

The embodiments described herein allow a user (such as a mixing operator) to move or change the shape of otherwise overlapping spatial audio fields in a VR environment. For example, an overlapping spatial audio field may be cropped so that no overlapped spatial audio field is present in resulting mixed audio. This may be achieved ‘live’ as the content is captured, for live consumption or for consumption later.

An example application is in a VR capture and rendering system in which video is also captured and rendered to provide an immersive user experience. Nokia's OZO® VR camera is used as an example of a VR capture device which comprises a microphone array to provide a spatial audio signal, but it will be appreciated that the embodiments are not limited to VR applications nor the use of microphone arrays at the video capture point. Local microphones (e.g. Lavalier microphones) or instrument pickups may be employed, for example.

Referring toFIG. 1, an overview of a VR capture scenario1is shown together with a capture, mixing and rendering system (CRS)15with an associated user interface16. The Figure shows in plan-view a real world space3which may be for example a concert hall or other music venue. The CRS15is applicable to any real world space, however. A VR capture device6for video and spatial audio capture may be supported on a floor5of the space3in front of multiple audio sources7,8, in this case two musicians and associated instruments; the position of the VR capture device6is known, e.g. through predetermined positional data or signals derived from a positioning tag on the VR capture device6. The VR capture device6in this example may comprise a microphone array configured to provide spatial audio capture.

As well as having an associated microphone or audio feed, the audio sources7,8may carry a positioning tag. A positioning tag may be any module capable of indicating through data its respective spatial position to the CRS15. For example the positioning tag may be a high accuracy indoor positioning (HAIP) tag which works in association with one or more HAIP locators20within the space3. HAIP systems use Bluetooth Low Energy (BLE) communication between the tags and the one or more locators20. For example, there may be four HAIP locators mounted on, or placed relative to, the VR capture device6. A respective HAIP locator may be to the front, left, back and right of the VR capture device6. Each tag sends BLE signals from which the HAIP locators derive the tag, and therefore, audio source location.

In general, such direction of arrival (DoA) positioning systems are based on (i) a known location and orientation of the or each locator, and (ii) measurement of the DoA angle of the signal from the respective tag towards the locators in the locators' local co-ordinate system. Based on the location and angle information from one or more locators, the position of the tag may be calculated using geometry.

The CRS15is a processing system having an associated user interface (UI)16which will be explained in further detail below. As shown inFIG. 1, the CRS15receives as input from the VR capture device6spatial audio and video data, and positioning data, through a signal line17.

Alternatively, the positioning data may be received from the HAIP locator20. The CRS15also receives as input from each of the audio sources7,8audio data and positioning data from the respective positioning tags, or the HAIP locator20, through separate signal lines18. The CRS15generates spatial audio data for output to a user device19, such as a VR headset with video and audio output.

The input audio data may be multichannel audio in loudspeaker format, e.g. stereo signals, 4.0 signals, 5.1 signals, Dolby Atmos® signals or the like. Instead of loudspeaker format audio, the input may be in the multi microphone signal format, such as the raw eight signal input from the Nokia OZO® VR camera, if used for the VR capture device6. The microphone signals can then be rendered to loudspeaker or binaural format for playback.

FIG. 2shows an example schematic diagram of components of the CRS15. The CRS15has a controller22, a touch sensitive display24comprised of a display part26and a tactile interface part28, hardware keys30, a memory32, RAM34and another (e.g. wired, serial or parallel) input interface36. The controller22is connected to each of the other components in order to control operation thereof. The touch sensitive display24is optional, and as an alternative a non-touch display may be used with the hardware keys30and/or a mouse peripheral used to control the CRS15by suitable means. The input interface36may be a microphone and speech recognition interface, allowing voice control.

The memory32may be a non-volatile memory such as read only memory (ROM) a hard disk drive (HDD) or a solid state drive (SSD). The memory32stores, amongst other things, an operating system38and software applications40. The RAM34is used by the controller22for the temporary storage of data. The operating system38may contain code which, when executed by the controller22in conjunction with RAM34, controls operation of each of the hardware components of the terminal.

The controller22may take any suitable form. For instance, it may be a microcontroller, plural microcontrollers, a processor, or plural processors. The controller includes circuitry.

In embodiments herein, the software application40is configured to provide video and distributed spatial audio capture, mixing and rendering to generate a VR environment, or virtual space, including the rendered spatial audio. In some embodiments, only spatial audio may be provided without the need for video.

The software application40also provides the UI16, through its output to the display24and receives user input through the tactile interface28or other input peripherals such as the hardware keys30or a mouse (not shown). Other embodiments may include a hand gesture input device for identifying hand movements for controlling the UI16. Here, the tactile interface28is not necessary. The hand gesture input device may for example include a glove having a number of sensors to detect movement of the hand, or an array of cameras for identifying and tracking a user's hand. One or more mixing controls may alternatively, or additionally, be provided as the input peripherals. For example, a rotatable knob may be associated with each audio source for controlling some aspect of the audio generated by that audio source. The mixing step of generating a VR environment may be performed manually through the UI16, through one or more other input peripherals, or all or part of said mixing step may be performed automatically. The software application40may render the virtual space, including the spatial audio, using known signal processing techniques and algorithms based on the mixing step.

The input interface36receives video and audio content data from the capture device6. The capture device6may be a VR capture device such as Nokia's OZO® device. The input interface36also receives audio content data from each of the audio sources7,8. The input interface36further receives the positioning data from (or derived from) the positioning tags on each of the VR capture device6and the audio sources7,8. From the positioning data may be made an accurate determination of the positions of the related components in the real world space3.

The software application40may be configured to operate in any of real-time (i.e. ‘live’), near real-time or even offline using pre-stored captured data.

The software application40is arranged to be operated to avoid or at least mitigate issues of unwanted audio masking when the rendered data is being consumed by a user. In this context, audio masking is the perceived effect of sounds from two or more audio sources overlapping. If a user is positioned in the virtual world at a location where there is spatial audio overlap, then one louder sound may interfere with, or block, a quieter sound. The rendered output may not create the desired perceived audio scene for the listener.

Each audio source7,8emitting a sound has an associated spatial audio field. The skilled person's common general knowledge includes the teaching of Pihlajamaki T., Santala O., & Pulkki V “Synthesis of Spatially Extended Virtual Sources with Time-Frequency Decomposition of Mono Signals”, J. Audio Eng. Soc., Vol. 62, No. 7/8, 2014. Here, it is explained how to create spatial audio fields. In one example, the method divides a sound into frequency bands using short-time Fourier transform (STFT) and then spatially distributes the frequency components to N discrete spatial locations around the desired spatial spread (for example, 90 degrees).

The spatial audio field is the two or three-dimensional space over which the audio source's7,8audio signals propagate at a given time. For ease of explanation, we will illustrate operation with respect to two-dimensional audio fields in top-plan view.

An audio field for a given audio source may change over time. For example, the audio field may move in correspondence with an audio source moving. If a musical artist is walking across a stage, and passes in front of another artist, then their respective audio fields will move in correspondence and may overlap with respect to a consuming user's position in the virtual space.

Additionally, or alternatively, a mixing operator may manually move or pan the audio field to suit a particular application or user experience.

Additionally, or alternatively, an audio field may be enlarged without the audio source necessarily moving. This may occur if the volume of the audio source increases. Additionally, or alternatively, mixing operator may widen the audio field to suit a particular application or user experience.

These examples may be collectively termed as audio field movement in this context because there is a spatial change from a current state. The movement may be caused by user input and/or through automatic adjustment. The following examples focus on movement due to user input, but automatic adjustment may result from the sound source nearing the VR capture device6and/or an algorithm whereby if the amplitude of an audio source exceeds a threshold, it is then made wider rather than louder.

Referring toFIGS. 3aand 3b, a potential masking scenario is represented. A two-dimensional spatial area50indicates the overall audio field around a user's current position52. Referring toFIG. 3a, two audio sources7,8are positioned with respective azimuth angles of approximately 30° and −30° to produce respective audio fields70,80which indicate the direction of audio signal propagation and which do not overlap. If the audio fields70,80are panned (rotated) relative to one another, as shown inFIG. 3b, overlap may occur resulting in an interference or masking zone58which may produce unwanted effects.

The software application40operates to alert the user to overlapping spatial audio fields. Alerting may occur by changing the displayed representation of the audio fields, but it may instead occur haptically, aurally or through a different visually perceivable indicator, or through some combination thereof. Any aural alert is separate from masking or distortion etc. that ordinarily results from overlapping spatial audio fields. Advantageously, the alert identifies the overlap between the spatial audio fields. Once the user is alerted to the overlap, they can decide what remedial action to take.

The software application40may be controlled by the user through the user interface16to remove (or at minimum significantly reduce) any overlapping zones and therefore avoid or mitigate masking.

The word “zone” may refer to any definable area of two- or three-dimensional space. The manner in which the modification is performed may take various forms, and some specific examples will be described with reference toFIGS. 6ato10.

Generally speaking, the software application stores mapping between user inputs and spatial audio field changes. In response to detecting a user input, the software application responds by performing the corresponding changing of one or more of the associated spatial audio fields.

The changing of the spatial audio fields may involve reducing a size of one or both of the overlapping spatial audio fields, for instance such that the spatial audio fields become contiguous or separated by a relatively small gap. The changing of the spatial audio fields may involve placing the first spatial audio field and the second spatial audio field adjacent to one another in a horizontal direction. The changing of the spatial audio fields may involve moving one or both of the first spatial audio field and the second spatial audio field in a vertical direction, to provide complete or partial separation between them. If one of the audio sources7,8is much quieter than the other one of the audio sources7,8and they lie in the same direction from the perspective of the user, the quieter audio source7,8may not be heard at all if their respective spatial audio fields70,80overlap. Generally, separating the sounds spatially helps the listener hear both sounds more clearly.

The changing of the spatial audio fields may involve splitting one spatial audio field into two portions and placing them either side of the other spatial audio field.

In addition to or alternatively to moving or splitting one or plural spatial audio fields, one or both may be re-sized. Re-sizing by reducing size allows overlapping to be avoided or reduced without altering the position of the spatial audio field as much as would otherwise be required.

The changing of the spatial audio fields may involve moving or re-sizing the first spatial audio field but not the second spatial audio field. Thus, priority of a dominant spatial audio field (such as one relating to a lead singer or lead musician) can be preserved in the mixed audio.

FIG. 5shows examples of spatial audio fields70,80from another perspective. Here, the perspective is of the user standing at the position of the VR capture device6. The view is towards two audio sources7,8. The first audio source7is a microphone. The second audio source is a keyboard8. Each audio source7,8has a respective spatial audio field70,80associated with it. The spatial audio fields70,80are shown as being rectangles. This should not be considered limiting. The spatial audio fields70,80may be any shape of polygon, such as a triangles or ellipses. Each of the spatial audio fields70,80may have different types of shapes. For example, the first spatial audio field70may be an ellipse, while the second spatial audio field may be a square.

The first spatial audio field70and the second spatial audio field80are shown here as being equal in length and width. However, the size of the spatial audio fields70,80varies according to volume and/or distance from the VR capture device6and/or spatial extent of the audio source7,8and/or the mixing operator's (i.e. the user's) perspective to the audio source in virtual reality.

FIG. 4shows an overview flow diagram of the capture, mixing and rendering stages of software application40. The mixing and rendering stages may be combined. First, video and audio capture is performed in step4.1. Next mixing is performed in step4.2. This is followed by rendering in step4.3. Mixing (step4.2) may be dependent on a manual or automatic control step4.4. Automatic control may be based on attributes of the captured video and/or audio.

In the embodiments described below, it is assumed that manual control is used to move and/or adjust one or more spatial audio fields, either through the touch sensitive display24or using one or more mixing controls. Other attributes may be used.

Step4.5indicates the user interface output which provides real- or near-real time visual feedback of the mixed output which will result from the commanded movement based on the operation of the software application40.

When an overlap incident occurs, the software application40is configured to alert the user. For example, the software application40causes the overlapping area to flash.

Advantageously, the embodiments described herein enable interactions in virtual reality on top of images of the sound sources7,8to correct their masking issues. It may be said that the interactions are performed ‘live’, rather than the spatial audio fields being preconfigured.

Examples of how the mixing step4.2may be performed by the software application40will now be described with reference toFIGS. 6ato10.

In the example shown inFIG. 6a, the first spatial audio field70and the second spatial audio field80partially overlap in an overlapping region58. The second spatial audio field80is shown as being slightly elevated above the first spatial audio field70, but this is merely to exemplify the overlapped region and make it clear that the second spatial audio field80is disposed behind the first spatial audio field70. In other words, as with the Figures that follow, the first spatial audio field70is to be considered the front (or dominant) spatial audio field, and the second spatial audio field80is to be considered the rear (not dominant) spatial audio field.

The overlapping region58is caused to be represented differently to non-overlapping regions. For instance, the overlapping region58may be caused to flash (alternate between different brightnesses) or change in colour. This allows the user to identify the full horizontal extent of the second spatial audio field80.

A user interaction, or gesture, is indicated by the arrow. The user gesture is a user interaction with the touch sensitive display24. Alternatively, the user gesture may be a user interaction with a mouse or other input device. For example, the user gesture may be a voice command received through a microphone. Alternatively again, the user gesture may be an interaction in the virtual reality environment. For example, instead of swiping or tapping on the touch sensitive display24, the user may swipe or tap virtual reality objects in free space. For example the user gesture may be a non-touch hand gesture, like a swipe gesture.

InFIG. 6a, the user gesture is a swipe gesture. The start of the swipe gesture is at a position in the representation of the overlapped region58and the movement of the gesture is in a downwards direction. As shown inFIG. 6b, the effect of the swipe gesture is to eliminate the overlapped region58by assigning half of the overlapped region58to the first spatial audio field and the other half of the overlapped area to the second spatial audio field80. In effect, each of the first spatial audio field70and second spatial audio field80are reduced in horizontal extent (or length) by an amount equal to half of the horizontal extent (or length) of the overlapped region58. The resulting first and second spatial audio fields70,80may not be equal in length, although the overlapped region is equally divided between them. In other words, each of the first and second spatial audio fields are cropped, or cut, such that both of the audio sources7,8can be heard when facing in a direction in which they overlap. The reduced first and second spatial audio regions are substantially horizontally adjacent each other.

In particular, the CRS15responds to the received user gesture by cropping each of the first and second spatial audio fields70,80. The CRS15crops the adjacent ends of the spatial audio fields70,80equally to remove the overlapping region58. This way it may be easier for the user to focus on the sounds generated by either of the audio sources7,8and hear them at different spatial locations. Sounds at horizontally separate locations are easier to perceive than sounds at vertically separate locations.

FIG. 7aillustrates another method of separating two overlapping spatial audio fields70,80. Here, the first spatial audio field70fully overlaps the second spatial audio field80, such that little or no sound would be heard from the second audio source80. The overlapped region58is the same length as the second spatial audio field80. As no part of the second spatial audio field80visible to the user from this perspective, the user is made aware of the overlap by the overlapping region58flashing, or changing in colour.

InFIG. 7a, the user gesture is a swipe-to-cut interaction. The gesture comprises two components: a first component includes leftward movement and a second component includes a rightward movement. Each component starts at a location on the representation of the first spatial audio field70, and in particular at opposite ends thereof. Essentially, the user swipes both ends of the first spatial audio field70to pull the first spatial audio field70apart.

As shown inFIG. 7b, the effect of the swipe-to-cut interaction is to create a gap in the first spatial audio field70through which the second audio source8can be heard. The first spatial audio field70is divided into two parts70a,70b. The two parts70a,7bare disposed adjacent the second spatial audio field80on opposite sides, such that there is no overlapping region.

In particular, the CRS15responds to the received user gesture by dividing the first spatial audio field70into two separated parts70a,70b. The two parts are spatially separated by the full length of the second spatial audio region80. In other words, the CRS15removes the section of the first spatial audio field70that overlaps the second spatial audio field80. Therefore, the second audio source8is not distorted by the first audio source7.

FIG. 8aillustrates another method of separating two overlapping spatial audio fields70,80. Here, the first spatial audio field70fully overlaps the second spatial audio field80, such that little or no sound can be heard from the second audio source80. The overlapped region58is the same length as the second spatial audio field80. The overlapping region58is caused to flash, or change in colour. This allows the user to identify the overlap even though no part of the second spatial audio field80visible to the user from this perspective.

InFIG. 8a, the user gesture is a swipe-to-gut interaction. Essentially, the user swipes a hole in the first spatial audio field70. This is achieved by a gesture starting at the top part of the first spatial audio field70and moving from the top to the bottom of the first spatial audio field70. The gesture may involve one or two fingers.

As shown inFIG. 8b, the effect of the user gesture is to split the first spatial audio field70into a first part70aand a second part70b. The first part70ais disposed adjacent one side of the second audio field80, while the second part70bis arranged to overlap part of the second audio field at the end of the second spatial audio region80that is opposite the first part70aof the first spatial audio region70.

Thus, the second sound source8, disposed behind the first sound source7, can be heard clearly when facing the arrangement from an angle where the first and second sound sources7,8overlap.

In particular, the CRS15responds to the received user gesture by removing a part of the first spatial audio field70at the overlapping region58. This reveals part of the second spatial audio field80. While there is still an overlapping region58, there is also a zone where the second audio source8can be heard without interference from the first audio source7, and a zone where there first audio source7can be heard without interference from the second audio source8.

FIG. 9ashows another example of an overlapping region58. Here, the first spatial audio field70partially overlaps the second spatial audio field80. The second spatial audio field80is fully overlapped in the horizontal direction, but is caused to be displayed extending from behind the first spatial audio field70in the vertical direction.

FIG. 9ashows the user gesture being a hold-and-move interaction. The user is able to touch part of the second spatial audio field80and move it above or below the first spatial audio field70, such that there is no longer an overlapping region. The effect is shown inFIG. 9b. This allows both audio sources7,8to be heard when facing in the shown perspective, although the second audio source8is be perceived to be at a different height than the first audio source7.

Responsive to it being determined that the first spatial audio field70overlaps the second spatial audio field80, the CRS15may be configured to overlay a representation of the second spatial audio field80on the first spatial audio field70. For example, the overlaid representation may be given a dotted perimeter line to indicate that the first spatial audio field70is actually in front of the second spatial audio field80. Without this representation, the second spatial audio field80would be completely masked such that the user could not interact with it.

The hold-and-move gesture shown inFIG. 9amay involve the user double tapping, or touching and holding, the representation of the second spatial audio field80in order to move it. Alternatively, where part of the second spatial audio field80is visible to the user, the user is able to interact directly with the second spatial audio field80rather than a representation thereof.

While the examples shown inFIGS. 6ato 9bindicate the user gestures being swipe actions in the vertical or horizontal directions, it would be readily apparent that the spatial audio fields70,80occur in three-dimensional space, and therefore a gesture from any direction could be used to remove the overlapped region58. For example, in the case ofFIGS. 7aand 7b, the swipe-to-cut interaction could be a vertical swipe, such that the first spatial audio field70is split into two parts70a,70barranged on the top and bottom sides of the second spatial audio field80. In further examples, the first and second spatial audio fields70,80are spatially separated by an increase in depth, rather than a horizontal or vertical displacement.

FIG. 10illustrates another method of handling overlapping spatial audio fields. Here, the positions of sound sources7,8are reversed in response to a user input. In other words, the audio source8at the rear of the first audio source7is moved to the front, and vice versa. Therefore, the audio source that in the real environment is to the rear now masks the audio source that in the real environment is to the front. The spatial order of the audio sources7,8may be changed for example by the user tapping on one of the first and second spatial audio fields70,80. By moving the rearmost audio source to the front, the audio source is no longer overlapped. In the case of the audio source that was at the front being larger (e.g. wider), the change results in both audio sources being present, whereas before the change one was wholly or largely masked.

Visually, this re-ordering is implemented by obtaining video images from multiple cameras and extracting image data for the audio sources7,8, such as performers. The audio source7originally at the front of another audio source8but moved to the back may be made semi-transparent in the rendered environment.

Where the second spatial audio field80is disposed behind the first spatial audio field70, at least part of the second spatial audio field80may be deleted from the overlapping region58. Therefore, instead of a hole being made in the first spatial audio field70as described above, the overlapping region58may be removed by deleting the overlapped part of the second spatial audio field80. This can effectively eliminate any impact on the first audio source7of sound generated by the second audio source8. Where the second spatial audio field80is spatially wider than the first spatial audio field70, the second audio source8can still be heard in the non-overlapping region. In other words, the remaining parts of the second spatial audio field80are arranged adjacent to opposite ends of the first spatial audio field70.

The choice of means for separating first and second spatial audio fields70will be determined by the mixing operator depending on sound types and desired artistic effect present in the scene or required. For example, the example shown inFIGS. 7aand 7bwhere one spatial audio field is split into two portions might be preferential for a scene containing a naturally wide sound (such as a drum set or keyboards), and a lead instrument which would naturally sound good as a narrow source (such as vocals).

A method of handling audio mixing will now be described with reference toFIG. 11.

In a first step1000, a plurality of audio signals corresponding with a plurality of audio sources7,8are received. The audio signals may be received from a VR capture device6or from microphones or other audio capture devices associated directly with the audio sources7,8. The audio sources7,8are for example singers or musicians and their associated microphones and instruments.

In step1010, the CRS15defines spatial audio fields70,80for each audio source7,8. The spatial extent and position of the spatial audio fields70,80are determined by at least the position of the respective audio sources7,8, by the mixing operator's (i.e. the user's) perspective to the audio source in virtual reality and by the spatial extent of the audio source7,8. The positions of the audio sources7,8are determined in any suitable way, for example using HAIP tags.

In step1020the CRS15determines whether one spatial audio field at least partially overlaps another spatial audio field from the mixing operator's perspective. If there is no overlap, then the process of handling audio masking ends, and the process of rendering the virtual reality environment continues.

If one of the spatial audio fields overlaps another spatial audio field, in an optional step1030a cue is generated to make the mixing operator aware of the overlapping region58. This may be advantageous when the mixing operator is unable to see one of the spatial audio fields, or is not aware of their full extent. The cue may be an audio or visual cue, or both. For example, the CRS15may cause the overlapping region58to flash (alternate in brightness and/or colour). Furthermore, the flashing overlapping region58may be made set such that the user interact with the covered part of the spatial audio field, as opposed to the foreground spatial audio field.

In step1040, the CRS15receives a touch interaction (such as a user gesture) through the touch sensitive display24. The touch interaction may be any one of those described previously with reference toFIGS. 6ato10. Alternatively, the touch interaction may be another form of user gesture, such as a manipulation of a mouse or a voice command received through a microphone.

In step1050, one of the overlapping or overlapped spatial audio regions70,80is controlled according to the touch interaction. This may include one of cropping, deleting, moving and reordering the controlled spatial audio region70,80.

It will be appreciated that the above described embodiments are purely illustrative and are not limiting. Other variations and modifications will be apparent to persons skilled in the art upon reading the present application and various variations and modifications are intended to be within the scope of the appended claims.

Moreover, the disclosure of the present application should be understood to include any novel features or any novel combination of features either explicitly or implicitly disclosed herein or any generalization thereof and during the prosecution of the present application or of any application derived therefrom, new claims may be formulated to cover any such features and/or combination of such features.