Patent Description:
The amount of multimedia content increases continuously. Users create and consume multimedia content, and it has a big role in modern society.

Publication <CIT> discloses a method for automatically producing a video and audio mix at a first portable electronic device. The method receives a request to capture video and audio, performs a network discovery process to find a second portable electronic device, and sends a message to the second device indicating when to start recording audio for a double system recording session. The method initiates the recording session, such that both devices record concurrently. In response to the first device stopping the recording of audio and sound, signaling the second device to stop recording for the identified recording session. In response to the first device receiving a first audio track from the second device that contains an audio signal recorded during the recording session, automatically generating a mix of video and audio, such that one of the audio signals from the first and second tracks is ducked relative to the other.

Publication <CIT> discloses a method, apparatus and computer program product are provided in order to facilitate spatial audio capture by utilizing multiple audio recording devices. In the context of a method, spatial characteristics of sound recorded by a first device are determined. The method also mapping, with a processor, sound recorded by a second device to a location and orientation of the first device. As such, the spatial characteristics of the sound recorded by the second device are modified to match the spatial characteristics of the sound recorded by the first device.

Publication <CIT> discloses systems, methods, and apparatus, including computer programs encoded on computer storage media, for network communication when recording audio and video (A/V) of a subject. In one aspect, a device may determine one or more internet of things (IoT) devices that are displayed within a field of view (FoV) of a camera of the device, and that are capable of remote audio recording and audio streaming. The device may select a first IoT device having a first microphone. The device may receive an audio stream from the first IoT device after selecting the first IoT device, the audio stream including the audio of the subject remotely recorded by the first IoT device for the device. When the FoV of the camera changes, the device may dynamically switch from receiving the audio stream from a first IoT device to receiving the audio stream from a second IoT device.

The examples and features, described in this specification that do not fall under the scope of the independent claims are to be interpreted as examples useful for understanding various embodiments of the invention.

Some example embodiments will now be described with reference to the accompanying drawings:.

Channel-based audio may, for example, comprise creating a soundtrack by recording a separate audio track (channel) for each loudspeaker or panning and mixing selected audio tracks between at least two loudspeaker channels. Common loudspeaker arrangements for channel-based surround sound systems are <NUM> and <NUM>, which utilize five and seven surround channels, respectively, and one low-frequency channel. A drawback of channel-based audio is that each soundtrack is created for a specific loudspeaker configuration such as <NUM> (stereo), <NUM> and <NUM>.

Object-based audio addresses this drawback by representing an audio field as a plurality of separate audio objects, each audio object comprising one or more audio signals and associated metadata. An audio object may be associated with metadata that defines a location or trajectory of that object in the audio field. Object-based audio rendering comprises rendering audio objects into loudspeaker signals to reproduce the audio field. As well as specifying the location and/or movement of an object, the metadata may also define the type of object, for example, acoustic characteristics of an object, and/or the class of renderer that is to be used to render the object. For example, an object may be identified as being a diffuse object or a point source object. Object-based renderers may use the positional metadata with a rendering algorithm specific to the particular object type to direct sound objects based on knowledge of loudspeaker positions of a loudspeaker configuration.

Scene-based audio combines the advantages of object-based and channel-based audio and it is suitable for enabling truly immersive VR audio experience. Scene-based audio comprises encoding and representing three-dimensional (3D) sound fields for a fixed point in space. Scene-based audio may comprise, for example, ambisonics and parametric immersive audio. Ambisonics comprises a full-sphere surround sound format that in addition to a horizontal plane comprises sound sources above and below a listener. Ambisonics may comprise, for example, first-order ambisonics (FOA) comprising four channels or higher-order ambisonics (HOA) comprising more than four channels such as <NUM>, <NUM>, <NUM>, <NUM>, or <NUM> channels. Parametric immersive audio may comprise, for example, metadata-assisted spatial audio (MASA).

Spatial audio may comprise a full sphere surround-sound to mimic the way people perceive audio in real life. Spatial audio may comprise audio that appears from a user's position to be assigned to a certain direction and/or distance. Therefore, the perceived audio may change with the movement of the user or with the user turning. Spatial audio may comprise audio created by sound sources, ambient audio or a combination thereof. Ambient audio may comprise audio that might not be identifiable in terms of a sound source such as traffic humming, wind or waves, for example. The full sphere surround-sound may comprise a spatial audio field and the position of the user or the position of the capturing device may be considered as a reference point in the spatial audio field. According to an example embodiment, a reference point comprises the centre of the audio field.

A device comprising a plurality of microphones may be used for capturing spatial audio information. For example, a user may capture spatial audio or video information comprising spatial audio when watching a performance of a choir. However, a position of the user capturing the spatial audio information might not be optimal in terms of the position being far away from the choir. If the distance between the capturing device and the sound source is long, the signal-to-noise ratio (SNR) is more deteriorated than a shorter distance between the capturing device and the sound source. Another problem is that it might not be possible to isolate, for example, the performance of a particular person in the choir from the overall capture. Isolating a particular sound source from a plurality of sound sources may be very challenging, especially if there are a plurality of spatially overlapping sound sources.

<FIG> is a block diagram depicting an apparatus <NUM> operating in accordance with an example embodiment of the invention. The apparatus <NUM> may be, for example, an electronic device such as a chip or a chipset. The apparatus <NUM> comprises one or more control circuitry, such as at least one processor <NUM> and at least one memory <NUM>, including one or more algorithms such as computer program code <NUM> wherein the at least one memory <NUM> and the computer program code are <NUM> configured, with the at least one processor <NUM> to cause the apparatus <NUM> to carry out any of example functionalities described below.

In the example of <FIG>, the processor <NUM> is a control unit operatively connected to read from and write to the memory <NUM>. The processor <NUM> may also be configured to receive control signals received via an input interface and/or the processor <NUM> may be configured to output control signals via an output interface. In an example embodiment the processor <NUM> may be configured to convert the received control signals into appropriate commands for controlling functionalities of the apparatus <NUM>.

The at least one memory <NUM> stores computer program code <NUM> which when loaded into the processor <NUM> control the operation of the apparatus <NUM> as explained below. In other examples, the apparatus <NUM> may comprise more than one memory <NUM> or different kinds of storage devices.

Computer program code <NUM> for enabling implementations of example embodiments of the invention or a part of such computer program code may be loaded onto the apparatus <NUM> by the manufacturer of the apparatus <NUM>, by a user of the apparatus <NUM>, or by the apparatus <NUM> itself based on a download program, or the code can be pushed to the apparatus <NUM> by an external device. The computer program code <NUM> may arrive at the apparatus <NUM> via an electromagnetic carrier signal or be copied from a physical entity such as a computer program product, a memory device or a record medium such as a Compact Disc (CD), a Compact Disc Read-Only Memory (CD-ROM), a Digital Versatile Disk (DVD) or a Blu-ray disk.

<FIG> is a block diagram depicting an apparatus <NUM> in accordance with an example embodiment of the invention. The apparatus <NUM> may be an electronic device such as a hand-portable device, a mobile phone or a Personal Digital Assistant (PDA), a Personal Computer (PC), a laptop, a desktop, a tablet computer, a wireless terminal, a communication terminal, a game console, a music player, an electronic book reader (e-book reader), a positioning device, a digital camera, a household appliance, a CD-, DVD or Blu-ray player, or a media player. In the examples below it is assumed that the apparatus <NUM> is a mobile computing device or a part of it.

In the example embodiment of <FIG>, the apparatus <NUM> is illustrated as comprising the apparatus <NUM>, a plurality of microphones <NUM>, one or more loudspeakers <NUM> and a user interface <NUM> for interacting with the apparatus <NUM> (e.g. a mobile computing device). The apparatus <NUM> may also comprise a display configured to act as a user interface <NUM>. For example, the display may be a touch screen display. In an example embodiment, the display and/or the user interface <NUM> may be external to the apparatus <NUM>, but in communication with it.

Additionally or alternatively, the user interface <NUM> may also comprise a manually operable control such as a button, a key, a touch pad, a joystick, a stylus, a pen, a roller, a rocker, a keypad, a keyboard or any suitable input mechanism for inputting and/or accessing information. Further examples include a camera, a speech recognition system, eye movement recognition system, acceleration-, tilt- and/or movement-based input systems. Therefore, the apparatus <NUM> may also comprise different kinds of sensors such as one or more gyro sensors, accelerometers, magnetometers, position sensors and/or tilt sensors.

According to an example embodiment, the apparatus <NUM> is configured to establish radio communication with another device using, for example, a Bluetooth, WiFi, radio frequency identification (RFID), or a near field communication (NFC) connection. For example, the apparatus <NUM> may be configured to establish radio communication with a wireless headphone, augmented/virtual reality device or the like.

The apparatus <NUM> is operatively connected to an audio device <NUM>. The apparatus <NUM> is wirelessly connected to the audio device <NUM>. For example, the apparatus <NUM> may be connected to the audio device <NUM> over a Bluetooth connection or the like. The audio device <NUM> comprises at least one microphone for capturing audio signals and at least one loudspeaker for playing back received audio signals. The audio device <NUM> may further be configured to filter out background noise and/or detect in-ear placement. The audio device <NUM> may comprise a single audio device <NUM> or a first audio device and a second audio device configured to function as a pair. An audio device <NUM> comprising a first audio device and a second audio device may be configured such that the first audio device and the second audio device may be used separately and/or independently of each other.

According to an example embodiment, the audio device <NUM> comprises a wireless headphone. The wireless headphone may be used independently of other wireless headphones and/or together with at least one other wireless headphone. For example, assuming the audio device <NUM> comprises a pair of wireless headphones, same or different audio information may be directed to each of the wireless headphones, or audio information may be directed to a single wireless headphone and the other wireless headphone may act as a microphone.

According to an example embodiment, the audio device <NUM> is configured to receive audio information from the apparatus <NUM>. The apparatus <NUM> may be configured to control provision of audio information to the audio device <NUM> based on characteristics of the audio device <NUM> or characteristics of the apparatus <NUM>. For example, the apparatus <NUM> may be configured to adjust one or more settings in the apparatus <NUM> and/or the audio device <NUM> when providing audio information to the audio device <NUM>. The one or more settings may relate to, for example, playback of the audio information, the number of loudspeakers available, or the like.

The audio information may comprise, for example, speech signals representative of speech of a caller or streamed audio information. According to an example embodiment, the audio device <NUM> is configured to render audio information received from the apparatus <NUM> by causing output of the received audio information via at least one loudspeaker.

According to an example embodiment, the audio device <NUM> is configured to transmit audio information to the apparatus <NUM>. The audio information may comprise, for example, speech signals representative of speech or some other type of audio information.

According to an example embodiment, the apparatus <NUM> is configured to receive spatial audio information captured by a plurality of microphones. The spatial audio information comprises at least one audio signal and at least one audio parameter for controlling the at least one audio signal. The at least one audio parameter may comprise, for example, an audio parameter corresponding to a direction and/or position of audio with respect to a reference point in a spatial audio field.

The apparatus <NUM> is configured to capture spatial audio information using the plurality of microphones <NUM>. The plurality of microphones <NUM> may be configured to capture audio signals around the capturing device. The plurality of microphones <NUM> is comprised by the apparatus <NUM>.

According to an example embodiment, the spatial audio information comprises spatial audio information captured during a voice or video call.

The apparatus <NUM> is configured to receive a captured audio object from an audio device wirelessly connected to the apparatus <NUM>. The captured audio object comprises an audio object captured by the at least one microphone comprised by the audio device <NUM>.

The audio object comprises audio data associated with metadata. Metadata associated with an audio object provides information on the audio data. Information on the audio data may comprise, for example, one or more properties of the audio data, one or more characteristics of the audio data and/or identification information relating to the audio data. For example, metadata may provide information on a position associated with the audio data in a spatial audio field, movement of the audio object in the spatial audio field and/or a function of the audio data.

According to an example embodiment, the audio object comprises a spatial audio object comprising one or more audio signals and associated metadata that defines a location and/or trajectory of the second audio object in a spatial audio field.

Without limiting the scope of the claims, an advantage of an audio object is that metadata may be associated with audio signals such that the audio signals may be reproduced by defining their position in a spatial audio field.

Receiving an audio object from the audio device may comprise decoding, using an audio codec, the received audio object. The audio codec may comprise, for example, an IVAS codec or a suitable Bluetooth audio codec.

According to an example embodiment, the audio object comprises an audio stream. An audio stream may comprise a live audio stream comprising real-time audio. An audio stream may be streamed together with other types of media streaming or audio may be streamed as a part of other types of media streaming such as video streaming. An audio stream may comprise, for example, audio from a live performance or the like.

The apparatus <NUM> is configured to determine an audio audibility value relating to the audio device <NUM>.

The audio audibility value may comprise a parameter value comprising information on a relation between the audio device <NUM> and the apparatus <NUM>. For example, the parameter value may comprise contextual information such as the position of the audio device <NUM> in relation to the position of the apparatus <NUM>. As another example, the parameter value may comprise information on characteristics of content captured by the audio device <NUM> in relation to characteristics of the content captured by the apparatus <NUM>.

According to an example embodiment, the audio audibility value relating to the audio device <NUM> depends upon a distance between the audio device <NUM> and the apparatus <NUM>. According to an example embodiment, the apparatus <NUM> is configured to update the audio audibility value in response to receiving information on a changed distance between the audio device <NUM> and the apparatus <NUM>. The apparatus <NUM> may receive information on a changed distance, for example, by detecting a change in the distance or in response to receiving information on a changed distance from a cloud server to which the apparatus <NUM> and the audio device <NUM> are operatively connected.

According to an example embodiment, the audio audibility value relating to the audio device <NUM> comprises the distance between the audio device <NUM> and the apparatus <NUM>. The distance may comprise an absolute distance or a relative distance.

The apparatus <NUM> may be configured to determine a distance between the apparatus <NUM> and the audio device <NUM> based on position information such as global positioning system (GPS) coordinates, based on a wireless connection between the apparatus <NUM> and the audio device <NUM>, based on an acoustic measurement such as a delay in detecting an event, or the like.

As another example, the apparatus <NUM> may be configured to determine a distance between the apparatus <NUM> and the audio device <NUM> based on information received from a cloud server. For example, if the location of the apparatus <NUM> and the audio device <NUM> is stored on a cloud server, the cloud server may inform the apparatus <NUM> about the respective locations or a distance between the apparatus <NUM> and the audio device <NUM>.

According to an example embodiment, the audio audibility value relating to the audio device <NUM> comprises a time of flight of sound between the audio device <NUM> and the apparatus <NUM>.

According to a non claimed example, the audio audibility value relating to the audio device <NUM> is adapted based on a sound pressure or noise level. The sound pressure comprises an overall sound pressure and the noise level comprises an overall noise level. According to another non claimed example, the audio audibility value relating to the audio device <NUM> is adapted based on a correlation measure between the spatial audio information and the audio object.

The apparatus <NUM> is configured to determine whether the audio audibility value fulfils at least one criterion. According to an example embodiment, determining whether the audio audibility value fulfils at least one criterion comprises comparing the audio audibility value with a corresponding threshold value.

According to an example embodiment, the at least one criterion comprises a threshold value dependent upon the distance between the audio device <NUM> and the apparatus <NUM>. For example, assuming the audio audibility value comprises a distance between the apparatus <NUM> and the audio device <NUM>, the threshold value comprises a threshold distance. As another example, assuming the audio audibility value comprises a time of flight of sound, the threshold value comprises a threshold time.

According to a non claimed example, the threshold value dependent upon the distance between the audio device <NUM> and the apparatus <NUM> is adapted based on a sound pressure or noise level. For example, a sound source that is relatively far away in a quiet environment may remain audible in a spatial audio capture using the apparatus <NUM>, whereas the sound source in a noisier environment needs to be closer to the apparatus <NUM> to be audible.

The apparatus <NUM> is configured to activate, in response to determining that the audio audibility value fulfils the at least one criterion, inclusion of the audio object captured by the audio device <NUM> in the spatial audio information captured by the plurality of microphones.

Activating inclusion of the audio object captured by the audio device <NUM> in the spatial audio information captured by the plurality of microphones may comprise activating a microphone associated with the audio device <NUM>, activating reception of audio signals from the audio device <NUM>, deactivating a loudspeaker associated with the audio device <NUM>, or the like.

Activating inclusion of the audio object in the spatial audio information may comprise controlling an operation of the audio device <NUM>. According to an example embodiment, the apparatus <NUM> is configured to switch the audio device <NUM> from a first mode to a second mode. The first mode may comprise, for example, a loudspeaker mode and the second mode may comprise, for example, a microphone mode. A loudspeaker mode comprises using the audio device <NUM> as a loudspeaker and a microphone mode comprises using the audio device <NUM> as a microphone.

According to an example embodiment, switching the audio device <NUM> from a first mode to a second mode comprises switching an audio output port of the audio device <NUM> into an audio input port of the audio device <NUM>.

According to an example embodiment, the apparatus <NUM> is configured to provide modified spatial audio information in response to activating inclusion of the audio object in the spatial audio information. The modified spatial audio information may comprise a combined representation of an audio scene comprising the spatial audio information and the audio object, or a representation of an audio scene in which the spatial audio information and the audio object are separate components. For example, modified spatial information may comprise the spatial audio information into which the audio object is downmixed. As another example, the modified spatial audio information may comprise the spatial audio information and the audio object as separate components.

Inclusion of the audio object in the spatial audio information may comprise controlling an audio encoder input by the apparatus <NUM>. For example, inclusion of the audio object in the spatial audio information may comprise including the audio object in an audio codec input format such that the same audio encoder is configured to encode the two audio signals jointly or packetize and deliver them together.

According to an example embodiment, the apparatus <NUM> is configured to include the audio object in an audio encoder input. According to another example embodiment, the apparatus <NUM> is configured to activate use of an audio object in an audio encoder input. According to a further example embodiment, the apparatus <NUM> is configured to renegotiate or reinitialize an audio encoder input such that the audio object is included in the encoder input. For example, if the audio encoder input was previously negotiated as first-order ambisonics (FOA), the audio encoder input may be renegotiated as FOA and the audio object. According to a yet further example embodiment, the apparatus <NUM> is configured to replace previous spatial audio information with modified spatial audio information.

Inclusion of the audio object in the spatial audio information may be performed based on metadata associated with the audio object.

Inclusion of the audio object in the spatial audio information may be activated for a period of time. In other words, the inclusion may also be terminated. According to an example embodiment, the apparatus <NUM> is configured to deactivate inclusion of the audio object captured by the audio device in the spatial audio information captured by the plurality of microphones.

According to an example embodiment, the apparatus <NUM> is configured to deactivate inclusion of the audio object captured by the audio device in the spatial audio information in response to determining the audio audibility value fulfils at least one criterion. The at least one criterion for deactivating the inclusion of the audio object may be different from the at least one criterion for activating the inclusion of the audio object.

Without limiting the scope of the claims, an advantage of different threshold values for activating and deactivating the inclusion of the audio object in the spatial audio information is that suitable hysteresis may be provided in order to prevent frequently activating and deactivating the inclusion of the audio object in the spatial audio information.

According to an example embodiment, deactivating inclusion of the audio object captured by the audio device <NUM> in the spatial audio information may comprise deactivating a microphone associated with the audio device <NUM>, deactivating reception of audio signals from the audio device <NUM>, activating a loudspeaker associated with the audio device <NUM>, instructing a microphone associated with the audio device to act as a loudspeaker or a combination thereof.

Deactivating inclusion of the audio object in the spatial audio information may comprise controlling an operation of the audio device <NUM>. According to an example embodiment, the apparatus <NUM> is configured to switch the audio device <NUM> from a second mode to a first mode. The first mode may comprise, for example, a loudspeaker mode and the second mode may comprise, for example, a microphone mode. A loudspeaker mode comprises using the audio device <NUM> as a loudspeaker and a microphone mode comprises using the audio device <NUM> as a microphone.

As mentioned above, the apparatus <NUM> may comprise a user interface for enabling a user to control and/or monitor the received spatial audio information and/or the received audio object. For example, the user interface may enable controlling and/or monitoring volume, locations of audio objects in a spatial audio field, balance or the like.

According to an example embodiment, the apparatus <NUM> is configured to provide a user interface based on available spatial audio objects. Therefore, the apparatus <NUM> may be configured to dynamically adapt the user interface.

According to an example embodiment, the apparatus <NUM> is configured to provide a control element for controlling the captured spatial audio information and, in response to determining that the audio audibility value fulfils the at least one criterion, adapt the user interface. Adapting the user interface may comprise, for example, modifying the contents of the user interface by adding, removing and/or modifying one or more user interface elements. Modifying the one or more user interface elements may comprise, for example, modifying the appearance and/or the operation of the one or more user interface elements. For example, the user interface may comprise a volume control for the captured spatial audio information and, in response to determining that the audio audibility value fulfils the at least one criterion, the user interface may be adapted to further comprise a volume control for the audio object.

According to an example embodiment, the apparatus <NUM> comprises means for performing the features of the claimed invention, wherein the means for performing comprises at least one processor <NUM>, at least one memory <NUM> including computer program code <NUM>, the at least one memory <NUM> and the computer program code <NUM> configured to, with the at least one processor <NUM>, cause the performance of the apparatus <NUM>. The means for performing the features of the claimed invention may comprise means for receiving spatial audio information captured by a plurality of microphones, means for receiving a captured audio object from an audio device wirelessly connected to the apparatus, means for determining an audio audibility value relating to the audio device, means for determining whether the audio audibility value fulfils at least one criterion, and means for activating, in response to determining that the audio audibility value fulfils the at least one criterion, inclusion of the audio object captured by the audio device in the spatial audio information captured by the plurality of microphones.

The apparatus <NUM> may further comprise means for deactivating inclusion of the audio object captured by the plurality of microphones. The apparatus <NUM> may further comprise means for switching the audio device <NUM> from a first mode to a second mode. The apparatus <NUM> may further comprise means for providing a control element for controlling the captured spatial audio information and means for, in response to determining that the audio audibility value fulfils the at least one criterion, adapting the user interface.

<FIG> illustrate an example system according to an example embodiment. In the examples of <FIG>, the apparatus <NUM> comprises an audio codec supporting user generated live content streaming.

In the example of <FIG>, a first user is in a voice or video call with a second user (not shown). For example, the first user <NUM> may use an apparatus <NUM> for capturing spatial audio information and receive audio from a second user using an audio device <NUM> such as a wireless headphone. The audio device <NUM> is wirelessly connected to the apparatus <NUM> using, for example, a Bluetooth connection. The audio device <NUM> comprises at least one loudspeaker and at least one microphone. In the example of <FIG>, audio received from the second user is illustrated with arrow <NUM>. The first user <NUM> captures spatial audio information for the second user. Captured spatial audio information is illustrated with arrow <NUM>. In the example of <FIG>, a third user <NUM> is a sound source of interest. For example, the third user <NUM> may be a person singing in a choir.

In the example of <FIG>, the first user <NUM> uses a single wireless headphone. In such as case, the headphone may be configured to act as a microphone or a loudspeaker by default.

In the example of <FIG>, the first user <NUM> has given the audio device <NUM> to the third user <NUM>. Assuming the third user <NUM> is a person singing in a choir, when the third user <NUM> moves to a venue, the distance between the audio device <NUM> and the apparatus <NUM> increases.

In the example of <FIG>, the distance <NUM> between the apparatus <NUM> and the audio device <NUM> increases. The apparatus <NUM> is configured to determine whether the distance <NUM> between the apparatus <NUM> and the audio device <NUM> is above a threshold value. The apparatus <NUM> is further configured to activate, in response to determining that the distance <NUM> between the apparatus <NUM> and the audio device <NUM> is above a threshold value, inclusion of an audio object captured by the audio device <NUM> in the spatial audio information captured by the apparatus <NUM>. If the audio device <NUM> acts as a microphone by default, activating inclusion of an audio object may comprise activating reception of audio signals from the audio device <NUM>. If the audio device <NUM> acts as a loudspeaker by default, activating inclusion of an audio object may comprise switching the audio device <NUM> from a loudspeaker mode to a microphone mode.

<FIG> illustrate another example system according to an example embodiment. In the examples of <FIG>, the apparatus <NUM> comprises an audio codec supporting user generated live content streaming.

In the example of <FIG>, a first user is in a voice or video call with a second user (not shown). For example, the first user <NUM> may use an apparatus <NUM> for capturing spatial audio information and receive audio from a second user using a pair of audio devices <NUM> such as a wireless headphone. The pair of audio devices <NUM> is wirelessly connected to the apparatus <NUM> using, for example, a Bluetooth connection.

The audio device <NUM> comprises at least one loudspeaker and at least one microphone. In the example of <FIG>, audio received from the second user is illustrated with arrow <NUM>. The first user <NUM> captures spatial audio information for the second user. Captured spatial audio information is illustrated with arrow <NUM>. In the example of <FIG>, a third user <NUM> is a sound source of interest. For example, the third user <NUM> may be a person singing in a choir.

In the example of <FIG>, the first user <NUM> uses a pair of wireless headphones. The pair of wireless headphones may comprise a first wireless headphone and a second wireless headphone. In such as case, one headphone may be configured to act as a microphone and one headphone may be configured to act as a loudspeaker.

In the example of <FIG>, the first user <NUM> has given one of the audio devices <NUM> to the third user <NUM>. In the following, it is assumed that the first user <NUM> uses the first wireless headphone and the third user <NUM> uses the second wireless headphone. Assuming the third user <NUM> is a person singing in a choir, when the third user <NUM> moves to a venue, the distance between the audio device <NUM> of the third user <NUM> and the apparatus <NUM> increases.

In the example of <FIG>, the distance <NUM> between the apparatus <NUM> and the audio device <NUM> (e.g. the second wireless headphone) increases. The apparatus <NUM> is configured to determine whether the distance <NUM> between the apparatus <NUM> and the audio device <NUM> of the third user <NUM> is above a threshold value. The apparatus <NUM> is further configured to activate, in response to determining that the distance <NUM> between the apparatus <NUM> and the audio device <NUM> of the third user <NUM> is above a threshold value, inclusion of an audio object captured by the audio device <NUM> in the spatial audio information captured by the apparatus <NUM>. Assuming the audio device <NUM> of the third user <NUM> is configured to act as a microphone, activating inclusion of an audio object may comprise activating reception of audio signals from the audio device <NUM> of the third user. On the other hand, assuming the audio device <NUM> of the third user <NUM> is configured to act as a loudspeaker, activating inclusion of an audio object may comprise sending an instruction to change the audio device <NUM> of the third user <NUM> from a first mode to a second mode. For example, activating inclusion of an audio object may comprise sending an instruction to change the audio device <NUM> of the third user <NUM> from a loudspeaker mode to a microphone mode. As another example, activating inclusion of an audio object may comprise sending an instruction to stop using the loudspeaker which may cause activating a microphone mode.

<FIG> illustrates example user interfaces according to an example embodiment. More specifically, example user interfaces in <FIG> illustrate user interfaces for controlling captured spatial audio information and example user interface in <FIG> illustrate dynamically adapting the user interfaces illustrated in <FIG> in response to determining that the audio audibility relating to an audio device <NUM> value fulfils at least one criterion for activating inclusion of an audio object in the spatial information.

In the example of <FIG>, the audio device <NUM> comprises a pair of wireless headphones. The pair of wireless headphones may comprise a first wireless headphone and a second wireless headphone. Similarly to the examples of <FIG>, it is assumed that the first user <NUM> uses the first wireless headphone and the third user <NUM> uses the second wireless headphone.

The apparatus <NUM> is configured to provide the user interfaces <NUM> and <NUM>. The apparatus <NUM> is further configured to provide one or more control elements presented on the user interface <NUM>, <NUM> and a representation of a spatial audio field <NUM>. In the examples of <FIG>, it is assumed that a reference point of the spatial audio field comprises the centre of the spatial audio field <NUM> and that the centre of the spatial audio field corresponds to the position of the apparatus <NUM>.

In the example of <FIG>, the first user <NUM> utilizes a spatial audio input. The user interface <NUM> comprises a control element <NUM> for controlling the volume of the spatial audio information. The user interface <NUM> is further configured to present a representation of a spatial audio field <NUM>. The representation of the spatial audio field <NUM> comprises indications of different directions such as front, right, back and left with respect to the reference point.

<FIG> illustrates an example where the first user <NUM> has given one wireless headphone, such as the second wireless headphone, to the third user <NUM> and the audio audibility value relating to an audio device <NUM> value fulfils at least one criterion for activating inclusion of an audio object in the spatial audio information.

In the example of <FIG>, the at least one criterion comprises a distance <NUM> between the wireless headphone <NUM> of the third user <NUM> (the second wireless headphone) and the wireless headphone <NUM> of the first user <NUM> (the first wireless headphone) or the apparatus <NUM>. When the distance <NUM> is above a threshold value, inclusion of an audio object in the spatial audio information is activated by the apparatus <NUM>. The apparatus <NUM> is configured to adapt the user interface <NUM> in order to enable controlling the audio object.

In the example of <FIG>, the user interface <NUM> comprises a control element <NUM> for controlling the volume of the received spatial audio information and a control element <NUM> for controlling the volume of the added audio object. The added audio object is indicated as a far source on the control element <NUM>. The location of the audio object <NUM> is indicated as being approximately in a front-right direction in the spatial audio field <NUM>.

Referring back to the example of <FIG>, the user interface <NUM> comprises a control element <NUM> for controlling the volume of the received spatial audio information and a control element <NUM> for controlling the volume of voice channel. For example, the first user <NUM> may capture spatial audio information and at the same time listen to audio from a second user or monitor the spatial audio capture. In other words, the first user <NUM> utilizes two audio inputs. The representation of the spatial audio field <NUM> comprises indications of different directions such as front, right, back and left with respect to the reference point and an indication that the position of the voice channel <NUM> is approximately towards left.

In the example of <FIG>, the user interface <NUM> comprises a control element <NUM> for controlling the volume of the received spatial audio information, a control element <NUM> for controlling the volume of voice channel and a control element <NUM> for controlling the volume of the added audio object. The added audio object is indicated as a far source on the user interface <NUM>. The location of the audio object <NUM> is indicated as being approximately in a front-right direction and the position of the voice channel <NUM> is indicated as being approximately towards left in the spatial audio field.

<FIG> illustrates an example method <NUM> incorporating aspects of the previously disclosed embodiments. More specifically the example method <NUM> illustrates activating inclusion of an audio object in spatial audio information. The method may be performed by the apparatus <NUM> such as a mobile computing device.

The method starts with receiving <NUM> spatial audio information captured by a plurality of microphones. The method continues with receiving <NUM> a captured audio object from an audio device <NUM> wirelessly connected to the apparatus <NUM>.

The method further continues with determining <NUM> an audio audibility value relating to the audio device <NUM>.

The method further continues with determining <NUM> whether the audio audibility value fulfils at least one criterion. If the audio audibility value does not fulfil the at least one criterion, the method returns to determining <NUM> whether the audio audibility value fulfils at least one criterion. If the audio audibility value fulfils the at least one criterion, the method continues with activating <NUM> inclusion of the audio object captured by the audio device <NUM> in the spatial audio information captured by the plurality of microphones.

<FIG> illustrate examples of audio audibility values and audio audibility threshold values. The apparatus <NUM> is configured to determine an audio audibility value based on a relationship between the apparatus <NUM> and the audio device <NUM>.

In the example of <FIG>, the audio audibility value is determined based on the distance between the apparatus <NUM> and the audio device <NUM>. According to an example embodiment, the distance between the apparatus <NUM> and the audio device <NUM> is used as the audio audibility value. In such a case, the distance may be compared to one or more threshold distance values.

<FIG> illustrates two example embodiments of audio audibility values and audio audibility threshold values. In the example of <FIG>, the audio audibility value is determined based on the distance between the apparatus <NUM> and the audio device <NUM> that is adapted based on a sound pressure level. Determining an audio audibility value based on a sound pressure level may comprise maintaining the sound pressure level as a fixed value and adapting the distance in dependence of the sound pressure value or determining an adaptive audio audibility threshold value that is dependent upon the sound pressure level.

Without limiting the scope of the claims, an advantage of activating inclusion of an audio object to spatial audio information is that it is possible to combine and/or isolate a sound source of interest in spatial audio information. Another advantage is that a user capturing spatial audio information can pick-up a sound source of interest even though a venue is crowded or the like. A further advantage is that a sound source that might not be audible due to distance or other factors can be included in the spatial audio information. A yet further advantage is that a sound source of interest may be included in the spatial audio information when necessary. A yet further advantage is that a regular accessory may be utilized without a need to invest in expensive and complex devices.

Without in any way limiting the scope, interpretation, or application of the claims appearing below, a technical effect of one or more of the example embodiments disclosed herein is that high quality spatial audio capture may be provided without complex arrangements. Another technical effect is that inclusion of an audio object may be activated automatically. A further technical effect is that computational resources and bandwidth may be saved when unnecessary inclusion of the sound source of interest in the spatial audio information is avoided.

As used in this application, the term "circuitry" may refer to one or more or all of the following: (a) hardware-only circuit implementations (such as implementations in only analog and/or digital circuitry) and (b) combinations of hardware circuits and software, such as (as applicable): (i) a combination of analog and/or digital hardware circuit(s) with software/firmware and (ii) any portions of hardware processor(s) with software (including digital signal processor(s)), software, and memory(ies) that work together to cause an apparatus, such as a mobile phone or server, to perform various functions) and (c) hardware circuit(s) and or processor(s), such as a microprocessor(s) or a portion of a microprocessor(s), that requires software (e.g., firmware) for operation, but the software may not be present when it is not needed for operation.

Claim 1:
An apparatus (<NUM>) comprising:
a plurality of first microphones (<NUM>) for capturing spatial audio information;
means for receiving spatial audio information captured by the plurality of microphones;
means for connecting wirelessly to an audio device (<NUM>) comprising at least one loudspeaker and at least one second microphone;
means for receiving an audio object captured by the audio device, wherein the audio object comprises one or more audio signals captured by the at least one second microphone and associated metadata, from the audio device (<NUM>) when the apparatus (<NUM>) is wirelessly connected to the audio device (<NUM>);
means for determining an audio audibility value relating to the audio device (<NUM>), wherein the audio audibility value is determined based on a distance between the apparatus (<NUM>) and the audio device (<NUM>) and the distance is used as the audio audibility value, or the audio audibility value is a time of flight of sound between the audio device (<NUM>) and the apparatus (<NUM>);
means for determining whether the audio audibility value fulfils at least one criterion by comparing the distance to a threshold distance value or by comparing the time of flight of sound to a threshold time; and
means for activating, in response to determining that the audio audibility value fulfils the at least one criterion by determining that the distance is above the threshold distance value or that the time of flight is above the threshold time, inclusion of the audio object captured by the audio device (<NUM>) in the spatial audio information captured by the plurality of the first microphones.