Patent Description:
An Augmented Reality (AR) system may provide a virtual object for user exploration during a consumption phase. Virtual objects may be any computer-generated version of an object, for example a captured real world object, which a user can view or interact with through a user device. The virtual object can be placed in a real world environment to provide an augmented reality experience to a user, in which the user hears sounds corresponding to both real and virtual objects and sees both real and virtual objects.

The user device may have a pair of displays and/or one or more (optionally a pair of) audio output transducers, e.g. earphones, headphones or loudspeakers. An AR headset is an example of a user device in this context.

<CIT> describes technology for providing realistic occlusion between a virtual object displayed by a head mounted, augmented reality display system and a real object visible to the user's eyes through the display. A spatial occlusion in a user field of view of the display is typically a three dimensional occlusion determined based on a three dimensional space mapping of real and virtual objects. An occlusion interface between a real object and a virtual object can be modeled at a level of detail determined based on criteria such as distance within the field of view, display size or position with respect to a point of gaze. Technology is also described for providing three dimensional audio occlusion based on an occlusion between a real object and a virtual object in the user environment.

<CIT> describes a method for rendering an audio signal in a virtual reality rendering environment. The method comprises rendering an origin audio signal of an audio source from an origin source position on an origin sphere around an origin listening position of a listener. Furthermore, the method comprises determining that the listener moves from the origin listening position to a destination listening position.

In addition, the method comprises determining a destination source position of the audio source on a destination sphere around the destination listening position based on the origin source position, and determining a destination audio signal of the audio source based on the origin audio signal. Furthermore, the method comprises rendering the destination audio signal of the audio source from the destination source position on the destination sphere around the destination listening position.

<CIT> describes an audio processing method and terminal equipment. The method comprises the steps of determining reverberation parameters of a real scene which is involved in augmented reality (AR) operation and/or an AR scene after operation; and according to the reverberation parameters of the real scene and/or the AR scene after operation, determining an AR audio corresponding to the AR scene after operation. The reverberation parameters of the real scene and the AR scene after operation can reflect the influence of the AR operation on the reverberation effect of the scenes, and the AR audio corresponding to the AR scene after operation is determined according to the reverberation parameters of the real scene and/or the AR scene after operation so that user can hear the sound matched with the AR scene.

According to a first aspect, this specification describes an apparatus according to independent claim <NUM>.

The apparatus may further comprise means for determining an estimated position of the user at a future time point. Optionally, the means for switching between the first mode and the second mode based on a location of the user relative to the occluded region comprises means for switching between the first mode and the second mode based on the current location of the user and the estimated position of the user.

No processing is performed on audio signals received from the one or more real audio sources in the first mode, such that the audio signals are heard by the user without intermediate processing.

The apparatus may further comprise means for processing audio signals received from the one or more real audio sources to compensate for an acoustic effect of the virtual object on the audio signals. Compensating for the acoustic effect of the virtual object may provide a more realistic and immersive augmented reality experience to a user.

The outputting means may be configured to output audio data associated with the virtual object in both the first and second modes. In this way, the user may hear virtual audio signals from the virtual object without interruption, and only real audio signals are processed or modified to compensate for the effect of the virtual object on the acoustics in the occluded region.

The means for switching may be configured to switch the outputting means between the first mode and the second mode based on a size of the occluded region.

According to a second aspect, this specification describes a method according to independent claim <NUM>.

Switching between the first mode and the second mode based on a location of the user relative to the occluded region may comprise: determining a current location of the user at a current time point; and operating the outputting means in the first mode if the current location of the user is outside of the occluded region; and operating the outputting means in the second mode if the current location of the user is within the occluded region.

The method may further comprise determining an estimated position of the user at a future time point. Optionally, switching between the first mode and the second mode based on a location of the user relative to the occluded region comprises switching between the first mode and the second mode based on the current location of the user and the estimated position of the user.

Optionally, switching the outputting means between the first mode and the second mode may comprise switching the outputting means based on a size of the occluded region.

The method may further comprise processing audio signals received from the one or more real audio sources to compensate for an acoustic effect of the virtual object on the audio signals. The outputting means may be configured to output audio data associated with the virtual object in both the first and second modes.

According to a third aspect, this specification describes a computer program comprising instructions for causing an apparatus to perform features of the second aspect.

According to a fourth aspect, this specification describes a computer-readable medium (such as a non-transitory computer-readable medium) comprising program instructions stored thereon for performing features of the second aspect.

According to a fifth aspect, this specification describes an apparatus comprising: at least one processor; and at least one memory including computer program code which, when executed by the at least one processor, causes the apparatus to perform the method of the second aspect.

Also disclosed herein is an apparatus comprising means for: determining, relative to one or more real audio sources in an environment, a relative position of a virtual object in the environment; determining, based on the relative position, an occluded region of the environment in which audio signals from the one or more real audio sources would be occluded by the virtual object; processing audio signals received, at the occluded region and from the one or more real audio sources, to generate audio data, the processing means configured to generate the audio data by at least partially removing those of the received audio signals which would be occluded by the virtual object; and outputting, to a user in the environment, the generated audio data.

By removing those of the received audio signals which would be occluded by the virtual object, the apparatus may compensate for an acoustic effect of the virtual object on the audio signals. Compensating for the acoustic effect of the virtual object may provide a more realistic and immersive augmented reality experience to a user.

The apparatus may further comprise means for receiving audio signals from the one or more real audio sources. Optionally, the receiving means may comprise one or more microphones and/or the outputting means comprises one or more audio output transducers.

The receiving means and the outputting means may be provided by an augmented reality device wearable by the user, the augmented reality device providing the virtual object. By having the receiving means in the same location as the user, the processing of audio signals may be computationally simpler than if the receiving means are located remote from the user. Optionally, the apparatus is an augmented reality device.

The means for determining, relative to the one or more real audio sources in an environment, the relative position of the virtual object in the environment may comprise: determining, based on the received audio signals, a position of the one or more real audio sources in the environment; and determining one or more of a size, a location and an orientation of the virtual object from associated augmented reality content.

The apparatus may further comprise means for providing the processed audio signals to the outputting means. The outputting means may be configured to output both audio data associated with the virtual object and the generated audio data. In this way, the user may hear virtual audio signals from the virtual object without interruption, and only real audio signals are processed or modified to compensate for the effect of the virtual object on the acoustics in the occluded region.

Also disclosed herein is a method comprising: determining, relative to one or more real audio sources in an environment, a relative position of a virtual object in the environment; determining, based on the relative position, an occluded region of the environment in which audio signals from the one or more real audio sources would be occluded by the virtual object; processing audio signals received, at the occluded region and from the one or more real audio sources, to generate audio data, the processing means configured to generate the audio data by at least partially removing those of the received audio signals which would be occluded by the virtual object; and outputting, to a user in the environment, the generated audio data.

Example embodiments of the apparatus may also provide any feature of the method.

This specification also describes a computer program comprising instructions for causing an apparatus to perform at least the above method and embodiments. This specification also describes a computer-readable medium (such as a non-transitory computer-readable medium) comprising program instructions stored thereon for performing at least the above method and embodiments. This specification describes an apparatus comprising: at least one processor; and at least one memory including computer program code which, when executed by the at least one processor, causes the apparatus to perform at least the above method and embodiments.

Example embodiments will now be described, by way of non-limiting example, with reference to the accompanying drawings, in which:.

Example embodiments relate to methods, computer programs and apparatuses for processing and rendering audio data, for example processing audio data to account for the occluding effect of virtual objects on real audio sources in a real world environment. A headset mode may be adapted (or switched) between different modes to render the processed audio to a user.

Example embodiments may relate to extended reality (XR) methods and apparatuses, for example virtual reality (VR), augmented reality (AR) and/or mixed reality (MR) methods and apparatuses. Example embodiments will focus on an AR system and method, but it should be understood that embodiments are applicable to any system or method that involves processing of audio such that real world sounds can be perceived by a listening user to substantially correspond to real world sounds if a virtual object presented or displayed to the user had a real, physical, presence in the environment.

<FIG> is a schematic illustration of an AR system <NUM> which represents user-end equipment. The AR system <NUM> includes a user device in the form of an AR headset <NUM> for outputting video and audio data for a virtual object, and an AR media player <NUM> for rendering the video and audio data at the AR headset <NUM>. The AR headset <NUM> may comprise one or more (here shown as two) video screens for displaying video data and one or more (here shown as two) audio output transducers, e.g. earphones, headphones or loudspeakers, for output of audio data. In some example embodiments, a separate user control (not shown) may be associated with the AR system <NUM>, e.g. a hand-held controller.

The AR headset <NUM> may receive the video and audio data from the AR media player <NUM>. The AR media player <NUM> may be part of a separate device which is connected to the AR headset <NUM> by a wired or wireless connection. For example, the AR media player <NUM> may comprise a games console, a PC, laptop or tablet computer configured to communicate the video and audio data to the AR headset <NUM>. Alternatively, the AR media player <NUM> may form part of the AR headset <NUM>. The AR headset may be of any suitable type.

The AR system <NUM> may include means for determining a position of the user. The position of the user may include the spatial position of the user and/or an orientation of the user or part of the user's body. For example, the AR system <NUM> may be configured to determine the spatial position of the user by means of determining the spatial position of the AR headset <NUM>. Over successive time frames, a measure of movement may therefore be determined based on the different spatial positions of the AR headset <NUM>.

For example, the AR headset <NUM> may comprise motion tracking sensors which may include one or more of gyroscopes, accelerometers and structured light systems. Alternatively, or additionally, the AR headset <NUM> may comprise a positioning receiver, such as a Global Navigation Satellite System (GNSS) receiver and/or another positioning system such as a WiFi positioning receiver or a cellular positioning receiver which operate based on, for example, measurement of Angle of Arrival (AoA), Time of Arrival (ToA) and/or Received Signal Strength Indication (RSSI) information.

Spatial position and movement of the user may also be determined using one or more cameras configured to detect and track user movement, possibly in conjunction with one or more markers or sensors carried by the user or placed on the AR headset <NUM>.

The AR system <NUM> may also comprise means for determining an orientation of part of the user's body, for example orientation of the user's head. This may also be determined by determining an orientation of the AR headset <NUM> using, for example, motion tracking sensors as mentioned above. Over successive time frames, a measure of a change in orientation of the user's head may therefore also be determined, e.g. to identify an amount of rotational change. Orientation of the user's head may be used to help determine user trajectory, for example.

The orientation of the user's head may also be determined using one or more cameras configured to detect and track head orientation, possibly in conjunction with one or more markers or sensors carried by the user or placed on the AR headset <NUM>.

The AR system <NUM> may also comprise means for determining an orientation of one part of the user's body with respect to another part of the user's body. For example, the AR system <NUM> may determine the orientation of an upper body part (e.g. torso) of the user with respect to a lower body part (e.g. legs) of the user. This may enable the AR system <NUM> to identify, for example, a downwards leaning motion based on a detected change in upper body orientation with respect to lower body orientation. The orientation of the user's upper body with respect to the lower body may also be determined using one or more cameras configured to detect and track the upper and lower body parts, possibly in conjunction with one or more markers or sensors carried by the user.

Determining the spatial position of the user and their head orientation enables the AR system <NUM> to track the user, for example to determine a current visual field-of-view (FOV) which may determine which video and audio data to output to the user via the AR headset <NUM>. Determining the spatial position of the user and/or other movements, e.g. orientation changes and movements of individual body parts, also enables the AR system <NUM> to track a user in an environment, and to estimate a future trajectory.

Data which represents, or from which can be derived, a spatial position of the user, an orientation of a user's body part and/or position of a user's body part may be referred to herein as position or positional data.

<FIG> indicates respective orientations of pitch <NUM>, roll <NUM> and yaw <NUM> and also translational movement in Euclidean space along side-to-side, front-to-back and up- and-down axes <NUM>, <NUM>, and <NUM>. These represent so-called six-degrees-of freedom (6DoF) which a user may change when exploring or consuming a virtual object represented by video and audio data.

Referring to <FIG>, a content provider <NUM> may store and transmit, e.g. by streaming, video and audio data representing a particular virtual space for output to the AR headset <NUM>. Responsive to receive or download requests sent by the AR media player <NUM>, the content provider <NUM> may stream the video and audio data over a data network <NUM>, which may be any network, for example an IP network such as the Internet. The remote content provider <NUM> may or may not be at the location or system where the video and audio data is captured, created and/or processed. For illustration purposes, we may assume that the content provider <NUM> also captures, encodes and stores the video and audio data, as well as streaming it responsive to signals from the AR display system <NUM>.

<FIG> is a schematic illustration of a real world environment <NUM> comprising a user <NUM> and a real audio source <NUM> (such as a speaker). A sound or audio source may be any object, e.g. a real world object, which produces a sound. An AR system is provided in the environment, the AR system including a user device (an augmented reality device wearable by the user <NUM>, or AR user device) in the form of an AR headset for outputting video and audio data for a virtual space, and an AR media player for rendering the video and audio data at the AR headset. The AR system can be the AR system <NUM> of <FIG>, for example.

The AR user device (or AR headset) may comprise one or more video screens <NUM> for displaying video data (shown here as a head mounted display, or HMD) and one or more audio output transducers <NUM>, e.g. earphones, headphones or loudspeakers, for output of audio data. In <FIG> there two audio output transducers <NUM> are shown, but the disclosure is not limited to this. The user <NUM> is shown as wearing the AR device. In some example embodiments, the audio output transducers <NUM> are provided separately from the video screens <NUM>, and may not be part of the AR headset. For example, the audio output transducers <NUM> may receive audio data for rendering from the AR system through any suitable wired or wireless connection.

The AR headset may be configured to provide video and audio data (received as part of AR content) to a user by means of the above-mentioned video screens <NUM> and audio output transducer(s) <NUM> (outputting means, whether integral to the AR headset or a connected but standalone component). In other words, the user sees AR visual content through the video screens <NUM> and hears AR audio content through the audio output transducer(s) <NUM> (outputting means, or means for outputting audio data). The outputting means are here provided by the augmented reality device wearable by the user <NUM>.

The video data can comprise one or more virtual objects <NUM> which (from the perspective of the user <NUM>) are placed within the real world environment <NUM> of the user. In other words, the augmented reality device wearable by the user can provide the virtual object to the user. For example, the virtual objects <NUM> can be overlaid with the real world environment <NUM>. In other examples, the virtual objects displayed to the user may be in a virtual space. In the context of this specification, a virtual space may be any computer-generated version of a space, for example a captured real world space, in which a user can be immersed. In some example embodiments, the virtual space may be entirely computer-generated, i.e. not captured.

The audio data is audio data associated with the one or more virtual objects <NUM>. The audio data can therefore be considered as virtual audio, or audio which is associated with a virtual audio source (for example, the one or more virtual objects <NUM>), and may be referred to herein as virtual audio data (though it will be understand that real sounds are rendered). Other virtual audio sources may also be present in the AR content. Moreover, in some implementations the outputting means <NUM> may be configured to output other audio data to the user <NUM>. In other words, the audio data output to a user through the outputting means can comprise audio data from one or more sources (real and/or virtual).

With reference to <FIG>, audio signals <NUM> are received, by the user <NUM>, from the real audio source <NUM>. The audio signals <NUM> (represented by the solid arrows) comprise both direct audio signals 112a and reflected audio signals 112b. The outputting means <NUM> are configured to operate in a first, or open, mode. In the first mode, the audio signals <NUM> are received at the user's ear (i.e. perceived or heard by the user) with substantially no processing by the AR system or by any other means. For example, there may be no processing of the audio signals <NUM> and the outputting means <NUM> can be configured as open headphones/earbuds etc. which let the user perceive the real-world sound as is (the audio signals <NUM> heard by the user <NUM> may be slightly muffled by the presence of the outputting means <NUM> in or over the user's ear). There is no latency in the communication of the audio signals <NUM>.

In other examples not the subject of the claims, the first (open) mode comprises the outputting means operating in a transparency, pass-through or hear-through mode, in which minimal processing is performed on the audio signals <NUM> to enable to the user to hear the real world as if they were not wearing the headphones/earbuds etc. (the outputting means <NUM>). This transparency mode provides a very low latency signal path from an outside microphone (receiving means) to the audio output transducer(s) (outputting means <NUM>). In other words, in the first or open mode the outputting means are configured to allow audio signals <NUM> to be captured and replayed to the user <NUM> with little to no intermediate processing.

In the first (open) mode of claim <NUM>, the outputting means <NUM> are configured so no audio data associated with the real audio source <NUM> output by the outputting means (the outputting means are passive in respect of external audio signals <NUM>). It can be understood that, in the first mode, substantially no processing is performed on audio signals received from the one or more real audio sources.

In the first mode, the outputting means are also configured to output (in combination with the first audio data, if appropriate) second audio data to the user <NUM>. The first audio data is output to the user as if virtual audio signals <NUM> have been received from the position of the virtual object <NUM>. As with the real audio signals <NUM>, the (virtual) audio signals comprise virtual direct audio signals 114a and virtual reflected audio signals 114b, each of which are presented to the user as part of the first audio data (the virtual audio signals <NUM> are represented by the dashed arrows). In other words, audio data from a virtual audio source associated with the virtual object <NUM> is rendered to a user through the outputting means <NUM>. In particular, the virtual direct audio signal 114a is rendered binaurally from the direction of the virtual audio source (here the virtual object <NUM>). Virtual reflected audio signals 114b are rendered to the user <NUM> based on a geometric model of the user's environment or surroundings (which geometric model is included as part of the AR content, in order to appropriately place the virtual object <NUM> in the environment <NUM>).

With reference to <FIG>, in some implementations the virtual direct audio signals 114a which would be directly received by the user <NUM> may be blocked by an obstruction <NUM>, such as a wall or piece of furniture. Provided that the obstruction <NUM> is included within the geometric model of the environment <NUM>, the second or virtual audio data rendered or output to the user may include only the virtual reflected audio signals 114b from the virtual object <NUM>. The user may continue to receive/hear direct audio signals 112a from the real audio source as well as reflected audio signals (not shown). The audio data output to the user <NUM> is therefore modified to account for real obstructions <NUM> in the user environment <NUM>. In other words, the audio data is processed such that sounds in the direction of the real audio source <NUM> are modified when output to the user device.

However, such modification is not performed in circumstances where the virtual object <NUM> is the obstruction <NUM> (as is the case in <FIG>). As shown, the virtual object <NUM> displayed by the AR system described above is located in the environment <NUM> such that, if the virtual object were replaced with a physical representation or manifestation, the audio signals <NUM> from the real audio source would be blocked or occluded by the virtual object <NUM> and thus prevented from reaching the user <NUM>. In other words, the user may not be able to hear the direct audio signals 112a if the virtual object <NUM> was a real object (or the direct audio signals 112a may be muffled, or otherwise distorted or affected by the presence of the object between the user and the real audio source <NUM>). The virtual object thus acts as an obstacle or obstruction <NUM>.

It may therefore be desirable to process the real audio signals <NUM> to account for the presence of the virtual object <NUM>, thereby to provide a more immersive AR experience to the user <NUM>. Such an approach may also have utility in various applications, such as in illustrating the effect of objects on noise reduction (for example, the effect of trees on reducing road noise may be illustrated to a user in example embodiments).

With reference to <FIG>, one example approach of processing audio data to account for the occluding effect of virtual objects <NUM> on audio signals <NUM> from real audio sources <NUM> in a real world environment <NUM> is described.

In <FIG>, the AR device is shown to include a system compromising one or more microphones (optionally an array of microphones, also referred to herein as receiving means) and the one or more outputting means <NUM>. The receiving means and the outputting means are provided by the augmented reality device wearable by the user (the AR device or AR user device). The outputting means of <FIG> are configured to operate in a closed mode. In the closed mode, the outputting means (such as headphones/earbuds) at least partially block the real audio signals <NUM>. In order for the user <NUM> to hear the real audio source <NUM>, the audio signals <NUM> need to be received by the microphone(s) and then reproduced to the user through the outputting means <NUM>. In particular, an array of microphones can record the soundscape <NUM> around the user <NUM> and render or output the received audio signals <NUM> as first audio data to the user using the outputting means <NUM>. As well as passive blocking of the audio signals <NUM> due to the presence of the outputting means in/over the user's ear, it will be understood that user devices which operate in such a closed mode can also be used for active noise cancellation to actively block audio signals <NUM> from reaching the user's ears.

With reference to <FIG>, this approach of active noise cancellation can be modified to take account of the location of the virtual object <NUM> which provides the obstacle or obstruction <NUM>. In particular, when the virtual object <NUM> is placed between the user <NUM> and the real audio source <NUM>, the recorded soundscape <NUM> can be modified to remove or reduce those audio signals which would be occluded by the virtual object <NUM> (e.g. if it were a real object). For example, direct audio signals 112a would be blocked by the virtual object <NUM>, but the reflected audio signals 112b would not. This may involve modifying the gain and/or diffuseness of recorded sounds in the direction of the real audio source <NUM>. The modifying may dynamically change (e.g. increase or decrease) in response to the position of the user changing, e.g. moving towards or way from the occluding virtual object <NUM>. For example, the gain of sounds in said direction may be decreased and/or the diffuseness increased as a user moves towards the occluding virtual object <NUM> and vice versa.

The area or region (or zone) of the environment in which one or more of audio signals <NUM> may be occluded by the presence of the virtual object <NUM> can be termed an "occluded region" of the environment <NUM>. An example occluded region <NUM> is shown in <FIG>. The occluded region can be defined as a volume in the user's AR content consumption space (here environment <NUM>) where a sound source may be occluded by an AR object. The occluded region <NUM> may change if the occluding virtual object <NUM> causing the obstruction <NUM> is a moving virtual object.

With reference to <FIG>, one example approach of using a closed (or second) mode to process audio data to account for the occluding effect of virtual objects <NUM> on audio signals <NUM> from real audio source <NUM> in environment <NUM> is described.

In one example implementation, the process can be performed by an apparatus comprising means for determining, relative to the one or more real audio sources <NUM> in the environment <NUM>, a position of the virtual object <NUM> in the environment. The position of the real audio source <NUM> can be obtained (S800), and the position (e.g. location, size and/or orientation) of the virtual object <NUM> can be obtained (S810).

The positions of real-life sound sources such as real audio source <NUM> can be estimated using the microphone array (receiving means) and the recorded soundscape <NUM>. Sound source localization or direction of arrival algorithms could be used. For example, time-difference of arrival techniques on the signals obtained at the different microphones of the array may be used. In addition, data from any cameras on the AR device may be fed into a visual object detection/recognition algorithm so that possible sound producing items are identified. This vision data can be combined with the sound localization techniques to improve audio source localization.

Information about the position of the virtual object <NUM>, and the geometry of the virtual object, can be obtained as part of the AR content provided to the AR device. The information may be presented as meshes or geometric primitives, for example. Information related to meshes or geometric primitives may include size, position and/or orientation information, which information is used to position the virtual objects <NUM> correctly in the AR scene and the user's AR consumption space (here environment <NUM>). A geometric model of the environment <NUM> may also be used to position the real audio source <NUM> relative to the virtual object <NUM>.

The apparatus can further comprise means for determining, based on the relative position (i.e. based on the positions obtained at S800 and S810), an occluded region <NUM> of the environment in which audio signals <NUM> from the one or more real audio sources <NUM> would be occluded by the virtual object. The occluded region is discussed above with reference to <FIG>. In other words, based on the location of the real audio source <NUM>, and the position and geometry of the virtual object <NUM> (i.e. the relative positions of the real audio source <NUM> and the virtual object <NUM>), the occlusion zone or occluded region <NUM> is determined (S820). The determination of the occluded region can be done in real time and/or or the determination can be done for all the set of real audio source locations in a given space before a user begins consuming AR content (i.e. a set of occluded regions <NUM> can be stored for a given AR content in a given environment).

The apparatus can further comprise means for processing audio signals <NUM> received at (or in) the occluded region <NUM> from the one or more real audio sources <NUM>. The processing means can be configured to generate audio data by at least partially removing those of the received audio signals which would be occluded by the virtual object to compensate for an acoustic effect of the virtual object on the audio signals. For example, the virtual object <NUM> may block, muffle or otherwise distort the audio signals received at the user <NUM> to compensate or otherwise account for the occluding presence of the virtual object <NUM>. The apparatus can further comprise outputting means <NUM>, to the user <NUM>, the generated audio data. When outputting means are operating in a closed mode as described above (and as described with reference to <FIG>), the second audio data associated with the virtual audio source (here the virtual audio source is associated with the virtual object <NUM>) may still be rendered to the user <NUM> through the outputting means in the same manner as described above with respect to <FIG> and <FIG>, for example, using a geometric model of the environment from the AR content. In other words, audio data output by the outputting means can comprise the first audio data and/or the second audio data, depending on an operating mode (first or second) of the outputting means.

The outputting of the audio data can be dependent on the user position. When the user <NUM> is positioned within the occluded region <NUM>, as shown in <FIG>, the received audio signals can be processed to remove those signals which would be blocked by a physical representation of the virtual object <NUM>. In other words, the audio data generated by at least partially removing those of the received audio signals which would be occluded by the virtual object <NUM> is output to the user through the outputting means <NUM>. For example, the user <NUM> may hear the reflected audio signals 112b through the outputting means <NUM> but not the direct audio signals 112a (as is discussed above with respect to the arrangement of <FIG>).

However, when the user is located outside of the occlusion zone, as in <FIG>, the audio data output to the user may comprise all of the received audio signals <NUM> (direct audio signals 112a and reflected audio signals 112b), since substantially none of the signals received at the user's current position are occluded by the virtual object. In other words, the received signals are not processed to remove any of the audio signals <NUM>, and the recorded signals are replayed or reproduced to the user.

In some examples, the user <NUM> may be moving. As seen in <FIG>, the user may be moving from a current location to a new position <NUM> in the environment along a trajectory <NUM>. The current location of the user is the location of the user in the environment <NUM> at a current time point (i.e. the point in time at which the location of the user is determined). The processing means of the apparatus can be configured to switch between simply replaying the audio signals <NUM> and processing the signals to remove one or more occluded signals at a point along the user's trajectory <NUM>. In other words, the switching is based on the user's position, using the positional data.

The current position or location of the user <NUM> can be obtained from sensor data obtained by the AR device, for example using a camera or other vision data, as discussed above; the resulting positional data may optionally be combined with the geometric model of the environment <NUM>. In addition, the user's trajectory <NUM> may optionally be estimated. If the user is moving, the trajectory <NUM> may be estimated based on the user's direction and speed of movement, which parameters can themselves be determined from the sensor data. These factors can be used to estimate the new position of the user at a future time point. In other words, an estimated position of the user at a future point in time may be determined. The determining of an estimated position of the user at a future time point may be based on the user's current location and the user's estimated trajectory. At the future point in time, the user may be at any suitable position along trajectory <NUM>, or at a position on a different trajectory, depending on the user's speed and direction at the selected future time point. Any other factors or variables may also be used in the determining of an estimated position of the user.

The above described closed, or second mode, of the outputting means <NUM> can increase latency and in some cases can change the overall sound (i.e., by removing noise), even when the audio signals <NUM> are being replayed to the user <NUM> in full. In other words, the processing required to operate in a closed mode can increase latency in the signal path as compared to the first or open mode, and can change the overall sound quality. The second mode can therefore be slower and more computationally expensive than the first mode.

Generation of the audio data to remove one or more of the audio signals can further increase latency in the second mode. It can therefore be desirable to only operate in the second mode in particular circumstances. It has been recognised that operation in the second mode may only be required in circumstances where the above-described processing to remove one of more of the real audio signals <NUM> is desirable.

With reference to <FIG>, the occluded region <NUM> (determined at S820) is obtained S830. It will be understood that parameters of the occluded region <NUM> may be stored after S820, and then subsequently obtained. Alternatively, S820 and S830 can be combined and performed in a single step. The user position (and optionally the user trajectory <NUM>) may be determined or otherwise obtained at S840, for example using the techniques described above.

The outputting means <NUM>, which can be of any suitable type, are configured to operate in both a first mode and in a second mode. The first mode is an open mode as described above, wherein substantially no processing is performed on audio signals <NUM> received from the one or more real audio sources <NUM>. In the second mode, which may be a closed mode, the outputting means is configured to output to the user <NUM> audio data generated by processing audio signals <NUM> received from the one or more real audio sources <NUM> to at least partially remove those audio signals which would be occluded by the virtual object <NUM>. The processing can be performed by any of the techniques described above.

The apparatus comprises means for switching the outputting means between the first mode and the second mode based on a location of the user relative to the occluded region <NUM>. In other words, the location or position of the user can be used to determine a mode of operation for the outputting means S850. For example, the mode of operation is determined based on whether the user <NUM> is in the occluded region <NUM> or not. When the user is in the occluded region <NUM> (as in <FIG>), the second (closed) mode of operation is used. When the user is outside of the occluded region <NUM> (as in <FIG>), the first (open) mode of operation is used. In this way, a more immersive experience may be provided to the user whilst at the same time latency and computational resources can be reduced.

This arrangement is described further with reference to <FIG>. In <FIG>, the outputting means <NUM> is configured to operate in the first mode, as the virtual object <NUM> does not occlude the direct audio signals 112a from the real audio source <NUM>. In <FIG>, two virtual objects 104a, 104b are present. 104b has no occluding effect, but virtual object 104a has an occluding effect on the direct audio signals 112a.

Virtual object 104a therefore acts as an obstruction <NUM> in the environment. It will therefore be understood that the occluded region <NUM> will differ between <FIG>; the switching between the first mode and the second mode of the outputting means is therefore determined not just on a position of a user relative to a static occluded region, but on the relative positions of the user <NUM> and one or more virtual objects <NUM> present in the environment.

In some example embodiments, other parameters or factors may be used by the means for switching in determining whether to switch the operating means between the first and second modes. The frequency of the switching may be chosen to minimize disruption of the immersive AR user experience. Some factors which may be used are discussed below.

The user trajectory <NUM> and the speed of movement of the user may be taken into account when switching between modes of operation. The trajectory and speed may be used to estimate the user's position <NUM> a few moments into the future (i.e. at a future time point). If the estimated position <NUM> is within the occluded region <NUM> when a current location of the user is outside of the occluded region <NUM> then the closed or second mode of operation may be faded in gradually.

If there is no occluded region <NUM>, or the area or volume of the region is smaller than a predetermined threshold value, the mode may be set to the open, first mode and not switched to the second mode. Alternatively, if the occluded region <NUM> covers all (or most) of the area of environment <NUM>, the mode may be set to the closed, second mode and not switched. This approach can reduce computational resources in switching the outputting means operating mode in cases where the effect on the user will be minimal.

After switching to the second mode, the subsequent switch back to the open, first mode can be delayed for a suitable period of time in order to maintain continuity of audio output and provide a smooth transition. For example, the switch to the first mode may be delayed until the user's <NUM> position is such that the occlusion by the virtual object <NUM> is insignificant. However, the switch to the second, closed, mode can be more immediate to provide intuitive activation of the occlusion effect to the user <NUM>. The immersive experience may therefore be improved, while efficiently managing computational resources.

If there are no real-life sound sources present (i.e. no real audio source <NUM>), the open mode may be used at all times. In some examples, the user's <NUM> interaction with the real-life sources (such as speaking to another person, or speaking to a home assistant) may override the switching and the system may remain in the first mode to allow low-latency (or no latency) communication by the user.

<FIG> is a flow diagram showing processing operations according to some example embodiments which may provide a more realistic and intuitive representation of a virtual object in an augmented reality environment. For example, the processing operations may be performed by hardware, software, firmware or a combination thereof. The processing operations may, for example, be performed by a processing system of the AR device processing system and/or by a processing system of the AR media player <NUM>.

A first operation <NUM> comprises determining, relative to a real audio source, a position of a virtual object. The audio source may be the real audio source <NUM> and the virtual object may be virtual object <NUM>.

A second operation <NUM> comprises determining, based on the relative position, an occluded region of an environment. This determination may be by means of using any of the above-mentioned methods of position determination.

A third operation <NUM> comprises processing audio signals, received at the occluded region from the real audio source, to generate audio data by at least partially removing audio signals which would be occluded by the virtual object.

A fourth operation <NUM> comprises outputting the generated audio data to a user (with outputting means <NUM>). The user device may be (but is not limited to) the AR headset <NUM> shown in <FIG>.

A fifth operation <NUM> comprises switching outputting means to an open mode and stopping the processing of audio signals of operation <NUM>. The switching may be based on a location of a user, as discussed with reference to <FIG>.

<FIG> shows an apparatus according to some example embodiments. The apparatus may be configured to perform the operations described herein, for example operations described with reference to any disclosed process. The apparatus comprises at least one processor <NUM> and at least one memory <NUM> directly or closely connected to the processor. The memory <NUM> includes at least one random access memory (RAM) 1201a and at least one read-only memory (ROM) 1201b. Computer program code (software) <NUM> is stored in the ROM 1201b. The apparatus may be connected to a transmitter (TX) and a receiver (RX). The apparatus may, optionally, be connected with a user interface (UI) for instructing the apparatus and/or for outputting data.

The at least one processor <NUM>, with the at least one memory <NUM> and the computer program code <NUM> are arranged to cause the apparatus to at least perform at least the method according to any preceding process, for example as disclosed in relation to the flow diagram of <FIG> and related features thereof. The apparatus may comprise some or all of the AR system of <FIG>.

<FIG> shows a non-transitory media <NUM> according to some embodiments. The non-transitory media <NUM> is a computer readable storage medium. It may be e.g. a CD, a DVD, a USB stick, a blue ray disk, etc. The non-transitory media <NUM> stores computer program code, causing an apparatus to perform the method of any preceding process for example as disclosed in relation to the flow diagram of <FIG> and related features thereof.

Names of network elements, protocols, and methods are based on current standards. In other versions or other technologies, the names of these network elements and/or protocols and/or methods may be different, as long as they provide a corresponding functionality. For example, embodiments may be deployed in <NUM>/<NUM>/<NUM>/<NUM> networks and further generations of 3GPP but also in non-3GPP radio networks such as WiFi.

A memory may be volatile or non-volatile. It may be e.g. a RAM, a SRAM, a flash memory, a FPGA block ram, a DVD, a CD, a USB stick, and a blue ray disk.

If not otherwise stated or otherwise made clear from the context, the statement that two entities are different means that they perform different functions. It does not necessarily mean that they are based on different hardware. That is, each of the entities described in the present description may be based on a different hardware, or some or all of the entities may be based on the same hardware. It does not necessarily mean that they are based on different software. That is, each of the entities described in the present description may be based on different software, or some or all of the entities may be based on the same software. Each of the entities described in the present description may be embodied in the cloud.

Implementations of any of the above described blocks, apparatuses, systems, techniques or methods include, as non-limiting examples, implementations as hardware, software, firmware, special purpose circuits or logic, general purpose hardware or controller or other computing devices, or some combination thereof. Some embodiments may be implemented in the cloud.

Claim 1:
An apparatus comprising headphones configured to operate in a first mode and a second mode and comprising means for:
determining, relative to one or more real audio sources in an environment (<NUM>), a relative position of a virtual object (<NUM>) in the environment;
determining, based on the relative position, an occluded region of the environment in which audio signals from the one or more real audio sources (<NUM>) would be occluded by the virtual object;
receiving audio signals (<NUM>) from the one or more real audio sources;
switching the headphones between the first mode and the second mode based on a location of a user (<NUM>) in the environment relative to the occluded region,
wherein when the location of the user is determined to be in the occluded region, the headphones operate in the second mode, wherein the second mode is a closed mode in which audio data is generated by processing the received audio signals to at least partially remove those audio signals which would be occluded by the virtual object, and in which the audio data is output to the user, and
wherein when the location of the user is determined to be outside the occluded region, the headphones operate in the first mode, wherein the first mode is an open mode in which the audio signals from the one or more real audio sources are received at the user ears with no processing performed on the received audio signals by the headphones and audio data associated with the one or more real audio sources is not output by the headphones