Patent Description:
"Mediated reality" describes when a user experiences a fully or partially artificial environment (a virtual space) as a virtual scene at least partially rendered by an apparatus to a user. The virtual scene is determined by a virtual position within the virtual space.

First person perspective-mediated reality is mediated reality in which the user's real position in a real space determines the virtual position within the virtual space.

In some implementations, the user's orientation in a physical space determines the virtual orientation within the virtual space but the user's location in the real space does not determine the virtual location within the virtual space.

In some implementations, the user's orientation in a physical space determines the virtual orientation within the virtual space and the user's location in the real space determines the virtual location within the virtual space.

Uncertainties in measuring the correct real position in the real space, may result in incorrect virtual positions within the virtual space.

<CIT> discloses embodiments relating to adapting sound fields in an environment.

<CIT> relates to a head-mounted device that can provide both three-dimensional visual and directional audio output. Movement of a user's head above a threshold level causes the audio orientation to be moved in response.

The invention relates to computer-implemented methods, apparatus, computer programs for enabling consumption of virtual content for mediated reality based on a point of view of a notional listener according to the appended claims.

For a better understanding of various examples that are useful for understanding the brief description, reference will now be made by way of example only to the accompanying drawings in which:.

"artificial environment" may be something that has been recorded or generated.

"virtual visual space" refers to fully or partially artificial environment that may be viewed, which may be three dimensional.

"virtual visual scene" refers to a representation of the virtual visual space viewed from a particular point of view within the virtual visual space.

'virtual visual object' is a visible virtual object within a virtual visual scene.

"sound space" (or "virtual sound space") refers to an arrangement of sound sources in a three-dimensional space. A sound space may be defined in relation to recording sounds (a recorded sound space) and in relation to rendering sounds (a rendered sound space).

"sound scene" (or "virtual sound scene") refers to a representation of the sound space listened to from a particular point of view within the sound space.

"sound object" refers to sound source that may be located within the sound space. A source sound object represents a sound source within the sound space, in contrast to a sound source associated with an object in the virtual visual space. A recorded sound object represents sounds recorded at a particular microphone or position. A rendered sound object represents sounds rendered from a particular position.

"virtual space" may mean a virtual visual space, mean a sound space or mean a combination of a virtual visual space and corresponding sound space. In some examples, the virtual space may extend horizontally up to <NUM>° and may extend vertically up to <NUM>°.

"virtual scene" may mean a virtual visual scene, mean a sound scene or mean a combination of a virtual visual scene and corresponding sound scene.

'virtual object' is an object within a virtual scene, it may be an artificial virtual object (e.g. a computer-generated virtual object) or it may be an image of a real object in a real space that is live or recorded. It may be a sound object and/or a virtual visual object.

"Virtual position" is a position within a virtual space. It may be defined using a virtual location and/or a virtual orientation. It may be considered to be a movable 'point of view'.

"Correspondence" or "corresponding" when used in relation to a sound space and a virtual visual space means that the sound space and virtual visual space are time and space aligned, that is they are the same space at the same time.

"Correspondence" or "corresponding" when used in relation to a sound scene and a virtual visual scene (or visual scene) means that the sound space and virtual visual space (or visual scene) are corresponding and a notional (virtual) listener whose point of view defines the sound scene and a notional (virtual) viewer whose point of view defines the virtual visual scene (or visual scene) are at the same location and orientation, that is they have the same point of view (same virtual position).

"real space" (or "physical space") refers to a real environment, which may be three dimensional.

"real scene" refers to a representation of the real space from a particular point of view within the real space.

"real visual scene" refers to a representation of the real space viewed from a particular point of view within the real space.

"mediated reality" in this document refers to a user visually experiencing a fully or partially artificial environment (a virtual space) as a virtual scene at least partially rendered by an apparatus to a user. The virtual scene is determined by a point of view within the virtual space. Displaying the virtual scene means providing it in a form that can be perceived by the user.

The virtual visual scenes <NUM> illustrated may be mediated reality scenes, virtual reality scenes or augmented reality scenes. A virtual reality scene displays a fully artificial virtual visual space <NUM>. An augmented reality scene displays a partially artificial, partially real virtual visual space <NUM>.

The mediated reality, augmented reality or virtual reality may be user interactive-mediated. In this case, user actions at least partially determine what happens within the virtual visual space <NUM>. This may enable interaction with a virtual object <NUM> such as a visual element <NUM> within the virtual visual space <NUM>. For example, a user may be able to select and move the virtual object <NUM>.

The mediated reality, augmented reality or virtual reality may be perspective-mediated. In this case, user actions determine the point of view <NUM> within the virtual visual space <NUM>, changing the virtual visual scene <NUM>. For example, as illustrated in <FIG>, <FIG> a location <NUM> of the point of view <NUM> within the virtual visual space <NUM> may be changed and/or a direction or orientation <NUM> of the point of view <NUM> within the virtual visual space <NUM> may be changed. If the virtual visual space <NUM> is three-dimensional, the location <NUM> of the point of view <NUM> has three degrees of freedom e.g. up/down, forward/back, left/right and the direction <NUM> of the point of view <NUM> within the virtual visual space <NUM> has three degrees of freedom e.g. roll, pitch, yaw. The point of view <NUM> may be continuously variable in location <NUM> and/or direction <NUM> and user action then changes the location and/or direction of the point of view <NUM> continuously. Alternatively, the point of view <NUM> may have discrete quantised locations <NUM> and/or discrete quantised directions <NUM> and user action switches by discretely jumping between the allowed locations <NUM> and/or directions <NUM> of the point of view <NUM>.

<FIG> illustrates an example of a real space <NUM> comprising real objects <NUM> that partially corresponds with the virtual visual space <NUM> of <FIG>. In this example, each real object <NUM> in the real space <NUM> has a corresponding virtual object <NUM> in the virtual visual space <NUM>, however, each virtual object <NUM> in the virtual visual space <NUM> does not have a corresponding real object <NUM> in the real space <NUM>. In this example, one of the virtual objects <NUM>, the computer-generated visual element <NUM>, is an artificial virtual object <NUM> that does not have a corresponding real object <NUM> in the real space <NUM>.

A linear mapping may exist between the real space <NUM> and the virtual visual space <NUM> and the same mapping exists between each real object <NUM> in the real space <NUM> and its corresponding virtual object <NUM>. The relative relationship of the real objects <NUM> in the real space <NUM> is therefore the same as the relative relationship between the corresponding virtual objects <NUM> in the virtual visual space <NUM>.

<FIG> illustrates an example of a real visual scene <NUM> that partially corresponds with the virtual visual scene <NUM> of <FIG>, it includes real objects <NUM> but not artificial virtual objects. The real visual scene is from a perspective corresponding to the point of view <NUM> in the virtual visual space <NUM> of <FIG>. The real visual scene content is determined by that corresponding point of view <NUM> and the field of view <NUM> in virtual space <NUM> (point of view <NUM> in real space <NUM>).

<FIG> may be an illustration of an augmented reality version of the real visual scene <NUM> illustrated in <FIG>. The virtual visual scene <NUM> comprises the real visual scene <NUM> of the real space <NUM> supplemented by one or more visual elements <NUM> displayed by an apparatus to a user. The visual elements <NUM> may be a computer-generated visual element. In a see-through arrangement, the virtual visual scene <NUM> comprises the actual real visual scene <NUM> which is seen through a display of the supplemental visual element(s) <NUM>. In a see-video arrangement, the virtual visual scene <NUM> comprises a displayed real visual scene <NUM> and displayed supplemental visual element(s) <NUM>. The displayed real visual scene <NUM> may be based on an image from a single point of view <NUM> or on multiple images from different points of view at the same time, processed to generate an image from a single point of view <NUM>.

In augmented reality, the virtual content <NUM> is one or more virtual objects <NUM>. The virtual scene <NUM> comprises the real scene <NUM>, augmented or not by virtual content in dependence upon the point of view <NUM> of the user <NUM>.

In virtual reality, the virtual content <NUM> is the whole of the virtual scene and all virtual objects <NUM> within it. The virtual scene <NUM> comprises only the virtual content <NUM> determined in dependence upon the point of view <NUM> of the user <NUM>.

<FIG> illustrates, from a top perspective, an example of a sound space <NUM> that corresponds to the virtual visual space <NUM>. <FIG> is a two-dimensional projection or cross-section of the three dimensional sound space <NUM>. The sound space <NUM> defines a sound scene <NUM>.

In some but not necessarily all examples, the virtual visual space <NUM> and the sound space <NUM> may be corresponding and form a combined virtual space <NUM>,<NUM>. "Correspondence" or "corresponding" when used in relation to a sound space and a virtual visual space means that the sound space <NUM> and virtual visual space <NUM> are time and space aligned as combined virtual space <NUM>,<NUM>, that is they are the same space at the same time.

The correspondence between virtual visual space <NUM> and sound space <NUM> results in correspondence between the virtual visual scene <NUM> and the sound scene <NUM> to form a combined virtual scene <NUM>,<NUM>. "Correspondence" or "corresponding" when used in relation to a sound scene <NUM> and a virtual visual scene <NUM> means that the sound space <NUM> and virtual visual space <NUM> are corresponding and a notional (virtual) listener whose point of view defines the sound scene <NUM> and a notional (virtual) viewer whose point of view defines the virtual visual scene <NUM> are at the same location and orientation, that is they have the same point of view <NUM>.

In <FIG>, the sound space <NUM> and the virtual visual space <NUM> form a combined virtual space <NUM>,<NUM>. The sound space <NUM> is an arrangement of sound sources <NUM> in a three-dimensional space. In this example, the sound space <NUM> is a rendered sound space and the sound sources <NUM> comprise sound objects <NUM>.

The sound space <NUM> defines a sound scene <NUM> that corresponds to the virtual visual scene <NUM>. The sound scene <NUM> and the virtual visual scene <NUM> form a combined virtual scene <NUM>, <NUM>. The sound scene <NUM> is a representation of the sound space <NUM> listened to from a particular point of view <NUM> of a virtual listener (user) <NUM> within the sound space <NUM>. The sound scene <NUM> is first person perspective-mediated. The user's real point of view <NUM> determines the point of view <NUM> within the sound space, changing the sound scene <NUM>.

In this example, the point of view <NUM> within the sound space <NUM> corresponds to the point of view <NUM> within the virtual visual space <NUM> and the same label is used. The virtual scene <NUM>,<NUM> is first person perspective-mediated. The user's real point of view <NUM> determines the point of view <NUM> of the virtual user <NUM> within the combined virtual space <NUM>, <NUM>, changing the combined virtual scene <NUM>, <NUM>.

Correspondence in this sense means that there is a one-to-one mapping between the sound space <NUM> and the virtual visual space <NUM> such that a position in the sound space <NUM> has a corresponding position in the virtual visual space <NUM> and a position in the virtual visual space <NUM> has a corresponding position in the sound space <NUM>. Correspondence in this sense means that there is a one-to-one mapping between the sound scene <NUM> and the virtual visual scene <NUM> such that a position in the sound scene <NUM> has a corresponding position in the virtual visual scene <NUM> and a position in the virtual visual scene <NUM> has a corresponding position in the sound scene <NUM>. Corresponding also means that the coordinate system of the sound space <NUM> /sound scene <NUM> and the coordinate system of the virtual visual space <NUM> /virtual visual scene <NUM> are in register such that an object is positioned as a sound object in the sound scene and as a visual object in the visual scene at the same common position from the perspective of a virtual user <NUM>.

In this illustrated example, the user actions determine the point of view <NUM> within the sound space <NUM> (and virtual visual space <NUM>), changing the sound scene <NUM> and the virtual visual scene <NUM> simultaneously. For example, a location <NUM> of the point of view <NUM> within the virtual space <NUM>, <NUM> may be changed and/or a direction or orientation <NUM> of the point of view <NUM> within the virtual space <NUM>, <NUM> may be changed. If the virtual space <NUM>, <NUM> is three-dimensional, the location <NUM> of the point of view <NUM> has three degrees of freedom e.g. up/down, forward/back, left/right and the direction <NUM> of the point of view <NUM> within the virtual visual space <NUM> has three degrees of freedom e.g. roll, pitch, yaw. The point of view <NUM> may be continuously variable in location <NUM> and/or direction <NUM> and user action then changes the location and/or direction of the point of view <NUM> continuously. Alternatively, the point of view <NUM> may have discrete quantised locations <NUM> and/or discrete quantised directions <NUM> and user action switches by discretely jumping between the allowed locations <NUM> and/or directions <NUM> of the point of view <NUM>.

The functionality that enables control of a virtual visual space <NUM> and the virtual visual scene <NUM> dependent upon the virtual visual space <NUM> and the functionality that enables control of a sound space and the sound scene <NUM> dependent upon the sound space <NUM> may be provided by the same apparatus, system, method or computer program.

<FIG> illustrates an example of an apparatus <NUM> that is operable to enable mediated reality and/or augmented reality and/or virtual reality.

The apparatus <NUM> comprises a rendering device or devices <NUM>, which may render information to a user visually via a display, aurally via one or more audio outputs <NUM>, for example via loudspeakers, and/or haptically via a haptic device.

The audio output device <NUM> may comprise one or more spatially distributed audio sources. For example, binaural loudspeakers may be separated in a head mounted audio (HMA) device, loudspeakers may be spatially separated in a sound bar or in a distributed loudspeaker arrangement e.g. <NUM> or <NUM> surround sound.

The display <NUM> is for providing at least parts of the virtual visual scene <NUM> to a user in a form that is perceived visually by the user. The display <NUM> may be a visual display that provides light that displays at least parts of the virtual visual scene <NUM> to a user. Examples of visual displays include liquid crystal displays, organic light emitting displays, emissive, reflective, transmissive and transflective displays, direct retina projection display, near eye displays etc. The display may be a head-mounted display (HMD), a hand-portable display or television display or some other display.

The rendering device or devices <NUM> are controlled in this example but not necessarily all examples by a controller <NUM>.

Implementation of a controller <NUM> may be as controller circuitry. The controller <NUM> may be implemented in hardware alone, have certain aspects in software including firmware alone or can be a combination of hardware and software (including firmware).

As illustrated in <FIG> the controller <NUM> may comprise a processor <NUM> configured to load computer program instructions <NUM> from a memory <NUM>. The controller <NUM> may be implemented using instructions that enable hardware functionality, for example, by using executable computer program instructions <NUM> in a general-purpose or special-purpose processor <NUM> that may be stored on a computer readable storage medium (disk, memory etc) to be executed by such a processor <NUM>.

The memory <NUM> stores at least a computer program <NUM> comprising computer program instructions (computer program code) that controls the operation of the apparatus <NUM> when loaded into the processor <NUM>. The computer program instructions, of the computer program <NUM>, provide the logic and routines that enables the apparatus to perform at least the methods illustrated in <FIG>. The processor <NUM> by reading the memory <NUM> is able to load and execute the computer program <NUM>.

The apparatus <NUM> may enable user-interactive mediation for mediated reality and/or augmented reality and/or virtual reality. The input circuitry <NUM> detects user actions using user input <NUM>. These user actions are used by the controller <NUM> to determine what happens within the virtual space. This may enable interaction with a visual element <NUM> within the virtual visual space <NUM>.

The apparatus <NUM> may enable perspective mediation for mediated reality and/or augmented reality and/or virtual reality. The input circuitry <NUM> detects user actions. These user actions are used by the controller <NUM> to determine the point of view <NUM> within the virtual space, changing the virtual scene. The point of view <NUM> may be continuously variable in location and/or direction and user action changes the location and/or direction of the point of view <NUM>. Alternatively, the point of view <NUM> may have discrete quantised locations and/or discrete quantised directions and user action switches by jumping to the next location and/or direction of the point of view <NUM>.

The apparatus <NUM> may enable first person perspective for mediated reality, augmented reality or virtual reality. The input circuitry <NUM> detects the user's real point of view <NUM> using point of view sensor <NUM>. The user's real point of view is used by the controller <NUM> to determine the point of view <NUM> within the virtual space, changing the virtual scene. Referring back to <FIG>, a user <NUM> has a real point of view <NUM>. The real point of view may be changed by the user <NUM>. For example, a real location <NUM> of the real point of view <NUM> is the location of the user <NUM> and can be changed by changing the physical location <NUM> of the user <NUM>. For example, a real direction <NUM> of the real point of view <NUM> is the direction in which the user <NUM> is looking and can be changed by changing the real direction of the user <NUM>. The real direction <NUM> may, for example, be changed by a user <NUM> changing an orientation of their head or view point and/or a user changing a direction of their gaze.

A head-mounted apparatus <NUM>, may be used to enable first-person perspective mediation by measuring a change in location and/or a change in orientation of the user's head and/or a change in the user's direction of gaze. The head-mounted apparatus <NUM> may, for example, operate as a head mounted audio (HMA) device, a head mounted display (HMD) device or a combined head mounted display and audio (HMDA) device,.

In some but not necessarily all examples, the apparatus <NUM> comprises as part of the input circuitry <NUM> point of view sensors <NUM> for determining changes in the real point of view.

For example, positioning technology such as GPS, HAIP (high-accuracy indoor positioning), triangulation (trilateration) by transmitting to multiple receivers and/or receiving from multiple transmitters, acceleration detection and integration may be used to determine a new physical location <NUM> of the user <NUM> and real point of view <NUM>.

For example, accelerometers, electronic gyroscopes or electronic compasses may be used to determine a change in an orientation of a user's head or view point and a consequential change in the real direction <NUM> of the real point of view <NUM>.

For example, pupil tracking technology, based for example on computer vision, may be used to track movement of a user's eye or eyes and therefore determine a direction of a user's gaze and consequential changes in the real direction <NUM> of the real point of view <NUM>.

The apparatus <NUM> may comprise as part of the input circuitry <NUM> image sensors <NUM> for imaging the real space <NUM>.

An example of an image sensor <NUM> is a digital image sensor that is configured to operate as a camera. Such a camera may be operated to record static images and/or video images. In some, but not necessarily all embodiments, cameras may be configured in a stereoscopic or other spatially distributed arrangement so that the real space <NUM> is viewed from different perspectives. This may enable the creation of a three-dimensional image and/or processing to establish depth, for example, via the parallax effect.

In some, but not necessarily all embodiments, the input circuitry <NUM> comprises depth sensors <NUM>. A depth sensor <NUM> may comprise a transmitter and a receiver. The transmitter transmits a signal (for example, a signal a human cannot sense such as ultrasound or infrared light) and the receiver receives the reflected signal. Using a single transmitter and a single receiver some depth information may be achieved via measuring the time of flight from transmission to reception. Better resolution may be achieved by using more transmitters and/or more receivers (spatial diversity). In one example, the transmitter is configured to `paint` the real space <NUM> with structured light, preferably invisible light such as infrared light, with a spatially dependent pattern. Detection of a certain pattern by the receiver allows the real space <NUM> to be spatially resolved. The distance to the spatially resolved portion of the real space <NUM> may be determined by time of flight and/or stereoscopy (if the receiver is in a stereoscopic position relative to the transmitter).

In some but not necessarily all embodiments, the input circuitry <NUM> may comprise communication circuitry <NUM> in addition to or as an alternative to one or more of the image sensors <NUM> and the depth sensors <NUM>. Such communication circuitry <NUM> may communicate with one or more remote image sensors <NUM> in the real space <NUM> and/or with remote depth sensors <NUM> in the real space <NUM>.

The apparatus <NUM> may enable mediated reality and/or augmented reality and/or virtual reality, for example using the method <NUM> illustrated in <FIG> or a similar method. The controller <NUM> stores and maintains a model <NUM> of the virtual space <NUM> and a mapping between the physical space and the virtual space.

The model may be provided to the controller <NUM> or determined by the controller <NUM>. For example, sensors in input circuitry <NUM> may optionally be used to create overlapping depth maps of the real space from different points of view, virtual content is added, to produce and change the model.

Each real location <NUM> in the physical space <NUM>, through the mapping <NUM>, has a corresponding virtual location <NUM> in the virtual space <NUM> and vice versa. Each real orientation <NUM> in the physical space <NUM>, through the mapping <NUM>, has a corresponding virtual orientation <NUM> in the virtual space <NUM> and vice versa.

There are many different technologies that may be used to create a depth map. An example of a passive system, used in the Kinect ™ device, is when an object is painted with a non-homogenous pattern of symbols using infrared light and the reflected light is measured using multiple cameras and then processed, using the parallax effect, to determine a location of the object.

At block <NUM> it is determined whether or not the model of the virtual space <NUM> has changed. If the model of the virtual visual space <NUM> has changed the method moves to block <NUM>. If the model of the virtual visual space <NUM> has not changed the method moves to block <NUM>.

At block <NUM> it is determined whether or not the point of view <NUM> in the virtual visual space <NUM> has changed. If the point of view <NUM> has changed the method moves to block <NUM>. If the point of view <NUM> has not changed the method returns to block <NUM>.

At block <NUM>, a two-dimensional projection of the three-dimensional virtual visual space <NUM> is taken from the location <NUM> and in the direction <NUM> defined by the current point of view <NUM>. The projection is limited by the field of view <NUM> to produce the virtual visual scene <NUM>. The projection may also define the sound scene. The method then returns to block <NUM>.

<FIG> illustrates an example of a method <NUM> for updating a model of the virtual visual space <NUM> for augmented reality. Where the apparatus <NUM> enables augmented reality, the virtual visual space <NUM> comprises objects <NUM> from the real space <NUM> and also visual elements <NUM> not present in the real space <NUM>. The combination of such visual elements <NUM> may be referred to as the artificial virtual visual space.

At block <NUM> it is determined whether or not the real space <NUM> has changed. If the real space <NUM> has changed the method moves to block <NUM>. If the real space <NUM> has not changed the method moves to block <NUM>. Detecting a change in the real space <NUM> may be achieved at a pixel level using differencing and may be achieved at an object level using computer vision to track objects as they move.

At block <NUM> it is determined whether or not the artificial virtual visual space has changed. If the artificial virtual visual space has changed the method moves to block <NUM>. If the artificial virtual visual space has not changed the method returns to block <NUM>. As the artificial virtual visual space is generated by the controller <NUM> changes to the visual elements <NUM> are easily detected.

At block <NUM>, the model of the virtual visual space <NUM> is updated.

The blocks illustrated in the <FIG> may represent steps in a method and/or sections of code in the computer program <NUM>. The illustration of a particular order to the blocks does not necessarily imply that there is a required or preferred order for the blocks and the order and arrangement of the block may be varied. Furthermore, it may be possible for some blocks to be omitted.

<FIG> illustrate examples of apparatus <NUM> that enable display of at least parts of the virtual visual scene <NUM> to a user and rendering of audio to a user.

<FIG> illustrates a handheld apparatus <NUM> comprising a display screen as display <NUM> that displays images to a user and is used for displaying the virtual visual scene <NUM> to the user. The apparatus <NUM> may be moved deliberately in the hands of a user in one or more of the previously mentioned six degrees of freedom. The handheld apparatus <NUM> may house the sensors <NUM> for determining changes in the real point of view from a change in orientation of the apparatus <NUM>. The handheld apparatus <NUM> may house the sensors <NUM> for determining changes in the real point of view from a change in a user controlled device such as, for example, actuation of buttons, virtual buttons, slider, joystick, etc. The handheld apparatus <NUM> may be or may be operated as a see-video arrangement for augmented reality that enables a live or recorded video of a real visual scene <NUM> to be displayed on the display <NUM> for viewing by the user while one or more visual elements <NUM> are simultaneously displayed on the display <NUM> for viewing by the user. The combination of the displayed real visual scene <NUM> and displayed one or more visual elements <NUM> provides the virtual visual scene <NUM> to the user.

If the handheld apparatus <NUM> has a camera mounted on a face opposite the display <NUM>, it may be operated as a see-video arrangement that enables a live real visual scene <NUM> to be viewed while one or more visual elements <NUM> are displayed to the user to provide in combination the virtual visual scene <NUM>.

<FIG> illustrates a head-mounted apparatus <NUM> comprising a display <NUM> and/or audio output <NUM> that renders content to a user. The head-mounted apparatus <NUM> may be moved automatically when a head of the user moves.

A head-mounted apparatus <NUM> comprising a display <NUM> may be referred to as a head-mounted display (HMD) device.

A head-mounted apparatus <NUM> comprising an audio output <NUM> (e.g. a loudspeaker) may be referred to as a head-mounted audio (HMA) device.

The head-mounted apparatus <NUM> may house the sensors <NUM> (not illustrated) for point of view detection that detect a location and orientation of the apparatus <NUM> or an orientation of the apparatus <NUM>.

The head-mounted apparatus <NUM> may house the sensors <NUM> (not illustrated) for gaze direction detection and/or selection gesture detection.

The head-mounted apparatus <NUM> may be a see-through HMD arrangement for augmented reality that enables a live real visual scene <NUM> to be viewed while one or more visual elements <NUM> are displayed by the display <NUM> to the user to provide in combination the virtual visual scene <NUM>. In this case a visor <NUM>, if present, is transparent or semi-transparent so that the live real visual scene <NUM> can be viewed through the visor <NUM>.

The head-mounted apparatus <NUM> may be operated as a see-video arrangement for augmented reality that enables a live or recorded video of a real visual scene <NUM> to be displayed by the display <NUM> for viewing by the user while one or more visual elements <NUM> are simultaneously displayed by the display <NUM> for viewing by the user. The combination of the displayed real visual scene <NUM> and displayed one or more visual elements <NUM> provides the virtual visual scene <NUM> to the user. In this case a visor <NUM> is opaque and may be used as display <NUM>.

Referring back to <FIG>, an apparatus <NUM> may enable user-interactive mediation for mediated reality and/or augmented reality and/or virtual reality. The input circuitry <NUM> detects user actions using user input <NUM>. These user actions are used by the controller <NUM> to determine what happens within the virtual visual space <NUM>. This may enable interaction with a visual element <NUM> within the virtual visual space <NUM>.

The detected user actions may, for example, be gestures performed in the real space <NUM>. Gestures may be detected in a number of ways. For example, depth sensors <NUM> may be used to detect movement of parts a user <NUM> and/or or image sensors <NUM> may be used to detect movement of parts of a user <NUM> and/or positional/movement sensors attached to a limb of a user <NUM> may be used to detect movement of the limb.

Object tracking may be used to determine when an object or user changes or moves. For example, tracking the object on a large macro-scale allows one to create a frame of reference that moves with the object. That frame of reference can then be used to track time-evolving changes of shape of the object, by using temporal differencing with respect to the object. This can be used to detect small scale human motion such as gestures, hand movement, finger movement, facial movement. These are scene independent user (only) movements relative to the user.

The apparatus <NUM> may track a plurality of objects and/or points in relation to a user's body, for example one or more joints of the user's body. In some examples, the apparatus <NUM> may perform full body skeletal tracking of a user's body. In some examples, the apparatus <NUM> may perform digit tracking of a user's hand.

The tracking of one or more objects and/or points in relation to a user's body may be used by the apparatus <NUM> in action recognition.

Referring to <FIG>, a particular action <NUM> in the real space <NUM> is an action user input used as a 'user control' event by the controller <NUM> to determine what happens within the virtual visual space <NUM>. An action user input is an action <NUM> that has meaning to the apparatus <NUM> as a user input.

Referring to <FIG>, illustrates that in some but not necessarily all examples, a corresponding representation of the action <NUM> in real space is rendered in the virtual visual scene <NUM> by the apparatus <NUM>. The representation involves one or more visual elements <NUM> moving <NUM> to replicate or indicate the action <NUM> in the virtual visual scene <NUM>.

An action <NUM> may be static or moving. A moving action may comprise a movement or a movement pattern comprising a series of movements. For example it could be making a circling motion or a side to side or up and down motion or the tracing of a sign in space. A moving action may, for example, be an apparatus-independent action or an apparatus-dependent action. A moving action may involve movement of a user input object e.g. a user body part or parts, or a further apparatus, relative to the sensors. The body part may comprise the user's hand or part of the user's hand such as one or more fingers and thumbs. In other examples, the user input object may comprise a different part of the body of the user such as their head or arm. Three-dimensional movement may comprise motion of the user input object in any of six degrees of freedom. The motion may comprise the user input object moving towards or away from the sensors as well as moving in a plane parallel to the sensors or any combination of such motion.

An action <NUM> may be a non-contact action. A non-contact action does not contact the sensors at any time during the action.

An action <NUM> may be an absolute action that is defined in terms of an absolute displacement from the sensors. Such an action may be tethered, in that it is performed at a precise location in the real space <NUM>. Alternatively an action <NUM> may be a relative action that is defined in terms of relative displacement during the action. Such an action may be un-tethered, in that it need not be performed at a precise location in the real space <NUM> and may be performed at a large number of arbitrary locations.

An action <NUM> may be defined as evolution of displacement, of a tracked point relative to an origin, with time. It may, for example, be defined in terms of motion using time variable parameters such as displacement, velocity or using other kinematic parameters. An un-tethered action may be defined as evolution of relative displacement Δd with relative time Δt.

A action <NUM> may be performed in one spatial dimension (1D action), two spatial dimensions (2D action) or three spatial dimensions (3D action).

<FIG> illustrates an example of a system <NUM> and also an example of a method <NUM> for controlling rendering of a sound space <NUM> for a notional (virtual) listener <NUM> at an arbitrary location <NUM> (the origin) and orientation <NUM> within the sound space <NUM> at a particular location and/or orientation from the listener <NUM>.

A sound space <NUM> is an arrangement of sound sources <NUM> in a three-dimensional space. A sound space <NUM> may be defined in relation to recording sounds (a recorded sound space) or in relation to rendering sounds (a rendered sound space).

The sound space <NUM> may optionally comprise one or more portable sound objects <NUM> and/or may optionally comprise one or more static sound objects <NUM>.

The relative location of a sound object from the origin may be represented by the vector z. The vector z therefore positions the sound object <NUM> relative to a notional (virtual) listener <NUM>.

The relative orientation <NUM> of the notional listener <NUM> at the origin may be represented by the value Δ. The orientation value Δ defines the notional listener's 'point of view' which defines the sound scene. The sound scene is a representation of the sound space listened to from a particular point of view <NUM> within the sound space <NUM>.

The audio signals <NUM> representing a static sound object <NUM> are, if necessary, coded by audio coder <NUM> into a multichannel audio signal <NUM>. If multiple static sound objects are present, the audio signals <NUM> for each would be separately coded by an audio coder into a multichannel audio signal.

The audio coder <NUM> may be a spatial audio coder such that the multichannel audio signals <NUM> represent the sound space and can be rendered giving a spatial audio effect. For example, the audio coder <NUM> may be configured to produce multichannel audio signals <NUM> according to a defined standard such as, for example, binaural coding, <NUM> surround sound coding, <NUM> surround sound coding etc or to change coding from one format to another.

The multichannel audio signals <NUM> are mixed by mixer <NUM> with multichannel audio signals <NUM> representing one or more portable sound objects <NUM> to produce a multi-sound object multichannel audio signal <NUM> that represents the sound scene relative to the origin and which can be rendered by an audio decoder corresponding to the audio coder <NUM> to reproduce a sound scene to a listener that corresponds to the sound scene when the listener is at the origin.

The multichannel audio signal <NUM> for the, or each, portable sound object <NUM> is processed before mixing to take account of any movement of the portable sound object relative to the origin.

The audio signals <NUM> are processed by the positioning block <NUM> to adjust for movement of the portable sound object <NUM> relative to the origin. The positioning block <NUM> takes as an input the vector z or some parameter or parameters dependent upon the vector z. The vector z represents the relative location of the portable sound object <NUM> relative to the origin.

The positioning block <NUM> may be configured to adjust for any time misalignment between the audio signals <NUM> and the audio signals <NUM> so that they share a common time reference frame. This may be achieved, for example, by correlating naturally occurring or artificially introduced (non-audible) audio signals that are present within the audio signals <NUM> with those within the audio signals <NUM>. Any timing offset identified by the correlation may be used to delay/advance the audio signals <NUM> before processing by the positioning block <NUM>.

The positioning block <NUM> processes the audio signals <NUM>, taking into account the relative orientation (Arg(z)) of that portable sound object relative to the origin.

The audio coding of the audio signals <NUM> to produce the multichannel audio signal <NUM> assumes a particular orientation of the rendered sound space relative to an orientation of the recorded sound space and the audio signals <NUM> are encoded to the multichannel audio signals <NUM> accordingly.

The relative orientation Arg (z) of the portable sound object <NUM> in the sound space is determined and the audio signals <NUM> representing the sound object are coded to the multichannels defined by the audio coding <NUM> such that the sound object is correctly oriented within the rendered sound space at a relative orientation Arg (z) from the listener. For example, the audio signals <NUM> may first be mixed or encoded into the multichannel signals <NUM> and then a transformation T may be used to rotate the multichannel audio signals <NUM>, representing the moving sound object, within the space defined by those multiple channels by Arg (z).

An orientation block <NUM> may be used to rotate the multichannel audio signals <NUM> by Δ, if necessary. Similarly, an orientation block <NUM> may be used to rotate the multichannel audio signals <NUM> by Δ, if necessary.

The functionality of the orientation block <NUM> is very similar to the functionality of the orientation function of the positioning block <NUM> except it rotates by Δ instead of Arg(z).

In some situations, for example when the sound scene is rendered to a listener through a head-mounted audio output device <NUM>, for example headphones using binaural audio coding, it may be desirable for a portion of the rendered sound space <NUM> to remain fixed in real space <NUM> when the listener turns their head in space. This means that the rendered sound space <NUM> needs to be rotated relative to the audio output device <NUM> by the same amount in the opposite sense to the head rotation. The orientation of the portion of the rendered sound space <NUM> tracks with the rotation of the listener's head so that the orientation of the rendered sound space <NUM> remains fixed in space and does not move with the listener's head.

The portable sound object signals <NUM> are additionally processed to control the perception of the distance D of the sound object from the listener in the rendered sound scene, for example, to match the distance |z| of the sound object from the origin in the recorded sound space. This can be useful when binaural coding is used so that the sound object is, for example, externalized from the user and appears to be at a distance rather than within the user's head, between the user's ears. The distance block <NUM> processes the multichannel audio signal <NUM> to modify the perception of distance.

<FIG> illustrates a module <NUM> which may be used, for example, to perform the method <NUM> and/or functions of the positioning block <NUM>, orientation block <NUM> and distance block <NUM> in <FIG>. The module <NUM> may be implemented using circuitry and/or programmed processors.

The Figure illustrates the processing of a single channel of the multichannel audio signal <NUM> before it is mixed with the multichannel audio signal <NUM> to form the multi-sound object multichannel audio signal <NUM>. A single input channel of the multichannel signal <NUM> is input as signal <NUM>.

The input signal <NUM> passes in parallel through a "direct" path and one or more "indirect" paths before the outputs from the paths are mixed together, as multichannel signals, by mixer <NUM> to produce the output multichannel signal <NUM>. The output multichannel signal <NUM>, for each of the input channels, are mixed to form the multichannel audio signal <NUM> that is mixed with the multichannel audio signal <NUM>.

The direct path represents audio signals that appear, to a listener, to have been received directly from an audio source and an indirect path represents audio signals that appear to a listener to have been received from an audio source via an indirect path such as a multipath or a reflected path or a refracted path.

The distance block <NUM> by modifying the relative gain between the direct path and the indirect paths, changes the perception of the distance D of the sound object from the listener in the rendered sound space <NUM>.

Each of the parallel paths comprises a variable gain device <NUM>, <NUM> which is controlled by the distance block <NUM>.

The perception of distance can be controlled by controlling relative gain between the direct path and the indirect (decorrelated) paths. Increasing the indirect path gain relative to the direct path gain increases the perception of distance.

In the direct path, the input signal <NUM> is amplified by variable gain device <NUM>, under the control of the distance block <NUM>, to produce a gain-adjusted signal <NUM>. The gain-adjusted signal <NUM> is processed by a direct processing module <NUM> to produce a direct multichannel audio signal <NUM>.

In the indirect path, the input signal <NUM> is amplified by variable gain device <NUM>, under the control of the distance block <NUM>, to produce a gain-adjusted signal <NUM>. The gain-adjusted signal <NUM> is processed by an indirect processing module <NUM> to produce an indirect multichannel audio signal <NUM>.

The direct multichannel audio signal <NUM> and the one or more indirect multichannel audio signals <NUM> are mixed in the mixer <NUM> to produce the output multichannel audio signal <NUM>.

The direct processing block <NUM> and the indirect processing block <NUM> both receive direction of arrival signals <NUM>. The direction of arrival signal <NUM> gives the orientation Arg(z) of the portable sound object <NUM> (moving sound object) in the recorded sound space and the orientation Δ of the rendered sound space <NUM> relative to the notional listener /audio output device <NUM>.

The location of the moving sound object changes as the portable object <NUM> moves in the recorded sound space and the orientation of the rendered sound space changes as a head-mounted audio output device rendering the sound space rotates.

The direct processing block <NUM> may, for example, include a system <NUM> that rotates the single channel audio signal, gain-adjusted input signal <NUM>, in the appropriate multichannel space producing the direct multichannel audio signal <NUM>. The system uses a transfer function to performs a transformation T that rotates multichannel signals within the space defined for those multiple channels by Arg(z) and by Δ, defined by the direction of arrival signal <NUM>. For example, a head related transfer function (HRTF) interpolator may be used for binaural audio. As another example, Vector Base Amplitude Panning (VBAP) may be used for loudspeaker format (e.g. <NUM>) audio.

The indirect processing block <NUM> may, for example, use the direction of arrival signal <NUM> to control the gain of the single channel audio signal, the gain-adjusted input signal <NUM>, using a variable gain device <NUM>. The amplified signal is then processed using a static decorrelator <NUM> and a static transformation T to produce the indirect multichannel audio signal <NUM>. The static decorrelator in this example uses a pre-delay of at least <NUM>. The transformation T rotates multichannel signals within the space defined for those multiple channels in a manner similar to the direct system but by a fixed amount. For example, a static head related transfer function (HRTF) interpolator may be used for binaural audio.

It will therefore be appreciated that the module <NUM> can be used to process the portable sound object signals <NUM> and perform the functions of:.

It should also be appreciated that the module <NUM> may also be used for performing the function of the orientation block <NUM> only, when processing the audio signals <NUM>. However, the direction of arrival signal will include only Δ and will not include Arg(z). In some but not necessarily all examples, gain of the variable gain devices <NUM> modifying the gain to the indirect paths may be put to zero and the gain of the variable gain device <NUM> for the direct path may be fixed. In this instance, the module <NUM> reduces to a system that rotates the recorded sound space to produce the rendered sound space according to a direction of arrival signal that includes only Δ and does not include Arg(z).

The positioning of a sound source <NUM>, for example a sound object <NUM>, has been described above as being dependent upon the vector z which represents the relative position of a virtual sound object <NUM> relative to the virtual listener <NUM>.

In the following, the vector zs(t) will be used in its place for the purpose of sound object positioning.

As previously described, changes in the virtual orientation <NUM> of the virtual listener <NUM> are controlled via Δ. If the virtual orientation has a unit vector o(t), then Δ is Arg[o(t)].

As illustrated in <FIG>, the vector z(t) will be used to represent the current relative position of a virtual sound object <NUM> relative to the virtual listener <NUM>. In some modes, but not all modes of operation, zs(t) may equal z(t), however in other modes of operation they are unequal.

The current location of a virtual sound object <NUM> is defined by vector rs(t). The current location of the virtual listener <NUM> is defined by vector ru(t). The current relative position of a virtual sound object <NUM> relative to the virtual listener <NUM> is: z(t)= rs(t) - ru(t).

"Current" means the most recent and closest to current time. It is the closest to real-time and is the most contemporaneous.

There is a mapping that maps a current location <NUM> of the user <NUM> in real space <NUM>, defined by a vector Ru(t), to the current location <NUM> of the virtual listener <NUM> in virtual space <NUM>, defined by vector ru(t), and that maps the current orientation <NUM> of the user <NUM> defined by a vector O(t) in real space <NUM> to the current orientation <NUM> of the virtual listener <NUM> defined by vector o(t) in virtual space <NUM>.

In a first mode of operation, the relative position zs(t) of a virtual sound object <NUM> relative to the virtual listener <NUM> has a first dependency upon the currently determined relative position z(t) of the virtual sound object <NUM> from the notional listener <NUM> in the virtual space <NUM> based on the current measured location Ru(t) of the user <NUM> in the real space <NUM>.

In a second mode of operation, different to the first mode of operation, the relative position zs(t) of a virtual sound object <NUM> relative to the virtual listener <NUM> has a second dependency upon the currently determined relative position z(t) of the virtual sound object <NUM> from the notional listener <NUM> in the virtual space <NUM> based on the current measured location Ru(t) of a user <NUM> in the real space <NUM>.

The first dependency is higher than the second dependency.

A higher (or greater) dependency with respect to a variable means a greater or larger magnitude of the differential with respect to that variable.

<FIG> illustrates an example of a method <NUM> for controlling a mode of operation of first-person-perspective mediated reality.

The method <NUM> comprises at block <NUM> enabling a first mode for controlling a relative position zs(t) of a virtual sound object <NUM> from a notional listener <NUM> in a virtual space <NUM> in dependence upon a point of view <NUM> of the notional listener <NUM>. The point of view <NUM> of the notional listener <NUM> changes with the point of view <NUM> of the user <NUM>. The point of view <NUM> of the user <NUM> is dependent upon the orientation <NUM> of the user <NUM> e.g. O(t) and may additionally be dependent upon the location <NUM> of the user <NUM> e.g. Ru(t).

During the first mode, there is a higher dependency of the relative position zs(t) of the virtual sound object <NUM> from the notional listener <NUM> upon a currently determined relative position z(t) of the virtual sound object <NUM> from the notional listener <NUM> in the virtual space <NUM> based on the current measured location Ru(t) of a user <NUM> in a real space <NUM>.

The method then comprises at block <NUM> automatically switching from the first mode to the second mode.

The method <NUM> then comprises at block <NUM> automatically enabling a second mode for controlling a relative position of the virtual sound object <NUM> from the notional listener <NUM> in the virtual space <NUM> in dependence upon the point of view <NUM> of the notional listener <NUM>. The point of view <NUM> of the notional listener <NUM> may change with the point of view <NUM> of the user <NUM>. The point of view <NUM> of the user <NUM> is dependent upon the orientation <NUM> of the user <NUM> e.g. O(t) and may additionally be dependent upon the location <NUM> of the user <NUM> e.g. Ru(t).

During the second mode, there is a lower dependency of the relative position zs(t) of the virtual sound object <NUM> from the notional listener <NUM> upon a currently determined relative position z(t) of the virtual sound object <NUM> from the notional listener <NUM> in the virtual space <NUM> based on the current measured location Ru(t) of a user <NUM> in a real space <NUM>.

Then the method <NUM> comprises at block <NUM> automatically switching from the second mode to the first mode.

In addition to controlling the dependency of zs(t) on the current measured location Ru(t) of a user <NUM> in a real space <NUM>, it may also be desirable to control Arg[zs(t)] - Arg[o(t)] }. This may be achieved by controlling Arg[zs(t)] and/or controlling Arg[zs(t)] +Δ.

During the second mode, there may be a lower dependency (compared to the first mode) of the adapted relative position of the virtual sound object <NUM> from the notional listener <NUM> upon a currently determined relative position z(t) of the virtual sound object <NUM> from the notional listener <NUM> in the virtual space <NUM> based on the current measured location Ru(t) of a user <NUM> in a real space <NUM> and the currently measured orientation O(t) of the user <NUM> in the real space <NUM>. The point of view <NUM> of the user <NUM> in the real space <NUM> is defined by the orientation of the user <NUM> and optionally the location <NUM> of the user <NUM>, that is, by the couplet Ru(t), O(t).

<FIG> illustrates an example of a state diagram in which a first state <NUM> represents the first mode and a second state <NUM> represents the second mode. A transition <NUM> causes a state transition from the state <NUM> (first mode) to the state <NUM> (second mode). A transition <NUM> causes a state transition from the state <NUM> (second mode) to the state <NUM> (first mode mode).

The first state <NUM> corresponds to block <NUM> of <FIG>. The second state <NUM> corresponds to block <NUM> of <FIG>. The transition <NUM> corresponds to block <NUM> of <FIG>. The transition <NUM> corresponds to block <NUM> of <FIG>.

The transition <NUM> from the first mode to the second mode has as a necessary and sufficient condition or as a necessary condition that a distance |zs(t)| of the virtual sound object <NUM> from the notional listener <NUM> in virtual space decreases below a threshold, for example, <NUM>. In some but not necessarily all examples, the transition <NUM> from the first mode to the second mode has as a necessary and sufficient condition or as a necessary condition that a distance |zs(t)| of the virtual sound object <NUM> from the notional listener <NUM> in virtual space decreases below a threshold and some other additional criterion or criteria is satisfied.

For example, one example of an additional criterion is how long a criterion or criteria are satisfied for. Thus, the transition <NUM> from the first mode to the second mode may have as a necessary and sufficient condition or as a necessary condition that a distance |zs(t)| of the virtual sound object <NUM> from the notional listener <NUM> in virtual space decreases below a threshold for a predetermined threshold time.

For example, one example of an additional criterion uses a rate of change of a displacement parameter (e.g. distance |zs(t)| and/or orientation Arg(zs(t)) ) of the virtual sound object <NUM> from the notional listener <NUM> in virtual space. Thus the transition <NUM> from the first mode to the second mode may occur when a distance |zs(t)| of the virtual sound object <NUM> from the notional listener <NUM> in virtual space decreases below a threshold AND the rate of change of the a displacement parameter (e.g. distance |zs(t)| and/or orientation Arg(zs(t)) ) of the virtual sound object <NUM> from the notional listener <NUM> in virtual space passes a threshold. Thus the transition <NUM> from the first mode to the second mode may require that the rate of change of the distance |zs(t)| of the virtual sound object <NUM> from the notional listener <NUM> in virtual space decreases below a threshold and/or the transition <NUM> from the first mode to the second mode may require that the rate of change of the orientation Arg(zs(t)) of the virtual sound object <NUM> from the notional listener <NUM> in virtual space increases above a threshold.

In some but not necessarily all examples, the transition <NUM> from the first mode to the second mode has as a necessary condition that a time variation of a displacement parameter (e.g. distance |zs(t)| and/or orientation Arg(zs(t)) ) of the virtual sound object <NUM> from the notional listener <NUM> in virtual space satisfies a criterion e.g. speed of closure between the notional listener and the sound object <NUM> |d zs (t)/ dt| is greater than or less than a threshold value or rate of change of orientation between the notional listener and the sound object <NUM> |d/dt { Arg[zs(t)] - Arg[o(t)]} | or |d/dt { Arg[zs(t)] } | is greater than a threshold value.

In some but not necessarily all examples, the transition <NUM> from the first mode to the second mode has as a necessary condition that a value for the uncertainty in correctly measuring the current measured location Ru(t) of a user <NUM> in a real space <NUM> exceeds a threshold. In some example, the threshold may be based upon a distance |zs(t)| of the virtual sound object <NUM> from the notional listener <NUM>.

In some but not necessarily all examples, the transition <NUM> from the second mode to the first mode has as a necessary and sufficient condition or as a necessary condition that a distance |zs(t)| of the virtual sound object <NUM> from the notional listener <NUM> in virtual space increases above a threshold, for example, <NUM>.

In some but not necessarily all examples, the transition <NUM> from the second mode to the first mode has as a necessary and sufficient condition or as a necessary condition that a distance |zs(t)| of the virtual sound object <NUM> from the notional listener <NUM> in virtual space increases above a threshold and some other additional criterion or criteria is satisfied. For example, one example of an additional criterion is how long a criterion or criteria are satisfied for. Thus, the transition <NUM> from the second mode to the first mode may occur when distance |zs(t)| of the virtual sound object <NUM> from the notional listener <NUM> in virtual space increases above a threshold for a predetermined threshold time.

For example, one example of an additional criterion uses a rate of change of a displacement parameter (e.g. distance |zs(t)| and/or orientation Arg(zs(t)) ) of the virtual sound object <NUM> from the notional listener <NUM> in virtual space. Thus the transition <NUM> from the second mode to the first mode may occur when a distance |zs(t)| of the virtual sound object <NUM> from the notional listener <NUM> in virtual space increases above a threshold AND the rate of change of the a displacement parameter (e.g. distance |zs(t)| and/or orientation Arg(zs(t)) ) of the virtual sound object <NUM> from the notional listener <NUM> in virtual space passes a threshold. Thus the transition <NUM> from the second mode to the first mode may require that the rate of change of the distance |zs(t)| of the virtual sound object <NUM> from the notional listener <NUM> in virtual space increases above a threshold and/or the transition <NUM> from the second mode to the first mode may require that the rate of change of the orientation Arg(zs(t)) ) of the virtual sound object <NUM> from the notional listener <NUM> in virtual space decreases below a threshold.

In some but not necessarily all examples, the transition <NUM> from the second mode to the first mode has as a necessary and sufficient condition or as a necessary condition that a time variation of a displacement parameter (e.g. distance |zs(t)| and/or orientation Arg(zs(t)) ) of the virtual sound object <NUM> from the notional listener <NUM> of the virtual sound object <NUM> from the notional listener <NUM> in virtual space satisfies a criterion e.g. speed of separation between the notional listener and the sound object <NUM> |d zs (t)/ dt| is less than or greater than a threshold value or rate of change of orientation between the notional listener and the sound object <NUM> |d/dt { Arg[zs(t)] - Arg[o(t)]} | or |d/dt { Arg[zs(t)] } | is less than a threshold value.

In some but not necessarily all examples, the transition <NUM> from the second mode to the first mode has as a necessary and sufficient condition or as a necessary condition that a value for the uncertainty in correctly measuring the current measured location Ru(t) of a user <NUM> in a real space <NUM> exceeds a threshold that may be based upon a distance |zs(t)| of the virtual sound object <NUM> from the notional listener <NUM>.

In some but not necessarily all examples, the condition C causes transition <NUM> and the condition NOT(C) causes transition <NUM>. The thresholds for the transitions <NUM>, <NUM> may therefore be the same- one transition occurs for passing the threshold in a first direction (e.g. exceeding the threshold) and the other transition occurs for passing the threshold in a second direction, opposite the first direction (e.g. no longer exceeding the threshold).

<FIG> illustrate examples of a notional listener <NUM> moving past a sound source <NUM> e.g. sound object <NUM>, during time t1, t2, t3. The closest approach min(|z(t)|) is less for <FIG> than for <FIG>.

<FIG> illustrates a variation of Arg[z(t)] - Arg[o(t)]} | with time. In this example, o(t) is constant so the Figure illustrates the variation of Arg[z(t)]. The trace A relates to the situation illustrated in <FIG>. The trace B relates to the situation illustrated in <FIG>.

<FIG> illustrates a portion of <FIG>, corresponding to the time of closest approach of notional listener <NUM> and sound object <NUM>, in greater detail. The rate of change of Arg(z(t)) is greater for trace B than trace A.

If there is some uncertainty concerning the measured point of view <NUM> of the user <NUM> in the real space defined by the orientation <NUM> of the user <NUM> and optionally the location <NUM> of the user <NUM>, that is, by the couplet Ru(t), O(t), then the effect will be greater for trace B than trace A. Switching from the first mode to the second mode mitigates this effect.

<FIG> are similar to <FIG> illustrate examples of a notional listener <NUM> moving past a sound source <NUM>, e.g. sound object <NUM>, during time t1, t2, t3. The closest approach min(z(t)) is less for <FIG> than for <FIG>.

If there is some uncertainty concerning the measured location <NUM> of the user <NUM>, there will be uncertainty in the location <NUM> of the notional listener <NUM>. This uncertainty is illustrated by circles of uncertainty <NUM> for the virtual location <NUM> of the notional listener and any corresponding uncertainty <NUM> associated with the point of view <NUM> of the notional listener <NUM>. The effect will be greater for <FIG> than <FIG>.

It is possible, for example, for the sound source <NUM> to switch it's bearing from the notional listener <NUM> randomly, the effect becoming more pronounced as the distance to the sound source <NUM> decreases and as the time spent near the sound source increases. It is possible, for example, for the sound source <NUM> to switch from the right hand side of the notional listener <NUM> to the left hand side of the listener <NUM> and back to the right hand side of the notional listener <NUM> because of uncertainty concerning the measured location <NUM> without any change in O(t) or o(t). Switching from the first mode to the second mode mitigates this effect.

<FIG> illustrates a variation of Arg[z(t)] - Arg[o(t)]} | with time. In this example, o(t) is constant so the Figure illustrates variation in Arg[z(t)].

Moving from left to right, up to point X the first mode is used, between X and Y the second mode is used and after Y the first mode is used again.

The dotted line T illustrates the relationship between relative orientation Arg[z(t) of the sound source <NUM> and the notional listener <NUM> according to the first mode. Arg[z(t) has a large variation between X at time tx and Y at time tx. The average gradient m between X and Y is (Y-X)/( ty - tx).

The hashed area illustrates the region occupied by the relationship between the relative orientation Arg[z(t) of the sound source <NUM> and the notional listener <NUM> according to the second mode.

During the second mode there is, for all or for the majority of the second mode, a lower dependency of the relative position zs(t) of the virtual sound object <NUM> from the notional listener <NUM> upon a currently determined relative position z(t) of the virtual sound object <NUM> from the notional listener <NUM> in the virtual space <NUM> based on the current measured location Ru(t) of a user <NUM> in a real space <NUM>. In the expression, |zs(t)| exp( i{ Arg[zs(t)] - Arg[o(t)]}, the value Arg[zs(t)] - Arg[o(t)] has a lesser gradient, d/dt{ Arg[zs(t)] - Arg[o(t)}, compared to the first mode, for all or for the majority of the second mode. d/dt{ Arg[zs(t)]} < m , assuming Arg[o(t)] is constant.

The rate of change of the vector between the notional listener <NUM> and the sound object <NUM>, d/dt{ Arg[zs(t)]}, is controlled between time tx and time ty.

As there is continuity at X and Y, there will be a discontinuity, for example, where the notional listener approaches closest to the sound source <NUM> between time ta and tb.

The discontinuity may be smoothed over time, but still represents a large change in Arg[zs(t)] - Arg[o(t)] over a time Δt= tb- ta. As Arg[o(t)] is constant: <MAT>.

In addition, on switching between the first mode and the second mode, the change in the relative position zs(t) of the virtual sound object <NUM> from the notional listener <NUM> is smoothed over time. The rate of change of rate of change of the relative position zs(t) does not exceed a threshold value n e.g. d<NUM>/dt<NUM>{ Arg[zs(t)] } < n.

Thus, during the second mode the relative position zs(t) of the virtual sound object <NUM> from the notional listener <NUM> is prevented from switching between a left hand side and a right hand side of the notional listener <NUM> except as a consequence of changes in orientation <NUM> of the notional listener <NUM> in the virtual space <NUM>.

In some but not necessarily all examples, during the second mode, changes in the relative position zs(t) of the virtual sound object <NUM> from the notional listener <NUM> in response to changes in the current measured position z(t) of user <NUM> in the real space <NUM> are damped but still dependent upon changes in orientation <NUM> of the notional listener <NUM> in the virtual space <NUM>. For example, the magnitude of the rate of change with time of the current measured position z(t) of user <NUM> in the real space <NUM> is prevented from exceeding a threshold value e.g. d/dt{ Arg[zs(t)]} < m , assuming Arg[o(t)] is constant, between tx and ta and between tb and ty.

In some but not necessarily all examples, during the second mode the relative position zs(t) of the virtual sound object <NUM> from the point of view <NUM> of the notional listener <NUM> is independent of the current measured position z(t) of user <NUM> in the real space <NUM> but still dependent upon changes in orientation <NUM> of the notional listener <NUM> in the virtual space <NUM>.

For example, as illustrated in <FIG>, d/dt{ Arg[zs(t)]} =<NUM>, assuming Arg[o(t)] is constant, between tx and ta and between tb and ty.

For example, during the second mode the relative position zs(t) of the virtual sound object from the notional listener is locked to an orientation of the notional listener in the virtual space e. g Arg[zs(t)] =constant, between tx and ta and between tb and ty.

Thus the rate of change of the vector between the notional listener <NUM> and the sound object <NUM>, d/dt{ Arg[zs(t)]}, is controlled between time tx and time ty. The distance represented by |zs(t)| exp( i{ Arg[zs(t)] - Arg[o(t)] }, is therefore changed as a consequence of controlling zs(t), which would unless compensated for result in a change in volume of the sound object <NUM>. In some but not necessarily all examples, during the second mode a volume of the sound object <NUM> is controlled to continue varying in dependence upon the relative position of the virtual sound object from the notional listener in the virtual space based on the current measured position of the user <NUM> in the real space <NUM>.

<FIG> Illustrates how the currently determined relative position zs(t) of the virtual sound object <NUM> from the notional listener <NUM> in the virtual space <NUM> can be based on the current measured position z(t) of the user <NUM> in the real space <NUM>, and a model-based position of the user. The model-based position of the user is produced using a Kalman filter <NUM>, or another historically based, predictive filter.

The model may, for example be a constant velocity model. During a prediction phase <NUM> an estimated position from the previous iteration and the model are used to produce a predicted position and during a following update phase <NUM> the predicted position and the measured position are used to produce an estimated position for this iteration. The method then iterates. The iteration rate may, for example, be less than <NUM>, it may for example be <NUM>-<NUM> for position measurement.

A predicted position may be used instead of a measured position in the examples described above, in the first mode and/or the second mode.

The predicted position may be used instead of a position z(t) based on a measured position to determine whether or not switch between the first mode and the second mode.

From the foregoing, it will be appreciated that the apparatus <NUM> can comprise:.

The computer program <NUM> may arrive at the apparatus <NUM> via any suitable delivery mechanism. The delivery mechanism may be, for example, a non-transitory computer-readable storage medium, a computer program product, a memory device, a record medium such as a compact disc read-only memory (CD-ROM) or digital versatile disc (DVD), an article of manufacture that tangibly embodies the computer program <NUM>. The delivery mechanism may be a signal configured to reliably transfer the computer program <NUM>. The apparatus <NUM> may propagate or transmit the computer program <NUM> as a computer data signal.

Although the memory <NUM> is illustrated in <FIG> as a single component/circuitry it may be implemented as one or more separate components/circuitry some or all of which may be integrated/removable and/or may provide permanent/semi-permanent/ dynamic/cached storage.

Although the processor <NUM> is illustrated in <FIG> as a single component/circuitry it may be implemented as one or more separate components/circuitry some or all of which may be integrated/removable.

The method <NUM> may be performed by the apparatus <NUM>, for example as previously described in relation to <FIG>, the controller <NUM> of the apparatus <NUM> or the computer program <NUM>. The apparatus <NUM> may a self-contained apparatus that performs all necessary functions itself or may be part of a system and delegate certain functions to other apparatuses or services.

The recording of data may comprise only temporary recording, or it may comprise permanent recording or it may comprise both temporary recording and permanent recording, Temporary recording implies the recording of data temporarily. This may, for example, occur during sensing or image capture, occur at a dynamic memory, occur at a buffer such as a circular buffer, a register, a cache or similar. Permanent recording implies that the data is in the form of an addressable data structure that is retrievable from an addressable memory space and can therefore be stored and retrieved until deleted or over-written, although long-term storage may or may not occur. The use of the term 'capture' in relation to an image relates to temporary recording of the data of the image. The use of the term 'store' in relation to an image relates to permanent recording of the data of the image.

Claim 1:
A computer-implemented method comprising:
switching between a first mode for controlling a relative position of a virtual sound object (<NUM>) from a notional listener (<NUM>) in a virtual space (<NUM>,<NUM>) in dependence upon a point of view (<NUM>) of the notional listener (<NUM>)
and a second mode for controlling the relative position of the virtual sound object (<NUM>) from the notional listener (<NUM>) in the virtual space (<NUM>,<NUM>) in dependence upon the point of view (<NUM>) of the notional listener (<NUM>), wherein a transition (<NUM>) from the first mode to the second mode occurs when a distance of a virtual sound object from the notional listener in virtual space decreases below a threshold, and wherein
during the first mode there is a first dependency of the relative position of the virtual sound object from the notional listener (<NUM>) upon a currently determined relative position of the virtual sound object (<NUM>) from the notional listener (<NUM>) in the virtual space (<NUM>,<NUM>) based on the current measured location of a user (<NUM>) in a real space (<NUM>) and
during the second mode there is a second dependency of the relative position of the virtual sound object (<NUM>) from the notional listener (<NUM>) upon a currently determined relative position of the virtual sound object (<NUM>) from the notional listener (<NUM>) in the virtual space (<NUM>,<NUM>) based on the current measured location of the user (<NUM>) in the real space (<NUM>),
wherein the first dependency is a higher dependency than the second dependency such that a magnitude of a differential of the relative position of the virtual sound object from the notional listener with respect to the currently determined relative position of the virtual sound object (<NUM>) from the notional listener (<NUM>) in the virtual space is larger in the first mode than in the second mode.