Patent Description:
Spatial audio allows a user to control where a sound source is rendered from. In some circumstances it is desirable for a user to add an audio source to displayed content.

<CIT> discloses an apparatus comprising means for: displaying content within a viewport; displaying a symbol representing a sound source within the viewport; detecting one or more gestures that place the displayed symbol in the displayed content; allowing a first placement method, performed for the symbol, that locks the sound source to a position that moves with displayed content moving within the viewport. <CIT> discloses rendering sound sources in a VR environment. Sound sources may be locked either to the content or to the viewport.

According to the invention there is provided an apparatus comprising means for:.

In some but not necessarily all examples, the disambiguating gesture of the first placement method comprises dragging the symbol relative to the viewport.

In some but not necessarily all examples, the first placement method comprises a prior gesture, before the disambiguating gesture, dragging the symbol relative to the viewport and the displayed content.

In some but not necessarily all examples, the apparatus comprises means for: determining a first position in the viewport defined by an end-point of the prior gesture dragging the symbol relative to the viewport and the displayed content; and determining a second position in the viewport defined by termination of respective first placement method or the second placement method; wherein a difference between the first position and the second position disambiguates the first placement method and the second placement method.

In some but not necessarily all examples, the first placement method comprises a placement method in which the symbol is dragged to content and is then dragged with content and the second placement method comprises a placement method in which the symbol is dragged to a position within the viewport and is then held at the position within the viewport.

In some but not necessarily all examples, the first placement method and the second placement method comprise: selection of the symbol using a gesture.

In some but not necessarily all examples, the first placement method and the second placement method comprise an initial gesture for the symbol relative to the viewport and the displayed content indicating a movement of the symbol relative to the viewport and the displayed content,
the first placement method additionally comprising:
a gesture relative to the viewport, for the symbol, indicating movement of the symbol relative to the viewport that disambiguates the first placement method from the second placement method.

In some but not necessarily all examples, the first placement method comprises an initial drag movement of the symbol relative to the viewport and the displayed content and a subsequent movement of the symbol relative to the viewport but without movement of the symbol relative to the displayed content.

In some but not necessarily all examples, the first placement method comprises an initial drag movement of the symbol relative to the viewport and the displayed content and a subsequent movement of the symbol relative to the viewport to a variable position that tracks the displayed content that is moving within the viewport to which the sound source is to be locked.

In some but not necessarily all examples, the displayed content that is tracked and to which the sound source is to be locked is first content of a visual scene.

In some but not necessarily all examples, the first content is a majority of the visual scene that forms a background.

In some but not necessarily all examples, the first content is a minority of the visual scene being a visual object that moves within the visual scene.

In some but not necessarily all examples, the content displayed within the viewport is a sub-set of available content, wherein the sub-set is defined by the viewport, and wherein the viewport is defined by a point of view and a field of view, wherein one or both of the displayed content and the point of view are time-variable.

In some but not necessarily all examples, the first placement method comprises a holding gesture that holds the displayed symbol at a fixed position relative to the content while the viewport moves and wherein the second placement method comprises a holding gesture that holds the displayed symbol at a fixed position relative to the viewport while the viewport moves.

In some but not necessarily all examples, the second placement method performed on the symbol locks the sound source to a position that tracks with the viewport, not displayed content that is moving relative to the viewport.

In some but not necessarily all examples, the second placement method comprises an initial drag movement of the symbol relative to the viewport and the displayed content and no subsequent movement of the symbol relative to the viewport for a threshold time period.

In some but not necessarily all examples, the viewport is a touch display screen and the symbol is a displayed icon on the touch display screen and wherein the one or more gestures that place the displayed symbol in the displayed content are touch-based gestures.

In some but not necessarily all examples, the icon is provided in a menu displayed on the touch display screen.

In some but not necessarily all examples, the sound source and the content are recorded contemporaneously.

According to the invention there is provided a method comprising:.

According to the invention there is provided a computer program comprising program instructions for causing an apparatus to perform at least the following:.

According to an example useful for understanding the invention there is provided a man-machine interface comprising means for:.

<FIG> illustrates an example of an apparatus <NUM> for controlling an addition of a sound source <NUM> to displayed content <NUM>.

The apparatus <NUM> comprises means for:.

The first placement method comprises a gesture <NUM> relative to the viewport <NUM>, for the symbol <NUM>, indicating movement of the symbol <NUM> relative to the viewport <NUM> that disambiguates the first placement method from the second placement method.

In the example illustrated, the apparatus <NUM> comprises.

The display means <NUM> can be any device for controlling or providing a visual output to a user. In some but not necessarily all examples, the apparatus <NUM> is a hand-portable apparatus and the display means <NUM> is a display screen of the apparatus <NUM>. In some but not necessarily all examples, the display means <NUM> projects light into the eye of the user, for example via a near eye display or a head-mounted display screen or is an output interface for controlling the projection of light into the eye of the user or the head-mounted display screen.

The detecting means <NUM> can be any device for detecting one or more gestures <NUM>. A gesture can be any movement of the user body. In some examples, the one or more gestures are touch gestures in which a user touches a touch sensitive screen, for example a touch sensitive display screen, using a finger of pointing device. In some examples, the one or more gestures are three dimensional gestures in which a user moves his or her body relative to the apparatus <NUM>. Three-dimensional gestures can be detected in a variety of ways, for example, using multiple detectors to detect spatial variations in projected electric fields, ultrasound fields, or light fields.

Spatial audio means <NUM> is any device capable of controlling a perceived position of a sound source when rendered to a user. The sound source can be modelled as a virtual loudspeaker that is positioned by controlling, for example, timing and amplitude differences between the sound as perceived at a user's left ear and right ear. Reverberation can also be used to externalise the sound outside of a user wearing headphones. The sound can be rendered using a suitable multi-loudspeaker set-up such as a surround sound loudspeaker or using headphones.

The viewport <NUM> is the display area visible to a user. Where the apparatus <NUM> uses a display screen as display means <NUM>, the viewport <NUM> is the display screen.

The displayed content <NUM> is what is displayed in the display area. In some, but not necessarily all examples, the content is video content. In some, but not necessarily all examples, the content displayed in the viewport <NUM> at a particular time is a sub-set of the content that is available to be displayed in the viewport <NUM> at a particular time. For example, the field of view of available content can be greater than a field of view of displayed content. This can for example occur if the viewport displays only part of a panoramic image or if the viewport <NUM> display mediated reality content. In first-person-perspective mediated reality, the user's real point of view (location and/or orientation) determines the point of view within a virtual space of a virtual user, changing a virtual scene (the displayed content). In some examples, the virtual space may extend horizontally up to <NUM>° and may extend vertically up to <NUM>°.

In some, but not necessarily all examples, the sound source <NUM> and the content are recorded contemporaneously.

The apparatus <NUM> in some examples is configured to enable adding of spatial audio effects (the sound source <NUM>) to captured content, for example captured videos.

The displayed content can, in some examples, have associated sound that is rendered when the content is displayed.

In some but not necessarily all examples, the apparatus <NUM> is a rendering apparatus. It can for example, be used for editing the content to add spatial audio effects.

<FIG> illustrate an example of a viewport <NUM>. Content <NUM> is displayed within the viewport <NUM>. The content <NUM> is moving within the viewport <NUM>. In these examples but not necessarily all examples, the content is stationary in <FIG>, then it moves to the left (relative to the viewport <NUM>) in <FIG>, then it moves to the right (relative to the viewport <NUM>) in <FIG>, then it moves to the left (relative to the viewport <NUM>) in <FIG>. The <FIG> are used to illustrate an example of the first placement method <NUM>.

In <FIG>, a symbol <NUM> representing a sound source <NUM> is displayed within the viewport <NUM>. A user <NUM> performs a selection gesture <NUM> for the symbol <NUM>.

<FIG> illustrates a user <NUM> performing a placement gesture <NUM>. This gesture <NUM> is relative to the viewport <NUM> and the displayed content <NUM> and indicates a movement of the symbol <NUM> relative to the viewport <NUM> and the displayed content <NUM>.

For example, the placement gesture <NUM> can be a drag movement of the symbol <NUM> relative to the viewport <NUM> and the displayed content <NUM> to a symbol position <NUM> in the viewport <NUM> (and the displayed content <NUM>).

<FIG> illustrate a user <NUM> continuing the placement gesture <NUM>. This continuing placement gesture <NUM> comprises movement of the symbol relative to the viewport <NUM> but there is not substantial movement of the symbol <NUM> relative to the displayed content <NUM>. The symbol position <NUM> varies with respect to the viewport <NUM> but does not vary or vary significantly with respect to the content <NUM>. The symbol position <NUM> follows the same portion of content <NUM>.

The symbol <NUM> has a variable position <NUM> that tracks the displayed content <NUM> that is moving within the viewport <NUM> to which the sound source <NUM> is to be locked.

For example, the continuing placement gesture <NUM> can be a drag movement of the symbol <NUM> relative to the viewport <NUM> that follows the displayed content <NUM>. There is not significant movement relative to the displayed content <NUM>.

The displayed content <NUM> that is tracked and to which the sound source <NUM> is to be locked is first content of a visual scene.

<FIG> illustrate that, after the placement gesture <NUM> terminates, the sound source <NUM> is locked to a position <NUM> that moves with displayed content <NUM> moving within the viewport <NUM>. The term 'locked' means that the sound source <NUM> is fixed to a position <NUM> without a requirement for continuing the placement gesture <NUM>. The locking is not necessarily permanent and can be time-limited, for example.

Although the sound source <NUM> is illustrated as a visual element in the viewport <NUM>, this is optional. In some examples the position <NUM> of the sound source <NUM> is not visually marked. In some examples, where the position <NUM> of the sound source <NUM> is visually marked using a visual element, the visual element can be the symbol <NUM> or a modified version of the symbol. In some examples, the user can make a user selection that causes the positions of one or more sound sources <NUM> to be visually indicated using a symbol <NUM>. The placement methods <NUM>, <NUM> can then be performed for any of those displayed symbols <NUM>.

<FIG> illustrate an example of a viewport <NUM>. Content <NUM> is displayed within the viewport <NUM>. The content <NUM> is moving within the viewport <NUM>. In these examples but not necessarily all examples, the content is stationary in <FIG>, then it moves to the left (relative to the viewport <NUM>) in <FIG>, then it moves to the right (relative to the viewport <NUM>) in <FIG>. The <FIG> are used to illustrate an example of the second placement method <NUM>.

To facilitate comparison of the first placement method <NUM> and the second placement method <NUM>, the same viewport <NUM>, same moving displayed content <NUM>, and same symbol <NUM> are used. However, it should be appreciated that this is only for the purpose of easy comparison and one or more of these features may differ between the methods.

<FIG> illustrates a user <NUM> performing a placement gesture <NUM>. This gesture is relative to the viewport <NUM> and the displayed content <NUM> and indicates a movement of the symbol <NUM> relative to the viewport <NUM> and the displayed content <NUM>.

For example, the placement gesture <NUM> can be a drag movement of the symbol relative to the viewport <NUM> and the displayed content <NUM> to a symbol position <NUM> in the viewport <NUM> (and the displayed content <NUM>).

<FIG> illustrates a user <NUM> continuing the placement gesture <NUM>. This continuing placement gesture <NUM> comprises movement of the symbol relative to the displayed content <NUM> but there is not substantial movement of the symbol <NUM> relative to the viewport <NUM>. The symbol position <NUM> varies with respect to the displayed content <NUM> but does not vary or vary significantly with respect to the viewport <NUM>.

The symbol <NUM> has a stationary position <NUM> within the viewport <NUM> to which the sound source <NUM> is to be locked. That position <NUM> does not track the displayed content <NUM> that is moving within the viewport <NUM>.

For example, the continuing placement gesture <NUM> can be a stationary gesture without drag movement of the symbol <NUM> relative to the viewport <NUM>. There is not significant movement relative to the viewport <NUM>.

The placement gesture <NUM> comprises an initial drag movement of the symbol <NUM> relative to the viewport <NUM> and the displayed content <NUM> (<FIG>) and no subsequent movement of the symbol relative to the viewport <NUM> for a threshold time period (<FIG>). The threshold period of time can, for example, be <NUM> or <NUM> seconds.

<FIG> illustrate that, after the placement gesture <NUM> terminates, the sound source <NUM> is locked to a position <NUM> that does not move with displayed content <NUM> moving within the viewport <NUM>. The term 'locked' means that the sound source <NUM> is fixed to a position <NUM> without a requirement for continuing the placement gesture <NUM>. The locking is not necessarily permanent and can be time-limited, for example.

Comparing the first placement method <NUM> and the second placement method <NUM>, the first placement method <NUM> additionally comprises a gesture <NUM> relative to the viewport <NUM>, for the symbol <NUM>, indicating movement of the symbol <NUM> relative to the viewport <NUM>. This gesture <NUM> (<FIG>) disambiguates the first placement method <NUM> from the second placement method <NUM>.

The first placement method <NUM>, in at least some examples, comprises placement method in which the symbol <NUM> is dragged to a position <NUM> within the displayed content <NUM> (C in <FIG>) and is then dragged with that content (<FIG>).

The second placement method <NUM>, in at least some examples, comprises a placement method in which the symbol <NUM> is dragged to a position <NUM> within the viewport <NUM> and is then held at that position <NUM> within the viewport <NUM> (<FIG>).

The disambiguation process can, for example comprise:.

In the first placement method <NUM>, the second position <NUM> is separated from the first position <NUM> in the viewport <NUM> by dragging the symbol <NUM> relative to the viewport <NUM>. In the second placement method <NUM>, the second position <NUM> is separated from the first position <NUM> in the content space by holding the symbol <NUM> in a substantially fixed position relative to the viewport <NUM>.

Consequently, a small position difference in content space (or a large position difference in the viewport space) indicates the first placement method <NUM> has occurred and a large position difference in content space (or a small position difference in the viewport space) indicates the second placement method <NUM> has occurred.

In some but not necessarily all examples, the drag gestures <NUM> are performed as movement of a contact point on a surface of a touch screen display. In these examples, the termination of a placement method <NUM>, <NUM> occurs when contact is terminated by lifting off the user's finger or the pointing device.

Where the viewport <NUM> is a touch display screen, the symbol <NUM> can be a displayed icon on the touch display screen. The one or more gestures <NUM> that place the displayed symbol <NUM> in the displayed content are touch-based gestures performed on the icon. The icon can be provided in a menu displayed on the touch display screen as part of a man machine interface for adding a spatial audio effect to visual content.

In some but not necessarily all examples, the displayed content <NUM> that is displayed within the viewport <NUM> is a sub-set of available content.

As illustrated in <FIG>, the sub-set is defined by the viewport <NUM>.

Each of <FIG> corresponds, respectively, with <FIG>. Each of <FIG> corresponds, respectively, with <FIG>.

<FIG> illustrate a relationship of the viewport <NUM> to the available content ABCDE. The displayed content <NUM> is that part of the available content that overlaps with the viewport <NUM>. The viewport <NUM> moves relative to the available content changing the displayed content <NUM>.

In the examples 4A to 4C, which correspond to the first placement method <NUM>, a position <NUM> of the gesture controlled symbol <NUM> remains static relative to a particular portion (C) of the displayed content <NUM> and moves within the viewport <NUM> in an opposite sense and to a same extent as how the viewport moves relative to the available content <NUM>. Detection of this placement method <NUM>, causes a position <NUM> of the sound source <NUM> to be locked to the particular portion (C) of the displayed content <NUM> defined by the placement method <NUM>. In the examples 4D to 4F, a position <NUM> of the sound source <NUM> remains static relative to a particular portion (C) of the displayed content <NUM> and moves within the viewport <NUM> in a opposite sense and to a same extent as how the viewport moves relative to the available content <NUM>.

The first placement method <NUM> locks the sound source <NUM> to a position <NUM> that moves with displayed content <NUM> not the viewport <NUM>.

In this example, the first placement method <NUM> comprises a holding gesture <NUM> that holds the displayed symbol 40at a fixed position <NUM> relative to the content while the viewport <NUM> moves.

In the examples 5A to 5C, which correspond to the second placement method <NUM>, a position <NUM> of the gesture-controlled symbol <NUM> remains static relative to a particular portion (X) of the viewport <NUM> and moves with the viewport <NUM>. The position <NUM> of the gesture-controlled symbol <NUM> moves within the displayed content <NUM> in a same sense and to a same extent as how the viewport moves relative to the available content <NUM>. Detection of this placement method <NUM>, causes a position <NUM> of the sound source <NUM> to be locked to the particular portion (X) of the viewport defined by the placement method <NUM>. In the examples 5D to 5F, a position <NUM> of the sound source <NUM> remains static relative to a particular portion (X) of the viewport <NUM> and moves within the displayed content <NUM> in the same sense and to the same extent as how the viewport moves relative to the available content <NUM>.

The second placement method <NUM> locks the sound source <NUM> to a position <NUM> that moves with the viewport <NUM> not the displayed content <NUM>. The sound source <NUM> tracks with the viewport <NUM>, not displayed content <NUM> that is moving relative to the viewport <NUM>.

In this example, the second placement method <NUM> comprises a holding gesture <NUM> that holds the displayed symbol 40at a fixed position <NUM> relative to the viewport while the viewport moves.

<FIG> and <FIG> illustrate examples in which the viewport <NUM> is a touch display screen of a handheld apparatus.

<FIG> illustrate an example of the first placement method <NUM> when the viewport <NUM> is a touch display screen of a handheld apparatus.

In <FIG>, a symbol <NUM> representing a sound source <NUM> is displayed within the viewport <NUM>. A user <NUM> performs placement gestures <NUM>, and optionally selection gestures <NUM>, on the symbol <NUM>. The user touches the symbol <NUM> with a finger or pointer and drags it to a desired position within the displayed content <NUM> of the viewport <NUM>.

In <FIG>, the user <NUM> continues the placement gesture <NUM>. The user continues to touch the symbol <NUM> with the finger or pointer and drags across the viewport <NUM>. This continuing placement gesture <NUM> comprises movement of the symbol relative to the viewport <NUM> but there is not substantial movement of the symbol <NUM> relative to the displayed content <NUM>. The symbol position varies with respect to the viewport <NUM> but does not vary or vary significantly with respect to the content <NUM>. The symbol position <NUM> follows the same portion of content <NUM>.

<FIG> illustrate an example of the second placement method <NUM> when the viewport <NUM> is a touch display screen of a handheld apparatus.

In <FIG>, the user <NUM> continues the placement gesture <NUM>. The user continues to touch the symbol <NUM> with the finger or pointer while keeping the finger or pointer substantially stationary with respect to the viewport <NUM>. This continuing placement gesture <NUM> comprises movement of the symbol relative to the displayed content <NUM> but there is not substantial movement of the symbol <NUM> relative to the viewport <NUM>. The symbol position varies with respect to the displayed content <NUM> but does not vary or vary significantly with respect to the viewport <NUM>.

<FIG> illustrate examples of how the viewport <NUM> can move relative to the available content changing the displayed content <NUM>.

In these examples, the viewport is defined by a point of view <NUM> and a field of view <NUM>.

One or both of the displayed content and the point of view <NUM> are time-variable.

In the example illustrated in <FIG>, the point of view <NUM> is variable by a change in location. This corresponds to a tracking shot of the available content.

In the example illustrated in <FIG>, the point of view <NUM> is variable by a change in orientation. This corresponds to a panning shot of the available content.

It will be appreciated that in these examples, the second placement method <NUM> locks the sound source <NUM> to the point of view <NUM>, such that a position <NUM> of the sound source <NUM> relative to the point of view <NUM> is invariant as the point of view <NUM> changes. The second placement method <NUM> locks the sound source <NUM> to a position <NUM> that moves with the viewport <NUM> not the displayed content <NUM>. The first placement method <NUM> locks the sound source <NUM> to the same displayed content as the point of view <NUM> changes. The first placement method <NUM> locks the sound source <NUM> to a position <NUM> that moves with displayed content <NUM> not the viewport <NUM>.

<FIG> illustrate some examples of how the point of view <NUM> can be varied.

<FIG> illustrates an example in which the display <NUM> used by the apparatus <NUM> is part of a handheld apparatus <NUM>.

The handheld apparatus <NUM> comprises a display <NUM> that is used for displaying the displayed content <NUM> and the displayed symbol <NUM> to the user <NUM>. The handheld apparatus <NUM> may be moved deliberately in the hands of a user <NUM> in one or more of the six degrees of freedom (three for orientation and three for translation) to define different points of view <NUM>.

If the handheld apparatus has a camera mounted on a face opposite the display, it may be operated as a see-video arrangement that enables a live real scene to be viewed while one or more visual elements, such as the symbol <NUM>, are displayed to the user to provide in combination a virtual scene.

The handheld apparatus may be or may be operated as a see-video arrangement for mediated reality that enables a live or recorded video of a real scene or other content to be displayed on the display <NUM> for viewing by the user while one or more visual elements, such as the symbol <NUM>, are simultaneously displayed on the display <NUM> for viewing by the user. The combination of the displayed real scene or other content and displayed one or more visual elements provides a virtual scene to the user.

In some but not necessarily all examples, the apparatus <NUM> is an image capturing apparatus, for, example a camera. The displayed content <NUM> can be content that is simultaneously displayed in the camera viewfinder and recorded as captured content. It can for example, be used for editing the content, as it is being captured, to add spatial audio effects. In this example, the content-locked sound source <NUM> comes from the same position in the world even when the camera is being panned. In this example, the viewport-locked sound source <NUM> follows the movement of the camera and its viewport. The sound source <NUM> when rendered comes from the same direction regardless of where the camera is pointing.

<FIG> illustrates an example in which the display <NUM> used by the apparatus <NUM> is head-mounted apparatus <NUM>.

The head-mounted apparatus <NUM> comprises a display <NUM> that is used for displaying the displayed content <NUM> and the displayed symbol <NUM> to the user <NUM>. The head-mounted apparatus may be moved automatically when a head of the user moves. The head-mounted apparatus <NUM> may be moved in one or more of the six degrees of freedom (three for orientation and three for translation) to define different points of view <NUM>.

The head-mounted apparatus <NUM> may be operated as a see-through arrangement for augmented reality that enables a live real scene to be viewed while one or more visual elements, such as the symbol <NUM>, are displayed by the display <NUM> to the user to provide in combination the virtual scene. In this case a visor, if present, is transparent or semi-transparent so that the live real scene can be viewed through the visor.

The head-mounted apparatus <NUM> may be operated as a see-video arrangement for mediated reality that enables a live or recorded video of a real scene or other content to be displayed by the display <NUM> for viewing by the user while one or more visual elements, such as the symbol <NUM>, are simultaneously displayed by the display for viewing by the user. The combination of the displayed scene and displayed one or more visual elements provides a virtual scene to the user. In this case a visor is opaque and may be used as display <NUM>.

Mediated reality means the rendering of mediated reality for the purposes of achieving mediated reality for a remote user, for example augmented reality or virtual reality. The mediated reality may support one or more of 3DoF, 3DoF+ or 6DoF.

A user in a real space has a real point of view defined by a location and an orientation. The location is a three-dimensional location and the orientation is a three-dimensional orientation.

In an example of 3DoF mediated reality, the user's real point of view (orientation) determines a virtual point of view <NUM>, within a virtual space, of a virtual user. An orientation of the user controls a virtual orientation of a virtual user. There is a correspondence between the real orientation of the user and the virtual orientation of the virtual user such that a change in the real orientation produces the same change in the virtual orientation.

The virtual orientation of the virtual user in combination with a virtual field of view defines a virtual visual scene within the virtual visual space. The virtual visual scene is presented to the user as the displayed content <NUM>. In some examples, the virtual visual scene can additionally define a virtual sound scene.

A virtual visual scene is that part of the virtual visual space (the available content) that is displayed to a user via the displayed content <NUM>. A virtual sound scene is that part of a virtual sound space that is rendered to a user. The virtual sound space and the virtual visual space correspond and form a combined virtual space. The virtual sound space and the virtual visual space correspond in that a point of view within the virtual sound space has an equivalent point of view within the virtual visual space. In 3DoF mediated reality, a change in the location of the user does not change the virtual location or virtual orientation of the virtual user.

In the example of 6DoF and 3DoF+ mediated reality, the user's real point of view (location and/or orientation) determines the virtual point of view (location and/or orientation) within the virtual space of a virtual user. The situation is as described for 3DoF and in addition it is possible to change the rendered virtual sound scene and the displayed virtual visual scene by movement of a location of the user. For example, there may be a mapping between the location of the user and the virtual location of the virtual user. A change in the location of the user produces a corresponding change in the virtual location of the virtual user. A change in the virtual location of the virtual user changes the rendered sound scene and also changes the rendered visual scene. In 6DoF, the change in location may arise from a postural change of the user and/or a translation of the user by walking or otherwise. In 3DoF+, the change in location may arise from a postural change of the user.

In some situations, it may be desirable for a portion of the virtual sound space to remain fixed in the virtual space when the virtual point of view of the virtual user changes.

Viewport-locked sounds, defined by placement method <NUM>, are fixed relative to the virtual point of view <NUM> of the virtual user so that when the user changes their point of view <NUM> the positions of the sound sources <NUM> relative to the user do not change.

Content-locked sounds, defined by placement method <NUM>, are fixed in space so that when the user changes their point of view <NUM>, the positions <NUM> of the sound sources <NUM> relative to the user also change.

<FIG> illustrate, in detail, examples of how the apparatus <NUM> enables a user <NUM> to indicate whether a sound source <NUM> (not illustrated), represented by symbol <NUM>, added into a virtual sound space is viewport-locked or content-locked.

<FIG> and <FIG> illustrate the virtual user who has a point of view <NUM> within a virtual space that comprises sound sources <NUM>. As previously described, in first-person perspective mediated reality, there is a correspondence between the virtual user and the user and the point of view of the virtual user and the point of view of the user. The user may vary the point of view <NUM> by changing an orientation of the user's head when wearing a head-mounted apparatus <NUM>. This changes the virtual point of view <NUM>.

<FIG> illustrate displayed content <NUM> within a viewport <NUM>. The displayed content <NUM> corresponds to the virtual visual scene defined by the virtual point of view <NUM> in respective <FIG>.

In the example of <FIG>, gesture <NUM> defines a sound source <NUM> to be added to a virtual sound space of the virtual space as a content-locked sound.

In <FIG>, the user <NUM> performs an initial gesture <NUM> to move the symbol <NUM> representing a sound source <NUM> to the desired position <NUM> in the viewport <NUM>. In the example of <FIG>, the user grabs the symbol <NUM> and drags it to the position <NUM> that corresponds to virtual object <NUM> in the displayed content <NUM>.

In <FIG>, the virtual point of view <NUM> of the virtual user changes, changing the displayed content <NUM> visible in the viewport <NUM>. The gesture <NUM> moves relative to the virtual point of view <NUM>. The symbol <NUM> is held static relative to the virtual object <NUM> by the gesture <NUM>.

In <FIG>, the gesture is terminated. The sound source <NUM> is then locked to the displayed content <NUM>. In the example of <FIG>, the sound source <NUM> is locked to the virtual object <NUM>.

In the example of <FIG>, gesture <NUM> defines a sound source <NUM> to be added to a virtual sound space of the virtual space as a viewport-locked sound.

In <FIG>, the user performs an initial gesture <NUM> to move the symbol <NUM> representing a sound source <NUM> to the desired position <NUM> in the viewport <NUM>. In the example of <FIG>, the user grabs the symbol <NUM> and drags it to the position <NUM> in the viewport <NUM>. For purposes of comparison, this position corresponds to virtual object <NUM> in the displayed content <NUM>.

In <FIG>, the virtual point of view <NUM> of the virtual user changes, changing the displayed content <NUM> visible in the viewport <NUM>. The gesture <NUM> moves with the virtual point of view <NUM>, that is with the viewport <NUM> and not with the displayed content <NUM>. The symbol <NUM> is held static relative to the viewport <NUM>/point of view <NUM> by the gesture <NUM>.

In <FIG>, the gesture is terminated. The sound source <NUM> is then locked to the point of view <NUM> of the user and the viewport <NUM>.

Sound sources <NUM> can be content-locked or viewport-locked. A content-locked sound source <NUM> has a position that moves with displayed content <NUM>. It can, for example, shift relative to the user along with a shift in a visual scene relative to the user or can, for example follow an object or other portion of displayed content that moves within the displayed content. A viewport-locked sound has a position that does not move with displayed content moving within the viewport but remains static relative to the viewport and user. It can, for example, remain stationary relative to a user with a shift in visual scene relative to the user.

The sounds rendered to a user can comprise only content-locked sounds, only viewport-locked sounds or a combination of content-locked sounds and viewport-locked sounds.

The user can use the described methods and apparatus to control whether a sound will be rendered as a content-locked sound or a viewport-locked sound.

A content-locked sound source <NUM> produced by placement method <NUM> and a viewport-locked sound source <NUM> produced by placement method <NUM> can be rendered to a user using the same or different audio output device. Examples of audio output devices include headphones and loudspeakers.

When a content-locked sound is rendered using loudspeakers, as the real orientation of the user changes, the sound source <NUM> remains static relative to the loudspeakers when the point of view of the virtual user changes.

When a content-locked sound is rendered using headphones the position and/or orientation of the sound source <NUM> needs to change relative to the headphones by the same amount in the opposite sense to the change in the virtual point of view of the virtual user. The position and/or orientation of the sound source <NUM> tracks opposite to the change in virtual point of view <NUM> of the virtual user so that the position and/or orientation of the sound source <NUM> remains fixed in the virtual space and does not move with the virtual point of view <NUM> of the virtual user.

When a viewport-locked sound is rendered using headphones, the sound source <NUM> remains static relative to the headphones when the point of view of the virtual user changes
When a viewport-locked sound is rendered using loudspeakers the position and/or orientation of the sound source <NUM> needs to change relative to the loudspeakers by the same amount in the same sense to the change in the virtual point of view of the virtual user. The position and/or orientation of the sound source <NUM> tracks the change in virtual point of view <NUM> of the virtual user so that the position and/or orientation of the sound source <NUM> remains fixed in the real space and moves with the virtual point of view <NUM> of the virtual user.

Channel-based audio, for example, surround sound (e.g. <NUM>, <NUM> or <NUM> surround sound) or binaural audio, can be used or scene-based audio, including spatial information about a sound field and sound sources, can be used.

Audio content may encode spatial audio as audio objects. Examples include but are not limited to MPEG-<NUM> and MPEG SAOC. MPEG SAOC is an example of metadata-assisted spatial audio.

Audio content may encode spatial audio as audio objects in the form of moving virtual loudspeakers.

Audio content may encode spatial audio as audio signals with parametric side information or metadata. The audio signals can be, for example, First Order Ambisonics (FOA) or its special case B-format, Higher Order Ambisonics (HOA) signals or mid-side stereo. For such audio signals, synthesis which utilizes the audio signals and the parametric metadata is used to synthesize the audio scene so that a desired spatial perception is created.

The parametric metadata may be produced by different techniques. For example, Nokia's spatial audio capture (OZO Audio) or Directional Audio Coding (DirAC) can be used. Both capture a sound field and represent it using parametric metadata. The parametric metadata may for example comprise: direction parameters that indicate direction per frequency band; distance parameters that indicate distance per frequency band; energy-split parameters that indicate diffuse-to-total energy ratio per frequency band. Each time-frequency tile may be treated as a sound source with the direction parameter controlling vector based amplitude panning for a direct version and the energy-split parameter controlling differential gain for an indirect (decorrelated) version.

3GPP IVAS (3GPP, Immersive Voice and Audio services), which currently under development, is expected to support new immersive voice and audio services, for example, mediated reality.

In some but not necessarily all examples amplitude panning techniques may be used to create or position a sound object. For example, the known method of vector-based amplitude panning (VBAP) can be used to position a sound source.

A sound object may be re-positioned by mixing a portion of a direct form of the object (an attenuated and directionally-filtered direct sound) with an indirect form of the object (e.g. positioned directional early reflections and/or diffuse reverberant).

<FIG> illustrates an example of a method <NUM> comprising:.

<FIG> illustrate an alternative example method of indicating whether a sound source <NUM> to be added to the displayed content <NUM> is a content-locked or viewport-locked.

In <FIG>, the user performs an initial gesture <NUM> to move the symbol <NUM> representing the sound source <NUM> to a desired position <NUM> in the viewport <NUM>.

In <FIG>, a prompt <NUM> is displayed in the viewport <NUM> inviting the user to make an input indicating whether the sound source <NUM> to be added to the displayed content <NUM> is a content-locked sound or a viewport-locked sound. In the example of <FIG>, the prompt is in the form of selectable icons <NUM> positioned close to the desired position <NUM> of the sound source <NUM> within the displayed content <NUM>.

In <FIG>, the user makes an input <NUM> indicating whether the sound source <NUM> is a content-locked sound or a viewport-locked sound. In the example of <FIG>, the user input is made by selecting one of the selectable icons <NUM> displayed by the prompt <NUM>. The sound source <NUM> is then defined as content-locked or viewport-locked depending on the input <NUM>.

The method illustrated in <FIG> is particularly useful when the displayed content <NUM> is static in the viewport <NUM>. In this circumstance it is not possible to disambiguate between the first and second placement methods <NUM>, <NUM> without further input. Holding the symbol <NUM> static relative to the displayed content <NUM> within the viewport <NUM> (placement method <NUM>) and holding the symbol <NUM> static relative to the viewport <NUM> (placement method <NUM>) cannot be disambiguated without further input when all the displayed content <NUM> is not moving relative to the viewport <NUM>. This is because following a visual object while the video is static is the same as holding still. The method illustrated in <FIG> provides the further input that enable disambiguation. It prompts the user to provide explicit disambiguation by selecting which effect (viewport-locked sound or content-locked sound) the user wishes to use for the added sound source <NUM>.

<FIG> illustrates a method <NUM> for determining whether or not to provide the prompt <NUM>.

At block <NUM> it is determined whether or not the displayed content <NUM> is static for the remaining time duration. If the displayed content <NUM> is static for the remaining time duration, the method moves to block <NUM>. If the displayed content <NUM> is not static for the remaining time duration, the method moves to block <NUM>.

At block <NUM>, the sound source <NUM> is added at the desired position <NUM> (content-locked and viewport-locked are the same).

At block <NUM>, it is determined whether or not the displayed content <NUM> is static at the current time.

If it is determined that the displayed content <NUM> is static at the current time, the prompt <NUM> is generated at block <NUM>.

If it is determined that the displayed content <NUM> is not static at the current time, then disambiguation as previously described can occur.

While the method <NUM> is described as occurring before the process of disambiguation, in other examples it could occur after the process of disambiguation, should it fail.

<FIG> illustrates an example of a controller <NUM>. Such a controller may be used to control performance of any or all of the functions of the apparatus <NUM> and/or perform any of the previously described methods.

Implementation of a controller <NUM> may be as controller circuitry. The controller <NUM> may be implemented in hardware alone, have certain aspects in software including firmware alone or can be a combination of hardware and software (including firmware).

The memory <NUM> stores a computer program <NUM> comprising computer program instructions (computer program code) that controls the operation of the apparatus <NUM> when loaded into the processor <NUM>. The computer program instructions, of the computer program <NUM>, provide the logic and routines that enables the apparatus to perform the methods illustrated in <FIG>. The processor <NUM> by reading the memory <NUM> is able to load and execute the computer program <NUM>.

The apparatus <NUM> can therefore comprises computer program means for.

As illustrated in <FIG>, the computer program <NUM> may arrive at the apparatus <NUM> via any suitable delivery mechanism <NUM>. The delivery mechanism <NUM> may be, for example, a machine readable medium, a computer-readable medium, a non-transitory computer-readable storage medium, a computer program product, a memory device, a record medium such as a Compact Disc Read-Only Memory (CD-ROM) or a Digital Versatile Disc (DVD) or a solid state memory, an article of manufacture that comprises or tangibly embodies the computer program <NUM>. The delivery mechanism may be a signal configured to reliably transfer the computer program <NUM>. The apparatus <NUM> may propagate or transmit the computer program <NUM> as a computer data signal.

In at least some examples, the computer program instructions are configured to cause an apparatus to perform at least the following:.

The term circuitry also covers, for example and if applicable to the particular claim element, a baseband integrated circuit for a hand portable device or a similar integrated circuit in a server, a cellular network device, or other computing or network device.

The blocks illustrated in the FIGS may represent steps in a method and/or sections of code in the computer program <NUM>. The illustration of a particular order to the blocks does not necessarily imply that there is a required or preferred order for the blocks and the order and arrangement of the block may be varied. Furthermore, it may be possible for some blocks to be omitted.

The above described examples find application as enabling components of: automotive systems; telecommunication systems; electronic systems including consumer electronic products; distributed computing systems; media systems for generating or rendering media content including audio, visual and audio visual content and mixed, mediated, virtual and/or augmented reality; personal systems including personal health systems or personal fitness systems; navigation systems; user interfaces also known as human machine interfaces; networks including cellular, non-cellular, and optical networks; ad-hoc networks; the internet; the internet of things; virtualized networks; and related software and services.

Although embodiments have been described in the preceding paragraphs with reference to various examples, it should be appreciated that modifications to the examples given can be made without departing from the scope of the claims.

Claim 1:
An apparatus (<NUM>) comprising means for:
displaying content (<NUM>) within a viewport (<NUM>);
displaying a symbol (<NUM>) representing a sound source (<NUM>) within the viewport (<NUM>);
detecting one or more gestures (<NUM>) that place the displayed symbol (<NUM>) in the displayed content (<NUM>);
disambiguating between a first placement method, performed for the symbol (<NUM>), that locks the sound source (<NUM>) to a position that moves with displayed content (<NUM>) moving within the viewport (<NUM>) and a second placement method performed for the symbol (<NUM>) that is different to the first placement method and locks the sound source (<NUM>) to a position that does not move with displayed content (<NUM>) moving within the viewport (<NUM>),
wherein the first placement method comprises a disambiguating gesture (<NUM>) relative to the viewport (<NUM>), for the symbol (<NUM>), indicating movement of the symbol (<NUM>) relative to the viewport (<NUM>) that disambiguates the first placement method from the second placement method.