Patent Description:
There exist apparatuses that have a user interface for enabling a user to select audio content and that respond to selection of audio content by rendering the audio content or otherwise processing the audio content.

However, these user interfaces have various limitations, particularly where different audio content is associated with different orientations relative to a user.

<CIT> discloses a user interface for enabling a user to select audio content, via virtual reality techniques. A sound space is divided into a series of non-overlapping volumes, each containing sound objects ("rooms") that lead off a "lobby space". When a virtual user is outside a room, a simplified sound space is rendered to the user. A simplified sound object is also rendered on a front face of each room. If a user enters a room, the sound objects within that room are rendered to the user.

<CIT> discloses the rendering of visual augmented reality content on the display of a hand-held electronic device, with aural augmented reality content output to the user of the hand-held device, based on the user's location. The user's location may be different to the hand-held device's location.

<CIT> discloses an electronic device with a 3D positional audio function.

A media library is represented as a virtual space, through which a user may navigate via an avatar. Different songs or genres of music may be selected by navigating the avatar through the virtual space.

In particular, different aspects of the invention are presented in independent claims <NUM>, <NUM> and <NUM> and favourable embodiments are presented in dependent claims <NUM>-<NUM>.

In at least some but not necessarily all examples, at least one of the multiple user-selectable visual elements, at a position indicative of an orientation from a user, is configured to have an appearance that comprises a visual extract of visual content for the orientation that is dependent upon audio content for the orientation.

In at least some but not necessarily all examples the visual extract of visual content for the orientation comprises an image for the orientation, a looped clip of video for the orientation, or a repeated sequence of images for the orientation.

In at least some but not necessarily all examples the visual content for an orientation is visual content that would be rendered to a user using mediated reality when a user changes a real world point of view of the user so the real world point of view of the user corresponds to the orientation, wherein mediated reality comprises rendering content in dependence upon a real world point of view of a user.

In at least some but not necessarily all examples, if audio content for an orientation is changed without changing the orientation, there is a change in appearance of the user-selectable visual element at the position in the user interface indicative of the orientation.

In at least some but not necessarily all examples audio content for an orientation is audio content that would be rendered to a user using mediated reality when a user changes a real world point of view of the user so that the real world point of view of the user corresponds to the orientation, wherein mediated reality comprises rendering content in dependence upon a real world point of view of a user.

In at least some but not necessarily all examples, the apparatus comprises a head-mounted apparatus for rendering the user interface.

In at least some but not necessarily all examples, a first user-selectable visual element of the arrangement has a first position in the user interface indicative of a first orientation from the user and has a first appearance dependent upon first audio content for the first orientation, wherein a second user-selectable visual element of the arrangement has a second position in the user interface indicative of a second orientation from the user and has a second appearance dependent upon second audio content for the second orientation,
wherein the apparatus is configured to: respond to actuation of the first user-selectable visual element to select the first audio content but not the second audio content, and to respond to user selection of the second user-selectable visual element to select the second audio content but not the first audio content.

In at least some but not necessarily all examples, one or more characteristics of an appearance of a user-selectable visual element is indicative of one or more characteristics of audio content selectable via the user-selectable visual element.

In at least some but not necessarily all examples, the user-selectable visual elements of the arrangement comprises a first portion that is asymmetric and indicative of an orientation from a user and a second portion that has an appearance dependent upon audio content for the orientation.

In at least some but not necessarily all examples, the first portion represents a segment of a circle having characteristics of appearance including one or more of segment size, width and length.

In at least some but not necessarily all examples, the second portion has characteristics of appearance including one or more of:
size, brightness, coloration, blurred edges, image content.

In at least some but not necessarily all examples, the apparatus comprises means for: responding to selection of a user-selectable visual element at a position in the user interface indicative of an orientation from the user to adapt the arrangement of multiple user-selectable visual elements to include more user-selectable visual elements, at positions in the user interface that are indicative of orientations from a user and have an appearance dependent upon audio content for the respective orientations; and responding to selection of a user-selectable visual element at a position in the user interface <NUM> indicative of an orientation from the user to select audio content for the orientation.

The user interface is particularly useful when audio content is associated with to both a front and a rear of a user.

In the following examples, audio content <NUM> is associated with different orientations θ relative to a user <NUM>. For example, audio content <NUM> can be rendered from the orientation θ it is associated with.

Multiple user-selectable visual elements <NUM> are arranged, in an arrangement <NUM>, at positions p in the user interface <NUM>. The positions p in the user interface <NUM> are indicative of different orientations θ from the user <NUM>. In some examples the position p is defined by a variable orientation θ from a point O and, optionally, a variable distance r from the point O.

The user-selectable visual elements <NUM> for orientations θ have an appearance <NUM> dependent upon audio content <NUM> associated with the respective orientations θ.

User selection of a user-selectable visual element <NUM> at a position p in the user interface <NUM> indicative of an orientation θ from the user <NUM> causes selection of the audio content <NUM> associated with that orientation θ for processing,.

<FIG> illustrate examples of a user interface <NUM> for enabling a user to select audio content. The user interface <NUM> comprises an arrangement <NUM> of multiple user-selectable visual elements <NUM> at positions p in the user interface <NUM>. The positions p in the user interface <NUM> are indicative of different orientations θ from a user <NUM> (not illustrated in the <FIG>).

In this example, the user can be considered to have a position O in the user interface <NUM> and the angle θ is measured from O. In some examples the position p is defined by a variable orientation θ from a point O and, optionally, a variable distance r from the point O. In this illustrated example, the position p is defined by a variable orientation θ from a point O and a fixed distance r from the point O.

In these figures, other figures and in the description, some reference numerals have suffixes and some do not. Suffixes are used to indicate different instances within a class of similar items. When a specific instance of an item is referred to, it will generally be referred to using the reference numeral with a suffix. When the general class of items is referred to it will generally be referred to without a suffix. Thus, the user interface <NUM> comprises an arrangement <NUM> of multiple user-selectable visual elements <NUM> at respective positions p in the user interface <NUM>. The positions p in the user interface <NUM> are indicative of different orientations θ from a user <NUM>. Or, alternatively, the user interface <NUM> comprises an arrangement <NUM> of multiple user-selectable visual elements <NUM>i, <NUM>j, <NUM>k at respective positions pi, pj, pk in the user interface <NUM>. The positions pi, pj, pk in the user interface <NUM> are indicative of different orientations θi, θj, θk from a user <NUM>.

In the examples illustrated, but not necessarily all examples, the angle of θ is defined relative to a current point of view <NUM> of the user. Therefore, as the point of view <NUM> of the user changes, the arrangement <NUM> of multiple user-selectable visual elements <NUM> changes. For example, if the point of view <NUM> rotates then the arrangement <NUM> of multiple user-selectable visual elements <NUM> rotates. This rotation is illustrated by comparing <FIG>. In these examples, the arrangement <NUM> of multiple user-selectable visual elements <NUM> is the same in both <FIG> except that it has been rotated about an axis through the point O and orthogonal to the change in orientation.

The multiple user-selectable visual elements <NUM> at positions p have an appearance <NUM> dependent upon audio content <NUM> for the orientations θ associated with the positions p. Thus, the different multiple user-selectable visual elements <NUM>i, <NUM>j, <NUM>k at respective positions pi, pj, pk have respective visual appearances <NUM>i, <NUM>j, <NUM>k dependent upon audio content 120i, 120j, <NUM> for the orientations θi, θj, θk associated with the positions pi, pj, pk.

The user interface <NUM> is part of an apparatus <NUM>, for example as illustrated in <FIG>. The user interface <NUM> and the apparatus <NUM> are configured to respond to selection of a user-selectable visual element <NUM>. If the selected user-selectable visual element <NUM>, is a user-selectable visual element <NUM>n at a position pn in the user interface <NUM> indicative of an orientation θn from the user <NUM> then audio content <NUM>n for the orientation θn is selected for processing. The suffix n can, for example, reference any of the multiple user-selectable visual elements <NUM> e.g. n= i, j or k in the illustrated example.

The processing can, for example, comprise the rendering of the respective audio content <NUM> to the user or other processing of the audio content <NUM> that has been selected, such as editing, modification, sharing, sending, inclusion, attachment, storage etc..

It should be appreciated that in some examples if audio content <NUM> for a respective orientation θ is changed without changing the orientation θ, the appearance <NUM> of the user-selectable visual element <NUM> at the position p in the user interface <NUM> indicative of the orientation θ changes. This is because the multiple user-selectable visual elements <NUM> at positions p have an appearance <NUM> dependent upon audio content <NUM> for the orientations θ associated with the positions p. Therefore a change in audio content <NUM>n can cause a change an appearance <NUM>n of the associated user-selectable visual element <NUM>n.

<FIG> illustrate examples of a real space in which a user <NUM> has a point of view <NUM>. The point of view <NUM> in the real space is measured relative to a reference <NUM>. <FIG> illustrate a change in the point of view, in the real space, of the user <NUM> relative to the reference <NUM>.

<FIG> illustrate an example of a virtual space. The user <NUM>, in the virtual space, has a point of view <NUM> that is measured relative to a reference <NUM>. A change in the point of view <NUM> in the virtual space, of the user <NUM>, is measured relative to the reference <NUM>. <FIG> illustrate different points of view <NUM> in the virtual space relative to the reference <NUM>.

Audio content <NUM> is associated with different orientations θ relative to a user <NUM>. For example, audio content <NUM> can be rendered from the orientation θ it is associated with. The user point of view <NUM> can define an orientation θ, and can in this way select audio content <NUM> that is associated with that orientation θ. Therefore, changing the point of view <NUM> of the user <NUM> in the virtual space changes the selected audio content <NUM>.

In some but not necessarily all examples, visual content <NUM> is associated with different orientations θ relative to the user <NUM>. For example, visual content <NUM> can be rendered from the orientation θ it is associated with. The user point of view <NUM> can define an orientation θ, and can in this way select visual content <NUM> that is associated with that orientation θ. Therefore, changing the point of view <NUM> of the user <NUM> in the virtual space changes the selected visual content <NUM>.

In some examples, the audio content <NUM> comprises sound sources in the virtual space. The positions within the virtual space of the sound sources can be a location of a sound object within the virtual space or can be a direction (orientation) of a sound source in the virtual space.

Thus, the user point of view <NUM> in the virtual space can be associated with visual content <NUM> that is dependent upon that point of view <NUM> (in addition to being associated with audio content <NUM> that is dependent upon the point of view <NUM>). Changing the point of view <NUM> of the user <NUM> in the virtual space changes the associated visual content <NUM> and audio content <NUM>.

<FIG> illustrates that the point of view <NUM> of the user <NUM> in the virtual space is directed towards a portion of the virtual space that corresponds to audio content <NUM>i and visual content <NUM>i. In <FIG>, the point of view <NUM> of the user <NUM> in the virtual space is directed towards a portion of the virtual space that corresponds to audio content <NUM>k and visual content <NUM>k.

In these examples, the audio content <NUM> is indicated as sound objects. Referring to <FIG>, the audio content <NUM>i is substantially aligned with the point of view <NUM> of the user <NUM> in the virtual space. The audio content <NUM>j has an orientation θi=<NUM> (or |θi | < δ) relative to the point of view <NUM> of the user <NUM> in the virtual space. The audio content <NUM>k has an orientation θk relative to the point of view <NUM> of the user <NUM> in the virtual space. The audio content <NUM>j has an orientation of θj relative to the point of view <NUM> of the user <NUM> in the virtual space.

Referring to <FIG>, the audio content <NUM>k is substantially aligned with the point of view <NUM> of the user <NUM> in the virtual space. The audio content <NUM>k has an orientation θk=<NUM> (or |θk | < δ) relative to the point of view <NUM> of the user <NUM> in the virtual space. The audio content <NUM>i has an orientation of θi relative to the point of view <NUM> of the user <NUM> in the virtual space. The audio content <NUM>k has an orientation θk relative to the point of view <NUM> of the user <NUM> in the virtual space.

<FIG> illustrates an example of the visual content <NUM>i associated with the point of view <NUM> illustrated in <FIG>. Although <FIG> illustrates the audio content <NUM>i as sound objects, it should be understood that these sound objects are not visible. In some circumstances the audio content <NUM>i is rendered audibly as sound objects as if positioned in the positions indicated in the figure. The positions of the audio content <NUM>i are included in <FIG> to illustrate that they are associated with corresponding visual features A, B in the visual content <NUM>i. That is, the visual content <NUM>i includes within it visual content that is associated with the audio content <NUM>i. In some circumstances the associated visual content may visualise the origin or source of the audio content <NUM>i.

<FIG> illustrates an example of the visual content <NUM>k associated with the point of view <NUM> illustrated in <FIG>. Although <FIG> illustrates the audio content <NUM>k as sound objects, it should be understood that these sound objects are not visible. In some circumstances the audio content <NUM>k is rendered as a sound object as if positioned in the position indicated in the figure. The position of the audio content <NUM>k is included in <FIG> to illustrate that it is associated with a corresponding visual feature C in the visual content <NUM>k. That is, the visual content <NUM>k includes within it visual content that is associated with the audio content <NUM>k. In some circumstances the associated visual content may visualise the origin or source of the audio content <NUM>k.

The user interface <NUM> illustrated in <FIG> can be used to represent the situation illustrated in <FIG> and the user interface <NUM> illustrated in <FIG> can be used to represent the situation illustrated in <FIG>.

The user interface <NUM> comprises an arrangement <NUM> of multiple user-selectable visual elements <NUM>i, <NUM>j, <NUM>k at respective positions pi, pj, pk in the user interface <NUM>. The positions pi, pj, pk in the user interface <NUM> are indicative of different orientations θi, θj, θk from a user <NUM>. In this example the user <NUM> in the virtual space has a position O in the user interface <NUM> and the angles θi, θj, θk are measured from O. The angles θi, θj, θk are defined relative to a current point of view <NUM> of the user <NUM> in the virtual space. As the point of view <NUM> of the user <NUM> in the virtual space rotates the arrangement <NUM> of multiple user-selectable visual elements <NUM> rotates. The multiple user-selectable visual elements <NUM>i, <NUM>j, <NUM>k have an appearance <NUM>i, <NUM>j, <NUM>k dependent upon respective audio content <NUM>i, <NUM>j, <NUM>k for the respective orientations θi, θj, θk. The appearance <NUM> of a user-selectable visual element <NUM> does not change just because of the rotation.

Any of the user-selectable visual elements <NUM>i, <NUM>j, <NUM>k can be selected by a user. The selection can occur in any suitable way.

If the selected user-selectable visual element <NUM> is at a position p in the user interface <NUM> indicative of an orientation θ from the user <NUM> then audio content <NUM> for the orientation θ is selected for processing.

In some examples, the user point of view <NUM> in the virtual space can be varied via user input to the user interface <NUM>. For example, the user can provide touch input, keystrokes, or similar.

In other examples, the user point of view <NUM> in the virtual space is controlled via mediated reality. In mediated reality, the user point of view <NUM> in the real space controls the user point of view <NUM> in the virtual space. Each change in the point of view <NUM> in the real space causes a corresponding change of the point of view <NUM> in the virtual space. The point of view <NUM> of the user <NUM> in virtual space illustrated in <FIG> corresponds to the point of view <NUM> of the user <NUM> in real space illustrated in <FIG>. The point of view <NUM> of the user <NUM> in virtual space illustrated in <FIG> corresponds to the point of view <NUM> of the user <NUM> in real space illustrated in <FIG>.

In all examples, the audio content <NUM> can be any suitable audio content. The audio content <NUM> can comprise only audio content or it can comprise audio content and metadata or it can comprise audio content and visual content <NUM>. The visual content <NUM> can be provided as part of the audio content <NUM>, for example, as multi-media content or can be provided separately. The visual content <NUM> can be a still image, a sequence of still images or a video. The visual content <NUM> and/or the metadata can be used to determine the visual appearance <NUM> of a user-selectable visual element <NUM>.

The audio content <NUM> and the visual content <NUM>, if present, can be panoramic. Panoramic in this context means that the audio and/or image extends beyond a current field of perception of the user. That is, the user <NUM> in the virtual space needs to adopt multiple different points of view <NUM> to access the whole of the audio content <NUM> and/or visual content <NUM> available at any one time.

If a user adapts a particular point of view <NUM>, that point of view positions the field of perception. The visual content <NUM> for that point of view <NUM> is defined by the field of perception (field of view). The audio content <NUM> for that point of view <NUM> is defined by the field of perception (may be the same or different to the field of view). In a play back mode, the visual content <NUM> and/or the audio content <NUM> for the point of view <NUM> defined by the field of perception is rendered. The audio content <NUM> can be rendered in playback mode from the orientation θ it is associated with which may be offset from the point of view <NUM>, for example.

The panoramic audio content <NUM> and/or visual content <NUM> can, for example, be two-dimensional and can extend like a cylinder for <NUM> degrees in a horizontal plane around the user <NUM> in the virtual space. In this circumstance, the point of view <NUM> of the user <NUM> is an angle in the azimuthal (horizontal) plane. The point of view <NUM> of the user cannot be moved out of the horizontal plane.

In other examples, the audio and/or visual content can be three-dimensional. In this example, the point of view <NUM> of the user <NUM> in the virtual space has two degrees of movement and can be moved in the horizontal plane and the vertical plane (polar and azimuth). In still other examples, the point of view <NUM> of the user <NUM> in the virtual space can have a further three degrees of freedom for defining a three-dimensional location of the point O.

If mediated reality is used to control the point of view <NUM> of the user <NUM> in the virtual space, then this may be achieved by, for example, using a 3DoF, 3DoF+, or 6DoF system. The user <NUM> in the real space can, for example, wear a head tracking device that tracks movement of the user's head in space. A change in orientation of the user's head can be tracked in the azimuthal and polar directions, for example. In addition, small spatial displacements of the user's head can be tracked in some examples (3DoF+) or large translations of the user's head can be measured in some examples (6DoF). Mediated reality comprises rendering content, which may be visual content <NUM> and/or audio content <NUM>, in dependence upon a real world point of view <NUM> of a user <NUM>. The audio content <NUM> for an orientation θ is audio content <NUM> that would be rendered to a user <NUM> using mediated reality when the user <NUM> changes a real world point of view <NUM> of the user so that the real world point of view <NUM> of the user corresponds to the orientation θ. The visual content <NUM> for an orientation θ is visual content <NUM> that would be rendered to a user <NUM> using mediated reality when the user <NUM> changes the real world point of view <NUM> of the user so the point of view <NUM> of the user in the virtual space corresponds to the orientation θ.

<FIG> illustrates an example of the apparatus <NUM> implemented as a head-mounted apparatus <NUM> that comprises head tracking circuitry <NUM>. In this example, the head-mounted apparatus <NUM> is a head-mounted display and also comprises a display <NUM> mounted in front of the user's eyes for displaying visual content <NUM> and audio output devices <NUM> adjacent the user's ears for rendering audio content <NUM>.

Referring back to <FIG>, it discloses an example of visual content <NUM>i for an orientation θi. The visual content <NUM>i is visual content that would be rendered to a user <NUM> when a user <NUM> in the virtual space changes the point of view <NUM> of the user so that point of view <NUM> corresponds to the orientation θi. In mediated reality, this occurs when the user <NUM> in the real space changes a point of view <NUM> of the user. In order to show the association between the visual content <NUM>i for an orientation θi and the audio content <NUM>i for an orientation θi, the audio content <NUM>i is also illustrated in <FIG>. In the example of <FIG> the audio content <NUM>i is associated with a portion A of the visual content <NUM>i and other audio content <NUM>i is associated with another portion B of the visual content <NUM>i.

Referring back to <FIG>, it discloses an example of visual content <NUM>k for an orientation θk. The visual content <NUM>k is visual content that would be rendered to a user <NUM> when a user <NUM> in the virtual space changes a point of view <NUM> of the user so the point of view <NUM> of the user corresponds to the orientation θ. In mediated reality, this occurs when the user <NUM> in the real space changes a point of view <NUM> of the user. In order to show the association between the visual content <NUM>k for an orientation θk and the audio content <NUM>k for an orientation θk, the audio content <NUM>k is also illustrated in <FIG>. In the example of <FIG> the audio content <NUM>k is associated with a portion C of the visual content <NUM>k.

Referring to <FIG>, the user-selectable visual element <NUM>i, <NUM>j, <NUM>k at a position pi, pj, pk indicative of an orientation θi, θj, θk from a user <NUM> has a visual appearance <NUM>i, <NUM>j, <NUM>k dependent upon visual content <NUM>i, <NUM>j, <NUM>k for the respective orientation θi, θj, θk.

The audio content <NUM> determines the orientations θi, θj, θk. The orientations θi, θj, θk in turn determine the visual content <NUM>i, <NUM>j, <NUM>k. Consequently, if a user-selectable visual element <NUM> has an appearance <NUM> dependent upon visual content <NUM> for an orientation θ then that appearance is also dependent upon audio content <NUM> for that orientation θ.

As illustrated in the example of <FIG>, the user-selectable visual element <NUM> at a position p indicative of an orientation θ from a user <NUM> has an appearance <NUM> that comprises a visual extract <NUM> of visual content <NUM> for the orientation θ. <FIG> illustrates a user-selectable visual element <NUM>i that comprises visual extract <NUM>i of visual content <NUM>i illustrated in <FIG>. In this example, the visual extract <NUM>i is a full field of view visual extract. The field of view in the visual extract <NUM>i corresponds to the field of view of the visual content <NUM>i, however, the resolution is significantly reduced. <FIG> illustrates a user-selectable visual element <NUM>k that comprises visual extract <NUM>k of visual content <NUM>k illustrated in <FIG>. In this example, the visual extract <NUM>k is a partial field of view visual extract. The field of view in the visual extract <NUM>k corresponds to only a cropped part of the field of view of the visual content <NUM>i (a cropped region). The resolution can be the same or reduced.

The visual extract <NUM> of visual content <NUM> for an orientation θ can comprise an image for the orientation θ, a looped clip of video for the orientation θ, a repeated sequence of images for the orientation θ or similar.

In some, but not necessarily all examples, characteristics of an appearance <NUM> of a user-selectable visual element <NUM> are indicative of characteristics of audio content <NUM> selectable via the user-selectable visual element <NUM>. For example, the characteristics of the audio content <NUM> can be dependent upon one or more parameters including: a number of sound sources, spatial separation of sound sources, amplitude of sound sources, direction of sound sources, quality of sound sources, changeability of sound sources.

The characteristics of the audio content <NUM> can for example be determined from metadata comprised in or associated with the audio content <NUM>. Alternatively, or in addition, characteristics of the audio content <NUM> can, for example, be determined by analysis of the audio content <NUM>. For example, beam-forming analysis of the audio content <NUM> can be used to identify and position sound sources and to then determine the number of sound sources, the spatial separation of sound sources, the amplitude of sound sources, the direction of sound sources, the quality of sound sources, the changeability of sound sources.

The characteristics of an appearance <NUM> of a user-selectable visual element <NUM> can include, for example, any one or more of a size of the user-selectable visual element <NUM>, brightness, colouration, blurred edges and the use of moving or still images.

<FIG> illustrates an example of a method <NUM> comprising: at block <NUM> providing a user interface <NUM> for enabling a user <NUM> to select audio content <NUM>. The user interface <NUM> comprises an arrangement <NUM> of multiple user-selectable visual elements <NUM> at positions p in the user interface <NUM> that are indicative of different orientations θ from the user <NUM> and have an appearance <NUM> dependent upon audio content <NUM> for the respective orientations θ.

The method <NUM> then comprises, at block <NUM>, responding to selection of a user-selectable visual element <NUM> at a position p in the user interface <NUM> indicative of an orientation θ from the user <NUM> to select, for processing, audio content <NUM> for the orientation θ. The selection of the user-selectable visual element <NUM> is by user actuation. Where the user interface <NUM> is displayed in mediated reality, the actuation may be a gesture within the virtual space or some other action within the virtual space. Where the user interface <NUM> is displayed on a touchscreen, the actuation may be touching a portion of the touchscreen that corresponds to a user-selectable visual element <NUM>.

Applying the method <NUM> to the example of the user interface <NUM> illustrated in <FIG>, a first user-selectable visual element <NUM>i has a first position pi in the user interface <NUM> indicative of a first orientation θi from the user <NUM> and has a first appearance <NUM>i dependent upon first audio content <NUM>i (not illustrated) for the first orientation θi, a second user-selectable visual element <NUM>j has a second position pj in the user interface <NUM> indicative of a second orientation θj from the user and has a second appearance <NUM>j dependent upon the second audio content <NUM>j (not illustrated) for the second orientation θj and a third user-selectable visual element <NUM>k has a third position pk in the user interface <NUM> indicative of a third orientation θk from the user <NUM> and has a third appearance <NUM>k dependent upon third audio content <NUM>k (not illustrated) with a third orientation θk.

The apparatus <NUM> is configured to respond to actuation of the first user-selectable visual element <NUM>i to select the first audio content <NUM>i (but not the second audio content <NUM>j or the third audio content <NUM>k); respond to user selection of the second user-selectable visual element <NUM>j to select the second audio content <NUM>j (but not the first audio content <NUM>i or the third audio content <NUM>k); and respond to user selection of third user-selectable visual element <NUM>k to select the third audio content <NUM>k (but not the first audio content <NUM>i or the second audio content <NUM>j).

In this example, the first audio content <NUM>i is different to the second audio content <NUM>j and the third audio content <NUM>k. The first user-selectable visual element <NUM>i has a first appearance <NUM>i different to the second appearance <NUM>j of the second user-selectable visual element <NUM>j and the third appearance <NUM>k of the third user-selectable visual element <NUM>k.

If only the first audio content <NUM>i is changed without changing the first orientation θi, the first appearance <NUM>i of the first user-selectable visual element <NUM>i changes and the second appearance <NUM>j of the second user-selectable visual element <NUM>j does not change and the third appearance <NUM>k of the third user-selectable visual element <NUM>k does not change. Likewise, if only the second audio content <NUM>j is changed, without changing the second orientation θj, the second appearance <NUM>j of the second user-selectable visual element <NUM>j changes and the appearance <NUM><NUM> of the first user-selectable visual element <NUM><NUM> does not change and the appearance <NUM>k of the third user-selectable visual element <NUM>k does not change.

As previously described, the user interface <NUM> comprises an arrangement of multiple user-selectable visual elements <NUM> at positions p in the user interface <NUM>. In some examples, the user-selectable visual elements <NUM> comprise a first portion <NUM> and a second portion <NUM> (for example as illustrated in <FIG>). The first portion <NUM> is an asymmetric wedge shape and is indicative of an orientation θ from a user. The midline of reflection symmetry of the wedge is aligned with orientation θ. The second portion <NUM> that has an appearance <NUM> dependent upon audio content <NUM> for the orientation θ. The appearance <NUM> of the second portion <NUM> of the user-selectable visual element <NUM> can be as described in the preceding paragraphs for the appearance of the user-selectable visual element <NUM>. In some, but not necessarily all examples, user selection of either the first portion <NUM> or the second portion <NUM> of a user-selectable visual element <NUM> causes user selection of the user-selectable visual element <NUM>. In other examples, user selection of the second portion <NUM> (but not necessarily the first portion <NUM>) of the user-selectable visual element <NUM> causes user selection of the user-selectable visual element <NUM>.

The user-selectable visual element <NUM> has the first portion <NUM> and the second portion <NUM> at positions in the user interface <NUM> that are indicative of an orientation from the user <NUM>. A line along orientation θ bi-sects both.

In some, but not necessarily all examples, the first portion <NUM> represents a segment of a circle. The characteristics of the appearance of the segment can include segment size (area), width (angle subtended at the point O), length (radial distance from the point O).

In the example illustrated, the segments <NUM> are regularly sized segments each of which corresponds to a particular orientation θ. Only some of the segments <NUM> are associated with audio content and have corresponding user-selectable visual elements <NUM>. Therefore, some of the segments <NUM> are empty and provide gaps between the other user-selectable visual elements <NUM>.

The upper portion of <FIG> illustrates a portion of visual content <NUM>. The visual content comprises visual content <NUM><NUM>, <NUM><NUM>, <NUM><NUM> associated with different respective points of view <NUM><NUM>, <NUM><NUM>, <NUM><NUM> (not illustrated in <FIG>) and with different respective audio content <NUM><NUM>, <NUM><NUM>, <NUM><NUM> (not illustrated in <FIG>). The visual content <NUM> is oriented with respect to a current (variable) point of view <NUM> (not illustrated in <FIG>) of the user. The respective points of view <NUM><NUM>, <NUM><NUM>, <NUM><NUM> have different orientations θ<NUM>, θ<NUM>, θ<NUM> (not illustrated in <FIG>) from the current point of view. In this example, the portion of visual content <NUM> illustrated comprises visual content <NUM><NUM>, <NUM><NUM>, (not visual content <NUM><NUM>). The visual content <NUM><NUM> visualizes an origin or source of the audio content <NUM><NUM> at orientation θ<NUM> and the visual content <NUM><NUM> visualizes an origin or source of the audio content <NUM><NUM> at orientation θ<NUM>. As the current point of view <NUM> of the user changes the orientations θ<NUM>, θ<NUM> change.

The lower portion of <FIG> illustrates a user interface <NUM> for enabling a user to select audio content <NUM><NUM>, <NUM><NUM>, <NUM><NUM>. The user interface <NUM> comprises an arrangement <NUM> of multiple user-selectable visual elements <NUM><NUM>, <NUM><NUM>, <NUM><NUM> at respective positions p<NUM>, p<NUM>, p<NUM> in the user interface <NUM> that are indicative of different respective orientations θ<NUM>, θ<NUM>, θ<NUM> from the user <NUM> (not illustrated in <FIG>).

The user-selectable visual element <NUM><NUM> at a position p<NUM> indicative of an orientation θ<NUM> from the user <NUM> has an appearance <NUM><NUM> dependent upon the audio content <NUM><NUM> for the respective orientation θ<NUM> because it comprises a visual extract <NUM> of the visual content <NUM><NUM> for the orientation θ<NUM>. This extract <NUM> visualizes an origin or source of the audio content <NUM><NUM> at orientation θ<NUM>.

User-selection of a user-selectable visual element <NUM><NUM>, <NUM><NUM>, <NUM><NUM> at a position p<NUM>, p<NUM>, p<NUM> in the user interface <NUM> indicative of an orientation θ<NUM>, θ<NUM>, θ<NUM> from the user <NUM> causes selection, for processing, of respective audio content <NUM><NUM>, <NUM><NUM>, <NUM><NUM> for the orientation θ<NUM>, θ<NUM>, θ<NUM>. Thus, user-selection of the user-selectable visual element <NUM><NUM> at position p<NUM> in the user interface <NUM> that is indicative of the orientation θ<NUM> causes selection, for processing, of the audio content <NUM><NUM> at orientation θ<NUM>. Thus, user-selection the user-selectable visual element <NUM><NUM> at position p<NUM> in the user interface <NUM> that is indicative of the orientation θ<NUM> causes selection, for processing, of the audio content <NUM><NUM> at orientation θ<NUM>. Thus, user-selection the user-selectable visual element <NUM><NUM> at position p<NUM> in the user interface <NUM> that is indicative of the orientation θ<NUM> causes selection, for processing, of the audio content <NUM><NUM> at orientation θ<NUM>.

In the example illustrated in <FIG>, the arrangement <NUM> of multiple user-selectable visual elements <NUM> comprises an ellipse representing a circle viewed from an elevated perspective. The effective user position O is at a midpoint between the focal points of the ellipse. This corresponds to the centre of the circle. As described previously, this circle can be segmented to produce first portions <NUM> of the multiple user-selectable visual elements <NUM>. Each of the multiple segments has a width and a centre vector through the point O. The centre vector is aligned with an orientation θ from a user.

In this example, the second portion <NUM> of a user-selectable visual element <NUM> is a visual cue to a consequence of user selection of the user-selectable visual element <NUM>. The characteristics of appearance <NUM> of the second portion <NUM> of the user-selectable visual element <NUM> include, for example size, brightness, colouration, blurred edges and whether or not the extracted visual content, the visual extract <NUM>, is video or a still image.

<FIG> illustrate an optional feature of the user interface <NUM>. <FIG> broadly corresponds to <FIG> and illustrates a user interface <NUM> that comprises a user-selectable visual element <NUM><NUM> that has an appearance <NUM><NUM>. This user-selectable visual element <NUM><NUM> corresponds to the user-selectable visual element <NUM>i illustrated in <FIG>. The other aspects of <FIG> are not illustrated for clarity of view. As illustrated in <FIG>, the appearance <NUM><NUM> of the user-selectable visual element <NUM><NUM> comprises a visual extract <NUM><NUM> (<FIG>) from the visual content for the orientation θ<NUM> (<FIG>). In this example the apparatus <NUM> does not respond to selection of the user-selectable visual element <NUM><NUM> by immediately and automatically processing the audio content <NUM><NUM> associated with the orientation θ<NUM>, as previously described, instead, the apparatus <NUM> responds to selection of the user-selectable visual element <NUM><NUM> at a position p<NUM> in the user interface <NUM> indicative of an orientation θ<NUM> from the user <NUM> to adapt the arrangement <NUM> of multiple user-selectable visual elements <NUM> to include more user-selectable visual elements <NUM><NUM>, <NUM><NUM> at positions p<NUM>, p<NUM> in the user interface <NUM> that are indicative of different orientations θ<NUM>, θ<NUM> from a user <NUM> and have an appearance dependent upon audio content <NUM><NUM>, <NUM><NUM> for the respective orientations θ<NUM>, θ<NUM>. The apparatus <NUM> is then configured to respond to selection of a user-selectable visual element <NUM><NUM>, <NUM><NUM> at a position p<NUM>, p<NUM> in the user interface <NUM> indicative of an orientation θ<NUM>, θ<NUM> from the user to select audio content <NUM><NUM>, <NUM><NUM> for the orientation θ<NUM>, θ<NUM>.

As was previously described in relation to <FIG>, a user-selectable visual element, such as the user-selectable visual element <NUM><NUM> in <FIG>, can be based upon visual content <NUM> that includes multiple visual features (e.g. A, B in <FIG>) associated with different audio content, for example, different sound objects. When this user-selectable visual element <NUM><NUM> is selected, the user-selectable visual element <NUM><NUM> is split into multiple different user-selectable visual elements <NUM><NUM>, <NUM><NUM>. The user-selectable visual element <NUM><NUM> is at an orientation θ<NUM> that is determined by a sound source corresponding to visual feature A and visual content <NUM> (e.g. feature A) determined by that sound source is illustrated in the user-selectable visual element <NUM><NUM>. The user-selectable visual element <NUM><NUM> is at an orientation θ<NUM> that is determined by a sound source corresponding to visual feature C and visual content <NUM> (e.g. feature B) determined by that sound source is illustrated in the user-selectable visual element <NUM><NUM>.

Referring back to the example of the user interface <NUM> illustrated in <FIG> (with reference also to <FIG>), each of the multiple user-selectable visual elements <NUM> of the user interface <NUM> has an appearance <NUM> determined by a visual extract <NUM> of visual content <NUM> where the extraction is by cropping and the visual extract <NUM> is a cropped region <NUM> of video segments of panoramic video (visual content <NUM>). The visual extracts (cropped regions) <NUM> are from directions θ of the panoramic video <NUM> corresponding to directions θ from where audio content <NUM> is heard in the panoramic video <NUM>. In the example illustrated in <FIG>, a van engine is making sounds to a rear of the user <NUM>. As a consequence, there is an sound source for audio content120 to the rear of the user <NUM>. As a consequence of this there is a user-selectable visual element <NUM> at a position p in the user interface <NUM> that corresponds with a position to a rear of the user. The appearance <NUM> of this user-selectable visual element <NUM> is dependent upon the audio content <NUM> at the rear of the user. In this example, it has an appearance <NUM> based upon a visual extract <NUM> (cropped regions of the panoramic video <NUM>) that includes the van that is making the sound heard from the rear of the user <NUM>. Thus, the back region of the video is cropped and shown on the user-selectable visual element <NUM> positioned on the ellipse defined by the user interface <NUM>. The user <NUM> can then click on the user-selectable visual element <NUM>, for example by clicking on the cropped video segment, or can click on a corresponding portion of the ellipse, to select a focus direction, that is to select the audio content <NUM> that corresponds to that direction θ.

In this example, the appearance <NUM> of the user-selectable visual element <NUM> (visual extract <NUM>) is rendered so that the properties of the selectable audio content <NUM> associated with that user-selectable visual element <NUM> are indicated. For example:.

As has been previously mentioned, in some but not necessarily all examples, the apparatus <NUM> is configured to analyse the audio content <NUM>. This analysis enables the appearance <NUM> of a user-selectable visual element <NUM> for orientation θ to be dependent upon audio content for the orientation θ.

For example, audio beam forming may be performed on the audio content <NUM> for beams <NUM> to several directions as illustrated in <FIG>. Signal energies are calculated for the beam forming signals. If the signal energy in a particular direction θ is greater than a set threshold, a visual extract <NUM> (e.g. a cropped region of the panoramic video <NUM>) is used as a user-selectable visual element <NUM> at the corresponding position θ on the ellipse of the user interface <NUM>. In this example, a sharpness (appearance <NUM>) of the edges of the visual extract <NUM> (the cropped region) can be determined by how much over the threshold the signal energy is. The higher the energy, the sharper the edge.

In some examples, the beams <NUM> are fixed and the beams have a fixed size, orientation θ and spacing. In other examples the size of the beams <NUM> can be varied and in some examples optimized for the audio content <NUM>.

In some examples, if adjacent beams <NUM> in different directions θ have high energy, then the multiple beams can be represented by a single wide user-selectable visual element <NUM> positioned on the ellipse of the user interface <NUM>. The width of the single wide user-selectable visual element <NUM> extends over the different directions θ of the adjacent beams <NUM>. In this example, the user <NUM> can select the wide user-selectable visual element <NUM> to select a wide audio segment that corresponds to the multiple beams. In this example, the user <NUM> can select different portions of the wide user-selectable visual element <NUM> to select one or more of the multiple beams. In this example, the user <NUM> can select the wide user-selectable visual element <NUM> to be presented with a user-selectable visual element <NUM> on the ellipse of the user interface <NUM> for each beam. Each of these narrow user-selectable visual elements <NUM> for a beam <NUM> extends over the direction θ for the beam <NUM>. The user <NUM> can select a narrow user-selectable visual element <NUM> to select a narrow audio segment that corresponds to a single beam. In some examples, the user interface <NUM> can indicate "component" directions θ of beams that are comprised within a group of beams, after the user-selectable visual element <NUM> for the group is selected by the user.

In this or other examples, audio metadata can be used to determine a sharpness of edges of a user-selectable visual element <NUM>.

For example, a variation (variants, or standard deviation or other suitable fluctuation metric) of a direction of arrival of a sound can be used to adjust a sharpness and/or a width of the user-selectable visual element <NUM>.

For example, the greater a ratio of direct sound energy to ambient or reverberant energy in a direction θ, the sharper the user-selectable visual element <NUM> for that direction θ.

For example, the smaller a ratio of direct sound energy to ambient or reverberant energy in a direction θ, the less sharp the user-selectable visual element <NUM> for that direction θ.

For example, parametric spatial audio data can comprise direction and ratio metadata at time-frequency tiles. Such metadata can be formed by dividing the audio content <NUM> into short time frames and the spectrum for each time frame is analysed yielding a time-frequency tile representation. For each time-frequency tile, at least one dominant direction of arrival (DOA) of dominant sound is analysed for example via time-delay of arrival to a defined microphone and at least one energy ratio parameter is analysed e.g. the ratio of direct sound energy to ambient sound energy. The strong directional sounds can be determined to be in the locations of most common direction of arrival values for example by creating a histogram for direction of arrival values. The variation (variants, or standard deviation or other suitable fluctuation metric) of the direction of arrival values around a determined sound source direction of arrival can be used for adjusting a sharpness and/or a width of the user-selectable visual element <NUM>. In addition, the radio metadata can affect an appearance <NUM> (sharpness) of the user-selectable visual element <NUM> so that the larger the ratio of direct sound energy to ambient energy in direction estimates corresponding to a determined source, the sharper the visual extract <NUM> ( the cropped region).

In the examples in <FIG>, the audio content <NUM> is provided by sound sources in the virtual space. In these examples, the virtual space and the sound sources are illustrated in two-dimensions. In other examples, the virtual space is three-dimensional and the sound sources have different three-dimensional positions within the virtual space.

A sound source (e.g. a person) that provides audio content <NUM> is somewhere in front of a recording device at O. Depending on how well the sound source audio is recorded in the different beamformed audio signals (focus signals) <NUM>, the visual extract <NUM> (e.g. crop region) of the user-selectable visual element <NUM> is created differently. In these examples, an appearance <NUM> of an edge of the user-selectable visual element <NUM> is used to indicate a quality of the associated audio content <NUM>. If the audio content <NUM> is of higher quality the edges are sharper and better defined. If the audio content is of lower quality the edges are less sharp and are blurred.

In <FIG>, the sound source that provides audio content <NUM> is directly in the middle of a beam <NUM> and is recorded well. The visual extract <NUM> (e.g. the cropped region) of the user-selectable visual element <NUM> is narrow and has sharp edges.

In <FIG>, the sound source that provides audio content <NUM> is in the middle of two beams <NUM>. The sound source that provides audio content <NUM> is recorded in both beams but not very loudly in either one (low energy). Thus, the visual extract <NUM> (e.g. the cropped region) <NUM> is wide and has blurry edges.

In <FIG>, the sound source that provides audio content <NUM> is farther away, and recorded partially in one beam <NUM>. The visual extract <NUM> (e.g. the cropped region) is narrow and has blurred edges. The user <NUM> can see that the sound source that provides audio content <NUM> is not centered in the focus beam <NUM> in this direction.

In <FIG>, audio is generated from a single source but is reflected from nearby walls. There are therefore three sound sources- one direct sound source D and two reflected sound sources R1, R2. The direct sound source D produces audio content <NUM>D associated with an orientation/direction θD The reflected sound source R1 produces audio content <NUM>R1 associated with an orientation/direction θR1 The reflected sound source R2 produces audio content <NUM>R2 associated with an orientation/direction θR2.

<FIG> illustrates the corresponding user interface <NUM>.

There is a user-selectable visual element <NUM>D at an orientation θD in the user interface <NUM> that has an appearance <NUM>D dependent upon audio content <NUM>D associated with the respective orientation θD. The visual extract <NUM>D in the user-selectable visual element <NUM>D shows a person speaking.

There is a user-selectable visual element <NUM>R1 at an orientation θR1 in the user interface <NUM> that has an appearance <NUM>R1 dependent upon audio content <NUM>R1 associated with the respective orientation θR1. The visual extract <NUM>R1 in the user-selectable visual element <NUM>R1 only shows a wall.

There is a user-selectable visual element <NUM>R2 at orientation θR2 in the user interface <NUM> that has an appearance <NUM>R2 dependent upon audio content <NUM>R2 associated with the respective orientation θR2. The visual extract <NUM>R2 in the user-selectable visual element <NUM>R2 only shows a wall.

The user can easily see from the appearance <NUM>R1, <NUM>RD, <NUM>R2 of the user-selectable visual elements <NUM>R1, <NUM>D, <NUM>R2 that two of the three focus directions θ are not pointing towards the sound source (the speaking person).

The user <NUM> is now able to select the user-selectable visual element <NUM>D where the person is seen for the focus direction θ. This selects the audio content <NUM>D for processing.

Alternatively, if the user <NUM> wants to also hear the reflections, the user <NUM> selects the user-selectable visual elements <NUM>R1, <NUM>D, <NUM>R2. This selects the audio content <NUM>R1, <NUM>D, <NUM>R2 for processing.

As will be appreciated from the foregoing, a direction estimate θ for a sound can be slightly off from a correct direction where the sound source can be seen (e.g. because of reverberation/reflections). In some but not necessarily all examples, a user-selectable visual elements <NUM> can be extended until the corresponding visual extract <NUM> includes an image of the sound source. The direction of extension can, for example, be determined using image processing and/or audio processing. For example, the user-selectable visual elements <NUM> can be extended towards the closest image of the sound source or towards a strong nearby sound source.

In some examples, for example as illustrated in <FIG>, the user interface <NUM> arranges the user-selectable visual elements <NUM> at a perimeter of an ellipse. There is a cylindrical visualization of panoramic video along at least parts of the perimeter of the ellipse. This is achieved by rendering visual extracts <NUM> (e.g. the cropped regions) <NUM> of the video on the perimeter of the ellipse. The visual extracts <NUM> (e.g. the cropped regions), their position p and their appearance <NUM> can be automatically determined based on analysis of the audio content <NUM> or its metadata. For example, the shape and sharpness of a visual extract <NUM> (e.g. a cropped region) for a direction θ can indicate quality of the audio content <NUM> for the direction θ. The user <NUM> can select the focus direction θ by selecting the user-selectable visual element <NUM>. Actual estimated directions θ are used for selecting focus direction θ.

<FIG> illustrates an example of an apparatus <NUM> comprising controller <NUM> that is configured to provide the user interface <NUM> and user input <NUM> for providing user input to the controller for user selection of a user-selectable visual element <NUM>. The user interface <NUM> is provided via a display <NUM>. In some but not necessarily all examples, the display <NUM> is a touch-sensitive display and also provides the user input <NUM>.

<FIG> illustrates an example of a controller <NUM>. Implementation of the controller <NUM> may be as controller circuitry. The controller <NUM> may be implemented in hardware alone, have certain aspects in software including firmware alone or can be a combination of hardware and software (including firmware).

The memory <NUM> stores a computer program <NUM> comprising computer program instructions (computer program code) that controls the operation of the apparatus <NUM> when loaded into the processor <NUM>. The computer program instructions, of the computer program <NUM>, provide the logic and routines that enables the apparatus to perform the methods illustrated in Figs. The processor <NUM> by reading the memory <NUM> is able to load and execute the computer program <NUM>.

As illustrated in <FIG>, the computer program <NUM> may arrive at the apparatus <NUM> via any suitable delivery mechanism <NUM>. The delivery mechanism <NUM> may be, for example, a machine readable medium, a computer-readable medium, a non-transitory computer-readable storage medium, a computer program product, a memory device, a record medium such as a Compact Disc Read-Only Memory (CD-ROM) or a Digital Versatile Disc (DVD) or a solid state memory, an article of manufacture that comprises or tangibly embodies the computer program <NUM>. The delivery mechanism may be a signal configured to reliably transfer the computer program <NUM>. The apparatus <NUM> may propagate or transmit the computer program <NUM> as a computer data signal.

Computer program instructions for causing an apparatus to perform at least the following or for performing at least the following:.

The blocks illustrated in the Figs may represent steps in a method and/or sections of code in the computer program <NUM>. The illustration of a particular order to the blocks does not necessarily imply that there is a required or preferred order for the blocks and the order and arrangement of the block may be varied. Furthermore, it may be possible for some blocks to be omitted.

The above described examples find application as enabling components of:
automotive systems; telecommunication systems; electronic systems including consumer electronic products; distributed computing systems; media systems for generating or rendering media content including audio, visual and audio visual content and mixed, mediated, virtual and/or augmented reality; personal systems including personal health systems or personal fitness systems; navigation systems; user interfaces also known as human machine interfaces; networks including cellular, non-cellular, and optical networks; ad-hoc networks; the internet; the internet of things; virtualized networks; and related software and services.

The term 'a' or 'the' is used in this document with an inclusive not an exclusive meaning. That is any reference to X comprising a/the Y indicates that X may comprise only one Y or may comprise more than one Y unless the context clearly indicates the contrary. If it is intended to use 'a' or 'the' with an exclusive meaning then it will be made clear in the context. In some circumstances the use of 'at least one' or 'one or more' may be used to emphasis an inclusive meaning but the absence of these terms should not be taken to infer and exclusive meaning.

The presence of a feature (or combination of features) in a claim is a reference to that feature or (combination of features) itself and also to features that achieve substantially the same technical effect (equivalent features). The equivalent features include, for example, features that are variants and achieve substantially the same result in substantially the same way. The equivalent features include, for example, features that perform substantially the same function, in substantially the same way to achieve substantially the same result.

Claim 1:
An apparatus (<NUM>) comprising means for:
providing a user interface (<NUM>), for enabling a user to select audio content (<NUM>), wherein the user interface (<NUM>) comprises an arrangement (<NUM>) of multiple user-selectable visual elements (<NUM>), at positions in the user interface (<NUM>) that are indicative of different orientations from a user,
wherein the user-selectable visual elements (<NUM>) have an appearance dependent upon audio content (<NUM>) for the respective orientations, and
at least one of the multiple user-selectable visual elements (<NUM>), at a position indicative of an orientation from a user, is configured to have an appearance that comprises a visual extract (<NUM>) of visual content (<NUM>) for the orientation, wherein the visual content (<NUM>) is dependent upon audio content (<NUM>) for the orientation and is visual content (<NUM>) that would be rendered to a user using mediated reality when a user changes a real world point of view of the user so the real world point of view of the user corresponds to the orientation, wherein mediated reality comprises rendering content in dependence upon a real world point of view of a user, and wherein the visual extract (<NUM>) is a cropped region (<NUM>) of the visual content (<NUM>);
and
responding to selection of a user-selectable visual element (<NUM>) at a position in the user interface (<NUM>) indicative of an orientation from the user to select, for processing, audio content (<NUM>) for the orientation.