Patent Description:
In some situations, it may be desirable to render content to a user in dependence upon a user perception direction.

For example, when a sound scene is rendered to a listener through a head-mounted audio output device, for example headphones using binaural audio coding, it may be desirable for the rendered sound space, the virtual space, to remain fixed in real space when the listener turns their head in space.

In some situations, for example when a visual scene is rendered to a viewer through a head-monitored visual output device, for example a head-mounted display, it may be desirable for the rendered visual space, the virtual space, to remain fixed in real space when the viewer turns their head in real space. This means that the rendered visual space needs to be rotated relative to the head-mounted visual output device by the same amount in the opposite sense to the head rotation. The orientation of the rendered visual space tracks with the rotation of the viewer's head so that the rendered visual space remains fixed in space and does not move with the viewer's head.

In some examples these two approaches may be combined to create a multimedia virtual reality experience.

In these examples, the sound space and the visual space extend beyond what is perceived at any one time by a user. The user can, for example, explore the visual space and/or the sound space changing their perception direction, by rotating their head, and sampling different portions of the visual space and/or sound space.

<CIT> discloses a head mounted display (HMD), in which the resolution may increase or decrease in certain areas of the display. This may be done by monitoring the user's eye movement to identify a focal-point of the user's gaze, and then increasing the resolution of an area surrounding the focal portion or decreasing the resolution elsewhere. This more accurately mimics a user's high resolution and low resolution field of view. The technique can be used to improve the effectiveness of the HMD, as well as the general experience and comfort of users of the HMD.

The invention is set forth in the appended set of claims.

In the following examples, a record is kept of which content, whether visual content and/or sound content, has been observed by a user. Content is information that is expressed through rendering in a form perceivable by the human senses of hearing (audio content) or sight (visual content): What is subsequently observable by a user may be reduced or restricted in dependence upon what has been recorded as observed by the user.

In this way, the information content that is available to a user will diminish over time. While content that has not yet been observed may retain its full information, content that has been observed may be adapted to lose information. This may allow a composer of the original content to control observation of the visual content and/or sound content. For example, in some but not necessarily all examples content that has been observed once may no longer be observable.

In the context of a virtual space (visual space and/or sound space) that is greater than what is observable by a user at any particular time, portions of the virtual space that have been observed are adapted so that they are no longer observable as initially observed, while unobserved portions of the virtual space remain for observation, in the future, by the user without adaptation. A virtual space is space defined by content that has a position within a space. The user's perspective (point of view) within the virtual space (and/or, optionally, the user's position within the virtual space) define what content is observed by a user. Changing the user's perspective (point of view) within the virtual space (and/or optionally changing the user's position within the virtual space) change what portion of the content is observed by a user.

The user perception direction may be measured in many different ways. Any technology that tracks the point of view of the user (head orientation and/or gaze direction) may be used. For example, accelerometers or gyroscopes comprised in a headset may track the movement and orientation of a headset and thus the movement and orientation of the user's head and thus the movement and orientation of the user's point of view.

<FIG> illustrate an example of a user perception direction <NUM>. In this example <FIG> illustrates, from a top perspective view, a component of the perception direction <NUM> in a horizontal plane (plane of paper) and <FIG> illustrates, from a side perspective view, a component of the perception direction <NUM> in a vertical plane (plane of paper). In <FIG>, the user perception direction <NUM> is defined by an azimuthal angle φ offset, in this example, from a vertical reference plane through a horizontal reference direction <NUM>. The azimuthal angle φ rotates the user perception direction up to +/- <NUM>° relative to the vertical reference plane parallel to the horizontal plane. In <FIG>, the user perception direction <NUM> is defined by a polar angle θ offset, in this example, from a horizontal reference plane through the horizontal reference direction <NUM>. The polar angle offset rotates the user perception direction up to +/-<NUM>° out of the horizontal reference plane. In will be appreciated that the azimuthal angle and polar angle are the defined in accordance with the convention for spherical polar co-ordinates and that other co-ordinate system may be used.

<FIG> illustrate a cross-section through an observation field of perception <NUM> and a rendering field of perception <NUM>.

A virtual space is space defined by content that has a position within a space. The user's perception direction <NUM> (point of view) within the virtual space define a position of the rendering field of perception <NUM> within the virtual space. The position of the rendering field of perception <NUM> within the virtual space determines what content is present in the rendering field of perception <NUM>. Changing the user's perception direction <NUM> (point of view) within the virtual space changes the position of the rendering field of perception <NUM> within the virtual space and consequently changes what portion of the content is observed by a user.

In this example, the cross-section may be through the perception direction <NUM> and parallel to the vertical plane or, alternatively, the cross-section may be through the perception direction <NUM> and parallel to the horizontal plane.

The observation field of perception <NUM> and a rendering field of perception <NUM> are areas in a plane orthogonal to the perception direction <NUM>. In one example, the observation field of perception <NUM> and the rendering field of perception <NUM> are rectangles in a plane orthogonal to the perception direction <NUM>. However, other geometries are possible for defining the two-dimensional observation field of perception <NUM> and the two-dimensional rendering field of perception <NUM>.

Both the observation field of perception <NUM> and the rendering field perception <NUM> are determined by the user perception direction <NUM>. In these examples, the observation field of perception <NUM> and the rendering field perception <NUM> are both centered on the user perception direction <NUM>.

The rendering field of perception <NUM> is equivalent to a field of view for rendered content, but the use of the term 'perception' indicates that the field and content is not necessarily limited to only visual content. For example, changing the user's perception direction <NUM> (point of view) within a virtual space changes the position of the rendering field of perception <NUM> within the virtual space and consequently changes what portion of the content is observed by a user. The visual content observable by a user is framed by the rendering field of perception <NUM> at a position determined by the user's perception direction <NUM> (point of view). The audio content observable by a user is determined by the rendering field of perception <NUM> at a position determined by the user's perception direction <NUM> (point of view) and/or audio content observable by a user is determined by the user's perception direction <NUM> (point of view) and not the rendering field of perception <NUM>. In some example embodiments, audio content observable by a user is audio coming from in front of the user and/or audio associated with an object in front of a user.

In some but not necessarily all examples, the rendering field of perception <NUM> is one of multiple different rendering fields of perception (defined by different user perception directions <NUM>) that partially overlap to cover a visual scene that extends horizontally and vertically. The visual scene subtends, in the horizontal plane, an angle that may be greater than <NUM>° for example it may be <NUM>°. The visual scene subtends, in the vertical plane, an angle that may be greater than <NUM>°, for example it may be <NUM>°.

As illustrated in <FIG>, the combination of the user perception direction <NUM> and the rendering field of perception <NUM> define content <NUM> for rendering. The combination of the user perception direction <NUM> and the observation field of perception <NUM> define observed content <NUM>.

As previously described, a virtual space is space defined by content that has a position within a space. The user's perspective (point of view) within the virtual space (and/or, optionally, the users position within the virtual space) define what (portion of the) content is observed by a user (but does not necessarily define that content). Changing the user's perspective (point of view) within the virtual space (and/or optionally changing the user's position within the virtual space) changes what (portion of the) content is observed by a user (but does not necessarily change the content).

The rendering field of perception <NUM> defines (determines) what portion of possible content is rendered and therefore what is or can be perceived from that perception direction <NUM>. The observation field of perception <NUM> defines what is considered to be observed from that perception direction <NUM>. In this example, the observation field of perception <NUM> is smaller than the rendering field of perception <NUM> and is centered within the rendering field of perception <NUM>. This arrangement is based upon a model that assumes that a user focuses mostly on what content (visual content or audio content) is directly in front of the user. However, other arrangements are possible and the size and potion of the observation field of perception <NUM> may be defined other than by using the observation field of perception <NUM> and the perception direction <NUM>. The observation field of perception <NUM> may be smaller than or the same size as the rendering field of perception <NUM>. The observation field of perception <NUM> may be positioned anywhere within a boundary of the rendering field of perception <NUM>.

For example, although <FIG> illustrates a relationship between the size of the observation field of perception <NUM> and the rendering field of perception <NUM>, in other examples one or more of the rendering field of perception <NUM> and the observation field of perception <NUM> may be rescaled to have a different dimension, for example as illustrated in <FIG>. The observation field of perception <NUM> is smaller than the rendering field of perception <NUM> in at least one direction, in this example the observation field of perception <NUM> is less than the rendering field of perception <NUM> in all directions.

<FIG> schematically illustrates time evolution of the defined observed content <NUM> and the defined content <NUM> for rendering as a user perception direction <NUM> varies over time. Although, for the sake of convenience of illustration the content is displayed on a notional single 'content' axis that maps non-overlapping content to a different value, alternatively the single content axis may be multiple orthogonal axes.

As an example, content in a three-dimensional space may have value C(x,y,z) where (x, y, z) represents a position in the space. The position in three-dimensional space map be mapped to a one dimensional space ¥, and the content in the one-dimensional space has value C'(¥).

As an example, pixels in a head-up display may have a values P(x,y) where x represents a column address and y represents a row address. There may be N columns (<NUM> ≤ x ≥ N) and M rows (<NUM> ≤ y ≥ M), giving NM pixels in total. Each pixel value may be represented as a value in a two dimensional space (x. y), alternatively each pixel value may be represented as a value in a one dimensional space, a single content axis ¥, where ¥= (y-<NUM>).

In this example embodiment the observation field of perception <NUM> and the rendering field of perception <NUM> remain of constant size and shape. However this is not essential and in other example embodiments one or both of the observation field of perception <NUM> and the rendering field of perception <NUM> may vary in time.

In this example embodiment the time axis is plotted horizontally and the content that can be rendered is plotted schematically along a vertical axis. It should be appreciated that for fixed content the content is fixed and there is a common fixed y-axis (y) for all time but for time-variable content the content varies with time and there is a different time-dependent axis y(t) for different times where different content is available.

It will be appreciated that at any point in time, the content <NUM> for rendering represents only a subset of all the content that is available for rendering across all perception directions <NUM>.

As the user perception direction <NUM> varies in time the content <NUM> for rendering changes with the user perception direction <NUM> and the observation field of perception <NUM> also changes with the user perception direction <NUM>.

Portions of the virtual space that have been observed are adapted so that they are no longer observable as initially observed, while unobserved portions of the virtual space remain for observation, in the future, by the user without adaptation In this way, the information content that is available to a user will diminish over time. While content that has not yet been observed may retain its full information, content that has been observed may be adapted to lose information.

When the content <NUM> for rendering is first rendered all of that content <NUM> is rendered in a non-adapted (non-restricted) form.

The content <NUM> when first rendered includes content within the observation field of perception <NUM>. Once content is rendered within the observation field of perception <NUM> it is considered observed and the content rendered within the observation field of perception <NUM> is observed content <NUM>. Once observed, the observed content <NUM> within the observation field of perception <NUM> may be designated in whole or in part as restricted content. Once content is designated as restricted, its use may be restricted, for example as illustrated in <FIG>.

In some examples, all of the observed content is designated as restricted content. In other examples a part of the observed content is designated restricted content. For example, only observed content that relates to foreground objects may be designated restricted content and/or for example, only observed content that relates to key objects (e.g. person, animal, particular person etc) may be designated restricted content.

A log may be stored in a memory recording what content <NUM> for rendering has been designated restricted content <NUM>.

Observed content <NUM> may be designated as restricted content <NUM> when a threshold is exceeded or based on satisfaction of some other criterion or criteria. The threshold for designating observed content <NUM> as restricted content <NUM> may, for example, be based upon a number N (N≥<NUM>) of distinct times content has been observed content <NUM> or based on a cumulative duration content has been observed content <NUM>. A distinct time may be a time separated from a preceding time by a minimum time duration or may be a time in a different session or may be a time separated from a preceding time by a minimum time duration and that is in a different session. A session is a period of persistent use without interruption it may, for example, be a content session (rendering of the same specific content only) or may be an application session (instantiation of the rendering application).

It may also be desirable to dynamically restrict content, for example, to motivate a user to look in a particular direction for a particular duration.

<FIG> schematically illustrates a time evolution of defined observed content <NUM> and the defined content <NUM> for rendering as a user perception direction <NUM> varies over time. Although, for the sake of convenience of illustration the content is displayed on a notional single 'content' axis that maps non-overlapping content to a different value, alternatively the single content axis may be multiple orthogonal axes.

As the user perception direction <NUM> varies in time, the content <NUM> for rendering changes with the user perception direction <NUM> and the observation field of perception <NUM> also changes with the user perception direction <NUM>.

In this example, as in <FIG>, the observation field of perception <NUM> and the rendering field of perception <NUM> remain of constant size and shape. However, this is not essential and in other examples one or both of the observation field of perception <NUM> and the rendering field of perception <NUM> may vary in time. In this example, as in <FIG>, the time axis is plotted horizontally and the content that can be rendered is plotted schematically along a vertical axis. It should be appreciated that for fixed content (e.g. a photograph) the content is fixed (unchanging) and there is a common fixed y-axis (y) for all time but for time-variable content (e.g. a video with or without audio or a photograph wilt audio) the content varies with time (e.g. different video/audio frames have different content) and there is a different time-dependent axis y(t) for different times where different content (e.g. different video/audio frames) is available.

<FIG> differs from <FIG> in that the time evolution of the defined observed content <NUM>/ defined content <NUM> for rendering varies differently over time compared to the preceding time evolution of the defined observed content <NUM> / defined content <NUM> for rendering illustrated in <FIG>. Although the time evolution illustrated in <FIG> is at a later time compared to <FIG>, where the content is time-variable, the time axis has been calibrated for the purposes of this illustration with respect to the same start time of the content so that the y-axis of <FIG> are synchronized, as they vary in time. The time evolution of the defined observed content <NUM>/ the defined content <NUM> for rendering illustrated in <FIG> are therefore comparable as they are measured and illustrated in the same way.

The time evolution of the content <NUM> arises when content, with audio content or visual content, changes over time. Each time slice has particular content that differs from time slice to time slice. If content is observed in a particular time slice it may be designated as observed content for that time slice only (<FIG>) or designated as observed content for all future time slices (<FIG>).

<FIG> differs from <FIG> in that the previous time evolution of defined observed content <NUM> illustrated in <FIG> has been designated as restricted content <NUM> in <FIG> (illustrated using cross-hatching). <FIG> therefore illustrates a time evolution of restricted content <NUM> that corresponds to the time-evolution of the observed content in <FIG>.

<FIG> illustrates the current content <NUM> for rendering defined by the perception direction <NUM> and the rendering field of perception <NUM> and also the observation field of perception <NUM> that defines observed content <NUM>. Where the current content <NUM> for rendering does not overlap with the restricted content <NUM>, then the current content <NUM> for rendering is rendered in a first state <NUM>. However, where the content <NUM> for rendering overlaps with the restricted content <NUM>, that content <NUM> is rendered in a second state <NUM>, different to the first state <NUM>.

In this example, the time evolution of the restricted content <NUM> exactly corresponds to the time-evolution of the observed content <NUM>. However, in other examples the time evolution of the restricted content <NUM> may be dependent upon but may not exactly match the time-evolution of the observed content <NUM>. For example, in some examples the time evolution of the restricted content <NUM> encompass the time-evolution of the observed content <NUM> but extends to a limited extent beyond the time-evolution of the observed content <NUM> in a manner that is variable in time and is context-dependent and/or content-dependent.

In an example embodiment, there is a <NUM> minute long video. First a user looks in direction A for the first <NUM> seconds observing content C(A[<NUM>-<NUM>]), then in a direction B for <NUM>-<NUM> seconds observing content C(B[<NUM>-<NUM>] and rest of the video to the direction A again observing content C(A[<NUM>-<NUM>]). During this first session, the content C(A[<NUM>-<NUM>]), C(B[<NUM>-<NUM>] and C(A[<NUM>-<NUM>]) is not restricted. However, in this example after a single viewing session the observed content becomes restricted. When the user looks at the same video again in a different session, the viewing direction A is restricted between <NUM>-<NUM> seconds and between <NUM>-<NUM> seconds, the content C(A[<NUM>-<NUM>]) and C(A[<NUM>-<NUM>]) is restricted, also the viewing direction B is restricted between <NUM>-<NUM> seconds, the content C(B[<NUM>-<NUM>]) and is restricted.

Applying a similar example to <FIG>, there is a <NUM> minute long video. The content may be plotted as a time on the x-axis and a direction on the y-axis. A user looks in direction A at x seconds observing content C(A[x]). This corresponds to the observed content <NUM> occupying point (x,A) in <FIG>. Then the user looks in a direction other than A (not A) until y seconds observing content C(B[x->y]) where B(t)= A̅. This corresponds to the observed content <NUM> occupying points (x<t<y, B(t)) in <FIG>. Then the user looks in direction A again observing content C(A[y]). This corresponds to the observed content <NUM> occupying point (y, A) in <FIG>. During a first session (<FIG>), the content C(A[x]), C(B[x->y]) and C(A[y]) is not restricted. However, in this example after a single viewing session, the observed content including C(A[x]) and C(A[y]), becomes restricted content <NUM> (<FIG>). When the user looks at the same video again in a different session (<FIG>), the viewing direction A is restricted at x seconds and y seconds, the content C(A[x]) and C(A[y]) is restricted content <NUM>, also the viewing direction B(t) is restricted between x-y seconds, the content C(B [x->y]) is restricted content <NUM>.

<FIG> illustrates an example embodiment of multiple different time-evolutions of restricted content <NUM> determined at different times being combined to provide a composite version of restricted content <NUM>. Although, for the sake of convenience of illustration the content is displayed on a notional single 'content' axis that maps non-overlapping content to a different value, alternatively the single content axis may be multiple orthogonal axes.

<FIG> illustrates an example embodiment in which, when content has been designated as restricted content <NUM>, the same or similar content is designated restricted content <NUM> for subsequent times. This may, for example, be achieved by using computer vision to identify portions of subsequent content that correspond to content previously designated as restricted <NUM>. This may, for example be achieved by using feature extract and comparison or other techniques. Although, for the sake of convenience of illustration the content is displayed on a notional single 'content' axis that maps non-overlapping content to a different value, alternatively the single content axis may be multiple orthogonal axes.

<FIG> illustrates an example of a method <NUM> for rendering content. At block <NUM>, the method comprises rendering in a first state <NUM> any portion of the content <NUM> for rendering that is not defined as restricted content <NUM> and rendering in a second state <NUM>, different to the first state <NUM> any portion of the content <NUM> for rendering that is defined as restricted content <NUM>.

At block <NUM>, the method <NUM> comprises using a combination of the user perception direction <NUM> and an observation field of perception <NUM> to define restricted content <NUM>. This definition of restricted content <NUM> is used for later iterations of the method <NUM> at block <NUM>.

At a first time, content <NUM> may be rendered in a first state <NUM>. The rendered content <NUM> that falls within the observation field of perception <NUM> positioned by the user perception direction <NUM> is observed content <NUM>. The observed content, as it has now been observed, may be designated restricted content <NUM>. Subsequently, that restricted content <NUM>, when rendered, is rendered in a second state <NUM>, different to the first state <NUM>.

As the method <NUM> iterates, the restricted content <NUM> will typically increase with each iteration.

The method also illustrates at block <NUM>, using a combination of a user perception direction <NUM> and a rendering field of perception <NUM> to define content <NUM> for rendering.

In block <NUM>, observed content <NUM> may be designated as restricted content <NUM> when a threshold is exceeded. The threshold for designating observed content <NUM> as restricted content <NUM> may, for example, be based upon a number N (N≥<NUM>) of distinct times content has been observed content <NUM> or based on a cumulative duration content has been observed content <NUM>.

In one example, each rendering up to and including an Nth rendering of the observed content <NUM> is in the first state <NUM> and each rendering after and including an (N+<NUM>)th rendering of the observed content <NUM> is in the second state <NUM>. In another example, a first rendering of the observed content <NUM> is in a first state <NUM> when the observed content <NUM> is first observed and any subsequent rendering of the observed content <NUM> is in the second state <NUM>. Subsequent rendering may require there to be a delay of at least a predetermined duration.

<FIG> illustrate example embodimens of rendered content <NUM>. In these examples, the rendered content is visual content, however, a similar description is applicable for sound content.

The figures illustrate rendering in a first state <NUM> any portion of content <NUM> for rendering that is not defined as restricted content <NUM> and rendering in a second state <NUM>, different to the first state <NUM> any portion of the content <NUM> for rendering that is defined as restricted content <NUM>. It should be appreciated that when there is rendering in a second state <NUM>, different to the first state <NUM> of at least portion <NUM> of the content <NUM> (the restricted content) there is simultaneous rendering in a first state <NUM> of at least one portion of the content <NUM> (the un-restricted content). The second state <NUM> is constrained relative to the first state <NUM>. For example, the second state <NUM> has less information content than the first state <NUM>. Where the content <NUM> is sound content, the second state <NUM> may indicate that the sound content has a lower volume or is muted. Where the content <NUM> is visual content, the second state <NUM> may for example indicate that the content is obscured, removed or otherwise modified, for example de-focused or blurred.

<FIG> illustrates an example embodiment in which the content <NUM> is fixed content, for example, a still image. The content is visual content <NUM> that is static and does not vary in time. In this example embodiment, a combination of the user perception direction <NUM> and an observation field of perception <NUM> over a cumulative viewing time or cumulative number of sessions is used to define restricted content <NUM> in the still image. For example, referring to <FIG>, the restricted content <NUM> of the still image is defined by the projection onto the y axis of the observed content <NUM> over time, as the time at which the content is observed is irrelevant for a still image. The method <NUM> then uses a combination of a user perception direction <NUM> and a display field of view <NUM> to define visual content for display, displays in a first state <NUM> any portion of the visual content <NUM> for display that is not defined as restricted visual content <NUM> and displays in a second state <NUM>, different to the first state <NUM>, any portion of the visual content <NUM> for display that is defined as restricted content <NUM> and uses a combination of the user perception direction <NUM> and an observation field of view <NUM> to define observed content <NUM> for this viewing which may be used to define restricted content <NUM> for subsequent views.

<FIG> illustrate the rendering of time-varying content <NUM>. Each of <FIG> corresponds to a time t<NUM> to t<NUM> as illustrated in <FIG>. In this example, as the content <NUM> is time varying, the content itself is defined by the relative time, during playback, at which it is rendered.

Each of <FIG> corresponds to a time t<NUM> to t<NUM> as illustrated in <FIG>. However, the times t<NUM> to t<NUM> are merely illustrative and the time-varying content <NUM> may additionally be rendered at times intermediate of times t<NUM> to t<NUM>. In the specific example illustrated in <FIG>, the user perception direction <NUM> moves over times t<NUM> to t<NUM> towards the bottom right, causing the current content <NUM> for rendering defined by the perception direction <NUM> and the rendering field of perception <NUM> to move towards the top left. The region of overlap, between the current content <NUM> for rendering and the restricted content <NUM>, that is rendered in the second state <NUM>, therefore enters bottom right and traverses the field of perception <NUM> towards the top right. In the example of <FIG> it happens to coincide with the observation field of perception <NUM> but this is not necessary. The position of the restricted content <NUM> in the second state <NUM> varies with the perception direction <NUM> and it may entirely avoid the observation field of perception <NUM> or partially fill the observation field of perception <NUM> depending upon the user perception direction <NUM>.

The time-varying content <NUM> comprises different content <NUM> at different relative times t.

The method <NUM> comprises: using a combination of a user perception direction <NUM> at a relative time tn and a rendering field of perception <NUM> to define content <NUM> for rendering at the relative time tn; and rendering in a first state <NUM> at the relative time tn any portion of the content <NUM> for rendering that is not defined as restricted content <NUM> and rendering in a second state <NUM> at the relative time tn, different to the first state <NUM>, any portion of the content <NUM> for rendering that is defined as restricted content <NUM>.

The combination of the user perception direction <NUM> and an observation field of perception define observed content <NUM> at the relative time tn. The observed content <NUM> may be used to define restricted content <NUM> for subsequent times tm (m>n).

In some examples, the time-varying content <NUM> may be a video comprising time-varying images at different relative times tn.

As previously described above, in some but not necessarily all examples, the rendering field of perception <NUM> is one of multiple different rendering fields of perception (defined by different user perception directions <NUM>) that partially overlap to cover a visual scene that extends horizontally and vertically. The visual scene subtends, in the horizontal plane, an angle that may be greater than <NUM>° for example it may be <NUM>°. The visual scene subtends, in the vertical plane, an angle that may be greater than <NUM>°, for example it may be <NUM>°.

In the illustrated examples of <FIG>, the content <NUM> for rendering is visual content. For visual content, the rendering field of perception <NUM> may be a front field of view for the user. The second state may, for example, be a state that is less visible relative to the first state <NUM>. For example it may obscure content, remove content or blur content.

The method <NUM> may, for example, comprise: using a combination of the user perception direction <NUM> at a first relative time and a display field of view <NUM> to define visual content <NUM> for display at the first time; displaying in a first state <NUM> at the first relative time any portion of the visual content <NUM> for display that is not defined as restricted visual content and displaying in a second state <NUM>, different to the first state <NUM>, at the first relative time any portion of the visual content <NUM> for display that is defined as restricted visual content <NUM>; and using a combination of the user perception direction <NUM> and an observation field of view <NUM> to define observed visual content <NUM> at the first relative time. This definition of observed visual content <NUM> may be used to update a definition of restricted visual content <NUM>. This updated definition of restricted visual content <NUM> is used for later iterations of this method <NUM>. As the method <NUM> iterates, the restricted content <NUM> will typically increase with each iteration.

Although in the example embodiments of <FIG>, the content <NUM> for rendering is visual content, in these example embodiments and in other example embodiments the time-varying content <NUM> for rendering may be sound content. For sound content, the rendering field of perception <NUM> may be a front field of hearing for the user. The second state may, for example, be a state that is less audible relative to the first state <NUM>. For example, it may obscure content, remove content or have a lower intensity/volume or may be muted.

The method <NUM> may, for example, comprise: using a combination of the user perception direction <NUM> at a first relative time and a front field of hearing <NUM> to define sound content <NUM> for rendering at the first relative time; rendering in a first state <NUM> at the first relative time any portion of the sound content <NUM> for rendering that is not defined as restricted content and rendering in a second state <NUM>, different to the first state <NUM>, at the first relative time any portion of the sound content <NUM> for rendering that is defined as restricted sound content <NUM>; and using a combination of the user perception direction <NUM> and an observation field of hearing <NUM> to define observed content <NUM> at the first relative time. This definition of observed sound content <NUM> may be used to update a definition of restricted sound content <NUM>. This updated definition of restricted sound content <NUM> is used for later iterations of this method <NUM>. As the method <NUM> iterates, the restricted sound content <NUM> will typically increase with each iteration.

In one example embodiment, there is a <NUM> minute long video. First a user looks in direction A for the first <NUM> seconds observing content C(A[<NUM>-<NUM>]), then immediately in a direction B for <NUM>-<NUM> seconds observing content C(B[<NUM>-<NUM>] and then immediately in the direction A again until the end of the video observing content C(A[<NUM>-<NUM>]). During this first session, the content C(A[<NUM>-<NUM>]), C(B[<NUM>-<NUM>] and C(A[<NUM>-<NUM>]) is not restricted. However, in this example after a single viewing session the observed content becomes restricted. When the user looks at the same video again in a different session, the viewing direction A is restricted between <NUM>-<NUM> seconds and between <NUM>-<NUM> seconds, the content C(A[<NUM>-<NUM>]) and C(A[<NUM>-<NUM>]) is restricted, also the viewing direction B is restricted between <NUM>-<NUM> seconds, the content C(B[<NUM>-<NUM>]) and is restricted.

In the example embodiments illustrated in <FIG>, content is designated as restricted content irrespective of its location within the rendering field of perception <NUM>. In other example embodiments, content may be conditionally designated as restricted content such that it is treated as restricted content only when it occupies a certain location within the rendering field of perception <NUM>. For example, content for rendering may only be restricted content if it occupies a defined area in a plane orthogonal to the perception direction <NUM>. For example, a whole collection of content (e.g. a visual object) for rendering may be restricted content if one part of the collection occupies a defined area in a plane orthogonal to the perception direction <NUM>. For example, a whole collection of content (e.g. a visual object) for rendering may be restricted content only if all of the collection occupies a defined area in a plane orthogonal to the perception direction <NUM>. In some or all of these examples, the defined area may be centered on the perception direction <NUM>. In some or all of these examples, the defined area may be the same as the observation field of perception <NUM>. Thus in some examples, the method <NUM> is as described above in relation to <FIG> but the definition of the observed visual content <NUM> at the first relative time is used to update a definition of restricted visual content <NUM> that is dependent upon when and also where the visual content is rendered. As a consequence, examples such as those illustrated in <FIG>, where there is restricted visual content <NUM> outside the defined area (e.g. the observation field of perception <NUM>) may not occur.

<FIG> illustrates an example embodiment of a headset <NUM> that may be used for rendering content to a user. The headset <NUM> may, for example, be used to render sound content and/or visual content.

When a sound scene is rendered to a listener through a headset <NUM> using binaural audio coding, it may be desirable for the rendered sound space, the virtual space, to remain fixed in real space when the listener turns their head in space. This means the rendered sound space needs to be rotated relative to the head-mounted audio output device by the same amount in the opposite sense to the head rotation. The orientation of the rendered sound space tracks with the rotation of the listener's head (user perception direction) so that the rendered sound space remains fixed in space and does not move with the listener's head. The system may use a transfer function to perform a transformation that rotates the sound objects within the sound space, for example, a head-related transfer function (HRTF) interpolator may be used for binaural audio. Vector-based amplitude panning, (VBAP) may be used for loudspeaker format (e.g. <NUM>) audio.

In this example embodiment, sound content may be rendered using head mounted loudspeakers <NUM> at or near the user's ears and visual content may be displayed to a user using a head-mounted display <NUM>.

The headset <NUM> may be configured to track the user perception direction <NUM>. The headset may for example be configured to track head movement, using for example inertial sensors such as accelerometers or gyroscopes. The headset may for example be configured to tracks eye movement (gaze direction) using a pupil detection system that measures pupil movement.

In some example embodiments, the headset <NUM> may additionally be used for recording sound content and/or visual content that may be provided to another user for operation of the method <NUM>. For example, the headset <NUM> may comprise one or more cameras <NUM> and one or more microphones <NUM>. The camera(s) <NUM> may be arranged to capture a wide field of view and the microphones <NUM> may be arranged to record a sound space with spatial diversity such that spatial audio processing can be performed.

The user of the headset may capture content <NUM> for rendering over a prolonged period by, for example, changing their head orientation such that they capture more than one visual scene and sound scene over time. This captured content <NUM> may be comprised within a message and sent via a transceiver <NUM> to another user with a similar headset <NUM>.

The receiver who receives the message <NUM> via the transceiver <NUM> may then have that message rendered to them using the method <NUM> described above. In this way the originating user is able to send a message to another user that does not last indefinitely. Access to the content is restricted with use.

The sender of the message and the receiver of the message may use different example embodiments of the same apparatus <NUM>.

In some example embodiments, the originating user may define via metadata settings that control the definition of observed content <NUM> and/or the thresholds for designating observed content <NUM> as restricted content and/or define the second state of the content and the extent to which it is restricted. For example, the originating user may specify that the content is for one view only.

<FIG> illustrates an example of a controller suitable for performing the method <NUM> described above.

Implementation of a controller <NUM> may be as controller circuitry. The controller <NUM> may be implemented in hardware alone, have certain aspects in software including firmware alone or can be a combination of hardware and software (including firmware).

The memory <NUM> stores a computer program <NUM> comprising computer program instructions (computer program code) that controls the operation of the apparatus <NUM> when loaded into the processor <NUM>. The computer program instructions, of the computer program <NUM>, provide the logic and routines that enables the apparatus to perform the methods illustrated in <FIG>. The processor <NUM> by reading the memory <NUM> is able to load and execute the computer program <NUM>.

The apparatus <NUM>, as previously described, may be a headset <NUM>. The apparatus <NUM> may part of another type of rendering engine. The apparatus <NUM> may part of a system that controls a headset or other type of rendering engine.

The controller <NUM> may be part of an apparatus <NUM>. The controller <NUM> may, for example, be configured to communicate with components of the apparatus <NUM> including, but not restricted to, one or more of: display <NUM>, audio output <NUM> (e.g. loudspeaker), camera(s) <NUM>, audio input <NUM> (e.g. microphones), user input interface <NUM> and transceiver <NUM>. In some but not necessarily all examples, the user input interface <NUM> may track the user perception direction <NUM>. It may for example comprise inertial sensors such as accelerometers or gyroscopes that track head movement or a pupil detection system that tracks eye movement (gaze direction).

The controller <NUM> may be configured to store a data structure <NUM> that records which content is restricted content. Preferably that data structure is not editable by the user but is only updated automatically as observed content is automatically designated as restricted content according to the methods described above. The data structure 305may be stored locally in the memory <NUM> or may be stored remotely, using transceiver <NUM>, at a remote server.

As illustrated in <FIG>, the computer program <NUM> may arrive at the apparatus <NUM> via any suitable delivery mechanism <NUM>. The delivery mechanism <NUM> may be, for example, a non-transitory computer-readable storage medium, a computer program product, a memory device, a record medium such as a compact disc read-only memory (CD-ROM) or digital versatile disc (DVD), an article of manufacture that tangibly embodies the computer program <NUM>. The delivery mechanism may be a signal configured to reliably transfer the computer program <NUM>. The apparatus <NUM> may propagate or transmit the computer program <NUM> as a computer data signal.

As used in this application, the term 'circuitry' refers to all of the following:.

The blocks illustrated in the Figs referred to above may represent steps in a method and/or sections of code in the computer program <NUM>. The illustration of a particular order to the blocks does not necessarily imply that there is a required or preferred order for the blocks and the order and arrangement of the block may be varied. Furthermore, it may be possible for some blocks to be omitted.

As used here 'module' refers to a unit or apparatus that excludes certain parts/components that would be added by an end manufacturer or a user. The controller <NUM> may, for example, be a module.

The preceding description describes a number of processes and methods. Except where specifically excluded any or all of the processes or methods or steps in the processes or methods may be performed automatically by the controller <NUM>.

If it is intended to use 'comprise' with an exclusive meaning then it will be made clear in the context by referring to "comprising only one" or by using "consisting".

Claim 1:
A method comprising:
obtaining a user perception direction (<NUM>), said user perception direction (<NUM>) being a direction of user orientation or a direction of user gaze;
rendering, in a first state (<NUM>), any portion of content (<NUM>) for rendering that is not defined as restricted content (<NUM>) and, in a second state (<NUM>), different to the first state (<NUM>), any portion of the content (<NUM>) for rendering that is defined as restricted content (<NUM>) wherein content (<NUM>) for rendering is visual content and/or sound content;
and
using a combination of the user perception direction (<NUM>) and an observation field of perception to define restricted content (<NUM>) wherein the observation field of perception defines what is considered to be observed from the user perception direction (<NUM>)
characterized in that
restricted content (<NUM>) is content (<NUM>) for rendering which is considered to be previously observed from a user perception direction (<NUM>).