Content browsing

A method, apparatus and computer program is provided. The method comprises: causing display of a panoramic menu in which a first panoramic menu item and a second panoramic menu item are simultaneously displayed, wherein the first panoramic menu item extends beyond a user's field of view and is associated with first mediated reality content, and the second panoramic menu item extends beyond a user's field of view and is associated with second mediated reality content, different from the first mediated reality content.

RELATED APPLICATION

This application was originally filed as Patent Cooperation Treaty Application No. PCT/FI2016/050454 filed Jun. 22, 2016 which claims priority benefit to European Patent Application No. 15175340.7, filed Jul. 3, 2015.

TECHNOLOGICAL FIELD

Embodiments of the present invention relate to content browsing. In particular, they relate to browsing mediated reality content.

BACKGROUND

Mediated reality refers to a user experiencing an environment that is fully or partially artificial/virtual.

Augmented reality is a form of mediated reality in which a user experiences a partially artificial/virtual, partially real environment. Virtual reality is a form of mediated reality in which a user experiences a fully artificial/virtual environment.

BRIEF SUMMARY

According to various, but not necessarily all, embodiments of the invention there is provided a method, comprising: causing display of a panoramic menu in which a first panoramic menu item and a second panoramic menu item are simultaneously displayed, wherein the first panoramic menu item extends beyond a user's field of view and is associated with first mediated reality content, and the second panoramic menu item extends beyond a user's field of view and is associated with second mediated reality content, different from the first mediated reality content.

According to various, but not necessarily all, embodiments of the invention there is provided computer program code that, when performed by at least one processor, causes at least the following to be performed: causing display of a panoramic menu in which a first panoramic menu item and a second panoramic menu item are simultaneously displayed, wherein the first panoramic menu item extends beyond a user's field of view and is associated with first mediated reality content, and the second panoramic menu item extends beyond a user's field of view and is associated with second mediated reality content, different from the first mediated reality content.

According to various, but not necessarily all, embodiments of the invention there is provided an apparatus, comprising: means for causing display of a panoramic menu in which a first panoramic menu item and a second panoramic menu item are simultaneously displayed, wherein the first panoramic menu item extends beyond a user's field of view and is associated with first mediated reality content, and the second panoramic menu item extends beyond a user's field of view and is associated with second mediated reality content, different from the first mediated reality content.

According to various, but not necessarily all, embodiments of the invention there is provided an apparatus, comprising: at least one processor; and at least one memory storing computer program code that is configured, working with the at least one processor, to cause the apparatus to perform at least: causing display of a panoramic menu in which a first panoramic menu item and a second panoramic menu item are simultaneously displayed, wherein the first panoramic menu item extends beyond a user's field of view and is associated with first mediated reality content, and the second panoramic menu item extends beyond a user's field of view and is associated with second mediated reality content, different from the first mediated reality content.

According to various, but not necessarily all, embodiments of the invention there is provided examples as claimed in the appended claims.

DESCRIPTION

Embodiments of the invention relate to content browsing. In particular, they relate to providing a panoramic menu in a mediated reality environment.

In embodiments of the invention, a panoramic menu is displayed which comprises at least first and second panoramic menu items that extend beyond a user's field of view. The first panoramic menu item is associated with first mediated reality content and the second menu item is associated with second mediated reality content.

The panoramic menu may be re-arranged based on an identified object of interest.

The panoramic menu provides a particularly efficient, intuitive way of browsing through and selecting mediated reality content and is described in further detail below.

Definitions

In this document, the following definitions apply:

“field of view” refers to extent of the observable world that is visible to a user at a particular instance in time;

“virtual space” refers to fully or partially artificial environment, which may be three dimensional;

“virtual scene” refers to a representation of the virtual space viewed from a particular point of view within the virtual space;

“real space” refers to a real environment, which may be three dimensional;

“real scene” refers to a representation of the real space viewed from a particular point of view within the real space;

“mediated reality” in this document refers to a user visually experiencing a fully or partially artificial environment (a virtual space) as a virtual scene at least partially displayed by a computer to a user. The virtual scene is determined by a point of view within the virtual space and a field of view. Displaying the virtual scene means providing it in a form that can be seen by the user;

“mediated reality content” is content which enables a user to visually experience a fully or partially artificial environment (a virtual space) as a virtual scene. Mediated reality content could include interactive content such as a video game or non-interactive content such as motion video;

“augmented reality” in this document refers to a form of mediated reality in which a user visually experiences a partially artificial environment (a virtual space) as a virtual scene comprising a real scene of a physical real world environment (real space) supplemented by one or more visual elements displayed by an apparatus to a user;

“augmented reality content” is a form of mediated reality content which enables a user to visually experience a partially artificial environment (a virtual space) as a virtual scene. Augmented reality content could include interactive content such as a video game or non-interactive content such as motion video;

“virtual reality” in this document refers to a form of mediated reality in which a user visually experiences a fully artificial environment (a virtual space) as a virtual scene displayed by an apparatus to a user;

“virtual reality content” is a form of mediated reality content which enables a user to visually experience a fully artificial environment (a virtual space) as a virtual scene. Virtual reality content could include interactive content such as a video game or non-interactive content such as motion video;

“perspective-mediated” as applied to mediated reality, augmented reality or virtual reality means that user actions determine the point of view within the virtual space, changing the virtual scene;

“first person perspective-mediated” as applied to mediated reality, augmented reality or virtual reality means perspective mediated with the additional constraint that the user's real point of view determines the point of view within the virtual space;

“user interactive-mediated” as applied to mediated reality, augmented reality or virtual reality means that user actions at least partially determine what happens within the virtual space; and

“displaying” means providing in a form that is perceived visually by the user.

DETAILED DESCRIPTION

FIGS. 1A-1C and 2A-2Cillustrate examples of mediated reality. The mediated reality may be augmented reality or virtual reality.

FIGS. 1A, 1B, 1Cillustrate the same virtual space20comprising the same virtual objects21, however, each Fig illustrates a different point of view24. The position and direction of a point of view24can change independently. The direction (but not the position) of the point of view24changes fromFIG. 1AtoFIG. 1B. The direction and the position of the point of view24changes fromFIG. 1BtoFIG. 1C.

FIGS. 2A, 2B, 2Cillustrate a virtual scene22from the perspective of the different points of view24of respectiveFIGS. 1A, 1B, 1C. The virtual scene22is determined by the point of view24within the virtual space20and a field of view26. The virtual scene22is at least partially displayed to a user.

The virtual scenes22illustrated may be mediated reality scenes, virtual reality scenes or augmented reality scenes. A virtual reality scene displays a fully artificial virtual space20. An augmented reality scene displays a partially artificial, partially real virtual space20.

The mediated reality, augmented reality or virtual reality may be user interactive-mediated. In this case, user actions at least partially determine what happens within the virtual space20. This may enable interaction with a virtual object21such as a visual element28within the virtual space20.

The mediated reality, augmented reality or virtual reality may be perspective-mediated. In this case, user actions determine the point of view24within the virtual space20, changing the virtual scene22. For example, as illustrated inFIGS. 1A, 1B, 1Ca position23of the point of view24within the virtual space20may be changed and/or a direction or orientation25of the point of view24within the virtual space20may be changed. If the virtual space20is three-dimensional, the position23of the point of view24has three degrees of freedom e.g. up/down, forward/back, left/right and the direction25of the point of view24within the virtual space20has three degrees of freedom e.g. roll, pitch, yaw. The point of view24may be continuously variable in position23and/or direction25and user action then changes the position and/or direction of the point of view24continuously. Alternatively, the point of view24may have discrete quantised positions23and/or discrete quantised directions25and user action switches by discretely jumping between the allowed positions23and/or directions25of the point of view24.

FIG. 3Aillustrates a real space10comprising real objects11that partially corresponds with the virtual space20ofFIG. 1A. In this example, each real object11in the real space10has a corresponding virtual object21in the virtual space20, however, each virtual object21in the virtual space20does not have a corresponding real object11in the real space10. In this example, one of the virtual objects21, the computer-generated visual element28, is an artificial virtual object21that does not have a corresponding real object11in the real space10.

A linear mapping exists between the real space10and the virtual space20and the same mapping exists between each real object11in the real space10and its corresponding virtual object21. The relative relationship of the real objects11in the real space10is therefore the same as the relative relationship between the corresponding virtual objects21in the virtual space20.

FIG. 3Billustrates a real scene12that partially corresponds with the virtual scene22ofFIG. 1B, it includes real objects11but not artificial virtual objects. The real scene is from a perspective corresponding to the point of view24in the virtual space20ofFIG. 1A. The real scene12content is determined by that corresponding point of view24and the field of view26.

FIG. 2Amay be an illustration of an augmented reality version of the real scene12illustrated inFIG. 3B. The virtual scene22comprises the real scene12of the real space10supplemented by one or more visual elements28displayed by an apparatus to a user. The visual elements28may be a computer-generated visual element. In a see-through arrangement, the virtual scene22comprises the actual real scene12which is seen through a display of the supplemental visual element(s)28. In a see-video arrangement, the virtual scene22comprises a displayed real scene12and displayed supplemental visual element(s)28. The displayed real scene12may be based on an image from a single point of view24or on multiple images from different points of view24at the same time, processed to generate an image from a single point of view24.

FIG. 4illustrates an example of an apparatus30that is operable to enable mediated reality and/or augmented reality and/or virtual reality.

The apparatus30comprises a display32for providing at least parts of the virtual scene22to a user in a form that is perceived visually by the user. The display32may be a visual display that provides light that displays at least parts of the virtual scene22to a user. Examples of visual displays include liquid crystal displays, organic light emitting displays, emissive, reflective, transmissive and transflective displays, direct retina projection display, near eye displays etc.

The display32is controlled in this example but not necessarily all examples by a controller42.

Implementation of a controller42may be as controller circuitry. The controller42may be implemented in hardware alone, have certain aspects in software including firmware alone or can be a combination of hardware and software (including firmware).

As illustrated inFIG. 4the controller42may be implemented using instructions that enable hardware functionality, for example, by using executable computer program instructions48in a general-purpose or special-purpose processor40that may be stored on a computer readable storage medium (disk, memory, etc.) to be executed by such a processor40.

The processor40is configured to read from and write to the memory46. The processor40may also comprise an output interface via which data and/or commands are output by the processor40and an input interface via which data and/or commands are input to the processor40.

The memory46stores a computer program48comprising computer program instructions (computer program code) that controls the operation of the apparatus30when loaded into the processor40. The computer program instructions, of the computer program48, provide the logic and routines that enables the apparatus to perform the methods illustrated inFIGS. 5 & 6. The processor40by reading the memory46is able to load and execute the computer program48.

The apparatus30therefore comprises: at least one processor40; and at least one memory46including computer program code48the at least one memory46and the computer program code48configured to, with the at least one processor40, cause the apparatus30at least to perform: causing display of a panoramic menu in which a first panoramic menu item and a second panoramic menu item are simultaneously displayed, wherein the first panoramic menu item extends beyond a user's field of view and is associated with first mediated reality content, and the second panoramic menu item extends beyond a user's field of view and is associated with second mediated reality content, different from the first mediated reality content.

The computer program48may arrive at the apparatus30via any suitable delivery mechanism. The delivery mechanism may be, for example, a non-transitory computer-readable storage medium, a computer program product, a memory device, a record medium such as a compact disc read-only memory (CD-ROM) or digital versatile disc (DVD), an article of manufacture that tangibly embodies the computer program48. The delivery mechanism may be a signal configured to reliably transfer the computer program48. The signal could, for example, be sent over a wireless connection (such as a radio frequency connection) or a wired connection in accordance with one or more protocols. The apparatus30may propagate or transmit the computer program48as a computer data signal.

Although the processor40is illustrated as a single component/circuitry it may be implemented as one or more separate components/circuitry some or all of which may be integrated/removable. The processor40may be a single core or multi-core processor.

As used in this application, the term ‘circuitry’ refers to all of the following:

(b) to combinations of circuits and software (and/or firmware), such as (as applicable):

(i) to a combination of processor(s) or (ii) to portions of processor(s)/software (including digital signal processor(s)), software, and memory(ies) that work together to cause an apparatus, such as a mobile phone or server, to perform various functions) and

(c) to circuits, such as a microprocessor(s) or a portion of a microprocessor(s), that require software or firmware for operation, even if the software or firmware is not physically present.

The apparatus30may enable mediated reality and/or augmented reality and/or virtual reality, for example using the method60illustrated inFIG. 5or a similar method. The controller42stores and maintains a model50of the virtual space20. The model may be provided to the controller42or determined by the controller42. For example, sensors in input circuitry44may be used to create overlapping depth maps of the virtual space from different points of view and a three dimensional model may then be produced.

At block62it is determined whether or not the model of the virtual space20has changed. If the model of the virtual space20has changed the method moves to block66. If the model of the virtual space20has not changed the method moves to block64.

At block64it is determined whether or not the point of view24in the virtual space20has changed. If the point of view24has changed the method moves to block66. If the point of view24has not changed the method returns to block62.

At block66, a two-dimensional projection of the three-dimensional virtual space20is taken from the location23and in the direction25defined by the current point of view24. The projection is then limited by the field of view26to produce the virtual scene22. The method then returns to block62.

Where the apparatus30enables augmented reality, the virtual space20comprises objects11from the real space10and also visual elements28not present in the real space10. The combination of such visual elements28may be referred to as the artificial virtual space.FIG. 6illustrates a method70for updating a model of the virtual space20for augmented reality.

At block72it is determined whether or not the real space10has changed. If the real space10has changed the method moves to block76. If the real space10has not changed the method moves to block74. Detecting a change in the real space10may be achieved at a pixel level using differentiating and may be achieved at an object level using computer vision to track objects as they move.

At block74it is determined whether or not the artificial virtual space has changed. If the artificial virtual space has changed the method moves to block76. If the artificial virtual space has not changed the method returns to block72. As the artificial virtual space is generated by the controller42changes to the visual elements28are easily detected.

At block76, the model of the virtual space20is updated.

The apparatus30may enable user-interactive mediation for mediated reality and/or augmented reality and/or virtual reality. The user input circuitry44detects user actions using user input43. These user actions are used by the controller42to determine what happens within the virtual space20. This may enable interaction with a visual element28within the virtual space20.

The apparatus30may enable perspective mediation for mediated reality and/or augmented reality and/or virtual reality. The user input circuitry44detects user actions. These user actions are used by the controller42to determine the point of view24within the virtual space20, changing the virtual scene22. The point of view24may be continuously variable in position and/or direction and user action changes the position and/or direction of the point of view24. Alternatively, the point of view24may have discrete quantised positions and/or discrete quantised directions and user action switches by jumping to the next position and/or direction of the point of view24.

The apparatus30may enable first person perspective for mediated reality, augmented reality or virtual reality. The user input circuitry44detects the user's real point of view14using user point of view sensor45. The user's real point of view is used by the controller42to determine the point of view24within the virtual space20, changing the virtual scene22. Referring back toFIG. 3A, a user18has a real point of view14. The real point of view may be changed by the user18. For example, a real location13of the real point of view14is the location of the user18and can be changed by changing the physical location13of the user18. For example, a real direction15of the real point of view14is the direction in which the user18is looking and can be changed by changing the real direction of the user18. The real direction15may, for example, be changed by a user18changing an orientation of their head or view point and/or a user changing a direction of their gaze. A head-mounted apparatus30may be used to enable first-person perspective mediation.

The apparatus30may comprise as part of the input circuitry44point of view sensors45for determining changes in the real point of view.

For example, positioning technology such as GPS, triangulation (trilateration) by transmitting to multiple receivers and/or receiving from multiple transmitters, acceleration detection and integration may be used to determine a new physical location13of the user18and real point of view14.

For example, accelerometers, electronic gyroscopes or electronic compasses may be used to determine a change in an orientation of a user's head or view point and a consequential change in the real direction15of the real point of view14.

For example, pupil tracking technology, based for example on computer vision, may be used to track movement of a user's eye or eyes and therefore determine a direction of a user's gaze and consequential changes in the real direction15of the real point of view14.

The apparatus30may comprise as part of the input circuitry44image sensors47for imaging the real space10.

An example of an image sensor47is a digital image sensor that is configured to operate as a camera. Such a camera may be operated to record static images and/or video images In some, but not necessarily all embodiments, cameras may be configured in a stereoscopic or other spatially distributed arrangement so that the real space10is viewed from different perspectives. This may enable the creation of a three-dimensional image and/or processing to establish depth, for example, via the parallax effect.

In some, but not necessarily all embodiments, the input circuitry44comprises depth sensors49. A depth sensor49may comprise a transmitter and a receiver. The transmitter transmits a signal (for example, a signal a human cannot sense such as ultrasound or infrared light) and the receiver receives the reflected signal. Using a single transmitter and a single receiver some depth information may be achieved via measuring the time of flight from transmission to reception. Better resolution may be achieved by using more transmitters and/or more receivers (spatial diversity). In one example, the transmitter is configured to ‘paint’ the real space10with light, preferably invisible light such as infrared light, with a spatially dependent pattern. Detection of a certain pattern by the receiver allows the real space10to be spatially resolved. The distance to the spatially resolved portion of the real space10may be determined by time of flight and/or stereoscopy (if the receiver is in a stereoscopic position relative to the transmitter).

In some but not necessarily all embodiments, the input circuitry44may comprise communication circuitry41in addition to or as an alternative to one or more of the image sensors47and the depth sensors49. Such communication circuitry41may communicate with one or more remote image sensors47in the real space10and/or with remote depth sensors49in the real space10.

In some but not necessarily all embodiments, the input circuitry44may comprise one or more keys/buttons. At least some of the keys/buttons may form part of a physical, hand held controller. Alternatively or additionally, if the apparatus30is head-mounted, at least some of the keys/buttons may be positioned on the head-mounted apparatus.

FIG. 7illustrates an example of the apparatus30in the form of head-mounted apparatus33comprising a display32that displays images to a user. The head-mounted apparatus33is moved when the head of the user moves.

The head-mounted apparatus33may be a see-through arrangement for augmented reality that enables a live real scene12to be viewed while one or more visual elements28are displayed by the display32to the user to provide in combination the virtual scene22. In this case a visor34, if present, is transparent or semi-transparent so that the live real scene12can be viewed through the visor34.

The head-mounted apparatus33may be operated as a see-video arrangement for augmented reality that enables a live or recorded video of a real scene12to be displayed by the display32for viewing by the user while one or more visual elements28are simultaneously displayed by the display32for viewing by the user. The combination of the displayed real scene12and displayed one or more visual elements28provides the virtual scene22to the user. In this case a visor34is opaque and may be used as display32.

Referring back toFIG. 4, an apparatus30may enable user-interactive mediation for mediated reality and/or augmented reality and/or virtual reality. The user input circuitry44detects user actions using user input43. These user actions are used by the controller42to determine what happens within the virtual space20. This may enable interaction with a visual element28within the virtual space20.

The detected user actions may, for example, be gestures performed in the real space10. Gestures may be detected in a number of ways. For example, depth sensors49may be used to detect movement of parts a user18and/or or image sensors47may be used to detect movement of parts of a user18and/or positional/movement sensors attached to a limb of a user18may be used to detect movement of the limb.

Object tracking may be used to determine when an object or user moves. For example, tracking the object on a large macro-scale allows one to create a frame of reference that moves with the object. That frame of reference can then be used to track time-evolving changes of shape of the object, by using temporal differencing with respect to the object. This can be used to detect small scale human motion such as gestures, hand movement, facial movement. These are scene independent user (only) movements relative to the user.

The apparatus30may track a plurality of objects and/or points in relation to a user's body, for example one or more joints of the user's body. In some examples, the apparatus30may perform full body skeletal tracking of a user's body.

The tracking of one or more objects and/or points in relation to a user's body may be used by the apparatus30in gesture recognition and so on.

FIG. 8Aillustrates a flow chart of a first method of displaying a panoramic menu, re-arranging menu items in the panoramic menu and enabling a user to select the menu items.

FIG. 9illustrates a user82viewing (visual) mediated reality content90displayed by the display32of the apparatus30. In the example described below and illustrated in the figures, the mediated reality content90is (visual) virtual reality content that enables the user to experience a fully artificial/virtual environment. In other examples, the mediated reality content90could be augmented reality content that enables the user to experience a partially artificial/virtual, partially real environment.

FIG. 9also illustrates a cylindrical co-ordinate system200. The cylindrical co-ordinate system is defined by a height dimension z, a radial dimension r and an azimuthal dimension ø. The user's field of view may be defined using the azimuthal and height dimensions ø, z.

The mediated reality content90extends around the body of the user82and beyond the field of view of the user82, in the azimuthal dimension ø, when it is viewed. This means that it is not possible for the user82to see the whole of the displayed mediated reality content90at any one instance in time. The mediated reality content90may extend around 180 degrees or more in the azimuthal dimension ø. In some examples, the mediated reality content90may extend around 360 degrees in the azimuthal dimension ø.

In the example illustrated inFIG. 9, the user82is wearing a head-mounted apparatus33that enables him to view the mediated reality content90in a first person perspective-mediated manner. The mediated reality content90extends 360 degrees around the user82in the azimuthal dimension ø, but this is not shown inFIG. 9for clarity purposes.

The whole of the mediated reality content90need not be displayed at the same time. That is, the angular range over which the mediated reality content90may be viewed by the user82by moving his head and/or eyes may be greater than the angular range over which the mediated reality content90is displayed by the display32at a particular instance in time.

As the user82experiences the mediated reality content90, the processor40continually cycles through the method illustrated inFIG. 5and, in the case of augmented reality content,FIG. 6.

The cross-hatched area inFIG. 9labelled with the reference numeral81indicates a central portion of the user's field of view. The central portion81of the user's field of view indicates what the user82is (directly) looking at. When the user82changes his point of view, the central portion81of his field of view changes. It can be seen inFIG. 9that the mediated reality content90includes a (virtual) object80. In this example the (virtual) object80is a person, but it need not be in other examples.

At block801inFIG. 8A, the processor40of the apparatus30identifies that the object80is of interest to the user82This may be at least partly based on a determination that the user82has changed his point of view to look directly towards the object80, as shown inFIG. 10, such that the central portion81of his field of view is aligned with the position of the object80in the mediated reality content90. Alternatively or additionally, the processor40may, for example, determine that the user has changed his point of view to look directly towards the object80by tracking the user's gaze, for instance by using the pupil tracking technology described above.

Alternatively or additionally, the apparatus30may enable the user82may identify that an object is of interest by enabling the user82to select an object using an alternative method of user input. For example, the apparatus30may enable the user to select an object by actuating one or more joysticks/keys/buttons, or by making a gesture.

In some embodiments, the mediated reality content90has associated metadata which identifies the presence and location of one or more objects in the mediated reality content90. That is, the metadata predefines the presence and location of one or more objects in the mediated reality content90. The metadata may identify the presence and location of the object(s) over a time period, and indicate how the presence and location of that/those object(s) in the mediated reality content90changes over the time period.

In some alternative embodiments, instead of using metadata to identify objects in the mediated reality content90the processor40may, for example, perform image processing on the mediated reality content90in order to identify objects80. The processor40of the apparatus30may continually perform image processing on the mediated reality content90such that objects, such as the object80illustrated inFIGS. 9 and 10, are automatically identified as the user82changes his point of view when viewing the mediated reality content90. In other embodiments, the image processing may only be performed in response to an input from the user, which acts as a prompt to perform image processing on objects within the user's field of view or, alternatively, objects within (only) the central portion81of the user's field of view.

After the object80has been identified to be an object of interest to the user82, the processor40determines whether the identified object of interest80is present in other mediated reality content which is different from that being viewed by the user82inFIGS. 8A and 9. The processing that is performed by the processor40in this respect may be done by analyzing metadata associated with the mediated reality content or performing image processing in the manner discussed above.

The processor40then causes a panoramic menu to be displayed by the display32in block802inFIG. 8A. The panoramic menu may be displayed after one or more inputs have been received from the user82.

In this example, if the processor40determines that the identified object of interest80is present in a mediated reality content item, a panoramic menu item for that mediated reality content is included in the panoramic menu100.

FIG. 11illustrates the panoramic menu100being displayed by the display32. In this example, the panoramic menu100comprises first, second and third panoramic menu items101,102and103, which are simultaneously displayed. Each of the panoramic menu items101,102,103is associated with (for example, represents) respective mediated reality content which includes the identified object of interest80, and each menu item101,102,103extends beyond the user's field of view.

The panoramic menu100is itself displayed in mediated reality, such as virtual reality or augmented reality. In the latter case, the user82is able to see one or more real objects in addition to the panoramic menu100while the panoramic menu100is being displayed by the display32. In the example illustrated inFIG. 11, the user82is able to view the panoramic menu100in a first person perspective-mediated manner via the head-mounted apparatus33. As the user82views the panoramic menu100, the processor40continually cycles through the method illustrated inFIG. 5(in the case of the panoramic menu100being shown in virtual reality) orFIG. 6(in the case of the panoramic menu being shown in augmented reality).

In the illustrated example, a panoramic menu item101,102,103for a particular mediated reality content item is a static image that is a spatial slice of that mediated reality content captured at an instance in time. Each panoramic menu item101,102,103has a reduced height in the height dimension z relative to the mediated reality content it represents.

For example, upper and lower portions of a static image derived from the mediated reality content item may be cut off to form the panoramic menu item for that mediated reality content item.

The panoramic menu items101,102,103are displayed vertically relative to one another and may form at least part of a virtual tube. The virtual tube could be a closed tube such that the panoramic menu items101,102,103extend around 360 degrees in the azimuthal dimension ø, or an open tube such that the panoramic menu items101,102,103extend around less than 360 degrees in the azimuthal dimension ø. The cross section of the virtual tube could, for example, be circular or elliptical. The length of the virtual tube is aligned with the height dimension z in the figures.

The second panoramic menu item102is positioned above the first panoramic menu item101. The third panoramic menu item103is positioned above the second panoramic menu item102.

In other examples, the panoramic menu items101,102,103may be displayed horizontally relative to one another and form at least part of a virtual tube that is orthogonal to that illustrated in the figures.

While three panoramic menu items101,102,103are illustrated inFIG. 11, more or fewer panoramic menu items101,102,103may be present in other examples.

The panoramic menu items101-103extend beyond the field of view of the user82in the azimuthal dimension ø when they are viewed by the user82. This means that it is not possible for the user82to see the whole of any individual one of the panoramic menu items101-103at any one instance in time. Each of the panoramic menu items101-103may, for example, extend around 180 degrees or more in the azimuthal dimension ø. In some examples, each of the panoramic menu items101-103may extend around 360 degrees in the azimuthal dimension ø.

InFIG. 11, the identified object of interest80is included in the first and second panoramic menu items101,102and is visible to the user82. The identified object of interest80is not currently visible to the user82in the third panoramic menu item103, but is present at a different location on the third panoramic menu item103which the user82cannot currently see.

In block803inFIG. 8A, the panoramic menu items101-103are (re)arranged based on the identified object of interest80.FIG. 12illustrates first, second and third panoramic menu items101,102,103being re-arranged. In the illustrated example, the panoramic menu items101-103are re-arranged based on the location of the object of interest80in the menu items101-103relative to the user's point of view.

InFIG. 12, it can be seen that the central portion81of the user's field of view is aligned with the location of the object of interest80in the first panoramic menu item101in both the height dimension z and the azimuthal dimension ø. The processor40determines that the location of the object of interest80in the first panoramic menu item101is aligned with the central portion81of the user's field of view in the azimuthal dimension ø, so it refrains from moving the first panoramic menu item101when the re-arrangement takes place.

The processor40determines that the object of interest80in both the second panoramic menu item102and the third panoramic menu item103is not aligned with the central portion81of the user's field of view in the azimuthal dimension ø. In response, the processor40moves the second panoramic menu item102by causing it to rotate in the azimuthal dimension ø as indicated by the arrow labelled with the reference numeral112inFIG. 12. The processor40also moves the third panoramic menu item103by causing it to rotate in the azimuthal dimension ø in as indicated by the arrow labelled with the reference numeral113inFIG. 12. It can be seen inFIG. 12that after the processor40has caused the second and third panoramic menu items102,103to rotate, the object of interest80in each of the first, second and third panoramic menu items101-103is aligned in the azimuthal dimension ø (and stacked above one another in the height dimension z).

The processor40may also re-order the panoramic menu items101-103in the height dimension z depending upon whether the object of interest80is present in mediated reality content associated with each of the panoramic menu items101-103. For example, panoramic menu items101-103that are associated with mediated reality content which includes the object of interest80may be positioned closer to the user82than panoramic menu items101-103which are associated with mediated reality content that does not include the object of interest80. In some embodiments, panoramic menu items101-103that are associated with mediated reality content that does not include the object of interest80may be removed from the panoramic menu100.

While the panoramic menu items101-103are re-ordered in the height dimension z in this illustrated example, in other examples the panoramic menu items101-103may be re-ordered in a different manner.

In block804inFIG. 8A, the processor40enables the user to browse the panoramic menu100after it has been re-arranged.FIG. 13illustrates the user browsing the panoramic menu100. The arrows labelled with the reference numerals121and122schematically illustrate the user's ability to browse the panoramic menu items101,102,103by redirecting his field of view in the height dimension z.FIG. 13illustrates a situation where the user82has re-directed the central portion81of his field of view so that it is aligned with the object of interest80in the second panoramic item102.

Alternatively or additionally, the apparatus30may enable the user82to browse the panoramic items101,102,103using an alternative method of user input. For example, the apparatus30may enable the user to select an object by actuating one or more joysticks/keys/buttons or making a gesture.

The user82then selects one of the panoramic menu items101-103. In block805ofFIG. 8A, the processor40responds to the user selection by causing mediated reality content91associated with the selected panoramic menu item to be displayed by the display32.FIG. 14illustrates the mediated reality content91being displayed by the display32.

The apparatus30/33may enable the user82may select one of the panoramic menu items101-103by providing user input in a number of different ways. For example, the apparatus30/33may enable the user to select a panoramic menu item101-103(merely) by aligning the central portion81of his field of view at a particular panoramic menu item101-103for more than a threshold period of time. In some examples, the user82may be required to confirm the selection by pressing a key/button or making a gesture. Alternatively, the central portion81of the user's field of view might not be used to perform the selection at all and instead the user82may perform his selection by solely making a gesture and/or pressing one or more keys/buttons.

FIG. 8Billustrates a flow chart of a second method of displaying a panoramic menu, re-arranging menu items in the panoramic menu and enabling a user to select the menu items.

In block806ofFIG. 8B, the processor40causes the display32to display the panoramic menu100in the same manner as that described above in relation toFIG. 8A.

FIG. 15illustrates a situation in which the displayed panoramic menu100includes fourth, fifth and sixth panoramic menu items104-106, which are simultaneously displayed. It can also be seen inFIG. 15that the fourth, fifth and sixth panoramic menu items104-106include a first object80and the fourth and sixth panoramic menu items104,106include a second object83. InFIG. 15, the central portion81of the user's field of view is presently aligned (in the height dimension z and the azimuthal dimension ø inFIG. 15) with the first object80in the fourth panoramic menu item104. This may be because the first object80has previously been identified to be an object of interest (for instance, as a consequence of user selection, as described above in relation to block805inFIG. 8A). The user82may browse the panoramic menu items104-106and select a panoramic menu item104-106in the same manner as that described above.

The user82then changes his point of view such that the central portion81of his field of view is aligned with the second object of interest83in the fourth panoramic menu item104, as illustrated inFIG. 16. In block807inFIG. 8B, the user82provides the necessary input(s) to select the second object83in the fourth panoramic menu item104such that it is identified by the processor40to be an object of interest. The second object83may replace the first object80as the object of interest, if the first object80had previously been identified to be the object of interest.

In block808inFIG. 8B, the processor40re-arranges the panoramic menu items104-106based on the identification of second object83as the object of interest, following the user's selection in block807inFIG. 8.

FIG. 17illustrates the panoramic menu items104-106being re-ordered based on the identification of the second object83as the object of interest. The order of the panoramic menu items104-106may, for instance, be changed in the vertical dimension z In this example, the processor40causes the sixth panoramic menu item106and the fifth panoramic menu item105to change places, such that the sixth panoramic menu item106is positioned beneath the fifth panoramic menu item105. This is because the processor40has determined that the mediated reality content associated with the sixth panoramic menu item106includes the identified (second) object of interest83(and, as a consequence, in this example the sixth panoramic menu item106also includes the identified object of interest83), whereas the mediated reality content associated with the fifth panoramic menu item105does not (and, as a consequence, in this example neither does the fifth panoramic menu item105).

The upwards movement of the fifth panoramic menu item105is illustrated by the arrow labelled with the reference numeral141inFIG. 17. The downwards movement of the sixth panoramic menu item106is illustrated by the arrow labelled with the reference numeral142inFIG. 17.

The sixth panoramic menu item106is brought closer to the user82than the fifth panoramic menu item105because the processor40has identified that its associated mediated reality content is more likely to be of interest to the user82than that associated with the fifth panoramic menu item105.

FIG. 18illustrates the panoramic menu items104-106being rotated based on the identification of second object83as the object of interest, following the user's selection, in the same manner as that described above in relation toFIG. 12. In this example, the processor40determines that the identified second object of interest83is not aligned with the central portion81of the user's field of view in the azimuthal dimension ø. In response, the processor40moves the sixth panoramic menu item106by causing it to rotate in the azimuthal dimension ø, as indicated by the arrow labeled with the reference numeral151inFIG. 18. It can be seen inFIG. 18that after the processor40has caused the sixth panoramic menu item106to rotate, the identified second object of interest83in the fourth and sixth panoramic menu items104,106is aligned with the central portion81of the user's field of view in the azimuthal dimension ø (and stacked above one another in the height dimension z).

An efficient and intuitive way of presenting and browsing menu items representing mediated reality content has been described above. The panoramic menu100described above provides a particularly immersive experience and therefore represents an appropriate gateway for mediated reality content.

Although embodiments of the present invention have been described in the preceding paragraphs with reference to various examples, it should be appreciated that modifications to the examples given can be made without departing from the scope of the invention as claimed. For example, while embodiments of the invention are described above in the context of a head-mounted apparatus33, a head-mounted apparatus is not necessarily required. For example, the apparatus30could include a holographic display that is not head-mounted. Alternatively, the mediated reality content90,91and the panoramic menu100could be displayed on or more physical displays, some or all of which could be curved.