Patent Description:
Modern computing and display technologies have facilitated development of "augmented reality" viewers. An augment reality viewer is a wearable device that presents the user with two images, one for the left eye and one for the right eye. Objects in the images for each eye are rendered with slightly different viewpoints that allows the brain to process the objects as three-dimensional objects. When the images constantly change viewpoints as the viewer moves, movement around synthetic three-dimensional content can be simulated.

An augmented reality viewer usually includes technology that allows the presentation of digital or virtual image information as an augmentation to visualization of the actual world around the user. In one implementation, the virtual image information is presented in a static location relative to the augmented reality viewer so that, if the user moves their head, and the augmented reality viewer with their head, the user is presented with an image that remains in a stationary position in front of them while real world objects shift in their view. This gives the user the appearance that the virtual image information is not fixed relative to the real world objects, but instead is fixed in the viewer's point of view. In other implementations, technologies exist to keep the virtual image information in a stationary position relative to the real world objects when the user moves their head. In the latter scenario, the user may be given some control over the initial placement of the virtual image information relative to the real world objects.

<CIT> discloses systems and methods for rendering images in a virtual or augmented reality system that may include capturing scene images of a scene in a vicinity of a first and a second projector, capturing spatial data with a sensor array in the vicinity of the first and second projectors, analyzing captured scene images to recognize body parts, and projecting images from each of the first and the second projectors with a shape and orientation determined based on the recognized body parts. Additional rendering operations may include tracking movements of the recognized body parts, applying a detection algorithm to the tracked movements to detect a predetermined gesture, applying a command corresponding to the detected predetermined gesture, and updating the projected images in response to the applied command. Improvement remain however desirable.

The invention provides an augmented reality viewer according to claim <NUM>. Further developments are according to dependent claims <NUM>-<NUM>.

The invention further provides an augmented reality viewing method according to claim <NUM>.

The invention is further described by way of example with reference to the accompanying drawings, wherein:.

The terms "surface" and "surface area" are used herein to describe two-dimensional areas that are suitable for use as display areas. Aspects of the invention may find application when other display areas are used, for example a display area that is a three-dimensional surface area or a display area representing a slice within a three-dimensional volume.

<FIG> of the accompanying drawings illustrates an augmented reality viewer <NUM> that a user uses to see a direct view of a real world scene, including real world surfaces and real world objects <NUM>, that is augmented with content <NUM> of the kind that is stored on, received by, or otherwise generated by a computer or computer network.

The augmented reality viewer <NUM> includes a display <NUM>, a data channel <NUM>, a content rendering module <NUM>, a projector <NUM>, a depth sensor <NUM>, a position sensor such as an accelerometer <NUM>, a camera <NUM>, an environmental calculation unit <NUM>, and a content placement and content orientation unit <NUM>.

The data channel <NUM> may be connected to a storage device that holds the content <NUM> or may be connected to a service that provides the content <NUM> in real time. The content <NUM> may for example be static images such as photographs, images that remain static for a period of time and can be manipulated by a user such as web pages, text documents or other data that is displayed on a computer display, or moving images such as videos or animations. The content <NUM> may be two-dimensional, three-dimensional, static, dynamic, text, image, video, etc. The content <NUM> may include games, books, movies, video clips, advertisements, avatars, drawings, applications, web pages, decorations, sports games, replays, <NUM>-D models or any other type of content as will be appreciated by one of skill in the art.

The content rendering module <NUM> is connected to the data channel <NUM> to receive the content <NUM> from the data channel <NUM>. The content rendering module <NUM> converts the content <NUM> into a form that is suitable for three-dimensional viewing. Various techniques exist for viewing two-dimensional planes in three-dimensional space depending on the orientation of the user, or viewing three-dimensional volumes in three dimensions by the user.

The projector <NUM> is connected to the content rendering module <NUM>. The projector <NUM> converts data generated by the content rendering module <NUM> into light and delivers the light to the display <NUM>. The light travels from the display <NUM> to eyes <NUM> of the user. Various techniques exist for providing the user with a three-dimensional experience. Each eye is provided with a different image and objects in the images are perceived by the user as being constructed in three dimensions. Techniques also exist for the user to focus on the objects at a field of depth that is not necessarily in the plane of the display <NUM> and is typically at some distance behind the display <NUM>. One way that virtual content can be made to appear to be at a certain depth is by causing light rays to diverge and form a curved wavefront in a way that mimics how light from real physical objects reaches an eye. The eye then focuses the diverging light beams onto the retina by changing shape of the anatomic lens in a process called accommodation. Different divergence angles represent different depths and are created using diffraction gratings on the exit pupil expander on the waveguides.

The display <NUM> is a transparent display. The display <NUM> allows the user to see the real world objects <NUM> through the display <NUM>. The user thus perceives an augmented reality view <NUM> wherein the real world objects <NUM> that the user sees in three-dimensions are augmented with a three-dimensional image that is provide to the user from the projector <NUM> via the display <NUM>.

The depth sensor <NUM> and the camera <NUM> are mounted in a position to capture the real world objects <NUM>. The depth sensor <NUM> typically detects electromagnetic waves in the infrared range and the camera <NUM> detects electromagnetic waves in the visible light spectrum. As more clearly shown in <FIG>, more than one camera <NUM> may be mounted on a frame <NUM> of the augmented reality viewer <NUM> in a world-facing position. In the particular embodiment, four cameras <NUM> are mounted to the frame <NUM> with two in a forward world-facing position and two in a left and right side or oblique world-facing position. The fields of view of the multiple cameras <NUM> may overlap. The depth sensor <NUM> and the cameras <NUM> are mounted in a static position relative to a frame <NUM> of the augmented reality viewer <NUM>. Center points of images that are captured by the depth sensor <NUM> and the camera <NUM> are always in the same, forward direction relative to the augmented reality viewer <NUM>.

The accelerometer <NUM> is mounted in a stationary position to the frame of the augmented reality viewer <NUM>. The accelerometer <NUM> detects the direction of gravitation force. The accelerometer <NUM> can be used to determine the orientation of the augmented reality viewer with respect to the Earth's gravitational field. The combination of the depth sensor <NUM> and a head pose algorithm that relies on visual simultaneous localization and mapping ("SLAM") and inertial measurement unit ("IMU") input, accelerometer <NUM> permits the augmented reality viewer <NUM> to establish the locations of the real world objects <NUM> relative to the direction of gravitation force and relative to the augmented reality viewer <NUM>.

The camera <NUM> captures images of the real world objects <NUM> and further processing of the images on a continual basis provides data that indicates movement of the augmented reality viewer <NUM> relative to the real world objects <NUM>. Because the depth sensor <NUM>, world cameras <NUM>, and the accelerometer <NUM> determine the locations of the real world objects <NUM> relative to gravitation force on a continual basis, the movement of the augmented reality viewer <NUM> relative to gravitation force and a mapped real world environment can also be calculated.

In <FIG>, the environmental calculation unit <NUM> includes an environment mapping module <NUM>, a surface extraction module <NUM> and a viewer orientation determination module <NUM>. The environment mapping module <NUM> may receive input from one or more sensors. The one or more sensors may include, for example, the depth sensor <NUM>, one or more world camera <NUM>, and the accelerometer <NUM> to determine the locations of the real world surfaces and objects <NUM>. The surface extraction module <NUM> is may receive data from the environment mapping module <NUM> and determines planar surfaces in the environment. The viewer orientation determination module <NUM> is connected to and receives input from the depth sensor <NUM>, the cameras <NUM> , and the accelerometer <NUM> to determine a user orientation of the user relative to the real world objects <NUM> and the surfaces that are identified by the surface extraction module <NUM>.

The content placement and content orientation unit <NUM> includes a surface vector calculator <NUM>, a surface selection module <NUM>, a content size determination module <NUM>, a content vector calculator <NUM> and a content orientation selection module <NUM>. The surface vector calculator <NUM>, the surface selection module <NUM> and content size determination module <NUM> may be sequentially connected to one another. The surface selection module <NUM> is connected to and provides input to the viewer orientation determination module <NUM>. The content vector calculator <NUM> is connected to the data channel <NUM> so as to be able to receive the content <NUM>. The content orientation selection module <NUM> connected to and receives input from the content vector calculator <NUM> and the viewer orientation determination module <NUM>. The content size determination module <NUM> is connected and provides input to the content orientation selection module <NUM>. The content rendering module <NUM> is connected and receives input from the content size determination module <NUM>.

<FIG> illustrates a user <NUM> who is wearing the augmented reality viewer <NUM> within a three-dimensional environment.

A vector <NUM> signifies a direction of gravitation force as detected by one or more sensors on the augmented reality viewer <NUM>. A vector <NUM> signifies a direction to the right from a perspective of the user <NUM>. A user orientation vector <NUM> signifies a user orientation, in the present example a forward direction in the middle of a view of the user <NUM>. The user orientation vector <NUM> also points in a direction that is to the center points of the images captured by the depth sensor <NUM> and camera <NUM> in <FIG>. <FIG> shows a further coordinate system <NUM> that includes the vector <NUM> to the right, the user orientation vector <NUM> and a device upright vector <NUM> that are orthogonal to one another.

The three-dimensional environment, by way of illustration, includes a table <NUM> with a horizontal surface <NUM>, surfaces <NUM> and <NUM>, objects <NUM> that provide obstructions that may make the surfaces <NUM> and <NUM> unsuitable for placement of content. For example, objects <NUM> that disrupt continuous surfaces <NUM> and <NUM> may include picture frames, mirrors, cracks in a wall, rough texture, a different colored area, a hole in the surface, a protrusion of the surface, or any other non-uniformity with respect to the planar surfaces <NUM>, <NUM>. In contrast, the surfaces <NUM> and <NUM> may be more suitable for placement of content because of their relatively large size and their proximity to the user <NUM>. Depending on the type of content being displayed, it may also be advantageous to find a surface having rectangular dimensions, although other shapes such as squares, triangles, circles, ovals, or polygons may also be used.

<FIG> illustrates the functioning of the depth sensor <NUM>, accelerometer <NUM> and environment mapping module <NUM> in <FIG>. The depth sensor <NUM> captures the depth of all features, including objects and surfaces in the three-dimensional environment. The environment mapping module <NUM> receives data, directly or indirectly, from one or more sensors on the augmented reality viewer <NUM>. For example, the depth sensor <NUM> and the accelerometer <NUM> may provide input to the environment mapping module <NUM> for mapping the depth of the three-dimensional environment in three dimensions.

<FIG> also illustrates the functioning of the camera <NUM> and the viewer orientation determination module <NUM>. The camera <NUM> captures an image of the objects <NUM> and surfaces <NUM>. The viewer orientation determination module <NUM> receives an image from the camera <NUM> and processes the image to determine that an orientation of the augmented reality viewer <NUM> that is worn by the user <NUM> is as represented by the user orientation vector <NUM>.

Other methods of mapping a three-dimensional environment may be employed, for example using one or more cameras that are located in a stationary position within a room. However, the integration of the depth sensor <NUM> and the environment mapping module <NUM> within the augmented reality viewer <NUM> provides for a more mobile application.

<FIG> illustrates the functioning of the surface extraction module <NUM> in <FIG>. The surface extraction module <NUM> processes the three-dimensional map that is created in <FIG> to determine whether there are any surfaces that are suitable for placement and viewing of content, in the present example two-dimensional content. The surface extraction module <NUM> determines a horizontal surface area <NUM> and two vertical surface areas <NUM> and <NUM>. The surface areas <NUM>, <NUM> and <NUM> are not real surfaces, but instead electronically represent two-dimensional planar surfaces oriented in a three-dimensional environment. The surface areas <NUM>, <NUM> and <NUM>, which are data representations, correspond respectively to the real surfaces <NUM>, <NUM> and <NUM> in <FIG> forming part of the real world objects <NUM> in <FIG>.

<FIG> illustrates a cube <NUM> and a shadow <NUM> of the cube <NUM>. These elements are used by the author to assist the viewer to track changes in the user orientation vector <NUM> and movement of the user <NUM> and the augmented reality viewer <NUM> in <FIG> through the three-dimensional space.

<FIG> also illustrates the functioning of the surface vector calculator <NUM> in <FIG>. The surface vector calculator <NUM> calculates a surface area orientation vector for each extracted surface of the mapped three-dimensional environment. For example, the surface vector calculator <NUM> calculates a surface area orientation vector <NUM> that is normal to a plane of the surface area <NUM>. Similarly, the surface vector calculator <NUM> calculates a surface area orientation vector <NUM> that is normal to the surface area <NUM> and a surface area orientation vector <NUM> that is normal to the surface area <NUM>.

Selection of a surface on which to display virtual content is done by a surface selection module <NUM> that calculates a relationship between the surface and the user. The surface selection module <NUM> in <FIG> calculates a dot product of the user orientation vector <NUM> and the surface area orientation vector <NUM>. The dot product of unit vectors a and b is represented by the following equation: <MAT> where <MAT> <MAT> θ is the angle between unit vectors a and b.

The user orientation vector <NUM> and the surface area orientation vector <NUM> are orthogonal to one another, which means their dot product is zero.

The surface selection module <NUM> also calculates a dot product of the user orientation vector <NUM> and the surface area orientation vector <NUM>. Because the user orientation vector <NUM> and the surface area orientation vector <NUM> are orthogonal their dot product is zero.

The surface selection module <NUM> also calculates a dot product of the user orientation vector <NUM> and the surface area orientation vector <NUM>. Because the user orientation vector <NUM> and the surface area orientation vector <NUM> are <NUM>° relative to one another, their dot product is -<NUM>. Because the dot product that includes the surface area orientation vector <NUM> is the most negative of the three dot products, the surface selection module <NUM> determines that the surface area <NUM> is the preferred surface area between the surface areas <NUM>, <NUM> and <NUM> for displaying content. The more negative the dot product is, the more likely it will be that content will be oriented to be directly facing the viewer. Because the surface area <NUM> is a vertical surface area, the content placement and content orientation unit <NUM> does not invoke the content orientation selection module <NUM> in <FIG>. The dot product is one of many surface characteristics that can be prioritized by the system or by the needs of the virtual content for choosing the best surface. For example, if the surface that has a dot product of -<NUM> is tiny and is far away from the user, it may not be preferable over a surface that has a dot product of -<NUM> but is large and near to the user. The system may choose a surface that has good contrast ratio properties when placing content, so it will be easier for the user to see. Next, the content size determination module <NUM> determines an appropriate size of content to display on the surface area <NUM>. The content has an optimal aspect ratio, for example an aspect ratio of <NUM> on a near edge and <NUM> on a side edge. The content size determination module <NUM> uses the ratio of the near edge to the side edge to determine the size and shape of the content, preserving this aspect ratio at all viewing angles so as not to distort content. The content size determination module <NUM> calculates the optimal height and width of the content with the optimal aspect ratio that will fit with the surface area <NUM>. In the given example, the distance between left and right edges of the surface area <NUM> determines the size of the content.

<FIG> illustrates the functioning of the content rendering module <NUM> and the projector <NUM> in <FIG>. The content rendering module <NUM> provides the content <NUM> in its calculated orientation to the projector <NUM> based on the size determination of the content size determination module <NUM> and the surface selection module <NUM>. The viewer views the content <NUM> as a rendering <NUM> that is placed in three-dimensional space on and coplanar with the surface area <NUM>. The content <NUM> is not rendered on the surface areas <NUM> and <NUM>. All other surface characteristics being equal, the surface area <NUM> provides an optimal area for the rendering <NUM> when compared to the surface areas <NUM> and <NUM>, because of the user orientation as represented by the user orientation vector <NUM>. The rendering <NUM> remains static on the surface area <NUM> when the user orientation vector changes by a small degree. If the viewer orientation determination module <NUM> in <FIG> senses that the user orientation vector changes by more than a predetermined threshold degree, for example by five degrees, the system automatically proceeds to recalculate all dot-products as described above and, if necessary, reposition and resize the content that is being rendered for display to the user. Alternatively, the system my routinely, e.g. every <NUM> seconds recalculate all dot-products and place content as described above.

Alternatively, the user may select the area <NUM> for the content to remain even when they change their orientation.

In <FIG>, the user <NUM> changes the inclination of their head. As a result, the user orientation vector <NUM> rotates in a downward direction <NUM>. A new user orientation is represented by a new user orientation vector <NUM>. The cameras <NUM> in <FIG> and <FIG> continually capture images of the real world objects <NUM>. Additional sensors such as the depth sensor <NUM> and the accelerometer <NUM> may also continually capture and provide updated information. The viewer orientation determination module <NUM> processes the images, along with other data captured by sensors on board the augmented reality viewer <NUM>, to determine relative movement of the real world objects <NUM> within a view of the camera <NUM>. The viewer orientation determination module <NUM> then processes such movement to determine the change of the user orientation vector from the user orientation vector <NUM> in <FIG> to the user orientation vector <NUM> in <FIG>. The system normally selects the surface with the most optimal dot-product, although there may be some tolerance/range allowable for the dot-product so that jitter and processing is reduced. By way of example, the system may move the content when there is another dot-product that is more optimal and if the dot-product that is more optimal is at least <NUM> percent better than the dot-product of the surface where the content is currently displayed.

Assuming that the user did not select the surface <NUM> for the content to remain after they change their orientation. the surface selection module <NUM> again calculates three dot products, namely between the user orientation vector <NUM> and the surface area orientation vector <NUM>, the user orientation vector <NUM> and the surface area orientation vector <NUM>, and the user orientation vector <NUM> and the surface area orientation vector <NUM>. The surface selection module <NUM> then determines which one of the three dot products is the most negative. In the present example, the dot product between the user orientation vector <NUM> and the surface area orientation vector <NUM> is the most negative. The surface selection module <NUM> determines that the surface area <NUM> is the preferred surface because its associated dot product is more negative than for the surface areas <NUM> and <NUM>. The system may also consider other factors as described above.

The content placement and content orientation unit <NUM> in <FIG> invokes the content vector calculator <NUM> and the content orientation selection module <NUM>. Following operation of the content orientation selection module <NUM>, the content size determination module <NUM> is again invoked.

The functioning of the content vector calculator <NUM>, content orientation selection module <NUM> and content size determination module <NUM> are better illustrated with the assistance of <FIG>.

<FIG> illustrates that the content rendering module <NUM> and projector <NUM> create a rendering <NUM> of the content <NUM> within and coplanar with the surface area <NUM>. The rendering on the surface area <NUM> is no longer displayed to the user <NUM>.

The rendering <NUM> has a far edge <NUM>, a near edge <NUM>, a right edge <NUM> and a left edge <NUM>. The content vector calculator <NUM> in <FIG> may calculate a content orientation vector <NUM>. The content orientation vector <NUM> extends from the near edge <NUM> to the far edge <NUM> and is orthogonal to both the near edge <NUM> and the far edge <NUM>.

The calculations that are made by the content vector calculator depend on the content that is provided on the data channel. Some content my already have a content orientation vector extends from the near edge to the far edge of the content, in which case the content vector calculator <NUM> simply identifies and isolates the content orientation vector within the code of the content. In other instances, a content orientation vector may be associated with the content and the content vector calculator <NUM> may have to re-orient the content orientation vector to extend from the near edge to the far edge of the content. In other instances, the content vector calculator <NUM> may generate a content orientation vector based on other data such as image analysis, the placement of tools in the content, etc..

The content orientation selection module <NUM> calculates a dot product between the user orientation vector <NUM> and the content orientation vector <NUM>. The dot product is calculated for four scenarios, namely when the content orientation vector <NUM> is oriented in the direction shown in <FIG>, when the content orientation vector <NUM> is oriented <NUM>° to the right, when the content orientation vector <NUM> is oriented <NUM>°, and when the content orientation vector <NUM> is oriented <NUM>° to the left. The content orientation selection module <NUM> then selects the dot product that is the lowest among the four dot products and places the rendering <NUM> so that the content orientation vector <NUM> is aligned in the direction with the lowest associated dot product. The near edge <NUM> is then located closer to the user <NUM> than the far edge <NUM> and the right and left edges <NUM> and <NUM> are located to the right and to the left from the orientation of the user <NUM> as depicted by the user orientation vector <NUM>. The content <NUM> is thus oriented in a manner that is easily viewable by the user <NUM>. For example, a photograph of a head and torso of a person is displayed with the head farthest from the user <NUM> and the torso closest to the user <NUM>, and a text document is displayed with the first lines farthest from the user <NUM> and the last lines closest to the user <NUM>.

The content size determination module <NUM> has determined an appropriate size for the rendering <NUM> with the right edge <NUM> and the left edge <NUM> defining the width of the rendering <NUM> within the surface area <NUM> and a distance between the far edge <NUM> and the near edge <NUM> being determined by the desired aspect ratio.

In <FIG>, the user <NUM> has moved in a direction <NUM> counterclockwise around the surface area <NUM>. The user <NUM> has also rotated their body counterclockwise by <NUM>°. The user <NUM> has now established a new orientation as represented by a new user orientation vector <NUM>. The user's head is still inclined downward toward the surface area <NUM> and the surface areas <NUM> and <NUM> are now located behind and to the right of the user <NUM>, respectively.

The surface selection module <NUM> again calculates a dot product associated with each one of the surface area orientation vectors <NUM>, <NUM> and <NUM>. The dot product of the user orientation vector <NUM> and the surface area orientation vector <NUM> has now become positive. The dot product between the user orientation vector <NUM> and the surface area orientation vector <NUM> is approximately zero. The dot product between the user orientation vector <NUM> and the surface area orientation vector <NUM> is the most negative. The surface selection module <NUM> in <FIG> selects the surface area <NUM> associated with the surface area orientation vector <NUM> as the preferred surface for positioning of a rendering of the content <NUM>.

The content orientation selection module <NUM> in <FIG> again calculates four dot products, each one associated with a respective direction of a content orientation vector, namely a dot product between the user orientation vector <NUM> and the content orientation vector <NUM> in the direction shown in <FIG>, and further dot products respectively between the user orientation vector <NUM> and content orientation vectors at <NUM>° to the right, <NUM>° and <NUM>° to the left relative to the content orientation vector <NUM> in <FIG>. The content orientation selection module <NUM> determines that the dot product associated with the content orientation vector <NUM> that is <NUM>° to the left relative to the direction of the content orientation vector <NUM> shown in <FIG> is the most positive of the four dot products.

The content size determination module <NUM> then determines an appropriate size for the rendering if the content orientation vector <NUM> is rotated <NUM>° to the left.

<FIG> illustrates how the content rendering module <NUM> creates the rendering <NUM> based on the user orientation as represented by the user orientation vector <NUM>. The rendering <NUM> is rotated <NUM>° counterclockwise so that the content orientation vector <NUM> is directed <NUM>° to the left when compared to <FIG>. The near edge <NUM> is now located closest to the user <NUM>. The content size determination module <NUM> in <FIG> has made the rendering <NUM> smaller than in <FIG> due to the available proportions of the surface area <NUM>. Renderings could snap between positions, smoothly rotate, fade in/fade out as selected by the content creator or by user preference.

In <FIG>, the user <NUM> has moved further around the surface area <NUM> in a direction <NUM> and has established a new user orientation as represented by a new user orientation vector <NUM>. The dot product between the user orientation vector <NUM> and the surface area orientation vector <NUM> is now positive. The dot product between the user orientation vector <NUM> and the surface area orientation vector <NUM> is approximately zero. The dot product between the user orientation vector <NUM> and the surface area orientation vector <NUM> is the most negative. The surface area <NUM> is thus the preferred surface for displaying content.

The dot product between the user orientation vector <NUM> and the content orientation vector <NUM> as shown in <FIG> is approximately zero. If the content orientation vector <NUM> is rotated <NUM>° clockwise, <NUM>° and <NUM>° counterclockwise, the respective dot products differ in magnitude with the dot product of the content orientation vector <NUM> that is <NUM>° to the left being the most positive. The rendering <NUM> should thus be rotated <NUM>° counterclockwise and be resized based on the proportions of the surface area <NUM>. <FIG> illustrates how the rendering <NUM> is rotated and resized due to the change in the user orientation vector <NUM> while remaining on the surface area <NUM>.

In <FIG>, the user <NUM> has moved in a direction <NUM> around the surface area <NUM> and has established a new user orientation as represented by a new user orientation vector <NUM>. A dot product of the user orientation vector <NUM> and the surface area orientation vector <NUM> is now negative. However, a dot product between the user orientation vector <NUM> and the surface area orientation vector <NUM> is more negative. The surface area <NUM> is thus the preferred surface area for creating a rendering of the content <NUM>.

A dot product between the user orientation vector <NUM> and the content orientation vector <NUM> as shown in <FIG> is approximately zero. A dot product between the user orientation vector <NUM> and the content orientation vector <NUM>, if it is rotated <NUM>° to the left, is positive. The rendering <NUM> should thus be rotated counterclockwise while remaining on the surface area <NUM>. <FIG> illustrates the placement, orientation and size of the rendering <NUM> as modified based on the new user orientation vector <NUM>.

<FIG> illustrates a new user orientation vector <NUM> that is established when the user <NUM> rotates their head in an upward direction <NUM>. A dot product between the user orientation vector <NUM> and the surface area orientation vector <NUM> is approximately zero. A dot product between the user orientation vector <NUM> and the surface area orientation vector <NUM> is also approximately zero. A dot product between the user orientation vector <NUM> and the surface area orientation vector <NUM> is, or approaches -<NUM> and is thus the most negative of the three surface-based dot products. The surface area <NUM> is now the preferred surface area for placement of a rendering of the content <NUM>. <FIG> illustrates a rendering <NUM> that is displayed to the user <NUM> on the surface area <NUM>. The rendering on the surface area <NUM> is no longer displayed to the user <NUM>. On vertical surface areas such as the surface area <NUM> and the surface area <NUM>, the near edge <NUM> is always at the bottom.

<FIG> illustrates the algorithm for carrying out the method as described above. At <NUM>, the three-dimensional space is mapped as described with reference to <FIG>. At 152A, B and C, the surface areas are extracted as described with reference to <FIG>. At 154A, B and C, the surface vectors are calculated as described with reference to <FIG>. At <NUM>, a user orientation vector is determined as described with reference to <FIG>. At 158A, B and C, a respective dot product is calculated between the user orientation vector and each respective surface area orientation vector, as described with reference to <FIG>. At <NUM>, a preferred surface area is determined as described with reference to <FIG>.

At <NUM>, a determination is made whether the preferred surface area is vertical. If the preferred surface area is not vertical then, at <NUM>, a direction of a content orientation vector relative far, near, right and left edges of the content is determined as described with reference to <FIG>. Following <NUM>, at 166A, B, C and D, content vectors are calculated at <NUM>°, <NUM>° right, <NUM>° and <NUM>° left as described with reference to <FIG>. At 168A, B, C and D, a dot product is calculated between the user orientation vector and the content orientation vectors calculated at 166A, B, C and D, respectively. At <NUM>, a content orientation is selected as described with reference to <FIG>.

At <NUM>, the size of the content is determined as described with reference to <FIG> and <FIG>. At <NUM>, the content is displayed as described with reference to <FIG> and <FIG>.

Following <NUM>, a new user orientation vector may be determined at <NUM> as described with reference to <FIG>, <FIG>, <FIG>, <FIG> and <FIG>. The process may then be repeated without again calculating the surface area orientation vectors at 154A, B and C.

Referring to <FIG> and <FIG>, an embodiment is shown in perspective view and in top view, respectively, with three-dimensional virtual content <NUM> rendered on a mapped surface <NUM> within an environment <NUM> for viewing by a user <NUM>. In such an embodiment, the principles described above are used to position the three-dimensional virtual content <NUM> that the user <NUM> can view the content as easily and naturally as possible.

The user orientation vector <NUM> is the same as a forward vector of the device <NUM> and is henceforth referred to as the "device forward vector <NUM>". Determining a surface on which to place three-dimensional virtual content <NUM> may rely, at least in part, on a dot product relationship between a device forward vector <NUM> and a surface normal vector <NUM> of mapped surfaces in the environment <NUM>. For optimal viewing of the three-dimensional virtual content <NUM>, one of many dot product relationships may be considered optimal depending on the content. For example, if the content is meant to be viewed from the side, it may be ideal for the dot product relationship between the device forward vector <NUM> and the surface normal vector <NUM> to be close to zero indicating that the user is nearly orthogonal to the mapped surface <NUM>. In such an embodiment, the three-dimensional virtual content <NUM> placed on the mapped surface <NUM> will be seen by the user from the side. Alternatively, a dot product relationship at or near -<NUM> may be more desirable if the three-dimensional virtual content <NUM> is meant to be viewed from above, as has been described herein with respect to other embodiments. The ideal dot product relationship may be an attribute set by the creator of the three-dimensional virtual content <NUM>, may be selected as a preference by the user, or may be otherwise determined by the augmented reality viewing system based on the type of content to be displayed.

Once a placement surface is determined, either by the system or by placement by a user, orientation of the three-dimensional virtual content <NUM> on the mapped surface <NUM> is determined with respect to the user. In the example shown, three-dimensional virtual content <NUM> is provided with a content orientation vector <NUM> that may be used to align the three-dimensional virtual content <NUM> to a reference vector of the user device. The three-dimensional virtual content <NUM> is the head of a character with a near edge of the character being where its mouth is. A far edge of the character will typically not be rendered for viewing by the user <NUM> because the far edge is on a side of the character that the user cannot see. The content orientation vector <NUM> is aligned parallel with the near edge of the character. The content orientation vector <NUM> may be used to align the three-dimensional virtual content <NUM> with the augmented reality viewer <NUM> such that the dot product between the content orientation vector <NUM> and the device right vector <NUM> is at or near <NUM>, indicating that the two vectors are pointing in substantially the same direction.

Referring to <FIG> and <FIG>, examples of three-dimensional content re-orientation based on a user's movement are shown. In <FIG>, the user <NUM> has moved clockwise around the table by a certain distance and angle with respect to <FIG>. As a result, the dot product relationship between the content orientation vector <NUM> and the device right vector <NUM> is less than <NUM>. In some embodiments, this change in position may not require re-orientation of three-dimensional virtual content <NUM>. For example, a content creator, a user, or software within the augmented reality viewer <NUM> may indicate that re-orientation of three-dimensional virtual content <NUM> is necessary only when the dot product between the content orientation vector <NUM> and a device reference vector is less than a predetermined threshold. Large or small threshold tolerances may be set depending on the type of content being displayed.

If the change in position of the user <NUM> from the location of <FIG> to the location of <FIG> triggers a re-orientation of three-dimensional virtual content <NUM>, the orientation module may re-render three-dimensional virtual content <NUM> such that the content orientation vector <NUM> aligns with the device right vector <NUM> to result in a dot product equal to or near <NUM> for the two vectors, as shown in <FIG>. As discussed above, re-orientation of three-dimensional virtual content <NUM> may also allow for resizing of the content; however, content may also remain the same size such that it appears only to re-orient about an axis normal to the mapped surface <NUM> as the user moves within the environment.

Referring to <FIG>, <FIG> and <FIG>, an example is shown of virtual content <NUM> re-orientation on a vertical surface <NUM>. In <FIG>, a user <NUM> is shown viewing virtual content <NUM> on a vertical surface <NUM> that is oriented vertically in the environment. The virtual content <NUM> may have at least one of a content right orientation vector <NUM> and a content upright orientation vector <NUM> which may be used to measure alignment with respect to the device right vector <NUM> and the device upright vector <NUM>, respectively. In <FIG>, the alignment between one of the content orientation vectors (<NUM>, <NUM>) and the corresponding device orientation vectors (<NUM>, <NUM>) results in a dot product value of approximately <NUM>. As discussed above, dot product values closer to <NUM> indicate more similar alignment between the two vectors being compared.

If the user <NUM> were to change positions, for example by lying down on a couch as shown in <FIG>, without re-orientation of the virtual content <NUM>, the alignment between content orientation vectors (<NUM>, <NUM>) and corresponding device orientation vectors (<NUM>, <NUM>) may be near zero, indicating a less optimal alignment between the user <NUM> and the virtual content <NUM> than the alignment shown in <FIG>. If a dot product relationship of zero is less than the required dot product relationship for the virtual content-to-user relative orientation, virtual content <NUM> may be re-rendered at a new orientation, as shown in <FIG>, such that the dot product relationships are within the predetermined thresholds. In some embodiments, re-rendering the virtual content <NUM> at a new orientation may re-establish optimal dot product relationships between content orientation vectors (<NUM>, <NUM>) and corresponding device orientation vectors (<NUM>, <NUM>).

<FIG> shows a diagrammatic representation of a machine in the exemplary form of a computer system <NUM> within which a set of instructions, for causing the machine to perform any one or more of the methodologies discussed herein, may be executed. In alternative embodiments, the machine operates as a standalone device or may be connected (e.g., networked) to other machines. Further, while only a single machine is illustrated, the term "machine" shall also be taken to include any collection of machines that individually or jointly execute a set (or multiple sets) of instructions to perform any one or more of the methodologies discussed herein.

The exemplary computer system <NUM> includes a processor <NUM> (e.g., a central processing unit (CPU), a graphics processing unit (GPU) or both), a main memory <NUM> (e.g., read only memory (ROM), flash memory, dynamic random access memory (DRAM) such as synchronous DRAM (SDRAM) or Rambus DRAM (RDRAM), etc.), and a static memory <NUM> (e.g., flash memory, static random access memory (SRAM), etc.), which communicate with each other via a bus <NUM>.

The computer system <NUM> may further include a disk drive unit <NUM>, and a network interface device <NUM>.

The disk drive unit <NUM> includes a machine-readable medium <NUM> on which is stored one or more sets of instructions <NUM> (e.g., software) embodying any one or more of the methodologies or functions described herein. The software may also reside, completely or at least partially, within the main memory <NUM> and/or within the processor <NUM> during execution thereof by the computer system <NUM>, the main memory <NUM> and the processor <NUM> also constituting machine-readable media.

The software may further be transmitted or received over a network <NUM> via the network interface device <NUM>.

While the machine-readable medium <NUM> is shown in an exemplary embodiment to be a single medium, the term "machine-readable medium" should be taken to include a single medium or multiple media (e.g., a centralized or distributed database, and/or associated caches and servers) that store the one or more sets of instructions. The term "machine-readable medium" shall also be taken to include any medium that is capable of storing, encoding, or carrying a set of instructions for execution by the machine and that cause the machine to perform any one or more of the methodologies of the present invention. The term "machine-readable medium" shall accordingly be taken to include, but not be limited to, solid-state memories, optical and magnetic media, and carrier wave signals.

Claim 1:
An augmented reality viewer (<NUM>) comprising:
a display (<NUM>) that permits a user to see real world objects;
a data channel (<NUM>) to hold content;
a user orientation determination module (<NUM>) to determine a user orientation vector (<NUM>, <NUM>, <NUM>, <NUM>), being a unit vector indicative of a first user orientation of a user relative to a first display area and to determine a second user orientation of the user relative to the first display area;
a projector (<NUM>) connected to the data channel (<NUM>) to display the content through the display (<NUM>) to the user within confines of the first display area while the user views the real-world objects;
a content vector calculator (<NUM>) to calculate a content orientation vector (<NUM>, <NUM>, <NUM>, <NUM>), being a unit vector relative to a near edge (<NUM>) of the content; and
a content orientation selection module (<NUM>) connected to the user orientation determination module (<NUM>) and the content vector calculator (<NUM>) to display the content in a first content orientation relative to the first display area so that the near edge (<NUM>) of the content is close to the user when the user is in the first user orientation, and display the content in a second content orientation relative to the first display area so that the near edge (<NUM>) is rotated closer to the user when the user is in the second user orientation,
characterised in that
the content orientation selection module (<NUM>) determines four dot products, respectively (i) of the user orientation vector (<NUM>, <NUM>, <NUM>, <NUM>) and the content orientation vector (<NUM>, <NUM>, <NUM>, <NUM>), (ii) of a vector <NUM> degrees to the left of the user orientation vector (<NUM>, <NUM>, <NUM>, <NUM>) and the content orientation vector (<NUM>, <NUM>, <NUM>, <NUM>), (iii) of a vector <NUM> degrees from the user orientation vector (<NUM>, <NUM>, <NUM>, <NUM>) and the content orientation vector (<NUM>, <NUM>, <NUM>, <NUM>), (iv) of a vector <NUM> degrees to the right of the user orientation vector (<NUM>, <NUM>, <NUM>, <NUM>) and the content orientation vector (<NUM>, <NUM>, <NUM>, <NUM>), and the content is rotated relative to the first display area from the first content orientation to a select orientation corresponding to a select one of the dot products that is the most positive.