Patent Description:
Augmented reality views change when the environment of an augmented reality device changes. <CIT> discloses a device which includes at least one processor, a camera accessible to the at least one processor, a display accessible to the at least one processor, and storage accessible to the at least one processor. The storage includes instructions executable by the at least one processor to receive input from the camera and, based on the input from the camera, identify information related to a first real-world object. The instructions are also executable to, based on the identification of the information related to the first real-world object, present at least one augmented reality (AR) image on the display at a location of the display that does not obstruct a user's view of the first real-world object while the device is worn. <CIT> discloses using previous object characteristics to determine new object characteristics at a new AR placetime.

As will be appreciated by one skilled in the art, aspects of the embodiments may be embodied as a system, method or program product.

Many of the functional units described in this specification have been labeled as modules, in order to more particularly emphasize their implementation independence. For example, a module may be implemented as a hardware circuit comprising custom VLSI circuits or gate arrays, off-the-shelf semiconductors such as logic chips, transistors, or other discrete components.

An identified module of code may, for instance, comprise one or more physical or logical blocks of executable code which may, for instance, be organized as an object, procedure, or function. Nevertheless, the executables of an identified module need not be physically located together, but may comprise disparate instructions stored in different locations which, when joined logically together, comprise the module and achieve the stated purpose for the module.

More specific examples (a non-exhaustive list) of the storage device would include the following: an electrical connection having one or more wires, a portable computer diskette, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or Flash memory), or any suitable combination of the foregoing.

Code for carrying out operations for embodiments may be written in any combination of one or more programming languages including an object oriented programming language such as Python, Ruby, R, Java, Java Script, Smalltalk, C++, C sharp, Lisp, Clojure, PHP, or the like, and conventional procedural programming languages, such as the "C" programming language, or the like, and/or machine languages such as assembly languages. The code may execute entirely on the user's computer, partly on the user's computer, as a stand-alone software package, entirely or partly on a head mounted display such as an augmented reality device, partly on the user's computer and partly on a remote computer or entirely on the remote computer or server.

The term "and/or" indicates embodiments of one or more of the listed elements, with "A and/or B" indicating embodiments of element A alone, element B alone, or elements A and B taken together.

This code may be provided to a processor of a general purpose computer, special purpose computer, or other programmable data processing apparatus to produce a machine, such that the instructions, which execute via the processor of the computer or other programmable data processing apparatus, create means for implementing the functions/acts specified in the schematic flowchart diagrams and/or schematic block diagrams block or blocks.

In this regard, each block in the schematic flowchart diagrams and/or schematic block diagrams may represent a module, segment, or portion of code, which comprises one or more executable instructions of the code for implementing the specified logical function(s).

<FIG> is a perspective drawing illustrating one embodiment of an augmented reality (AR) device <NUM>. In the depicted embodiment, the AR device <NUM> includes a camera <NUM>. The AR device <NUM> may be worn by a user who views a display within the AR device <NUM>. The AR view of the AR device <NUM> may include both the environment and images that are presented on the display by the AR device <NUM>.

<FIG> is a perspective drawing illustrating one embodiment of an AR view <NUM>. The AR view <NUM> is presented to the user by a display of the AR device <NUM>. The AR view <NUM> includes the environment 103a. The AR view <NUM> further includes a window <NUM> that comprises video, graphics, text, and the like that is superimposed over the environment 103a by the AR device <NUM>. Although the window <NUM> is depicted as a flat screen, the window <NUM> may be presented with any shape and in either two or three dimensions. In one embodiment, the window <NUM> is presented as though disposed on a surface of the environment 103a. For example, a television window <NUM> may be presented as though disposed on a wall of a room.

<FIG> is a perspective drawing illustrating one alternate embodiment of an AR view <NUM>. In the depicted embodiment, a new environment 103b is captured by the camera <NUM> at a different AR position and/or AR time from the given AR position and time of the environment 103a of <FIG>. The new environment 103b includes objects <NUM> that were not in the previous environment 103a. As a result, displaying the window <NUM> at the same location in the new environment 103b as was used in the original environment 103a may not be desirable.

Unfortunately, the user may wish to view the window <NUM>. Because presenting the window <NUM> at the same location of the previous environment 103a would degrade the user's viewing experience, the embodiments determine a new location and other window characteristics for the window <NUM> in the new environment 103b. The embodiments detect a new AR placetime for the AR device <NUM> and calculate new window characteristics for the window <NUM> at the new AR placetime based on previous window characteristics at a previous environment 103a. In addition, the embodiments display the window <NUM> with the new characteristics. As a result, the window <NUM> is automatically repositioned to a new location within the new environment 103b, improving the efficiency and efficacy of the AR device <NUM> as will be described hereafter.

<FIG> is a schematic block diagram illustrating one embodiment of AR data <NUM>. The AR data <NUM> may be used to calculate the new window characteristics <NUM> for the window <NUM> and display the window <NUM> with the new window characteristics <NUM>. The AR data <NUM> may be organized as a data structure in a memory. In the depicted embodiment, the AR data <NUM> includes window preferences <NUM>, one or more window characteristics <NUM>, thresholds <NUM>, and current activity <NUM>.

The window preferences <NUM> may record user preferences for the placement of windows <NUM> within an AR view <NUM>. The window preferences <NUM> may include a preferred window angular size, a docking preference, a preferred window angular orientation, and/or a preferred window density. The preferred window angular size may specify a maximum window angular size, a minimum angular window size, and/or an average window angular size. The window angular size is an apparent maximum window dimension measured in degrees of the window <NUM> as seen from the user's eye. Alternatively, the window angular size may be measured in pixels.

The docking preference may specify whether a window <NUM> is presented as docked, wherein the window <NUM> appears disposed on a surface such as a wall, or presented undocked as though floating in the air. The preferred window angular orientation may specify a maximum angular difference between a vector from the user's eye to the center of the window <NUM>, and an apparent normal vector radiating from the center of the window <NUM>. The preferred window density may specify a percentage of the AR view <NUM> over which windows <NUM> are overlaid. In one embodiment, the preferred window density is for a specified view angle such as a <NUM> degree angle as measured from the user's eye. Alternatively, the preferred window density may be for the total pixels presented by the AR device <NUM>.

The window characteristics <NUM> describe presentation characteristics of a window <NUM>. The window characteristics <NUM> are described in more detail in <FIG>. The thresholds <NUM> include one or more thresholds used to make automatic decisions. The current activity <NUM> records the current activity of the user with the AR device <NUM> in the environment <NUM>.

<FIG> is a schematic block diagram illustrating one embodiment of the window characteristics <NUM>. The window characteristics <NUM> maybe organized as a data structure in a memory. In the depicted embodiment, the window characteristics <NUM> include a window size <NUM>, a window angle <NUM>, a window distance <NUM>, a window orientation <NUM>, the AR position <NUM>, the AR time <NUM>, AR people <NUM>, an AR context <NUM>, a window timestamp <NUM>, a surface relationship <NUM>, surface characteristics <NUM>, a reposition status <NUM>, a usage history <NUM>, and user activity <NUM>. An AR placetime <NUM> may comprise the AR position <NUM>, AR time <NUM>, AR people <NUM>, and/or AR context <NUM>.

The window size <NUM> may record the size of the window <NUM> as an angle is viewed from the user's eye and/or in pixels. For example, a window <NUM> may have a window size <NUM> of <NUM> degrees and/or <NUM> pixels. The window angle <NUM> may specify an angle of an apparent normal vector from a center of the window to a vector from the user's eye to the center of the window <NUM>. The window distance <NUM> may be an apparent distance in the environment <NUM> from the user's eye to the window <NUM>. The window orientation <NUM> may record whether the window <NUM> is presented in landscape or portrait.

The AR position <NUM> specifies the position of the window <NUM> within the AR view <NUM>. The AR time <NUM> may record a time interval when the window <NUM> was displayed at the AR position <NUM>. The AR people <NUM> may record other people in the AR view <NUM> when the window <NUM> was displayed. The AR context <NUM> records the context of usage of the AR device <NUM> and/or activity displayed in the AR view <NUM>.

The window timestamp <NUM> records when the window <NUM> was last displayed using the window characteristics <NUM>. The surface relationship <NUM> records whether the window <NUM> was docked to a surface such as a wall. The surface characteristics <NUM> record characteristics of any surface that the window <NUM> was docked to. The surface characteristics <NUM> may include a surface color, a surface uniformity, and object density, and the like. The surface color may record the average color of the surface. The surface uniformity may record an average deviation from the surface color across the surface. The object density may record a percentage of the surface area of the surface that includes objects that form discontinuities in the surface.

The reposition status <NUM> records details of the last repositioning of the window <NUM> within the AR view <NUM>. The usage history <NUM> records uses of the window <NUM>. The user activity <NUM> records other interactions of the user with the AR view <NUM> while the window <NUM> was active.

<FIG> is a perspective drawing illustrating one embodiment of an AR view <NUM> with new window characteristics <NUM>. The AR view <NUM> of the new environment 103b of <FIG> is shown. The window <NUM> is displayed with new window characteristics <NUM> that conform to the new, current environment 103b. As a result, the user may view the window <NUM> in the new environment 103b, giving user access to customary information in the window <NUM> and improving the function and efficiency of the AR device <NUM>.

<FIG> is a schematic block diagram illustrating one embodiment of the AR device <NUM>. In the depicted embodiment, the AR device <NUM> includes a processor <NUM>, a memory <NUM>, communication hardware <NUM>, the camera <NUM>, and a display <NUM>. The memory <NUM> may comprise a semiconductor storage device. The memory <NUM> stores code. The processor <NUM> executes the code. The communication hardware <NUM> communicates the camera <NUM>, the display <NUM>, and/or with other devices such as a network, server, and/or computer. The display <NUM> may be disposed in the AR device <NUM> and may be an optical engine of the AR device <NUM>.

<FIG> is a schematic flow chart diagram illustrating one embodiment of a window display method <NUM>. The method <NUM> may display the window <NUM> with the new window characteristics <NUM>. The method <NUM> may be performed by the AR device <NUM> and/or the processor <NUM>.

The method <NUM> starts, and in one embodiment, the processor <NUM> records <NUM> the window characteristics <NUM> for the AR view <NUM> in the previous environment 103a. The processor <NUM> may record a plurality of previous window characteristics <NUM> for each environment <NUM> in which the AR device <NUM> is employed.

The processor <NUM> further detects <NUM> a new AR placetime <NUM> using the camera <NUM>. The new AR placetime <NUM> may be different from the most recent previous AR placetime <NUM>. In addition, the new AR placetime <NUM> may be different from all previous AR placetimes <NUM>. In one embodiment, the new AR placetime <NUM> is detected from a new AR position <NUM>. For example, if the AR device <NUM> is used at a new AR position <NUM>, the new AR placetime <NUM> may be detected.

In addition, the new AR placetime <NUM> may be detected <NUM> from a new AR time <NUM>. For example, if the AR device <NUM> is used at a new AR time <NUM>, the new AR placetime <NUM> may be detected <NUM>. The new AR placetime <NUM> may be detected <NUM> in response to elapsed time from the window timestamp <NUM> exceeding an elapsed time threshold <NUM>.

In one embodiment, the new AR placetime <NUM> is detected <NUM> in response to new AR people <NUM> in the environment <NUM>. For example, if a new colleague is visible in the AR view <NUM>, a new AR placetime <NUM> may be detected <NUM>.

The new AR placetime <NUM> may also be detected <NUM> from a change in AR context <NUM>. For example, if the AR device <NUM> was moving in the previous environment 103a and is stationary in the new, current environment 103b, the new AR placetime <NUM> may be detected <NUM>. In one embodiment, the new AR placetime <NUM> is detected <NUM> from a change in any of the window characteristics <NUM>. The new AR placetime <NUM> may be automatically detected <NUM>.

In response to detecting <NUM> the new AR placetime <NUM>, the processor <NUM> determines <NUM> whether to redeploy the window <NUM> within the AR view <NUM>. The window <NUM> may be redeployed with new window characteristics <NUM>. In one embodiment, the processor <NUM> queries to reposition the window <NUM>. The processor <NUM> may query the user as to whether to reposition the window <NUM>. For example, the processor <NUM> may present a "Reposition Window" option to the user. The processor <NUM> may determine <NUM> to redeploy the window <NUM> in response to an affirmative reply from the user. Thus, the window <NUM> may be displayed with the new window characteristics <NUM> in response to being directed to reposition the window <NUM>.

In one embodiment, the processor <NUM> determines <NUM> to redeploy the window <NUM> within the AR view <NUM> if the usage history <NUM> and/or user activity <NUM> of a previous window characteristic <NUM> is similar to the current activity <NUM>. For example, if the user was previously performing an inspection user activity <NUM> recorded in a previous window characteristic <NUM> and the user is now performing an inspection current activity <NUM>, the processor <NUM> may determine <NUM> to redeploy the window <NUM>.

In one embodiment, the processor <NUM> determines <NUM> to redeploy the window <NUM> within the AR view <NUM> if the current time is within a daily time interval of the AR time <NUM> of a previous window characteristic <NUM>. As used herein, a daily time interval is a range of times on any time and/or a specific day. For example, <NUM>:<NUM> to <NUM>:<NUM> on Monday may be a daily time interval.

In one embodiment, the processor <NUM> automatically determines <NUM> to redeploy each window <NUM> from a most recent previous environment <NUM> within the AR view <NUM>. For example, if a first window <NUM> was displayed the last time the AR device <NUM> was used, the first window <NUM> is determined to be redeployed.

In one embodiment, the processor <NUM> determines <NUM> not to redeploy the window <NUM> if the elapsed time from the window timestamp <NUM> exceeds a significant time threshold <NUM>. If the processor <NUM> determines <NUM> not to redeploy the window <NUM>, the processor <NUM> queries <NUM> the user for new window characteristics <NUM>. For example, the processor <NUM> may present a user interface that queries <NUM> for the new window characteristics <NUM>. The processor <NUM> may further receive the new window characteristics <NUM> from the user.

If the processor <NUM> determines <NUM> to redeploy the window <NUM>, the processor <NUM> determines <NUM> the window preferences <NUM> for the window <NUM> displayed at one or more previous AR placetimes <NUM>. The window preferences <NUM> are calculated to conform to one or more elements of one or more previous window characteristics <NUM>. In addition, the window preferences <NUM> may be received from the user.

The processor <NUM> may calculate <NUM> the new window characteristics <NUM> for the window <NUM>. The new window characteristics <NUM> are calculated <NUM> for the new AR placetime <NUM>.

In one embodiment, the new window characteristics <NUM> are based on one instance of previous window characteristics <NUM>. The one instance of previous window characteristics <NUM> may be selected based on the current activity <NUM>. For example, the one instance of previous window characteristics <NUM> may record user activity <NUM> that is equivalent to the current activity <NUM> of the user.

In a certain embodiment, the new window characteristics <NUM> are based on an instance of the previous window characteristics <NUM> with the usage history <NUM> that conforms to the current activity <NUM>. In addition, the new window characteristics <NUM> may be based on an instance of the previous window characteristics <NUM> with a reposition status <NUM> that is most similar to the current activity <NUM>. In one embodiment, the new window characteristics <NUM> are based on an instance of the previous window characteristics <NUM> with surface characteristics <NUM> that are most similar to the surface characteristics <NUM> of the current, new environment 103b.

In one embodiment, the new window characteristics <NUM> are calculated <NUM> based on an average, median, and/or mean of the plurality of previous window characteristics <NUM>. In addition, the new window characteristics <NUM> may be calculated <NUM> based on an average, median, and/or mean of one or more elements of the plurality of previous window characteristics <NUM>.

The processor <NUM> calculates of <NUM> the new window characteristics <NUM> for the window <NUM> at a new AR placetime <NUM> based on previous window characteristics <NUM>. For example, one or more of the window size <NUM>, the window angle <NUM>, the window distance <NUM>, and/or window orientation <NUM> may be within a similarity threshold <NUM> of the window size <NUM>, the window angle <NUM>, the window distance <NUM>, and/or window orientation <NUM> of the previous window characteristics <NUM>. The similarity threshold may be plus or minus <NUM> to <NUM> percent. In addition, the new window characteristics <NUM> may conform to the window preferences <NUM>.

The processor <NUM> displays <NUM> the window <NUM> with the new window characteristics <NUM> and the method <NUM> ends. The window <NUM> may be displayed <NUM> in the AR view <NUM> for the new environment 103b.

The embodiments detect the new AR placetime <NUM> and calculate new window characteristics for the window <NUM> at the new AR placetime <NUM> based on previous window characteristics <NUM>. The window <NUM> is displayed with the new window characteristics <NUM>. As a result, the window <NUM> and information displayed thereon is automatically available to the user through the AR device <NUM> when the user moves between environments <NUM>, improving the efficiency and effectiveness of the AR device <NUM> in presenting information to the user.

Claim 1:
An apparatus (<NUM>) comprising:
a processor (<NUM>);
a memory (<NUM>) that stores code executable by the processor to:
detect (<NUM>) a new augmented reality, AR, placetime comprising a new AR position and/or a new AR time of an AR device;
in response to detecting the new AR placetime, determine (<NUM>) whether to redeploy a window;
in response to determining to redeploy the window:
calculate (<NUM>) new window characteristics for the window at the new AR placetime based on previous window characteristics (<NUM>) stored in memory; and
display (<NUM>) the window with the calculated new window characteristics;
in response to determining to not redeploy the window:
query (<NUM>) a user for new window characteristics; receive new window characteristics from the user; and
display (<NUM>) the window with the new window characteristics received from the user.