System and method for placement of augmented reality information for users based on their activity

Systems and methods described herein are provided for receiving location information for a real-world object in a user's augmented reality (AR) view, determining a direction of travel of a user, determining an activity zone shape associated with the direction of travel, responsive to a determination that the real-world object is within the activity zone, rendering on an AR display AR information associated with the real-world object at a location outside an area of the AR display used to display the activity zone. Some embodiments continue to display AR information in the same location if the user changes his or her gaze to read the AR information. Some embodiments render on an AR display subtle highlighting if an object is inside an activity zone and prominent highlighting if an object is outside an activity zone.

BACKGROUND

When a user is walking around with augmented reality (AR) glasses, applications may display annotations and/or animations overlaid on top of a real-world view and block or obscure a user's ability to see both the real world and the augmentations.

Different techniques have been considered for determining which AR annotations should be displayed to a user. One such technique is to display AR information about real-world (or real-life) objects that are in the users field of view, including objects that may be too far away to be discerned by the user. Another such technique is to display AR information about real-world objects on which the user's gaze rests. The first technique may overwhelm the user with AR information, especially in environments where there are many real-world objects in the user's field of view for which information is available to be displayed to the user. The second technique may display AR information about fewer real-world objects than the first system, though the second technique displays AR information for any real-world object upon which the user may happen to gaze. The AR information for these objects may interfere with the very scene that the user is viewing.

SUMMARY

Systems and methods described herein relate to user interfaces that display relevant augmented reality (AR) information in a manner that is unobtrusive to the user and does not interfere with the user's main activity. For example, AR information may be restaurant or coffee icons or directions to such locations. For some embodiments, such AR information may be displayed on an optical see-through device as overlay graphics that highlight a real-world object, such as a sign or a building, associated with a restaurant or store. For some embodiments, AR information (such as locations and distances to local coffee shops) may be retrieved from an external server and displayed to a user on an AR display, such as an AR optical see-through device.

A user's main activity may correspond to part of the users visual field (e.g., the center region of where the user may be looking and moving). Systems and methods described herein display augmented information outside an activity zone so the information does not interfere with the user's activity. The user may turn his or her gaze without causing changes in the placement of the augmented information. Thus, the user may view and interact with augmented information without the information intruding into the users main activity. A device or system application may display AR information and objects in a manner that does not obscure an active activity zone.

Systems and methods herein provide dynamic determinations of how and when AR information about real-world objects may be displayed to a user. Some embodiments determine how such information may be displayed to a user whose direction of gaze and direction of motion may be approximately, but imperfectly, aligned without interfering with the user's interaction with reality.

The entities, connections, arrangements, and the like that are depicted in—and described in connection with—the various figures are presented by way of example and not by way of limitation. As such, any and all statements or other indications as to what a particular figure “depicts,” what a particular element or entity in a particular figure “is” or “has,” and any and all similar statements—that may in isolation and out of context be read as absolute and therefore limiting—may only properly be read as being constructively preceded by a clause such as “In at least one embodiment . . . .” For brevity and clarity of presentation, this implied leading clause is not repeated ad nauseum in the detailed description of the drawings.

DETAILED DESCRIPTION

In exemplary embodiments described herein, augmented reality annotations are displayed at a position as not to obstruct a user's view of a determined activity zone (or region). For example, when a user is moving in a particular direction, the augmented reality annotations may be positioned so as not to obstruct the user's view in that direction of motion. For some exemplary embodiments, the user guides the direction of motion, e.g., while walking, riding a bicycle, driving a car, or riding a Segway. For such embodiments, the users direction of gaze may continually approximate the user's direction of motion, even though the two directions may not be identical. If a user shifts his or her gaze to another direction, e.g., to read the AR information or to look at a highlighted object near the periphery, exemplary embodiments of methods or systems described herein will not cause AR information or associated graphics to move out of the way of the users gaze. Such embodiments may have greater stability to the user by displaying information out of the way. When a user is a passenger in a train or car, though, a user's direction of gaze may be decoupled from the user's direction of motion.

One embodiment of a method disclosed herein tracks a direction of travel of a user of an augmented reality (AR) display, identifies an object having associated AR information, determines whether the direction of travel is substantially towards the object, selects a location for display of the AR information, wherein selecting the location comprises: selecting a first location at an offset from the object while the direction of travel is substantially towards the object, and selecting a second location that at least partially obscures the object while the direction of travel is not substantially towards the object, and displays the AR information at the selected location using the AR display.

For one or more embodiments, a method for displaying (or rendering) AR information about a real-world object in a users view (or AR display) may comprise multiple steps: receiving real-world object data, determining a users activity zone, highlighting a real-world object based on where the real-world object lies with respect to the activity zone, displaying a real-world object's AR information outside an activity zone, and displaying an association of the AR information with a real-world object. An AR system may receive data regarding a real-world object, along with the object's coordinates in a system's visual field and associated augmented information. For one embodiment, determining a users activity zone may comprise determining the user's direction of motion as a straight line and determining a zone as a preset view angle centered on the users direction of motion. For another embodiment, determining a users activity zone may be based on a nonlinear direction of motion, such as a user walking on a curved path. Determining the user's direction of motion as a straight line may comprise predicting motion based on (as appropriate): the major dimension of movement of the user, the orientation of the user's body (including chest and feet), and the direction in which the user leans. For one embodiment, highlighting a real-world object based on where a real-world object lies with respect to an activity zone may comprise displaying subtle highlighting (such as a silhouette or a soft glow overlay) for the object if the object is displayed within the activity zone and displaying prominent (or strong) highlighting (such as hatching) for the object if the object is displayed outside the activity zone. For one embodiment, a real-world object's AR information may be displayed at a point along the outside periphery of the activity zone closest to the object. For one embodiment, displaying an association of AR information with a real-world object may comprise highlighting the AR information in a way that matches the highlighting of the real-world object and displaying a line linking the AR information with the real-world object, which may comprise subtle lines for within-zone objects and prominent lines for outside-zone objects.

Prominent highlighting may differ from subtle highlighting in one or more of various different properties. For example, as compared to subtle highlighting, prominent highlighting may have one or more of a greater opacity, greater line thickness, greater brightness, greater color saturation, or other properties indicative of prominence.

FIG. 1shows a process interface diagram of an AR system100. A scene sensor150, such as a camera or LIDAR system, captures image data. For one embodiment, scene sensors150measure information about a scene162, e.g., as a point cloud, and communicate such scene information to an Activity Zone Determiner154and to augmented reality applications. User-specific sensors152may provide information about a user, such as orientation of a users chest or the angle of a users shoes. Augmented Reality applications160(or apps) or services external to systems and methods described herein may receive data related to identified real-world objects along with information to display in an augmented manner. An AR application160may communicate identified objects and augmented information172to an augmented reality and object locator and information placer158. An Activity Zone Determiner154may determine an activity zone168for a scene and communicate information used to describe the activity zone to an augmented reality object locator and information placer158. A user-specific sensor152may measure user-specific data and communicate user activity information166to an Activity Zone Determiner154. An Activity Zone Determiner154may use data from user-specific sensors152to determine a user's activity zone168comprising of the region of the visual field that may be relevant or most relevant to a user's activity. For one embodiment, the activity zone may be determined from the user's direction of motion by estimating where the user may arrive within a small duration (such as five seconds) with a diameter corresponding to an inner part of the visual field. For one embodiment, such an inner part of the visual field may be a “near peripheral” area that is a wedge-shaped area centered on an axis of motion with angles of 30° clockwise and counterclockwise from a center direction directly in front on a user's eye. For example, if a user is walking, an Activity Zone Determiner154may determine a zone as the part of the visual field that includes where the user may be within ten or fewer steps. An Augmented Reality Object Locator and Information Placer158may locate identified objects with respect to an activity zone168and determine subtle or prominent highlights170, as appropriate. An Augmented Reality Object Locator and Information Placer158may determine locations outside an activity zone168for augmented information170associated with these objects that minimizes the distance between the augmented information170and the associated objects and may determine a display style for a connection170between an identified object and an identified object's associated augmented information170. For one embodiment, object highlight data, augmented information, and connection line display data170is communicated to an AR display156(or display). An AR display156may display augmented information to a user. For one embodiment, augmented information, connection line display data, and object highlight data may be used to display data to a user.

For some embodiments, an AR device comprises the components illustrated inFIG. 1: scene sensor, Activity Zone Determiner154, user-specific sensor(s), augmented reality application(s), Augmented Reality Object Locator and Information Placer158, and an AR display156. For some embodiments, for example, scene sensors150may use a camera integrated into a smart phone. Other scene and user-specific sensors may be integrated into a headset or AR glasses. For some embodiments, one or more scene sensors may be mounted to the top of a headset, along with user-specific sensors. For some embodiments, mounted to or otherwise coupled with a headset may be a processor for running software that implements an Activity Zone Determiner154, an Augmented Reality Object Locator and Information Placer158, and one or more AR applications160. User-specific sensors may measure heartrate, walking or running speed, temperature, and these sensors may be integrated into a smart phone or other AR device. For some embodiments, an Activity Zone Determiner154and an Augmented Reality Object Locator and Information Placer158may be processes running on a processor within an AR device. For some embodiments, an AR display156may be a screen on a smart phone. For some embodiments, an AR display156may be a see-through display device, such as an AR-enabled vehicle windshield or AR optical see-through glasses or headset. For some embodiments, scene sensor data162,164is measured by scene sensor150external to an AR device and communicated to an AR device. For some embodiments, user-specific sensor data is measured by devices external to an AR device and communicated to an AR device. For some embodiments, augmented reality applications160running on a device external to an AR device may communicate identified objects and augmented information to an AR device. For some embodiments, an AR device comprises an Activity Zone Determiner154and an Augmented Reality Object Locator and Information Placer158, while AR applications160, scene sensor(s)150, user-specific sensor(s)152, and an AR display156are external.

FIG. 2shows an example display200within an AR device not using systems and methods described herein. AR information202,204,206,208,210,212is scattered across the display area and obstructs the center of the user's vision.

FIG. 3shows an example display300within an AR device using systems and methods described herein. For one embodiment, real-world objects316,318,320are highlighted either subtly or prominently based on whether an object is inside or outside an activity zone. For one embodiment shown inFIG. 3, the big circle is the periphery of an activity zone314. AR information302,304,306,308,310,312associated with real-world objects316,318,320are displayed on the periphery of the activity zone314.

FIG. 4shows an example AR display400with an activity zone surrounding a person402in a users field of view. AR information404regarding the person's gender, height, and weight are displayed to the left of the person. AR information406,408related to other objects is displayed to the right of the person.

FIG. 5shows another example AR display. For this example, a user is walking around a city center. AR information is displayed along the top and right sides of the AR display. For this example, icons are displayed showing the locations of the nearest Ladurée bakery, McDonald's fast food restaurants, Starbuck's coffee, and taxi stand, along with distances to each of these locations underneath each icon.

For some embodiments, an AR display is an optical see-through device. For some embodiments, an AR display is provided on a windshield of a vehicle. For some embodiments, an AR display is embedded in glasses or goggles. For some embodiments, an AR display renders all objects and graphics seen on a screen. For some embodiments, an AR display renders graphical overlays on top of optically-seen real-world objects.

ForFIGS. 2 and 3, some embodiments may display the example white boxes, circles, and lines as graphical overlays on top of the optically-seen real-world objects represented by the black and white photograph. Similarly, forFIG. 4, some embodiments may display the text and white outline of the activity zone displayed around the person in the foreground as graphical overlays on top of the optically-seen real-world objects represented by the black and white photograph. ForFIG. 5, one embodiment500may display a bakery icon502, two fast food restaurant icons504,506, a coffee shop icon508, and a taxi stand icon510, along with distances524,526,528,530to each of these locations as graphical overlays on top of optically-seen real-world objects512,514,516,518,520,522represented by black and white drawings of real-world objects. For some embodiments, everything seen inFIGS. 2 to 5may be rendered by an AR display.

FIGS. 6A and 6Bare perspective schematic diagrams600,650for the placement of AR information604,654at two moments in time. At time t1, a user has been continually walking towards a real-world object602, shown as a cube inFIG. 6A. AR information604related to the object is displayed to the left of the object602and the users direction of motion606. At time t2, the user has substantially stopped moving656towards the object652, as shown inFIG. 6B. AR information654related to the object652is displayed towards the top of the object652. For some embodiments, responsive to a determination that the user has substantially stopped moving in the direction of travel, AR information associated with a real-world object may be rendered on an AR display at a location on the AR display that at least partially obscures the real-world object.

FIGS. 7A and 7Bare perspective schematic diagrams700,750for the placement of AR information704,754at two moments in time when a user changes direction of motion. At time t1, a user is walking towards a real-world object702, shown as a cube inFIG. 7A. AR information704related to the object702is displayed to the left of the object and the user's direction of motion706. At time t2, the user turns to the left and starts walking to the left of the object752, as shown inFIG. 7B. AR information754related to the object752is displayed towards the top of the object752and to the right of the users direction of motion756.

FIG. 8is a flowchart800for an exemplary embodiment for determining a screen location for displaying AR information related to a real-world object. For one embodiment, the direction of motion of the user is determined802. A user frame of reference to location <0, 0, 0> within a coordinate system for the users motion may be set804, where for one embodiment, positive x-axis values are to the right, positive y-axis values are up, and negative z-axis values are in the direction of the user's motion. For one embodiment, an activity zone is determined806as a cone (or conical region) centered on the z-axis with an inner angle of 60 degrees centered on the z-axis and the point of the cone centered on the user. For one embodiment, the coordinates of a real-world object are transformed808to the coordinate system for the frame of reference of the activity zone. For one embodiment, the real-world object's transformed coordinates are set808to <x0, y0, z0>. For one embodiment, the activity zone is further determined810to be a cone of depth z0with a radius rcsuch that the cone's inner angle is 60 degrees. For one embodiment, a process determines812if x02+y02≤rc2. If x02+y02≤rc2, the real-world object is inside the activity zone's cone and subtle highlighting is added814to the real-world object on the AR display. Otherwise (for the same embodiment), the real-world object is outside the activity zone's cone and prominent highlighting is added816to the real-world object on the AR display.

For some embodiments, subtle highlighting comprises displaying a non-clear transparency on top of the real-world object. For some embodiments, subtle highlighting comprises displaying transparent colors with an intensity level less than a predetermined threshold. For some embodiments, subtle highlighting comprises displaying a transparent color associated with a highlighter, such as light pink, light orange, light yellow, light green, light blue, or light purple. For some embodiments, subtle highlighting comprises displaying a thin-line graphic around the real-world object or displaying a thin line connecting the real-world object to augmented information.

For some embodiments, prominent highlighting comprises displaying an opaque color on top of the real-world object. For some embodiments, prominent highlighting comprises displaying opaque colors with an intensity level greater than a predetermined threshold. For some embodiments, prominent highlighting comprises displaying an opaque color, such as solid red, solid orange, solid yellow, solid green, solid blue, or solid purple. For some embodiments, prominent highlighting comprises displaying a thick-line graphic around the real-world object or displaying a thick line connecting the real-world object to augmented information.

For one embodiment, xposand xnegare calculated818as the two solutions to Eqn. 1.

x=±rc21+y02x02Eqn.⁢1
For one embodiment, a process compares820the distance from <x0, y0, z0> to

〈xpos,(y0x0)⁢(xp⁢o⁢s),z0〉
with the distance from

〈x0,y0,z0〉⁢⁢to⁢⁢〈xneg,(y0x0)⁢(xn⁢e⁢g),z0〉.
If the distance from

〈x0,y0,z0〉⁢⁢to⁢⁢〈xpos,(y0x0)⁢(xpos),z0〉
is smaller than the distance from

〈x0,y0,z0〉⁢⁢to⁢⁢〈xneg,(y0x0)⁢(xn⁢e⁢g),z0〉,
the augmented reality information for the real-world object is displayed822at

〈xpos,(y0x0)⁢(xp⁢o⁢s),z0〉.
Otherwise (for the same embodiment), the augmented reality information for the real-world object is displayed824at

〈xneg,(y0x0)⁢(xn⁢e⁢g),z0〉.
For one embodiment, a connection graphic is displayed826to connect the AR information to the real-world object.

FIG. 9shows a message sequence diagram900for messages communicated between processes and devices for displaying AR information for an exemplary embodiment. A location is listed above each device or process for an exemplary embodiment. A Scene Sensor904transmits914,916scene data (such as point cloud) to an Activity Zone Determiner906and to an AR Application (which may be an information service). A User Sensor910may also transmit918user information to an Activity Zone Determiner906. For one embodiment, an Activity Zone Determiner906determines920an activity zone and transmits922a message describing the activity zone to an AR Object Locator and Information Placer908. An AR Object Locator and Information Placer908also may receive924a message from an AR Application912with location data for real-world objects in a scene and augmented information about those real-world objects. An AR Object Locator and Information Placer908determines926a location of the real-world object with respect to the activity zone and for one embodiment, generates928scene data that may be used to display augmented information on the periphery of an activity zone. The AR Object Locator and Information Placer908may send930a message to an AR Display902with scene data augmented with AR information.

FIG. 10shows a plan view schematic1000for a user1008and an activity zone1004. For this example, real-world objects1010,1012are shown as dark, bold circles, while the direction of travel1002is shown with an arrow pointing directly up towards a normal width circle. For this example, a sample activity zone1004is shown as three-dimensional oval centered on the direction of activity (or travel). For this embodiment, an inner angle1006of 60 degrees is shown for a cone associated with the activity zone1004. For this embodiment, the cone's apex is centered upon the user1008.

For an exemplary embodiment, an activity zone1004is determined using a users chest and foot orientation based on sensor readings located on a users body. Another embodiment uses camera image data in the infrastructure surrounding the user. One embodiment uses one or more motion sensors, such as a GPS, a compass, an accelerometer, or a gyroscope, on a user's device to calculate a main direction of travel. For one embodiment, if an accelerometer or gyroscope is used, a GPS or other location device may be used to calculate a main direction of travel. For one embodiment, a compass may be used to determine a main direction of travel. For one embodiment, two or more GPS readings may be used to determine a main direction of travel. An example of techniques that may be used to determine a user's direction of travel include those described in Gabor Paller,Motion Recognition with Android Devices, SFONGELTD. (Oct. 7, 2011), http://www.slideshare.net/paller/motion-recognition-with-android-devices (see slide33).

For some embodiments, an activity zone1004is calculated based on a direction of travel. For some embodiments, an activity zone's cone1014has an opening angle is 60 degrees and whose apex is at the center of a user's device (alternatively, the center may be at the users eyes or the device's camera) and which is centered about the ray corresponding to the users direction of motion. The height (in math terminology), which is the depth in the present setting, of the cone1014corresponds to how far the user may see. For some embodiments, there is a cone1014for each height (e.g., one cone for 3 feet and another cone for 3.1 feet). Some real-world objects project inside and some do not.

FIG. 11shows a three-dimensional coordinate system1100with a frame of reference for a user1102with an origin aligned with the user1102and the z-axis along the user's direction of motion. For some embodiments, coordinates for an activity zone1108and coordinates received for a real-world object1106may be used with geometric equations to determine whether a real-world object1106is inside or outside an activity zone1108. For one embodiment, the labeling of x, y, and z coordinates follow the Android convention, which are described in Freescale Semiconductor's Application Note 4317 found at http://cache.freescale.com/files/sensors/doc/app_note/AN4317.pdf (see page 2, FIG. 1). For the example Android coordinate convention, the screen of the device is perpendicular to ground and facing the user. The x-axis is to the right; the y-axis is toward the sky; and the z-axis is parallel to the ground and toward the user. Points behind the screen have negative z-axis values. Some embodiments have the origin at the bottom left of the screen. For an exemplary embodiment used herein, the origin is in the middle of the screen. The device's screen may be small relative to the real world in target applications, so the error between these two embodiments may not be significant.

For some embodiments, scene data is received, which may be a point cloud. The scene data may correspond to what a user sees naturally in the real-world environment. For an object in a scene, a ray may exist that goes from the coordinate system's origin to the object. For objects with a nonzero extent, multiple rays may exist that go from the origin to the different sides of the object. For some embodiments, the coordinate system may be transformed from the received scene data (which may be a point cloud) to a coordinate system aligned with the user direction of motion, which may have negative z-axis values in the direction of motion. The angle between an object's ray and a z-axis aligned with the user's direction of motion may be calculated. If this calculated angle is less than half the opening angle of the cone, the real-world object may be determined to lie inside the activity zone. Otherwise, for this embodiment, the real-world object may be determined to lie outside the activity zone. Some embodiments may make these determinations based on calculations using a real-world object's coordinates without calculating the angle between the two rays described above.

For one embodiment shown inFIG. 11, an activity zone1108has at least one cone with an apex at the origin and a base centered on the Z axis. For one embodiment, the radius/height1104equals the tangent of 30 degrees or 0.577. That is,

One embodiment has a real-world object at coordinates <x0, y0, z0>. This embodiment may have a cone that has the same depth as the object (zc=z0). For this example, the object is within the user's activity zone if x02+y02≤rc2. If the object is within the user's activity zone, the AR information corresponding to the object may be displayed as a point along the outside periphery of the activity zone closest to the object. For one embodiment, augmented information is displayed at a point on the periphery of the activity zone closest to the real-world object. For one embodiment, this point is determined through use of the Pythagorean Theorem and geometric calculation. A line linking the real-world object to the associated augmented information may be displayed between the two items.

FIG. 12is a three-dimensional coordinate system1200centered on a user1202showing a real-world object1206and placement of augmented information. For one embodiment, a real-world object1206is at <x0, y0, z0> and the center of a cone's base is at <0, 0, z0>.FIG. 12shows one object1214located inside and one object1206located outside an example activity zone1208. For this example, both real-world objects and the cone's base are in the same plane, located at z=z0, and the equations that follow are for the x-axis and y-axis values. The radius1204of the cone's base is rc. For this example, the slope s of the line from the object to the center of the base of the cone is

s=y0x0.
This line intersects the circle at the base of the cone at two points1210,1212that satisfy the following Pythagorean Theorem and straight-line equations 2 to 6:

For this example, the equations may be used to calculate two solutions, which are the positive (xpos) and negative (xneg) values of x that lie on the circle and the corresponding values of y, which may be positive or negative depending upon the slope. For this example, the equation solutions are <xpos, (s)(xpos)> and <xneg, (s)(xneg)>. Substituting

y0x0
for the slope s gives

〈xp⁢o⁢s,(y0x0)⁢(xp⁢o⁢s)〉⁢⁢and⁢⁢〈xn⁢e⁢g,(y0x0)⁢(xn⁢e⁢g)〉.
The augmented information may be displayed at the <x, y> location that has a shorter distance to the real-world object. For this example, the calculations are the same if the object is inside or outside the cone.

FIG. 13is a plan view schematic1300(or bird's eye view) of a user1306and two real-world objects1308,1312. For one embodiment, a graphic is displayed in front of a user1308to indicate the periphery of the user's activity zone1304. For another embodiment, a graphic is not displayed in front of a user1306to indicate the periphery of the user's activity zone1304. For this example, the users gaze1302is aligned with the user's activity zone1304. For exemplary embodiments, augmented information, except for subtle highlights and linkages to AR information displayed along the periphery of the activity zone1304, is not displayed within the activity zone1304. AR information1310,1312for remote objects may be displayed outside but near the activity zone. As shown in the embodiment ofFIG. 13, AR information1310,1314is displayed away from a users gaze direction1302. One object1308located within an activity zone1304and within a user's gaze direction1302is highlighted subtly, while another object1312located outside an activity zone1304and outside a user's gaze direction1302is highlighted prominently.

FIG. 14is a schematic plan view1400(or bird's eye view) of a user1406and two real-world objects1408,1412. For this embodiment, a user's activity zone1404is displayed aligned with a direction of activity1402in front of the user1406, but the users gaze1416is off center and to the right. The user1406may be looking at a highlighted object1412(as shown inFIG. 14), AR information1414for an object1412, or an object without highlighting. Similar to the embodiment shown inFIG. 13, the activity zone1404remains clear of augmented information except for subtle highlights and linkages to AR information on the periphery of the activity zone. AR information1410,1414for the remote objects1408,1412may be displayed outside but near the activity zone1404. The observation zone1418is the part of the AR display where the user1406is looking that is outside of the activity zone1404. The observation zone1418may be cluttered with AR information while the activity zone1404remains free of clutter. As shown in the embodiment ofFIG. 14, the activity zone1404remains stable. For this embodiment, if a user's gaze1416shifts, AR information1414shown within the scene may not move (or “jump around”). For this embodiment, the activity zone1404may become peripheral to the user's vision but the activity zone1404continues to the clear. For one embodiment, if the user1406looks at AR information1414outside an activity zone1404, that AR information1414does not move (or “bounce away”), which would occur if AR information1414always was displayed outside the users direction of gaze1416. The embodiment shown inFIG. 14uses both an activity zone1404and an observation zone1418in determining where to display AR information for a seamless AR user experience.

For one embodiment, an activity zone is determined to be a cone-shaped area with an apex centered on a user's (or viewer's) head with an opening angle of 60 degrees and a z-axis plane aligned with a displayed object located at a maximum distance from the user in the direction of travel (or as far as the user is able to see in the direction of travel). For one embodiment, no visual demarcation is displayed for the activity zone, unlikeFIG. 3, which shows the activity zone as a thick circle. For some embodiments, a graphic may be rendered on an AR display that indicates a periphery of an activity zone. For one embodiment, an activity zone may be determined to be a distorted (or non-symmetrical) shape (instead of a symmetrical, conical, or circular shape) based on a user's activity. For example, an activity zone may be wider than the height if a users activity is focused on the lower part of the scene. For one embodiment, an activity zone is a rectangle. For one embodiment, an activity zone is a plurality of disconnected shapes. For one embodiment, an activity zone is a three-dimensional shape. For one embodiment, determination of a user's activity zone is based not on a two-dimensional model but a three-dimensional model incorporating depth, where the activity zone may be a conical shape or three-dimensional part of the scene. For one embodiment, determination of a user's activity zone may be based, not on the users direction of motion, but on the salience of a part of the scene, such as, what objects appear in what part of the scene. For one embodiment, a users activity zone may have one or more salient objects based on the users activity. For one embodiment, an activity zone is not a contiguous region (or is a plurality of disconnected shapes). For one embodiment with real-world objects outside of an activity zone, associated AR information is displayed (or placed) near the object along with a marker at the periphery of the activity zone that indicates some AR information is available if the user looks toward the real-world object. For some embodiments, additional AR information may be displayed if the user looks toward current AR information.

Consumer Use Cases

For an example consumer use case, John uses some AR applications as he walks through New York City. His activity is recognized as walking and an associated activity zone is determined to be the path in front of him up to about 3 feet ahead. John is able to view his walking path because AR information received from his AR applications is not displayed in his direction of travel. For example, if John sees interesting people, such as celebrities, some AR apps display information about those people. This information is displayed outside the activity zone.

For another consumer user case, John runs into Mary while walking through Times Square. They have an animated conversation. His activity is determined to be chatting with Mary, and the associated activity zone changes to Mary's body frame (which may be a silhouette slightly larger than Mary). As John stands and chats with Mary, he looks left and right from time to time. Different real-world objects may come within view either within his activity zone (e.g., next to Mary's ear) or outside his activity zone (e.g., to the right and down the street). The AR information for these objects is placed outside John's activity zone. If John turns to read this information or to examine a real-world object in some detail, the AR information may change but none of the AR information obstructs his view of Mary.

Network Architecture

A wireless transmit/receive unit (WTRU) may be used as a AR display device in embodiments described herein. The AR device shown inFIGS. 15A to 15Bmay be used to retrieve AR information.

FIG. 15Ais a system diagram of an example WTRU102. As shown inFIG. 15A, the WTRU102may include a processor118, a transceiver120, a transmit/receive element122, a speaker/microphone124, a keypad126, a display/touchpad128, a non-removable memory130, a removable memory132, a power source134, a global positioning system (GPS) chipset136, and other peripherals138. The transceiver120may be implemented as a component of decoder logic119. For example, the transceiver120and decoder logic119may be implemented on a single LTE or LTE-A chip. The decoder logic may include a processor operative to perform instructions stored in a non-transitory computer-readable medium. As an alternative, or in addition, the decoder logic may be implemented using custom and/or programmable digital logic circuitry.

It will be appreciated that the WTRU102may include any sub-combination of the foregoing elements while remaining consistent with an embodiment. Also, embodiments contemplate that base stations and/or the nodes that base stations may represent, such as but not limited to transceiver station (BTS), a Node-B, a site controller, an access point (AP), a home node-B, an evolved home node-B (eNodeB), a home evolved node-B (HeNB), a home evolved node-B gateway, and proxy nodes, among others, may include some or all of the elements depicted inFIG. 15Aand described herein.

The transmit/receive element122may be configured to transmit signals to, or receive signals from, a base station over the air interface116. For example, in one embodiment, the transmit/receive element122may be an antenna configured to transmit and/or receive RF signals. In another embodiment, the transmit/receive element122may be an emitter/detector configured to transmit and/or receive IR, UV, or visible light signals, as examples. In yet another embodiment, the transmit/receive element122may be configured to transmit and receive both RF and light signals. It will be appreciated that the transmit/receive element122may be configured to transmit and/or receive any combination of wireless signals.

FIG. 15Bdepicts an example network entity190that may be used within the communication system100ofFIG. 15A. As depicted inFIG. 15B, network entity190includes a communication interface192, a processor194, and non-transitory data storage196, all of which are communicatively linked by a bus, network, or other communication path198.

Communication interface192may include one or more wired communication interfaces and/or one or more wireless-communication interfaces. With respect to wired communication, communication interface192may include one or more interfaces such as Ethernet interfaces, as an example. With respect to wireless communication, communication interface192may include components such as one or more antennae, one or more transceivers/chipsets designed and configured for one or more types of wireless (e.g., LTE) communication, and/or any other components deemed suitable by those of skill in the relevant art. And further with respect to wireless communication, communication interface192may be equipped at a scale and with a configuration appropriate for acting on the network side—as opposed to the client side—of wireless communications (e.g., LTE communications, Wi-Fi communications, and the like). Thus, communication interface192may include the appropriate equipment and circuitry (which may include multiple transceivers) for serving multiple mobile stations, UEs, or other access terminals in a coverage area.

Processor194may include one or more processors of any type deemed suitable by those of skill in the relevant art, some examples including a general-purpose microprocessor and a dedicated DSP.

Data storage196may take the form of any non-transitory computer-readable medium or combination of such media, some examples including flash memory, read-only memory (ROM), and random-access memory (RAM) to name but a few, as any one or more types of non-transitory data storage deemed suitable by those of skill in the relevant art may be used. As depicted inFIG. 15B, data storage196contains program instructions197executable by processor194for carrying out various combinations of the various network-entity functions described herein.

In some embodiments, the network-entity functions described herein are carried out by a network entity having a structure similar to that of network entity190ofFIG. 15B. In some embodiments, one or more of such functions are carried out by a set of multiple network entities in combination, where each network entity has a structure similar to that of network entity190ofFIG. 15B.

Note that various hardware elements of one or more of the described embodiments are referred to as “modules” that carry out (i.e., perform, execute, and the like) various functions that are described herein in connection with the respective modules. As used herein, a module includes hardware (e.g., one or more processors, one or more microprocessors, one or more microcontrollers, one or more microchips, one or more application-specific integrated circuits (ASICs), one or more field programmable gate arrays (FPGAs), one or more memory devices) deemed suitable by those of skill in the relevant art for a given implementation. Each described module may also include instructions executable for carrying out the one or more functions described as being carried out by the respective module, and those instructions may take the form of or include hardware (or hardwired) instructions, firmware instructions, software instructions, and/or the like, and may be stored in any suitable non-transitory computer-readable medium or media, such as commonly referred to as RAM or ROM.