Patent Description:
This application also claims priority to <CIT>.

Placing an augmented reality (AR) object in the proper context within an image of a real-world scene viewed through a mobile device of a user can be complicated. Specifically, placing the AR object in the proper location and/or orientation within the display can be difficult to achieve. A global positioning system (GPS) of a mobile device of a user can be used to identify a location of the user and the location of the user can then be used to place AR objects associated with objects within the display of the user. <CIT> describes mixed-reality systems for using anchor graphs within a mixed-reality environment. These systems utilize anchor vertexes that comprise at least one first key frame, a first mixed-reality element, and at least one first transform connecting the at least one first key frame to the first mixed-reality element. Anchor edges comprising transformations connect the anchor vertexes.

The invention is set forth in the independent claims. Specific embodiments are presented in the dependent claims.

Placing an augmented reality (AR) object in the proper location and/or orientation within an image of a real-world scene viewed through a mobile device of a user can be difficult to achieve. A global positioning system (GPS) of a mobile device of a user can be used to identify a location of the user and the location of the user can then be used to place AR objects associated with objects within the display of the user. However, GPS may not be available and/or sufficiently accurate in some situations (e.g., in a building with multiple floors). For example, when a device is indoors, GPS generally may not be used to localize the device position accurately (e.g., accurately to a particular floor). Also, many venues are not open to the public and/or may not be well documented. Some information about a venue may not be reliable because a proprietor of the venue may not have the resources to maintain accurate information about the venue. Information about a venue may only be produced with expensive equipment and/or specialized technology. After such information is produced it may be relatively static and difficult to modify or update. Without accurate mapping, location, and/or orientation information associated with a venue, an application cannot properly place AR objects of the place and/or event within the display of the device.

The technical solutions described herein are related to processing of multiple perception signals to display augmented reality content (e.g., AR objects) for, for example, wayfinding and/or discovery at a venue (e.g., a location, physical space, region, area). Specifically, the accurate positioning and orientation of place and/or event information enables the use of augmented reality displays for use with, for example, wayfinding and/or information discovery. In some implementations, the contextual display in AR assists users in wayfinding at unfamiliar places and/or discovering events or places of interest when on location.

To achieve accurate placement of AR objects (also can be referred to as points of interest (POIs)), a scale-accurate digital 3D representation of the venue is generated and a location and/or orientation of a user can be localized to the scale-accurate digital 3D representation via AR anchors (an AR anchor has a fixed location with respect to an origin, wherein the origin is a predefined, fixed location in a real-world physical area). The 3D representation can then be transformed into the view space of a device of a user and the AR objects can be displayed in proper context within the real world using augmented reality. In some implementations, the AR object rendered appears anchored to the physical element the AR object is pointing to, labeling, and/or so forth.

In some implementations, physical signage can be used to facilitate resolving location and/or orientation. Signs that exist in physical space, are defined and placed in the digital map representation, which are then uniquely identified by usage of perception technologies (e.g., image and/or text recognition). In such implementations, the methods and apparatus described herein may not require an operator to 3D map the space, and can instead rely on a floorplan and information of where signs are positioned and oriented.

The methods and apparatus described herein have technical advantages over existing mapping applications that use, for example, GPS and Wi-Fi to localize the device. Specifically, the solutions described herein are configured to precisely localize a user device when a GPS signal is not available (or cannot be used when multiple floors in an interior space are involved), and does not require extra networking equipment to function. The methods and apparatus described herein also have advantages over use of the magnetometer sensor to orient the device's direction (e.g., the magnetometer sensor may be relatively inaccurate, and/or may be impaired by local magnetic fields). The methods and apparatus described herein have advantages over existing augmented reality platform technologies and over existing machine learning technologies, which recognize and read text. In some implementations, the methods and apparatus described herein allow for a proprietor of a venue to update information about a venue (e.g., locations of AR objects and physical objects) without the need for operators to scan the space. The methods and apparatus described herein have advantages over products that rely primarily on GPS that fail to localize a user device position and/or orientation accurately when GPS is not available.

<FIG> is a diagram of a user <NUM> within a real-world physical area <NUM> (e.g., a real-world venue) viewing an AR object P through a mobile device <NUM>. The location (e.g., location and/or orientation, and/or distance and orientation) of the user <NUM> is localized against a location of an AR anchor B, which has a fixed location with respect to an origin O. The origin O is at a fixed location within the real-world physical area <NUM>, which is modeled by a <NUM>:<NUM> scale model or representation. The <NUM>:<NUM> scale model (e.g., model map) of the real-world physical area <NUM> can be referred to as a full-scale model (e.g., a scale-accurate model or representation). The AR object P has a fixed location within the full-scale model of the real-world physical area <NUM>. Unless otherwise indicated, references to a model are considered the same as a reference to a full-scale model.

In some implementations, a location can include a location in X, Y, Z coordinates, and an orientation can include a directional orientation (e.g., direction(s) or angle(s) that an object or user is facing, a yawl, pitch, and roll). Accordingly, a user (e.g., user <NUM>) and/or an AR object (e.g., AR object P) can be at a particular X, Y, Z location and facing in particular direction as an orientation at that X, Y, Z location.

The AR object P is displayed properly within (e.g., on a display screen of) the mobile device <NUM> utilizing a combination of localization of the mobile device <NUM> of the user <NUM> (can be referred to as localization of the user <NUM> and/or localization of the mobile device <NUM>) to the AR anchor B, the origin O, the full-scale model of the real-world physical area <NUM>, and the fixed location of the AR object P within the full-scale model of the real-world physical area <NUM>. The origin O is a common origin (e.g., anchor) to which the AR anchor B and the full-scale model of the real-world physical area <NUM> is oriented (e.g., fixedly tied, bound). In addition, AR objects such as AR object P can also be included (at fixed locations and orientations (e.g., X, Y, and Z coordinate orientations)) within the full-scale model of the real-world physical area <NUM>. Accordingly, the origin O can be used to reconcile (e.g., translate, transform) the locations and/or orientations of AR objects to the mobile device <NUM> (of the user <NUM>) when the mobile device <NUM> is localized to the AR anchor B.

For example, in some implementations, a representation of a real-world scene from the real-world physical area <NUM> can be captured by the user <NUM> using a camera of the mobile device <NUM>. The real-world scene can be a portion of the real-world physical area <NUM> captured by a camera (e.g., the camera of the mobile device <NUM>). A location (and/or orientation) of the mobile device <NUM> can be associated with the AR anchor B based on a comparison (e.g., matching of features) of the representation of the real-world scene with a portion of a full-scale model of the real-world physical area <NUM>. In some implementations, localizing can include determining the location and orientation of the mobile device <NUM> with respect to the AR anchor B. In some implementations, the location and orientation can include a distance from the AR anchor B and direction the mobile device <NUM> is facing with respect to the AR anchor B. Because the AR anchor B has a fixed location with respect to the origin O and because the real-world physical area <NUM> has a fixed location with respect to the origin O, the location and orientation of the mobile device <NUM> with respect to the real-world physical area <NUM> can be determined. Thus, the location and the orientation of the mobile device <NUM> with respect to the AR object P can be determined by way of the AR object P having a fixed location and orientation within the real-world physical area <NUM>. In other words, through localization with the AR anchor B, the orientation of the full-scale model of the real-world physical area <NUM> and the AR object P around the user <NUM> can be determined via the origin O. The AR object P can then be displayed, at the proper location and orientation, within the mobile device <NUM> to the user <NUM>. Changes in the location and orientation of the mobile device <NUM> can be determined through sensors (e.g., inertial measurement units (IMU's), cameras, etc.) and can be used to update locations and/or orientations of the AR object P (and/or other AR objects).

<FIG> is a block diagram illustrating a system <NUM> configured to implement the concepts described herein (e.g., the generic example shown in <FIG>), according to an example implementation. The system <NUM> includes the mobile device <NUM> and an AR server <NUM>. <FIG> illustrates details of the mobile device <NUM> and the AR server <NUM>. Using the system <NUM>, one or more AR objects can be displayed within a display device <NUM> of the mobile device <NUM> utilizing a combination of localization of the mobile device <NUM> to an AR anchor, an origin, a full-scale model of a real-world physical area, and a fixed location and orientation of the AR object within the full-scale model of the real-world physical area. The operations of the system <NUM> will be described in the context of <FIG> and other of the figures.

The mobile device <NUM> may include a processor assembly <NUM>, a communication module <NUM>, a sensor system <NUM>, and a memory <NUM>. The sensor system <NUM> may include various sensors, such as a camera assembly <NUM>, an inertial motion unit (IMU) <NUM>, and a global positioning system (GPS) receiver <NUM>. Implementations of the sensor system <NUM> may also include other sensors, including, for example, a light sensor, an audio sensor, an image sensor, a distance and/or proximity sensor, a contact sensor such as a capacitive sensor, a timer, and/or other sensors and/or different combinations of sensors. The mobile device <NUM> includes a device positioning system <NUM> that can utilize one or more portions of the sensor system <NUM>.

The mobile device <NUM> also includes the display device <NUM> and the memory <NUM>. An application <NUM> and other applications <NUM> are stored in and can be accessed from the memory <NUM>. The application <NUM> includes an AR anchor localization engine <NUM>, a map reconciliation engine <NUM>, an AR object retrieval engine <NUM>, a map and anchor creation engine <NUM>, AR anchor presentation engine <NUM>, and a user interface engine <NUM>. In some implementations, the mobile device <NUM> is a mobile device such as a smartphone, a tablet, and/or so forth.

The system illustrates details of the AR server <NUM>, which includes a memory <NUM>, a processor assembly <NUM> and a communication module <NUM>. The memory <NUM> is configured to store a model map <NUM> (can also be referred to as a model), AR anchors A, and AR objects P.

Although the processing blocks shown in AR server <NUM> and the mobile device <NUM> are illustrated as being included in a particular device, the processing blocks (and processing associated therewith) can be included in different devices, divided between devices, and/or so forth. For example, at least a portion of the map reconciliation engine <NUM> can be included in the AR server <NUM>.

The model map <NUM> stored in the memory can be a three-dimensional (3D) representation (e.g., with depth data) of the real-world physical area <NUM>. In some implementations, the model map <NUM> can be a black and white, or color image (e.g., with depth data). In some implementations, the model map <NUM> can be, or can include a panorama (e.g., with depth data). As an example, the panorama may include an image or a set of images (captured at one location) that extend over a wide angle, e.g., over at least <NUM> degrees, over at least <NUM> degrees, or even over <NUM> degrees. In some implementations, the model map <NUM> can be a point cloud representation that includes points (e.g., a point cloud) in a 3D space that represent the features (e.g., edges, densities, buildings, walls, signage, planes, objects, textures, etc.) within the real-world physical area <NUM>. As described above, the model map <NUM> can be a <NUM>:<NUM> full scale map of the real-world physical area <NUM>. The model map <NUM> (and real-world physical area <NUM>) can be a venue (e.g., a park, a portion of a city, a building (or a portion thereof), a museum, a concert hall, and/or so forth).

<FIG> illustrates a representation of an example model map <NUM> associated with a real-world physical area <NUM>. The representation shown in <FIG> is a two-dimensional (2D) top-down view of the model map <NUM> that includes buildings, streets, trees, etc. An origin O of the model map <NUM> is shown in <FIG>, and the origin O functions as the origin for the relative coordinate system of the model map <NUM>. In other words, the model map <NUM> has a coordinate system that is based on the origin O rather than a GPS coordinate system or another absolute coordinate system tied to the actual location of the real-world physical area <NUM> within the Earth. However, the distances represented within the model map <NUM> can be real-world distances (e.g., meters). The origin O can be an arbitrary point selected or identified within the model map <NUM>. However, the origin O can be used for reconciliation (e.g., coordinate translations, coordinate transformations) with other coordinate systems.

In some implementations, the model map <NUM> can be created by capturing video of a real-world physical area <NUM> using the camera assembly <NUM> and the map and anchor creation engine <NUM>. In some implementations, the model map <NUM>, which is an accurately scaled (e.g., real-world distances (e.g., meters, centimeters) and scale) digital map can be created from a digital map of a location, an architectural diagram, a floorplan (e.g., technical floorplan) of a venue (e.g., an indoor location, planned build out of an event space, and/or so forth), and so forth. In some implementations, a 2D map can be used (e.g., at least partially used) to generate the 3D model map <NUM>. In some implementations, the model map <NUM> can be quickly created (e.g., in under an hour) via the mobile device <NUM> and walk through of the area. This is contrasted with methods that required expensive and complex image capture equipment with specialized capture data. The model map <NUM>, after being captured, can be stored in the AR server <NUM>.

AR objects P1-P9 (e.g., points of interest) are overlaid on the model map <NUM> shown in <FIG>. The AR objects P1-P9 (which can collectively be referred to as AR objects P) have a fixed location (e.g., X, Y, Z location) and orientation (e.g., direction) within the model map <NUM>. The AR objects P have a fixed location and orientation with respect to the origin O (as illustrated by the dashed lines). In some implementations, the model map <NUM> includes AR objects P that are relevant to (e.g., associated with, designed for, identify) a place, event, location, and so forth.

In some implementations, at least one of the AR objects P can be configured to move as the mobile device <NUM> moves user moves or can move even if the mobile device <NUM> does not move. For example, one of the AR objects P, such as a navigation guide (e.g., a wayfinding arrow) used to guide a user, can have a starting point near (e.g., at, in front of) a location and orientation of the mobile device <NUM>. As the mobile device <NUM> moves, the navigation guide can also move (e.g., rotate, move in front of the user) to navigate a user to a desired location.

In some implementations, the AR objects P can each be a fixed locations and orientations within a coordinate space of the model map <NUM>. The AR objects P can each be independent of a real-world coordinate space (e.g., latitude and longitude, a GPS coordinate space). Because the AR objects P are at fixed locations and orientations within the coordinate space of the model map <NUM>, the AR objects P are at full-scale locations and orientations. In other words, the AR objects P can each be at fixed locations and orientations within a coordinate space of the model map <NUM>. In some implementations, the AR objects P can be at fixed locations and orientations (in real-world distances) with respect to the origin O. In some implementations, the AR objects P can be within a coordinate space that is independent of that of the model map <NUM> (but has origin O as a common origin).

In some implementations, the AR objects P can be a label, a 3D model, an interactive immersive model, etc. In some implementations, the AR objects P can be placed within the model map <NUM>. In some implementations, the AR objects P can be placed within the model map <NUM> to facilitate discovery and/or wayfinding using the AR objects P within the real-world physical area <NUM>.

AR anchors A1-A3 are overlaid on the model map <NUM> shown in <FIG>. The AR objects P are also shown. The AR anchors A1-A3 (which can collectively be referred to as AR anchors A) have a fixed location (e.g., X, Y, Z location) and orientation (e.g., direction) within the model map <NUM>. The AR anchors P have a fixed location and orientation with respect to the origin O (as illustrated by the dashed lines). As noted above, the origin O can be an arbitrarily selected origin.

The AR anchors A (which each are unique) can each be a fixed locations (and/or orientations) within a coordinate space of the model map <NUM>. Because the AR anchors A are at fixed locations (and/or orientations) within the coordinate space of the model map <NUM>, the AR anchors A are at full-scale locations (and/or orientations). The AR anchors A can each be a fixed locations (and/or orientations) within a coordinate space of the model map <NUM>. The AR anchors P are at fixed locations (and/or orientations) with respect to the origin O. In some implementations, the AR anchors P can be within a coordinate space that is independent of that of the model map <NUM>. In some implementations, at a minimum each of the AR anchors P have a location (without an orientation) within the model map <NUM>.

The AR anchors A can be used to localize a user <NUM> (e.g., a mobile device <NUM> of the user) to the model map <NUM>. The AR anchors can be considered AR activation markers. The AR anchors A can be created so that the mobile device <NUM> of the user can be localized to one or more of the AR anchors A. For example, the AR anchors A can be an image and/or a representation associated with a location (e.g., point and/or an area) with the real-world physical area <NUM> that corresponds with the full-scale model map <NUM>. Here, the AR anchors A (like in some implementations the model map <NUM>) are a collection of points (namely, a point cloud) that represent features (e.g., edges, densities, buildings, walls, signage, planes, objects, textures, etc.) at or near a location (e.g., point and/or an area) within the model map <NUM>. In some implementations, the AR anchors A can be a spherical image (e.g., color image) or panorama associated with a location within the model map <NUM>. In some implementations, one or more of the AR anchors A can be an item of content. In some implementations, the AR anchors A can be one or more features associated with a location within the model map <NUM>.

Because the AR anchors A can be, for example, an image or representation associated with a location (e.g., point and/or an area) within the model map <NUM>, each of the AR anchors A can be considered as having their own, independent coordinate system (rather than a unified coordinate system). In some implementations, the AR anchors A can be a part of a coordinate space that is relative to the AR anchors A (and independent of other coordinate systems). The AR anchors A can each be independent of a real-world coordinate space (e.g., latitude and longitude, a GPS coordinate space). The locations associated with the AR anchors A are relative (in real-world distances), however, to the origin O. In other words, the AR anchors can be defined with a coordinate space that has an origin common with origin O.

In some implementations, one or more of the AR anchors A can be created by capturing a feature (e.g., an image or a set of images (e.g., a video), a panorama) while the user <NUM> (holding mobile device <NUM>) physically stands a point and/or an area within a real-world physical area <NUM>. The creation of the AR anchors A can be performed using the map and anchor creation engine <NUM>. The captured feature(s) can then be mapped to a location (e.g., collection of features associated with a location) within the full-scale model map <NUM> as an AR anchor A. This information can be stored in the AR server <NUM>.

In some implementations, one or more of the AR anchors A within the model map <NUM> can include uniquely identifiable signs (e.g., physical signs) which will be used as AR activation markers. In some limitations, the signs can include text, QR, custom-designed visual scan codes, and/or so forth. In some implementations, the AR anchors A can be uniquely identifiable physical signs that are connected by location and/or orientation within, for example, the model map <NUM>. The physical signage in a real-world physical area can be used to precisely calibrate the location and/or orientation of the mobile device <NUM>.

As noted above, in some implementations, the model map <NUM>, each of the AR anchors A, and the AR objects P are associated with or are defined within different (e.g., different and independent) coordinates spaces. Accordingly, each of these elements (model map <NUM>, AR anchors A, AR objects P) can be updated dynamically without affecting, in an adverse fashion, the other elements. For example, one or more of the AR anchors A and/or AR objects P can be modified (e.g., updated, deleted, changed) in a desirable fashion. More details regarding dynamic updating are discussed in connection with FIGA. <NUM> and <NUM>. Because of the independent nature of these coordinate spaces, the locations and orientations of the AR objects P with respect to the mobile device <NUM> are resolved (e.g., translated, transformed) by a common tie to the model map <NUM> (and origin O) with the AR anchors A to which the mobile device <NUM> is localized when in use. This system and method can operate accurately even when the captured data during setup is not complete, has inaccuracies, etc. This is contrasted with other systems which may require complete and very accurate, unified data capture during setup.

Referring back to <FIG>, the AR anchor localization engine <NUM> can be configured to determine a location of the mobile device <NUM> based on a comparison (e.g., matching of features) of a representation of a real-world scene with a portion of the full-scale model map <NUM> of the real-world physical area. The comparison can include comparison of features (e.g., edges, densities, buildings, walls, signage, planes, objects, textures, etc.) captured through the mobile device <NUM> with features included in or represented within, for example, the model map <NUM>. In some implementations, the comparison can include comparison of portions of an image captured through the mobile device <NUM> with portions of an image associated with the model map <NUM>.

The camera assembly <NUM> can be used to capture images or videos of the physical space such as a real-world scene from the real-world physical area around the mobile device <NUM> (and user <NUM>) for localization purposes. The camera assembly <NUM> may include one or more cameras. The camera assembly <NUM> may also include an infrared camera. In some implementations, a representation (e.g., an image) of a real-world scene from the real-world physical area <NUM> can be captured by the user <NUM> using the camera assembly <NUM> camera of the mobile device <NUM>. The representation of the real-world scene can be a portion of the real-world physical area <NUM>. In some implementations, features (e.g., image(s)) captured with the camera assembly <NUM> may be used to localize the mobile device <NUM> to one of the AR anchors <NUM> stored in the memory <NUM> of the AR server <NUM>.

Based on the comparison of features, the AR localization engine <NUM> can be configured to determine the location and/or orientation of the mobile device <NUM> with respect to one or more of AR anchors A. The location (and/or orientation) of the mobile device <NUM> can be localized against the location of the AR anchor A through a comparison of an image as viewed through the mobile device <NUM>. Specifically, for example, an image captured by a camera of the mobile device <NUM> can be used to determine a location and orientation of the mobile device <NUM> with respect to the AR anchor A.

An example of localization is illustrated in <FIG>. As shown in <FIG>, the user <NUM> is at a location C1. The location of the user <NUM> is shown in <FIG> within the model map <NUM> for purposes of explanation and by way of example. But, in reality, the user <NUM> is in the real-world physical area <NUM> associated with the model map <NUM> and is merely represented within <FIG>. The user <NUM> is using the mobile phone <NUM> to capture an image of an area (e.g., scene) within the real-world physical area <NUM> using the mobile device <NUM>. The captured image (as an example) of the area (e.g., scene) can be compared with the model map <NUM> to determine the location C1 of the user and the orientation of the user at that location C1. The location and orientation can include determining a distance D1 that the user <NUM> is located from the AR anchor A2, and the direction U that the user is facing, which is toward building <NUM> and to the left of AR anchor A2. The AR anchor A2 can be associated with an image capture that can be compared with the capture of the mobile device <NUM> along direction U. Based on the comparison of the capture along direction U and the capture associated with the AR anchor A2, the AR anchor localization engine <NUM> can determine that the mobile device <NUM> is at distance D1 (and location C1) and facing in direction U relative to the AR anchor A2. Because the AR anchor A2 has a fixed location with respect to the origin O and because the real-world physical area <NUM> represented within the model map <NUM> has a fixed location with respect to the origin O, the location and orientation of the mobile device <NUM> with respect to the real-world physical area <NUM> can be determined.

In some implementations, the localization of the mobile device <NUM> to an AR anchor A can be updated based on movement of the user. For example, if the user moves from location C1 in <FIG> to location C2 in <FIG>, the AR localization engine <NUM> can be configured to determine the location and/or orientation of the mobile device <NUM> with respect to the AR anchor A1 as the user moves to location C2 and away from AR anchor A2. In this example, the location of the mobile device <NUM> is closer to AR anchor A1 than AR anchor A2 when at location C2. The mobile device <NUM> is a distance D2 from (and facing a direction with respect to) the AR anchor A1.

The updating of the localization can facilitate accuracy of display of the AR objects P within the display of the mobile device <NUM> of the user <NUM>. As the mobile device <NUM> moves within the real-world physical area (which corresponds with the model map <NUM>), the location of the user can be inaccurate because of drift in inherent in the sensor systems <NUM>. Dynamically updating the localization of the mobile device <NUM> against the AR anchors A, the inaccuracies due to drift can be reduced or eliminated.

Another example of localization is illustrated in <FIG> where the mobile device <NUM> captures a portion of a corner of a wall and a part of a painting <NUM> (e.g., inside of a building, inside of a building on a particular floor (e.g., of a plurality of floors) of the building). The captured area is shown as area <NUM>. This captured area <NUM> can be used to localize the mobile device <NUM> to the AR anchor E1, which was previously captured (e.g., captured by another mobile device) from a different angle and includes overlapping area <NUM> as illustrated by dash-dot lines. Specifically, the features of the captured area <NUM> can be compared with the features of the captured area <NUM> associated with the AR anchor E1, to localize the mobile device <NUM> to the AR anchor E1.

In some implementations, the AR localization engine <NUM> can be configured to determine the location and/or orientation of the mobile device <NUM> with respect to one or more of AR anchors A by attempting to localize against more than one (e.g., all) of the AR anchors A. In some implementations only one AR anchor A is selected for localization when the user is at a specified location (or area) at a given time (or over a time window). The best match AR anchor A can be selected for localization. In some implementations, the best match can be the AR anchor A closest to the mobile device <NUM>. In some implementations, the best match can be the AR anchor A with the most features matched to the model map <NUM>. In some implementations, the AR anchor localization engine <NUM> determines a confidence score for a recognized AR anchor A. A higher confidence score may indicate that the feature (e.g., image, representation, extracted text, barcode, QR code) from an image is more likely to be associated with the determined AR anchor A than if a lower confidence score is determined.

Even after localizing at one of the AR anchors A, the precise location and orientation of the mobile device <NUM> within the physical real-world may not be known. Only the relative location and orientation of the mobile device <NUM> with respect to the AR anchor A (and within the model map <NUM> by way of the AR anchor A) is known. The ad-hoc capture of feature (e.g., image) information by the mobile device <NUM> is used to determine the relative location of the mobile device <NUM>. Further reconciliation may be required (e.g., with the mobile map <NUM>) to determine the location and orientation of the mobile device <NUM> with respect to the AR objects P.

In some implementations, images captured with the camera assembly <NUM> may also be used by the AR localization engine <NUM> to determine a location and orientation of the mobile device <NUM> within a physical space, such as an interior space (e.g., an interior space of a building), based on a representation of that physical space that is received from the memory <NUM> or an external computing device. In some implementations, the representation of a physical space may include visual features of the physical space (e.g., features extracted from images of the physical space). The representation may also include location-determination data associated with those features that can be used by a visual positioning system to determine location and/or position within the physical space based on one or more images of the physical space. The representation may also include a three-dimensional model of at least some structures within the physical space. In some implementations, the representation does not include three-dimensional models of the physical space.

In some implementations, multiple perception signals (from one or more of the sensor systems <NUM>) can be used by the AR localization engine <NUM> to uniquely identify signage. In some implementations, these include, but are not limited to: image recognition and tracking, text recognition and tracking, AR tracked oriented points, GPS position, Wifi signals, QR codes, custom designed visual scan codes, and/or so forth. In some implementations, the AR anchor localization engine <NUM> identifies signage for localization. In some implementations, uniquely identifiable signage associated with the model map <NUM> can correspond with uniquely identifiable physical signs. In some implementations, the AR anchor localization engine <NUM> identifies one or more codes, such as a barcode, QR code, or another type of code, within an image. The code may then be mapped to an AR anchor A. In some implementations, this mapping of the digital and physical representations allow for precise localization of the mobile device <NUM>.

After the mobile device <NUM> of the user <NUM> has been localized to an AR anchor A, the map reconciliation engine <NUM> is configured to reconcile the coordinate spaces of the model map <NUM>, the AR objects P, and the AR anchors A. In some implementations, the map reconciliation engine <NUM> is configured to reconcile the coordinate spaces of the model map <NUM>, the AR objects P, and the AR anchors A based on the common origin O.

For example, the locations and orientations of the AR objects P (which have fixed locations and orientations within the model map <NUM>) with respect to the mobile device <NUM> can be determined based on the localization of the mobile device <NUM> to at least one of the AR anchors A, which has a fixed relationship to the origin O and a fixed relationship within the model map <NUM>. In other words, the locations and orientations of the AR objects P with respect to the mobile device <NUM> are resolved (e.g., translated, transformed) by a common tie to the model map <NUM> (and origin O) with the AR anchors A to which the mobile device <NUM> is localized. By doing so, one or more of the AR objects P can be displayed within the mobile device <NUM> based on the location and orientation of the mobile device <NUM>, even when the mobile device <NUM> moves within the real-world physical area (and corresponding model map <NUM>).

For example, as shown in <FIG>, the location and the orientation of the mobile device <NUM> with respect to the AR object P6 can be determined by way of the AR object P6 having a fixed location and orientation within the real-world physical area <NUM> represented by model map <NUM>. In other words, through localization with the AR anchor A2, the orientation of the full-scale model of the real-world physical area <NUM> and location and orientation of the AR object P6 around the user <NUM> can be determined via the origin O. The AR object P6 can then be displayed within the mobile device <NUM> to the user <NUM>. As shown in <FIG>, for example, the mobile device <NUM> (via the user <NUM>) is facing in the direction U as determined through the localization process with the AR anchor A2, and the AR object P6 can be displayed within the display device <NUM> of the mobile device <NUM> based on the direction U. Accordingly, the locations and orientations of the AR objects P are displayed within the mobile device <NUM> as resolved (e.g., transformed, translated) by a common tie to the model map <NUM> (and origin O) and the AR anchors A to which the mobile device <NUM> is localized.

Changes in the location and orientation of the mobile device <NUM> can be determined through sensors (e.g., inertial measurement units (IMU's), cameras, etc.) and can be used to update locations and/or orientations of the AR object P6 (and/or other AR objects P1-P5, P7-P9). For example, if the mobile device <NUM> is moved to a direction different than direction U, the display of the AR object P6 can be modified within the display device <NUM> of the mobile device <NUM> accordingly.

Referring back to <FIG>, the AR object retrieval engine <NUM> can be configured to retrieve one or more AR objects P from the AR server <NUM>. For example, the AR object retrieval engine <NUM> may retrieve AR objects P within the model map <NUM> based on the reconciliation of the coordinate spaces of the AR objects P, the model map <NUM>, and the AR anchors A performed by map reconciliation engine <NUM>.

The AR object presentation engine <NUM> presents or causes one or more AR objects P to be presented on the mobile device <NUM>. For example, the AR object presentation engine <NUM> may cause the user interface engine <NUM> to generate a user interface that includes information or content from the one or more AR objects P to be displayed by the mobile device <NUM>. In some implementations, the AR object presentation engine <NUM> is triggered by the AR object retrieval engine <NUM> retrieving the one or more AR objects P. The AR object presentation engine <NUM> may then trigger the display device <NUM> to display content associated with the one or more AR objects P.

The user interface engine <NUM> can be configured to generate user interfaces. The user interface engine <NUM> may also cause the mobile device <NUM> to display the generated user interfaces. The generated user interfaces may, for example, display information or content from one or more of the AR objects P. In some implementations, the user interface engine <NUM> generates a user interface including multiple user-actuatable controls that are each associated with one or more of the AR objects P. For example, a user may actuate one of the user-actuatable controls (e.g., by touching the control on a touchscreen, clicking on the control using a mouse or another input device, or otherwise actuating the control).

An example of an AR object <NUM> displayed within a real-world scene <NUM> is shown in <FIG>. The AR object <NUM> can be stored at an AR server <NUM>. The real-world scene <NUM> without the AR object <NUM> is shown in <FIG>.

An example of AR objects <NUM> and <NUM> displayed within a real-world scenes <NUM>, <NUM> are shown in <FIG>, respectively. Specifically, AR objects <NUM> and <NUM> are related to wayfinding. Such AR objects <NUM>, <NUM> can be stored in and accessed from the AR server <NUM>.

<FIG> illustrate additional real-world scenes <NUM>, <NUM> associated with AR objects <NUM> through <NUM> for wayfinding within a building (e.g., a specific floor of the building). In this example implementations, the AR objects <NUM> (an arrow pointing the direction), <NUM> (a destination marker) can be updated as a user is moving until the user has arrived at the location as shown by AR object <NUM>. The AR objects <NUM> through <NUM> can be stored in and accessed from the AR server <NUM>.

<FIG> is a diagram illustrating dynamic addition of an AR anchor within the model map <NUM>. Specifically, in this example, an AR anchor A4 is added to the set of AR anchors A. The AR anchor A4 can be added to a specific location to the model map <NUM> even after the other AR anchors A have been created (e.g., curated) and used in discovery and wayfinding. The AR anchor A4 can be used for further localization and used to identify and display AR objects P.

Being able to dynamically add AR anchors as separate localization points avoids having to re-create a full, complete, and final model of a real-world, which may be required by other systems. Additional localization AR anchors can be quickly added and used to improve the processing of the system and to localize a user <NUM>.

<FIG> is a diagram illustrating dynamic addition of an AR anchor within the model map <NUM>. Specifically, in this example, an AR object P10 is added to the set of AR objects P. The AR objects P10 can be added to a specific location to the model map <NUM> even after the other AR objects P10 have been created (e.g., curated) and used in discovery and wayfinding. In some implementations, one or more AR objects P can also be moved.

Being able to dynamically add AR objects to the model map <NUM> makes updating the AR world convenient. Additional AR objects can be quickly added dynamically and used to improve the AR experience without having to re-create all aspects of the model.

<FIG> illustrates a method of discovery and/or wayfinding as described herein. As shown in <FIG>, the method includes receiving (by, for example, the AR anchor localization engine <NUM> shown in <FIG>) a representation of a real-world scene captured (e.g., by a user) using a mobile device (block <NUM>). The real-world scene can be a portion of a real-world physical area (e.g., a venue). The capture of the real-world scene can be performed by the sensor system <NUM> shown in at least <FIG>. The method includes associating a location of the mobile device with an AR anchor based on a comparison of the representation of the real-world scene with a portion of a model of the real-world physical area (block <NUM>). The associating of the location (e.g., localization) can be performed by, for example, the AR anchor localization engine <NUM> shown in <FIG>. The method also includes triggering display of an AR object associated with the model of the real-world physical area within the mobile device based on the location of the mobile device (block <NUM>). The display can be triggered by, for example, one or more of the AR object retrieval engine <NUM>, the AR presentation engine <NUM>, and the user interface engine <NUM> shown in <FIG>.

<FIG> illustrates a method of discovery and/or wayfinding as described herein. As shown in <FIG>, the method includes receiving a representation of a real-world scene captured by a mobile device, the real-world scene being a portion of a real-world physical area (block <NUM>). The real-world scene can be a portion of a real-world physical area (e.g., a venue). The capture of the real-world scene can be performed by the sensor system <NUM> shown in at least <FIG>. The method includes localizing the mobile device with an AR anchor from a plurality of AR anchors based on a comparison of a feature of the real-world scene with a feature of a model map of the real-world physical area (block <NUM>). The associating of the location (e.g., localization) can be performed by, for example, the AR anchor localization engine <NUM> shown in <FIG>. The method also includes identifying a location and orientation of an AR object with respect to the mobile device based on a fixed location of the AR object within the model and a fixed location of the AR anchor within the model (block <NUM>). The identifying can be performed by, for example, one or more of the map reconciliation engine <NUM> or the AR object retrieval engine <NUM> shown in <FIG>.

<FIG> illustrates a method of creating a model, and associated elements, for discovery and/or wayfinding as described herein. The flowchart elements can be performed by the map and anchor creation engine <NUM> shown in <FIG>.

The method can include capturing first features associated with a real-world physical area as a model (block <NUM>), and associating an AR object with a fixed location within the model (block <NUM>). In some implementations, the AR object can be associated with a fixed location and fixed orientation within the model. The method can include capturing second features associated with a real-world location corresponding with a portion of the real-world physical area (block <NUM>), and associating the captured second features with a location in the model, corresponding with the real-world location, as an AR anchor (block <NUM>). In some implementations, one or more of the features can be captured in an image or a point cloud.

Referring back to <FIG>, the IMU <NUM> can be configured to detect motion, movement, and/or acceleration of the mobile device <NUM>. The IMU <NUM> may include various different types of sensors such as, for example, an accelerometer, a gyroscope, a magnetometer, and other such sensors. An orientation of the mobile device <NUM> may be detected and tracked based on data provided by the IMU <NUM> or GPS receiver <NUM>.

The GPS receiver <NUM> may receive signals emitted by GPS satellites. The signals include a time and position of the satellite. Based on receiving signals from several satellites (e.g., at least four), the GPS receiver <NUM> may determine a global position of the mobile device <NUM>.

The other applications <NUM> include any other applications that are installed or otherwise available for execution on the mobile device <NUM>. In some implementations, the application <NUM> may cause one of the other applications <NUM> to be launched.

The device positioning system <NUM> determines a position of the mobile computing device <NUM>. The device positioning system <NUM> may use the sensor system <NUM> to determine a location and orientation of the mobile computing device <NUM> globally or within a physical space.

The AR anchor localization engine <NUM> may include a machine learning module that can recognize at least some types of entities within an image. For example, the machine learning module may include a neural network system. Neural networks are computational models used in machine learning and made up of nodes organized in layers with weighted connections. Training a neural network uses training examples, each example being an input and a desired output, to determine, over a series of iterative rounds, weight values for the connections between layers that increase the likelihood of the neural network providing the desired output for a given input. During each training round, the weights are adjusted to address incorrect output values. Once trained, the neural network can be used to predict an output based on provided input.

In some implementations, the neural network system includes a convolution neural network (CNN). A convolutional neural network (CNN) is a neural network in which at least one of the layers of the neural network is a convolutional layer. A convolutional layer is a layer in which the values of a layer are calculated based on applying a kernel function to a subset of the values of a previous layer. Training the neural network may involve adjusting weights of the kernel function based on the training examples. Typically, the same kernel function is used to calculate each value in a convolutional layer. Accordingly, there are far fewer weights that must be learned while training a convolutional layer than a fully-connected layer (e.g., a layer in which each value in a layer is a calculated as an independently adjusted weighted combination of each value in the previous layer) in a neural network. Because there are typically fewer weights in the convolutional layer, training and using a convolutional layer may require less memory, processor cycles, and time than would an equivalent fully-connected layer.

The communication module <NUM> includes one or more devices for communicating with other computing devices, such as the AR server <NUM>. The communication module <NUM> may communicate via wireless or wired networks, such as the network <NUM>. The communication module <NUM> of the AR server <NUM> may be similar to the communication module <NUM>. The network <NUM> may be the Internet, a local area network (LAN), a wireless local area network (WLAN), and/or any other network.

The display device <NUM> may, for example, include an LCD (liquid crystal display) screen, an LED (light emitting diode) screen, an OLED (organic light emitting diode) screen, a touchscreen, or any other screen or display for displaying images or information to a user. In some implementations, the display device <NUM> includes a light projector arranged to project light onto a portion of a user's eye.

The memory <NUM> can include one or more non-transitory computer-readable storage media. The memory <NUM> may store instructions and data that are usable by the mobile device <NUM> to implement the technologies described herein, such as to generate visual-content queries based on captured images, transmit visual-content queries, receive responses to the visual-content queries, and present a digital supplement identified in a response to a visual-content query. The memory <NUM> of the AR server <NUM> may be similar to the memory <NUM> and may store data instructions that are usable to implement the technology of the AR server <NUM>.

The processor assembly <NUM> and/or processor assembly <NUM> includes one or more devices that are capable of executing instructions, such as instructions stored by the memory <NUM>, to perform various tasks. For example, one or more of the processor assemblies <NUM>, <NUM> may include a central processing unit (CPU) and/or a graphics processor unit (GPU). For example, if a GPU is present, some image/video rendering tasks, such as generating and displaying a user interface or displaying portions of a digital supplement may be offloaded from the CPU to the GPU. In some implementations, some image recognition tasks may also be offloaded from the CPU to the GPU.

Although <FIG> does not show it, some implementations include a head-mounted display device (HMD) (e.g., glasses that are AR enabled). The HMD may be a separate device from the mobile device <NUM> or the mobile device <NUM> may include the HMD. In some implementations, the mobile device <NUM> communicates with the HMD via a cable. For example, the mobile device <NUM> may transmit video signals and/or audio signals to the HMD for display for the user, and the HMD may transmit motion, position, and/or orientation information to the mobile device <NUM>.

The mobile device <NUM> may also include various user input components (not shown) such as a controller that communicates with the mobile device <NUM> using a wireless communications protocol. In some implementations, the mobile device <NUM> may communicate via a wired connection (e.g., a Universal Serial Bus (USB) cable) or via a wireless communication protocol (e.g., any WiFi protocol, any BlueTooth protocol, Zigbee, etc.) with a HMD (not shown). In some implementations, the mobile device <NUM> is a component of the HMD and may be contained within a housing of the HMD.

<FIG> shows an example of a generic computer device <NUM> and a generic mobile computer device <NUM>, which may be used with the techniques described herein. Computing device <NUM> is intended to represent various forms of digital computers, such as laptops, desktops, tablets, workstations, personal digital assistants, televisions, servers, blade servers, mainframes, and other appropriate computing devices. Computing device <NUM> is intended to represent various forms of mobile devices, such as personal digital assistants, cellular telephones, smart phones, and other similar computing devices.

The processor <NUM> can be a semiconductor-based processor. The memory <NUM> can be a semiconductor-based memory.

Various implementations of the systems and techniques described herein can be realized in digital electronic circuitry, integrated circuitry, specially designed ASICs (application specific integrated circuits), computer hardware, firmware, software, and/or combinations thereof.

To provide for interaction with a user, the systems and techniques described herein can be implemented on a computer having a display device (e.g., a CRT (cathode ray tube) or LCD (liquid crystal display) monitor) for displaying information to the user and a keyboard and a pointing device (e.g., a mouse or a trackball) by which the user can provide input to the computer.

The systems and techniques described herein can be implemented in a computing system that includes a back end component (e.g., as a data server), or that includes a middleware component (e.g., an application server), or that includes a front end component (e.g., a client computer having a graphical user interface or a Web browser through which a user can interact with an implementation of the systems and techniques described herein), or any combination of such back end, middleware, or front end components.

Claim 1:
A method, comprising:
receiving, by an augmented reality, AR, anchor localization engine (<NUM>) of a mobile device (<NUM>), a representation of a real-world scene (<NUM>; <NUM>, <NUM>; <NUM>, <NUM>) captured using the mobile device (<NUM>), the real-world scene (<NUM>; <NUM>, <NUM>; <NUM>, <NUM>) being a portion of a real-world physical area (<NUM>; <NUM>);
associating, by the AR anchor localization engine (<NUM>), a location of the mobile device (<NUM>) with an AR anchor (A1-A4; B; E1) from a plurality of AR anchors (A1-A4; B; E1) based on a comparison of the representation of the real-world scene (<NUM>; <NUM>, <NUM>; <NUM>, <NUM>) with a portion of a model map (<NUM>) of the real-world physical area (<NUM>; <NUM>), including determining the location and orientation of the mobile device (<NUM>) with respect to the AR anchor (A1-A4; B; E1),
wherein the AR anchors (A1-A4; B; E1) are a point cloud representing features at a location within the model map (<NUM>) and each of the AR anchors is unique,
wherein the model map (<NUM>) has a coordinate space that is based on an origin (O) and the AR anchors (A1-A4; B; E1) are defined within a coordinate space that is independent of that of the model map (<NUM>);
performing, by a map reconciliation engine (<NUM>) of the mobile device (<NUM>), a coordinate translation or transform of the location and/or orientation of an AR object (P1-P9) to the mobile device (<NUM>) using the origin (O), wherein the locations of the AR anchors (A1-A4; B; E1) are fixed relative to the origin (O) and the AR object (P1-P9) has a fixed location relative to the origin (O), the origin (O) being a predefined, fixed location in the real-world physical area (<NUM>; <NUM>); and
triggering display of the AR object (P1-P9) associated with the model map (<NUM>) of the real-world physical area (<NUM>; <NUM>) within the mobile device (<NUM>) based on the location of the mobile device (<NUM>).