Status Quo of AR — Part 1. AR Technology
By: James Zhang
Tags: Facebook, Mobile, Google, Augmented Reality, Technology

Facebook Livemap

AR Technology

Augmented Reality, abbreviated as AR, is the technology that overlays virtual information on top of the real one. For instance, we may see pokemon on the ground dodging the pokemon balls through a smartphone screen, or we may see the exaggerated virtual cosmic cover on faces. This requires machines to understand the environment so well that they figure out how the present information correctly to fool us. In the following sections, I will write about how machines perceive environments from low-level information to a high-level ones.

Positioning

First and foremost, for an AR device likes smartphones or glasses to display 3D virtual content in the real world, they need to be aware of their positions and orientations, which is achieved by SLAM (Simultaneously Localization and Mapping). The machine first extract visual distinct point in the environment called feature points, positions of feature points are tracked and used to infer to the camera position. Combined with IMU (Inertial Measurement Unit, used to measure orientation), the poses of a camera can be estimated.

Position Tracking

If we collect and cluster detected feature points, we can recognize some primitive geometries such as horizontal or vertical planes. After detecting planes, we can do some simple interaction with the real environment by placing virtual on top of a plane.

Occlusion

By simply estimating camera pose and primitive geometry, the occlusion issue is unsolved. In the below picture, we can track the pose of an object. But as long as a real object is in front of the virtual one, immersion will be broken. Because the geometry of the person’s legs is not densely reconstructed, the algorithm can’t decide which part of the pokemon is behind the legs to hide the occluded parts.

Occlusion

To tackle this problem, one of the solutions is to compute the depth value of each pixel by adding depth sensors like Lidar, or through an algorithm to approximate the depth values.

Depth information can also be used to construct environmental mesh and simulate physics effortlessly.

from 6D.ai

Lighting

In terms of realism, a component is still missing — the lighting. In the real world environment, there are lighting from light sources and inter reflections of objects. To display virtual items realistically, we need to reconstruct environmental lighting for rendering so that virtual objects can blend in.

Semantic Information

The above methods only extract physical information. Empowered by AI algorithms, AR devices can detect semantic information such as facial, object, or text. The high-level information facilitates applications such as facial filters or text translation.

Google Translation AR

The state-of-the-art of AR technology is the capacities of machines to recognition location. This task is called feature matching, where algorithms can determine whether two images belong to the same location and even estimated the relative pose between them.

If feature points of a specific location are stored on a database which sometimes is called AR cloud, we can query the saved feature whenever a camera looks at the same location. And the algorithm computes the relative position and recovers the virtual objects placed in the previous sections.

Location recognition coupled with AR cloud can further bridge the gap between the virtual and real world.

Imagine you can place a note in front of a door, and other people can scan the door with smartphones to retrieve the note as a reminder. This persistent experience is essential for AR as a social tool, which is the focal point of many companies.

Placenote Persistent AR

Contextual Information

Beyond existing technologies, imagine machines can understanding arbitrary context. Your AR classes can understand the layout of your home and display information at the hall to remind you to bring an umbrella or recommend your diet choices or health information at kitchens or refrigerators. Integrating digital information from high-level context to low-level poses, Facebook Reality Labs envisions LiveMaps as a representation of the real world, which could radically change how humans behave in a meaningful manner with AR technology.