Patent Description:
For example, testing of holographic apps is described in <NPL>.

The subject-matter of the claims is presented. A method, apparatus, and computer-readable storage medium for testing Augmented Reality (AR) applications are described. In an embodiment the method comprises:.

Augmented Reality (AR) systems do not provide support for testing AR applications. The testing of an AR application may be defined as a process to evaluate the functionality of the AR application to find out whether the AR application meets the specified requirements and/or to ensure the AR application is free from defects (e.g., evaluate functionality, robustness, quality, etc. of the AR application). A proposed solution to this technical problem includes using a device simulator running on the AR system to test AR applications. In an example implementation, the device simulator may be configured to run on a computing device configured with the AR system. A physical model that runs on the device simulator may be created and the movement of the physical model controlled by a user. The physical model when driven by the user generates data that is similar to data generated by real movement of a real device. The generated data is forwarded to a tracking framework of the AR system which may be displayed to the user so that the user can verify the performance of the AR application. The technical advantages of using a simulated device described above to test AR applications include the flexibility in testing AR applications from the comfort of one's desk, in various virtual environments, and different device configurations. This achieves efficient use of time and resources and results in cost savings and improved quality of AR applications.

<FIG> illustrates a block diagram of a computing device <NUM> which includes a device simulator <NUM>, according to at least one example implementation. In some example implementations, the device simulator <NUM> can be used for testing AR applications, for example, AR application <NUM>, as described herein.

The computing device <NUM> includes a processor <NUM>, a memory <NUM>, an AR framework <NUM>, application programming interfaces (APIs) <NUM>, and/or the device simulator <NUM>. The device simulator <NUM> receives input from, for example, from a user, via user input <NUM> and the computing device <NUM> outputs graphics to a display via, for example, display to user <NUM>.

The AR framework <NUM> includes an AR tracking system <NUM> to support AR experiences by integrating virtual content with the real world as seen through a device's camera using the APIs <NUM>. For example, in some implementations, the AR tracking system <NUM> may support AR experiences by motion tracking, environmental understanding, light estimation, etc. The motion tracking may allow a device to understand and track the device's position relative to the word. The environmental understanding may allow a device to detect the size and location of all types of surfaces, e.g., horizontal, vertical, and angled surfaces, etc. The light estimation may allow a device to estimate the environment's lighting conditions.

The AR tracking system <NUM> tracks the position of a device (e.g., a mobile device) as the device moves in the real world (space) and builds its own understanding of the real world. The AR tracking system <NUM> uses sensors (e.g., camera, accelerometer, gyroscope, inertial measurement unit (IMU), global positioning system (GPS), etc.) in the device to identify interesting points (e.g., key points, features, etc.) and tracks how such points move over time. The AR tracking system <NUM> may determine position, orientation, etc. of the device as the device moves through the real world based on a combination of the movement of these points and readings from the device's sensors.

In addition to identifying the interesting points, the AR tracking system <NUM> may detect flat surfaces (e.g., table, floor, etc.) and may also estimate the average lighting in the surrounding areas (or environment). These capabilities of the AR tracking system <NUM> may combine to enable the AR framework <NUM> to build its own understanding of the real world around it. Further, the understanding of the real world lets a user place objects, annotations, or other information to seamlessly integrate with the real world. For example, a user may place objects, e.g., a napping kitten on the corner of a coffee table, annotate a painting with biographical information about the artist, etc. The motion tracking functionality of the AR tracking system <NUM> allows the user to move around and view these objects from any angle. For instance, if the user turns around and leaves the room, and comes back later, the objects (e.g., napping kitten on the corner of the coffee table or the annotations on the painting) will be where the user placed them.

The AR framework <NUM> may include an abstraction layer <NUM> (for example, a hardware abstraction layer. The abstraction layer <NUM> represents an interface between the operating system (OS, not shown) of the computing device <NUM> and the device simulator <NUM>. That is, the abstraction layer <NUM> provides the interface which enables the OS of the computing device <NUM> to be agnostic about lower-level driver implementations. In an example implementation, the abstraction layer <NUM> may support functionality to be implemented in the device simulator <NUM> without affecting or modifying the higher level system (e.g., AR framework <NUM> and/or AR tracking system <NUM>). That is, the abstraction layer <NUM> allows APIs <NUM> provided by the computing device <NUM> (or the OS of the computing device <NUM>) to assume that the OS is interacting with a real device (and not a simulated device such as device simulator <NUM>). The APIs <NUM> provided by the OS of the computing device <NUM> may include Graphics APIs, Sensor APIs, Camera APIs, etc..

The device simulator <NUM> simulates a mobile device (e.g., a mobile phone, a tablet, etc.) on the computing device <NUM> (e.g., a desktop computer). In an example implementation, the device simulator <NUM> may be an application running on the computing device <NUM>. The device simulator <NUM> can provide most of the capabilities of a real mobile device and/or may have pre-defined/pre-configured configurations for different devices or device types. In some implementations, for example, a user may also configure the device simulator <NUM> to simulate, for example, a location of the device, network speeds, rotation, sensors (e.g., camera, accelerometer, gyroscope, IMU, GPS, etc.), etc..

In some implementations, each instance of the device simulator <NUM> may use a virtual device configuration to configure size, form factor, OS version, and other desired hardware characteristics, and may function as an independent device with its own private storage. For example, the device simulator <NUM> may store user data, SD card data, and/or cache associated with a virtual device in a directory specific to that virtual device. When a user launches the device simulator <NUM>, the device simulator <NUM> loads the user data, SD card data, and/or the cache from the associated virtual device directory. The device simulator <NUM>, in some implementations, may further include, for example, user controls <NUM>, simulated sensors <NUM>, and /or a physical model <NUM>. Some or all of the components (e.g., <NUM>, <NUM>, <NUM>, and <NUM>) of the device simulator <NUM>, together or in combinations, may be stored in the memory <NUM> and/or implemented by machine-readable instructions executed by the processor <NUM>.

The physical model <NUM>, in some implementations, may be used to simulate the movement of a real device. That is, the physical model may be an application running on the device simulator <NUM> that simulates inertial movement of an object through an environment. For example, a user, using the user input <NUM> and via the user controls <NUM> may control the movement (e.g., behavior) of the physical model <NUM> in space. The user may control the behavior of the physical model <NUM> by sending instructions via the user input <NUM> to the user controls <NUM>. In an example implementation, the user controls <NUM> may be controlled via the user input <NUM> which may include, for example, WASD keys, mouse controls, arrow keys, joysticks, trackpads, game controllers, etc. When a user engages the user controls <NUM> using, for example, using a W key of a key board to move the physical model <NUM> forward, the physical model <NUM> moves forward in a way that simulates movement of a real device in space. For example, the movement of the physical model <NUM> may represent a real device moving inertially (e.g., with certain acceleration and velocity, no jumps, etc.) in space. In other words, the physical model <NUM> may simulate movement of a real device moving in space in such a way that the position of the physical model <NUM> is continuous through a second derivative (e.g., acceleration) and the rotation of the physical model <NUM> is continuous through a first derivative (e.g., angular velocity).

As the AR tracking system <NUM> assumes that the device being tracked (e.g., device simulator <NUM>) is a physical device moving through the real world, the movement of the physical model <NUM> is modeled in order to simulate realistic physical movement. A simulation of this physical movement may be managed and exposed to other components via the physical model <NUM>. The physical model <NUM> smoothly interpolates physical parameters describing the current state of the system and expose the ways of controlling the system to other components to allow these components to drive movement of the physical model <NUM> (e.g., instruct the physical model <NUM> to move smoothly to a particular position and rotation) such that position, velocity, acceleration, rotation, angular velocity, etc. of the physical model <NUM> are all continuous.

For example, if the user via user controls <NUM> instructs the physical model <NUM> to move at a velocity of <NUM> meter/second, a real device cannot suddenly move at a velocity of <NUM>/s (starting from a velocity of <NUM>/s), but instead takes some time to move at a velocity of <NUM>/s. In some implementations, the physical model <NUM> may be considered as a master controller as the physical model <NUM> can compute its own location, velocity, acceleration, rotational velocity, etc. The physical model <NUM> computes this information in such a way that they are realistic in nature (e.g., smooth/continuous, no jumps, etc.). In some implementations, the physical model <NUM> may be considered to be acting as a mediator between the user controls <NUM> and other components (e.g., simulated sensors <NUM>) of the device simulator <NUM>. For example, upon receiving instructions (e.g., to simulate movement) via the user input <NUM>, the physical model <NUM> may generate its own data and/or interact with the simulated sensors <NUM> to correct the generated data (e.g., feedback mechanism) to ensure the data generated by the physical model <NUM> represents real movement of a real device.

The simulated sensors <NUM> may include any type of simulated sensors, for example, a virtual camera, an accelerometer, IMU, etc. In some implementations, for example, the feed from a virtual camera may support camera APIs that are built in a way such that the view matrix (e.g., position, orientation, etc.) of the camera feed is set based on the real-time position reported by the physical model <NUM>. This may ensure a highly realistic scene of sufficient complexity is rendered to provide features to the AR tracking system <NUM>. Additionally, in some implementations, the simulated sensors <NUM> report simulated IMU data through the APIs <NUM> (e.g., IMU API) <NUM> at a high frequency and based on the real-time state of the physical model <NUM>. The user controls <NUM> may drive the physical model <NUM> in real-time such that a user can easily and comfortably move the camera through the world.

In some implementations, in order to calculate simulated IMU readings (e.g., accelerometer, gyroscope measurements), acceleration and angular velocity in a device's frame of reference have to be instantaneously computed. A polynomial interpolation (e.g., in 3D space for position; in 4D space for rotation) with a high enough degree is implemented so that the acceleration and angular velocity are continuous, and interpolation to the user-control set target position and rotation over a fixed time period is achieved. This way all derivatives are always available for IMU calculations.

In addition, for tracking to work properly, a sensor (e.g., a virtual camera) cannot be rotated to a point. Instead, the virtual camera may be placed at an offset from the center of rotation. For example, the location of the virtual camera may have a minor offset from the location of the virtual device that the user can control and rotate using a user interface of the device simulator <NUM>. That is, for the tracking mechanism <NUM> to work properly, the physical model <NUM> and the simulated sensors <NUM> may be computed or calculated in accordance with the device requirements for the AR framework <NUM>. In an example implementation, this may require particular offsets between virtual cameras and simulated sensor positions. In another example, the physical model <NUM> may simulate the layout of different sensors on a device, e.g., position, rotation, etc. of the IMU relative to the device.

In some implementations, a user may engage the user controls <NUM> to drive the physical model <NUM> forward to test an AR application using the device simulator <NUM>. As the physical model <NUM> moves forward, the physical model <NUM> may generate data related to, for example, position, velocity, acceleration, rotation, etc. of the physical model <NUM>. The data generated by the physical model <NUM> may be shared with the simulated sensors <NUM>. The simulated sensors <NUM> report the generated data via the abstraction layer <NUM> to the AR tracking system <NUM>. In other words, a user may control the physical model <NUM> using user controls <NUM> (via user input <NUM>). Upon receiving input from the user controls <NUM>, the physical model <NUM> may generate data (e.g., position, velocity, acceleration, rotation, etc.) for the physical model <NUM>. The data generated by the physical model <NUM> is shared with the simulated sensors <NUM> via the abstraction layer <NUM> to be forwarded to the AR tracking system <NUM>. This allows the AR tacking system <NUM> to assume that the data (e.g., position, velocity, acceleration, rotation, etc.) is being received from a real device. This allows a user to conveniently/efficiently test AR applications using the device simulator <NUM> from the comfort of their own desks.

In some implementations, for example, the device simulator <NUM> may be used to introduce errors/imperfections in the data generated by the physical model <NUM> to simulate/reproduce error/failure scenarios, for example, sensor noise and error, camera limitations (e.g., lens distortions, motion blur, reduced contrast from conditions like low-light levels, etc.), and miscalibration and misalignment between camera and sensor data, etc., that would be encountered on a real device when running an AR application. This will assist in simulating AR tracking system failure conditions, e.g., loss of tracking (for instance, when walking down a dark hallway), drift between tracked and physical position (objects appear to slide), etc..

The example processor <NUM> of <FIG> may be in the form of a microcontroller, a central processing unit (CPU), an ASIC, a digital signal processor (DSP), an FPGA, a graphics processing unit (GPU), etc. programmed or configured to execute machine-readable instructions stored in the memory <NUM>. The instructions, when executed, may cause the processor <NUM> and/or the components described above, among other things, control the device simulator <NUM> to simulate a real device and the physical model <NUM> to simulate realistic movement of a mobile device in the real world. In some examples, more than one processor <NUM> or more than one memory <NUM> may be included in the computing device <NUM>.

<FIG> illustrates a block diagram of the device simulator <NUM>, according to at least one example implementation. As shown in <FIG>, the simulated sensors <NUM> may include various types of sensors, for example, camera <NUM>, accelerometer <NUM>, IMU <NUM>, etc. As described above, the simulated sensors provide support to the physical model <NUM> so that the physical mode <NUM> can correct the data generated by it. In some example implementations, the physical model <NUM> can use the data from the simulated sensors <NUM> to correct drifting.

<FIG> is a flowchart <NUM> of a method of testing AR applications using a device simulator, in accordance with an embodiment described herein.

At block <NUM>, a user initiates an AR application targeted for testing on a computing device. The user initiates (e.g., launch, start, etc.) the AR application <NUM> on the computing device <NUM>. The user may select the AR application <NUM> to run on the device simulator <NUM> so that the user can test the AR application <NUM> using the device simulator <NUM>. As described above, this allows the user to efficiently test AR applications.

During the initiation of the AR application <NUM>, the user may select a virtual scene (e.g., virtual environment, simulated real-world scene, etc.) to be loaded which is used for testing the AR application. This supports the testing of the AR application using various virtual indoor/outdoor environments depending on the testing requirements. In some implementations, the virtual scenes may be available on the computing device <NUM> for use by the simulated device <NUM>. The virtual scenes provide the capability to test AR application <NUM> in various virtual environments to improve the quality of AR applications and/or user experience.

In some implementations, the user may select the configuration of the device simulator <NUM> based on virtual device configuration, for example, that may be available. The virtual device configuration allows the device simulator <NUM> to configure size, form factor, OS version, memory size, and other desired hardware characteristics, of the device simulator <NUM>. This provides flexibility to the user to test AR applications using various device configurations.

At block <NUM>, the user controls a physical model to simulate movement of a simulated device in a simulated real-world space. The user controls the physical model <NUM> to simulate movement of the device simulator <NUM> in a simulated real-world space. In response to the user controls moving the physical model <NUM>, the physical model <NUM> generates data which represents (or similar to) data generated by real movement of a real device in space. The data generated by the physical model <NUM> is shared with the AR tracking system <NUM> via the abstraction layer <NUM> for testing the AR application.

For example, a user may use the user controls <NUM> to control the physical model <NUM> which generates data that represents a realistic movement of a real device, for example, in a virtual environment. For example, the user initializes AR application <NUM> on the device simulator <NUM> and a loads a virtual scene of a family room. The user may use the user controls <NUM> to simulate movement of the physical model <NUM> and generate data which is forwarded the AR tracking system <NUM> which processes the data and displays to the user via API <NUM>. The user views the performance of the AR application <NUM> via the camera API using a display (e.g., display to user <NUM>) to determine whether the AR application <NUM> is working or performing as intended.

The above defined capability allows the user to test AR applications in any virtual environments (depending on the availability of the virtual environment) from the comfort of his/her desk and with any device configuration by configuring the device simulator <NUM> with the configuration the user wants. This functionality not only provides the capabilities to test AR applications but to efficiently test them.

<FIG> shows an example of a computing device <NUM> and a mobile computing device <NUM>, which may be used with the techniques described here.

For example, computing device <NUM> may be a device on which the emulator <NUM> is configured to run and/or mobile computer device <NUM> may be a mobile device on which applications are run. Computing device <NUM> is intended to represent various forms of digital computers, such as laptops, desktops, tablets, workstations, personal digital assistants, televisions, servers, blade servers, mainframes, and other appropriate computing devices. Computing device <NUM> is intended to represent various forms of mobile devices, such as personal digital assistants, cellular telephones, smart phones, and other similar computing devices.

The processor <NUM> can be a semiconductor-based processor. The memory <NUM> can be a semiconductor-based memory. Also, multiple computing devices <NUM> may be connected, with each device providing portions of the necessary operations (e.g., as a server bank, a group of blade servers, or a multiprocessor system).

In addition, short-range communication may occur, such as using a Bluetooth, Wi-Fi, or other such transceiver (not shown).

These computer programs (also known as programs, software, software applications, software modules, software components, or code) include machine instructions for a programmable processor, and can be implemented in a high-level procedural and/or object-oriented programming language, and/or in assembly/machine language.

A number of embodiments have been described. Nevertheless, it will be understood that various modifications may be made without departing from the scope of the invention.

In addition, the logic flows depicted in the figures do not require the particular order shown, or sequential order, to achieve desirable results. In addition, other steps may be provided, or steps may be eliminated, from the described flows, and other components may be added to, or removed from, the described systems. Accordingly, other embodiments are within the scope of the following claims.

Implementations of the various techniques described herein may be implemented in digital electronic circuitry, or in computer hardware, firmware, software, or in combinations of them. Implementations may be implemented as a computer program product, i.e., a computer program tangibly embodied in an information carrier, e.g., in a machine-readable storage device (computer-readable medium), for processing by, or to control the operation of, data processing apparatus, e.g., a programmable processor, a computer, or multiple computers. Thus, a computer-readable storage medium can be configured to store instructions that when executed cause a processor (e.g., a processor at a host device, a processor at a client device) to perform a process.

A computer program, such as the computer program(s) described above, can be written in any form of programming language, including compiled or interpreted languages, and can be deployed in any form, including as a stand-alone program or as a module, component, subroutine, or other unit suitable for use in a computing environment. A computer program can be deployed to be processed on one computer or on multiple computers at one site or distributed across multiple sites and interconnected by a communication network.

Method steps may be performed by one or more programmable processors executing a computer program to perform functions by operating on input data and generating output.

Processors suitable for the processing of a computer program include, by way of example, both general and special purpose microprocessors, and any one or more processors of any kind of digital computer. The processor and the memory may be supplemented by, or incorporated in special purpose logic circuitry.

To provide for interaction with a user, implementations may be implemented on a computer having a display device, e.g., a cathode ray tube (CRT), a light emitting diode (LED), or liquid crystal display (LCD) monitor, for displaying information to the user and a keyboard and a pointing device, e.g., a mouse or a trackball, by which the user can provide input to the computer.

Claim 1:
A computer-implemented method, comprising:
initiating (<NUM>), on a computing device, an augmented reality, AR, application (<NUM>) targeted for testing;
controlling (<NUM>) a physical model (<NUM>) to simulate movement of a simulated device in a simulated real-world space, the physical model (<NUM>) interpolating physical parameters between a current state and a user control set target position and rotation, and the simulated movement of the physical model (<NUM>) generating data for testing the AR application (<NUM>); and
forwarding the generated data to an AR tracking system (<NUM>) of the AR application (<NUM>) which is adapted to process the generated data and displays the generated data to the user via an application programming interface, API, (<NUM>) to determine whether the AR application (<NUM>) is performing as intended.