SYSTEMS, METHODS, AND GRAPHICAL USER INTERFACES FOR AUGMENTED REALITY SENSOR GUIDANCE

Systems and methods for real-time environmental sensor data gathering is enhanced using augmented reality, with a virtual target object being presented to the user of the sensor device that guides the user where to move the sensor device next. A combination of pose data for the sensor and data modeling of the sensor data allows for users with minimal training to make optimized environmental readings.

BACKGROUND

Handheld sensors, such as flow field measurement devices, allow for determining sensor values in an area of interest. As data is gathered, the sensor can be moved to different locations within the area to get more data about the entire field.

In engineering applications, measuring three-dimensional fields is an essential part of the development process. New designs are discovered by adapting an experiment's parameters and iteratively optimizing the outcome, examined by measurements, to meet the desired specifications. Predictions of previously run physics simulations are often measured and validated in real-life experiments at later stages of a development cycle. Especially in the field of aerodynamics, to this date, it is challenging to forecast the complex state of flow fields. In environments where substantial uncertainties about the boundary conditions of the governing equations exist, the results of physics simulations are also subject to considerable uncertainty.

Augmented Reality (AR) allows for the overlay of virtual symbols and images over a view of a real-world region.

SUMMARY

The system presented herein provides for an improved manner to gather data using handheld sensors by combining a pose determination of the sensor with augmented reality and sensor location optimization software. By determining the current location of the sensor and the optimal location to move the sensor to, an augmented reality system can provide visualization to allow the sensor holder to know where to move the sensor to optimize data gathering, without any special skill or knowledge from the user.

Novel measurement systems and techniques for quantifying environmental fields based on an Augmented Reality (AR) system and Active Learning algorithms are presented herein. Environmental sampling is an essential tool across various fields in engineering applications, such as site monitoring, environmental protection initiatives, scientific research, engineering, and agriculture. Environmental sampling refers to different methods in different fields but may be summarized as moving sensors at various locations in space and sequentially collecting data points. The task of the sampling system is to deliver the best possible measurement results, reconstruct an optimal prediction of the sampled quantity with respect to its surroundings, and provide the basis for understanding the physical effects or documenting the present conditions. Sampling many different quantities is possible, but they can be formalized in either a scalar or a vector field, depending on the physical effect and the sensor or measurement technique used.

According to a first aspect, a system for taking sensor readings of a region is disclosed, the system comprising: a sensor device configured to take the sensor readings and to provide pose information of the sensor device; computer software on a non-transient medium configured to, when run on a computer, determine a location to take a next sensor reading based on previous sensor readings and the pose information; and a display device configured to display a virtual object at the location overlayed on a view of the region.

According to a second aspect, a method for taking sensor readings of a region is disclosed, the method comprising: taking readings from the region using a sensor device; computing pose information of the sensor device producing pose data; computing a location in the region for a next sensor reading based on previous sensor readings and the pose data; and displaying on a display device a virtual indicator overlaid on a view of the region, such that the virtual indicator is at the location.

Further aspects are disclosed in the descriptions and drawings herein, as understood by one skilled in the art.

DETAILED DESCRIPTION

In embodiments herein, a sensor operator's Augmented Reality device provides a virtual spatial reference indicator to a computer-determined measuring point and offers novel intuitive interaction techniques for sampling environmental fields in real time that is herein referred to as Spatial Sensing. Integrating the operator into a closed-loop Active Learning framework during the measurement shifts the expertise about the measurement process to the expert algorithm (ML/AI). It allows many operators or frontline workers to use the proposed method to make decisions informed by real-time quantitative feedback.

As used herein, “augmented reality” refers to the combination of real-world views/video with virtual objects/icons/text (computer generated) viewed together. In some embodiments, the real-world is viewed through a camera or array of cameras and the virtual objects are added into the video. In some embodiments, the real-world is viewed directly through a transparent screen and the virtual objects are projected or otherwise displayed on the screen.

As used herein, a “display device” or “display generating device” is any device capable of viewing an augmented reality.

As used herein, a “head mounted display” is a display device that the viewer wears on their head.

As used herein, a “sensor device” is a device that is either hand-held, user guided, or wearable that is capable of sampling some environmental factor and converting it to data. Examples of types of environmental factors are described herein.

In an example, an operator with a head-mounted display (HMD) holds a sensor system in her hand. The sensor system has inside-out tracking capability. A pattern of infrared LEDs mounted on the HMD is referenced by the sensor system's cameras, allowing it to calculate a relative pose between the sensor and the operator's head and, combined with the global pose of the HMD, derive a global position and orientation of the environmental sensor. As the computing capabilities of the current HMD devices are limited, a server is connected wirelessly, hosting the data model and analyzing the stream of measured data in real time. If the HMD hardware allows, the data model can also run on the device itself. The task is to sample an environmental field (e.g., a flow field, gas concentration, etc.) by moving the sensor through the field. Holographic overlays indicate to the operator the current measurement state of the sensor, the domain under investigation, and the optimal next sampling location calculated by the data model. A virtual object (target) is presented in the AR overlay to let the operator know where to move the sensor for optimal measurements, as determined by a computer algorithm.

FIG.1Aillustrates an example user interface on a head-mounted device with a display generation component100(e.g., head-mounted display) worn by the human operator105, blending the view of the physical world with the sensor-system110and a virtual indicator125visualizing the current suggested location for the user105to move the sensor head115to. The system can also include virtual indications of the senor readings120, such as arrows showing flow direction and strength. The virtual indicator125can be any shape, such as a circle (shown), square, cross- hairs, point, diamond, etc. and can optionally include side indicators130such as chevrons that can indicate distance from the user by changing size or distance from the central indicator (e.g. close to the central indicator means close to the user, far from the central indicator means further from the user). The AR display can also show other information, such as text or icons displaying device battery life, current field strength at the sensor device, time, warnings, etc. A visual indication (e.g., a progress bar) can be included that displays the measurement's current progress and stage (e.g., exploration or exploitation).

For the determination of the proposed measurement location125, the algorithms from the families of Uncertainty Quantification and/or Data Assimilation are intended (Gaussian Process Regression, Statistical Methods, Kalman Filtering, etc.). This algorithm can be composed of layers of functions and methods, including traveling salesman-like minimization problems for determining the order of sampling of the proposed locations. The results of this algorithm are dynamically updated as the measurement progresses, and additional data is available.

FIG.1Bshows a system similar toFIG.1A, except that the display generation component is a mobile device150with a camera155either built-in or attached to the device (note that the head-mounted display could also use either a built-in camera or external camera).

FIG.2shows an example sensor device for some embodiments. The hand-held sensor device210includes a sensor head215that takes the sensor readings and a handle220to be held by the user. The sensor can include a haptic feedback module225that can, through vibrations, indicate and signal sensor alignment, measurement progress or other information on the system state to the user, thereby augmenting the AR experience.

FIG.3Ashows the system includes a user301, a head-mounted device with a display generation component302(e.g., a head-mounted device (HMD), a display, a projector, a touchscreen, etc.), a user-guided sensor-system303, which includes but is not limited to, an environmental sensor or a subsystem of an environmental measurement system304, a passive component for spatial referencing (e.g., optical motion capture system, magnetic motion capture system, ultrasonic motion capture system, camera inside-out or outside-in tracking, object pose or hand pose detection)305and a communication module (e.g., Universal Serial Bus, Ethernet, Wi-Fi, Bluetooth, etc.)306. In this example, an active component of the spatial referencing system309tracks the location of the passive component305. An onsite computer unit (e.g., personal computer, workstation, etc.)312combines the measurement of the environmental sensor with the location of the sensor-system. The data can then be fed forward to another computer unit or server308, which might or might not be located onsite. A data model executed on one of the computers308or312processing the sensor stream provides a suggested location and orientation of the sensor device303. Therefore, a virtual indicator object307is displayed in the field of view of the user indicating the current optimal location and orientation of the sensor304. This provides an optimized method to sample a scalar or vector field311(e.g., pressure, temperature, fluid velocity, magnetic field, light intensity, gas concentration, radiation, etc.). The communication between303,302,312,309, and308can be through an external wireless network (e.g., 5G, Wi-Fi, Bluetooth), or wired, or a combination thereof. Additional (one or more) external fixed sensors304aand304bcan be used to provide further environmental sensor data for the system.

FIG.3Bshows a system similar to that ofFIG.3A, except in this embodiment the user301uses a mobile device322, such as a computer tablet or smartphone, to view the field311, sensor device303, and the virtual indicator object307, as well as other data/images used in the AR experience.

FIG.3Cshows a system similar to that ofFIG.3A, except in this embodiment the user301uses a wearable sensor device323that is to be moved to the indicator object307displayed on the display component302for the field311. In some embodiments, the wearable sensor (e.g., as depicted inFIG.3C) is used with a mobile device (e.g., as depicted inFIG.3B).

FIG.3Dshows a system similar to that ofFIG.3A, except in this embodiment the user301uses a sensor device343that is to be moved to the indicator object307displayed on the display component302for the field311, and the display component302includes markers333that can be used by cameras345on the sensor device343to help the system determine pose information of the sensor device343. In some embodiments, markers can be light emitting diodes (LEDs), such as infrared LEDs, or specifically colored dots/balls, or retroreflective elements.

Some embodiments include, as the display device, a head mounted display (HMD) capable of its own visual-inertial-odometry/simultaneous localization and mapping algorithm, providing a coordinate system within the user movement. Also, markers333visible to the sensing system345are rigidly attached to the HMD. The markers might be LEDs (visible or invisible spectrum, i.e., infrared), fiducial markers or recognizable shapes or distinct locations. These markers are tracked by the system of343, establishing a reference between the sensor location and the HMD. As the HMD tracks itself, a global location of the marker can be calculated. Further, the system343includes a communication module (e.g., Universal Serial Bus, Ethernet, Wi-Fi, Bluetooth, etc.). One or more processors of the device302or subcomponents of it combine the current measurement of the environmental system with the current location of the sensor-system. The data is then fed forward to a computer unit or server (see308ofFIG.3A), which might or might not be located onsite (e.g., on a cloud server). A data model executed on the computer processing the sensor stream provides a suggested sampling location and orientation of the sensor-system343. Therefore, a virtual object307is displayed in the field of view of the user indicating the current optimal sampling location and orientation. The goal of the method is to sample the scalar or vector field311(e.g., pressure, temperature, fluid velocity, magnetic field, light intensity, gas concentration, radiation, etc.).

FIG.4Ashows an example simplified system according to some embodiments. The sensor device403can include an environmental sensor, processors, communication module, and passive spatial referencing device (e.g., optical motion capture system, magnetic motion capture system, ultrasonic motion capture system, camera inside-out or outside-in tracking). The spatial referencing system409tracks the passive spatial referencing device through its own active spatial referencing device (e.g., cameras/magnetic sensors/acoustic sensors/etc.). The display generation system402can include cameras (to view the surrounding area for AR), pose sensors (to determine the pose of the display), processors, and a display generation component (e.g., screen). The system can also include an external processing device412, a server408, and/or a wireless networking system410to enable communication between the systems.

FIG.4Bshows an example simplified system according to some embodiments. The sensor device413can include an environmental sensor, processors, communication module, and an active spatial referencing device415(e.g., cameras). The display generation system412can include cameras (to view the surrounding area for AR), pose sensors (to determine the pose of the display), processors, and a display generation component (e.g., screen). The system can also include an external server418and a wireless networking system420.

FIG.4Cshows an example simplified system according to some embodiments. The sensor device423can include an environmental sensor424a, processors, communication module, and an active spatial referencing device415(e.g., cameras). The display generation system422can include cameras (to view the surrounding area for AR), pose sensors (to determine the pose of the display), processors, and a display generation component (e.g., screen). The system can also include an external server428and a wireless networking system430. The system can also include one or more external environmental sensor systems424band424c.

FIG.4Dshows an example simplified system according to some embodiments. The sensor device433can include an environmental sensor, processors, communication module, and an active spatial referencing device435(e.g., cameras). The display generation system432can include cameras (to view the surrounding area for AR), pose sensors (to determine the pose of the display), processors, and a display generation component (e.g., screen). The system can also include a wireless networking system420.

FIG.4Eshows an example of a pose sensor system445, which can include one or more of accelerometers, gyroscopes, magnetometers, and cameras.

FIG.4Fshows and example of a sensor device453that includes a haptic feedback module491, in addition to the other components. In some embodiments, the sensor device can also include a microphone or an array of microphone for audio data. Examples of environmental sensor types include, but are not limited to: vector field flow (velocity and/or pressure), temperature, radiation levels, pollution levels, sound and/or light intensity, magnetic flux, gas concentration.

FIG.5Ashows an example of a measurement sequence. As the time of the measurement progresses, different suggested locations of the sensor-system are visualized from507a,507b,507c, and finally,507d. The illustration shows the physical world operator and the location of the virtual object on the left. On the right, the field of view502vof the human operator in augmented reality/mixed reality blending the physical world with the virtual object as viewed from the display generation system502.

The system is aware of the distance of the sensor-system503to the suggested location507and can adjust its behavior accordingly. As an example, the virtual object507of the suggested location only moves once the sensor system503has been placed close enough to its location (and orientation). In another example, the virtual object507a-dis moved once the background process of the data model provides an update, regardless of the position of the sensor-system503.

FIG.5Bshows illustrates the sample sequence ofFIG.5Awith the addition of a physical object within the region of interest of the sampling process. The data model is aware of the object/shape/surface/subsurface/texture511in the physical world due to either scene awareness of the device or registration and tracking with or a-priori knowledge of the scene due to user input. After the sampling process, the measurement data might be stored together with the surroundings' object/surface/texture/shape (e.g., spatial mapping mesh, Neural Radiance Field, Gaussian Splatting), providing valuable context.

The data model can now suggest sampling locations (virtual object locations) in consideration of the object/surface/texture/shape551so as to avoid sensor collision with the object/surface/shape551.

FIG.6Ashows an example simplified diagram of a wearable sensor device603. The device can include an environmental sensor604, a spatial referencing system605, a communications module606, and haptic feedback module631. Examples of wearable sensor devices include wrist worn devices, rings, gloves, armbands, etc.

FIG.6Bshows an example simplified diagram of a hand-held sensor device613. The device can include an environmental sensor614, a spatial referencing system615, a communications module616, and haptic feedback module632. Examples of hand-held devices include wands, rings, tablets, spheres, etc.

FIG.7shows, in some embodiments, different coordinate systems of the presented measurement system and method must be combined for determining the virtual location of the targeting object707. For example, the coordinate system771is provided by the device702and its pose estimation capabilities, while the coordinate system772of the sensor-system703is provided either by the external spatial referencing709or the internal active spatial referencing705. The measurement system may include a way of initial, or periodical, hand-eye-calibration of the display generation device702and the sensor-system703to generate a relation between the two coordinate systems771and772. This calibration process can include, for example computer vision methods of the display device702recognizing a visual target (e.g., QR, active LED patterns, etc.) or object patterns and shapes on the sensor device703. Other feasible methods for example are proximity sensors or movement patterns of703recognized by the presented system or subcomponents of it (e.g., the display device702).

In some embodiments, only the coordinate system771provided by the device702is utilized.

In some embodiments, the system is applied in a closed environment, such as indoors or in a vehicle such as a car, plane, or spacecraft.

In some embodiments, the system is applied outdoors, or in a larger scale environment.

Depending on the setting, the spatial localization of the display device702might be changed (e.g., from inside-out tracking to global positioning system (GPS) or other local ranging techniques).

FIG.8shows an example block model with data streams. The sensor device gathers data803, which is combined805with pose data804from spatial referencing of the sensor device, providing location data for the environmental readings. This is fed into a data model801(e.g. machine learning, neural network, artificial intelligence, data assimilation system) that feeds into an acquisition function algorithm802to determine the optimal location in the AR field to place a virtual object (target)807to guide the user to move the sensor device to next. Optionally, in some embodiments, the shapes/surfaces of surrounding objects have their location (pose) data are included in the data model801to prevent the system from instructing the user to move the sensor device in a way that would cause a collision.

FIG.9shows an example of a display device. The device in this embodiment includes a head-mounted display905for viewing the AR image with straps906to hold the device to the user's head. For spatial location, the device includes markers such as infrared LED markers910a,910b,910c,910d,910eand/or reflective spheres915a,915b,915c.

FIG.10shows an example of a wearable sensor device. The device in this embodiment includes cameras1005for pose measurement with the display device, a processor1010for sensor data processing and/or pose calculation, and reflective elements1015for sensor pose/location determination by external devices.

Sensor Device Motion Capture

The rich sensor suite on modern Augmented Reality headsets offers a way to get a local pose estimation of the user's head (HMD) with respect to the shape and texture of the surrounding space based on Visual-Inertial Odometry (VIO) and Simultaneous Localization and Mapping (SLAM) algorithms. For general AR applications, the accuracy of these algorithms ensures the holographs' consistency and persistence in space between sessions.

Sensor Pose Estimation

In some embodiments, to estimate the pose of the sensor device with respect to the display device, a set of sensors similar to the approach in consumer devices for user interaction is utilized. A precise pattern of infrared LEDs is rigidly attached to the operator's display device. These LEDs are then tracked by cameras on the sensor device. Data from the cameras build the inside-out tracking of the system, looking for the pattern on the head of the operator for pose estimation.

In some embodiments, An Inertial Measurement Unit (IMU) is placed on the sensor device, allowing it to run its own VIO algorithm. This not only increases the accuracy of the sensor pose estimation but also provides more usability for the system.

The examples set forth above are provided to those of ordinary skill in the art as a complete disclosure and description of how to make and use the embodiments of the disclosure and are not intended to limit the scope of what the inventor/inventors regard as their disclosure.

Modifications of the above-described modes for carrying out the methods and systems herein disclosed that are obvious to persons of skill in the art are intended to be within the scope of the following claims. All patents and publications mentioned in the specification are indicative of the levels of skill of those skilled in the art to which the disclosure pertains. All references cited in this disclosure are incorporated by reference to the same extent as if each reference had been incorporated by reference in its entirety individually.