Augmented reality device, system, and method for safety

Disclosed is a device, system, and method utilizing augmented-reality (AR) for safety. The AR safety system generates hazard-response messages in response to sensor data in order to help users understand and response to hazards in work environments in an intuitive and effective way. The AR safety system fuses a plurality of technologies. These technologies include person location/tracking, hazard detection/tracking, and augmented reality (i.e., AR).

FIELD OF THE INVENTION

The present invention relates to safety systems and, more specifically, to an augmented reality (AR) hazard response system and display.

BACKGROUND

Industrial environments typically have safety systems that include alarms for notifying workers of hazardous conditions (i.e., hazards). Commonly, these alarms present limited information and are often merely audio/visual alarm signals that indicate the detection of some hazard.

Workers may follow prescribed exit plans in response to the alarm signals. When these exit plans are blocked, the worker must choose an alternate route using only their knowledge of the workplace and their knowledge of the hazard. Often, it is impossible for a worker to understand the nature or extent of a hazard, and as a result, the worker must guess.

Augmented reality (AR) displays are becoming more common in industrial workplaces. The AR displays present a user with a live direct or indirect view of the user's environment. This view is supplemented with computer-generated information (e.g., sound, video, graphics, text, data, etc.), which may be overlaid with objects/areas in the displayed view to provide contextual information. This contextual information helps a worker understand and avoid hazards in the workplace.

Therefore, a need exists for personal safety system or display that utilizes augmented reality to help a user understand and respond to hazards.

SUMMARY

Accordingly, in one aspect, the present invention embraces an augmented reality (AR) hazard display. The display includes a user-tracking subsystem for locating a user's position in real-time. The display also includes a communication subsystem for receiving real-time hazard data from at least one sensor. The display further includes a processing subsystem configured to convert the user's position and the hazard data into augmented reality messages, and an interface subsystem to convey the AR messages to the user.

In an exemplary embodiment, the AR hazard display's user-tracking subsystem includes a global positioning system (GPS) receiver.

In another exemplary embodiment, the AR hazard display's user-tracking subsystem includes at least one camera.

In another exemplary embodiment, The AR hazard display's user-tracking subsystem includes a wireless-network receiver.

In another exemplary embodiment, the AR hazard display's interface subsystem includes speakers for creating three-dimensional (3D) audio sounds.

In another exemplary embodiment, the AR hazard display's interface subsystem includes a head mounted display (HMD).

In another exemplary embodiment, the AR hazard display's augmented reality (AR) messages include visual images of a map showing the user's position and a route to a safe place.

In another exemplary embodiment, the AR hazard display's augmented reality (AR) messages include prompts to guide a user's movement.

In another aspect, the present invention embraces an augmented reality (AR) safety system. The system includes a hazard-tracking subsystem including at least one sensor for gathering hazard data. The system also includes a user-tracking subsystem for tracking a user's position. Further, an AR hazard display is communicatively coupled to the hazard tracking subsystem and the user-tracking subsystem for generating and presenting hazard-response messages to the user.

In an exemplary embodiment, the AR safety system's AR hazard display is a smartphone.

In another exemplary embodiment, the AR safety system's at least one sensor is worn by the user.

In another exemplary embodiment, the AR safety system's at least one sensor is in a fixed position and not attached to the user.

In another exemplary embodiment, the AR safety system's at least one sensor is a gas sensor for detecting an invisible gas.

In another exemplary embodiment, the AR safety system's at least one sensor is a gas sensor for detecting an invisible gas, and the hazard-response messages include graphics to help a user visualize the invisible gas.

In another exemplary embodiment, the AR safety system's at least one sensor is a gas sensor for detecting an invisible gas, and the hazard-response messages include graphics to help a user visualize the direction in which the invisible gas is moving.

In another exemplary embodiment, the AR safety system's hazard-response messages include a map indicating hazardous areas.

In another exemplary embodiment, the AR safety system's hazard-response messages include an alert that the user is near a hazardous area or has entered into a hazardous area.

In another exemplary embodiment, the AR safety system's hazard-response messages include navigational prompts to guide a user to a safe area.

In another exemplary embodiment, the AR safety system's user-tracking subsystem includes at least one camera.

In another aspect, the present invention embraces a computer-implemented method for generating an augmented-reality (AR) hazard-response message. The method includes the steps of gathering hazard data from at least one sensor; gathering user-position data from at least one user-tracking sensor; and capturing images from a camera that is aligned with a user's sight line. The method also includes the step of generating an AR hazard-response message from the hazard data, the user-position data, and the captured images.

DETAILED DESCRIPTION

The present invention embraces a safety system that utilizes augmented-reality (AR) hazard-response messages on an AR hazard display to help users understand and response to hazards in work environments (e.g., inside a work facility). The AR safety system fuses a plurality of technologies. These technologies include person (i.e., worker, user) tracking, hazard tracking, and augmented reality (i.e., AR).

User Tracking

A user-tracking subsystem is included in the safety system to help locate/track a user. Typically, this location/tracking information is combined with contextual information (e.g., a floor plan of a building) to produce an indication of the user's position.

The user-tracking subsystem is worn, held, or otherwise possessed by a user. In an exemplary embodiment, the user-tracking subsystem includes the processing electronics necessary to track its position from signals transmitted by a network of stationary transmitters. Here, the stationary transmitters are indifferent to user's user-tracking subsystem and serve merely as beacons. In another exemplary embodiment, however, the user may be located/tracked by an infrastructure system using a network of sensors distributed around a workplace. Here, the user-tracking subsystem may include the processing/electronics necessary to transmit an identification signal to the infrastructure system and, in return, receive the user's position. In yet another exemplary embodiment, the infrastructure system may include a network of cameras (e.g., visible cameras, infrared cameras, etc.) that collect images of users. The images may be processed using recognition software running on the infrastructure system to identify a user. Here, the user-tracking subsystem need not transmit any signal and can just receive the user's position from the infrastructure system.

The user-tracking subsystem may be configured to receive radio frequency (i.e., RF) signals, and include the necessary antennas, RF filtering, RF amplification, and digitization electronics to convert the RF signals into user-position data. Alternatively, the received signals may be optical signals (e.g., infrared or visible), in which case the user-tracking subsystem may include the necessary lenses, optical filters, and detectors necessary for converting the optical signals into user-position information.

In an exemplary embodiment, the user-tracking subsystem includes a global position system (GPS) receiver for receiving signals from satellites. GPS signals are not typically available indoors (e.g., due to signal blockage and/or multipath interference), and so within buildings other subsystem may be employed. In general, it may be advantageous for the user-tracking subsystem to include the electronics and processing necessary for location/tracking in a variety of environments (e.g., indoor and outdoor).

In another exemplary embodiment, the user-tracking subsystem may include a BLUETOOTH™ receiver for receiving low energy signals (e.g., Bluetooth low energy, BLE, signals) from transceivers placed throughout a workplace (e.g., the interior of a building). These BLUETOOTH™ transceivers may transmit messages over a short range. The limited range of the BLE transceivers helps to facilitate an understanding of position. For example, a user-tracking subsystem receiving BLE messages from a particular BLE transceiver implies that the user-tracking subsystem is in proximity with the particular BLE transceiver.

A BLUETOOTH™ enabled user-tracking subsystem may also include the necessary processing and electronics to exchange information with the BLUETOOTH™ transceivers. For example, the user-tracking subsystem may broadcast messages that cause a BLUETOOTH™ transceiver to respond. Further, the user-tracking subsystem may identify a user by transmitting an identification message to a BLE transceiver.

In another exemplary embodiment, the user-tracking subsystem may include a wireless local area network (i.e., WLAN) receiver (e.g., Wi-Fi receiver) for determining a user's position. Here, the user-tracking subsystem may include a WLAN receiver and the necessary electronics and processing to measure received signal strength. The user-tracking subsystem may use the received signal strength and knowledge of the WLAN transmitter's location to determine a user's position.

In another exemplary embodiment, the user-tracking subsystem may include variety of sensors (e.g., laser range finders, LIDAR, SONAR, cameras, etc.) in addition to one or more of the user-tracking subsystem embodiments described above for simultaneous localization and mapping (i.e., SLAM). Here the user-tracking subsystem gathers information from the sensors and receivers and constructs/updates maps of the workplace.

As mentioned previously, the location/tracking information may be combined with maps of a workplace to produce an indication of a user's position. These maps of the workplace may also include additional information regarding hazardous and/or restricted areas (e.g., construction zones, high voltage areas, hazardous material storage areas, etc.). This additional information may also be indicated on the maps of the workplace.

In an exemplary embodiment, a plurality of user positions may be indicated on the maps of the workplace. For example, a map showing the location of workers, the floor plan of the workplace, and hazards in the workplace may be displayed to a worker or to a third party (e.g., a supervisor, security personnel, etc.) to facilitate situational awareness. This situational awareness may help during a disaster (e.g., locate workers for rescue).

Hazard Tracking

A hazard-tracking subsystem is included in the safety system to help locate/track hazards in the workplace. The hazard-tracking subsystem includes at least one sensor for gathering hazard data. This sensor (or sensors) may be worn, held, or otherwise possessed by a user (i.e., body worn). Alternatively, sensors may be located in fixed positions throughout a workplace. Each sensor may indicate the sensor's position, a hazard (e.g., gas leak), and in some cases additional aspects of the hazard (e.g., gas concentration).

The hazard-tracking subsystem's plurality of sensors may be spaced apart but otherwise identical in order to improve coverage. Alternatively, the hazard-tracking subsystem's sensors may be spaced apart and different in order to improve coverage and/or to provide complimentary/supporting information. For example, a plurality of stationary gas detectors may detect different concentration levels of gas in different areas. These concentration levels may be mapped to help understand the source of the gas leak. In another example, a gas sensor's data may be combined with a weather sensor's wind data in order to estimate the movement of the gas.

The hazard-tracking subsystem's sensors may sense a hazard directly (e.g., by monitoring the environment) or indirectly through some action taken by a user. For example, user may indicate an alarm by setting the position of an alarm switch.

The hazard-tracking subsystem may reconfigure (e.g., switch on/off) sensors in response to a hazard. For example, if a hazard in a specific area is detected then additional sensors in the hazard area may be enabled to understand other aspects of the hazard.

Augmented Reality

An augmented reality (AR) hazard display is included in the safety system to help a user understand and respond to a hazard. The AR hazard display is an interface device that provides a user with hazard-response messages that augment the user's perception of his/her environment. These hazard-response messages contain information relevant to the user's position and relevant to the hazard. The hazard response messages may change in real-time as either the user's position or the hazard changes. To accomplish this, the AR hazard display typically includes a processor, display (e.g., projection, LCD screens, etc.), sensors (e.g., camera, accelerometer, gyroscope, compass, etc.), and input devices (e.g., speech recognition device, gesture recognition device, stylus, keyboard, pointer, glove, etc.) to sense a user's movement/position and to adjust the information presented on the display in response.

Hazard-response messages may be customized for a specific user, location, and/or time. Alternatively, hazard-response messages may be generalized.

The hazard-response message may include visual, tactile, and/or audio prompts. Visual prompts may include text, graphics, and/or images. The visual prompts may be colored to convey information (e.g., severity of hazard). In some possible embodiments, the visual prompts may have a three-dimensional (3D) aspect. The position of the visual prompts on a display may convey an association between the visual prompt and some location/object in the environment.

Tactile prompts may include pressure (e.g., vibrations) applied to a user in order to information (e.g., direction). For example, a vibration on the user's right side could convey that the user should turn to the right. Further, the amplitude of the pressure and/or the frequency of a vibration may also be used to convey information (e.g., proximity).

Audio prompts may be produced by audio sources arranged stereoscopically to produce a three-dimensional audio effect (i.e., 3D audio). For example, 3D audio sounds (e.g., (prompts instructing a user to move toward the sound) can guide a user in environments with low or no visibility. Exemplary audio sources include speakers, earphones, and bone conduction devices.

In one possible embodiment, the AR hazard display is a smartphone configured with a camera, display, processing, and software necessary to capture images of an environment, to display the images of the environment in real time, and to display (e.g., simultaneously) hazard-response messages.

In another possible embodiment, the AR hazard display includes a near-to-eye (NTE) device. The NTE device includes a wearable display configured with a screen worn in proximity to the user's eye. The screen is transparent, allowing the user to see through the screen. The user may also observe images projected by the NTE device and guided to the user's eye via optical waveguides, polarization optics, reflection optics, and/or diffraction optics.

In another possible embodiment, the AR hazard display includes a head-mounted display (HMD). Like the NTE device, the HMD is worn on a user's head. Unlike the NTE device, the user does not observe his/her environment directly. Instead, the HMD is typically configured with one or more cameras (e.g., two cameras in a stereoscopic arrangement) that capture real time images of a user's environment. One or more displays present the images to a user.

In another possible embodiment, the AR hazard display includes a projector. The projector may be configured with a light source (e.g., lasers, LEDs, filament source, etc.), an imager (e.g., digital micro-mirror device, scanning mirrors, liquid crystal spatial light modulator, etc.), and projection optics (e.g., lenses) to project hazard-response messages onto a wall, ceiling, or floor.

Hazard-response messages may contain content to help a user understand a hazard. Visual prompts may be formed from sensor data to illustrate aspects (e.g., extent, concentration, predicted movement, etc.) of an invisible hazard (e.g., gas plume). In another example, visual prompts may indicate virtual barriers (i.e., geo-fence) surrounding hazardous areas (e.g., construction areas). More specifically, a geo-fence could be created in response to a hazard. Here the hazard tracking system could use sensor data to establish the perimeter of a hazard. The safety system could combine this hazard perimeter information with a user's position data to create a message (e.g., an AR message) when the user entered into a hazardous area. Hazard response messages may display probabilistic information regarding the hazard (e.g., most like direction the hazard is moving). This information may be displayed via colors and/or numbers.

Hazard-response messages may contain content to help a user respond to a hazard. For example, navigational messages may direct a user to the nearest exit or safe place in an emergency. The system may obtain the destination and user routing algorithms to guide the user along the best route.

Exemplary Embodiments

An exemplary AR hazard display is shown inFIG. 1. The display includes a user-tracking subsystem1for locating a user's position. Typically, the user-tracking subsystem is configured to receive positioning information2from beacons or sensors in the workplace. The display also includes a communication subsystem3for receiving real-time hazard data4from at least one sensor. A processing subsystem5, communicatively coupled to the user-tracking subsystem1and the communication subsystem2. The processing subsystem includes a computer readable memory (e.g., RAM, ROM, etc.) and a processor (e.g., controllers, processors, etc.) for converting the user's position and the hazard information into AR messages. The processing subsystem is also communicatively coupled to an interface subsystem6that presents the AR messages to a user. The interface subsystem6may also receive information from the user (e.g., head position, sight line, etc.) and convey this information to the processing subsystem5so only AR messages relevant to the user's position are generated and displayed.

An exemplary AR safety system is shown inFIG. 2. The system includes a hazard-tracking subsystem10. The hazard-tracking subsystem detects and tracks hazards via a sensor or sensors. The sensors may be in fixed locations as part of a workplace infrastructure or may be body worn by a user. The system also includes a user-tracking subsystem11for tracking a user's movements. The user-tracking subsystem11may track a user's movements through installed cameras or sensors in the workplace. Alternatively, a user may possess the user-tracking subsystem11. The user-tracking subsystem11and the hazard tracking subsystem10send information to a communicatively coupled AR hazard display12, typically worn or held by the user. The AR hazard display12processes this information and generates hazard-response messages. The AR hazard display presents the hazard-response messages to the user.

An exemplary computer-implemented method for generating a hazard-response message is shown inFIG. 3. The method includes the step of gathering data (e.g., hazard information) from sensors20. The method further includes the step of gathering user position data from sensors21. In addition, the method includes the step of capturing images (e.g., images of a field of view corresponding with a user's head position)22. Finally, the method includes the step of generating AR hazard response messages from the gathered/captured data23.