Systems and methods for integrating haptics overlay in augmented reality

Systems and methods for integrating haptics overlay in augmented reality are disclosed. One illustrative system described herein includes a haptic output device. The system also includes a display configured to output a visual effect. The system also includes a sensor for tracking a position of a proxy object. The system also includes a processor configured to: determine a modified visual effect based in part on data received from the sensor, determine a haptic effect based in part on data received from the sensor, transmit a display signal associated with the modified visual effect to the display, transmit a haptic signal associated with the haptic effect to the haptic output device; and output the haptic effect using the haptic output device.

FIELD OF THE INVENTION

The present application relates to the field of user interface devices. More specifically, the present application relates to integrating haptics overlay in augmented reality.

BACKGROUND

Virtual and Augmented Reality (“VR” and “AR”) applications have become increasingly popular. VR applications typically incorporate a dedicated controller to allow the user to feel a touch sensation of the virtual object. For example, a VR user may experience a gun shooting in his hand in the VR application and may simultaneously feel a haptic effect simulating that shooting through the dedicated controller. In AR space, it is difficult to provide a haptic effect to the user. Accordingly, systems and methods for integrating haptics overlay in augmented reality are needed.

SUMMARY

Embodiments of the present disclosure comprise systems and methods for integrating haptics overlay in augmented reality. In one embodiment, a system comprises a haptic output device. The system also comprises a display configured to output a visual effect. The system also comprises a sensor for tracking a position of a proxy object. The system also comprises a processor configured to: determine a modified visual effect based in part on data received from the sensor, determine a haptic effect based in part on data received from the sensor, transmit a display signal associated with the modified visual effect to the display, transmit a haptic signal associated with the haptic effect to the haptic output device; and output the haptic effect using the haptic output device.

In another embodiment, a method for integrating haptics overlay in augmented reality comprises determining a modified visual effect based in part on data received from a sensor, determining a haptic effect based in part on data received from the sensor, transmitting a display signal associated with the modified visual effect to a display, transmitting a haptic signal associated with the haptic effect to a haptic output device; and outputting the haptic effect using the haptic output device.

In yet another embodiment, a non-transitory computer readable medium may comprise program code, which when executed by a processor is configured to perform such methods.

These illustrative embodiments are mentioned not to limit or define the limits of the present subject matter, but to provide examples to aid understanding thereof. Illustrative embodiments are discussed in the Detailed Description, and further description is provided there. Advantages offered by various embodiments may be further understood by examining this specification and/or by practicing one or more embodiments of the claimed subject matter.

DETAILED DESCRIPTION

Reference will now be made in detail to various and alternative illustrative embodiments and to the accompanying drawings. Each example is provided by way of explanation, and not as a limitation. It will be apparent to those skilled in the art that modifications and variations can be made. For instance, features illustrated or described as part of one embodiment may be used in another embodiment to yield a still further embodiment. Thus, it is intended that this disclosure include modifications and variations as come within the scope of the appended claims and their equivalents.

Illustrative Example of Integrating Haptics Overlay in Augmented Reality

Embodiments provide for haptic feedback on virtual objects in AR space. Such embodiments may allow a user to see and feel a texture on a real world object using a haptics overlay on that object. Alternatively, such an embodiment may seamlessly incorporate a wearable peripheral device that is also capable of providing haptic feedback. In other embodiments, just the wearable peripheral device may be used to provide haptic feedback. In some embodiments, AR space comprises VR space.

In one illustrative embodiment, a user is wearing an AR headset and is looking at a proxy object, e.g., a shoe, in real space. The user may choose a material from a library of materials (e.g., materials such as one or more of leather, plastic, rubber, metal, burlap, suede, denim, wood, cloth, ceramic, foamed material, or glass). This material will then be displayed on the proxy object via the AR headset by saying a voice command out loud (e.g., apply leather material) or inputting instructions via a visual menu in a VR or AR headset, a hand controller, a wearable device, or gesture recognition. The illustrative AR display of the AR headset is modified based on the command so that the appearance of the proxy object is modified through the AR display to show the proxy object made of the selected material.

In one illustrative embodiment, a haptic effect associated with the proxy object is output by a haptic output device, e.g., an actuator, that outputs a haptic effect (e.g., a vibration or ESF effect) simulating a texture of the material selected from the library. In still other illustrative embodiments, a haptic effect is produced by a wearable peripheral device, such as a glove. The user interacting with the proxy object feels the haptic effect, and the haptic effect simulates different textures on the surface of the proxy object (e.g., the haptic effect may simulate the texture of one or more of leather, plastic, rubber, metal, burlap, suede, denim, wood, cloth, ceramic, foamed material, or glass).

In one illustrative embodiment, information from a sensor(s) is passed to a processor that is also processing and outputting haptic signals associated with haptic effects associated with a texture. The modified image of the proxy object through the AR display, and the texture output by the haptic output device, can then be modified in real-time to reflect any movement of the proxy object. In addition, or alternatively, the processor can use information from a sensor(s) to tell when a wearable peripheral device comes into “contact” with an AR object. These sensors may further detect movement of the user associated with the proxy object and modify visual and haptic effects based on the user's movement.

This illustrative example is given to introduce the reader to the general subject matter discussed herein and the disclosure is not limited to this example. The following sections describe various additional non-limiting examples of the present disclosure.

Illustrative Systems for Integrating Haptics Overlay in Augmented Reality

FIG. 1shows an illustrative system100for integrating environmental haptics in virtual reality. Particularly, in this example, system100comprises a computing device101having a processor102interfaced with other hardware via bus106. A memory104, which can comprise any suitable tangible (and non-transitory) computer-readable medium such as RAM, ROM, EEPROM, or the like, embodies program components that configure operation of the computing device101. In this example, computing device101further includes one or more network interface devices110, input/output (I/O) interface components112, and additional storage114.

Network device110can represent one or more of any components that facilitate a network connection. Examples include, but are not limited to, wired interfaces such as Ethernet, USB, IEEE 1394, and/or wireless interfaces such as IEEE 802.11, Bluetooth, or radio interfaces for accessing cellular telephone networks (e.g., transceiver/antenna for accessing a CDMA, GSM, UMTS, or other mobile communications network(s)).

I/O components112may be used to facilitate connection to devices such as one or more displays, such as VR and AR headsets or touch screen displays, keyboards, mice, speakers, microphones, cameras, and/or other hardware used to input data or output data. Storage114represents nonvolatile storage such as magnetic, optical, or other storage media included in device101. In some embodiments, I/O components112may comprise VR controllers or AR input devices. In other embodiments, I/O components may comprise a controller or input device in a transportation device, such as a car, or boat. In yet other embodiments, the controllers or input devices may be the user's hands, and sensors108may be able to detect the movements and gestures in free space.

Audio/visual output device(s)122comprise one or more devices configured to receive signals from processor(s)102and provide audio or visual output to the user. For example, in some embodiments, audio/visual output device(s)122may comprise a display such as a touch-screen display, LCD display, plasma display, CRT display, projection display, or some other display known in the art. For use in augmented or virtual reality, audio/visual output device122may comprise a headset comprising a display for each eye, a mobile device, e.g., a mobile phone or tablet, a windshield of a vehicle, or some other display known in the art. Further, audio/visual output devices may comprise one or more speakers configured to output audio to a user.

System100further includes one or more sensors108. The sensors108are configured to detect a haptic output device118and provide appropriate data for use by processor102. Any suitable number, type, or arrangement of sensors can be used. For example, different embodiments may include cameras, lasers, radars, accelerometers, gyrometers, pressure sensors, magnetic sensors, light sensors, microphones, capacitive sensors, touch sensors, tracking sensors, or any combination of such sensors. In one embodiment, a camera, laser mapping, or radar scanning is used to identify the haptic output device118. Such an embodiment may utilize artificial intelligence (“AI”) to make the identification. An accelerometer may be used to detect vibration, displacement, and speed. A gyrometer may be used to sense rotation. A pressure sensor may be used to determine altitude and a magnetic sensor to determine direction or orientation. A light sensor may be used to determine perceived luminosity. And a microphone may be used to detect sound. Any of these sensors may be used in combination with any other sensor.

In other embodiments, the sensor108may comprise an LED detector. In some embodiments, the processor102is in communication with a single sensor108, in other embodiments, the processor102is in communication with a plurality of sensors108.

Although a single haptic output device118is shown here, embodiments may use multiple haptic output devices of the same or different type to output haptic effects. For example, haptic output device118may comprise one or more of, for example, a piezoelectric actuator, an electric motor, an electro-magnetic actuator, a voice coil, a shape memory alloy, an electroactive polymer, a solenoid, an eccentric rotating mass motor (ERM), or a linear resonant actuator (LRA), a low profile haptic actuator, a haptic tape, or a haptic output device configured to output an electrostatic effect, such as an Electrostatic Friction (ESF) actuator. In some embodiments, haptic output device118may comprise a plurality of actuators, for example a low profile haptic actuator, a piezoelectric actuator, and an LRA. Further, haptic output device118may be integrated into a proxy object or into the user's clothing or a wearable device.

In some embodiments, haptic output device118may be used to output thermal and or kinesthetic properties to the user. For example, the thermal conductivity of different materials could be conveyed by overlaying heating and cooling transducers, e.g., Peltier devices. In other embodiments, the force feedback properties could be overlaid with transducers that modify the stiffness of a material, e.g., using electroactive polymer tendons.

Turning to memory104, exemplary program components124,126, and128are depicted to illustrate how a device may be configured to determine and output haptic effects. In this example, a detection module124configures processor102to monitor sensor(s)108to determine characteristics of haptic output device118. For example, detection module124may sample sensor108in order to track one or more of the location, path, velocity, acceleration, pressure, and/or other characteristics of haptic output device118over time.

Haptic effect determination module126represents a program component that analyzes data regarding audio and video characteristics to select a haptic effect to generate. Particularly, module126comprises code that determines, based on the characteristics of haptic output device118and AR environment, one or multiple effects to generate and output by the haptic output device118. Haptic effect determination module126may further comprise code that selects one or more existing haptic effects to output in response to detection of a change in characteristics of haptic output device118. Different haptic effects may be selected based on various combination of these features.

Haptic effect generation module128represents programming that causes processor102to generate and transmit a haptic signal to haptic output device118, which causes haptic output device118to generate the selected haptic effect. For example, generation module128may access stored waveforms or commands to send to haptic output device118. As another example, haptic effect generation module128may receive a desired type of haptic effect and utilize signal processing algorithms to generate an appropriate signal to send to haptic output device118. As a further example, a desired haptic effect may be indicated along with target coordinates for the texture and an appropriate waveform sent to one or more actuators to generate appropriate displacement of the surface (and/or other device components) to provide the haptic effect. Some embodiments may utilize multiple haptic output devices in concert to simulate a feature. For example, a vibration may be utilized to convey a specific texture to the user while, simultaneously, a kinesthetic effect is output to indicate the stiffness of an object, such as the stiffness of a shoe upper's material. Such a configuration is merely illustrative and not the sole way in which such a system may be constructed.

FIG. 2shows another illustrative system for integrating haptics overlay in augmented reality. In the embodiment shown inFIG. 2, a user206is wearing a wearable AR device204and interacting with a proxy object202. Though this embodiment shows the user wearing a wearable AR device204, any type of AR display system may be used. The illustrative system ofFIG. 2may incorporate the system illustrated inFIG. 1for sensing conditions of proxy object202as well as for determining, generating and outputting haptic effects. The wearable AR device204may include a visual display as well as headphones or speakers.

In some embodiments, proxy object202may take the form of any object, including, but not limited to, a shoe, a bag, an article of clothing, an electronic device, a display, or an interface object. In other embodiments, proxy object202may comprise neutral physical characteristics, like color and texture, e.g. a proxy object made of white color with a smooth surface material. In still other embodiments, proxy object202may comprise a high fidelity visual prototype, e.g., a high fidelity visual shoe prototype with photorealistic graphic stenciling, so a visual overlay using wearable AR device204is unnecessary.

The user206may choose any texture from a given library of textures, e.g., leather, plastic, rubber, metal, burlap, suede, denim, wood, cloth, ceramic, foamed material, or glass, and an augmented reality visual effect will modify the appearance of the proxy object through the wearable AR device204. In some embodiments, the user will choose the texture using an input capture device such as an audio capture device, a touchscreen, a keyboard, a mouse, a joystick, or an augmented reality control board. Processor102will receive an input signal from the input capture device and determine the haptic effect and visual effect modification that must be transmitted via a haptic signal and display signal, respectively.

In some embodiments, once the display signal is transmitted, the visual effect will be output through wearable AR device204. So, the user206will be able to view the proxy object with the chosen texture through the wearable AR device204. For example, if the user206chooses a red leather, then the user206will see the proxy object202made of that same red leather through the wearable AR device204.

As the AR visual effect is changed through the choice of texture, a haptic signal is transmitted to the haptic output device(s)118contained in proxy object202. These haptic output devices output a haptic effect associated with the selected texture, which enables user206to feel the chosen texture through the haptic effect produced in proxy object202as the user touches proxy object202.

In some embodiments, sensors108track proxy object202so the texture produced by the haptic effect in proxy object202and the AR visual effect may be felt and seen, respectively, regardless of any movement or rotation of proxy object202by user206. In other embodiments, the speakers or headphones of wearable AR device204may emit audio feedback to convey the sounds user206would hear if an actual textured object were being touched. In still other embodiments, no audio feedback is emitted.

FIG. 3shows another illustrative system for integrating haptics overlay in augmented reality. In the embodiment shown inFIG. 3, a user is wearing a wearable AR device306and a wearable peripheral device304and interacting with a proxy object302. In some embodiments, proxy object302may comprise any of the same features as proxy object202.

Wearable peripheral device304may comprise a glove that covers the user's whole hand, a glove that covers only some of the user's hand and fingers, e.g., only the user's thumb and index finger, pieces that only cover the user's fingertips, a shoe, a shirt, a pair of pants, or any other article of clothing. Wearable peripheral device304and proxy object302comprise one or more haptic output device(s)118, e.g., actuators, capable of outputting a haptic effect to the user.

In some embodiments, the user will not interact with proxy object302. Instead, a chosen object will be displayed as a visual effect through the AR display of the wearable AR device306. The user may again choose a texture and that texture will be displayed on the object being displayed through the AR display. For example, the user may choose a red leather shoe and a red leather shoe may be displayed through the AR display of the wearable AR device306.

A sensor108may be used to track the position or other characteristics of wearable peripheral device304. This allows the system to determine when wearable peripheral device304makes “contact” with the object being displayed through the AR display. Once this “contact” is made a haptic signal will be transmitted to wearable peripheral device304so a haptic effect corresponding to the object and texture chosen may be output by wearable peripheral device304. For example, using the red leather shoe example from above, the user may go to feel the red leather shoe. As the user touches the object based on the AR display of the wearable AR device306, a haptic effect is output to wearable peripheral device304so the user feels as if he is touching an actual red leather shoe.

In other embodiments, a user will use both a proxy object302and a wearable peripheral device304. The chosen texture is displayed through the AR display of wearable AR device306so proxy object302appears to have the chosen texture. In some embodiments, proxy object302and wearable peripheral device304comprise at least one haptic output device118configured to output a haptic effect. One or more sensors108track characteristics of both proxy object302and wearable peripheral device304. Through the use of proxy object302and wearable peripheral device304, a high-fidelity touch sensation may be achieved.

FIG. 4shows another illustrative system for integrating haptics overlay in augmented reality. In the embodiment shown inFIG. 4, a user is wearing a wearable AR device406and a wearable peripheral device404and interacting with a system of proxy objects402, which comprises at least one proxy object. In some embodiments, the proxy objects that make up the system of proxy objects402may comprise any of the same features as proxy object202.

The system of proxy objects402may be arranged to resemble any number of areas, including the interior of a car, the cockpit of a plane, the desk of an office, etc. A user may select a variety of textures to be output throughout the system of proxy objects402. For example, the user might select a black leather texture for the steering wheel of a car with a cherry walnut wooden center console.

In the embodiment ofFIG. 4, a haptic effect may be output via only a system of proxy objects402or via a combination of a system of proxy objects402and wearable peripheral device404. For example, a vehicle dealership may have a system of proxy objects402set up to simulate a certain style of vehicle in a show room. Potential buyers may interact with the system by choosing different materials to be output in different areas of the simulated vehicle. The system of proxy objects402may be able to accurately output haptic effects simulating the texture of the different materials without the potential buyers wearing a wearable peripheral device404. In other embodiments, combining a system of proxy objects402with a wearable peripheral device404. For example, in a gaming environment, a system of proxy objects402may be used to simulate a recurring environment throughout the game (e.g., in a racing game, the system of proxy objects402may simulate the interior of a vehicle). Further incorporating a wearable peripheral device404could allow the use to explore gaming environments outside of the system of proxy objects402as well as enhance the user experience when interacting with a system of proxy objects402.

Illustrative Methods for Integrating Haptics Overlay in Augmented Reality

FIG. 5is a flow chart of method steps for one example embodiment for integrating haptics overlay in augmented reality. In some embodiments, the steps inFIG. 5may be implemented in program code that is executed by a processor, for example, the processor in a general purpose computer, a mobile device, virtual reality or augmented reality control system, or a server. In some embodiments, these steps may be implemented by a group of processors. In some embodiments one or more steps shown inFIG. 5may be omitted or performed in a different order. Similarly, in some embodiments, additional steps not shown inFIG. 3may also be performed. The steps below are described with reference to components described above with regard to computing device101shown inFIG. 1and the system shown inFIG. 2.

In the embodiment shown, a process500begins at step502when a processor, such as processor102, determines a modified visual effect. The processor102receives sensor signals from one or more sensors108. Using the sensor signals, the processor102analyzes the environment to identify characteristics of proxy object202. Based on these characteristics, processor102may determine a modified visual effect. For example, processor102may determine that a blue leather material to display on a proxy object202, in this case a shoe. Based on data from sensors108, processor102may analyze any movement of the shoe by the user and determine the appropriate modified visual effect to display the blue leather material on the shoe.

At step504the processor102next determines a haptic effect based on signals received from sensors108. Similar to step502, the processor102receives sensor signals from sensors108, and using those sensor signals, analyzes the environment to identify characteristics of proxy object202to determine a haptic effect. In some embodiments, processor102determines the haptic effect using a library of textures. For example, processor102may determine the appropriate texture of the blue leather material of the shoe to output as a haptic effect. In some embodiments, processor102may determine the haptic effect to output to simulate a blue leather material on a shoe based on a blue leather haptic effect stored in the library of textures in memory114. The processor102may also receive data from sensor108monitoring any movement of the shoe to determine the haptic effect to output to ensure the movement of the shoe does not interfere with blue leather texture produced by the haptic effect.

At step506the processor102transmits a display signal associated with the modified visual effect to the augmented reality display, e.g., wearable AR device204. For example, when viewed through the wearable AR device204, the shoe will appear to be made of blue leather material. The augmented reality display may comprise at least one of a wearable device or a mobile device. In some embodiments, the modified visual effect modifies the appearance of the proxy object202through the augmented reality display.

At step508the processor102transmits a haptic signal associated with the haptic effect to the haptic output device. For example, the haptic output device(s) in the shoe may output a haptic effect that simulates the texture of blue leather on the shoe. In some embodiments, the proxy object202may comprise one or more haptic output devices118. The proxy object202may be one or more of a shoe, an article of clothing, an electronic device, a display, an interface object, or an interior of a vehicle.

At step510processor receives an input signal from an input capture device. In some embodiments, a user may use an input capture device, e.g., an audio capture device, a touchscreen, a keyboard, a mouse, a joystick, or an augmented reality control board, to instruct the system on what texture needs to be displayed. For example, a user may say, “apply red suede material to shoe” into a microphone. The processor102receives an input signal from the microphone representing this instruction. The processor102may then determine any necessary modifications to the visual or haptic effect based on this signal. So, the display of the wearable AR device204shows the shoe change from a blue leather material to a red suede material and the haptic effect output by the shoe changes from a blue leather texture to a red suede texture.

Some embodiments described herein provide a more immersive, better optimized experience by combining an AR space with real proxy objects featuring real-world haptic effects. In other embodiments, proxy objects are combined with wearable peripheral devices to enable more compelling haptics because not all haptic feedback needs to be rendered by haptic actuators. Due to the ease of changing the texture and appearance of proxy object, product design becomes more efficient and cost effective.

General Considerations

Embodiments in accordance with aspects of the present subject matter can be implemented in digital electronic circuitry, in computer hardware, firmware, software, or in combinations of the preceding. In one embodiment, a computer may comprise a processor or processors. The processor comprises or has access to a computer-readable medium, such as a random access memory (RAM) coupled to the processor. The processor executes computer-executable program instructions stored in memory, such as executing one or more computer programs including a sensor sampling routine, selection routines, and other routines to perform the methods described above.

Such processors may comprise a microprocessor, a digital signal processor (DSP), an application-specific integrated circuit (ASIC), field programmable gate arrays (FPGAs), and state machines. Such processors may further comprise programmable electronic devices such as PLCs, programmable interrupt controllers (PICs), programmable logic devices (PLDs), programmable read-only memories (PROMs), electronically programmable read-only memories (EPROMs or EEPROMs), or other similar devices.

Such processors may comprise, or may be in communication with, media, for example tangible computer-readable media, that may store instructions that, when executed by the processor, can cause the processor to perform the steps described herein as carried out, or assisted, by a processor. Embodiments of computer-readable media may comprise, but are not limited to, all electronic, optical, magnetic, or other storage devices capable of providing a processor, such as the processor in a web server, with computer-readable instructions. Other examples of media comprise, but are not limited to, a floppy disk, CD-ROM, magnetic disk, memory chip, ROM, RAM, ASIC, configured processor, all optical media, all magnetic tape or other magnetic media, or any other medium from which a computer processor can read. Also, various other devices may include computer-readable media, such as a router, private or public network, or other transmission device. The processor, and the processing, described may be in one or more structures, and may be dispersed through one or more structures. The processor may comprise code for carrying out one or more of the methods (or parts of methods) described herein.