Source: https://patents.justia.com/patent/10416774
Timestamp: 2019-10-16 00:12:24
Document Index: 158962147

Matched Legal Cases: ['art\n8000825', 'arty\n20030181813', 'Application No. 2014', 'Application No. 14', 'Application No. 14', 'Application No. 201410447159', 'Application No. 2014', 'Application No. 201410447159', 'Application No. 201410447159']

US Patent for Automatic remote sensing and haptic conversion system Patent (Patent # 10,416,774 issued September 17, 2019) - Justia Patents Search
Justia Patents US Patent for Automatic remote sensing and haptic conversion system Patent (Patent # 10,416,774)
Jan 5, 2018 - IMMERSION CORPORATION
A system is provided that automatically converts an input into one or more haptic effects in near real-time. The system senses the input in near real-time. The system automatically converts the sensed input into one or more haptic signals in near real-time. The system generates the one or more haptic effects based on the one or more haptic signals.
The present application is a continuation of U.S. patent application Ser. No. 15/226,299, filed Aug. 2, 2016 and issued as U.S. Pat. No. 9,910,495, which is a continuation of U.S. patent application Ser. No. 14/019,606, filed Sep. 6, 2013 and issued as U.S. Pat. No. 9,443,401, the entire contents of which are incorporated by reference herein in their entirety.
Devices can be configured to coordinate the output of haptic effects with the output of other content, such as audio, so that the haptic effects are incorporated into the other content. For example, in a gaming context, when a game is developed, an audio effect developer can develop audio effects that are associated with the game and represent an action occurring within the game, such as machine gun fire, explosions, or car crashes. In another context, audio input, or other types of real-world input, can be captured by a microphone, or other type of sensor. A haptic effect developer can subsequently author a haptic effect for the device, and the device can be configured to output the haptic effect along with the other content. However, such a process is generally not instantaneous, as the process requires a time period for the haptic effect developer to analyze the audio input, or other input, and author an appropriate haptic effect that can be coordinated with the audio input, or other input.
One embodiment is a system that can automatically convert an input into one or more haptic effects in near real-time. The system can sense the input in near real-time. The system can further automatically convert the sensed input into one or more haptic signals in near real-time. The system can further generate one or more haptic effects based on the one or more haptic signals.
FIG. 2 illustrates a block diagram of a system that automatically converts an input into one or more haptic effects in near real-time, according to an embodiment of the invention.
FIG. 3 illustrates a flow diagram of the functionality of an automatic haptic conversion module, according to one embodiment of the invention.
One embodiment is a system that can include a haptic device that delivers haptic feedback converted from an input, such as an audio signal, that can be sensed at a remote location by a sensor, such as a microphone or camera, or an array of microphones or cameras, and transmitted wirelessly to the haptic device. The haptic feedback can be output within the haptic device by a haptic output device or a set of haptic output devices, such as an actuator or a set of actuators, which can be programmed to deliver haptic effects that are generated based on the input and transmitted in near real-time. More specifically, the system can convert the sensed input into one or more haptic signals in near real-time, and the system can further send the one or more haptic signals to the haptic output device(s) to generate the one or more haptic effects. In one embodiment, the sensor can be configured to filter out and deliver only specific frequency bands, features, or other characteristics of the input to the haptic device. Further, according to the embodiment, the haptic output device(s) within the haptic device can be tuned to extract different frequency bands, features, or other characteristics of the input, controlling and/or enhancing the haptic feedback in a way to modulate the attention of the user. Thus, the user can experience haptic feedback that accentuates certain frequency bands, features, or other characteristics of the input that is sensed at the remote location. In one embodiment, the haptic device can be a wearable haptic device.
As understood by one of ordinary skill in the art, “near real-time,” in telecommunications and computing, refers to a time delay introduced by automated data processing or network transmission. Thus, in an embodiment where the input is sensed by the system in near real-time, the input is sensed at the time the input is generated by a remote source, in addition to any time delay associated with sensing the input. Further, in an embodiment where the system generates one or more haptic effects based on the sensed input in near real-time, the system can convert the sensed input into one or more haptic signals used to generate the one or more haptic effects at the time the sensed input is received by the system, in addition to any time delay associated with converting the sensed input into the one or more haptic signals. In certain embodiments, there can be no time delay, and thus, in these embodiments, “near real-time” can be identical to “real-time.”
Such a system can have several applications, according to certain embodiments of the invention. For example, the system can be utilized in a military scenario, where the system can allow for stealth remote monitoring of specific input, such as specific auditory signals, video signals, or other type of sensed input, in near real-time, where the system can deliver the monitored input regarding specific events at a remote location using a haptic output, and where the haptic output can be delivered in a completely silent manner. The haptic output may also deliver enough descriptive information to service in an “alert” capacity. More specifically, the haptic output can inform an individual of specific kinds of activity that are taking place, such as human speech events, vehicle activity or gunfire.
As another example, the system can be utilized in a scientific scenario, where the system can allow for scientists to do remote monitoring of biological or geological events having a distinctive input, such as a distinct audio signal, video signal, or other type of sensed input. For example, the system can allow an ornithologist to use a remote microphone to monitor for a distinct bird call that can be transmitted, converted into a haptic output, and played on a haptic device, such as a wearable haptic device. Thus, the ornithologist can be informed that a certain species of bird has returned to a nesting location, or some other instance of a certain behavior at a certain time.
As yet another example, the system can be utilized in an “immersive” or entertainment scenario, where the system can allow for real-time information, such as auditory information, to be ported to a viewer and converted to one or more haptic effects. For example, in a motocross race, a viewer can feel what a racer is experiencing in the event in near real-time.
FIG. 1 illustrates a block diagram of a system 10 in accordance with one embodiment of the invention. In one embodiment, system 10 is part of a mobile device, and system 10 provides an automatic haptic conversion functionality for the mobile device. In another embodiment, system 10 is part of a wearable device, and system 10 provides an automatic haptic conversion functionality for the wearable device. Examples of wearable devices include wrist bands, headbands, eyeglasses, rings, leg bands, arrays integrated into clothing, or any other type of device that a user may wear on a body or can be held by a user. Some wearable devices can be “haptically enabled,” meaning they include mechanisms to generate haptic effects. In another embodiment, system 10 is separate from the device (e.g., a mobile device or a wearable device), and remotely provides the automatic haptic conversion functionality for the device. Although shown as a single system, the functionality of system 10 can be implemented as a distributed system. System 10 includes a bus 12 or other communication mechanism for communicating information, and a processor 22 coupled to bus 12 for processing information. Processor 22 may be any type of general or specific purpose processor. System 10 further includes a memory 14 for storing information and instructions to be executed by processor 22. Memory 14 can be comprised of any combination of random access memory (“RAM”), read only memory (“ROM”), static storage such as a magnetic or optical disk, or any other type of computer-readable medium.
In one embodiment, memory 14 stores software modules that provide functionality when executed by processor 22. The modules include an operating system 15 that provides operating system functionality for system 10, as well as the rest of a mobile device in one embodiment. The modules further include an automatic haptic conversion module 16 that automatically converts an input into one or more haptic effects, as disclosed in more detail below. In certain embodiments, automatic haptic conversion module 16 can comprise a plurality of modules, where each module provides specific individual functionality for automatically converting an input into one or more haptic effects. System 10 will typically include one or more additional application modules 18 to include additional functionality, such as Integrator™ software by Immersion Corporation.
System 10, in one embodiment, further includes an actuator 26. Processor 22 may transmit a haptic signal associated with a generated haptic effect to actuator 26, which in turn outputs haptic effects such as vibrotactile haptic effects, electrostatic friction haptic effects, or deformation haptic effects. Actuator 26 includes an actuator drive circuit. Actuator 26 may be, for example, an electric motor, an electro-magnetic actuator, a voice coil, a shape memory alloy, an electro-active polymer, a solenoid, an eccentric rotating mass motor (“ERM”), a linear resonant actuator (“LRA”), a piezoelectric actuator, a high bandwidth actuator, an electroactive polymer (“EAP”) actuator, an electrostatic friction display, or an ultrasonic vibration generator. In alternate embodiments, system 10 can include one or more additional actuators, in addition to actuator 26 (not illustrated in FIG. 1). Actuator 26 is an example of a haptic output device, where a haptic output device is a device configured to output haptic effects, such as vibrotactile haptic effects, electrostatic friction haptic effects, or deformation haptic effects, in response to a drive signal. In alternate embodiments, actuator 26 can be replaced by some other type of haptic output device. Further, in other alternate embodiments, system 10 may not include actuator 26, and a separate device from system 10 includes an actuator, or other haptic output device, that generates the haptic effects, and system 10 sends generated haptic signals to that device through communication device 20.
FIG. 2 illustrates a block diagram of a system 200 that automatically converts an input into one or more haptic effects in near real-time, according to an embodiment of the invention. In accordance with the embodiment, system 200 includes remote sensing device 210 and haptic device 220. Remote sensing device 210 can be operably coupled to haptic device 220 via connection 230. In certain embodiments, connection 230 represents a wireless network connection. In other alternate embodiments, connection 230 represents a wired network connection. Further, in alternate embodiments, system 200 can include one or more additional remote sensing devices in addition to remote sensing device 210 and/or one or more additional haptic devices in addition to haptic device 220 (not illustrated in FIG. 2). In some embodiments, connection 230 can be omitted, and sensing device 210 and haptic device 220 can be housed within a single device of system 200, where sensing device 210 and haptic device 220 can be operably coupled by a bus, or other communication mechanism for communicating information. Further, in some embodiments, haptic device 220 can be a wearable haptic device.
According to the embodiment, remote sensing device 210 includes sensor 211. Sensor 211 is configured to sense input in near real-time. More specifically, sensor 211 is configured to detect input that is generated in proximity of remote sensing device 210 and to convert the detected input into a signal in near real-time. In an alternate embodiment, the sensing of the input is not required to be in real-time or near real-time. Instead, in this alternate embodiment, the input can be buffered in one or more buffers (not illustrated in FIG. 2). In certain embodiments, the input can be an audio signal, or other type of audio input, that includes audio data. In other alternate embodiments, the input can be a video signal, or other type of video input, that includes video data. In yet other alternate embodiments, the input can be an acceleration signal, or other type of acceleration input, that includes acceleration data. In yet other alternate embodiments, the input can be an orientation signal that includes orientation data, an ambient light signal that includes ambient light data, or another type of signal that can be sensed by a sensor. In certain embodiments, sensor 211 can be identical to sensor 30 of FIG. 1.
In embodiments where the input is an audio signal, sensor 211 can be a microphone, or some other type of device configured to sense an audio signal. In embodiments where the input is a video signal, sensor 211 can be a camera or some other type of device configured to sense a video signal. In embodiments where the input is an acceleration signal, sensor 211 can be an accelerometer or some other type of device configured to sense an acceleration signal. In embodiments where the input is a movement signal, sensor 211 can be a pedometer or some other type of device configured to sense a movement signal. In embodiments where the input is a temperature signal, sensor 211 can be a temperature sensor or some other type of device configured to sense a temperature signal. Further, in certain embodiments, remote sensing device 210 can include one or more additional sensors, in addition to sensor 211 (not illustrated in FIG. 2). In some of these embodiments, sensor 211 and the one or more additional sensors may be part of a sensor array (such as a microphone array).
In accordance with the embodiment, remote sensing device 210 optionally includes filter 212. Filter 212 is configured to filter the sensed input. More specifically, filter 212 is configured to filter out certain frequency bands, features, or other characteristics of the sensed input. For example, where the sensed input is an audio signal, or other type of audio input, filter 212 can filter out characteristics of the sensed input that represent background sound, so that the only remaining characteristics of the sensed input represent speech or some other type of sound. As an alternate example, in a scenario where speech is constantly detected, filter 212 can filter out characteristics of the sensed input that represent speech, so that the only remaining characteristics of the sensed input represent background sound. As yet another example, where the sensed input is another type of input, filter 212 can filter out characteristics of the sensed input so as to remove events of small magnitude, and so that the only remaining characteristics of the sensed input represent events of a significant magnitude, such as environment changes, frequency transitions, etc. In certain embodiments, filter 212 can be a high-pass filter. Further, in alternate embodiments, filter 212 can be omitted from remote sensing device 210, and the corresponding filtering can be omitted.
According to the embodiment, remote sensing device 210 further includes transmitter 213. Transmitter 213 is configured to transmit the sensed input to haptic device 220 using connection 230. In embodiments where connection 230 is a wireless connection, transmitter 213 can be a wireless transmitter configured to wirelessly transmit the sensed input to haptic device 220 using the wireless connection. In alternate embodiments where connection 230 is a wired connection, transmitter 213 can be a wired transmitter configured to transmit the sensed input to haptic device 220 using the wired connection. In alternate embodiments, transmitter 213 can be omitted from remote sensing device 210, and the corresponding transmitting can be omitted.
According to the embodiment, haptic device 220 includes receiver 221. Receiver 221 is configured to receive the sensed input that is transmitted from remote sensing device 210 using connection 230. In embodiments where connection 230 is a wireless connection, receiver 221 can be a wireless receiver configured to wirelessly receive the sensed input from remote sensing device 210 using the wireless connection. In alternate embodiments where connection 230 is a wired connection, receiver 221 can be a wired receiver configured to receive the sensed input from remote sensing device 210 using the wired connection. In alternate embodiments, receiver 221 can be omitted from haptic device 210, and the corresponding receiving can be omitted.
In accordance with the embodiment, haptic device 220 further includes controller 222. Controller 222 is configured to automatically convert the sensed input into one or more haptic signals in near-real time. More specifically, controller 222 is configured to receive the sensed input from receiver 221 and to automatically convert the received sensed input into one or more haptic signals in near real-time. In certain embodiments, this means that the sensed input is directly converted into the one or more haptic signals without storing the sensed input in a memory, or some other type of storage. In an alternate embodiment, the conversion of the received sensed input into the one or more haptic signals is not required to be in real-time or near real-time, and can involve storing the sensed input in a memory, or some other type of storage.
Controller 222 can use any haptic conversion algorithm that is known to one or ordinary skill in the relevant art to convert the sensed input into the one or more haptic signals. Example haptic conversion algorithms are described in the following patents or patent applications (all of which are hereby incorporated by reference in their entirety): U.S. Pat. Nos. 7,979,146; 8,000,825; 8,378,964; U.S. Pat. App. Pub. No. 2011/0202155; U.S. Pat. App. Pub. No. 2011/0215913; U.S. Pat. App. Pub. No. 2012/0206246; U.S. Pat. App. Pub. No. 2012/0206247; U.S. patent application Ser. Nos. 13/439,241; 13/661,140; 13/743,540; 13/767,129; 13/785,166; 13/788,487; 13/803,778; and 13/799,059.
In some embodiments, controller 222 can further identify and/or extract one or more frequency bands, features, or characteristics of the sensed input. Examples of the characteristics can include: amplitude, frequency, duration, envelope, density, magnitude, and strength. A characteristic can include a numeric value, where the numeric value can define a characteristic of the sensed input. Controller 222 can further modify the one or more haptic signals based on the one or more identified characteristics. More specifically, controller 222 can modify one or more haptic parameters of the one or more haptic signals to match the one or more identified characteristics. According to the embodiment, a haptic signal can include one or more haptic parameters, where a haptic parameter is a parameter that can define the haptic signal used to generate a haptic effect, and thus, can also define the haptic effect to be generated. More specifically, a haptic parameter is a quantity of a haptic effect quality, such as magnitude, frequency, duration, amplitude, strength, envelope, density, or any other kind of quantifiable haptic parameter. According to the embodiment, a haptic effect can be defined, at least in part, by the one or more haptic parameters of the haptic signal, where the one or more haptic parameters can define characteristics of the haptic effect. A haptic parameter can include a numeric value, where the numeric value can define a characteristic of the haptic signal, and thus, can also define a characteristic of the haptic effect generated by the haptic signal. Examples of haptic parameters can include: an amplitude haptic parameter, a frequency haptic parameter, a duration haptic parameter, an envelope haptic parameter, a density haptic parameter, a magnitude haptic parameter, and a strength haptic parameter.
As an example, controller 222 can identify and/or extract one or more specific frequency bands within an audio signal, or other type of audio input. Controller 222 can then modify one or more haptic parameters of a generated haptic signal, so that the haptic signal includes one or more specific frequency bands that are identical or similar to the identified frequency bands within the audio signal, or other type of audio input. As is described below in greater detail, the modified haptic signal can then be used to generate one or more distinctive haptic effects, where the distinctive haptic effects can identify the audio signal/input.
As another example, controller 222 can identify and/or extract a time and/or frequency signature from an audio input. Controller 222 can then modify one or more haptic parameters of a generated haptic signal, so that the haptic signal includes a time and/or frequency signature that is identical or similar to the identified time and/or frequency signature within the audio input. As is described below in greater detail, the modified haptic signal can then be used to generate one or more distinctive haptic effects, where the distinctive haptic effects can identify the audio input. In one particular example, the audio input may be a type of bird call, and the time and/or frequency signature within the bird call can uniquely identify the type of bird call. Thus, the modified haptic signal may be used to generate a haptic effect that identifies the type of bird call based on the corresponding time and/or frequency signature of the modified haptic signal. In another particular example, the audio input may include human activity and non-human activity (such as human speech and vehicle noise), and the identified characteristics within the audio input may uniquely identify the human activity as distinguished from the non-human activity. Thus, the modified haptic signal may be used to generate a first haptic effect that identifies the human activity, and a second haptic effect that identifies the non-human activity. An individual that experiences both the first haptic effect and second haptic effect can identify that the first haptic effect identifies human activity, and the second haptic effect identifies non-human activity.
Further, in an alternate embodiment, rather than automatically converting the sensed input into one or more haptic signals, controller 222 can select one or more haptic signals that are stored within controller 222 based on the one or more identified characteristics of the sensed input. More specifically, controller 222 can select one or more haptic signals that include one or more haptic parameters that are identical or most similar to the one or more identified characteristics of the sensed input. The one or more haptic signals can be stored within a library, where the library is stored within controller 222.
According to the illustrated embodiment, controller 222 can include processor 233, memory 224 and automatic haptic conversion module 225. Processor 233 may be any type of general or specific purpose processor. In certain embodiments, processor 233 can be identical to processor 22 of FIG. 1. Memory 224 can be configured to store information and instructions to be executed by processor 233. Memory 224 can be comprised of any combination of RAM, ROM, static storage such as a magnetic or optical disk, or any other type of computer-readable medium. In certain embodiments, memory 224 can be identical to memory 14 of FIG. 1. Automatic haptic conversion module 225 can be configured to perform the aforementioned functionality for automatically converting an input into one or more haptic effects. In certain embodiments, automatic haptic conversion module 225 can comprise a plurality of modules, where each module provides a specific individual portion of the aforementioned functionality for automatically converting an input into one or more haptic effects. Further, in certain embodiments, automatic haptic conversion module 225 is identical to automatic haptic conversion module 16 of FIG. 1.
Also according to the embodiment, haptic device 220 further includes haptic output device 226. Controller 222 is configured to send the one or more haptic signals to haptic output device 226. In turn, haptic output device 226 is configured to output one or more haptic effects, such as vibrotactile haptic effects, electrostatic friction haptic effects, or deformation haptic effects, in response to the one or more haptic signals sent by controller 222. In certain embodiments, haptic output device 226 is identical to actuator 26 of FIG. 1.
FIG. 3 illustrates a flow diagram of the functionality of an automatic haptic conversion module (such as automatic haptic conversion module 16 of FIG. 1, and automatic haptic conversion module 225 of FIG. 2), according to one embodiment of the invention. In one embodiment, the functionality of FIG. 3 is implemented by software stored in memory or other computer-readable or tangible media, and executed by a processor. In other embodiments, the functionality may be performed by hardware (e.g., through the use of an application specific integrated circuit (“ASIC”), a programmable gate array (“PGA”), a field programmable gate array (“FPGA”), etc.), or any combination of hardware and software.
The flow begins and proceeds to 310. At 310, input is sensed. In certain embodiments, the input can be sensed in near-real time. Further, in certain embodiments, the input can include audio data. In other embodiments, the input can include video data. In other embodiments, the input can include acceleration data. In other embodiments, the input can include another type of data. The flow then proceeds to 320.
At 320, the sensed input is filtered. In certain embodiments, 320 can be omitted. The flow then proceeds to 330.
At 330, the sensed input is transmitted to a controller. In some embodiments, the sensed input can be wirelessly transmitted to the controller. In certain embodiments, 330 can be omitted. The flow then proceeds to 340.
At 340, one or more characteristics of the sensed input are identified. In certain embodiments, the one or more characteristics of the sensed input can include at least one of: an amplitude, a frequency, a duration, an envelope, a density, a magnitude, or a strength. In certain embodiments, 340 can be omitted. The flow then proceeds to 350.
At 350, the sensed input is automatically converted into one or more haptic signals. The sensed input can be automatically converted into the one or more haptic signals using a haptic conversion algorithm. In certain embodiments, the sensed input can be automatically converted into the one or more haptic signals in near real-time. The flow then proceeds to 360.
At 360, the one or more haptic signals are modified based on the one or more identified characteristics of the sensed input. In certain embodiments, one or more haptic parameters of the one or more haptic signals can be modified to match the one or more identified characteristics of the sensed input. In some of these embodiments, the one or more haptic parameters can include at least one of: an amplitude haptic parameter, a frequency haptic parameter, a duration haptic parameter, an envelope haptic parameter, a density haptic parameter, a magnitude parameter, or a strength haptic parameter. In certain embodiments, 360 can be omitted. The flow then proceeds to 370.
At 370, one or more haptic effects are generated based on the one or more haptic signals. In certain embodiments, the one or more haptic signals can be sent to one or more haptic output devices to generate the one or more haptic effects. In some of these embodiments, at least one of the one or more haptic output devices is an actuator. Further, in certain embodiments, a wearable haptic device can generate the one or more haptic effects based on the one or more haptic signals. The flow then ends.
Thus, according to an embodiment, a system is provided that automatically converts sensed input, such as audio signals, into haptic effects, where the sensing of the input and the automatic conversion is performed in near real-time. Thus, the system can automatically generate haptic content from other input, such as audio signals. Thus, the system can make it significantly easier to create haptic content.
1. A wearable haptic device, comprising:
a haptic output device configured to generate a haptic effect;
a controller communicably coupled to the microphone via a communication bus, and configured to receive a sensed audio signal from the microphone, to identify a numeric value of a parameter of the sensed audio signal, to convert the sensed audio signal into a haptic signal by causing a haptic parameter of the haptic signal to have a numeric value which is the same as the numeric value of the parameter of the sensed audio signal, to communicate the haptic signal to the haptic output device, wherein the haptic output device is configured to generate the haptic effect based on the haptic signal, and to cause the speaker to be silent when the haptic output device generates the haptic effect.
2. The wearable haptic device of claim 1, wherein the wearable haptic device includes a wrist band.
3. The wearable haptic device of claim 1, wherein the wearable haptic device includes at least one of a head band, eyeglasses, a ring, or a leg band.
4. The wearable haptic device of claim 1, wherein the haptic parameter of the haptic signal and the parameter of the audio signal have a same parameter type, the parameter type being at least one of: an amplitude, frequency content, a duration, an envelope, or a time signature.
5. The wearable haptic device of claim 4, wherein the wearable haptic device is configured to generate the haptic effect in near real time as the sensed audio signal is received from the microphone, such that any time delay between the receiving of the sensed audio signal and the generating of the haptic effect includes only a time delay for converting the sensed audio signal to the haptic signal.
6. The wearable haptic device of claim 1, wherein the controller is configured to convert the sensed audio signal to the haptic signal by identifying a frequency content or a time signature of the audio signal, and by causing the haptic signal to have an identical frequency content or time signature as the audio signal.
7. The wearable haptic device of claim 1, wherein the controller includes a processor and a memory.
8. A wearable haptic device, comprising:
a haptic output device configured to generate a haptic effect; and
a controller configured to receive a sensed audio signal generated by a microphone, to identify a frequency content of the sensed audio signal, to convert the sensed audio signal into a haptic signal by causing the haptic signal to have a frequency content that is identical to the frequency content of the sensed audio signal, to communicate the haptic signal to the haptic output device, wherein the haptic output device is configured to generate the haptic effect based on the haptic signal.
9. The wearable haptic device of claim 8, wherein the microphone is remote from the wearable haptic device, and the sensed audio signal is received via a network, wherein the wearable haptic device is configured to generate the haptic effect in near real-time as the sensed audio signal is generated by the microphone, such that any time delay between the microphone generating the sensed audio signal and the wearable haptic device generating the haptic effect includes only a first time delay introduced by network transmission of the sensed audio signal from the microphone to the wearable haptic device, and a second time delay associated with converting the sensed audio signal to the haptic signal.
10. The wearable haptic device of claim 8, wherein the microphone is part of the wearable haptic device.
11. A method of generating a haptic effect, the method comprising:
receiving a sensed audio signal that is generated by a microphone;
identifying a time signature of the sensed audio signal;
converting the sensed audio signal to a haptic signal, by causing the haptic signal to have a time signature that is identical to the time signature of the sensed audio signal;
communicating the haptic signal to a haptic output device, wherein the haptic output device is configured to generate the haptic effect based on the haptic signal.
12. The method of claim 11, further comprising performing frequency filtering on the sensed audio signal to filter out characteristics of the sensed audio signal that represents a background sound, wherein the frequency filtering is performed before converting the sensed audio signal to the haptic signal.
5699284 December 16, 1997 Muramatsu
6157316 December 5, 2000 Okayama
6384814 May 7, 2002 Kobayashi et al.
7307820 December 11, 2007 Henson
7969287 June 28, 2011 Tschirhart
8000825 August 16, 2011 Ullrich
8315652 November 20, 2012 Grant et al.
8325144 December 4, 2012 Tierling et al.
8380561 February 19, 2013 Viegas et al.
8436839 May 7, 2013 Takashima et al.
8743057 June 3, 2014 Rosenberg
8783264 July 22, 2014 Levendowski et al.
8832804 September 9, 2014 Casey
8860563 October 14, 2014 Cruz-Hernandez et al.
8874345 October 28, 2014 Nandedkar et al.
9092059 July 28, 2015 Bhatia
9261960 February 16, 2016 Lacroix et al.
9342882 May 17, 2016 Dale
9368005 June 14, 2016 Cruz-Hernandez et al.
9760278 September 12, 2017 Chan
9910495 March 6, 2018 Weddle
20020016916 February 7, 2002 Natarajan
20020033795 March 21, 2002 Shahoian et al.
20030018835 January 23, 2003 Nonaka
20030068053 April 10, 2003 Chu
20030073518 April 17, 2003 Marty
20030181813 September 25, 2003 Ogawa
20040030383 February 12, 2004 Havey
20040030419 February 12, 2004 Miyasaka
20040127198 July 1, 2004 Roskind
20040130526 July 8, 2004 Rosenberg
20040212507 October 28, 2004 Zweig
20050113167 May 26, 2005 Buchner
20050156892 July 21, 2005 Grant
20050228207 October 13, 2005 Ardizzone
20060017691 January 26, 2006 Cruz-Hernandez et al.
20060066569 March 30, 2006 Eid et al.
20060143342 June 29, 2006 Kim et al.
20060267949 November 30, 2006 Rosenberg
20070005835 January 4, 2007 Grant et al.
20070016425 January 18, 2007 Ward
20070057913 March 15, 2007 Eid et al.
20070182711 August 9, 2007 Grant et al.
20070242040 October 18, 2007 Ullrich et al.
20070279401 December 6, 2007 Ramstein et al.
20070300142 December 27, 2007 King
20080045804 February 21, 2008 Williams
20080062143 March 13, 2008 Shahoian
20080159569 July 3, 2008 Hansen
20080253695 October 16, 2008 Sano
20080276793 November 13, 2008 Yamashita et al.
20080296072 December 4, 2008 Takashima
20090021473 January 22, 2009 Grant et al.
20090064031 March 5, 2009 Bull et al.
20090096632 April 16, 2009 Ullrich
20090106655 April 23, 2009 Grant et al.
20090231271 September 17, 2009 Heubel et al.
20090231276 September 17, 2009 Ullrich et al.
20090312817 December 17, 2009 Hogle et al.
20090313542 December 17, 2009 Cruz-Hernandez et al.
20090325645 December 31, 2009 Bang et al.
20100004033 January 7, 2010 Choe et al.
20100116562 May 13, 2010 Cruz-Hernandez et al.
20100152586 June 17, 2010 Grant et al.
20100172591 July 8, 2010 Ishikawa
20100179423 July 15, 2010 Ramstein et al.
20100179587 July 15, 2010 Grant et al.
20100245232 September 30, 2010 Birnbaum et al.
20110054822 March 3, 2011 Bauschke
20110119224 May 19, 2011 Mangione-Smith
20110191674 August 4, 2011 Rawley
20110202155 August 18, 2011 Ullrich et al.
20110215913 September 8, 2011 Ullrich et al.
20110222782 September 15, 2011 Kashiwagi
20110234517 September 29, 2011 Ando
20110285637 November 24, 2011 Karkkainen
20120075168 March 29, 2012 Osterhout
20120200520 August 9, 2012 Harris
20120206247 August 16, 2012 Bhatia
20120229248 September 13, 2012 Parshionikar
20120245406 September 27, 2012 Aghamohammadi
20120286944 November 15, 2012 Forutanpour et al.
20120327006 December 27, 2012 Israr et al.
20130038792 February 14, 2013 Quigley
20130050062 February 28, 2013 Kim et al.
20130113715 May 9, 2013 Grant et al.
20130147971 June 13, 2013 Flynn, III et al.
20130194219 August 1, 2013 Modarres et al.
20130207918 August 15, 2013 Kono et al.
20130222311 August 29, 2013 Pesonen
20130227411 August 29, 2013 Das et al.
20130268172 October 10, 2013 Nandedkar et al.
20130311881 November 21, 2013 Birnbaum et al.
20140035736 February 6, 2014 Weddle et al.
20140051360 February 20, 2014 Kim
20140062682 March 6, 2014 Birnbaum et al.
20140139450 May 22, 2014 Levesque et al.
20140139451 May 22, 2014 Levesque et al.
20140253302 September 11, 2014 Levesque et al.
20140256438 September 11, 2014 Grant et al.
20140266647 September 18, 2014 Visitacion et al.
20140267069 September 18, 2014 Dionne et al.
20140267076 September 18, 2014 Birnbaum et al.
20140267911 September 18, 2014 Grant et al.
20140292501 October 2, 2014 Lim
20140315642 October 23, 2014 Grant et al.
20140320393 October 30, 2014 Modarres et al.
20140320436 October 30, 2014 Modarres et al.
20140340209 November 20, 2014 Lacroix et al.
20140362014 December 11, 2014 Ullrich et al.
20150009168 January 8, 2015 Levesque et al.
20150042573 February 12, 2015 Grant et al.
20150054768 February 26, 2015 Grant et al.
20150070144 March 12, 2015 Weddle et al.
20150070260 March 12, 2015 Saboune
20150070261 March 12, 2015 Saboune et al.
20150177840 June 25, 2015 Kankaanpaa et al.
2629176 August 2013 EP
H10-506508 June 1986 JP
2006-154213 June 2006 JP
2008-283305 November 2008 JP
2011-188342 September 2011 JP
96/09617 March 1996 WO
2008/146616 December 2008 WO
Any copies of information that are not included with this Information Disclosure Statement can be found in U.S. Appl. No. 14/019,606 and U.S. Appl. No. 15/226,299.
Notice of Reasons for Refusal dated Jan. 23, 2018 in corresponding Japanese Patent Application No. 2014-181041.
Corrected Notice of Allowability dated Feb. 5, 2018 in co-pending U.S. Appl. No. 15/226,299.
Decision to refuse a European Patent Application dated Feb. 6, 2018 in corresponding European Patent Application No. 14 002 753.
Provision of the minutes in accordance with Rule 124(4) EPC dated Feb. 6, 2018 in corresponding European Patent Application No. 14 002 753.
Notification of First Office Action dated Apr. 4, 2018 in corresponding Chinese Patent Application No. 201410447159.8.
Notice of Reasons for Refusal dated Aug. 21, 2018 in corresponding Japanese Patent Application No. 2014-181041.
Second Office Action issued in Chinese Application No. 201410447159.8, dated Dec. 17, 2018.
The Third Office Action issued in Chinese Application No. 201410447159.8, dated Jun. 27, 2019.
Patent number: 10416774
Patent Publication Number: 20180129291
Inventors: Amaya Becvar Weddle (San Luis Obispo, CA), Danny Grant (Laval)
Application Number: 15/862,732
Current U.S. Class: 331/113.0R
International Classification: G06F 1/16 (20060101); G06F 3/00 (20060101); G06F 3/01 (20060101); G08B 6/00 (20060101); H04W 4/80 (20180101); G06F 3/041 (20060101);