Source: https://patents.justia.com/patent/10073526
Timestamp: 2019-06-27 05:03:35
Document Index: 427487226

Matched Legal Cases: ['Application No. 61', 'Application No. 61', 'Application No. 61', 'Application No. 10', 'Application No. 10', 'Application No. 201080011708', 'Application No. 201080011708', 'Application No. 201080011743', 'Application No. 201080011743', 'Application No. 201080011744', 'Application No. 201080011744', 'Application No. 201080011744', 'Application No. 201080011744', 'Application No. 10', 'Application No. 10712201', 'Application No. 10787610', 'Application No. 201510760216']

US Patent for Systems and methods for friction displays and additional haptic effects Patent (Patent # 10,073,526 issued September 11, 2018) - Justia Patents Search
Justia Patents US Patent for Systems and methods for friction displays and additional haptic effects Patent (Patent # 10,073,526)
May 22, 2017 - Immersion Corporation
This patent application is a continuation of and claims priority to U.S. application Ser. No. 12/696,900, entitled “Systems and Methods for Friction Displays and Additional Haptic Effects,” filed on Jan. 29, 2010, which claims priority to U.S. Provisional Patent Application No. 61/159,482, entitled “Locating Features Using a Friction Display,” filed Mar. 12, 2009 and also claims priority to U.S. Provisional Patent Application No. 61/262,041, entitled “System and Method for Increasing Haptic Bandwidth in an Electronic Device” filed Nov. 17, 2009, which is incorporated by reference herein in its entirety, and also claims priority to U.S. Provisional Patent Application No. 61/262,038, entitled “Friction Rotary Device for Haptic Feedback” filed Nov. 17, 2009, which is incorporated by reference herein in its entirety, the entirety of all of which is hereby incorporated herein by reference.
This application is related to U.S. patent application Ser. No. 12/697,010, which was filed the same day as application Ser. No. 12/696,900 and is entitled “Systems and Methods for a Texture Engine,” which is incorporated by reference herein in its entirety.
This application is related to U.S. patent application Ser. No. 12/697,042, filed the same day as application Ser. No. 12/696,900 and is entitled “Systems and Methods for Using Multiple Actuators to Realize Textures,” which is incorporated by reference herein in its entirety.
This patent application is related to U.S. patent application Ser. No. 12/697,037, filed the same day as application Ser. No. 12/696,900 and is entitled “Systems and Methods for Using Textures in Graphical User Interface Widgets,” which is incorporated by reference herein in its entirety.
This patent application is related to U.S. patent application Ser. No. 12/696,893, filed the same day as application Ser. No. 12/696,900 and is entitled “Systems and Methods for Providing Features in a Friction Display,” which is incorporated by reference herein in its entirety.
This patent application is related to U.S. patent application Ser. No. 12/696,908, filed the same day as application Ser. No. 12/696,900 and is entitled “Systems and Methods for Interfaces Featuring Surface-Based Haptic Effects,” which is incorporated by reference herein in its entirety.
Touch-enabled devices have become increasingly popular. For instance, mobile and other devices may be configured with touch-sensitive displays so that a user can provide input by touching portions of the touch-sensitive display. As another example, a touch-enabled surface separate from a display may be used for input, such as a trackpad, mouse, or other device.
Embodiments include systems and methods for providing composite haptic effects comprising a variation in a coefficient of friction of a touch surface and one or more other outputs. By providing such surface-based effects, a device can provide a more compelling user experience than may otherwise have been achieved.
In one embodiment, a method comprises detecting, using at least one sensor, a touch occurring in a touch area when an object contacts a touch surface. The method can further comprise selecting a composite haptic effect to generate in response to the touch, the composite haptic effect including at least one surface-based haptic effect and at least one other effect. Based on the selected composite haptic effect, a first haptic signal can be sent to a first actuator to cause the first actuator to vary a coefficient of friction of the touch surface. The method can further comprise, based on the selected composite haptic effect, sending a second haptic signal to cause an actuator to provide a second haptic output in addition to the variation in the coefficient of friction. The second haptic signal can be sent to a second actuator or the same actuator(s) used to vary the coefficient of friction can generate the second haptic output. Another embodiment comprises a tangible computer storage medium embodying program code executable by a computing system for carrying out such a method.
These illustrative embodiments are mentioned not to limit or define the limits of the present subject matter, but to provide examples to aid understanding thereof. Additional embodiments include systems and devices configured to provide composite haptic effects and computer storage media embodying program code for providing composite haptic effects. Illustrative embodiments are discussed in the Detailed Description, and further description is provided there. Advantages offered by various embodiments may be further understood by examining this specification and/or by practicing one or more embodiments of the claimed subject matter.
FIG. 1A is a block diagram showing an illustrative system configured to provide composite haptic effects.
FIGS. 1B and 1C show additional illustrative systems configured to provide composite haptic effects.
FIG. 2A shows another illustrative system configured to provide composite haptic effects.
FIG. 2B is a cross-sectional view of the system shown in FIG. 2A.
FIG. 3A shows another illustrative system configured to provide composite haptic effects, while FIG. 3B is a cross sectional view of the system shown in FIG. 3A.
FIG. 4 is a flowchart showing illustrative steps in a method of providing composite haptic effects.
FIGS. 5A-5D show illustrative features that can be simulated using surface-based haptic effects.
FIGS. 6A-6H show illustrative features, specifically textures, that can be simulated using surface-based haptic effects.
FIG. 7 is a flowchart showing illustrative steps in another method of providing composite haptic effects.
FIG. 8 a diagram illustrating an example of a reference file for use in generating composite haptic effects.
As the user interacts with the device, one or more actuators are used to provide tactile effects, including (but not limited to) composite haptic effects. A composite haptic effect can comprise an effect that utilizes a variation in a coefficient of friction of a touch surface along with one or more other haptic outputs presented in combination with the variation in coefficient of friction. The other output(s) may be presented at the same time as the variation in the coefficient of friction, shortly before, and/or shortly after. The composite effect may be perceived as a single effect or may be perceived as a plurality of related effects.
For example, as a user moves a finger across the device, the coefficient of friction of the screen can be varied based on the position, velocity, and/or acceleration of the finger. Depending on how the friction is varied, the user may perceive a feature and/or a texture. As a particular example, the friction may be varied so that the user perceives a bump, border, or other obstacle corresponding to an edge of an on-screen button. The composite haptic effect may further include tactile feedback as the on-screen button is pressed. As another example, a vibration-based effect may be generated so that a user perceives a feature such as a bump, border, or obstacle defining a control area, with a variation in the coefficient of friction used to indicate when the control is selected/changed. Still further, a combination of variation in friction and vibrotactile effects may be used in combination to create the perception of a single texture or other feature.
The variation in the coefficient of friction and other effect(s) may be generated by the same actuator or by different actuators working in concert. For instance, the coefficient of friction can be varied using a piezoelectric actuator, while vibrotactile effects are generated using a linear resonant actuator. As another example of examples of a haptic output that can be used alongside variations in the coefficient of friction, a portion of a touch surface can be raised/lowered (e.g., using an actuator or shape memory alloy).
FIG. 1A shows an illustrative system 100 for providing a composite haptic effect. Particularly, in this example, system 100 comprises a computing device 101 featuring a processor 102 interfaced with other hardware via bus 106. A memory 104, which can comprise any suitable tangible (and non-transitory) computer-readable medium such as RAM, ROM, EEPROM, or the like, embodies program components that configure operation of the computing device. In this example, computing device 101 further includes one or more network interface devices 110, input/output (I/O) interface components 112, and additional storage 114.
System 100 further includes a touch surface 116, which is in this example integrated into device 101. Touch surface 116 represents any surface that is configured to sense tactile input of a user. One or more sensors 108 are configured to detect a touch in a touch area when an object contacts a touch surface and provide appropriate data for use by processor 102. Any suitable number, type, or arrangement of sensors can be used. For example, resistive and/or capacitive sensors may be embedded in touch surface 116 and used to determine the location of a touch and to determine other information, such as touch pressure. As another example, optical sensors may be used.
In this example, a first actuator 118 in communication with processor 102 is coupled to touch surface 116. In some embodiments, first actuator 118 is configured to output a haptic output varying a coefficient of friction of the touch surface in response to a first haptic signal. Additionally or alternatively, first actuator 118 may provide haptic output by moving the touch surface in a controlled manner. Some haptic outputs may utilize an actuator coupled to a housing of the device, and some haptic outputs may use multiple actuators in sequence and/or in concert. For example, the coefficient of friction can be varied by vibrating the surface at different frequencies and/or amplitudes. Different combinations/sequences of variance can be used to simulate the feeling of a texture, presence of another feature such as an obstacle or protrusion, or to provide another effect.
Although a single first actuator 118 is shown here, embodiments may use multiple actuators of the same or different type to vary the coefficient of friction of the touch surface. For example, a piezoelectric actuator is used in some embodiments to displace some or all of touch surface 116 vertically and/or horizontally at ultrasonic frequencies.
FIG. 1A also shows a second actuator 119. Second actuator 119 can be used to provide a second haptic output in addition to the variation in the coefficient of friction as provided by the first actuator. For example, the second haptic output may comprise a vibrotactile haptic effect such as a vibration of the device and/or touch surface 116. As another example, the second haptic output may comprise a change in the touch surface 116, such as a change in temperature, color, or texture generated by raising/lowering portions of touch surface 116.
Actuators 118 and 119 may each comprise multiple actuators, and may be of the same or different types. Suitable actuator types include, but are not limited to, piezoelectric actuators, shape memory alloys, electroactive polymers, flexible composite piezoelectric actuators (e.g., an actuator comprising a flexible material), electromagnetic actuators, eccentric rotational mass actuators, linear resonant actuators, voice coil actuators, electrostatic actuators, and/or magnetostrictive actuators. Although shown as separate elements 118 and 119, a single actuator capable of varying the coefficient of friction of touch surface 116 and generating another haptic effect could be used instead of separate first and second actuators. Additionally, in practice one or more of the actuators could be embedded within touch surface 116.
Turning to memory 104, exemplary program components 124, 126, and 128 are depicted to illustrate how a device can be configured in some embodiments to provide composite haptic effects. In this example, a detection module 124 configures processor 102 to monitor touch surface 116 via sensor(s) 108 to determine a position of a touch. For example, module 124 may sample sensor 108 in order to track the presence or absence of a touch and, if a touch is present, to track the location, path, velocity, acceleration, pressure and/or other characteristics of the touch over time.
Haptic effect determination module 126 represents a program component that analyzes data regarding touch characteristics to select a composite haptic effect to generate. For example, in some embodiments, an input gesture comprising a sequence of one or more touches may be recognized and correlated to one or more composite haptic effects. As another example, some or all of the area of touch surface 116 may be mapped to a graphical user interface. Different composite haptic effects may be selected based on the location of a touch in order to simulate the presence of a feature displayed on a screen by varying the friction of touch surface and providing one or more other haptic outputs so that the feature is “felt” when a portion of touch surface 116 mapped to a screen location containing the feature is encountered. However, haptic effects may be provided via touch surface 116 even if a corresponding element is not displayed in the interface (e.g., a haptic effect may be provided if a boundary in the interface is crossed, even if the boundary is not displayed).
Haptic effect generation module 128 represents programming that causes processor 102 to generate and transmit a haptic signal to actuator(s) 118/119 to generate the selected composite haptic effect at least when a touch is occurring. For example, generation module 128 may access stored waveforms or commands to send to actuator 118 and 119. As another example, haptic effect generation module 128 may receive a desired coefficient of friction and utilize signal processing algorithms to generate an appropriate signal to send to actuator(s) 118. Module 128 may receive a desired haptic effect type and select an actuator 119 and command the selected actuator to provide the desired haptic effect.
As a further example, a desired texture may be indicated along with target coordinates for the texture. An appropriate waveform can be sent to a piezoelectric or other actuator to vary the coefficient of friction using high-frequency displacement, with another waveform sent to one or more vibrotactile actuators to generate appropriate displacement of the touch surface (and/or other device components) at a (relatively) lower frequency, or to provide a pop, ping, or other effect when the feature is encountered.
In this example, a composite haptic effect is selected based on the content of a graphical user interface 130, which in this example shows a 12-key keypad such as may be provided by a mobile device. Particularly each key may include a border 132 and interior 134. A composite haptic effect can be selected to generate a variation in friction corresponding to interior 134 and a different haptic output can be used to indicate when a user has reached border 132. In this example, a feature 136 is included in the middle key (corresponding to the “5” key in a standard numeric keypad). For instance, feature 136 may comprise a different texture from a texture of interior 134 or a simulated feature such as a bump or gap serving as a “centering” feature. As a user moves a finger across the keypad, composite effects can be used to denote the border between keys and to indicate when the “5” key is reached by the perception of feature 136.
Returning to FIG. 1C, in this example a graphical user interface 138 is shown, comprising a menu bar 140 and buttons 142, 144, and 146. For example, the graphical user interface may comprise an on-screen menu for a home entertainment device, while touch surface 116 is included on a remote control or peripheral. As shown at 148, an area of the touch surface is mapped to menu bar 140. When a user contacts area 148, a composite haptic effect is provided to indicate that the menu bar is selected. For example, the coefficient of friction may differ as between area 148 and areas 150, 152, and 154 (corresponding to buttons 142, 144, and 146, respectively), with vibrotactile or other feedback provided as a user moves through area 148 and crosses into one of areas 150, 152, and 154. As another example, vibrotactile or other haptic output can be used to generate different simulated textures in areas 148, 150, 152, and 154, with the coefficient of friction varied within each area as buttons 142, 144, and/or 146 are actuated.
FIG. 2A shows another computing device 201 featuring a touch-enabled display 202, which again corresponds to a touch surface. FIG. 2B shows a cross-sectional view of device 201. As can be seen in FIG. 2B, in this example two actuators 218-1 and 282-2 are used to vary the coefficient of friction of touch surface 202 while a second actuator 219 provides another haptic output via touch surface 202. For instance, actuator 219 may provide a vibrotactile output such as a pop or ping through touch surface 202 or may raise/lower one or more portions of the touch surface 202. Additionally, FIG. 2B shows another exemplary actuator 222. For example, actuator 222 may be used in addition to or instead of one or more of actuators 218-1, 218-2, or 219. For example, if actuator 219 raises/lowers portions of display 202, actuator 222 may be used to provide a low-frequency vibration output.
Returning to FIG. 2A, an example of providing a composite effect is illustrated at 220. Specifically, a finger 226 contacts the display and moves to the position shown at 228, encountering a simulated feature 230. As can be seen in FIG. 2B, as the finger moves, it initially contacts a first area 232 of the display, then a second area 234, and a third area 236. A composite haptic effect may be used to simulate feature 230. For instance, while the finger encounters area 232 and moves toward area 234, the coefficient of friction of display 202 may be increased by adjusting output of actuators 218-1/218-2. As area 234 is approached and then encountered, actuator 219 may be used to provide a vibrotactile “pop” effect. As the finger moves across area 236, the coefficient of friction of display 202 may be decreased.
A composite haptic effect may comprise a group of effects. For example, lines 238 and 240 indicate a path followed by the finger moving from 226 to 228. In one embodiment, the composite haptic effect includes output generated using actuator 219 and/or 222 if the finger moves outside the path. For example, a variation in friction may be used to simulate feature 230, with a vibration output provided while feature 230 is simulated if the user's finger crosses either of lines 238 and 240.
FIGS. 3A-3B depict an illustrative hardware architecture for a device that can provide composite haptic effects. In this example, the touch surface comprises glass panel 302, although another transparent material (or non-transparent) material could be used. For instance, rather than glass, a touchpad (i.e. touch-sensitive device) could be used. A pair of piezoelectric benders 318-1 and 318-2 are bonded to the glass. Use of glass or another transparent material along with free space between the piezoelectric benders can allow for use of a display (not shown) beneath the glass. In some embodiments, the piezoelectric benders can be commanded to reduce the static coefficient of friction of glass 302 by 42%. Some embodiments utilize a bipolar pulse width modulated signal at 24 kHz, with varying amplitude used to vary the coefficient of friction. As an example, voltage can vary between −80 and +80 Volts at a frequency above 20 kHz, with friction variation from 0 to 60% depending on voltage magnitude (or PWM magnitude to produce the voltage magnitude). These example voltage, frequency, and variation ranges are for purposes of example only and are not intended to be limiting.
In the cross-sectional view of FIG. 3B, a plurality of second actuators 319-1, 319-2, and 319-3 are illustrated. For instance, actuator 319-2 may be bonded to panel 302 and can be used to provide low-frequency vibration effects using an eccentric rotating mass (ERM) actuator and/or haptic pops, jolts, and the like via a linear resonant actuator (LRA). Actuators 319-1 and 319-3 may be bonded to the case or housing of device 301 and may comprise the same type of actuator as 319-2 or may be different from actuator 319-2 (and/or one another). For example, in one embodiment, actuator 319-1 comprises an ERM actuator, actuator 319-2 comprises a shape memory alloy used to move panel 302, and actuator 319-3 comprises an LRA actuator. Either or both actuators 319-1 and 319-3 could be bonded to both panel 302 and the housing of device 301.
FIG. 4 is a flowchart showing an illustrative method 400 for providing an interface with composite haptic effects including at least one surface-based haptic effects. Block 402 represents determining a position of a touch in a touch area. For example, a processor may utilize one or more sensors embedded in or viewing a touch-enabled display or surface to track a position of a touch on the surface. Based on the current and/or past position of the touch, an interaction with a graphical user interface mapped to the touch area can be determined.
Based on the interaction, one or more composite haptic effects can be selected at block 404, with a composite effect achieved by varying the friction of the touch surface along with providing one or more other haptic outputs.
For example, the touch may occur at a position in the touch surface mapped to a particular texture or feature in a graphical user interface. The composite haptic effect may comprise a simulation of the texture or feature. Additionally or alternatively, the composite haptic effect may comprise an ambient effect (e.g., a texture/feature) and dynamic effect, such as output provided when a feature is initially encountered.
As an example, a gesture can be recognized as a sequence of one or more touches or patterns of touch, such as based on a direction and length of a swipe across the screen, a sequence of discrete touches in a pattern, or another recognizable interaction. If a gesture is recognized, a composite haptic effect associated with the gesture can be selected. For instance, a “Z”-shaped touch trajectory may be recognized as a type of input gesture based on pattern recognition carried out by a processor of the device while the gesture is in progress. One or more composite haptic effects may be associated with the “Z”-gesture in data accessible to the processor indicating an effect to output while the gesture is in progress and/or after the gesture is complete. For example, the data may provide for the surface to take on a texture or a change in friction as the gesture nears completion. Additionally or alternatively, a texture or coefficient of friction of the display may change after the gesture is recognized in order to confirm input of the gesture.
At block 406, one or more haptic signals are accessed and/or generated in order to provide a variation in the coefficient of friction and to generate the second haptic output (or outputs). For example, a processor may access drive signals stored in memory and associated with particular haptic effects. As another example, a signal may be generated by accessing a stored algorithm and inputting parameters associated with an effect. For example, an algorithm may output data for use in generating a drive signal based on amplitude and frequency parameters. As another example, a haptic signal may comprise data sent to an actuator to be decoded by the actuator. For instance, the actuator may itself respond to commands specifying parameters such as amplitude and frequency.
As an example, for a simulated feature such as a gap, obstacle, or texture the current pixel location and/or a projected pixel location for the touch based on a velocity of the touch can be compared to a bitmap specifying composite haptic effects for various pixel positions. Based on the composite haptic effect(s), suitable haptic signals can be accessed/generated to provide the output specified in the bitmap.
As another example, a current or projected location of a touch can be compared to data identifying the location of graphical user interface (GUI) features such as controls, textual content, boundaries, and the like. Then, if a GUI feature is identified at the location, data associating one or more composite haptic effects to the feature can be accessed. For instance, a processor may track the location of a touch and determine the touch is at or approaching a position in the touch area mapped to a particular control (e.g., a button) in the graphical user interface. The processor can then consult a listing of interface elements to determine a composite haptic effect (e.g., a texture, a friction variation) associated with the button and, based on the haptic effect, take further actions to generate the composite haptic effect.
Block 408 represents transmitting the haptic signal or signals to one or more actuators to achieve the composite haptic effect by varying the coefficient of friction and providing one or more other haptic outputs. For instance, if an analog drive signal is to be provided, a processor can utilize an onboard D/A converter to create the signal. If a digital command is provided to the actuator, an appropriate message can be generated by an I/O bus of the processor. The haptic effect may be felt at the point of the touch and/or elsewhere. For example, if a two-finger input gesture is provided, the texture/coefficient of friction at the first finger may be changed in response to recognizing movement of the second finger.
FIGS. 5A-5D each depict an illustrative simulated feature. FIG. 5A shows a simplified example in which the white area represents an area where actuators will be activated, such as by using a non-zero voltage PWM signal. For example, the white area may correspond to a virtual button in the middle of a touch pad, where a user's finger (or another object in contact with a surface) will encounter a lower friction value. FIG. 5B represents an inverses situation—the finger/object may navigate freely in the white area, but may be slowed or stopped at the high friction (black) area. This may, for example, allow a user to more easily locate a button or other location in the touch area.
FIG. 5C illustrates a simulated feature comprising a plurality of grooves. As a user's finger or another object moves horizontally across the stripes, the finger/object will encounter increasing and decreasing friction that is perceived as a series of grooves.
As was noted above, a computing system comprising a touch surface configured to provide surface-based haptic effects may determine effects and signals in real time. For example, for any of FIGS. 5A-5D, the system may first determine if the touch position is inside the circle and, if so, provide a suitable output value (FIG. 5A) or cease output (FIG. 5B). Similarly, the system may provide the feature of FIG. 5C by determining if the touch occurs in an area with desired high-friction and, if so, drive the actuator(s).
FIG. 5D presents a more complex pattern. For instance, the pattern in FIG. 5D may correspond to desired features associated with an array of keys, such as an array of mobile phone keys, a simulated keyboard, or other controls. Although real time rendering could be used for any of FIGS. 5A-5D, more complex logic may be needed to render each specific circle/button in the pattern. These and even more arbitrary patters may increase the complexity of programming and computation time. Thus, in some embodiments, the surface-based haptic effects can be determined ahead of time and stored in a file. At runtime, the file can be accessed based on a touch position to allow for faster determination and generation of appropriate haptic signals. For FIG. 5D, such a file could include data to drive the actuators to provide a first haptic effect (e.g., high friction) when the touch position is mapped to the circles, and the file could include data to drive the actuators to provide a second effect (e.g., low friction) when the touch position is mapped to a location outside the circles
The features of FIGS. 5A-5D may be simulated using other types of actuators in addition to or instead of piezoelectric actuators. For example, vibrotactile actuators may be used to provide a vibration, pop, pulse, click, or the like when the circles of FIG. 5A, 5B, or 5D are encountered and/or when the pattern of FIG. 5C is encountered (e.g., a pop or click at each transition from black to white). The haptic effects that can be produced by the actuators can vary depending on the current, voltage, frequency as well as start and stop times. Such haptic effects include, but are not limited to, vibrations, pulses, pops, clicks, damping characteristics, and varying textures. In an embodiment, the multiple actuators are utilized to generate different haptic effects for different applications. For example, the two actuators are configured to provide a vibration or pop upon the user's finger or stylus passing over the boundaries of a graphical object (e.g. keyboard keys), as discussed above.
The features of FIGS. 5A-5D may be simulated using composite haptic effects. For example, a variation of friction may be used along with a click, pop, or other response when an object touching the surface moves from black to white or vice-versa.
Vibrotactile effects and/or variations in friction may additionally or alternatively be used to simulate various textures. Additional detail regarding generation and use of textures can be found in U.S. patent application Ser. Nos. 12/697,010, 12/697,042, and 12/697,037, referenced above and entitled “Systems and Methods for a Texture Engine,” “Systems and Methods for Using Multiple Actuators to Realize Textures,” and “Systems and Methods for Using Textures in Graphical User Interface Widgets,” respectively.
For instance, patterns of differing friction or patterns of vibration may be provided to mimic the feeling of textures such as brick, rocks, sand, grass, fur, various fabric types, water, molasses, and other fluids, leather, wood, ice, lizard skin, metals, and other texture patterns. Other textures not analogous to real-world textures may also be used, such as high-magnitude vibrotactile or other feedback when a “danger” texture is desired.
Generally, a texture can simulate a surface property by providing an output perceived as a spatially varying force. Thus, a texture is contrasted to a vibration output intended to be directly perceived as a time varying force, though of course vibrations can be used in simulating texture.
As an example, a texture may simulate the force felt when a stick or finger is moved over a grating. In one embodiment, the texture force can be characterized by a programmer/developer using parameters that can include a magnitude and a grit. The magnitude specifies the amount of force applied to the object encountering the touch surface (e.g., at each “bump” of the grating). The grit is basically the spacing between forces (e.g., spacing between each of the grating bumps). Alternatively, additional command parameters can be provided to control the position of the “bumps” of the texture force. For example, information can be included to instruct the distance between bumps to vary exponentially over a distance, or vary according to a specified formula. Additionally, a given texture may be defined as a plurality of superimposed grits and magnitudes.
FIG. 6A is an illustration of one of the textures that a texture engine may generate according to one embodiment of the present invention. The embodiment shown in FIG. 6A comprises brick. The texture of brick is characterized by having a rough irregular texture from bricks, punctuated with the feel of gritty valleys from the mortar. A system for a texture engine may generate the rough irregular texture of brick by driving an actuator, such as a LRA, LPA, or FPA, with a random signal with medium to high maximum variance while the user's finger is moving. In some embodiments, this variance may be adjusted for different roughness. In some embodiments, the transition from brick to mortar may be rendered by a high duration pop created by an ERM. Additionally, the coefficient of friction may be varied between the brick and mortar portions. If the mortar is thick enough, a fine texture may be rendered by driving an actuator with a lower magnitude signal with a higher variance than that used to drive the actuator outputting the texture of the brick.
FIG. 6B is an illustration of one of the textures that a texture engine may generate according to one embodiment of the present invention. The embodiment shown in FIG. 6B comprises rocks. The texture of rocks is characterized by smooth surfaces punctuated with transitions as the user moves from rock to rock. In order to output the texture of a rock, an actuator, such as an FPA and/or piezoelectric actuator can be used to create patches of low friction. Individual rocks may be rendered by a non-visual edge map of the displayed image, and outputting a high magnitude haptic signal to an actuator, such as an LRA, LPA or ERM, when the touch-sensitive interface detects the user's movement. For example, a high-magnitude output can be provided whenever the touch-sensitive interface detects that the user's finger is transitioning from one rock to another. The coefficient of friction may be varied as between rocks, as well.
FIG. 6C is an illustration of one of the textures that a texture engine may generate according to one embodiment of the present invention. The embodiment shown in FIG. 6C comprises sand or sandpaper. Sand is characterized by a rough, gritty feel as well as the sensation a pile of sand particles building up in front of the user's finger. In order to output the rough gritty texture, of sand, an actuator, such as an LRA, LPA or FPA is driven with a random signal with high maximum variance while the user's finger is moving. In some embodiments, the processor may adjust the variance of the signal to create different degrees of roughness. To create the feeling of sand piling up, an actuator such as an FPA may be used. In such an embodiment, when user moves their finger across the touch screen, the processor will drive the actuator with a signal that starts with a low intensity and builds as the user moves their finger in one direction. A piezoelectric or other actuator can be used to increase the coefficient of friction to simulate sand build-up as well.
In another embodiment, the texture shown in FIG. 6C may comprise sandpaper. Sandpaper is characterized by having a rough, gritty feel. To create the rough, gritty feel the processor drives an actuator, such as an LRA, LPA or FPA with a random signal with high maximum variance. In some embodiments, this signal is output only while the user's finger is moving across the surface the touch sensitive interface. In some embodiments, the processor may adjust the variance of the signal to change the level of roughness. Additionally, the roughness may be generated or emphasized through abrupt changes in the coefficient of friction.
FIG. 6D is an illustration of one of the textures that a texture engine may generate according to one embodiment of the present invention. In the embodiment shown in FIG. 6D, the texture comprises the texture of grass. Grass is characterized by a periodic light sensation that almost tickles the user's finger. In order to create the sensation of grass, the processor may drive an actuator, such as an FPA or piezoelectric actuator, with a signal configured to create patches of low friction overlaid with patches of grass. In some embodiments, the processor may render individual grass blades by having a non-visual edge map of the displayed image and outputting a low magnitude signal to an actuator such as an LPA or ERM when the user interface detects movement between the simulated blades.
FIG. 6E is an illustration of one of the textures that a texture engine may generate according to one embodiment of the present invention. In the embodiment shown in FIG. 6E, the texture comprises the texture of fabric. Fabric is characterized by a light smooth sensation. In order to create the sensation of the texture of fabric, the processor may drive an actuator such as an LPA or an LRA with low magnitude high frequency signals as the user's finger moves across the surface of the touch-sensitive interface. The “grain” of the fabric may be simulated by varying friction levels of the display to provide lower friction when moving along the grain.
FIG. 6F is an illustration of one of the textures that a texture engine may generate according to one embodiment of the present invention. In the embodiment shown in FIG. 6F, the texture comprises the texture of water or molasses. Water is characterized by having almost no sensation. However, water that is disturbed may splash around and hit against he user's finger. To emulate the texture of water, the processor may drive an actuator such as an FPA to reduce the friction on the surface of the touch-sensitive interface. To emulate the water sloshing, the processor may output the haptic signal only when the user is touching the screen. To emulate the texture of a more viscous fluid, such as molasses, or oil, the processor may drive the actuator with a signal configured to increase the friction on the user's finger as it moves across the surface of the touch-sensitive interface.
FIG. 6G is an illustration of one of the textures that a texture engine may generate according to one embodiment of the present invention. In the embodiment shown in FIG. 6G, the texture comprises the texture of leather. Leather is characterized by an overall smooth feeling that comprises the bumps and valleys of the surface of the leather. In order to create the sensations of the texture of leather, the processor may drive an actuator, such as an FPA or piezoelectric actuator, with a signal configured to output a haptic effect that reduces friction as the user's finger moves across the surface of the touch-sensitive interface. The processor can output the cracks and bumps by driving the actuator with a very short low magnitude haptic signal at times when the touch-sensitive interface detects that the user's finger is moving.
FIG. 6H is an illustration of one of the textures that a texture engine may generate according to one embodiment of the present invention. In the embodiment shown in FIG. 6E, the texture comprises the texture of wood. Wood may be characterized by an irregular bumpy texture punctuated by a sharp transition as the user moves from board to board. In order to create the irregular bumpy texture, the processor may drive an actuator such as an LRA, LPA, or FPA with a non-visual edge map of the displayed image and drive the actuator with a very short low magnitude signal at various times when the user's finger is moving. To output the transition from plank to plank, the processor may output a haptic signal configured to cause the actuator to generate a high magnitude, short duration, pop. The friction of the touch surface can be varied when moving between planks, but can be maintained (or only slightly varied) when moving along the grain of the wood.
In other embodiments, haptic effects associated with different textures may be output. For example, in one embodiment, the processor may transmit a haptic signal configured to cause the actuator to output a haptic effect configured to cause the user to feel a texture associated with the texture of ice. Ice is characterized by low friction, in some embodiments, ice has a completely smooth texture, in other embodiments, ice comprises a fine low magnitude gritty texture. In order to create the texture of ice, the processor may determine a haptic signal configured to cause the actuator to reduce the friction as much as possible while the user moves their finger across the surface of the touch-sensitive interface. In another embodiment, the processor may drive an actuator such as an LPA or LRA, with a haptic signal configured to output low magnitude effects while the user moves their finger. These low magnitude effects may be associated with imperfections or grit on the surface of the ice.
In another embodiment, the processor may drive the actuator with a signal configured to cause the actuator to output a haptic effect approximating the texture of lizard skin. Lizard skin is characterized by an overall smooth sensation punctuated by transitions from bump to bump on the skin. In order to implement a haptic effect comprising the texture of lizard skin, the processor may drive an actuator with a haptic signal configured to cause the actuator to create patches of low friction on the touch-sensitive interface. The processor may render cracks on the surface of the skin by outputting high magnitude haptic signals periodically when the touch-sensitive interface detects that the user's finger is moving across its surface. These high magnitude signals may approximate the cracks in the surface of the skin.
In yet another embodiment, the processor may drive the actuator with a signal configured to cause the actuator to output a haptic effect approximating the texture of fur. Fur is characterized by a periodic light sensation that is very soft to the touch. In order to implement a haptic effect comprising the texture of fur, the processor may drive the actuator with a haptic signal configured to cause the actuator to output a haptic effect configured to reduce the friction the user feels on the surface of the touch-sensitive interface. The processor may further render individual hairs outputting a low magnitude pulsing haptic signals as the touch-sensitive interface detects the user's movement.
In yet another embodiment, the processor may drive the actuator with a signal configured to cause the actuator to output a haptic effect approximating the texture of metal. Metal is characterized by a smooth low friction surface that, in some embodiments, includes light grit. In order to implement a haptic effect comprising the texture of metal, the processor may drive the actuator with a signal configured to lower the friction the user feels on the surface of the touch-sensitive interface. In some embodiments, the processor may render individual bumps by outputting brief high magnitude haptic signals when the touch-sensitive interface detects that the user is moving over its surface. These brief high magnitude signals may approximate grit on the surface of the metal.
In yet another embodiment, the processor may drive the actuator with a signal configured to cause the actuator to output a haptic effect approximating another sensation, for example, heat. In such an embodiment, the processor may output a haptic signal configured to cause the actuator to output a high frequency jolting effect when the user touches elements of the display that are associated with heat.
FIG. 7 is a flowchart showing illustrative steps in another method 700 of providing composite haptic effects. FIG. 7 is a flowchart showing an exemplary method 700 for providing a simulated feature by creating and using a reference file. FIG. 8 shows an example of a reference file comprising an array of pixels. Blocks 702 and 704 represent preprocessing-activities that occur prior to use of a reference file to determine a composite haptic effect. In this example, a single reference file is used to determine friction values and at least one other haptic output used in providing a composite haptic effect. Additionally, a “reference file” may comprise multiple files used together.
Block 702 represents creating a layout of a location and block 704 represents storing the layout in an image file, such as an array of pixels in a bitmap or other image file. For example, arbitrary shapes may be “drawn” in order to specify desired friction values. In FIG. 8, white pixels are shown to indicate where no friction adjustment or other output is intended, while shaded pixels indicate a value of a desired coefficient of friction or even a value usable to drive an actuator (e.g., a desired PWM voltage level, frequency, etc.). Alternatively, white pixels may indicate maximum drive, while various degrees of shading indicate lower drive values, with black representing zero drive. In an embodiment, white pixels and black pixels only are used, with the colors corresponding to on/off states of the actuators of a device.
In this example, different degrees of shading are represented by cross-hatching. In practice, each pixel may comprise multiple values (e.g., each pixel may have an RGB value), with the multiple values providing different data, such as drive levels for different actuators and the like. Additionally, a reference file may include multiple layers for specifying various parameters for each pixel position. For example, one layer may be used to define variations in the coefficient of friction, with one or more other layers used to define drive values or other information for use in outputting a vibrotactile or other haptic output. This example shows a relatively small number of pixels; in practice, the array may comprise thousands or millions of pixels.
In this example, three buttons 802, 804, and 806 are shown, with button borders indicated by solid shading. Respective interiors 808, 810, and 812 have different shading and are intended to represent different textures, friction values, or other effects corresponding to the button interiors. A menu bar is shown with different shading at 814, along with two menu commands 816 and 818 with the same shading as one another, corresponding to different haptic effects that can be used to indicate when the menu bar is encountered and then when menu items are encountered. Although not shown here, transition areas between low and high (or high and low) friction or other effects could be included.
Returning to FIG. 7 and method 700, once a reference file is created, it can be loaded into memory and read as shown at block 706 to determine composite haptic effects for output. For example, some or all of the pixel array may be maintained in working memory of a processor carrying out a position detection and feature simulation routine. In an embodiment, the pixel array is distributed alongside a corresponding image of a graphical user interface. In additional embodiments, the pixel array is a layer or component of the graphical user interface image, and in further embodiments the array is a separate file not associated with a graphical user interface.
As a more complex example, each pixel address may be associated with three intensity values (i.e., RGB). Each of the three intensity values can be associated with a signal intensity/frequency for a corresponding actuator in some embodiments. As another example, some values may specify intensity and others specify duration of operation for the same actuator. As a further example, different pixel intensity values may be correlated to different desired textures or components used to drive actuators to simulate a single texture. For example, textures 808, 810, and 812 may be achieved using a combination of friction variations and vibrations. When a touch occurs at a location mapped to the areas, the appropriate actuator(s) can be used to generate the textures.
Method 700 may determine touch locations mapped to multiple pixels in the image file. For example, a large touch may correspond to a range of pixel addresses in the image file. Values from the range of pixel addresses may be considered together, or analysis may be made to “pinpoint” the touch location and use values from a corresponding single pixel address.
In this example, method 700 checks at 714 for one or more events tied to a haptic effect. As was noted above, a composite haptic effect may utilize one or more actuators that provide a haptic output to confirm a selection, indicate a change in status, or otherwise provide feedback to a user. Block 716 represents using one or more actuators to provide haptic output corresponding to the event, and then continuing back to 708 to determine the touch position.
For example, a user may move a finger or other object across a region of a touch surface that is mapped to button 802 and on to a region mapped to button 804. Texture 808 may be generated and then, as the finger/object moves to button 804, a click, pop, or other effect may be output to indicate the button borders. A click or button-press event may be registered if the user lingers over a button for a predetermined time and/or if the touch-enabled system registers a touch or increase in pressure. In response to the click event, a click, pop, or other effect can be output to simulate the response of a physical button (or another response).
a processor in communication with the sensor and configured to: determine a first haptic effect based in part on data received from the sensor by mapping a location of the touch to a mapping file comprising friction values at locations on the touch surface, the mapping file comprising data associated with multiple layers of a user interface, each layer associated with a state of the user interface, the first haptic effect configured to vary a coefficient of friction of the touch surface; and transmit a first haptic signal associated with the first haptic effect; and
a first haptic output device in communication with the processor and configured to receive the first haptic signal and output the first haptic effect.
2. The system of claim 1, further comprising a second haptic output device, and wherein the processor is further configured to determine a second haptic effect and transmit a second haptic signal associated with the second haptic effect to the second haptic output device.
3. The system of claim 2, wherein the first haptic output device comprises a piezoelectric actuator, and wherein the second haptic output device comprises one or more of: a second piezoelectric actuator, a shape memory alloy, an electroactive polymer, a composite piezoelectric actuator, an electromagnetic actuator, an eccentric rotational mass actuator, a linear resonant actuator, or a voice coil actuator.
4. The system of claim 3, wherein the first haptic effect is configured to vary the coefficient of friction on the touch surface to simulate a background texture, and the second haptic effect is configured to simulate a feature overlaying the background texture.
5. The system of claim 3, wherein the mapping file comprises a plurality of values, including a first value used to generate the first haptic signal and a second value used to generate the second haptic signal.
6. The system of claim 1, wherein the processor is further configured to determine a pressure of the touch based on data from the sensor, and wherein the first haptic effect is determined based at least in part on the pressure of the touch.
7. The system of claim 1, further comprising a display, the display comprising a plurality of pixels and defining a display area, the display configured to output an image based at least in part on a display signal, and wherein the touch surface corresponds to the display or a material above the display.
8. The system of claim 7, wherein the mapping file further comprises data associated with the plurality of pixels.
9. The system of claim 1, wherein varying the coefficient of friction of the touch surface comprises increasing the coefficient of friction of the touch surface.
receiving a sensor signal from a sensor configured to detect a touch in a touch area when an object contacts a touch surface;
determining a first haptic effect based in part on the sensor signal by mapping a location of the touch to a mapping file comprising friction values at locations on the touch surface, the mapping file comprising data associated with multiple layers of a user interface, each layer associated with a state of the user interface, the first haptic effect configured to vary a coefficient of friction of the touch surface; and
transmitting a first haptic signal associated with the first haptic effect to a first haptic output device.
11. The method of claim 10, further comprising determining a second haptic effect and transmitting a second haptic signal associated with the second haptic effect to a second haptic output device.
12. The method of claim 11, wherein the first haptic output device comprises a piezoelectric actuator, and wherein the second haptic output device comprises one or more of: a second piezoelectric actuator, a shape memory alloy, an electroactive polymer, a composite piezoelectric actuator, an electromagnetic actuator, an eccentric rotational mass actuator, a linear resonant actuator, or a voice coil actuator.
13. The method of claim 12, wherein the first haptic effect is configured to vary the coefficient of friction on the touch surface to simulate a background texture, and the second haptic effect is configured to simulate a feature overlaying the background texture.
14. The method of claim 12, wherein the mapping file comprises a plurality of values, including a first value used to generate the first haptic signal and a second value used to generate the second haptic signal.
15. The method of claim 10, further comprising determining a pressure of the touch based on data from the sensor, and wherein the first haptic effect is determined based at least in part on the pressure of the touch.
16. The method of claim 10, wherein varying the coefficient of friction of the touch surface comprises increasing the coefficient of friction of the touch surface.
17. A non-transitory computer readable medium embodying program code executable by a processor, the program code, when executed, configured to cause the processor to:
receive a sensor signal from a sensor configured to detect a touch in a touch area when an object contacts a touch surface;
determine a first haptic effect based in part on the sensor signal by mapping a location of the touch to a mapping file comprising friction values at locations on the touch surface, the mapping file comprising data associated with multiple layers of a user interface, each layer associated with a state of the user interface, the first haptic effect configured to vary a coefficient of friction of the touch surface; and
transmit a first haptic signal associated with the first haptic effect to a first haptic output device.
18. The non-transitory computer readable medium of claim 17, further comprising program code, which when executed by processor is configured to cause the processor to: determine a second haptic effect and transmitting a second haptic signal associated with the second haptic effect to a second haptic output device.
19. The non-transitory computer readable medium of claim 18, wherein the first haptic output device comprises a piezoelectric actuator, and wherein the second haptic output device comprises one or more of: a second piezoelectric actuator, a shape memory alloy, an electroactive polymer, a composite piezoelectric actuator, an electromagnetic actuator, an eccentric rotational mass actuator, a linear resonant actuator, or a voice coil actuator.
20. The non-transitory computer readable medium of claim 17, wherein varying the coefficient of friction of the touch surface comprises increasing the coefficient of friction of the touch surface.
5144187 September 1, 1992 Culp
5198732 March 30, 1993 Morimoto
5939816 August 17, 1999 Culp
6046527 April 4, 2000 Roopnarine et al.
6046730 April 4, 2000 Bowen et al.
6219032 April 17, 2001 Rosenberg
6252579 June 26, 2001 Rosenberg
6285351 September 4, 2001 Rosenberg et al.
6433711 August 13, 2002 Chen
6697086 February 24, 2004 Rosenberg et al.
6850222 February 1, 2005 Rosenberg
7027032 April 11, 2006 Rosenberg et al.
7109967 September 19, 2006 Hioki et al.
7815436 October 19, 2010 Cunningham et al.
8004498 August 23, 2011 Meridian
8264465 September 11, 2012 Grant et al.
8494860 July 23, 2013 Asakawa et al.
8677274 March 18, 2014 Runov et al.
20030063128 April 3, 2003 Salminaa et al.
20030184518 October 2, 2003 Numata et al.
20040107407 June 3, 2004 Henson et al.
20040169674 September 2, 2004 Linjama
20040233162 November 25, 2004 Kobayashi
20040251780 December 16, 2004 Goodson et al.
20060046031 March 2, 2006 Janevski
20060061558 March 23, 2006 Grant et al.
20060101347 May 11, 2006 Runov et al.
20060106767 May 18, 2006 Adcock et al.
20060119573 June 8, 2006 Grant et al.
20060209037 September 21, 2006 Wang et al.
20060226298 October 12, 2006 Pierson
20060267416 November 30, 2006 Suzuki
20070021961 January 25, 2007 Oh
20070236474 October 11, 2007 Ramstein
20080048974 February 28, 2008 Braun et al.
20080068648 March 20, 2008 Benz et al.
20080216578 September 11, 2008 Takashima et al.
20090046054 February 19, 2009 Olien
20090063472 March 5, 2009 Pell et al.
20090112572 April 30, 2009 Thorn et al.
20090134744 May 28, 2009 Yoon et al.
20090207129 August 20, 2009 Ullrich et al.
20090227296 September 10, 2009 Kim
20100026976 February 4, 2010 Meehan et al.
20100073304 March 25, 2010 Grant et al.
20100145934 June 10, 2010 Tran et al.
20100188327 July 29, 2010 Frid et al.
20100315212 December 16, 2010 Radivojevic
20120154133 June 21, 2012 Kyung et al.
20120182248 July 19, 2012 Kobayashi et al.
1496549 May 2004 CN
1646833 July 2005 CN
101467118 June 2009 CN
101506758 August 2009 CN
101523329 September 2009 CN
101616213 December 2009 CN
0899 650 March 1999 EP
1748350 January 2007 EP
2001-255993 September 2001 JP
2003-256105 September 2003 JP
2006-509289 June 2004 JP
2004-265281 September 2004 JP
2005-258666 September 2005 JP
2008-515002 April 2006 JP
2006-163206 June 2006 JP
2006-228151 August 2006 JP
2007-133698 May 2007 JP
2008-027223 February 2008 JP
2008-516348 May 2008 JP
2008-209689 September 2008 JP
2008-225690 September 2008 JP
2009-003867 January 2009 JP
WO 2001/054109 July 2001 WO
WO 2002/073587 September 2002 WO
WO 2004/051451 June 2004 WO
WO 2006/042309 April 2006 WO
2007/059172 May 2007 WO
WO 2007/091053 August 2007 WO
WO 2007/120562 October 2007 WO
WO 2007117418 October 2007 WO
WO 2008/037275 April 2008 WO
WO 2008/103535 August 2008 WO
WO 2008/144108 November 2008 WO
WO 2009/002605 December 2008 WO
WO 2009/026337 February 2009 WO
2009/042424 April 2009 WO
Biet, M. et al., “Discrimination of Virtual Square Gratings by Dynamic Touch on Friction Based Tactile Displays,” Haptic Interfaces for Virtual Environment and Teleoperator Systems, 2008, Symposium, IEEE, Piscataway, NJ, XP 031339918, pp. 41-48, Mar. 2008.
De Witt, A., “Designing Sonification of User Data in Affective Interaction,” Master of Science Thesis Stockholm, Sweden, XP 002551466, at hppt://w3.nada.kth.se/utbildning/grukth/exjobb/rapportlistor/2007/de_witt_anna_07142, as available via the Internet and printed Oct. 20, 2009.
Kaaresoja, T. et al., “Snap-Crackle-Pop: Tactile Feedback for Mobile Touch Screens,” Proceedings of the Eurohaptics 2006, XP 002551465, at http://lsc.univ-evry.fr/(eurohaptics/upload/cd/papers/f80, as available via the Internet and printed Oct. 20, 2009.
Maeno, T. et al., “Tactile Display of Surface Texture by use of Amplitude Modulation of Ultrasonic Vibration,” IEEE Ultrasonics Symposium, 2006, pp. 62-65.
Minsky, Margaret Diane Rezvan, “Computational Haptics: The Sandpaper System for Synthesizing Texture for a Force-Feedback Display,” Ph.D. Dissertation, Massachusetts Institute of Technology, Jun. 1995, 217 pages.
Oakley, I. et al., Contact IM: Exploring Asynchronous Touch over Distance, Palpable Machines Research Group, Media Lab Europe, XP 007910188, at http://people.cs.vt.edu/[wangr06/touch%20review%origization/OAK002, as available via the Internet and printed Oct. 20, 2009.
Williamson, J. et al., “Shoogle: Excitatory Multimodal Interaction on Mobile Devices,” CHI 2007 Proceedings-Shake, Rattle and Roll: New Forms of Input and Output, 2007, pp. 121-124, XP002549378.
Sekiguchi, Y. et al., “Haptic Interface using Estimation of Box Contents Metaphor,” Proceedings of ICAT 2003, Tokyo, Japan, XP 002551467, at http://www.vrsj.org/ic-at/papers/2003/00947_00000, as available via the Internet and printed Oct. 20, 2009.
Watanabe, T. et al., “A Method for Controlling Tactile Sensation of Surface Roughness Using Ultrasonic Vibration,” IEEE International Conference on Robotics and Automation, 0-7803-1965-6/95, 1995, pp. 1134-1139.
Search Report and Written Opinion dated Jun. 8, 2010 for corresponding PCT Application No. PCT/US2010/026909.
Search Report and Written Opinion dated Jun. 8, 2010 for corresponding PCT Application No. PCT/US2010/026897.
Search Report and Written Opinion dated Jun. 8, 2010 for corresponding PCT Application No. PCT/US2010/026905.
Search Report and Written Opinion dated Jun. 8, 2010 for corresponding PCT Application No. PCT/US2010/026894.
Search Report and Written Opinion dated Jun. 8, 2010 for corresponding PCT Application No. PCT/US2010/026907.
Search Report and Written Opinion dated Jun. 8, 2010 for corresponding PCT Application No. PCT/US2010/026900.
International Preliminary Report on Patentability dated Sep. 22, 2011 for corresponding International Application No. PCT/US2010/026909.
International Preliminary Report on Patentability dated Sep. 22, 2011 for corresponding International Application No. PCT/US2010/026897.
International Preliminary Report on Patentability dated Sep. 22, 2011 for corresponding International Application No. PCT/US2010/026905.
International Preliminary Report on Patentability dated Sep. 22, 2011 for corresponding International Application No. PCT/US2010/026894.
International Preliminary Report on Patentability dated Sep. 22, 2011 for corresponding International Application No. PCT/US2010/026907.
International Preliminary Report on Patentability dated Sep. 22, 2011 for corresponding International Application No. PCT/US2010/026900.
Corel Paint Shop Pro Photo X2 Reviewer's Guide, Copyright 2007, web page at http:/ /web.archive.org/web/20071102133023/http://www.corel.com/conte nt!vpk!psppx2/PS P PX2Reviewer_Guide.pdf, as available via the Internet and accessed Apr. 7, 2012.
Office Action dated Nov. 2, 2012 for corresponding U.S. Appl. No. 12/947,321.
Chowdhury, M.A. et al., “The Effect of Amplitude of Vibration on the Coefficient of Friction for Different Materials,” Tribology International, 2008, 41:307-314.
Tang, H. et al., “A Microfabricated Electrostatic Haptic Display for Persons with Visual Impairments,” IEEE Transactions on Rehabilitation Engineering, Sep. 1998, 6(3):241-314.
Office Action dated Nov. 15, 2012 for corresponding U.S. Appl. No. 12/947,532.
Office Action dated Jan. 15, 2013 for corresponding U.S. Appl. No. 12/696,908.
Office Action dated Jan. 15, 2013 for corresponding U.S. Appl. No. 12/697,037.
Levin et al., “Tactile-Feedback Solutions for an Enhanced User Experience,” Information Display, Oct. 2009, pp. 18-21.
Office Action dated Mar. 12, 2013 for corresponding U.S. Appl. No. 12/947,532.
Office Action dated Mar. 18, 2013 for corresponding U.S. Appl. No. 12/947,321.
Japanese Patent Office, Notice of Reasons for Rejection, Application No. JP 2014-171836, dated Nov. 4, 2015.
Japanese Patent Office, Notice of Reasons for Rejection, Application No. JP 2011-554174, dated Jan. 28, 2014.
Korean Patent Office, Notice of Preliminary Rejection, Application No. 10-2011-7023927, dated Sep. 30, 2015.
Korean Patent Office, Notice of Preliminary Rejection, Application No. 10-2011-7023926, dated Sep. 30, 2015.
State Intellectual Property Office of the Peoples Republic of China, Notification of the Second Office Action, Application No. 201080011708 dated Jul. 10, 2014.
State Intellectual Property Office of the Peoples Republic of China, Notification of the Third Office Action, Application No. 201080011708 dated Jan. 26, 2015.
Japanese Patent Office, Office Action, Application No. JP 2015-083829 dated Feb. 16, 2016.
Japanese Patent Office, Notice of Reasons for Rejection, Application No. JP 2011-554180 dated Jan. 7, 2014.
Japanese Patent Office, Decision of Rejection, Application No. JP 2011-554180 dated Dec. 16, 2014.
State Intellectual Property Office of the Peoples Republic of China, Notification of the Second Office Action, Application No. 201080011743 dated Apr. 3, 2014.
State Intellectual Property Office of the Peoples Republic of China, Decision on Rejection, Application No. 201080011743 dated Nov. 3, 2014.
State Intellectual Property Office of the Peoples Republic of China, Notification of the First Office Action, Application No. 201080011744 dated Sep. 23, 2013.
Japanese Patent Office, Decision of Rejection, Application No. JP 2011-554175 dated Mar. 1, 2016.
Japanese Patent Office, Notice of Reasons for Rejection, Application No. JP 2015-020413 dated Mar. 1, 2016.
Japanese Patent Office, Notice of Reasons for Rejection, Application No. JP 2015-019308 dated Feb. 26, 2016.
Japanese Patent Office, Notice of Reasons for Rejection, Application No. JP 2011-554172 dated Jan. 28, 2014.
State Intellectual Property Office of the Peoples Republic of China, Notification of the Second Office Action, Application No. 201080011744 dated Jun. 24, 2014.
State Intellectual Property Office of the Peoples Republic of China, Notification of the Third Office Action, Application No. 201080011744 dated Dec. 16, 2014.
State Intellectual Property Office of the Peoples Republic of China, Notification of the Fourth Office Action, Application No. 201080011744 dated Aug. 12, 2015.
Korean Patent Office, Notice of Preliminary Rejection, Application No. 10-2011-7023987 dated Aug. 21, 2015.
United States Patent and Trademark Office, Office Action, U.S. Appl. No. 12/696,893 dated Feb. 29, 2012.
United States Patent and Trademark Office, Office Action, U.S. Appl. No. 12/696,893 dated Sep. 6, 2012.
United States Patent and Trademark Office, Office Action, U.S. Appl. No. 12/696,893 dated Jun. 6, 2013.
United States Patent and Trademark Office, Office Action, U.S. Appl. No. 12/696,893 dated Oct. 22, 2013.
United States Patent and Trademark Office, Office Action, U.S. Appl. No. 12/696,893 dated Mar. 11, 2014.
Meyer, D. et al., Fingertip Friction Modulation due to Electrostatic Attraction, IEEE World Haptics Conference Apr. 14-18, 2013, Korea, pp. 43-48.
United States Patent and Trademark Office, Office Action, U.S. Appl. No. 12/697,010 dated May 20, 2016.
United States Patent and Trademark Office, Office Action, U.S. Appl. No. 12/697,042 dated May 17, 2016.
United States Patent and Trademark Office, Office Action, U.S. Appl. No. 12/697,037 dated Jun. 6, 2016.
United States Patent and Trademark Office, Office Action, U.S. Appl. No. 12/696,908 dated May 26, 2016.
Japanese Patent Office, Notice of Reasons for Rejection, Application No. JP 2015-020413 dated Aug. 30, 2016.
United States Patent and Trademark Office, Office Action, U.S. Appl. No. 12/696,908 dated Nov. 10, 2016.
United States Patent and Trademark Office, Office Action, U.S. Appl. No. 12/697,010 dated Sep. 16, 2016.
United States Patent and Trademark Office, Office Action, U.S. Appl. No. 12/947,532 dated Sep. 9, 2016.
United States Patent and Trademark Office, Office Action, U.S. Appl. No. 12/697,042 dated Nov. 30, 2016.
United States Patent and Trademark Office, Office Action, U.S. Appl. No. 12/697,037 dated Jan. 17, 2017.
United States Patent and Trademark Office, Office Action, U.S. Appl. No. 12/947,532 dated Feb. 2, 2017.
United States Patent and Trademark Office, Office Action, U.S. Appl. No. 12/697,010 dated Feb. 22, 2017.
European Patent Office, Examination Report, Application No. 10712201.2 dated Mar. 2, 2017.
Japanese Patent Application 2015-020413, Final Office Action dated Apr. 4, 2017, 4 pages.
U.S. Appl. No. 12/696,908, Non-Final Office Action dated Sep. 21, 2017, 20 pages.
U.S. Appl. No. 12/947,532, Final Office Action dated Sep. 8, 2017, 34 pages.
European Patent Office Application No. 10787610.4, Communication Pursuant to Article 94(3) EPC dated Aug. 21, 2017, 7 pages.
U.S. Appl. No. 12/947,321, Final Office Action dated Oct. 3, 2017, 36 pages.
State Intellectual Property Office of the Peoples Republic of China Application No. 201510760216.2, Office Action dated Nov. 3, 2017, 10 pages.
“MicroTouch Capacitive TouchSense System,” 3M Touch System; Methuen, MA.; Published 2008.
U.S. Appl. No. 12/696,908, Final Office Action dated Jun. 29, 2017.
U.S. Appl. No. 12/947,321, Non-Final Office Action dated Jun. 16, 2017.
KR 10-2016-7023565, “Office Action”, dated May 18, 2018, 6 pages.
U.S. Appl. No. 15/988,359 , “Non Final Office Action”, dated Jun. 29, 2018, 14 pages.
JP 2017-149800, “Office Action”, dated Jun. 26, 2018, 5 pages.
CN 201510760216.2, “Office Action”, dated Jun. 26, 2018, 6 pages.
Patent number: 10073526
Patent Publication Number: 20170255268
Application Number: 15/601,580
International Classification: G06F 3/01 (20060101); G06F 3/0488 (20130101); G08B 6/00 (20060101);