Haptic device with multitouch display

A method, apparatus, and/or computer program product integrates one or more haptic devices with a multi-touch display. One or more haptic devices are placed upon a multi-touch display surface to form a set of identified haptic devices. The set of identified haptic devices is calibrated upon the multi-touch display surface, where calibration provides localized haptic interaction over subsets of the multi-touch display surface, thus enabling feedback to the identified haptic devices.

BACKGROUND

This disclosure relates generally to input devices in a data processing system and more specifically to calibrating one or more haptic devices with a multi-touch display in the data processing system.

The term “haptics” refers to the sense of touch. Haptic technology involves human computer interaction devices that interface using a sense of touch applying forces, vibrations, and/or motions to the user. Multi-touch displays support sensing of multiple interaction points on a display surface. Example multi-touch systems typically include smart devices such as phones with touch sensitive screens and kiosks with touch screens. Multi-touch sensing extends beyond the detection of human fingertips and can include detection of a variety of physical objects and visual markers, depending on the sensing technology adopted.

The combination of one or more haptic devices with a visual display is known as a “hapto-visual system”. Such systems may be collocated, where the haptic interaction and visual display volumes overlap. Several hapto-visual systems have been developed since the late 1960s. Most hapto-visual systems comprise a liquid crystal display (LCD) or cathode ray tube (CRT) desktop monitor reflected in a half-silvered mirror with a haptic device mounted behind the mirror. The user observes a stereoscopic image projected by the monitor by looking at the mirror, such that the virtual object represented by the image appears to be located behind the mirror, and holds the haptic device. The combined effect is one of “hands on” interaction with a virtual object or environment, in which the user can see and touch virtual objects as if the virtual objects were physical objects. Typical large-scale systems however use a different design where a large haptic device is situated in front of a large display surface.

The hapto-visual systems described above incorporate a robotic device for force-feedback. Alternative haptic devices have been integrated with display devices, including: tactile displays, such as technology that provides either limited display deformation or vibration feedback; physical widgets, such as silicone illuminated active peripheral (SLAP) widgets that use passive resistance to simulate control widgets on the display surface; and the use of magnetic induction to move objects on a display surface, such as provided by a Proactive Desk.

Personal hapto-visual systems typically limit collaboration between physically adjacent colleagues. Only one user can interact with such a system at a given moment. Personal hapto-visual systems also limit visual context due to their use of small-scale display devices. Increasing the number of haptic devices in a personal hapto-visual system, for example to support bimanual interaction, is difficult due to overlapping physical work volumes and a high probability of collision between haptic devices. Large-scale hapto-visual systems limit the work volume accessible by the haptic device to a fixed subset of the visual display surface and additionally obscure substantial portions of the visual display with parts of the haptic device. Large-scale hapto-visual systems also pose safety risks due to the close proximity of a large robotic (haptic) device to the body and head of a user while the attention of the user is focused on the visual display.

Tactile displays and active peripherals limit the haptic working volume to the display plane. A user only feels haptic feedback while the fingers of the user are in contact with the display surface, which is a severe restriction in stereoscopically projected visual virtual environments.

In the field of haptic devices, several current solutions are available including a two dimensional hapto-visual system using an electro-magnetic device or a resistive ballpoint in a stylus (pen) for obtaining force-feedback. Interaction with multiple styluses is described. The two dimensional hapto-visual system is essentially an “active peripheral”, however, with limitations.

Another example of a solution may be viewed as a trivial extension of a vibro-tactile display to accommodate multi-touch sensing. A similar variation of the previously stated vibro-tactile display system provides an emphasis on non-visual feedback in portable devices, for example, by providing a capability to locate graphic user interface widgets without examining the display.

In another example of a current solution, a variant of a tactile display is presented in which the surface of the display deforms either in response to a touch of the user or to emphasize two dimensional graphic user interface (GUI) elements such as active button widgets. The example solution has similar limitations as other tactile displays currently available.

In another example of current solutions, a combination of displays (some touch-sensitive, some not) and physical buttons (such as those on a game controller) provides a capability in which the device as a whole may vibrate or the physical buttons may provide force-feedback. The motion or feedback may be, for example, refusing to depress, in response to the combination of something displayed on one of the displays and a physical button or touch-screen press from a user.

In another example a haptic stylus provides a variety of vibration effects. For example, a vibrating haptic stylus is similar to tactile display systems except the actuator providing haptic feedback is mounted in the stylus rather than in the display. A further variation on tactile multi-touch displays, as previously described, includes a tactile element attached to the fingers of a user rather than the stylus or the display as stated previously.

Existing hapto-visual systems limit the scale of collocated hapto-visual interaction, impede collaboration between multiple simultaneous users and, in larger systems, risk safety and increase cost through placement of large robotic devices in close proximity to the head of a user.

SUMMARY

According to one embodiment, a computer-implemented process/method for integrating one or more haptic devices with a multi-touch display is presented. The computer-implemented process/method identifies one or more haptic devices to form a set of identified haptic devices upon a multi-touch display surface and calibrates the identified haptic devices upon the multi-touch display surface, wherein calibration provides localized haptic interaction over subsets of the multi-touch display surface. The computer-implemented process/method further enables feedback to the identified haptic devices.

According to another embodiment, a computer program product for integrating one or more haptic devices with a multi-touch display is presented. The computer program product comprises a computer recordable-type media containing computer executable program code stored thereon. The computer executable program code comprises computer executable program code for identifying one or more haptic devices to form a set of identified haptic devices upon a multi-touch display surface, computer executable program code for calibrating the identified haptic devices upon the multi-touch display surface, wherein calibration provides localized haptic interaction over subsets of the multi-touch display surface and computer executable program code for enabling feedback to the identified haptic devices.

According to another embodiment, an apparatus for integrating one or more haptic devices with a multi-touch display is presented. The apparatus comprises a communications fabric, a memory connected to the communications fabric, wherein the memory contains computer executable program code, a communications unit connected to the communications fabric, an input/output unit connected to the communications fabric, a display connected to the communications fabric and a processor unit connected to the communications fabric. The processor unit executes the computer executable program code to direct the apparatus to identify one or more haptic devices to form a set of identified haptic devices upon a multi-touch display surface, calibrate the identified haptic devices upon the multi-touch display surface, wherein calibration provides localized haptic interaction over subsets of the multi-touch display surface and enable feedback to the identified haptic devices.

DETAILED DESCRIPTION

Although an illustrative implementation of one or more embodiments is provided below, the disclosed systems and/or methods may be implemented using any number of techniques. This disclosure should in no way be limited to the illustrative implementations, drawings, and techniques illustrated below, including the exemplary designs and implementations illustrated and described herein, but may be modified within the scope of the appended claims along with their full scope of equivalents.

A computer-readable signal medium may include a propagated data signal with the computer-readable program code embodied therein, for example, either in baseband or as part of a carrier wave. Such a propagated signal may take a variety of forms, including but not limited to electro-magnetic, optical or any suitable combination thereof. A computer readable signal medium may be any computer readable medium that is not a computer readable storage medium and that can communicate, propagate, or transport a program for use by or in connection with an instruction execution system, apparatus, or device.

Program code embodied on a computer-readable medium may be transmitted using any appropriate medium, including but not limited to wireless, wire line, optical fiber cable, RF, etc. or any suitable combination of the foregoing.

Turning now toFIG. 1a block diagram of an exemplary data processing system operable for various embodiments of the disclosure is presented. In this illustrative example, data processing system100includes communications fabric102, which provides communications between processor unit104, memory106, persistent storage108, communications unit110, input/output (I/O) unit112, and display114.

Memory106and persistent storage108are examples of storage devices116. A storage device is any piece of hardware that is capable of storing information, such as, for example without limitation, data, program code in functional form, and/or other suitable information either on a temporary basis and/or a permanent basis. Memory106, in these examples, may be, for example, a random access memory or any other suitable volatile or non-volatile storage device. Persistent storage108may take various forms depending on the particular implementation. For example, persistent storage108may contain one or more components or devices. For example, persistent storage108may be a hard drive, a flash memory, a rewritable optical disk, a rewritable magnetic tape, or some combination of the above. The media used by persistent storage108also may be removable. For example, a removable hard drive may be used for persistent storage108.

Communications unit110, in these examples, provides for communications with other data processing systems or devices. In these examples, communications unit110is a network interface card. Communications unit110may provide communications through the use of either or both physical and wireless communications links.

Input/output unit112allows for input and output of data with other devices that may be connected to data processing system100. For example, input/output unit112may provide a connection for user input through a keyboard, a mouse, and/or some other suitable input device. Further, input/output unit112may send output to a printer. Display114provides a mechanism to display information to a user.

Instructions for the operating system, applications and/or programs may be located in storage devices116, which are in communication with processor unit104through communications fabric102. In these illustrative examples the instructions are in a functional form on persistent storage108. These instructions may be loaded into memory106for execution by processor unit104. The processes of the different embodiments may be performed by processor unit104using computer-implemented instructions, which may be located in a memory, such as memory106.

These instructions are referred to as program code, computer usable program code, or computer readable program code that may be read and executed by a processor in processor unit104. The program code in the different embodiments may be embodied on different physical or tangible computer readable media, such as memory106or persistent storage108.

Program code118is located in a functional form on computer readable media120that is selectively removable and may be loaded onto or transferred to data processing system100for execution by processor unit104. Program code118and computer readable media120form computer program product122in these examples. In one example, computer readable media120may be in a tangible form, such as, for example, an optical or magnetic disc that is inserted or placed into a drive or other device that is part of persistent storage108for transfer onto a storage device, such as a hard drive that is part of persistent storage108. In a tangible form, computer readable media120also may take the form of a persistent storage, such as a hard drive, a thumb drive, or a flash memory that is connected to data processing system100. The tangible form of computer readable media120is also referred to as computer recordable storage media. In some instances, computer readable media120may not be removable.

Alternatively, program code118may be transferred to data processing system100from computer readable media120through a communications link to communications unit110and/or through a connection to input/output unit112. The communications link and/or the connection may be physical or wireless in the illustrative examples. The computer readable media also may take the form of non-tangible media, such as communications links or wireless transmissions containing the program code.

In some illustrative embodiments, program code118may be downloaded over a network to persistent storage108from another device or data processing system for use within data processing system100. For instance, program code stored in a computer readable storage medium in a server data processing system may be downloaded over a network from the server to data processing system100. The data processing system providing program code118may be a server computer, a client computer, or some other device capable of storing and transmitting program code118.

As another example, a storage device in data processing system100may be any hardware apparatus that may store data. Memory106, persistent storage108and computer readable media120are examples of storage devices in a tangible form.

According to an illustrative embodiment, a computer-implemented process for integrating one or more haptic devices with a multi-touch display is presented. Using data processing system100ofFIG. 1as an example, an illustrative embodiment provides the computer-implemented process stored in memory106, executed by processor unit104, for integrating one or more haptic devices with a multi-touch display. Processor unit104identifies one or more haptic devices to form a set of identified haptic devices upon a multi-touch display surface and calibrates the identified haptic devices upon the multi-touch display surface, wherein calibration provides localized haptic interaction over subsets of the multi-touch display surface. Processor unit104further enables feedback to the identified haptic devices. In another example, a computer-implemented process, using program code118stored in memory106or as a computer program product122, for integrating one or more haptic devices with a multi-touch display is presented.

In an alternative embodiment, program code118containing the computer-implemented process may be stored within computer readable media120as computer program product122. In another illustrative embodiment, the process for integrating one or more haptic devices with a multi-touch display may be implemented in an apparatus comprising a communications fabric, a memory connected to the communications fabric, wherein the memory contains computer executable program code, a communications unit connected to the communications fabric, an input/output unit connected to the communications fabric, a display connected to the communications fabric, and a processor unit connected to the communications fabric. The processor unit of the apparatus executes the computer executable program code to direct the apparatus to perform the process.

With reference toFIG. 2, a block diagram of a haptic visual feedback system, in accordance with various embodiments of the disclosure is presented. System200is an example of a haptic visual feedback system used to calibrate haptic and visual feedback. System200provides a capability to calibrate haptic and visual feedback with respect to how a haptic work volume relates to a display. For example, a surgical simulation system that represents a scalpel via a haptic device and represents a human organ visually via a display surface may calibrate haptic and visual feedback such that observed intersection of the tip of the haptic device end effector with the visual presentation of the human organ will correspond with a visual update presenting an incision in the displayed human organ at the point of intersection and force feedback through the haptic device presenting contact between the scalpel and the human organ.

The components of system200provide a capability of placing one or more small-scale, commodity haptic devices directly on a sensing “multi-touch” display surface, and dynamically moving the haptic devices on the surface. Embodiments of system200typically solve the problems of scale in collocated hapto-visual systems, collaboration between multiple simultaneous users, improving safety and reducing cost when integrating haptic devices with large-scale visual displays.

System200is a combination of commodity haptic devices202with a multi-touch display surface204enabling localized haptic interaction over subsets of the display surface. One or more haptic devices202are placed on multi-touch display surface204and multi-touch display surface204senses a presence on the surface using a mechanism such as detection208. System200automatically calibrates haptic and visual feedback, recalibrating when one or more haptic devices202are relocated, and haptic feedback is disabled when presence of one or more haptic devices202are no longer sensed by multi-touch display surface204.

Haptic work volume projections206are projected areas on the surface of multi-touch display surface204. The projections are regions of the display in which physical interaction with displayed images is possible through feedback to effectors, including handles, of an associated haptic device. Identification210determines which particular haptic device is active and calculates a respective work volume projection accordingly. Identification210may also provide a distinguishing mark for an occurrence of a haptic device in a set of homogeneous devices to distinguish one device from another similar device. Shapes/marks212provides a storage repository to store, maintain and look-up shapes and marks associated with haptic devices. For example, a detected shape may be compared with a stored shape in shapes/marks212to identify a specific haptic device from a plurality of haptic devices. Feedback notification may be communicated to the detected haptic device in accordance with specifications of the haptic device.

Compared with previous available personal hapto-visual workbenches, system200enables multiple users to collaborate around the surface of the display table, each user having a capability for haptic interaction with visual information, where the haptic interaction occurs over a larger display area. Support for the use of haptic device202with a large multi-touch display surface204enables multiple haptic devices to be used simultaneously without physical conflict among the workspaces associated with respective haptic devices. Additionally, a large display surface of multi-touch display surface204typically provides more visual context for user interaction than a personal hapto-visual workbench.

Compared with current larger scale hapto-visual displays, system200typically has a significantly lower cost, due to the use of commodity haptic devices, haptic devices202, and improved safety. Large-scale hapto-visual display systems typically place users in close proximity to a large robotic device while the users focus attention on a visual display. In contrast with such systems, system200enables automatic disengagement of haptic feedback when a haptic device is removed from the display surface, use of smaller commodity haptic devices capable of exerting lower forces and no longer places a haptic device in close proximity to the head of a user. System200also provides high fidelity haptic interaction over an entire display surface, due to an ability to dynamically relocate a haptic device, whereas large scale hapto-visual displays feature a “sweet spot” for haptic interaction and therefore do not completely cover the display surface, due to a fixed base and limited reach of the haptic device.

Compared with tactile displays, such as those from Pacinian Corporation, and active peripherals, such as silicone illuminated active peripherals widgets, system200provides high fidelity haptic interaction in3dimensions, with multiple degrees of freedom. Commodity haptic devices typically allow more general haptic interaction than tactile displays, including simulation of handheld tools in virtual reality simulations.

With reference toFIG. 3, a pictorial diagram of a haptic device in operation with a multi-touch surface, in accordance with one embodiment of the disclosure is presented. System300depicts an example interaction between a user and a haptic device using a multi-touch surface.

User302manipulates haptic device306on multi-touch surface304typically using a handle or other control mechanism. Haptic work volume308is a three dimensional spatial region reachable by the end effector of haptic device306without moving the base of haptic device306on multi-touch surface304.

With reference toFIG. 4, a pictorial diagram of haptic work volume projections on a multi-touch display, in accordance with one embodiment of the disclosure is presented.FIG. 4is an example of a combination of haptic devices and a multi-touch display table ofFIG. 3providing localized haptic interaction over subsets of the display area.

View400represents a combination of a multi-touch display table, multi-touch display surface402, with two commodity haptic devices, shown as haptic devices404, to provide localized haptic interaction over defined subsets of multi-touch display surface402. Haptic devices404are low-cost devices with a small work volume and high fidelity sensing capable of providing haptic feedback in three or more degrees of freedom. Examples of the relatively low-cost devices include the PHANTOM Omni® (“PHANTOM Omni” is a registered trademark of SensAble Technologies, Inc. in the United States and/or other countries) and the Falcon® (“Novint Falcon” is a registered trademark of Novint Technologies, Inc. in the United States and/or other countries). Examples of multi-touch display tables including the DiamondTouch and the Microsoft Surface® (“Microsoft” and “Surface” are registered trademarks in the United States and/or other countries) provide a larger display area than a working volume of commodity haptic devices. For example, the DiamondTouch has an active display area of 86 cm×65 cm, whereas the PHANTOM Omni has a working volume of 16 cm×7 cm with a height of 12 cm. Accordingly, when a commodity haptic device, such as one of haptic devices404, is placed on a multi-touch display table, the haptic working volume includes only a subset of the active visual display area of the multi-touch display table. Placing multiple haptic devices on the multi-touch display table provides haptic feedback over several subsets of the display depicted as haptic work volume projections406.

One or more haptic devices404are placed on multi-touch display surface402and the table senses the presence of haptic devices404on the surface. Multi-touch display surface402is capable of sensing shapes of objects placed on the display surface. Haptic devices404such as the PHANTOM Omni and the Falcon have distinctively shaped bases, enabling placement of a haptic device on the display surface to be easily distinguished from other contact devices or elements, such as fingertips. The simultaneous presence of multiple, heterogeneous devices can be detected, by a same standard means, to support haptic feedback over several subsets of the display. Multiple homogenous devices can be distinguished using markers such as those described for a thin form-factor interactive surface technology such as a ThinSight display surface from Microsoft® (Microsoft is a registered trademark of Microsoft Corporation in the United States and/or other countries).

The system automatically calibrates haptic and visual feedback, recalibrating when a haptic device is relocated. Similarly haptic feedback is disabled when presence of a haptic device is not sensed by the display. The distinctive shape of a haptic device base further enables the system, such as system400, to identify position and orientation of haptic devices404on multi-touch display surface402, which in turn allows the sensed position and orientation of the haptic device end effector to be calibrated with respect to a displayed image, providing collocated hapto-visual interaction within the working volume of the haptic device.

Haptic devices404can be dynamically relocated on multi-touch display surface402by lifting a device base and placing the device elsewhere on the table, or by sliding the device base. Once the haptic device base has come to rest, a new position and orientation is sensed by means previously described. Haptic feedback is re-calibrated to provide collocated hapto-visual interaction for a possibly different subset of the display surface. When haptic devices404are removed from multi-touch display surface402, the working volumes associated with haptic devices404are not collocated with any portion of multi-touch display surface402. In this situation, haptic feedback may be disabled to avoid force feedback from unseen sources and thereby increase safety. Similarly, when haptic devices404are slid across multi-touch display surface402, haptic devices404may interact unexpectedly with virtual objects displayed on multi-touch display surface402, accordingly haptic feedback may be disabled as well and re-enabled when position and orientation of the device bases are sensed to have stabilized, for example, by remaining at a same location for a defined period of time.

With reference toFIG. 5, a pictorial diagram of specific haptic device footprints, in accordance with one embodiment of the disclosure are presented. Shapes500are examples representative of typical haptic device footprint shapes currently available.

Shape502is representative of the footprint distinguishing the object as a Phantom Omni device base outline. Shape504is representative of the footprint distinguishing the object as a Falcon device base outline. Multi-touch display surface402ofFIG. 4is capable of sensing shapes of objects placed on a respective display surface. Haptic devices404such as PHANTOM Omni and Novint Falcon have distinctively shaped bases, as shown inFIG. 5, therefore placement of a haptic device on a display surface can typically be easily distinguished from other contact elements, such as fingertips of an operator. The capability to distinguish a sensed position and orientation of a haptic device further enables the haptic device end effector to be calibrated.

With reference toFIG. 6, a flowchart of a process of integrating a haptic device with a multi-touch display is presented. Process600is an example of a process using system200ofFIG. 2.

Process600begins (step602) and determines whether a presence of a haptic device is detected (step604). When a determination is made that a presence of a haptic device is not detected, process600loops back to perform step604. When a determination is made that a presence of one or more haptic devices is detected, process600identifies the one or more detected haptic devices to form a set of identified haptic devices (step606). A set contains one or more haptic devices.

Process600calibrates the set of identified haptic devices placed upon the multi-touch display surface (step608). Calibration provides localized haptic interaction over subsets of the multi-touch display surface. The subsets of the multi-touch display surface are calculated as a set of haptic work volumes associated with respective identified haptic devices. Process600enables physical feedback through the identified haptic devices (step610). Additional auditory, visual or a combination of sensory feedback appropriate to the haptic devices and scenario may be provided through other devices. Feedback is meant to be meaningful to a user to indicate a condition of the haptic devices with respect to the multi-touch display surface.

Process600determines whether the position or orientation of an identified haptic device has changed (step612). When a determination is made that the position or orientation of an identified haptic device has changed, (yes), process600loops back to perform step608as before. When a determination is made that the position and orientation of the identified haptic device has not changed, process600determines whether the position or orientation of the identified haptic device is changing (step614).

When a determination is made that the position or orientation of the identified haptic device is changing, (yes), process600disables feedback for the identified haptic device and waits for the position and orientation to stabilize (step616). The wait time is a predetermined duration that is configurable for a haptic device and may be set as a default time period. Process600proceeds with step618.

When a determination is made that the position and orientation of the identified haptic device is not changing, process600determines whether the presence of the identified haptic device is detected (step618). When a determination is made that the presence of the identified haptic device is detected process600terminates (step622). When a determination is made that the presence of the identified haptic device is not detected process600disables feedback for the identified haptic device (step620) and terminates thereafter (step622). When process600determines an identified haptic device is not detected, process600presumes the haptic device is no longer in the presence of the multi-touch display surface and therefore no longer should be considered for feedback communication.

Thus is presented in one embodiment a computer-implemented process for integrating one or more haptic devices with a multi-touch display. The computer-implemented process identifies one or more haptic devices to form a set of identified haptic devices upon a multi-touch display surface. The computer-implemented process further calibrates the identified haptic devices upon the multi-touch display surface, wherein calibration provides localized haptic interaction over subsets of the multi-touch display surface and enables feedback to the identified haptic devices.