Abstract:
A system and method of providing a surgical training apparatus which comprises a plurality of mechanical interfaces each of which is configured to receive a user tool operable by a user. Each mechanical interface of the plurality is configured to allow movement of its respective user tool in a rotational degree of freedom and a linear degree of freedom. The apparatus includes one or more sensors which are coupled to the mechanical interfaces and configured to simultaneously sense positional information of each user tool during movement. A computer coupled to the one or more sensors and configured to run a software application simulating each user tool as a respective simulated surgical tool operating on a simulated body part in a displayed graphical environment, wherein the computer updates movement of each simulated user tool in the graphical environment based on said positional information.

Description:
PRIORITY CLAIM 
     The present application is a continuation of U.S. patent application Ser. No. 11/301,831, filed Dec. 12, 2005 which is a continuation of U.S. patent application Ser. No. 10/196,563, filed Jul. 15, 2002, now U.S. Pat. No. 7,056,123 which claims the benefit of U.S. Provisional Patent Application Ser. No. 60/305,957, filed on Jul. 16, 2001, entitled “Interface Apparatus With Cable-Driven Force Feedback And Four Grounded Actuators,” commonly owned herewith. 
    
    
     TECHNICAL FIELD 
     The present relates generally to interface devices between humans and computers, and more particularly to computer interface devices that provide force feedback to the user. 
     BACKGROUND 
     Virtual reality computer systems provide users with the illusion that they are part of a “virtual” environment. A virtual reality system will typically include a computer processor, virtual reality software, and virtual reality I/O devices such as head mounted displays, sensor gloves, three dimensional (“3D”) pointers, etc. 
     Virtual reality computer systems may be used for training. In many fields, such as aviation and vehicle and systems operation, virtual reality systems have been used successfully to allow a user to learn from and experience a realistic “virtual” environment. The appeal of using virtual reality computer systems for training relates, in part, to the ability of such systems to allow trainees the luxury of confidently operating in a highly realistic environment and making mistakes without “real world” consequences. For example, a virtual reality computer system allows a doctor-trainee or other human operator or user to “manipulate” a scalpel or probe within a computer-simulated “body,” and thereby perform medical procedures on a virtual patient. In this instance, the I/O device, which is typically a 3D pointer, stylus, or the like, is used to represent a surgical instrument such as a scalpel or probe. As the “scalpel” or “probe” moves within a provided space or structure, results of such movement are updated and displayed in a body image displayed on the screen of the computer system so that the operator gains the experience of performing such a procedure without practicing on an actual human being or a cadaver. In other applications, virtual reality computer systems allow a user to handle and manipulate the controls of complicated and expensive vehicles and machinery for training and/or entertainment purposes. 
     For virtual reality systems to provide a realistic (and therefore effective) experience for the user, sensory feedback and manual interaction should be as natural as possible. In addition to sensing and tracking a user&#39;s manual activity and feeding such information to the controlling computer to provide a 3D visual representation to the user, a human interface mechanism should also provide force or tactile (“haptic”) feedback to the user. The need for the user to obtain realistic haptic information is extensive in many kinds of simulation and other applications. For example, in medical/surgical simulations, the “feel” of a probe or scalpel simulator is important as the probe is moved within the simulated body. It would be invaluable to a medical trainee to learn how an instrument moves within a body, how much force is required depending on the operation performed, the space available in a body to manipulate an instrument, etc. Other applications similarly benefit from the realism provided by haptic feedback. A “high bandwidth” interface system, which is an interface that accurately responds to signals having fast changes and a broad range of frequencies as well as providing such signals accurately to a control system, is therefore desirable in these and other applications. 
     Several existing devices provide multiple degrees of freedom of motion of an instrument or manipulatable object and include haptic feedback. Many of these devices, however, are limited in how many degrees of freedom that forces are provided, and may also be less accurate and realistic than desired for a particular application. Devices having greater realism yet reasonable cost are desired for medical and other virtual simulation applications. 
     Overview 
     In an aspect, a system and method of providing a surgical training apparatus which comprises a plurality of mechanical interfaces each of which is configured to receive a user tool operable by a user. Each mechanical interface of the plurality is configured to allow movement of its respective user tool in a rotational degree of freedom and a linear degree of freedom. The apparatus includes one or more sensors which are coupled to the mechanical interfaces and configured to simultaneously sense positional information of each user tool during movement. A computer coupled to the one or more sensors and configured to run a software application simulating each user tool as a respective simulated surgical tool operating on a simulated body part in a displayed graphical environment, wherein the computer updates movement of each simulated user tool in the graphical environment based on said positional information. 
     Other features and advantages will be understood upon reading and understanding the description of the preferred exemplary embodiments, found hereinbelow, in conjunction with reference to the drawings, in which like numerals represent like elements. 
    
    
     
       BRIEF DESCRIPTION OF THE DRAWINGS 
         FIG. 1  is a schematic illustration of the system being used for medical simulation purposes; 
         FIGS. 2A and 2B  are perspective view of a mechanical simulation apparatus; 
         FIGS. 2C and 2D  are elevational views of the base structure and portions of the linkage mechanisms of the mechanical simulation apparatus; 
         FIG. 2E  is a rear view of the mechanical simulation of the apparatus; 
         FIG. 2F  is a top view of the mechanical simulation apparatus; 
         FIG. 2G  is a close up of top surfaces of the mechanical simulation apparatus; 
         FIG. 3A  is a perspective view of a mechanical linkage of the mechanical simulation of the apparatus; 
         FIG. 3B  is a top view of the mechanical linkage; 
         FIG. 3C  is a side view of the mechanical linkage; 
         FIG. 3D  is a front view of the mechanical linkage; 
         FIG. 3E  is a bottom view of the mechanical linkage; 
         FIGS. 4A and 4B  are perspective views of the mechanical linkage; 
         FIGS. 5A-5D  are side sectional views of the mechanical linkage; 
         FIGS. 6A and 6B  are bottom and perspective bottom views, respectively of the mechanical linkage; 
         FIGS. 7A-7C  are additional sectional perspective views of the mechanical linkage; and 
         FIGS. 8A and 8B  are sectional perspective and front views, respectively, of the mechanical linkage. 
     
    
    
     DETAILED DESCRIPTION 
       FIG. 1  illustrates an example of the use of the system for medical simulation purposes. A virtual reality system  10  used to simulate a medical procedure includes a human/computer interface apparatus  12 , an electronic interface  14 , and a host computer  16 . The illustrated virtual reality system  10  is directed to a virtual reality simulation of a laparoscopic surgery procedure. 
     The handle  26  of a laparoscopic tool  18  used is manipulated by an operator and virtual reality images are displayed on a display device  20  of a digital processing system in response to such manipulations. For example, when the tool  18  is moved by the user, a graphical representation of the tool or a part of the tool may be moved correspondingly within a graphical environment displayed on device  20 . Display device  20  may be a standard display screen or CRT, 3-D goggles, or any other visual interface. The digital processing system is typically a host computer  16 . The host computer can be a personal computer or workstation or other computer device or processor, such as a home video game system commonly connected to a television set, such as systems available from Nintendo, Sega, or Sony; a “set top box” which may be used, for example, to provide interactive television functions to users; an arcade game; a portable computing device, etc. Multiple tools  18 , each manipulatable by the user, may also be provided, as in a preferred embodiment described below. 
     Host computer  16  implements a host application program with which a user is interacting via peripherals and interface device  14 . For example, the host application program may be a video game, medical simulation, scientific analysis program, or even an operating system or other application program that utilizes force feedback. Typically, the host application provides images to be displayed on a display output device, as described below, and/or other feedback, such as auditory signals. The medical simulation example of  FIG. 1  includes a host medical simulation application program. Suitable software for such applications is available from Immersion® Corporation of San Jose, Calif. Alternatively, display screen  20  may display images from a game application program or other program. 
     One example of a human/interface apparatus  12  as illustrated herein is used to simulate a laparoscopic medical procedure. In addition to the handle of a standard laparoscopic tool  18 , the human/interface apparatus  12  may include a barrier  22  and a standard laparoscopic trocar  24  (or a facsimile of a trocar). The barrier  22  is used to represent a portion of the skin covering the body of a patient. Trocar  24  is inserted into the body of the virtual patient to provide an entry and removal point from the body of the patient for the laparoscopic tool  18  and to allow the manipulation of the laparoscopic tool. Barrier  22  and trocar  24  may be omitted from apparatus  12  in other embodiments if desired. Preferably, the laparoscopic tool  18  is modified; in one embodiment, the shaft is replaced by a linear axis member, as described below. In other embodiments, the end of the shaft of the tool (such as any cutting edges) may be removed. The distal end of the laparoscopic tool  18  may not be required for the virtual reality simulation. 
     The laparoscopic tool  18  includes a handle or “grip” portion  26  and a shaft portion  28 . The shaft portion is an elongated mechanical object, described in greater detail below. In one embodiment, the system is concerned with tracking the movement of the shaft portion  28  in three-dimensional space, e.g. four degrees of freedom. The shaft  28  is constrained at some point along its length such that it may move with four degrees of freedom within the simulated patient&#39;s body. 
     A mechanical apparatus  25  for interfacing mechanical input and output is shown within the “body” of the patient in phantom lines. When an interaction is simulated on the computer, the computer will send feedback signals to the tool  18  and mechanical apparatus  25 , which has actuators for generating forces in response to the position of the virtual laparoscopic tool relative to surfaces or features displayed on the computer display device. Mechanical apparatus  25  is described in greater detail below. Signals may be sent to and from apparatus  25  via interface  30 , which may be similar to interface  72  described below. 
     While one embodiment will be discussed with reference to the laparoscopic tool  18 , it will be appreciated that a great number of other types of objects may be used with the method and apparatus. In fact, the present may be used with any mechanical object where it is desirable to provide a human/computer interface with one to six degrees of freedom. Such objects may include endoscopic or other similar surgical tools used in medical procedures, catheters, hypodermic needles, wires, fiber optic bundles, styluses, joysticks, screw drivers, pool cues, hand grips, etc. 
     The electronic interface  14  is a component of the human/computer interface apparatus  12  and may couple the apparatus  12  to the host computer  16 . Electronic interface  14  may be included within a housing of mechanical apparatus  25 , within host computer  16 , or may be provided as a separate unit. More particularly, interface  14  is used in preferred embodiments to couple the various actuators and sensors of apparatus  25  (described in detail below) to computer  16 . In some embodiments, the interface may include a microprocessor local to the apparatus  25  to handle sensor data and actuator control. Suitable electronic configurations are described, for example, in U.S. Pat. Nos. 5,623,582; 5,821,920; 5,731,804; 5,734,373; 5,828,197; and 6,024,576. 
     Signals may be sent to and from interface  14  and computer  16  by a standard interface  32  (RS-232, USB, Firewire, serial, parallel, etc.) or by wireless transmission and reception. In various embodiments, interface  14  may serve solely as an input device for the computer  16 , solely as an output device for the computer  16 , or as an input/output (I/O) device for the computer  16 . The interface  14  may also receive inputs from other input devices or controls that are associated with apparatus  12  and may relay those inputs to computer  16 . For example, commands sent by the user activating a button on apparatus  12  may be relayed to computer  16  to implement a command or cause the computer  16  to output a command to the apparatus  12 . 
     In  FIGS. 2A and 2B , perspective views of mechanical apparatus  25  for providing mechanical input and output are shown. Apparatus  25  may include two or more tools  18  (only one is shown) to allow a user to realistically simulate an actual surgical procedure using laparoscopic instruments. A user may manipulate each of the tools  18  independently, where each tool is independently sensed and actuated. 
     Each tool  18  is coupled to a linear axis member  40 , which is coupled to a mechanical linkage  38 , which will be described in more detail below. The user object  44 , such as a handle, is preferably coupled to linear axis member  40 . The mechanical linkage is grounded via a base structure  46 . The actuators, such as DC motors, which output the forces on each linear axis member  40  and tool  18 , are in the described embodiment located within the base structure  46 , and are therefore all grounded. This configuration allows high fidelity and efficient haptic feedback to be produced with the apparatus  25 . The actuators may also include sensors which sense the rotation of the actuators and thus, detect the motion of the tool in the four degrees of freedom. In other embodiments, sensors may be coupled to parts of the linkage  38  to sense the motion of the tool more directly. 
     In the described embodiment, each linear axis member  40 /tool  18  may be moved in four degrees of freedom, shown as the insert degree of freedom  50 , the twist degree of freedom  52 , the first rotation (yaw)  54 , and the second rotation (pitch)  56 . Other embodiments may limit the degrees of freedom to a lesser number, or provide additional degrees of freedom. 
       FIGS. 2C and 2D  further illustrate the base structure  46  and portions of the linkage mechanisms  38  that are rotatably coupled to the base structure. 
       FIG. 2E  illustrates a rear view of the apparatus  25  showing many of the actuators and some of the sensors of the described embodiment. A rotary actuator  62 , such as a DC motor, drives the insert degree of freedom  50 , a rotary actuator  64  drives the yaw degree of freedom  54 , and a rotary actuator  66 , positioned behind actuator  64  in  FIG. 2E , drives the twist degree of freedom  52 . An actuator-sensor pair  70  drives-the pitch degree of freedom  56 . 
       FIG. 2F  illustrates a top view of the apparatus  25  and  FIG. 2G  is a close up of the top surfaces of the apparatus. A pulley  72  is coupled to actuator  62  and has a cable  160  wrapped around it. A pulley  74  is coupled to the actuator  64  and has a cable  106  wrapped around it. A pulley  76  is coupled to the actuator  66  and has a cable  130  wrapped around it. These cables are described in greater detail below. The cables are all routed to the mechanical linkage  38  through an aperture  77  in the side of the base structure. In the described embodiment, the cables may each be wrapped around its own central spindle  78  before being routed to their respective pulleys  72 ,  74 , or  76 . In the described embodiment, a sensor  65  senses the motion of the shaft of actuator  64 , a sensor  67  senses the motion of the spindle  78  connected to the shaft of actuator  62 , and a sensor  69  senses the motion of the shaft of actuator  66 . The sensors are optical encoders having emitters and detectors sensing marks on an encoder wheel coupled to the pulley or spindle, as shown. In the described embodiment, the sensor for the pitch degree of freedom  56  is provided on the housing of actuator/sensor  70  to measure the actuator shaft rotation directly. 
     Other types of sensors and actuators, which essentially serve as transducers for the system, may be used in other embodiments, such as analog potentiometers, Polhemus (magnetic) sensors, lateral effect photo diodes, etc. Alternatively, sensors may be positioned at other locations of relative motion or joints of mechanical apparatus  25 . It should be noted that the present may utilize both absolute and relative sensors. The actuators may also be of various types, such as active actuators and/or passive actuators. Active actuators may include linear current control motors, stepper motors, pneumatic/hydraulic active actuators, stepper, motor, brushless DC motors, pneumatic/hydraulic actuators, a torquer (motor with limited angular range), a voice coil, and other types of actuators that transmit a force to move an object. Passive actuators may also be used. Magnetic particle brakes, friction brakes, or pneumatic/hydraulic passive actuators may be used in addition to or instead of a motor to generate a damping resistance or friction in a degree of motion. In addition, in some embodiments, passive (or “viscous”) damper elements may be provided on the bearings of apparatus  25  to remove energy from the system and in intentionally increase the dynamic stability of the mechanical system. In other embodiments, this passive damping may be introduced by using the back electromotive force (EMF) of the actuators to remove energy from the system. In addition, in the voice coil embodiments, multiple wire coils may be provided, where some of the coils may be used to provide back EMF and damping forces. 
     The actuators and sensors are decoupled, meaning that these transducers are directly coupled to ground member  46  which is coupled to a ground surface  47 , i.e. the ground surface carries the weight of the transducers, not the user handling tool  18 . The weights and inertia of the transducers are thus substantially negligible to a user handling and moving the tool. This provides a more realistic interface to a virtual reality system, since the computer may control the transducers to provide substantially all of the forces felt by the user in these degrees of motion. In contrast, in typical prior art arrangements of multi-degree of freedom interfaces, one actuator “rides” upon another actuator in a serial chain of links and actuators. This low bandwidth arrangement causes the user to feel the inertia of coupled actuators when manipulating an object. 
     Optionally, additional transducers may be added to apparatus  25  to provide additional degrees of freedom for the tool  18 . For example, a transducer may be added to the grip of laparoscopic tool  18  to sense and/or output forces to the degree of freedom provided by the user moving two portions of a tool  18  relative to each other to simulate extending the cutting blade of the tool. 
       FIGS. 3A  (perspective view),  3 B (top view),  3 C (side view),  3 D (front view), and  3 E (bottom view) illustrate the mechanical linkage  38  of the apparatus  25 . The linkage  38  is rotatably coupled to the base structure  46  to allow the second rotation  56 , where cables from various moving parts of the linkage  38  extend to the actuators of the base structure, as detailed below. Linear axis member  40  may be moved relative to the linkage  38  to provide two degrees of freedom  50  and  52 , and moves with portions of the linkage to provide two other degrees of freedom  54  and  56 . 
       FIGS. 4A and 4B  shows a perspective view of mechanical linkage  38 . The second rotation (pitch)  56  is provided by a mechanical bearing positioned between the linkage  38  and the base structure  46 . To provide forces in the second rotation  56  from grounded actuator  70 , a capstan drive  80  may be a mechanical transmission transmitting forces from the actuator to the linkage  38 . A capstan pulley  82  may be rigidly coupled to the rotating shaft  71  of the actuator  70 , where the pulley has an axis of rotation parallel to the axis of rotation A of the linkage  38  for the degree of freedom  56  and the pulley is positioned adjacent to a drum  84  that is rigidly coupled to the linkage  38  as shown. A cable  86  is connected at one end of the drum  84 , routed along the edge of the drum, around the pulley  82  one or more times, and is routed along the remaining edge of the drum to its other side. The cable may be tensioned using tensioning nut  88 , for example. Other types of transmissions may be used in other embodiments, e.g. gears, friction wheels, belt drives, etc. 
     The first rotation (yaw)  54  of linkage  38  is provided by a different cable drive  100 . Cable drive  100  includes a drum  102  which is rigidly coupled to linkage member  110 , which rotates about degree of freedom  54  about axis B with respect to linkage member  112 . Two idler pulleys  104   a  and  104   b  are rotatably coupled to linkage member  112  and rotating about axes parallel to axis B. A cable  106 , shown as a dashed line, is routed from one end of drum  102 , around idler pulley  104   a , through the linkage member  38  and out to the base structure and driven pulley  74  of actuator  64 , where it is wrapped multiple times. The cable then is routed back into and through the linkage  38 , around the idler pulley  104   b , and along the edge of drum  102  to the tensioner  114 . This configuration allows the actuator to rotate the linkage member  110  by pulling the desired side of the drum  102  with the cable  106 . 
       FIGS. 5A ,  5 B,  5 C, and  5 D are other side sectional views of the linkage  38 , where examples of extremes of rotation of the linkage member  110  with respect to the linkage member  112  are shown. The motion may be limited by stops provided in the path of movement of the drum  102 . For example, as shown in  FIG. 5A , an opening  118  may be placed in the drum  102 . A stop member  120 , such as a cylinder, may be coupled to the linkage member  112  and positioned within the opening  118 , so that the stop member  120  will engage the ends of the opening  118  to provide the limits of motion of the drum. 
       FIGS. 6A and 6B  are bottom and perspective bottom views, respectively, of the linkage mechanism  38 . To allow forces to be output in the twist degree of freedom  52 , a first end of cable  130  (represented by a dashed line) is routed from directly-driven pulley  76  of the actuator  66  in the base structure  46  and through the linkage mechanism  38 . The cable  130  is routed around an idler pulley  132 , around another idler pulley  134 , and around another idler pulley  136 . The cable  130  is then wrapped counterclockwise (as viewed in  FIG. 6   a ) around a rotatable drum  138  and connected to the drum at a point  140  (point  140  may be located elsewhere in other embodiments). The other, second end of the cable  130  is also connected to the drum  138  at point  140  and may be wrapped counterclockwise (as viewed in  FIG. 6   a ) on the remaining side around the drum  138  to the pulley  142 . Cable  130  is routed from the second end around idler pulley  142  and then idler pulley  144 , where idler pulley  144  and idler pulley  134  are positioned adjacent to each other and have the same axis of rotation. Cable  130  is then routed around idler pulley  146 , which is positioned adjacent to and has the same axis of rotation as pulley  132 . The cable  130  is then routed through the linkage member  38 , both ends represented by line  130 , to the actuator  66  in the base structure, where it is wrapped multiple times around the pulley  76  directly-driven by the actuator  66 . 
     In operation, the actuator  66  may rotate the drum  138  in either direction, thereby rotating the linear axis member  40  and tool  18 . When the actuator shaft is rotated in one direction, the first end of cable  130  around pulley  136  is pulled, causing the drum to rotate about center point  170  in the corresponding direction. When the actuator shaft is rotated in the opposite direction, the second end of cable  130  is pulled around pulley  142 , causing the drum to rotate about central point  170  in its other direction. 
     To allow forces to be output in the linear insert degree of freedom  50 , a first end of cable  160  (represented by dashed line in  FIG. 6   a ) is routed from directly-driven pulley  72  of actuator  62  in the base structure  46  through the linkage mechanism  38 . The cable  160  is routed around idler pulley  162 , around idler pulley  164 , and then around idler pulley  166 . This first end  161  of cable  160  is then routed around pulley  169  (shown in  FIG. 7   a ) and is coupled to the linear axis member  40 . The second end  162  of the cable  160  is coupled to the linear axis member  40  on the other side of the central pivot point  170 . The cable  160  is routed from the second end, around pulley  168 , around pulley  172  which is adjacent to and rotates about the same axis as pulley  164 , and around pulley  174  which is adjacent to and rotates about the same axis as pulley  162 . The cable is then routed through the linkage mechanism  38  to the pulley  72  driven by the actuator  62 , where it is wrapped multiple times. 
     In operation, the actuator  62  may rotate its driven pulley in either direction to correspondingly pull on the first end or the second end of the cable  160 . If the first end is pulled, a downward force on the linear axis member  40  (as oriented in  FIG. 3 ) is output, while if the second end is pulled, an upward force on the linear axis member is output. 
       FIGS. 7A-7C  are additional sectional perspective views of the linkage mechanism  38  and the cables and pulleys described above, illustrating the mechanism of the insert degree of freedom  50 . 
       FIGS. 8A and 8B  are sectional perspective and front views of the linkage mechanism  38  showing features described above. 
     Thus, the mechanism preferably provides four grounded actuators to provide forces in four degrees of freedom of the tool  18 . To make the actuators grounded, cables are used to allow the actuators to output forces to a remote mechanical motion, i.e. the rotated drums or moved linear axis member is located far from the driven pulley, unlike standard capstan drives. The three cables (six ends) routed through the interior of the mechanical linkage and out to the base structure are bent in various ways around idler pulleys and about their lengthwise axes; however, this does not cause significant stretching in the cables. The six ends of the cables are preferably arranged close together close to the pitch axis A so as to minimize bending of the cables. For example, the six cable lengths may be arranged so that their cross sections approximately form a circle around the rotation axis A. 
     While the system has been described in terms of several preferred embodiments, it is contemplated that alterations, modifications and permutations thereof will become apparent to those skilled in the art upon a reading of the specification and study of the drawings. For example, the linked members of apparatus  25  may take a number of actual physical sizes and forms while maintaining the disclosed linkage structure. Likewise, other types of gimbal mechanisms or different mechanisms providing multiple degrees of freedom may be used with the drive mechanisms disclosed herein to reduce inertia, friction, and backlash in a system. A variety of devices may also be used to sense the position of an object in the provided degrees of freedom and to drive the object along those degrees of freedom. In addition, the sensor and actuator used in the transducer system having desired play may take a variety of forms. Similarly, other types of couplings may be used to provide the desired play between the object and actuator. Furthermore, certain terminology has been used for the purposes of descriptive clarity and not to limit.