Patent ID: 12201390

DETAILED DESCRIPTION

This description and the accompanying drawings that illustrate inventive aspects, embodiments, implementations, or applications should not be taken as limiting—the claims define the protected invention. Various mechanical, compositional, structural, electrical, and operational changes may be made without departing from the scope of this description and the claims. In some instances, well-known circuits, structures, or techniques have not been shown or described in detail in order not to obscure the invention. Like numbers in two or more figures represent the same or similar elements.

Elements described in detail with reference to one embodiment, implementation, or application may, whenever practical, be included in other embodiments, implementations, or applications in which they are not specifically shown or described. For example, if an element is described in detail with reference to one embodiment and is not described with reference to a second embodiment, the element may nevertheless be claimed as included in the second embodiment. Thus, to avoid unnecessary repetition in the following description, one or more elements shown and described in association with one embodiment, implementation, or application may be incorporated into other embodiments, implementations, or aspects unless specifically described otherwise, unless the one or more elements would make an embodiment or implementation non-functional, or unless two or more of the elements provide conflicting functions.

Aspects of the invention are described primarily in terms of an implementation using a da Vinci® Surgical System (specifically, a Model IS4000, marketed as the da Vinci® Xi™ HD™ Surgical System), commercialized by Intuitive Surgical, Inc. of Sunnyvale, California.

Knowledgeable persons will understand, however, that inventive aspects disclosed herein may be embodied and implemented in various ways, including robotic and, if applicable, non-robotic embodiments and implementations. Implementations on da Vinci® Surgical Systems (e.g., the Model IS4000 da Vinci® Xi™ Surgical System, the Model IS3000 da Vinci Si® Surgical System) are merely exemplary and are not to be considered as limiting the scope of the inventive aspects disclosed herein.

In accordance with various aspects, the present disclosure describes a surgical planning tool that includes a medical device configured to video record the performance of surgical procedures. The video recordings can be embedded with various metadata. e.g., highlights made by a medical person. Additionally, the video recordings can be tagged with various metadata, e.g., text annotations describing certain subject matter of the video, the identity of the patient to whom the video recording corresponds, biographical or medical information about the patient, and the like. In one aspect, tagged metadata is embedded in the video recordings.

In accordance with further aspects, the present disclosure describes a teleoperated medical device that includes a surgical instrument used to perform at least one surgical activity during a surgical procedure. A surgical instrument typically has a limited useful lifetime during which it can be reliably used to perform the surgical activity. In some embodiments, a lifetime is indicated as a count of a number of surgeries in which it is permissible to use a surgical instrument. For example, an instrument having a lifetime count of ten is permitted to be used in ten more surgeries, Conversely, for example, an instrument having a lifetime count of five is permitted to be used in five more surgeries.

Different surgical instruments are used to perform different surgical activities. For example, a scalpel is used for dissecting, a needle is used for suturing, and a heat source is used for cauterizing. The use of a surgical instrument to perform its surgical activity during a surgical procedure imparts wear to the surgical instrument. Surgical instrument wear accumulates so as to degrade the utility of the surgical instrument over the course of several surgical procedures to a degree that it is no longer reliably useful for its activity. For example, wear upon a scalpel can involve dulling of the scalpel cutting edges. A surgical instrument that is worn down a degree that it is no longer reliably useful typically is discarded or if feasible, refurbished. In accordance with some embodiments, a record is maintained of remaining useful lifetime of a surgical instrument. Incremental wear imparted to the surgical instrument during a surgical procedure is tracked. The record of remaining useful lifetime of the surgical instrument is updated based upon the incremental wear imparted during the surgical procedure.

In a teleoperated surgical system, different instruments may be used at different stages of a surgical procedure. Moreover, the same instrument may be used in different actuator states at different stages of a surgical procedure. As used herein, the term actuator state refers to a mechanical disposition of a surgical instrument as determined by an actuator, such as a motor, in response to input commands received from a surgeon or other surgical team member.

The video recordings and information structures that associate surgical instrument actuator states with surgical guidance or actuator safety state information can be archived on an electronic medical record database implemented locally or on a cloud data storage service. The video recordings can be made available to interested health care providers. The information structures can be made available for use with the teleoperated medical device to provide surgical guidance and to control surgical instrument actuator state during performance of at least one surgical activity during a surgical procedure.

Health care providers can search the medical device database based upon surgeon skill level for videos and information structure relationships of interest using the metadata tags described above. Additionally, in one aspect, the surgical planning tool includes a computer-based pattern matching and analysis algorithm. In one aspect, the pattern-matching algorithm culls through the videos stored on the electronic medical record database to identify correlations between visual characteristics in the video recordings and associated metadata tags made by medical persons. The surgical planning tool can apply these correlations to newly encountered anatomy, and thereby assist medical persons performing a procedure in making determinations about patient anatomy, preferred surgical approaches, disease states, potential complications, etc.

In another aspect, a pattern matching algorithm culls through videos stored on the electronic medical record database to identify correlations between visual characteristics in the video recordings to identify surgical activities that contribute to instrument degradation. Some routine surgical activities result in predictable rates of instrument degradation so that an instrument can be designated for efficacy and safety reasons as suitable for a fixed number of surgical uses, referred to as “lifetimes”. For example, an instrument that has ‘x’ number of lifetimes for its intended use is eligible for its intended use in ‘x’ surgeries before it must be refurbished or discarded. Certain irregular surgical uses of an instrument can accelerate its degradation. Surgical uses of individual instruments are tracked. A surgical planning tool can apply correlations between surgical activities and instrument wear-rate for to individual surgical instruments for inventory planning purposes to determine when to replace or refurbish individual surgical instruments based upon their individual remaining lifetimes.

Minimally Invasive Teleoperated Surgical System

Referring now to the drawings, in which like reference numerals represent like parts throughout the several views.FIG.1is a plan view of a minimally invasive teleoperated surgical system10, typically used for performing a minimally invasive diagnostic or surgical procedure on a patient12who is lying on an operating table14. The system includes a surgeon's console16for use by a surgeon18during the procedure. One or more assistants20may also participate in the procedure. The minimally invasive teleoperated surgical system10further includes a patient-side cart22and an electronics cart24. The patient-side cart22can manipulate at least one removably coupled surgical instrument26through a minimally invasive incision in the body of the patient12while the surgeon18views the surgical site through the surgeon's console16. An image of the surgical site can be obtained by an endoscope28, such as a stereoscopic endoscope, which can be manipulated by the patient-side cart22to orient the endoscope28. Computer processors located on the electronics cart24can be used to process the images of the surgical site for subsequent display to the surgeon18through the surgeon's console16. The number of surgical instruments26used at one time will generally depend on the diagnostic or surgical procedure and the space constraints within the operating room among other factors. If it is necessary to change one or more of the surgical instruments26being used during a procedure, an assistant20can remove the surgical instrument26from the patient-side cart22, and replace it with another surgical instrument26from a tray30in the operating room.

FIG.2is a perspective view of the surgeon's console16. The surgeon's console16includes a left eye display32and a right eye display34for presenting the surgeon18with a coordinated stereoscopic view of the surgical site that enables depth perception. The console16further includes one or more control inputs36. One or more surgical instruments installed for use on the patient-side cart22(shown inFIG.1) move in response to surgeon18's manipulation of the one or more control inputs36. The control inputs36can provide the same mechanical degrees of freedom as their associated surgical instruments26(shown inFIG.1) to provide the surgeon18with telepresence, or the perception that the control inputs36are integral with the instruments26so that the surgeon has a strong sense of directly controlling the instruments26. To this end, position, force, and tactile feedback sensors (not shown) may be employed to transmit position, force, and tactile sensations from the surgical instruments26back to the surgeon's hands through the control inputs36.

The surgeon's console16is usually located in the same room as the patient so that the surgeon can directly monitor the procedure, be physically present if necessary, and speak to a patient-side assistant directly rather than over the telephone or other communication medium. But, the surgeon can be located in a different room, a completely different building, or other remote location from the patient allowing for remote surgical procedures.

FIG.3is a perspective view of the electronics cart24. The electronics cart24can be coupled with the endoscope28and includes a computer processor to process captured images for subsequent display, such as to a surgeon on the surgeon's console, or on another suitable display located locally and/or remotely. For example, if a stereoscopic endoscope is used, a computer processor on electronics cart24can process the captured images to present the surgeon with coordinated stereo images of the surgical site. Such coordination can include alignment between the opposing images and can include adjusting the stereo working distance of the stereoscopic endoscope. As another example, image processing can include the use of previously determined camera calibration parameters to compensate for imaging errors of the image capture device, such as optical aberrations. Optionally, equipment in electronics cart may be integrated into the surgeon's console or the patient-side cart, or it may be distributed in various other locations in the operating room.

FIG.4diagrammatically illustrates a teleoperated surgical system50(such as the minimally invasive teleoperated surgical system10ofFIG.1). A surgeon's console52(such as surgeon's console16inFIG.1) can be used by a surgeon to control a patient-side cart54(such as patent-side cart22inFIG.1) during a minimally invasive procedure. The patient-side cart54can use an imaging device, such as a stereoscopic endoscope, to capture images of a surgical site and output the captured images to a computer processor located on an electronics cart56(such as the electronics cart24inFIG.1). The computer processor typically includes one or more data processing boards purposed for executing computer readable code stored in a non-volatile memory device of the computer processor. In one aspect, the computer processor can process the captured images in a variety of ways prior to any subsequent display. For example, the computer processor can overlay the captured images with a virtual control interface prior to displaying the combined images to the surgeon via the surgeon's console52.

Additionally or in the alternative, the captured images can undergo image processing by a computer processor located outside of electronics cart56. In one aspect, teleoperated surgical system50includes an optional computer processor58(as indicated by dashed line) similar to the computer processor located on electronics cart56, and patient-side cart54outputs the captured images to computer processor58for image processing prior to display on the surgeon's console52. In another aspect, captured images first undergo image processing by the computer processor on electronics cart56and then undergo additional image processing by computer processor58prior to display on the surgeon's console52. Teleoperated surgical system50can include an optional display60, as indicated by dashed line. Display60is coupled with the computer processor located on the electronics cart56and with computer processor58, and captured images processed by these computer processors can be displayed on display60in addition to being displayed on a display of the surgeon's console52.

FIG.5is a perspective view of a patient-side cart500of a minimally invasive teleoperated surgical system, in accordance with embodiments of the present invention. The patient-side cart500includes one or more support assemblies510. A surgical instrument manipulator512is mounted at the end of each support assembly510. Additionally, each support assembly510can optionally include one or more unpowered, lockable setup joints that are used to position the attached surgical instrument manipulator512with reference to the patient for surgery. As depicted, the patient-side cart500rests on the floor. In other embodiments, operative portions of the patient-side cart can be mounted to a wall, to the ceiling, to the operating table526that also supports the patient's body522, or to other operating room equipment. Further, while the patient-side cart50) is shown as including four surgical instrument manipulators512, more or fewer surgical instrument manipulators512may be used.

A functional minimally invasive teleoperated surgical system will generally include a vision system portion that enables a user of the teleoperated surgical system to view the surgical site from outside the patient's body522. The vision system typically includes a camera instrument528for capturing video images and one or more video displays for displaying the captured video images. In some surgical system configurations, the camera instrument528includes optics that transfer the images from a distal end of the camera instrument528to one or more imaging sensors (e.g., CCD or CMOS sensors) outside of the patient's body522. Alternatively, the imaging sensor(s) can be positioned at the distal end of the camera instrument528, and the signals produced by the sensor(s) can be transmitted along a lead or wirelessly for processing and display on the one or more video displays. One example of a video display is the stereoscopic display on the surgeon's console in surgical systems commercialized by Intuitive Surgical, Inc., Sunnyvale, California.

Referring toFIG.5, mounted to each surgical instrument manipulator512is a surgical instrument520that operates at a surgical site within the patient's body522. Each surgical instrument manipulator512can be provided in a variety of forms that allow the associated surgical instrument to move with one or more mechanical degrees of freedom (e.g., all six Cartesian degrees of freedom, five or fewer Cartesian degrees of freedom, etc.). Typically, mechanical or control constraints restrict each manipulator512to move its associated surgical instrument around a center of motion on the instrument that stays stationary with reference to the patient, and this center of motion is typically located at the position where the instrument enters the body.

In one aspect, surgical instruments520are controlled through computer-assisted teleoperation. A functional minimally invasive teleoperated surgical system includes a control input that receives inputs from a user of the teleoperated surgical system (e.g., a surgeon or other medical person). The control input is in communication with one or more computer-controlled teleoperated actuators, such as one or more motors to which surgical instrument520is coupled. In this manner, the surgical instrument520moves in response to a medical person's movements of the control input. In one aspect, one or more control inputs are included in a surgeon's console such as surgeon's console16shown atFIG.2. A surgeon can manipulate control inputs36of surgeon's console16to operate teleoperated actuators of patient-side cart500. The forces generated by the teleoperated actuators are transferred via drivetrain mechanisms, which transmit the forces from the teleoperated actuators to the surgical instrument520.

Referring toFIG.5, in one aspect, a surgical instrument520and a cannula524are removably coupled to manipulator512, with the surgical instrument520inserted through the cannula524. One or more teleoperated actuators of the manipulator512move the surgical instrument512as a whole. The manipulator512further includes an instrument carriage530. The surgical instrument520is detachably connected to the instrument carriage530. In one aspect, the instrument carriage530houses one or more teleoperated actuators inside that provide a number of controller motions that the surgical instrument520translates into a variety of movements of an end effector on the surgical instrument520. Thus the teleoperated actuators in the instrument carriage530move only one or more components of the surgical instrument520rather than the instrument as a whole. Inputs to control either the instrument as a whole or the instrument's components are such that the input provided by a surgeon or other medical person to the control input (a “master” command) is translated into a corresponding action by the surgical instrument (a “slave” response).

In an alternate embodiment, instrument carriage530does not house teleoperated actuators. Teleoperated actuators that enable the variety of movements of the end effector of the surgical instrument520are housed in a location remote from the instrument carriage530, e.g., elsewhere on patient-side cart500. A cable-based force transmission mechanism or the like is used to transfer the motions of each of the remotely located teleoperated actuators to a corresponding instrument-interfacing actuator output located on instrument carriage530. In some embodiments, the surgical instrument520is mechanically coupled to a first actuator, which controls a first motion of the surgical instrument such as longitudinal (z-axis) rotation. The surgical instrument520is mechanically coupled to a second actuator, which controls second motion of the surgical instrument such as two-dimensional (x, y) motion. The surgical instrument520is mechanically coupled to a third actuator, which controls third motion of the surgical instrument such as opening and closing or a jaws end effector.

FIG.6Ais a side view of a surgical instrument520, which includes a distal portion650and a proximal control mechanism640coupled by an elongate tube610having an elongate tube centerline axis611. The surgical instrument520is configured to be inserted into a patient's body and is used to carry out surgical or diagnostic procedures. The distal portion650of the surgical instrument520can provide any of a variety of end effectors654, such as the forceps shown, a needle driver, a cautery device, a cutting tool, an imaging device (e.g., an endoscope or ultrasound probe), or the like. The surgical end effector654can include a functional mechanical degree of freedom, such as jaws that open or close, or a knife that translates along a path. In the embodiment shown, the end effector654is coupled to the elongate tube610by a wrist652that allows the end effector to be oriented relative to the elongate tube centerline axis611. Surgical instrument520can also contain stored (e.g., on a semiconductor memory device660associated with the instrument) information, which may be permanent or may be updatable by a surgical system configured to operate the surgical instrument520. In some embodiments, a semiconductor memory device660associated with an instrument tracking device662is associated with the instrument520. In some embodiments, the instrument tracking device662includes a radio frequency identification device (RFID) device. Accordingly, the surgical system may provide for either one-way or two-way information communication between the surgical instrument520and one or more components of the surgical system.

FIG.6Bis an illustrative drawing representing an RFID tracking device662associated with an instrument520in accordance with some embodiments. The RFID device662includes a storage device660that includes an information structure664that associates device identifier information666with remaining device lifetime information668. The RFID device662includes a transceiver circuitry670to send and receive information wirelessly. The RFID device662includes control logic circuitry672to control the storage device660and the transceiver670.

FIG.7is a perspective view of surgical instrument manipulator512. Instrument manipulator512is shown with no surgical instrument installed. Instrument manipulator512includes an instrument carriage530to which a surgical instrument (e.g., surgical instrument520) can be detachably connected. Instrument carriage530houses a plurality of teleoperated actuators. In some embodiments, an RFID reader535is disposed upon the instrument manipulator512in a location to read the contents of the storage device660within an RFID device662associated with an instrument attached to the manipulator512. Each teleoperated actuator includes an actuator output705. When a surgical instrument is installed onto instrument manipulator512, one or more instrument inputs (not shown) of an instrument proximal control mechanism (e.g., proximal control mechanism640atFIG.6) are mechanically coupled with corresponding actuator outputs705. In one aspect, this mechanical coupling is direct, with actuator outputs705directly contacting corresponding instrument inputs. In another aspect, this mechanical coupling occurs through an intermediate interface, such as a component of a drape configured to provide a sterile barrier between the instrument manipulator512an associated surgical instrument.

In one aspect, movement of one or more instrument inputs by corresponding teleoperated actuators results in a movement of a surgical instrument mechanical degree of freedom. For example, in one aspect, the surgical instrument installed on instrument manipulator512is surgical instrument520, shown atFIG.6. Referring toFIG.6, in one aspect, movement of one or more instrument inputs of proximal control mechanism640by corresponding teleoperated actuators rotates elongate tube610(and the attached wrist652and end effector654) relative to the proximal control mechanism640about elongate tube centerline axis611. In another aspect, movement of one or more instrument inputs by corresponding teleoperated actuators results in a movement of wrist652, orienting the end effector654relative to the elongate tube centerline axis611. In another aspect, movement of one or more instrument inputs by corresponding teleoperated actuators results in a movement of one or more moveable elements of the end effector654(e.g., a jaw member, a knife member, etc.). Accordingly, various mechanical degrees of freedom of a surgical instrument installed onto an instrument manipulator512can be moved by operation of the teleoperated actuators of instrument carriage530.

Annotating a Recorded Video

FIG.8shows a schematic diagram of an exemplary surgical planning tool800. In one aspect, surgical planning tool800includes a teleoperated surgical system850in data communication with an electronic medical device record database830and an instrument inventory management system860. Teleoperated surgical system850shown here is similar to teleoperated surgical system850shown atFIG.4. In one aspect, electronic medical record database830includes the medical records of patients that have undergone treatment at a particular hospital. Database830and instrument inventory management system860can be implemented on a server located on-site at the hospital. The medical record entries contained in the database830and management system860can be accessed from hospital computers through an intranet network. Alternatively, database830and management system860can be implemented on a remote server located off-site from the hospital, e.g., using one of a number of cloud data storage services. In this case, medical record entries of database830and management system860are stored on the cloud server, and can be accessed by a computer with internet access.

In one aspect, a surgical procedure is performed on a first patient using teleoperated surgical system850. An imaging device associated with teleoperated surgical system850captures images of the surgical site and displays the captured images as frames of a video on a display of surgeon's console52. In one aspect, a medical person at surgeon's console52highlights or annotates certain patient anatomy shown in the displayed video using an input device of surgeon's console52. An example of such an input device is control input36shown atFIG.2, which is coupled to a cursor that operates in conjunction with a graphic user interface overlaid onto the displayed video. The graphic user interface can include a QWERTY keyboard, a pointing device such as a mouse and an interactive screen display, a touch-screen display, or other means for data or text entry. Accordingly, the medical person can highlight certain tissue of interest in the displayed image or enter a text annotation.

In one aspect, the surgical site video is additionally displayed on a display located on electronics cart56. In one aspect, the display of electronics cart is a touch-screen user interface usable by a medical person to highlight and annotate certain portions of patient anatomy shown on an image that is displayed for viewing on the display on the electronics cart. A user, by touching portions of patient anatomy displayed on the touch-screen user interface, can highlight portions of the displayed image. Additionally, a graphic interface including a QWERTY keyboard can be overlaid on the displayed image. A user can use the QWERTY keyboard to enter text annotations.

In one aspect, the surgical site video captured by the imaging device associated with teleoperated surgical system850is recorded by the teleoperated surgical system850, and stored on database830, in addition to being displayed in real time or near real time to a user. Highlights and/or annotations associated with the recorded video that were made by the user can also be stored on database830. In one aspect, the highlights made by the user are embedded with the recorded video prior to its storage on database830. At a later time, the recorded video can be retrieved for viewing. In one aspect, a viewer of the recorded video can select whether the highlights are displayed or suppressed from view. Similarly, annotations associated with the recorded video can also be stored on database830. In one aspect, the annotations made by the user are used to tag the recorded video, and can be used to provide as a means of identifying the subject matter contained in the recorded video. For example, one annotation may describe conditions of a certain disease state. This annotation is used to tag the recorded video. At a later time, a person desiring to view recorded procedures concerning this disease state can locate the video using a key word search.

Retrieval of Stored Video

In some cases, it is desirable for a medical person to be able to view video recordings of past surgical procedures performed on a given patient. In one aspect, a patient who previously underwent a first surgical procedure to treat a medical condition subsequently requires a second surgical procedure to treat recurrence of the same medical condition or to treat anatomy located nearby to the surgical site of the first surgical procedure. In one aspect, the surgical site events of the first surgical procedure were captured in a surgical site video recording, and the video recording was archived in database830as part of the patient's electronic medical records. Prior to performing the second surgical procedure on the patient, a medical person can perform a search of database830to locate the video recording of the patient's earlier surgical procedure.

In some cases, it is desirable for a medical person planning to perform a surgical procedure on a patient to be able to view video recordings of similar surgical procedures performed on persons having certain characteristics similar to the patient. In one aspect, surgical site video recordings of surgical procedures can be tagged with metadata information such as the patient's age, gender, body mass index, genetic information, type of procedure the patient underwent, etc., before each video recording is archived in database830. In one aspect, the metadata information used to tag a video recording is automatically retrieved from a patient's then-existing medical records, and then used to tag the video recording before the video recording is archived in database830. Accordingly, prior to performing a medical procedure on a patient, a medical person can search database830for video recordings of similar procedures performed on patients sharing certain characteristics in common with the patient. For example, if the medical person is planning to use teleoperated surgical system850to perform a prostatectomy on a 65 year-old male patient with an elevated body mass index using, the medical person can search database830for surgical site video recordings of prostatectomies performed using teleoperated surgical system850on other males of similar age and having similarly elevated body mass index.

In one aspect, a video recording of a surgical procedure is communicated by database830to an optional personal computer820(as indicated by dashed line), and made available for viewing by a medical person who plans to perform a surgical procedure. Additionally or in the alternative, the video recording of the earlier surgical procedure can be communicated by database830to teleoperated surgical system850, and made available for viewing preoperatively or intraoperatively. In one aspect, the video recording is displayed by teleoperated surgical system850on a display located on surgeon's console52. In another aspect, the video recording of the first surgical procedure is displayed on a display located on electronics cart56.

Cloud-Based Video Database

In one aspect, database830is implemented on a remote server using a cloud data storage service and is accessible by multiple health care providers. Referring toFIG.8, as shown by dashed line, surgical planning tool800optionally includes teleoperated surgical system850(as indicated by dashed line) and personal computer840(as indicated by dashed line). In one aspect, teleoperated surgical system850is similar to teleoperated surgical system850and personal computer840is similar to personal computer820, except that teleoperated surgical system850and personal computer820are located at a first health care provider and teleoperated surgical system850and personal computer840are located at a second health care provider. In one aspect, a first patient requires surgical treatment of a medical condition, and undergoes a surgical procedure using teleoperated surgical system850at the first health care provider. A video recording of the surgical procedure is archived on database830. At a later time, a second patient requires surgical treatment of the same medical condition, and plans to receive surgical treatment using teleoperated surgical system850at the second health care provider. Prior to performing the surgical procedure on the second patient, a medical person accesses database830through a secure internet connection and searches database830for surgical site video recordings of similar procedures. In one aspect, the medical person treating the second patient is able to retrieve from database830the video recording of first patient's surgical procedure, without acquiring knowledge of the identity of the first patient. In this manner, the privacy of the first patient is maintained. In one aspect, the video recording of the first patient's surgical procedure includes highlights and/or annotations made by the medical person who treated the first patient.

Computer Based Pattern Matching and Analysis

Surgical planning tool800can includes a pattern matching and analysis algorithm implemented in the form of computer executable code. In one aspect, the pattern matching and analysis algorithm is stored in a non-volatile memory device of surgical planning tool800, and is configured to analyze the video recordings archived in database830. As discussed previously, each of the video recordings archived in database830can be tagged and/or embedded with certain metadata information. This metadata information can include patient information such as patient age, gender, and other information describing the patient's health or medical history. Additionally, as discussed previously, the metadata information can include highlights or annotations made by a medical person. In one aspect, these highlights and annotations are embedded with the video recording and archived together with the video in database830.

In one aspect, pattern matching and analysis algorithm includes an image analysis component that identifies patterns in shapes and colors that are shared amongst multiple video recordings stored on database830. The pattern matching and analysis algorithm then reviews the tagged metadata associated with this subset of video recordings to determine whether any words or phrases are frequently associated with videos within this subset. These analyses performed by pattern matching and analysis algorithm can be used to assist medical persons in making determinations about patient anatomy, preferred surgical approaches, disease states, potential complications, etc.

A Method of Using a Surgical Planning Tool

FIG.9shows a method900of using a surgical planning tool. In one aspect, the surgical planning tool is similar to surgical planning tool800atFIG.8. At910, a fact or characteristic describing a medical patient, e.g., a medical condition suffered by a patient, is received by a medical device. Medical device can receive this fact or circumstance via a user interface located on a teleoperated surgical system (e.g., teleoperated surgical system10atFIG.1or teleoperated surgical system50atFIG.4), or alternatively, through a personal computer similar to personal computer820atFIG.2. At920, the medical device uses the fact or characteristic received at910to retrieve at least one relevant video recording of a surgical procedure from a medical device database. At930, the medical device uses the video recordings to determine surgical planning information. In one aspect, the surgical planning information includes the types of instruments used in the recorded procedure. At940, the medical device displays to a user the surgical planning information determined at930.

A Method of Surgical Instrument Inventory Management Based upon Observation of Surgical Instrument Usage

Chart 1 identifies several example surgical instruments and the corresponding incremental lifetime degradation resulting from routine in-surgery use, irregular in-surgical use, and sterilization in accordance with some embodiments.

CHART 1LifetimeLifetimeTotalTotaldecrement-decrement-LifetimeLifetimesLifetimes forper routinePer irregulardecrement-Instrumentfor newrefurbishedin-surgeryin-surgeryPerNameinstrumentinstrumentusageusagesterilizationSuturinginstrScalpelinstrCauterizinginstrScissorsinstrOther?

In some embodiments, an instrument can be refurbished to add lifetimes i.e. additional surgical uses. However a refurbished instrument may start with fewer lifetimes than a new instrument. In some embodiments, the number of surgeries in which a surgical instrument can be used varies with the use of the instrument. Irregular use can result in more rapid degradation of the instrument. For example, an irregular use of a scissors instrument to perform suturing can result in accelerated degradation. Moreover, a sterilization of an instrument can involve high temperatures and chemical treatment, which can result in instrument wear.

FIG.10is an illustrative drawing representing storage atlas in a computer readable storage device1004in accordance with some embodiments. The storage atlas1002includes first information structures1006that indicate instances of previously performed surgical procedures. Second information structures1008associate surgical procedures with surgical activities performed and surgical instruments used during the surgical procedure. Third information structures1010associate surgical activities with surgical instrument actuation states. Fourth information structures1012associate routine surgical instrument use, lifetime decrements with surgical instrument actuation states. Fifth information structures1014associate non-routine surgical instrument use, lifetime decrements with surgical instrument actuation states. Sixth information structures1016associate surgical instrument sterilization, lifetime decrements with sterilization events. Seventh information structures1018that associate video images of surgical scenes recorded during surgical procedures with instrument actuation states recorded during the surgical procedures. In some embodiments, the various information structures1004-1018are evaluated to produce eighth information structures1020that associate surgical activities with surgical instrument activity states and with surgical instrument lifetime decrements.

FIG.11is an illustrative drawing representing an example instance of the seventh information structure1018included within the atlas1002in the storage device1004, which associates recorded video images from an individual surgery with corresponding surgical instrument actuator state information in accordance with some embodiments. In one aspect, video images of patient anatomy structures and instruments used to operate upon those anatomical structures during a surgery and corresponding surgical instrument actuator states are recorded and time stamped (t1, t2 . . . tn) during a surgery to produce a chronological record of surgical activities and corresponding surgical instrument actuator states during the surgical procedure. The time stamps are used to temporally align video images with surgical instrument actuator states.

During a surgery, a user may annotate the video recording and the surgical instrument actuation state recording with metadata that indicate corresponding surgical activity. The annotation may include one or more of or a combination of written notes tagged to video information and/or surgical instrument actuation state information, coloring or highlighting (e.g., telestration) of images in the video recordings, for example. The annotations may be time stamped for use to temporally align them with corresponding video recording information and corresponding recorded surgical instrument state information.

During a teleoperated surgical procedure, a surgical activity can which involves use of at least one surgical instrument. During the surgical activity, the surgical instrument is operated, under surgeon control, in one or more actuator states. Operation of the surgical instrument in support of the surgical activity in the one or more surgical states can result in degradation of the instrument's efficacy for its intended use. As explained more fully below, to keep track of this reduction in efficacy, a record indicating the instrument's remaining lifetime is modified, e. g., a lifetime count may be decremented in response to the instrument's usage in the surgical activity.

FIGS.12A-12Care illustrative drawings showing an example surgical instrument1202and an actuator assembly1203in which the surgical instrument is shown in three different example operational states in accordance with some embodiments. The example instrument1202includes a jaw end effector1204that can transition between open and closed states and a continuum of partially opened/partially closed states in between. The example instrument1202also includes a two degree of freedom (2-dof) wrist1206that can move between different two-dimensional (x, y) positional states. The example actuator assembly1203includes a first actuator1208, which in some embodiments includes a jaw motor (JM) used to actuate the jaw end effector1204. The example actuator assembly1203includes a second actuator1210, which in some embodiments includes a wrist motor (WM) used to actuate the wrist1206. During a surgery, the surgical instrument1202may transition through multiple actuation states corresponding to different activities during a surgical procedure. As represented inFIG.12A, for example, a surgical procedure may involve a first surgical activity in which the first actuator1208(the JM) disposes the jaw end effector1204to a fully open state and the second actuator1210the (WM) disposes the wrist1206to a first positional state (x1, y1). As represented inFIG.12B, for example, the surgical procedure may involve a second surgical activity in which the first actuator1208transitions the jaw end effector1204to a fully closed state and the second actuator1210transitions the wrist1206to a second positional state (x2, y2). As represented inFIG.12C, for example, the surgical procedure may involve a third surgical activity in which the first actuator1208disposes the jaw end effector1104in a partially open/partially closed state and the second actuator1210transitions the wrist1206to a third positional state (x3, y3).

FIG.13is an illustrative drawing representing an example instance1020iof the eighth information structure1020of the atlas1002stored in the computer readable storage device1004in accordance with some embodiments. The example eighth information structure instance1020iassociates surgical activities during a selected surgical procedure with surgical instrument actuator states that are possible during the activity. The example eighth information structure instance1020iassociates the possible surgical instrument actuator states with corresponding surgical instrument lifetime decrements.

A first column of the example eighth information structure instance1020iindicates a list of surgical activities, A1, A2, A3. . . AN to be performed during the example surgical procedure. A second column of the information structure instance1020iindicates instruments I1, I2, I3to be used during surgical activities. A third column indicates first possible surgical instrument actuation states that can occur during corresponding surgical activities. A fourth column indicates second possible surgical instrument actuation states that can occur during corresponding surgical activities. Referring to the third and fourth columns, the first instrument I1can operate in either of two possible surgical instrument actuator (SIA) states, SIA11and SIA12. The second instrument I2can operate in only one surgical instrument actuator state, SIA21. The third instrument I3can operate in either of two possible surgical instrument actuator states, SIA31and SIA32. A fifth column indicates lifetime decrements corresponding to corresponding first possible surgical instrument actuation states. A sixth column indicates lifetime decrements corresponding to corresponding second possible surgical instrument actuation states. Referring to the fifth and sixth columns, the first instrument I1operating in the first instrument's first actuator state SIA11is associated with the first instrument's first decrement count I1C1, and the first instrument I1operating in the first instrument's second actuator state SIA21is associated with the first instrument's second decrement count I1C2. The second instrument I2operating in the second instrument's first actuator state SIA21is associated with the second instrument's first decrement count I2C1, and the second instrument I2operating in the second instrument's second actuator state SIA22is associated with the second instrument's second decrement count I2C2. The third instrument I3operating in the third instrument's first actuator state SIA31is associated with the third instrument's first decrement count I3C1, and the third instrument I3operating in the third instrument's second actuator state SIA32is associated with the third instrument's second decrement count I3C2.

It is noted that some surgical activities may involve use of an instrument such as an endoscope, for example, which is not worn down or degraded as result of its use. Also, some instruments such as instrument I2, are operable in only in a single routine actuator state, and are not operable in an alternative second actuator state, and therefore, is associated with only a single category of lifetime decrement. Moreover, in some embodiments, a lifetime decrement associated with an instrument actuator state is determined as a function of energy use over time during an instrument actuator state, and therefore the lifetime decrement for such instrument can be variable depending upon energy usage. In accordance with some embodiments, the more energy is used, the larger the lifetime decrement. For example, operation of an instrument at higher speed during an actuator state can result in use more energy than operation of the same instrument in the same actuator state at a lower speed.

Referring to the first row of the example information structure instance1020iofFIG.13, for example, during activity A1, the first instrument I1can be operated in the first instrument's first actuation state SIA11, which results in decrementing the first instrument's remaining lifetime668by the first decrement count I1C1. Alternatively, during activity A1, the first instrument I1can be operated in the first instrument's second actuation state SIA12, which results in decrementing the first instrument's remaining lifetime668by the second decrement count I1C2.

Referring to the second row of the example information structure instance1029iofFIG.13, for example, during the surgical, the first activity A2of the surgery can involve the second instrument I2operated in the second instrument's first actuation state SIA21, which results in decrementing the second instrument's remaining lifetime668by the first decrement count I2C1. It is noted that there is no second instrument second actuation state in this example. Referring to the Nth row of the example information structure instance1020iofFIG.13, for example, during the Nth activity AN, the third instrument I3can be operated in the third instrument's first actuation state SIA31, which results in decrementing the third instrument's remaining lifetime668by the first decrement count I3C1. Alternatively, during activity AN, the third instrument I3can be operated in the third instrument's second actuation state SIA32, which results in decrementing the third instrument's remaining lifetime668by the second decrement count I3C2.

FIG.14is an illustrative flow diagram representing a process1402to configure processor58to determine surgical instrument wear incurred during a surgical procedure in accordance with some embodiments. In block1404, a surgical instrument identification is received at an input to a computer processing system associated with the electronics cart56. The surgical instrument identification includes information to identify each individual instrument slated for use in a surgical procedure and corresponding remaining lifetime information identified instruments. In block1406, an identification of a surgical procedure is received at an input of the computer processing system associated with the electronics cart56. In block1408, information included within the atlas1002within information structures1008,1010,1012,1014and1016is used to produce an instance of the eighth information structure1020associating surgical activities with surgical instrument activity states and with surgical instrument lifetime decrements.

During performance of the identified surgical procedure, block1410tracks operational state of a surgical instrument actuator. In decision block1412, a determination is made as to whether a current instrument actuator state matches an actuator state that is associated with a lifetime decrement for the instrument. In response to no match, control loops back to block1410and tracking continues. In response to a match, block1414decrements the identified instrument's remaining lifetime668based upon a lifetime decrement associated with the matching instrument actuator state. Decision block1416determines whether the surgical procedure is done. In response to a determination that the surgical procedure is not yet done, control next flows back to block1410, which continues to track surgical instrument actuator state based upon other identified actuator state transition information, for example. In response to a determination that the surgical procedure is done, block1418uses the RFID reader535to read instrument identifier information666and corresponding remaining lifetime information668from the storage device660within the RFID662associated with the instrument and sends remaining lifetime information for the identified instrument to the inventory management system860.

Assuming for example, that the process1402ofFIG.14that the is performed for using the example information structure instance1020iofFIG.13, during performance of the surgical procedure, block1410tracks operational state of surgical instrument actuators for each of instruments I1, I2and I3. In decision block1412, a determination is made as to whether a current instrument actuator state of any one or more of the three surgical instrument actuators matches an actuator state that is associated with a lifetime decrement for the corresponding instrument. In response to no match, control loops back to block1410and tracking continues. In response to a match, block1414decrements an identified instrument's remaining lifetime668based upon a lifetime decrement associated with the matching instrument actuator state. For example, while tracking during surgical activity A1, the decision block1412determines whether a current actuator state of instrument I1matches either of actuator state SIA11or SAI21. In response to a match between the actuator state of I1and SIA11, a lifetime668for I1is decremented by I1C1. In response to a match between the actuator state of I1and SIA21, a lifetime668for I1is decremented by I1C2. Control next flows to decision block1416, which determines whether there are additional surgical states to be performed. In this example, surgical states A2through AN are to be performed after surgical state A1. Accordingly, control flows back to block1410, which continues to track surgical instrument actuator state during surgical activities A2through AN based upon other identified actuator state transition information, for example. Upon completion of all surgical activities A1through AN, block1418configures the processor to send an updated lifetime information for the identified instrument to the inventory management system860.

FIG.15is an illustrative drawing representing a sterilization chamber1502with a surgical instrument520disposed inside it in accordance with some embodiments. The surgical instrument is associated with an RFID device662and a storage device660. In operation, following sterilization of an instrument prior to use of the instrument in a next surgery, the RFID reader535associated with the sterilization chamber1502reads instrument identifier information666and corresponding remaining lifetime information668from memory660for transmission to the inventory management system860.

Although illustrative embodiments have been shown and described, a wide range of modification, change and substitution is contemplated in the foregoing disclosure and in some instances, some features of the embodiments may be employed without a corresponding use of other features. For example, in some embodiments, the processor58is coupled to a memory device such as storage device1004that includes an instruction set executable on the processor58to cause the processor58to perform operations. In some embodiments, the operations include providing in a first information structure in the memory device that associates a surgical instrument identifier with remaining useful lifetime of the identified surgical instrument. The operations further include tracking surgical instrument actuator state of the identified surgical instrument during performance of a surgical procedure. The operations further include reducing the associated remaining useful lifetime of the identified surgical instrument by a surgical instrument lifetime reduction amount in response to the tracked surgical instrument actuator state matching a surgical instrument wear-down actuation state during the performance of the surgical procedure.

One of ordinary skill in the art would recognize many variations, alternatives, and modifications. Thus, the scope of the disclosure should be limited only by the following claims, and it is appropriate that the claims be construed broadly and in a manner consistent with the scope of the embodiments disclosed herein.