Patent Publication Number: US-2021186460-A1

Title: Method of spatially locating points of interest during a surgical procedure

Description:
BACKGROUND 
     During a minimally invasive surgery (MIS) an intraoperative ultrasound probe can be used to provide two-dimensional (2D) cross-sectional views of a surgical site. During MIS, a clinician typically holds the ultrasound probe, either with a surgical grasper tool or with the ultrasound probe being part of its own dependent tool shaft. The ultrasound probe is placed in contact with a tissue region of interest and moved about so that the 2D cross-sectional view image of the surgical site is seen on an ultrasound display. The ultrasound display is typically distinct from an endoscope display which is showing images captured from an endoscope being used to directly observe the surgical site. The endoscope display may be used to direct manipulation of the ultrasound probe. 
     The 2D cross-sectional views can reveal information about the state of the structures below the tissue surface at/or adjacent the surgical site. Typically, a clinician manipulates the ultrasound probe and mentally notes the structures at/or adjacent the surgical site. After the clinician removes the ultrasound probe to begin or continue the surgical procedure, the clinician must remember location of the structures at/or adjacent the surgical site. If during the surgical procedure the clinician requires a reminder of the 2D cross-sectional views, the surgical procedure is paused and the ultrasound probe is reactivated to reacquire 2D cross-sectional views and refresh a clinician&#39;s memory. This pausing of the surgical procedure can cause a disruption in the flow of the surgical procedure. This disruption in flow of the surgical procedure may encourage a clinician not to pause the surgical procedure to reacquire the 2D cross-sectional views with the ultrasound probe. By not pausing during a surgical procedure to reacquire the 2D cross-sectional views, quality of decision making during a surgical procedure may be reduced. 
     There is a need to allow a clinician to view ultrasound images during a surgical procedure at points of interest during the surgical procedure. By identifying points of interest during a surgical procedure, surgical decision making can be improved. 
     SUMMARY 
     In an aspect of the present disclosure, a method of visualizing a surgical site includes scanning a surgical site with an ultrasound system, marking a first area or point of interest within cross-sectional views of the surgical site with a first tag, and viewing the surgical site with a camera on a second display. The second display displaying a first indicia representative of the first tag. 
     In aspects, scanning the surgical site with the ultrasound system includes inserting an ultrasound probe into a body cavity of a patient. Displaying the first indicia representative of the first tag may include displaying information relevant to the first area or point of interest on the second display. The method may include toggling the first indicia to display information relevant to the first area or point of interest on the second display. 
     In some aspects, viewing the surgical site with the camera on the second display incudes a control unit locating the first tag within images captured by the first camera. Locating the first tag within images captured by the camera may include determining a depth of the first tag within the surgical site from multiple images captured by the camera. Locating the first tag within images captured by the camera may include using pixel-based identification of images from the camera to determine the location of the first tag within the images captured by the camera. 
     In particular aspects, the method includes freezing the first display such that a particular cross-sectional view of the surgical site is viewable on the first display. Viewing the surgical site with the camera on the second display may include removing distortion from the images of the surgical site captured with the camera before displaying the images of the surgical site on the second display. 
     In certain aspects, the method includes marking a second area or point of interest within the cross-sectional views of the surgical site with a second tag and viewing a second indicia representative of the second tag on the second display. Viewing the second indicia representative of the second tag includes displaying information relevant to the second area or point of interest on the second display. The method may include toggling the second indicia to display information relevant to the second area or point of interest on the second display. The method may also include toggling the first indicia to display information relevant to the first area or point of interest on the second display independent of toggling the second indicia. 
     In another aspect of the present disclosure, a surgical system includes an ultrasound system, an endoscopic system, and a processing unit. The ultrasound system includes an ultrasound probe and an ultrasound display. The ultrasound probe is configured to capture cross-sectional views of a surgical site. The ultrasound display is configured to display the cross-sectional views of the surgical site captured by the ultrasound probe. The endoscopic system includes an endoscope and an endoscope display. The endoscope has a camera that is configured to capture images of the surgical site. The endoscope display is configured to display the images of the surgical site captured by the camera. The processing unit is configured to receive a location of a first area or point of interest within a cross-sectional view of the surgical site and to display a first indicia representative of the first area or point of interest on the second display. 
     In aspects, the ultrasound display is a touchscreen display that is configured to receive a tag that is indicative of the location of the first area or point of interest within the cross-sectional view of the surgical site. The processing unit may be configured to remove distortion from images of the surgical site captured with the camera before displaying the images of the surgical site on the second display. The processing unit may be configured to locate the first area or point of interest within images captured by the camera using pixel-based identification of images from the camera. 
     Further, to the extent consistent, any of the aspects described herein may be used in conjunction with any or all of the other aspects described herein. 
    
    
     
       BRIEF DESCRIPTION OF THE DRAWINGS 
       Various aspects of the present disclosure are described hereinbelow with reference to the drawings, which are incorporated in and constitute a part of this specification, wherein: 
         FIG. 1  is a perspective view of an ultrasound system in accordance with the present disclosure including an ultrasound probe, a positional field generator, a processing unit, an ultrasound display, and an endoscope display; 
         FIG. 2  is a cut-away of the detail area shown in  FIG. 1  illustrating the ultrasound probe shown in  FIG. 1  and an endoscope within a body cavity of a patient; 
         FIG. 3  is view of the ultrasound display of  FIG. 1  illustrating a two-dimensional cross-sectional image of a surgical site; and 
         FIG. 4  is view of the endoscope display of  FIG. 1  illustrating an image of the surgical site and a distal portion of a surgical instrument within the surgical site. 
     
    
    
     DETAILED DESCRIPTION 
     Embodiments of the present disclosure are now described in detail with reference to the drawings in which like reference numerals designate identical or corresponding elements in each of the several views. As used herein, the term “clinician” refers to a doctor, a nurse, or any other care provider and may include support personnel. Throughout this description, the term “proximal” refers to the portion of the device or component thereof that is closest to the clinician and the term “distal” refers to the portion of the device or component thereof that is farthest from the clinician. 
     Referring now to  FIG. 1 , a surgical system  1  provided in accordance with the present disclosure includes an ultrasound imaging system  10  and an endoscopic system  30 . The ultrasound imaging system  10  includes a processing unit  11 , an ultrasound display  18 , and an ultrasonic probe  20 . 
     The ultrasound imaging system  10  is configured to provide 2D cross-sectional views or 2D image slices of a region of interest within a body cavity of a patient “P” on the ultrasound display  18 . A clinician may interact with the ultrasound imaging system  10  and an endoscope  36 , which may include a camera, to visualize surface and subsurface portions of a surgical site “S” of the patient “P” during a surgical procedure as detailed below. 
     The ultrasound probe  20  is configured to generate 2D cross-sectional views of the surgical site “S” from a surface of a body cavity of the patient “P” and/or may be inserted through an opening, either a natural opening or an incision, to be within the body cavity adjacent the surgical site “S”. The processing unit  11  receives 2D cross-sectional views of the surgical site “S” and transmits a representation of the 2D cross-sectional views on the ultrasound display  18 . 
     The endoscopic system  30  includes a control unit  31 , an endoscope  36 , and an endoscope display  38 . With additional reference to  FIG. 2 , the endoscope  36  may include a camera  33  and a sensor  37  which are each disposed on or in a distal portion of the endoscope  36 . The camera  33  is configured to capture images of the surgical site “S” which are displayed on the endoscope display  38 . The control unit  31  is in communication with the camera  33  and is configured to transmit images captured by the camera  33  to the endoscope display  38 . The control unit  31  is in communication with the processing unit  11  and may be integrated with the processing unit  11 . 
     Referring to  FIGS. 1-4 , the use of the ultrasound system  10  and the endoscopic system  30 , to image the surgical site “S”, is described in accordance with the present disclosure. Initially, the ultrasound probe  20  is positioned adjacent the surgical site “ 5 ”, either within or outside of a body cavity of the patient, to capture 2D cross-sectional views of the surgical site “S”. The ultrasound probe  20  is manipulated to provide 2D cross-sectional views of the areas or points of interest at or adjacent the surgical site “S”. It will be appreciated that the entire surgical site “S” is scanned while the ultrasound probe  20  is within the view of the camera  33  of the endoscope  36  such that the position of ultrasound probe  20  can be associated with the 2D cross-sectional views of the surgical site “S” as the 2D cross-sectional views are acquired. While the surgical site “S” is scanned, the processing unit  11  and/or the control unit  31  record the 2D cross-sectional views and associate the 2D cross-sectional views with the position of the ultrasound probe  20  within the surgical site “S” at the time each 2D cross-sectional view was acquired. 
     When the endoscope  36  views the surgical site “S”, the camera  33  of the endoscope  36  captures real-time images of the surgical site “S” for viewing on the endoscope display  38 . After the surgical site “S” is scanned with the ultrasound probe  20 , other surgical instruments, e.g., a surgical instrument in the form of a grasper or retractor  46 , may be inserted through the same or a different opening from the endoscope  36  to access the surgical site “S” to perform a surgical procedure at the surgical site “S”. 
     As detailed below, the 2D cross-sectional views of the surgical site “S” recorded during the scan of the surgical site “S” are available for view by the clinician during the surgical procedure. As the camera  33  captures real-time images, the images are displayed on the endoscope display  38 . The clinician may select an area or point of interest of the surgical site “S” to review on the endoscope display  38 . When the an area or point of interest is selected on the endoscope display  38 , the control unit  31  determines the position of the area or point of interest within the surgical site “S” and sends a signal to the processing unit  11 . The processing unit  11  receives the signal from the control unit  31  and displays a recorded 2D cross-sectional view taken when the ultrasound probe  20  was position at/or near the area or point of interest during the scan of the surgical site “S”. The recorded 2D cross-sectional view can be a fixed image or can be a video clip of the area or point of interest. 
     When the recorded 2D cross-sectional view is a video clip of the area or point of interest the video clip may have a duration of about  1  second to about  10  seconds. The duration of the video clip may be preset or may be selected by the clinician before or during a surgical procedure. It is envisioned that the video clip may be looped such that it continually repeats. 
     To indicate the area or point of interest on the endoscope display  38 , the clinician may electronically or visually “mark” or “tag” the area or point of interest in the image on the endoscope display  38 . To electronically or visually mark the area or point of interest in the image on the endoscope display  38 , the clinician may use any known means including, but not limited to, touching the display with a finger or stylus; using a mouse, track pad, or similar pointing device to move an indicator on the endoscope display  38 ; using a voice recognition system; using an eye tracking system; typing on a keyboard; and/or a combination thereof. 
     To determine the position of the area or point of interest within the surgical site “S”, the control unit  31  processes the real-time images from the camera  33 . The control unit  31  may remove distortion from the real-time images to improve accuracy of determining the position of the area or point of interest. It is envisioned that the control unit  31  may utilize a pixel-based identification of the real-time images from the camera  33  to identify the location of the area or point of interest within the real-time images from the camera  33 . Additionally or alternatively, the location of the area or point of interest may be estimated from multiple real-time images from the camera  33 . Specifically, multiple camera images captured during movement of the endoscope  36  about the surgical site “S” can be used to estimate a depth of an area or point of interest within the surgical site “S”. 
     In embodiments, a stereoendoscope can be used to determine a depth of structures within the surgical site “S” based on the depth imaging capability of the stereoendoscope. The depth of the structures can be used to more accurately estimate the location of the area or point of interest in the images from the camera  33 . 
     With the location of the area or point of interest of the surgical site “S” determined, the processing unit  11  displays a 2D cross-sectional view, recorded during the scan of the surgical site “S” detailed above, that is associated with the identified location of the area or point of interest. The clinician can observe the 2D cross-sectional view to visualize subsurface structures at the area or point of interest. By visualizing the subsurface structures at the area or point of interest, the clinician&#39;s situational awareness of the area or point of interest is improved without the need for rescanning the area or point of interest with the ultrasound probe  20 . 
     Additionally or alternatively, during a surgical procedure, a clinician may rescan an area or point of interest within the surgical site “S” with the ultrasound probe  20  to visualize a change effected by the surgical procedure. It is envisioned that the clinician may visualize the change on the ultrasound display  18  by comparing the real-time 2D cross-sectional views with the recorded 2D cross-sectional views at the area or point of interest. To visualize the changes on the ultrasound display  18 , the clinician may overlay either the real-time or recorded 2D cross-sectional view with the other, 
     Before, during, or after viewing 2D cross-sectional views, the clinician may “tag” areas or points of interest within images on the endoscope display  38 , as represented by tags  62 ,  64 ,  66  in  FIG. 4 . The tags  62 - 66  may include information about the area or point of interest which may not be apparent when the surgical site “S” is viewed with the endoscope  36 , e.g., nerve, blood vessel, scar tissue, blood flow, etc., about the area or point of interest. It is contemplated that the clinician may freeze the image on the endoscope display  38  before, after, or during tagging of the area or point of interest. With the area or point of interest tagged on the endoscope display  38 , the clinician may continue the surgical procedure. Similar to marking the area or point of interest, the clinician may use any known means to tag an area or point of interest on the endoscope display  38 . 
     Additionally, while viewing the ultrasound display  18 , the clinician may identify an area or point of interest at or adjacent the surgical site “S”. When the clinician identifies an area or point of interest on the display  18 , the clinician may electronically or visually “mark” or “tag” the area or point of interest in the image on the display  18  as represented by tag  68  in  FIG. 3 . To electronically or visually tag the area or point of interest in the image on the display  18 , the clinician may use any known means as detailed above. The tag  68  may include information about the area or point of interest which may not be apparent when the surgical site “S” is viewed with the endoscope  36 , e.g., nerve, blood vessel, scar tissue, blood flow, etc., about the area or point of interest. It is contemplated that the clinician may freeze the image on the ultrasound display  18  before, after, or during tagging of the area or point of interest. With the area or point of interest tagged on the ultrasound display  18 , the clinician may continue to scan the surgical site “S” with the ultrasound probe  20  and electronically or visually tag subsequent areas or points of interest on the display  18  indicative of areas or points of interest at or adjacent the surgical site “S”. With the area or point of interest tagged on the ultrasound display  18 , the clinician may continue to scan the surgical site “S” with the ultrasound probe  20  and electronically or visually tag subsequent areas or points of interest on the ultrasound display  18  indicative of areas or points of interest at or adjacent the surgical site “S”. 
     When an area or point of interest is tagged on ultrasound display  18 , e.g., tag  68 , the location of the ultrasound probe  20  within the surgical site “S” is marked on the endoscope display  38  with a tag, e.g., tag  68 ′, to represent the tag on the ultrasound display  18 . 
     Providing tags  62 ,  64 ,  66 ,  68 ′ with information of areas or points of interest at or adjacent a surgical site during a surgical procedure without requiring a clinician to pause a procedure may increase a clinician&#39;s situational awareness during a surgical procedure and/or may decrease a clinician&#39;s cognitive loading during a surgical procedure. Increasing a clinician&#39;s situational awareness and/or decreasing a clinician&#39;s cognitive loading may improve surgical outcomes for patients. 
     As shown, the tags  62 ,  64 ,  66 ,  68 ′ can be displayed in a variety of shapes including a sphere, a cube, a diamond, an exclamation point. The shape of the tags  62 ,  64 ,  66 ,  68 ′ may be indicative of the type of information pertinent to the associated tags  62 ,  64 ,  66 ,  68 ′. In addition, the tags  62 ,  64 ,  66 ,  68 ′ may have a color indicative of the information contained in the tag. For example, the tag  62  may be blue when the information of the tag is pertinent to a blood vessel or may be yellow when the information of the tag is pertinent to tissue. 
     It is contemplated that the tags  62 ,  64 ,  66 ,  68 ′ may be saved for subsequent surgical procedures. Before a surgical procedure on a patient, a clinician can load a profile of the patient into the processing unit  11  and/or the control unit  31  including tags from a previous procedure. As the camera  33  of the endoscope  36  captures real-time images, the control unit  31  identifies structures within the surgical site “S” to locate and place tags, e.g., tags  62 ,  64 ,  66 ,  68 ′ from previous surgical procedures. When similar structures are identified within the surgical site “S” the control unit  31  places a tag within the image on the endoscope display  38  to provide the clinician with additional information about and/or 2D cross-sectional views of the area or point of interest from the previous surgical procedure in a similar manner as detailed above. 
     As detailed above and with reference back to  FIG. 1 , the surgical system  1  includes an ultrasound display  18  and a separate endoscope display  38 . However, the surgical system  1  can include a single monitor having a split-screen of multiple windows and/or panels with each of the ultrasound display  18  and the endoscope display  38  viewable in a respective one of the windows or panels on the monitor. 
     While several embodiments of the disclosure have been shown in the drawings, it is not intended that the disclosure be limited thereto, as it is intended that the disclosure be as broad in scope as the art will allow and that the specification be read likewise. Any combination of the above embodiments is also envisioned and is within the scope of the appended claims. Therefore, the above description should not be construed as limiting, but merely as exemplifications of particular embodiments. Those skilled in the art will envision other modifications within the scope of the claims appended hereto.