Patent Publication Number: US-2022217286-A1

Title: Correction of horizon tilt in an endoscopic image

Description:
This application claims the benefit of U.S. Provisional Application No. 62/133,409, filed Jan. 3, 2021. 
    
    
     BACKGROUND 
     An angled endoscope is one in which the optical axis of the scope is angularly offset from the longitudinal axis of the scope. It is a tool commonly used in laparoscopic surgery. The angled scope allows for visualization of target anatomy and provides additional options for trocar port and instrument placement. 
     Pan and zoom motions of an angled scope can be challenging, as the view is not centered on the axis of the endoscope shaft. Additionally, the angle of the scope is typically oriented downward, which makes visualizing port sites challenging. Typical practice when the surgeon wishes to view a port site is to roll the scope, thus rolling the angle about the endoscope axis. This motion has an effective panning motion, as the angle of the scope provides a view to the left or right of the original view. However, the rotation of the scope creates an incorrect horizon which can make instrument manipulation challenging. 
     This application describes features enabling digital rotation of an on-screen image to compensate for rotation of the camera. 
    
    
     
       BRIEF DESCRIPTION OF THE DRAWINGS 
         FIG. 1  schematically shows a system according to the disclosed invention; 
         FIG. 2  is a sequence of images showing a body cavity, and possible orientations of a view box of a laparoscopic camera positioned to capture images of that cavity. The images at the lower left illustrate how the view boxes would be displayed on an endoscope display using conventional techniques. The images at the right illustrate the process of detecting rotation of the scope and adjusting the display to account for the rotation. 
         FIG. 3  illustrates an image of a body cavity, and shows viewing boxes in three rotational orientations. 
         FIG. 4  schematically depicts rotation orientation of the viewing boxes for three rotational positions of the endoscope. 
         FIG. 5  illustrates a comparison of the endoscope view versus the monitor view for each of the rotational orientations shown in  FIG. 4 . 
     
    
    
     DETAILED DESCRIPTION 
     System 
     A system useful for performing the disclosed methods is depicted in  FIG. 1 . The system may optionally be used in conjunction with a robot-assisted surgical system in which surgical instruments are maneuvered within the surgical space using one or more robotic components (e.g., robotic manipulators that move the instruments and/or camera, and/or robotic actuators that articulate joints, or cause bending, of the instrument or camera shaft). 
     The system may comprise an angled endoscopic or laparoscopic camera  10 , one or more computing units  12 , and a display  14 . The camera  10  is one suitable for capturing images of the surgical site within a body cavity. It may be a 3D or 2D endoscopic or laparoscopic camera. The angle may be a fixed angle, or the scope may be one moveable to one or more angled positions. 
     The computing unit  12  is configured to receive the images/video from the camera. It may also be configured to receive data from other sources  16 , where the data allows the computing unit to receive data corresponding to changes in the rotational orientation of the camera around its longitudinal axis, as described in this application. The data may be input from which the computing unit can determine or derive the changes in the rotational orientation, or it may be input providing the computing unit with the degree of change in the rotation orientation. 
     In some embodiments, the input may be received from components of the camera. For example, an inertial measurement unit (IMU) on the camera might provide such data. As a second example, where the camera includes integral electromechanical features for rotating the camera or a distal portion of the camera about the longitudinal axis, the input representing changes in the rotational orientation of the imaging head is obtained from the corresponding components of the endoscope. In embodiments where the camera is rotated using a robotic manipulator, the system may be configured so that the computing unit receives kinematic information from the robotic manipulator or associated robotic components. 
     Still other embodiments might use computer vision applied to features captured in the endoscope image to determine the degree to which the scope has been rotated about its longitudinal axis. Where the system uses an image processing/computer vision algorithm to detect whether the scope is an angled scope that is rotating, the algorithm is used to track motion of areas detected in the image data. The system analyses whether the image data shows both a panning motion as well as rotation. Image processing algorithm analysis is used to determine if the rotation is resulting in a panning motion as well as a rotation. Pure rotation would suggest that an angled scope is not in use. Pan and rotation suggests an angled scope is in use. The image processing detects landmarks in the endoscopic view and then maintains those landmarks at a fixed horizon. 
     An algorithm stored in memory accessible by the computing unit is executable to, depending on the particular application, use the input data to perform one or more of the functions described with respect to the disclosed embodiments. 
     Referring to  FIG. 2 , block  110  shows a surgical site, and block  112  shows a border around the view of the surgical site that would be captured (“viewbox”) by the angled endoscope. It is also how the image would be displayed on the image display. The circle or dot D above the border aligns with the longitudinal axis of the scope. 
     Block  114  represents the scene after the camera has been rotated 90 degrees. The solid border represents the current viewbox, and the dashed border represents the pre-rotation viewbox from Block  112 . In conventional laparoscopy, the viewbox would be displayed in the horizontal/landscape orientation as illustrated in Block  116   a,  with the solid border representing the current viewbox, rotated to a landscape horizon. Block  118   a  shows how that current viewbox would thus be conventionally presented on the image display without the currently disclosed features. See  FIG. 3 , which shows the scene at the surgical site. The three rectangles with the light borders show the endoscope views with the angled camera at 0 degrees, 45 degrees and 90 degrees rotation, respectively, around its longitudinal axis (which is aligned with the small circle D that is shown). This is also schematically depicted in the three schematic drawings of the endoscope and the captured view shown in  FIG. 4 . Using conventional methods, each of these viewboxes would be displayed horizontally, thus making its horizon incorrect relative to the scene. 
     Method 
     In use, the system determines the amount by which the scope has been rotated. In this determining step, the system may derive the amount from received input, or it may receive input specifying the amount. In either case the determining step uses at least one of (i) image data from the camera image, (ii) telemetry data from a robotic manipulator that maneuvers the scope, or from electrometrical components that rotate a distal part of the scope, and (iii) an inertial momentum unit (IMU) sensor on the camera. If the scope is determined to have rotated, the system automatically corrects for the degree of horizon line incongruity that would occur as a result of the rotation by rotating the endoscopic image (for display) by an equal and opposite amount as the amount of derived or input rotation. The result is a “digital pan” where the scene appears to pan left or right but the only endoscope motion is an axial rotation. By maintaining the horizon line, it is easier for the surgeon to maintain cognitively oriented with respect to the image, and so instrument manipulation within the surgical site is simpler and more natural. 
     In  FIG. 5 , the endoscope image view and the displayed or “monitor” view are shown for each of the three scope angles represented in  FIG. 4 . In each case, the endoscope image view is depicted with a light border, while the boundaries of the monitor view are depicted with a dark border. The greater the degree of roll of the camera to the left, the further the rotated endoscopic image appears to the left of the monitor view, and the greater the degree of roll to the right, the further to the right it appears. 
     At zero degrees rotation, the image view fills the monitor view. As the endoscope is rotated and the resulting image is counter-rotated to provide a consistent horizon, the resulting image will not fill the monitor if a camera system with a rectangular image output is used. In this case, two methods of presenting the image may be used. In one method, shown on the left side of  FIG. 5 , the image processing unit may mask areas of the screen for which no image data has been received for display. In the other method, shown on the right side of  FIG. 5 , the image processing unit may stitch previous frames to the edges of the live, real time, (rotated) frame using image processing technique. A border B around the endoscope view may be displayed as an overlay on the monitor, serving as a clear indicator that the stitched portion of the image (i.e., the portion outside border B) is no not a live image but it is based on previous frames prior to the rotation of the angled scope. 
     Whereas a typical roll motion of the scope allows an angled scope to center on a point lateral to center, such a motion also results in the undesirable shift of the horizon line on the viewer&#39;s monitor. The solution described in this application allows correction of this horizon tilt, thereby delivering on a surgeon&#39;s need to roll to angle scope to a point desired, while simultaneously rotating the image on the monitor to provide an image with the correct horizon line.