Method of guiding an endoscope for performing minimally invasive surgery

In a method of guiding an endoscope for performing minimally invasive surgery, wherein a surgical instrument is automatically tracked by an electrically driven and controlled guide system (EGS), three base steps are principally followed: the computer controlled processing of fault tolerances, the intuitive use of the equipment by the surgeon and the sovereignty of the operating surgeon. In this way, a high degree of reliability during operation is achieved and the surgeon is relieved from the tasks of performing the tracking procedures which requires a high level of concentration and from carrying out tasks of relatively low priority.

SYSTEM DESCRIPTION In medical apparatus, the safety standards are very high. The core of the automatic endoscope tracking is therefore the error-tolerant method, which operates with multiple redundancy and therefore ensures the required safety. Additional safety occurs with the relief of the operating surgeon who is freed from technical procedures. Various degrees of automatic tracking support and the surgeon as he or she desires. As a result, the surgeon can operate the instruments necessary for an operation intuitively and in a sovereign manner. This is ensured by the still guidance by the speed limit during tracking and the voice system by which the surgeon is kept informed by way of: MMI-Monitor, LCD display or voice information concerning errors and critical conditions of the system such as soiling of the endoscope. In this way, the safety and acceptance is, in comparison with presently available systems, substantially improved because the surgeon or an assistant can eliminate the causes for malfunction effectively and rapidly for example by cleaning the optical system or by returning the instrument to the proper image area. In addition, unexpected reactions of the tracking system are substantially reduced. Sovereignty further means: The surgeon uses the monitor which does not depend on the tracking systems, that is, the original monitor and has the hierarchic possibility to switch off the tracking system at any time. FIG. 1 shows this structured requirement and it also shows the hierarchy structure starting with the central requirement for safety. The error tolerance is achieved by one or more measures: Object recognition and control as a unity, multiple treatment of possible error conditions resulting from individual components of the image processor and the control as well as a superior surveillance unit. Multi-sensor concept, adaptive feature adaptation, and 3-D reconstruction. The advantage of the uniform treatment of the object recognition and control resides in the fact that the causes for errors can be pinpointed. If, for example, the last setting actions are known the likely positions of the instrument markings can be assumed with relatively high accuracy, whereby an improved recognition safety can be achieved. A determination of the reasons for errors has, in addition to an improved communication with the surgeon, the advantage that adequate system reactions can be determined. A system configuration of the endoscope guide system is schematically presented for example by the system structure of FIG. 2 and comprises the following blocks which are interconnected by a cable, the basic EGS with four degrees of freedom, left/right, top/bottom, rotating and in/out including the electronic control and the limit switches on the respective axes of the degrees of freedom, the 2-D video endoscope with video output (red/yellow/blue-output, RYB), original monitor and light source, the computer (PC) with MMI monitor for the Man-Machine Interface (MMI) and the digital output card for the control of the logic interface (TTL), the additional components for the image processing, so-called frame grabber, the operation interface in the form of a manual switch, the joystick, for the manual operation. The tracking control consists of the following components: Image processing, Track control and, Surveillance. It processes the input values: B 1 &equals;Binary Input “Tracking in”, B 1 &equals;Binary Input “Tracking stop”, and The video signal with three channels (RGB) and synchronization. The output values are: 2×4×BO (Binary Output) for changing the positions of the axes by addressing a second digital interface, status and error messages. The main object of the automatic tracking function resides in the fact that the momentarily needed instrument tip is to be maintained in the center area of the monitor (see FIG. 4 ). The control procedure required therefor is presented in the condition graph of FIG. 3 . The release switching for the automatic tracking is initiated within the system. The automatic tracking is initiated in the present case by the operating surgeon by way of the ring switch at the operating unit (see FIG. 6 ) It remains activated until it is stopped either by pressing the stop button or by joystick actuation or automatically. The tracking is automatically stopped, when no instrument is recognized within the image either because none is present or because of soiling of the system, when the image becomes blurry because the instrument is too close to the camera, when the instrument cannot be recognized within the required reaction time, when no video signal is present, when the image processing, the tracking control the surveillance or the control recognizes electronic or program errors. Any errors are indicated on the MMT monitor. After a stop, the tracking can again be initiated. The automatic tracking operates with predetermined limited adjustment speeds up to 10 cm/see or respectively, 30°/sec, which can be adjusted depending on applications (belly; lungs; heart surgery for example) in an individual-dependent manner in such a way that the surgeon can react to undesired situations. Furthermore, there is a control limit for the positions of the axes, which keeps tilting and pivoting within predetermined limits, which limits the translatory movement along the trocar axis and which does not permit a full rotation about the shaft axis (see FIG. 7 ). From the camera image of the O-monitor (see FIG. 4 ), the possibly additionally marked instrument tip is automatically recognized by comparison with an image thereof stored in the computer and its average position by the x and y locations in the two-dimensional camera image, recognition probability, size of the identified instrument tip and additional information for error recognition are supplied to the control. The recognition of the instrument tip operates automatically and is independent on the tracking release. The image processing ( FIG. 2 ) recognizes any errors such as no instrument in the image frame, several instruments in the image frame, and stops the automatic tracking in such cases. When the instrument tip leaves the admissible image area ( FIG. 4 ) the automatic tracking system will change the position of the camera or the endoscope such that instrument tip is again in the center area of the image. This task is solved by the track control (see FIG. 2 ), which continuously processes the measured position of the instrument tip in the camera image. When the instrument tip is again within the smaller area (almost in the center— FIG. 4 ) around the center of the image, no position adjustment is initiated until the instrument tip again leaves the larger admissible area in the image. With this reservation in the movement by area-dependent suppression of the tracking movement a still picture is generated on the O-monitor. The status of the automatic tracking and any error messages are displayed on the monitor, while the image is displayed so that the image transmission to the monitor is not interrupted. In order to obtain depth recognition, generally a 3-D position determination is employed. But, since in that case, two cameras would be necessary and arranged at different observation angles, a depth recognition on the basis of 2-D image data using only one camera is preferably used. Employing the simple beam-optical relation between image and subject distances permits the determination of the distance g&equals;f ( G/B&plus; 1) wherein g&equals;the distance of the object, G&equals;the size of the object, B&equals;image size, f&equals;focal length of the endoscope lens. The most important object of the depth estimation is the determination of the size of the object in the image. The “object”, may also be represented by easily recognizable markings at the sharp edges on the object. The most simple recognition method resides in the determination of the diameter of segmented marking regions. However, this has been found to be inaccurate since, with different orientation of the endoscope and because of the properties of the central projection, there may be deformations which do not permit an accurate determination of the width of the object. A better method for determining the instrument width at the tip segments, in a first step, the edges of the object and then determines the distance from the calculated center point. This has the advantage that the width of the object is determined independently of the orientation of the object and unaffected by the particular projection. The object edges can be detected in several steps: First, a filter, for example, a 3×3 Sobel filter is applied to the transformed shading values of the image in order to subsequently begin an edge determining algorithm. The edges determined in this way however have the disadvantage, that their width may vary substantially. However, a thin edge line is required which has the width of a pixel in order to facilitate a determination of the distances from the edge in an accurate manner. This is achieved by replacing the segmented edges by approximated straight lines. This is achieved fastest by a linear regression analysis, wherein the relation between the x and y values of a quantity of line point are formulated in the formulated in the form of a linear model. In this way, the edges can be mathematically defined which facilitates the determination of the size of the object in a next step. This is done either by way of the distance between two parallel straight lines or by way of the distance of a straight line from the center point of the object by transformation of the line equations into the Hesse normalized form and insertion of the center point. FIG. 5 is an overview showings the method including the four essential steps. These are: 1. Generation of the gradient image of the marked instrument using the Sobel filter, then 2. Segmenting the edges of the object, tracking the edges, then 3. Calculating the straight edge lines by means of linear regression and finally 4. Calculating the distance: Straight line—center point of markings. It is noted that the accuracy of the distance determination depends essentially on the quality of the edge extraction.