ENDOSCOPE SYSTEM, ENDOSCOPE MOVEMENT CONTROL METHOD, AND RECORDING MEDIUM

An endoscope system includes an endoscope including an insertion portion, an imaging sensor disposed at a proximal end thereof so as to be rotatable about a longitudinal axis, and an optical element that tilts an optical axis, a robotic arm, and at least one processor. The processor transmits a first signal to the moving device, detects a first moving direction of an object, estimates a first amount of rotation of the optical element by using first two images before and after the transmission with respect to the imaging sensor based on the first movement direction, transmits a second signal to the robotic arm, detects a second moving direction of the object by using second two images before and after the transmission, and estimates a second amount of rotation between the robotic arm and the endoscope based on the second moving direction and the first amount of rotation.

TECHNICAL FIELD

The present disclosure relates to an endoscope system, an endoscope movement control method, and a recording medium.

BACKGROUND

An endoscope system in which an endoscope is moved by a moving device such as an electric holder has been conventionally known. In order to move the endoscope accurately, it is desirable that the coordinate system of the endoscope and the coordinate system of the moving device match each other.

For example, when an operator remotely operates an electric holder using a user interface, the operator inputs a desired moving direction of the endoscope into a user interface based on an endoscopic image. When the coordinate system of the endoscope and the coordinate system of the moving device match each other, the user can intuitively and accurately move the endoscope in a desired direction. On the other hand, when the coordinate system of the endoscope and the coordinate system of the moving device do not match each other, the actual moving direction of the endoscope is different from the moving direction input to the user interface, which makes it difficult for the user to intuitively and accurately move the endoscope in a desired direction.

In a robot system in which a camera system is attached to a movable arm, there is known a method for correcting the transformation between the coordinate system of a robot and the coordinate system of a camera system (see, for example, PTL 1). In PTL 1, an image of a target is captured by a camera system, and the transformation is determined from the position of the movable arm when the image is captured and the position of a feature point of the target in the image.

SUMMARY

According to an aspect of the present disclosure, an endoscope system comprises: an insertion portion extending in a longitudinal axis direction;an imaging sensor disposed at a proximal end of the insertion portion so as to be rotatable about a longitudinal axis; an optical element provided in the insertion portion to tilt an optical axis in a direction offset from the longitudinal axis direction; a robotic arm that moves and holds the endoscope such that the endoscope is rotatable about the longitudinal axis; and at least one processor comprising hardware, wherein the processor is configured to: transmit, to the robotic arm, a first signal for moving the endoscope in the longitudinal axis direction, detect a first moving direction of an object within first two images by using the first two images captured by the imaging sensor before and after the transmission of the first signal, estimate a first amount of rotation of the optical element around the longitudinal axis with respect to the imaging sensor based on the first movement direction, transmit, to the moving device, a second signal for moving the endoscope perpendicular to the longitudinal axis direction, detect a second moving direction of the object within second two images by using the second two images captured by the imaging sensor before and after the transmission of the second signal, and estimate a second amount of rotation about the longitudinal axis between the robotic arm and the endoscope based on the second moving direction and the first amount of rotation.

According to another aspect of the present disclosure, there is provided an endoscope movement control method for controlling a robotic arm for moving an endoscope, the endoscope comprising an insertion portion extending in a longitudinal axis direction, an imaging sensor disposed at a proximal end of the insertion portion so as to be rotatable about a longitudinal axis, and an optical element that is provided in the insertion portion to tilt an optical axis in a direction offset from the longitudinal axis direction, and the robotic arm holding the endoscope such that the endoscope is rotatable about the longitudinal axis, the endoscope movement control method comprising: moving the endoscope in the longitudinal axis direction by the robotic arm; detecting a first moving direction of an object in first two images by using the first two images captured by the imaging sensor before and after the movement in the longitudinal axis direction; estimating a first amount of rotation of the optical element about the longitudinal axis with respect to the imaging sensor based on the first moving direction; moving the endoscope in a direction perpendicular to the longitudinal axis direction by the robotic arm; detecting a second moving direction of the object in second two images using two images captured by the imaging sensor before and after the movement in the direction perpendicular to the longitudinal axis direction; and estimating a second amount of rotation about the longitudinal axis between the robotic arm and the endoscope based on the second moving direction and the first amount of rotation.

According to another aspect of the present disclosure, there is provided a non-transitory computer-readable recording medium in which an endoscope movement control program for controlling a robotic arm for moving an endoscope is stored, the endoscope comprising an insertion portion extending in a longitudinal axis direction, an imaging sensor disposed at a proximal end of the insertion portion so as to be rotatable about a longitudinal axis, and an optical element that is provided in the insertion portion to incline an optical axis in a direction offset from the longitudinal axis direction, the robotic arm holding the endoscope such that the endoscope is rotatable about the longitudinal axis, and the endoscope movement control program causing a computer to: move the endoscope in the longitudinal axis direction by the robotic arm; detect a first moving direction of an object in first two images by using two images captured by the imaging sensor before and after the movement in the longitudinal axis direction; estimate a first amount of rotation of the optical element about the longitudinal axis with respect to the imaging sensor based on the first moving direction; move the endoscope in a direction perpendicular to the longitudinal axis direction by the robotic arm; detect a second moving direction of the object in the second two images by using the second two images captured by the imaging sensor before and after the movement in the direction perpendicular to the longitudinal axis direction; and estimate a second amount of rotation about the longitudinal axis between the robotic arm and the endoscope based on the second moving direction and the first amount of rotation.

DESCRIPTION OF EMBODIMENTS

First Embodiment

An endoscope system1, an endoscope movement control device5, a method, a program, and a recording medium according to a first embodiment of the present disclosure will be described with reference to the drawings.

As shown inFIG.1, the endoscope system1according to the present embodiment includes an endoscope2, a moving device3for holding and moving the endoscope2, an operating device4to be operated by a user, a control device (endoscope movement control device)5for controlling the moving device3based on an operation signal from the operating device4, and a display device6.

The endoscope2is a rigid endoscope including an elongated and rigid lens tube portion (insertion portion)2a, an optical system (optical element)2bdisposed in the lens tube portion2a, and a camera head2cdisposed at a base end of the lens tube portion2a. Furthermore, the endoscope2is an oblique-viewing type endoscope2having a visual axis (optical axis) B which is tilted at a predetermined angle by an optical system2bwith respect to a longitudinal axis A extending along the center in the radial direction of the lens tube portion2a.

The lens tube portion2ais attached to the camera head2cso as to be rotatable around the longitudinal axis A. An operation ring2dis fixed to the lens tube portion2a. An operator can rotate the lens tube portion2aaround the longitudinal axis A with respect to the camera head2cby rotating the operation ring2daround the longitudinal axis A while holding the camera head2c. Alternatively, the operator can also rotate the lens tube portion2aaround the longitudinal axis A with respect to the camera head2cby rotating the camera head2caround the longitudinal axis A while holding the operation ring2d.

An imaging element2efor imaging light focused by the optical system2bis fixed inside the camera head2c. The imaging element2eincludes an imaging surface which is arranged perpendicularly to the longitudinal axis A such that the longitudinal axis A passes through the center thereof. The imaging element2eis an image sensor such as a CCD (Charge Coupled Device) image sensor or a CMOS (Complementary Metal Oxide Semiconductor) image sensor.

The endoscope2is inserted into a body together with one or more treatment tools7, and an endoscopic image (image) C including the one or more treatment tools7(seeFIG.3) is captured by the imaging element2e, and transmitted through the control device5to the display device6. The display device6is any display such as a liquid crystal display or an organic EL display. The operator manipulates the treatment tools7while observing the endoscopic image C displayed on the display device6.

The moving device3is a five-degree-of-freedom robot arm having a plurality of, for example, three driving joints (joints)3M and two passive joints3P. The moving device3includes, for example, a base3ainstalled on a horizontal plane, and a first link3bto be rotationally driven with respect to the base3aaround a vertical first axial line J1. Further, the moving device3includes a second link3cto be rotationally driven with respect to the first link3baround a horizontal second axial line J2, and a third link3dto be rotationally driven with respect to the second link3caround a horizontal third axial line J3. Further, the moving device3includes a fourth link3ethat is supported to be freely rotatable with respect to the third link3dabout a fourth axial line J4extending along a plane that is orthogonal to the third axial line J3and contains the first axial line J1, and a fifth link3fwhich is supported to be rotatable with respect to the fourth link3earound a fifth axial line J5perpendicular to the fourth axial line J4.

The moving device3includes a holder3gthat is provided at the distal end of the fifth link3fand holds the lens tube portion2aof the endoscope2so as to make the lens tube portion2arotatable around the longitudinal axis A. The holder3gholds the lens tube portion2asuch that the longitudinal axis A of the lens tube portion2apasses through the intersection point between the fourth axial line J4and the fifth axial line J5and is perpendicular to the fifth axial line J5. The moving device3has five-degree-of-freedom. However, since the holder3gholds the lens tube portion2aso as to make the lens tube portion2arotatable around the longitudinal axis A, the moving device3has six-degree-of-freedom as a whole, so that the endoscope2can be arranged in a desired posture and at a desired position.

The endoscope2is moved by an interlocking operation of the three driving joints3M and the two passive joints3P of the moving device3, whereby the position and posture of the endoscope2are three-dimensionally changed.

When using the endoscope2, as shown inFIG.4, the lens tube portion2aof the endoscope2is inserted into a through-hole8aof a trocar8that has been attached to an abdominal wall W of a patient by penetrating the trocar8through the abdominal wall W, thereby observing the inside of the patient's abdominal cavity. The trocar8is swingable about a pivot point X near the abdominal wall W, and the inner diameter of the through-hole8ais set to be slightly larger than the outer diameter of the lens tube portion2a. The gap between the through-hole8aand the lens tube portion2ais hermetically sealed by a sealing member made of an elastic material (not shown).

The three-dimensional position of the distal end of the third link3dis uniquely determined by driving the three driving joints3M. The moving direction of the endoscope2is restricted by the trocar8. Therefore, when the distal end position of the third link3dis changed, the endoscope2swings the trocar8around the pivot point X or move the lens tube portion2ain the longitudinal axis direction A inside the through-hole8aof the trocar8, and passively rotates the two passive joints3P so as to follow the above movement. The rotation of the lens tube portion2aaround the longitudinal axis A relative to the holder3gis maintained in a state manually adjusted by the operator.

Each of the three driving joints3M of the moving device3is equipped with an angle sensor3hfor detecting the rotation angle thereof. The angle sensor3his, for example, an encoder, a potentiometer, or a Hall sensor provided at each driving joint3M.

The operating device4has a user interface4aincluding input devices such as keys, a joystick, buttons, and a touch panel. The operator can input an instruction for moving the endoscope2to the operating device4by operating the user interface4a. The operating device4transmits an operating signal based on an operation of the user interface4ato the control device5.

Further, the user interface4acan accept a trigger input from the operator. As described later, the trigger is used for executing processing of estimating the amounts of rotation amounts β and θ of the endoscope2.

As shown inFIG.2, the control device5includes at least one processor5a, a memory5b, a storage unit5c, an input interface5d, and an output interface5e.

The control device5is connected to the surrounding endoscope2, moving device3(robotic arm), operating device4, and display device6via the input interface5dand the output interface5e, and transmits and receives an endoscopic image C, information on the rotation angle of the driving joints3M, signals, etc. via the interfaces5dand5e.

The memory5bis, for example, a semiconductor memory including ROM (read-only memory) or RAM (random access memory) area.

The storage unit5cis a computer-readable non-transitory recording medium, and is, for example, a nonvolatile recording medium including a hard disk or a semiconductor memory such as a flash memory.

The processor5acontrols the moving device3based on the operation signal from the operating device4, thereby moving the endoscope2according to an instruction input to the user interface4aby the operator.

Here, the operator can rotate the endoscope2around the longitudinal axis A according to two ways by operating the endoscope2attached to the holder3g. First, the lens tube portion2ais rotated around the longitudinal axis A with respect to the camera head2cby manipulating the operation ring2d, which makes it possible to change the relative angle (first amount of rotation) β between the camera head2cand the lens tube portion2a. Second, the lens tube portion2ais rotated around the longitudinal axis A with respect to the holder3g, which makes it possible to change the angle (second amount of rotation) θ representing the direction of the visual axis B of the lens tube portion2awith respect to the holder3g.

In order for the operator to attach the endoscope2to the moving device3and intuitively move the endoscope2vertically and horizontally while watching the endoscopic image C displayed on the display device6, it is necessary that the amounts of rotation β and θ between the moving device3and the endoscope2have been grasped.

In the endoscope system1according to the present embodiment, the processor5aestimates the amounts of rotation β and θ of the endoscope2by executing a program (endoscope movement control program) which is recorded in the storage unit5cand read out to the memory5b.

Hereinafter, an endoscope movement control method by the endoscope system1according to the present embodiment will be described with reference to the drawings.

The processing of estimating the amounts of rotation β and θ of the endoscope2is started by the operator inputting a trigger to the user interface4a.

As shown inFIG.5A, the processor5afirst determines whether a trigger has been received (step S1), and if accepted, the processor5ainitializes the numbers of loops NB and Nθ, respectively (step S2). Next, the processor5adetermines whether Nβ=1 is satisfied (step S3), and if Nβ=1 is satisfied, the processor5astores a first endoscopic image C acquired by the endoscope2at that time into the memory5b(step S4). Further, the processor5arecognizes treatment tools (moving objects)7in the first endoscopic image C by using a known method such as image recognition using artificial intelligence, and stores areas of the treatment tools7as treatment tool areas D into the memory5b(step S5). Then, the processor5aincrements the number of loops Nβ(step S6), and repeats the processing from step S3.

If the processor5adetermines in step S3that Nβ≠1 is satisfied, the processor5atransmits a first signal for activating the moving device3so as to move the endoscope2in one direction along the longitudinal axis A (Step S7). It is determined whether the amount of movement of the endoscope2caused by the moving device3being activated with the first signal is equal to or more than a predetermined threshold value (step S8). If it is equal to or more than the threshold value, the processing is terminated.

When determining that the amount of movement of the endoscope2is smaller than the threshold value, the processor5astores a second endoscopic image C acquired by the endoscope2at that time into the memory5b(step S9). Further, the processor5arecognizes the treatment tools (moving objects)7in the second endoscopic image C, and stores the areas of the treatment tools7as treatment tool areas D into the memory5b(step S10).

A first moving direction M1of an object S in the endoscopic image C is detected using the first endoscopic image C and the second endoscopic image C (step S11). Specifically, the processor5auses a known method such as optical flow to estimate a movement vector vobjof the object S in another area E obtained by excluding the treatment tool areas D from the two endoscopic images C captured before and after the endoscope2is moved by the moving device3. As shown inFIG.6, the movement vector vobjis a two-dimensional vector that indicates the moving direction of each of a plurality of different feature points in the other area E within the endoscopic image C.

Next, the processor5adetermines whether the magnitude of the estimated movement vector vobjis equal to or more than a threshold value (step S12). The determination of the magnitude of the movement vector vobjis performed using the magnitude of any of estimated movement vectors vobjor the average value of the magnitudes of the estimated movement vectors vobj.

When the magnitude of the movement vector vobjis smaller than the threshold value, the processor5arepeats the processing from step S7. When the magnitude of the movement vector vobjis equal to or more than the threshold value, the processor5aestimates the amount of rotation β of the endoscope2from the estimated movement vector vobj(step S13).

The estimation of the amount of rotation β of the endoscope2is performed as follows.

First, the processor5acalculates the intersection point (hereinafter also referred to as a vanishing point) of straight lines along the directions of a plurality of movement vectors vobjestimated for a plurality of feature points on the endoscopic image C. Specifically, as shown inFIG.7, the processor5aarranges a plurality of temporary vanishing points Q1to Q12at equal intervals on a circumference of a radius L from the center O (hereinafter referred to as an image center) of the endoscopic image C, and stores the respective positions of the temporary vanishing points Q1to Q12.FIG.7shows an example in which the vanishing points Q1to Q12are arranged at intervals of 30°.

The radius L is the distance from the image center O to the vanishing point, and can be calculated using Formula (1).

Here, as shown inFIG.8, H represents the height (pixel) of the endoscopic image C, α represents the angle of the visual axis B with respect to the longitudinal axis A of the endoscope2, and φ represents the field of view of the endoscope2. For example, the angle α may be 30 degrees.

The processor5auses, as a reference line R, a straight line which connects the position of a vanishing point Q1for the amount of rotation β=0° and the image center O, and sets the temporary vanishing points Q1to Q12, for example, at intersection points between the circle of the radius L and straight lines obtained by rotating the reference line R by every 30° around the image center O. As a result, for example, when the temporary vanishing point Q2at a position where rotation of 30° is performed with respect to the reference line R is selected as the vanishing point, the amount of rotation β of the endoscope2can be estimated to be β=30°.

The processor5acalculates, for each temporary vanishing point Qj, the inner product of each movement vector viand a vector (Qj−pi) connecting the origin piof each movement vector viand the temporary vanishing point Qj.FIG.9illustrates a vector (Q2-p6) connecting the temporary vanishing point Q2and the origin p6of the vector v6and a vector (Q12-p6) connecting the temporary vanishing point Q12and the origin p6of the vector v6. Then, for example, as shown in Formula (2), the processor calculates the sum of inner products for all movement vectors vias an evaluation function Uj, and selects, as the vanishing point, a temporary vanishing point Qjwhere the evaluation function Ujis minimized.

In the example shown inFIG.9, the temporary vanishing point Q12is selected as the vanishing point.

As a result, the processor5aestimates the rotation angle corresponding to the selected vanishing point, 330° in the example shown inFIG.9, as the amount of rotation β of the endoscope2.

Next, as shown inFIG.5B, the processor5adetermines whether the number of loops Nθsatisfies Nθ=1 (step S14). In case of Nθ=1, the third endoscopic image C captured by the endoscope2at that time is stored in the memory5b(step S15). Further, the processor5astores the areas of the treatment tools (moving objects)7in the third endoscopic image C as the treatment tool areas D in the memory5bin the same manner as in step S5(step S16). Then, the processor5aincrements the number of loops Ne (step S17), and repeats the processing from step S14.

In step S14, when determining that Nθ≠1 is satisfied, the processor5atransmits a second signal for causing the moving device3to operate so as to move the endoscope2in one direction perpendicular to the longitudinal axis A (step S18). The processor5adetermines whether the amount of movement of the endoscope2caused by the moving device3being activated with the second signal is equal to or more than a predetermined threshold value (step S19), and when it is equal to or more than the threshold value, the processor5aterminates the rotation amount estimation processing.

When the movement amount of the endoscope2is smaller than the threshold value, the processor5auses the amount of rotation β estimated in step S13to calculate a movement vector vsysof the object S in the endoscopic image C, which is assumed to be caused by transmission of the second signal when the amount of rotation θ is equal to a predetermined value, for example, 0° (step S20).FIG.10illustrates the movement vector vsysin case of (β, θ)=(−90°, 0°).

Then, the processor5astores, into the memory5b, the fourth endoscopic image C captured after the endoscope2is moved with the second signal (step S21). Further, the processor5arecognizes the treatment tools (moving objects)7in the fourth endoscopic image C, and stores the areas of the treatment tools7as the treatment tool areas D into the memory5b(step S22).

The second moving direction M2of the object S in the endoscopic image C is detected using the third endoscopic image C and the fourth endoscopic image C (step S23). Specifically, the processor5auses a known method such as optical flow to estimate a movement vector vrealof the object S in another area E obtained by excluding the treatment tool areas D from the two endoscopic images C captured before and after the endoscope2is moved by the moving device3.

Next, the processor5adetermines whether the magnitude of the estimated movement vector vrealis equal to or more than a threshold value (step S24). The determination of the magnitude of the movement vector vrealis performed using the magnitude of any of estimated movement vectors vrealor the average value of the magnitudes of the estimated movement vectors vreal.

When the magnitude of the movement vector vrealis smaller than the threshold value, the processor5arepeats the processing from step S18. When the magnitude of the movement vector vrealis equal to or more than the threshold value, the processor5aestimates the angle between the assumed movement vector vsysand the estimated movement vector vreal, which is 180° in the example shown inFIG.10, as the amount of rotation θ of the endoscope2(step S25), and terminates the processing.

When the amounts of rotation β and θ of the endoscope2are estimated, the processor5auses the estimated amounts of rotation β and θ to correct a holder coordinate system Σr so that the holder coordinate system Σr and an endoscope coordinate system e match each other. The endoscope coordinate system Ce is a coordinate system fixed for the imaging element2e, and the holder coordinate system Σr is a coordinate system fixed for the distal end of the holder3g.

In one example, the endoscope coordinate system Ce is a rectangular coordinate system having Xe-axis, Ye-axis, and Ze-axis which are orthogonal to one another, and the holder coordinate system Σr is a rectangular coordinate system having Xr-axis, Yr-axis, and Zr-axis which are orthogonal to one another. The Xe-axis and the Xr-axis match the longitudinal axis A, the Ye-axis and the Yr-axis are parallel to the horizontal direction (left-right direction) of the endoscopic image C, and the Ze-axis and the Zr-axis are parallel to the vertical direction (up-and-down direction) of the endoscopic image C. The holder coordinate system Σr and the endoscope coordinate system Σe matching each other means that the directions of the Xe-axis and Xr-axis match each other, the directions of the Ye-axis and Yr-axis match each other, and the directions of the Ze-axis and Zr-axis match each other.

However, when the endoscope2is rotated by the amounts of rotation β and θ around the longitudinal axis A, a shift occurs between the endoscope coordinate system Ce and the holder coordinate system Σr. When the endoscope coordinate system Ce does not match the holder coordinate system Σr as described above, it is difficult for the operator who is observing the endoscopic image C displayed on the display device6to intuitively and accurately move the endoscope2in a desired direction by operating the user interface4a.

The processor5amakes the holder coordinate system Σr match the endoscope coordinate system Ce by correcting the holder coordinate system Σr based on the amounts of rotation β and θ. Specifically, the processor5acorrects a DH (Denavit-Hartenberg) parameter of the moving device3based on the amounts of rotation β and @ so that the holder coordinate system Σr matches the endoscope coordinate system Ce.

The processor5acontrols the moving device3based on the corrected holder coordinate system Σr.

As described above, according to the present embodiment, the amounts of rotation β and θ which are manually adjusted by the operator are estimated, and the holder coordinate system Σr is corrected using the estimated amounts of rotation β and θ. Therefore, even in the case of the endoscope2consisting of an oblique-viewing endoscope, the operator can intuitively and accurately move the endoscope2in a desired direction by operating the user interface4awhile observing the endoscopic image C.

Furthermore, the amounts of rotation β and θ of the endoscope2can be calculated from the two endoscopic images C obtained before and after the moving device3is activated, respectively. In other words, it is not necessary to add any equipment such as a sensor to detect the amounts of rotation β and θ, and the amounts of rotation β and θ can be estimated without any sensor.

If a device such as a sensor for detecting the amounts of rotation β and θ of the endoscope2is added to the moving device3, it would be possible to easily measure the amounts of rotation amounts β and θ. However, in this case, it is impossible to use any existing moving device3, and it is necessary to improve the moving device3. Moreover, the size and weight of the moving device3increase, and the moving device3may become an obstacle to the operator. Therefore, according to the present embodiment, there is no need to improve the moving device3, and the moving device3can be easily miniaturized.

Further, according to the present embodiment, the area E other than the treatment tool areas D in the endoscopic image C is used for estimating the movement vector vobj. As a result, even when a moving treatment tool7exists in the endoscopic image C, a highly accurate movement vector vobjthat accurately represents the moving direction of the endoscope2can be estimated. The amounts of rotation β and θ of the endoscope2can be calculated based on such a highly accurate movement vector vobj, and the holder coordinate system Σr can be accurately corrected. Furthermore, based on the accurately corrected holder coordinate system Σr, the endoscope2can be moved by the moving device3in a direction that exactly corresponds to a direction input into the operating device4by the operator.

In the present embodiment, as shown inFIG.11, a treatment tool area estimator51may estimate the moving direction of the object S based on another area E′ excluding areas obtained by adding margins F to the treatment tool areas D.

The margin F is an area that extends along the outline of the treatment tool area D and surrounds the treatment tool area D, and is, for example, a belt-shaped area having a predetermined width.

In the vicinity of the treatment tool7, the movement of the object S may be affected by the movement of the treatment tool7. For example, the object S may partially move in the vicinity of the treatment tool7due to the object S being pushed or pulled by the treatment tool7.

By excluding the area obtained by adding the margin F to the treatment tool area D, it is possible to improve the estimation accuracy of the movement vectors vobjand vrealOf the object S and more accurately estimate the amounts of rotation β and θ of the endoscope2.

Further, in the present embodiment, a plurality of temporary vanishing points Qjcorresponding to the amount of rotation β is set, and the vanishing point is selected using the evaluation function Ujshown in Formula 2. In this case, the temporary vanishing points are set at every 30°, but they may be set at every smaller angles. As a result, the amount of rotation β can be estimated with higher accuracy.

Further, instead of setting a plurality of temporary vanishing points Qjcorresponding to the amount of rotation β, functions of a plurality of straight lines along movement vectors vobjof a plurality of feature points in the endoscopic image C are determined, and as shown inFIG.12, the intersection of these straight lines may be calculated as the vanishing point. Then, the angle of a straight line connecting the calculated intersection point and the image center O with respect to the reference line R may be calculated as the amount of rotation β. As a result, the amount of rotation β can be estimated with higher accuracy.

Further, as described above, a gap exists in the radial direction between the lens tube portion2aof the endoscope2and the through-hole8aof the trocar8. When the moving device3has the passive joint3P, there may occur a situation that even if an attempt is made to move the lens tube portion2ain the longitudinal axis direction A, the lens tube portion2ashifts in a direction intersecting the longitudinal axis A by the amount corresponding to the gap. The optical flow when the lens tube portion2ais moved in the longitudinal axis direction A is used in order to estimate the amount of rotation β. Therefore, if the lens tube portion2ashifts in the direction intersecting the longitudinal axis A, the accuracy of estimating the vanishing point and thus the amount of rotation β deteriorates.

Therefore, the lens tube portion2ais moved in the longitudinal axis direction A sufficiently larger than the gap between the lens tube portion2aand the trocar8(for example, 20 mm or more when the gap is 2 to 3 mm). As a result, it is possible to reduce the ratio of the deviation amount of the movement vector vobjcaused by the gap, so that the estimation accuracy of the amount of rotation β can be improved.

Further, in order to accurately arrange the lens tube portion2aof the endoscope2at a desired position and in a desired posture, a moving device3having six-degree-of-freedom may required. The moving device3in general as a robotic arm that can have 6 degrees of freedom, however, a moving device having 3 degrees of freedom or more can be used. In the present embodiment, as described above, the six-degree-of-freedom holder3gholds the endoscope2so that the endoscope2is manually rotatable around the longitudinal axis A. For this reason, for example, some posture of the endoscope2may cause the entire endoscope2to rotate around the longitudinal axis A when the lens tube portion2aof the endoscope2is moved in a direction perpendicular to the longitudinal axis A. In this case, the endoscopic images C obtained before and after the movement rotates, which makes it difficult to correctly calculate the moving direction of the object S.

Therefore, as shown inFIG.13, when determining in step S14that Nθ=1 is satisfied, the processor5asets a target speed of the distal end of the moving device3for operating the moving device3(step S26). Based on the set target speed, the processor5acalculates the angular velocity of each of the joints3M,3P from inverse kinematics on the assumption that the moving device3has six-degree-of-freedom, and calculates a second signal for achieving the calculated angular velocity (step S27).

In this case, a second signal for a joint which rotates the endoscope2around the longitudinal axis A, but does not actually exist is also calculated. When the thus-calculated second signal is transmitted to the moving device3, the entire endoscope2would be rotated around the longitudinal axis A to compensate for the posture of the endoscope2if the moving device3has six-degree-of-freedom. However, the posture of the endoscope2is not compensated because the moving device3of the present embodiment does not have a driving joint around the longitudinal axis A.

Therefore, the processor5arotates the endoscopic image C stored in step S21after the activation of the moving device3by only the rotation angle corresponding to the second signal calculated for the joint around the longitudinal axis A which does not actually exist (step S28). As a result, the same endoscopic image C as an endoscopic image C which would be captured when the moving device3has the six-degree-of-freedom can be acquired as the endoscopic image C to be captured by the endoscope2, and at least one of the two endoscopic images C acquired before and after the transmission of the second signal is rotated by image processing so that the angles around the longitudinal axis A of the two endoscopic images C acquired before and after the transmission of the second signal match each other, whereby it is possible to accurately calculate the moving direction of the object S before and after the movement.

Further, as described above, in the present embodiment, the two joints at the distal end of the moving device3are passive joints3P, and have neither a motor nor an angle sensor. Therefore, when the three driving joints3M are driven, the lens tube portion2aof the endoscope2swings the trocar8, and is moved along the through-hole8aof the trocar8by the reaction force received from the trocar8. Following this movement, the passive joints3P are passively activated.

However, as described above, a gap exists between the inner surface of the through-hole8aof the trocar8and the outer surface of the lens tube portion2aof the endoscope2. Therefore, as shown inFIG.4, in a state where the lens tube portion2ais receiving no reaction force from the trocar8, even if the endoscope2is attempted to be moved in the longitudinal axis direction A by transmitting the first signal, the lens tube portion2amay move in a direction intersecting the longitudinal axis direction A after the movement is started. In this case, since a correct movement vector vobjcannot be obtained, the estimation accuracy of the amount of rotation β of the endoscope2deteriorates.

Therefore, as shown inFIG.14, before acquiring the first endoscopic image C, the processor5atransmits a third signal to the moving device3(step S30) to move the endoscope2to a position where the lens tube portion2areceives a reaction force equal to or more than a predetermined threshold value from the trocar8. For example, by transmitting the third signal, the inclination angle of the lens tube portion2amay be changed as shown inFIG.15to elastically deform a sealing member (not shown) between the lens tube portion2aand the through-hole8aand/or the abdominal wall W, thereby generating a reaction force equal to or more than a predetermined threshold value. Thereafter, by executing the processing from step S3, it is possible to estimate the amount of rotation β with high accuracy.

Further, in a case where the moving device3has the passive joint3P, a problem may occur when the amount of rotation θ of the endoscope2is estimated. In other words, when there is a gap between the lens tube portion2aand the through-hole8aof the trocar8, there may occur a situation that even if an attempt is made to move the endoscope2straight in a direction perpendicular to the longitudinal axis A, the passive joint3P shifts unrestrainedly by the amount corresponding to the gap, so that the passive joint3P cannot be moved straight.

Therefore, as shown inFIG.16, when it is determined in step S14that Nθ=1 is not satisfied, the processor5acalculates a second signal for achieving a moving direction of the endoscope2which allows the passive joint3P to move in order to move the endoscope2straight (step S31). The processor5amay transmit the thus-calculated second signal to the moving device3.

Since the endoscope2is moved in the direction in which the passive joint3P moves, the endoscope2can be moved straight in the direction perpendicular to the longitudinal axis A, and the accuracy of estimating the amount of rotation θ can be improved.

Further, in the present embodiment, in step S20, the assumed movement vector vsysis calculated, and in step S25, the intersection angle between the movement vector vsysand the estimated movement vector vrealis estimated as the amount of rotation θ of the endoscope2. Instead of this, as shown inFIG.17, the processor5amay perform simulation for a plurality of amounts of rotation θ, calculate the movement vector of the object S on the endoscopic image C, and adopt, as an estimation result, the amount of rotation θ used in the simulation in which the moving direction matches that of the real movement vector vreal.

In other words, first, the processor5astores the angle of each driving joint3M of the moving device3at the time when the first endoscopic image C and the treatment tool area D are stored in steps S15and S16(step S32). Then, at the time point when the real movement vector vrealhaving sufficient magnitude is calculated, the processor5astores the angle of each driving joint3M of the moving device3again (step S33). Thereafter, the processor5acalculates respective movement vectors vsimuof the object S through simulations in which the amount of rotation θ is changed (step S34). The processor5aselects a simulation in which a movement vector vsimuwhose moving direction matches that of the real movement vector vrealis calculated, and adopts the amount of rotation θ used in the selected simulation as an estimated value (step S35).

In the simulation, the angle of each of the joints3M stored in steps S32and S33, the amounts of rotation β estimated in step S13, the length of each of the links3b,3c, and3d, and the distance from the distal end of the endoscope2to the object S are used as fixed values. Then, the processor5aperforms a plurality of simulations to calculate the movement vector vsimuof the object S using the amount of rotation θ as a parameter.

According to the present embodiment, since the simulation is performed using the real angles of the driving joints3M of the moving device before and after movement, it is possible to calculate the movement vector vsimuin consideration of the variation in posture of the endoscope2caused by the moving device having five-degree-of-freedom and including the passive joints3P. Therefore, as compared with the movement vector vsysof the first embodiment which is calculated on the assumption that the distal end of the endoscope2moves straight in the direction perpendicular to the longitudinal axis A, the movement vector vsimucan be calculated with high accuracy, and the estimation accuracy of the amount of rotation θ can be improved.

Further, the case where the sixth to eighth modifications are implemented individually has been described, but instead of this case, the modifications may be implemented while combining at least two of these modifications.

Furthermore, when the endoscope2is inserted via the trocar8as described above, if the endoscope2is excessively advanced along the longitudinal axis A in order to estimate the amount of rotation β, it is considered the distal end of the endoscope2comes into contact with the object S inside the abdominal cavity. On the other hand, if the endoscope2is excessively retreated along the longitudinal axis A, it is considered that the distal end of the endoscope2enters the trocar8, and sufficient optical flow cannot be obtained.

Therefore, the operator operates the moving device3while checking the endoscopic image C in advance, moves the endoscope2to a position and posture in which the distal end thereof does not come into contact with the object S inside the body cavity, and registers the three-dimensional position of the distal end of the endoscope2at that time as a registration point T. The processing of estimating the amount of rotation β may be performed by moving the endoscope2in a retreating direction along the longitudinal axis A from a state in which the distal end of the endoscope2is disposed at the registration point T.

Specifically, as shown inFIG.19, the processor5aregisters the three-dimensional position of the distal end of the endoscope2as a registration point T (step S36). The number of registration points T may be one or more. As shown inFIG.20, when registering a plurality of registration points T, the processor5adetermines whether the registration is terminated (step S37), and repeats the processing of step S36at an arbitrary number of times.

When the processing of estimating the amounts of rotation β and θ is started in a state where the distal end of the endoscope2is disposed at an arbitrary position, as shown inFIG.21, the processor5acalculates a distance L1between the distal end of the endoscope2and the pivot point X (step S38). Then, the processor5adetermines whether the distance L1is equal to or less than a threshold value (step S39), and if the distance L1is larger than the threshold value, the processor5aexecutes the processing from step S2. In this case, the operating direction of the moving device3according to the first signal transmitted in step S7may be either the direction in which the endoscope2is advanced along the longitudinal axis A or the direction in which the endoscope2is retreated.

When it is determined in step S39that the distance L1is equal to or less than the threshold value, the processor5acalculates a distance L2between the distal end of the endoscope2and the registration point T for all registration points T (step S40). Then, the processor5amoves the distal end of the endoscope2to the registration point T which provides the smallest calculated distance L2(step S41), and then executes the processing from step S2. In this case, the operating direction of the moving device3according to the first signal transmitted in step S7is limited to the direction in which the endoscope2is retreated along the longitudinal axis A.

As described above, according to the present embodiment, in the processing of estimating the amount of rotation β, it is possible to prevent occurrence of the inconvenience of entering the trocar8and efficiently estimate the amount of rotation β. In particular, simply by registering the registration points T, the estimation processing can be performed regardless of the position of the distal end of the endoscope2at the time of starting the estimation. Therefore, there is an advantage that the operator does not have to manually move the endoscope2to a position where the distance L11is ensured to be larger than the threshold value each time.

Furthermore, when the distal end of the endoscope2is moved to the registration point T by the operation of the moving device3, the processor5amoves the distal end of the endoscope2as follows. First, the endoscope2is retreated along the longitudinal axis A until the distance L1between the distal end of the endoscope2and the pivot point X becomes a constant distance or less. Next, the posture of the endoscope2is adjusted so that the longitudinal axis A of the endoscope2is parallel to a straight line connecting a registration point T to which the endoscope2is requested to be moved and the pivot point X. Finally, the endoscope2is moved along the longitudinal axis A until the distal end of the endoscope2is at a distance equal to or less than a predetermined threshold value from the registration point T. As a result, even when there is an obstacle such as tissue or a treatment tool between the distal end of the endoscope2and the registration point T to which the endoscope2is requested to be moved at the start time of the processing of estimating the amount of rotation β, it is possible to move the distal end of the endoscope2to the registration point T without causing the distal end of the endoscope2to come into contact with the obstacle.

Moreover, in the present embodiment, the moving device3including three driving joints3M and two passive joints3P has been described. Instead of this moving device3, a moving device3including five or more driving joints3M may be adopted. As a result, the position and posture of the endoscope2can be controlled more accurately. However, since the driving joint3M requires a motor and an angle sensor, the moving device3becomes larger in size. Therefore, according to the present embodiment, the minimum number of driving joints3M that can achieve the position and posture of the endoscope2are provided, so that the moving device3can be miniaturized and the moving device3can be prevented from interfering with surgery.

Furthermore, in the present embodiment, the imaging element2ein the camera head2cis arranged so as to be orthogonal to the longitudinal axis A of the lens tube portion2aand such that the longitudinal axis A passes through the center of the imaging surface. Instead of this, in a case where an optical axis B is inclined by the optical element2bsuch as a mirror or a prism in the camera head2c, the imaging element2emay be arranged so as to be perpendicular to the optical axis B which is inclined with respect to the longitudinal axis A and such that the optical axis B passes through the center of the imaging surface.

Second Embodiment

Next, an endoscope system1, an endoscope movement control device5, a method, a program, and a recording medium according to a second embodiment of the present disclosure will be described. In the present embodiment, points different from the first embodiment will be described, and with respect to configurations common to the first embodiment, the same reference signs are appended, and description thereof will be omitted.

The endoscope system1according to the present embodiment is different from the first embodiment in that the endoscope2has a two-degree-of-freedom curved joint2fat the distal end of the lens tube portion2aas shown inFIG.22andFIG.23instead of the optical element2bwhich is provided in the lens tube portion2ato incline the visual axis B. Since the curved joint2fhas two-degree-of-freedom, there is no operation ring2dfor rotating the inclined direction of the visual axis B around the longitudinal axis A by the optical element2b.

In the description of the present embodiment, as shown inFIG.22, an inclination angle of the visual axis B with respect to the longitudinal axis A of the lens tube portion2adue to the curvature of the curved joint2fis defined as an amount of rotation γ, and as shown inFIG.23, an angle which indicates the bending direction of the curved joint2fwith the longitudinal axis A as the center is defined as an amount of rotation δ. The two-degree-of-freedom of the curved joint2fof the endoscope2is defined by these amounts of rotation γ and δ. Further, similarly to the first embodiment, the amount of rotation θ is an attachment angle of the lens tube portion2aof the endoscope2to the holder3g, and it can be arbitrarily changed manually by the operator without using the moving device3.

In the endoscope system1according to the present embodiment, the processor5aexecutes a program (endoscope movement control program) which is recorded in the storage unit5cand read out to the memory5b, thereby estimating the amounts of rotation γ, δ, and θ of the endoscope2.

An endoscope movement control method using the endoscope system1according to the present embodiment will be described below with reference to the drawings.

As shown inFIG.24, the endoscope movement control method according to the present embodiment is different in that the number of loops Nγδis used instead of the number of loops Nβin steps S42, S43, and S44, but the processing up to step S12is performed in the same manner as in the first embodiment.

The present embodiment differs from the first embodiment in estimating the amounts of rotation γ and δ (step S45).

The amount of rotation γ can be determined by setting θ=0° and calculating Formula (3) using the distance L between the vanishing point and the image center O in the positional relation shown inFIG.22.

Here, the distance L is determined as follows.

First, as shown inFIG.25, an equation of a straight line indicating the direction of each vector is determined using movement vectors vobjof two or more feature points on the endoscopic image C calculated in step S11. From the equations of these straight lines, the intersection point of the straight lines is determined as a vanishing point. As a result, the distance L from the image center O to the vanishing point is determined.

The amount of rotation γ can be estimated by substituting the determined distance L into Formula (3). Further, the angle of a straight line connecting the calculated vanishing point and the image center O with respect to the reference line R can be estimated as the amount of rotation δ. Further, the amount of rotation θ can be estimated in the same manner as in the first embodiment.

As described above, according to the present embodiment, it is possible to estimate the amounts of rotation γ and θ of the two-degree-of-freedom of the curved joint2fof the endoscope2to be manually adjusted by the operator, and the attachment angle of the endoscope2to be manually adjusted, and correct the holder coordinate system Σr using the estimated amounts of rotation γ, δ, and θ. Therefore, even when the endoscope2has a curved joint2f, the operator can intuitively and accurately move the endoscope2in a desired direction by operating the user interface4awhile observing the endoscopic image C.

REFERENCE SIGNS LIST