Patent Description:
Conventionally, an oblique-viewing endoscope capable of capturing an image at an angle inclined with respect to a central axis is used as an insertion unit of an endoscope system for insertion into a living body. In an endoscope system using the oblique-viewing endoscope, an image of a subject can be captured at another angle by rotating the oblique-viewing endoscope and changing an orientation of an observation window at a distal end of the oblique-viewing endoscope.

However, in a case where the oblique-viewing endoscope is simply rotated, a region of interest of a user (hereinafter, referred to as a region of interest) moves to an end portion of an image or goes out of an angle of view, and the user may lose sight of the region of interest. Therefore, in a case of rotating the oblique-viewing endoscope, the user needs to carefully adjust the angle and the like of the oblique-viewing endoscope while checking the image so as not to lose sight of the region of interest.

In this regard, conventionally, there has been proposed a method of rotating the insertion unit so that the observation optical axis of the insertion unit draws a conical locus having a point on an affected part of a patient as a vertex (see, for example, Patent Document <NUM>).

<CIT> discloses a medical observation system, a control device for an endoscope system, and a control method. The medical observation system comprises: a multilink structure having a plurality of links coupled to each other by joints; a medical observation device attached to the multilink structure; a control unit that controls the multilink structure or the medical observation device; and a user interface unit for a user to input instructions about changes to the visual field range of the medical observation device. The control unit controls the multilink structure or the medical observation device in accordance with user instructions, sent via the user interface unit, about the up/down/left/right movement of the visual field range of the medical observation device, the rotation of an oblique-viewing endoscope, and changes to the zoom magnification for the visual field range.

However, in the method described in Patent Document <NUM>, since it is necessary to adjust the position of an entire rotation mechanism in advance in accordance with the region of interest of the affected part of the patient, the region of interest cannot be easily changed.

The present technology has been made in view of such a situation, and enables easy change of a direction in which an image of a subject is captured in a case where imaging is performed using an optical member whose observation optical axis is inclined with respect to a central axis.

According to a first aspect the present invention provides an imaging control device in accordance with independent claim <NUM>. According to a second aspect the present invention provides an imaging control method in accordance with independent claim <NUM>. According to a third aspect the present invention provides a program according to the independent claim <NUM>. According to a fourth aspect the present invention provides an imaging system in accordance with pseudo-dependent claim <NUM>. Further aspects are set forth in the dependent claims, the drawings, and the following description.

An imaging control device according to a first aspect of the present technology includes: a rotation axis setting unit that sets a rotation axis on the basis of a pivot point and a point of interest on a subject, the pivot point serving as a fulcrum of a rod-shaped optical member whose observation optical axis is inclined with respect to a central axis and being not positioned on the observation optical axis; and a posture control unit that rotates the optical member around the rotation axis.

An imaging control method according to the first aspect of the present technology performed by an imaging control device includes: setting a rotation axis on the basis of a pivot point and a point of interest on a subject, the pivot point serving as a fulcrum of a rod-shaped optical member whose observation optical axis is inclined with respect to a central axis and being not positioned on the observation optical axis; and rotating the optical member around the rotation axis.

A program according to the first aspect of the present technology causes a computer to execute processing of: setting a rotation axis on the basis of a pivot point and a point of interest on a subject, the pivot point serving as a fulcrum of a rod-shaped optical member whose observation optical axis is inclined with respect to a central axis and being not positioned on the observation optical axis; and rotating the optical member around the rotation axis.

An imaging system according to a second aspect of the present technology includes: an imaging unit that includes a rod-shaped optical member whose observation optical axis is inclined with respect to a central axis; a rotation axis setting unit that sets a rotation axis on the basis of a pivot point and a point of interest on a subject, the pivot point serving as a fulcrum of the optical member and being not positioned on the observation optical axis; and a posture control unit that rotates the optical member around the rotation axis.

According to the first aspect of the present technology, the rotation axis is set on the basis of the pivot point that serves as a fulcrum of the rod-shaped optical member whose observation optical axis is inclined with respect to the central axis, and is not positioned on the observation optical axis, and the point of interest on the subject, and the optical member is rotated around the rotation axis.

According to the second aspect of the present technology, the rotation axis is set on the basis of the pivot point that serves as a fulcrum of a rod-shaped optical member of an imaging unit that includes the optical member whose observation optical axis is inclined with respect to the central axis, and is not positioned on the observation optical axis, and the point of interest on the subject, and the optical member is rotated around the rotation axis.

Hereinafter, modes for carrying out the present technology will be described. Descriptions will be provided in the following order.

First, an embodiment of the present technology will be described with reference to <FIG>.

<FIG> illustrates an example of a configuration of an imaging system <NUM> to which the present technology is applied.

The imaging system <NUM> is, for example, an endoscope system used in the medical field, and is used for imaging and observation in a living body. The imaging system <NUM> includes an input unit <NUM>, an imaging control unit <NUM>, an imaging unit <NUM>, an image processing unit <NUM>, a display unit <NUM>, and an arm mechanism <NUM>.

The input unit <NUM> includes an operation device such as a button, a switch, or a touch panel, and receives a user operation. The input unit <NUM> supplies an input signal input by the user operation to the imaging control unit <NUM>.

The imaging control unit <NUM> controls imaging of a subject performed by the imaging unit <NUM>. The imaging control unit <NUM> includes an imaging processing control unit <NUM> and an imaging posture control unit <NUM>.

The imaging processing control unit <NUM> controls imaging processing (for example, an imaging timing or the like) of the imaging unit <NUM> and sets various imaging parameters and the like on the basis of the input signal supplied from the input unit <NUM>, a captured image supplied from the imaging unit <NUM>, and the like.

The imaging posture control unit <NUM> controls the imaging unit <NUM> and the arm mechanism <NUM> on the basis of the input signal supplied from the input unit <NUM>, the captured image supplied from the imaging unit <NUM>, and the like, thereby controlling the posture (hereinafter, referred to as an imaging posture) of the imaging unit <NUM>. As a result, a position (hereinafter, referred to as an imaging position) and a direction (hereinafter, referred to as an imaging direction) at and in which the imaging unit <NUM> performs imaging are controlled.

The imaging unit <NUM> captures an image of the subject under the control of the imaging control unit <NUM>. The imaging unit <NUM> supplies the image (hereinafter, referred to as a captured image) obtained by performing imaging to the imaging control unit <NUM> and the image processing unit <NUM>.

The image processing unit <NUM> performs various types of image processing on the captured image, and supplies the captured image after the image processing to the display unit <NUM>.

The display unit <NUM> includes, for example, a display or the like. The display unit <NUM> displays an image based on the captured image. The image displayed on the display unit <NUM> is used as, for example, a monitor image for a doctor or the like to check an operative field or the like.

The arm mechanism <NUM> is implemented by, for example, a robot arm. The arm mechanism <NUM> supports the imaging unit <NUM> and changes the posture of the imaging unit <NUM> by moving the imaging unit <NUM> under the control of the imaging control unit <NUM>.

<FIG> schematically illustrates an example of a configuration of the imaging unit <NUM> of <FIG>.

The imaging unit <NUM> includes a camera <NUM>, a rotary actuator <NUM>, an insertion unit <NUM>, and observation windows 54a and 54b. The camera <NUM> and the insertion unit <NUM> are connected via the rotary actuator <NUM>. As illustrated in the enlarged view, the observation window 54a and the observation window 54b are provided in a distal end surface 53A of the insertion unit <NUM>.

The camera <NUM> is implemented by, for example, an imaging device including an imaging element such as a complementary MOS (CMOS). The camera <NUM> captures an image of the subject by using light (hereinafter, referred to as subject light) from the subject incident via the insertion unit <NUM>, and supplies the obtained captured image to the imaging control unit <NUM> and the image processing unit <NUM>. Furthermore, the camera <NUM> includes, for example, two imaging elements (not illustrated), and can perform stereo imaging.

The rotary actuator <NUM> rotates the camera <NUM> around an optical axis (hereinafter, referred to as an imaging optical axis) of the camera <NUM> with respect to the insertion unit <NUM>. That is, the camera <NUM> is provided at one end of the insertion unit <NUM> so as to be rotatable around the imaging optical axis via the rotary actuator <NUM>. As the camera <NUM> is rotated around the imaging optical axis, a coordinate system of the camera <NUM> (hereinafter, referred to as a camera coordinate system) is rotated around the imaging optical axis.

Note that, hereinafter, an x axis of the camera coordinate system represents a lateral direction of the camera <NUM>, a y axis represents a height direction of the camera <NUM>, and a z axis represents a direction of the imaging optical axis.

The insertion unit <NUM> includes an optical system such as a lens or a mirror, and is a rod-shaped cylindrical optical member to be inserted into a living body. Note that the rod-shaped optical member includes a curved optical member and a linear optical member. Examples of the insertion unit <NUM> include an oblique-viewing endoscope implemented by a rigid endoscope.

The insertion unit <NUM> is connected to the camera <NUM> via the rotary actuator <NUM> so that a central axis thereof coincides with the imaging optical axis of the camera <NUM>. The distal end surface 53A of the insertion unit <NUM> is inclined with respect to the central axis of the insertion unit <NUM>. The distal end surface 53A of the insertion unit <NUM> is provided with the observation window 54a and the observation window 54b each implemented by a lens. The observation window 54a and the observation window 54b are laterally arranged at the same height in an inclination direction of the distal end surface 53A.

The subject light incident on the insertion unit <NUM> from each of the observation window 54a and the observation window 54b passes through the insertion unit <NUM> and is incident on a light receiving surface of each of different imaging elements of the camera <NUM>. Then, the camera <NUM> captures a captured image (hereinafter, referred to as a left captured image) based on the subject light incident from the observation window 54a and captures a captured image (hereinafter, referred to as a right captured image) based on the subject light incident from the observation window 54b. That is, the camera <NUM> can perform stereo imaging. The camera <NUM> supplies the left captured image and the right captured image to the imaging control unit <NUM> and the image processing unit <NUM>.

Here, an observation optical axis of the insertion unit <NUM>, which is the optical axis of the observation window 54a and the observation window 54b, is perpendicular to the distal end surface 53A and inclined with respect to the central axis of the insertion unit <NUM>. Therefore, the imaging unit <NUM> captures an image in an oblique direction with respect to the central axis of the insertion unit <NUM> via the insertion unit <NUM>.

Note that, hereinafter, in a case where it is not necessary to individually distinguish the observation window 54a and the observation window 54b, they are simply referred to as the observation window <NUM>. In addition, hereinafter, in a case where it is not necessary to distinguish the left captured image and the right captured image individually, they are simply referred to as the captured image.

<FIG> is a diagram illustrating an example of a method of inserting the insertion unit <NUM> into a living body.

In this example, the insertion unit <NUM> is inserted into the abdomen of the patient by using a trocar <NUM>. Specifically, the trocar <NUM> is inserted into a skin <NUM> of the abdomen. Then, the insertion unit <NUM> is inserted from an insertion port of the trocar <NUM>, and a distal end of the insertion unit <NUM> protrudes from a distal end port of the trocar <NUM>. As a result, the insertion unit <NUM> is inserted into the living body, and the position of the insertion unit <NUM> is stabilized.

<FIG> schematically illustrates an example of a configuration of the arm mechanism <NUM> of <FIG>. Note that, in <FIG>, the imaging unit <NUM> is also schematically illustrated, and the rotary actuator <NUM> is not illustrated. Note that, also in the following similar drawings, the imaging unit <NUM> is basically schematically illustrated, and the rotary actuator <NUM> is not illustrated.

The arm mechanism <NUM> includes a base portion <NUM> and an arm portion <NUM>. The arm portion <NUM> includes links <NUM>-<NUM> to <NUM>-<NUM> and joint portions <NUM>-<NUM> to <NUM>-<NUM>. One end of the link <NUM>-<NUM> is connected to the base portion <NUM> via the joint portion <NUM>-<NUM>, and the other end of the link <NUM>-<NUM> is connected to one end of the link <NUM>-<NUM> via the joint portion <NUM>-<NUM>. The other end of the link <NUM>-<NUM> is connected to one end of the link <NUM>-<NUM> via the joint portion <NUM>-<NUM>. The other end of the link <NUM>-<NUM> is connected to the camera <NUM> via the joint portion <NUM>-<NUM>.

The base portion <NUM> supports, for example, the arm portion <NUM>, has all or a part of the imaging control unit <NUM> of the imaging system <NUM> incorporated therein, and controls the movement of the arm portion <NUM>.

The joint portions <NUM>-<NUM> to <NUM>-<NUM> are independently driven under the control of the imaging control unit <NUM>. As a result, the posture of the arm portion <NUM> is changed, and the imaging posture is changed as the imaging unit <NUM> moves.

Note that, hereinafter, in a case where it is not necessary to individually distinguish the links <NUM>-<NUM> to <NUM>-<NUM>, they are simply referred to as the link <NUM>. Furthermore, hereinafter, in a case where it is not necessary to individually distinguish the joint portions <NUM>-<NUM> to <NUM>-<NUM>, they are simply referred to as the joint portion <NUM>.

<FIG> illustrates an example of a configuration of functions of the imaging posture control unit <NUM> of the imaging system <NUM> of <FIG>.

The imaging posture control unit <NUM> includes a distance detection unit <NUM>, an observation optical axis detection unit <NUM>, a pivot point setting unit <NUM>, a rotation axis setting unit <NUM>, and a posture control unit <NUM>. The posture control unit <NUM> includes a target posture setting unit <NUM>, a camera posture control unit <NUM>, and an arm posture control unit <NUM>.

The distance detection unit <NUM> detects a distance to the subject on the basis of the left captured image and the right captured image supplied from the camera <NUM>. For example, the distance detection unit <NUM> detects a distance between an arbitrary point (hereinafter, referred to as a point of interest) on the subject and the observation window <NUM>. The distance detection unit <NUM> supplies data indicating a detection result to the rotation axis setting unit <NUM>.

The observation optical axis detection unit <NUM> detects the posture (position and orientation) of the insertion unit <NUM> (in particular, the observation window <NUM>) on the basis of data indicating the posture of the arm portion <NUM> (hereinafter, referred to as arm posture data) supplied from the arm posture control unit <NUM>. In addition, the observation optical axis detection unit <NUM> detects the observation optical axis on the basis of the posture of the observation window <NUM>. The observation optical axis detection unit <NUM> supplies data indicating a detection result to the rotation axis setting unit <NUM> and the target posture setting unit <NUM>.

The pivot point setting unit <NUM> sets the position of a pivot point on the basis of the left captured image and the right captured image supplied from the camera <NUM> and the arm posture data supplied from the arm posture control unit <NUM>. The pivot point is a point serving as a fulcrum in a case where the insertion unit <NUM> is rotated. The pivot point setting unit <NUM> supplies data indicating the position of the pivot point to the rotation axis setting unit <NUM> and the target posture setting unit <NUM>.

The rotation axis setting unit <NUM> sets a rotation axis on the basis of a distance to the point of interest on the subject, the posture of the observation optical axis, and the position of the pivot point. This rotation axis is an axis around which the insertion unit <NUM> is rotated in a case of changing a direction in which an image of the subject is captured. The rotation axis setting unit <NUM> supplies data indicating the posture (position and orientation) of the rotation axis to the target posture setting unit <NUM>.

In a case where the user performs an operation of changing the imaging posture via the input unit <NUM>, the target posture setting unit <NUM> sets a target imaging posture (hereinafter, referred to as a target posture) on the basis of a content of the operation. In addition, the target posture setting unit <NUM> calculates the posture of the arm portion <NUM> (hereinafter, referred to as a target arm posture) and the posture of the camera <NUM> (hereinafter, referred to as a target camera posture) for realizing the target posture on the basis of the posture of the observation window <NUM>, the posture of the observation optical axis, the position of the pivot point, the posture of the rotation axis, and the like. The target posture setting unit <NUM> supplies data indicating the target camera posture to the camera posture control unit <NUM>. Furthermore, the target posture setting unit <NUM> supplies data indicating the target arm posture to the arm posture control unit <NUM>.

The camera posture control unit <NUM> drives the rotary actuator <NUM> so that the camera <NUM> is in the target camera posture.

Furthermore, the camera posture control unit <NUM> detects the posture of the camera <NUM> on the basis of data indicating a rotation angle supplied from the rotary actuator <NUM>. Note that the rotation angle of the rotary actuator <NUM> is detected using, for example, a potentiometer or the like included in the rotary actuator <NUM>.

The arm posture control unit <NUM> drives actuators <NUM>-<NUM> to <NUM>-n for driving the joint portions <NUM>-<NUM> to <NUM>-<NUM> of the arm portion <NUM> so that the arm portion <NUM> is in the target arm posture. Furthermore, the arm posture control unit <NUM> detects the posture of the arm portion <NUM> on the basis of data indicating rotation angles supplied from the actuators <NUM>-<NUM> to <NUM>-n.

Note that the rotation angles of the actuators <NUM>-<NUM> to <NUM>-n are detected using, for example, a potentiometer or the like included in each of the actuators <NUM>-<NUM> to <NUM>-n.

Furthermore, hereinafter, in a case where it is not necessary to individually distinguish the actuators <NUM>-<NUM> to <NUM>-n, they are simply referred to as the actuator <NUM>.

Next, imaging direction change processing performed by the imaging system <NUM> will be described with reference to a flowchart of <FIG>.

Here, the imaging direction change processing is processing of changing the direction in which an image of the subject is captured, in particular, processing of changing the direction in which an image of a region of interest on the subject is captured.

For example, this processing starts once a power supply of the imaging system <NUM> is turned on, and ends once the power supply is turned off.

Note that examples illustrated in <FIG> will be described as specific examples as appropriate.

In Step S1, the camera <NUM> starts imaging. Specifically, under the control of the imaging control unit <NUM>, the camera <NUM> starts imaging and starts supplying a captured image to the imaging control unit <NUM> and the image processing unit <NUM>.

In Step S2, the pivot point setting unit <NUM> sets the pivot point.

Here, an example of a pivot point setting method will be described with reference to <FIG> illustrates an example in which the insertion unit <NUM> is inserted from an insertion port 202A of a skin <NUM> of the patient and an image of the inside of the living body of the patient is captured.

For example, the user operates the arm portion <NUM> via the input unit <NUM> to align the distal end of the insertion unit <NUM> with the vicinity of the center of the insertion port 202A. In this state, the user presses a button <NUM> included in the input unit <NUM>. The input unit <NUM> supplies an input signal indicating that the button <NUM> has been pressed to the imaging posture control unit <NUM>.

The arm posture control unit <NUM> detects the posture of the arm portion <NUM> in a global coordinate system on the basis of the rotation angle of each actuator <NUM>. Here, the global coordinate system is a coordinate system representing an entire three-dimensional space in which the imaging system <NUM> is present, and is also referred to as a world coordinate system. The arm posture control unit <NUM> supplies the arm posture data indicating the detected posture of the arm portion <NUM> to the pivot point setting unit <NUM>.

The pivot point setting unit <NUM> calculates the position of the distal end of the insertion unit <NUM> in the global coordinate system on the basis of the posture of the arm portion <NUM>. Then, the pivot point setting unit <NUM> sets the calculated position of the distal end of the insertion unit <NUM> as a pivot point P1. As a result, the pivot point P1 is set substantially at the center of the insertion port 202A. The pivot point setting unit <NUM> supplies data indicating the position of the pivot point P1 in the global coordinate system to the rotation axis setting unit <NUM> and the target posture setting unit <NUM>.

Thereafter, the user further inserts the insertion unit <NUM> into the living body by operating the arm portion <NUM> via the input unit <NUM>.

For example, as illustrated in <FIG>, the insertion unit <NUM> is inserted into the living body from the insertion port 202A. At this time, for example, the insertion unit <NUM> is inserted into the living body so that the pivot point P1 is positioned on the insertion unit <NUM>. Therefore, the pivot point P1 is a point that is positioned on the insertion unit <NUM> and not on an observation optical axis A1.

In Step S3, the input unit <NUM> determines whether or not a rotation operation, that is, an operation of rotating the insertion unit <NUM> to change the direction in which an image of the subject is captured has been performed. This determination processing is repeated until it is determined that the rotation operation has been performed, and in a case where it is determined that the rotation operation has been performed, the processing proceeds to Step S4.

Note that, as the rotation operation, for example, the user designates an angle and a direction of rotation of the insertion unit <NUM> around the rotation axis via the input unit <NUM>.

At this time, the angle of rotation of the insertion unit <NUM> may be expressed by either a relative angle or an absolute angle. In a case where the angle of rotation of the insertion unit <NUM> is represented by a relative angle, for example, the angle and the direction of rotation of the insertion unit <NUM> are designated in a form such as an angle of x degrees in a clockwise or counterclockwise direction from the current position. On the other hand, in a case where the angle of rotation of the insertion unit <NUM> is represented by an absolute angle, for example, the angle of rotation of the insertion unit <NUM> is designated in a form such as a direction of x degrees based on a predetermined reference position.

In Step S4, the observation optical axis detection unit <NUM> detects the observation optical axis. Specifically, the input unit <NUM> supplies an input signal indicating the content of the rotation operation to the imaging control unit <NUM>. The arm posture control unit <NUM> detects the posture of the arm portion <NUM> in a global coordinate system on the basis of the rotation angle of each actuator <NUM>. The arm posture control unit <NUM> supplies the arm posture data indicating the detected posture of the arm portion <NUM> to the observation optical axis detection unit <NUM>.

The observation optical axis detection unit <NUM> calculates the posture of the insertion unit <NUM> in the global coordinate system, in particular, the posture of the observation window <NUM> on the basis of the posture of the arm portion <NUM>. In addition, the observation optical axis detection unit <NUM> calculates the posture (position and direction) of the observation optical axis in the global coordinate system on the basis of the posture of the observation window <NUM>. The observation optical axis detection unit <NUM> supplies data indicating the postures of the observation window <NUM> and the observation optical axis to the rotation axis setting unit <NUM> and the target posture setting unit <NUM>.

In Step S5, the distance detection unit <NUM> detects the distance to the point of interest. For example, the distance detection unit <NUM> sets, as the point of interest, a point on the subject corresponding to the center of the captured image. As a result, for example, in the example of <FIG>, an intersection point P2 between the observation optical axis A1 and a surface of a subject <NUM> is set as the point of interest.

Then, the distance detection unit <NUM> detects a distance between the observation window <NUM> and the point of interest on the basis of the left captured image and the right captured image. The distance detection unit <NUM> supplies data indicating a detection result to the rotation axis setting unit <NUM>. For example, in the example of <FIG>, a distance between the observation window <NUM> and the point of interest P2 is detected.

Note that any method can be adopted as a method of detecting the distance between the observation window <NUM> and the point of interest.

In Step S6, the rotation axis setting unit <NUM> sets the rotation axis. Specifically, the rotation axis setting unit <NUM> sets a straight line connecting the pivot point and the point of interest as the rotation axis. For example, in the example of <FIG>, a straight line connecting the pivot point P1 and the point of interest P2 is set as a rotation axis A2. The rotation axis setting unit <NUM> supplies data indicating the posture of the rotation axis in the global coordinate system to the target posture setting unit <NUM>.

Here, in the example of <FIG>, the position of the observation window <NUM> and the posture of the observation optical axis A1 in the global coordinate system, and the distance between the observation window <NUM> and the point of interest P2 are known. Therefore, the rotation axis setting unit <NUM> can calculate the position of the point of interest P2 in the global coordinate system. Further, the position of the pivot point P1 in the global coordinate system is known. Therefore, the rotation axis setting unit <NUM> can calculate the posture of the rotation axis A2 connecting the pivot point P1 and the point of interest P2.

In Step S7, the target posture setting unit <NUM> sets a target posture of the imaging unit <NUM>.

Specifically, the target posture setting unit <NUM> sets a direction and an amount (rotation angle) of rotation of the insertion unit <NUM> around the rotation axis on the basis of the content of the rotation operation.

Furthermore, as illustrated in A of <FIG>, in a case where the insertion unit <NUM> is rotated around the rotation axis A2, the camera <NUM> (camera coordinate system) is also rotated. As a result, before and after the rotation of the insertion unit <NUM>, the subject <NUM> may greatly rotate in the captured image, and visibility may deteriorate.

On the other hand, as illustrated in B of <FIG>, the target posture setting unit <NUM> calculates a direction and an amount (rotation angle) of rotation of the camera <NUM> around the imaging optical axis (z axis) so that directions of the x axis and the y axis of the camera coordinate system in the global coordinate system become substantially the same before and after the rotation of the insertion unit <NUM>. Note that the direction of rotation of the camera <NUM> around the imaging optical axis is opposite to the direction of rotation of the insertion unit <NUM> around the rotation axis A2. For example, in a case where the direction of rotation of the insertion unit <NUM> around the rotation axis A2 is the clockwise direction, the direction of rotation of the camera <NUM> around the imaging optical axis is the counterclockwise direction.

The target posture setting unit <NUM> supplies data indicating the direction and the amount of rotation of the insertion unit <NUM> around the rotation axis to the arm posture control unit <NUM>. In addition, the target posture setting unit <NUM> supplies data indicating the direction and the amount of rotation of the camera <NUM> around the imaging optical axis to the camera posture control unit <NUM>.

In Step S8, the posture control unit <NUM> changes the posture of the imaging unit <NUM>. Specifically, the arm posture control unit <NUM> drives each actuator <NUM>, moves the arm portion <NUM>, and rotates the insertion unit <NUM> around the rotation axis in the rotation direction and by the rotation amount, the rotation direction and the rotation amount being set by the target posture setting unit <NUM>. In this case, the observation window <NUM> is rotated on a circumference around an intersection point between a perpendicular line drawn from the observation window <NUM> to the rotation axis and the rotation axis.

As a result, for example, the insertion unit <NUM> is rotated around the rotation axis A2, and is changed from the state illustrated in <FIG> to the state illustrated in <FIG>. Here, since the rotation axis A2 connects the pivot point P1 and the point of interest P2, a state in which the observation optical axis A1 intersects with the subject <NUM> at the point of interest P2 is maintained while the insertion unit <NUM> is rotated around the rotation axis A2. Therefore, during and after the rotation of the insertion unit <NUM>, a state in which a point corresponding to the point of interest P2 is positioned substantially at the center in the captured image is maintained.

In addition, the camera posture control unit <NUM> drives the rotary actuator <NUM> to rotate the camera <NUM> around the imaging optical axis in the rotation direction and by the rotation amount, the rotation direction and the rotation amount being set by the target posture setting unit <NUM>. As a result, the camera <NUM> is rotated around the imaging optical axis so that the rotation of the camera <NUM> accompanying the rotation of the insertion unit <NUM> is canceled according to the rotation of the insertion unit <NUM> around the rotation axis A2. As a result, as illustrated in <FIG> described above, even after the insertion unit <NUM> is rotated, the directions of the x axis and the y axis of the camera coordinate system in the global coordinate system are kept substantially constant. Then, the subject is prevented from greatly rotating around the imaging optical axis in the captured image.

Note that it is desirable that the arm posture control unit <NUM> and the camera posture control unit <NUM> are operated in a cooperative manner to make a timing of rotation of the insertion unit <NUM> and a timing of rotation of the camera <NUM> be the same as each other, such that the directions of the x axis and the y axis of the camera coordinate system in the global coordinate system are kept substantially constant even during the rotation of the insertion unit <NUM>.

Thereafter, the processing returns to Step S3, and the pieces of processing after Step S3 are performed.

As described above, the direction in which an image of the subject is captured can be easily changed. That is, under a situation where the movement of the insertion unit <NUM> is limited by the pivot point (insertion port), the user can change the direction in which an image of the subject is captured without finely adjusting the movement of the insertion unit <NUM>. In addition, even in a case where the direction in which an image of the subject is captured is changed, movement of the point of interest in the captured image and vertical inversion of the subject are suppressed. As a result, the user can easily observe the region of interest from different directions without losing sight of the region of interest on the subject.

Next, a modified example of the capturing direction change processing of <FIG> will be described with reference to <FIG>.

In Steps S1 to S4, processing similar to the processing described above is performed.

In Step S5, the distance detection unit <NUM> detects the distance to the point of interest. Here, for example, the user specifies an arbitrary point in the captured image. Then, the distance detection unit <NUM> detects a distance between the observation window <NUM> and a point of interest P11 on the subject <NUM> corresponding to the point specified by the user on the basis of the left captured image and the right captured image. The distance detection unit <NUM> supplies data indicating a detection result to the rotation axis setting unit <NUM>.

In Step S6, the rotation axis setting unit <NUM> sets the rotation axis.

Specifically, the rotation axis setting unit <NUM> calculates the position of the point of interest P11 in the global coordinate system on the basis of the position of the observation window <NUM> and the posture of the observation optical axis A1 in the global coordinate system, and the distance between the observation window <NUM> and the point of interest P11. In addition, as illustrated in A of <FIG>, the rotation axis setting unit <NUM> calculates the position of an intersection point P12 between a perpendicular line L11 drawn from the point of interest P11 to the observation optical axis and the observation optical axis A1 in the global coordinate system on the basis of the position of the point of interest P11 and the posture of the observation optical axis A1 in the global coordinate system.

Then, the rotation axis setting unit <NUM> sets a straight line connecting the pivot point P1 and the intersection point P12 as a rotation axis A11, and calculates the posture of the rotation axis A11 in the global coordinate system. The rotation axis setting unit <NUM> supplies data indicating the posture of the rotation axis A11 in the global coordinate system to the target posture setting unit <NUM>.

Specifically, the target posture setting unit <NUM> sets a direction and an amount of rotation of the insertion unit <NUM> around the rotation axis A11 on the basis of the content of the rotation operation. Further, the target posture setting unit <NUM> calculates a direction and an amount of rotation of the camera <NUM> around the imaging optical axis so that directions of the x axis and the y axis of the camera coordinate system in the global coordinate system become substantially the same before and after the rotation of the insertion unit <NUM>.

Moreover, the target posture setting unit <NUM> sets a target position of the point of interest P11 in the captured image after the rotation of the insertion unit <NUM> and the camera <NUM>. The target position of the point of interest P11 may be set by the user, or may be automatically set by the target posture setting unit <NUM>, for example. Then, the target posture setting unit <NUM> calculates a direction and an amount (rotation angle) of rotation of the insertion unit <NUM> around the pivot point P1 so that the point of interest P11 is moved to the target position in the captured image after the rotation of the insertion unit <NUM> and the camera <NUM>.

The target posture setting unit <NUM> supplies, to the arm posture control unit <NUM>, data indicating the direction and the amount of rotation of the insertion unit <NUM> around the rotation axis A11 and the direction and the amount of rotation of the insertion unit <NUM> around the pivot point P1. In addition, the target posture setting unit <NUM> supplies data indicating the direction and the amount of rotation of the insertion unit <NUM> around the imaging optical axis to the camera posture control unit <NUM>.

In Step S8, the posture control unit <NUM> changes the posture of the imaging unit <NUM>. Specifically, the arm posture control unit <NUM> drives each actuator <NUM>, moves the arm portion <NUM>, and rotates the insertion unit <NUM> around the rotation axis A11 in the rotation direction and by the rotation amount as illustrated in B of <FIG>, the rotation direction and the rotation amount being set by the target posture setting unit <NUM>. In addition, the camera posture control unit <NUM> drives the rotary actuator <NUM> to rotate the camera <NUM> around the imaging optical axis in the rotation direction and by the rotation amount, the rotation direction and the rotation amount being set by the target posture setting unit <NUM>. Note that, as described above, it is desirable that the arm posture control unit <NUM> and the camera posture control unit <NUM> are operated in a cooperative manner to make a timing of rotation of the insertion unit <NUM> and a timing of rotation of the camera <NUM> be the same as each other.

Next, the arm posture control unit <NUM> drives each actuator <NUM>, moves the arm portion <NUM>, and rotates the insertion unit <NUM> around the pivot point P1 in the rotation direction and by the rotation amount as illustrated in C of <FIG>, the rotation direction and the rotation amount being set by the target posture setting unit <NUM>. In this case, the observation window <NUM> is rotated on a circumference around the pivot point P1. As a result, the point of interest P11 reaches the target position in the captured image.

As described above, the rotation of the insertion unit <NUM> around the rotation axis A11 and the rotation of the insertion unit <NUM> around the pivot point P1 are controlled, whereby the position of the point of interest P11 in the captured image is controlled.

For example, in a case where the target position of the point of interest P11 in the captured image is set to the same position as that before the rotation, the direction in which an image of the subject <NUM> is captured can be changed without substantially moving the point of interest P11 on the subject <NUM> in the captured image.

Furthermore, for example, in a case where the target position of the point of interest P11 in the captured image is set at the center of the captured image, the posture of the insertion unit <NUM> is adjusted so that the observation optical axis A1 passes through the point of interest P11 as illustrated in <FIG>. As a result, the point of interest P11 on the subject <NUM> can be moved to the center of the captured image while changing the direction in which an image of the subject <NUM> is captured.

Note that, for example, by rotating the insertion unit <NUM> around the pivot point P1 in parallel with the rotation of the insertion unit <NUM> around the rotation axis A11, the point of interest P11 may be moved to the target position in the captured image in parallel with the change of the direction in which an image of the subject is captured.

For example, A of <FIG> illustrates the same state as A of <FIG>.

Next, as illustrated in B of <FIG>, the insertion unit <NUM> is rotated around the rotation axis A11 by a predetermined amount. At this time, the camera <NUM> is rotated around the imaging optical axis in accordance with the rotation of the insertion unit <NUM> around the rotation axis A11.

Next, as illustrated in C of <FIG>, the insertion unit <NUM> is rotated by a predetermined amount around the pivot point P1 so that the position of the point of interest P11 in the captured image approaches the target position. Note that, in a case where the target position of the point of interest P11 is set to the same position as that before the rotation, the insertion unit <NUM> is rotated around the pivot point P1 so that the position of the point of interest P11 in the captured image is moved to the target position.

Then, by alternately repeating the processing in B of <FIG> and the processing in C of <FIG>, the point of interest P11 is moved to the target position in the captured image substantially at the same time as the rotation of the insertion unit <NUM> around the rotation axis A11 ends.

Note that the processing in B of <FIG> and the processing in C of <FIG> are performed in synchronization with a frame interval of the camera <NUM>. For example, every time the processing in B of <FIG> and the processing in C of <FIG> end, the captured image is obtained.

As a result, for example, in a case where the target position of the point of interest P11 is the same as that before the rotation, the point of interest P11 is fixed at substantially the same position in the captured image during and after the rotation of the insertion unit <NUM>. On the other hand, in a case where the target position of the point of interest P11 is different from that before the rotation, the point of interest P11 is gradually brought close to the target position in the captured image. Therefore, for example, the user is prevented from losing sight of the region of interest around the point of interest P11.

Note that, in C of <FIG>, as the insertion unit <NUM> is rotated around the pivot point P1, the intersection point between the perpendicular line drawn from the point of interest P11 to the observation optical axis A1 and the observation optical axis is moved. Therefore, for example, it is desirable to change, after the processing in C of <FIG> and before the processing in B of <FIG>, the rotation axis A11 so as to connect the pivot point P1 and the intersection point after the movement.

As described above, the user can easily change the direction in which an image of the subject is captured, set an arbitrary point on the subject as the point of interest, and move the point of interest to an arbitrary target position in the captured image.

Hereinafter, modified examples of the above-described embodiment of the present technology will be described.

For example, as illustrated in <FIG>, a marker <NUM> is provided around the insertion port 202A of the skin <NUM> of the patient, and the camera <NUM> captures an image of the periphery of the marker <NUM> before the insertion of the insertion unit <NUM>. Note that the marker <NUM> is implemented by an arbitrary method such as bonding, printing, or irradiation with pattern light. Then, a distance between the observation window <NUM> and the center of the insertion port 202A in the marker <NUM> is detected on the basis of the left captured image and the right captured image. Furthermore, the position of the observation window <NUM> in the global coordinate system is detected on the basis of the posture of the arm portion <NUM>. Then, the position of the center of the insertion port 202A in the global coordinate system is detected on the basis of the position of the observation window <NUM> in the global coordinate system and the distance between the observation window <NUM> and the center of the insertion port 202A in the marker <NUM>. Then, the position of the center of the insertion port 202A in the global coordinate system is set as the pivot point.

Furthermore, for example, as illustrated in <FIG>, an insertion unit <NUM> is provided instead of the insertion unit <NUM> of the imaging unit <NUM>. An observation window <NUM> is provided on a distal end surface of the insertion unit <NUM> similarly to the insertion unit <NUM>. Furthermore, a marker (not illustrated) having a predetermined pattern is provided at a distal end of the insertion unit <NUM>. Moreover, a camera <NUM> captures an image of the periphery of the insertion port 202A, and supplies the obtained captured image to the pivot point setting unit <NUM> in the base portion <NUM>.

In a process of inserting the insertion unit <NUM> into the living body from the insertion port 202A, the pivot point setting unit <NUM> detects a moment at which the distal end of the insertion unit <NUM> enters the insertion port 202A and the marker at the distal end of the insertion unit <NUM> becomes invisible in the captured image from the camera <NUM>. The pivot point setting unit <NUM> calculates the position of the distal end of the insertion unit <NUM> in the global coordinate system on the basis of the posture of the arm portion <NUM> at the time of obtaining the captured image at the moment at which the marker at the distal end of the insertion unit <NUM> becomes invisible. Then, the pivot point setting unit <NUM> sets the calculated position of the distal end of the insertion unit <NUM> as the pivot point.

Note that, for example, a light emitting element may be provided at the distal end of the insertion unit <NUM> instead of the marker, and the pivot point setting unit <NUM> may detect a moment at which the light emitting element becomes invisible in the captured image from the camera <NUM>.

Furthermore, for example, in a case where the pivot point is fixed, for example, in a case where imaging is performed by inserting the insertion unit from a predetermined insertion port of a fixed machine, the pivot point setting processing may be omitted.

Moreover, for example, in a case where there is a possibility that the pivot point is moved, for example, in the processing in <FIG>, the pivot point setting processing may be performed every time the rotation operation is performed.

For example, as illustrated in A of <FIG>, a case where the point of interest is moved from a point P21 on a subject <NUM> to a point P22 on a subject <NUM> by rotating the insertion unit <NUM> around the pivot point P1 will be described.

In this case, in a case where the insertion unit <NUM> is rotated around the pivot point P1 as it is, the angle of the observation optical axis A1 with respect to a normal vector at the point P22 of the subject <NUM> is increased. That is, an image of the periphery of the point P22 is captured in a direction greatly inclined obliquely from the front. Therefore, the visibility of the periphery of the point P22 in the captured image may deteriorate.

On the other hand, in a case where the insertion unit <NUM> is rotated around the pivot point P1, the insertion unit <NUM> may be rotated after the observation window <NUM> (observation optical axis) is oriented in a direction in which the observation window <NUM> (insertion unit <NUM>) is rotated.

Specifically, for example, first, as illustrated in B of <FIG>, the insertion unit <NUM> is rotated around a rotation axis A21 connecting the pivot point P1 and a point P21 so that the orientation of the observation window <NUM> (the orientation of the observation optical axis A1) approaches the direction of rotation of the observation window <NUM> (the insertion unit <NUM>) around the pivot point P1.

Then, as illustrated in C of <FIG>, the insertion unit <NUM> is rotated around the pivot point P1 so that the observation optical axis A1 intersects with the point P22 on the subject <NUM>.

Note that an insertion unit <NUM>', an observation window <NUM>', and an observation optical axis A1' in C of <FIG> indicate the positions of the insertion unit <NUM>, the observation window <NUM>, and the observation optical axis A1 in a case where the insertion unit <NUM> is rotated around the pivot point P1 without being rotated around the rotation axis A21. The observation optical axis A1 has a smaller angle with respect to the normal vector at the point P22 of the subject <NUM> than the observation optical axis A1'. That is, by rotating the insertion unit <NUM> around the rotation axis A21, an image of the periphery of the point P22 can be captured in a direction closer to the front, and the visibility is improved.

Note that the processing of orienting the observation window <NUM> (observation optical axis A1) in the rotation direction of the observation window <NUM> may be automatically performed or may be performed by a user operation.

In the latter case, for example, in a case where a stick 351A of a stick controller <NUM> included in the input unit <NUM> is tilted by a predetermined angle or more, or is continuously tilted in the same direction for a predetermined time or more, processing of orienting the observation window <NUM> in the rotation direction of the observation window <NUM> may be performed.

Furthermore, in both a case where the orientation processing is automatically performed and a case where the orientation processing is performed by a user operation, on/off of a function of orienting the observation window <NUM> in the rotation direction of the observation window <NUM> may be switched.

Moreover, similarly to the example of <FIG> described above, the rotation of the insertion unit <NUM> around the pivot point P1 and the rotation of the insertion unit <NUM> around the rotation axis A21 may be alternately repeated little by little.

In the above description, an example in which the distance between the observation window <NUM> and the point of interest is detected using a stereo camera system has been described, but other detection methods may be used.

For example, the distance to the point of interest may be detected using a laser, an ultrasonic sensor, an infrared sensor, a depth sensor, or the like.

Furthermore, a distance to each point around the point of interest may also be detected according to the degree of unevenness of the surface of the subject, and an average value of the detected distances or the like may be used as the distance to the point of interest.

Furthermore, for example, in a case where a positional relationship between the insertion unit <NUM> and the subject and a three-dimensional shape of the subject are known in advance, the processing of detecting the distance to the point of interest may be omitted. Such a case is assumed to be, for example, a case where a three-dimensional shape of the inside of the body of the patient is known by a computed tomography (CT) scan or the like.

In the above description, an example in which the posture of the arm portion <NUM> is detected on the basis of the rotation angle of each actuator <NUM> has been described, but other detection directions may be used.

For example, a marker may be provided on each part of the arm portion <NUM>, the position of each marker may be detected by a motion capture system or the like, and the posture of the arm portion <NUM> may be detected on the basis of the position of each marker.

Furthermore, the configuration of the arm portion is not limited to the above-described configuration, and can be arbitrarily changed. For example, it is possible to increase the number of joints or change the direction or angle in or at which the joints are bent.

In the above description, an example in which the insertion unit is implemented by a rigid endoscope has been described, but the present technology can also be applied to a case where the insertion unit is implemented by a flexible endoscope such as a fiberscope, for example.

Furthermore, for example, the imaging unit may be provided at the distal end of the insertion unit. That is, the imaging unit may be provided so that the optical axis (imaging optical axis = observation optical axis) is inclined with respect to the central axis of the insertion unit.

The present technology can be generally applied to a case of capturing an image by rotating a rod-shaped optical member whose observation optical axis is inclined with respect to a central axis around a pivot point as a fulcrum, in addition to the case of capturing an image of the inside of a living body described above.

The series of pieces of processing described above can be performed by hardware or can be executed by software. In a case where the series of pieces of processing is performed by software, a program constituting the software is installed in a computer. Here, the computer includes a computer incorporated in dedicated hardware, a general-purpose personal computer capable of executing various functions by installing various programs, and the like, for example.

<FIG> is a block diagram illustrating an example of a configuration of hardware of a computer <NUM> performing the series of pieces of processing described above by using a program.

In the computer <NUM>, a central processing unit (CPU) <NUM>, a read only memory (ROM) <NUM>, and a random access memory (RAM) <NUM> are connected to one another by a bus <NUM>.

Moreover, an input/output interface <NUM> is connected to the bus <NUM>. An input unit <NUM>, an output unit <NUM>, a storage unit <NUM>, a communication unit <NUM>, and a drive <NUM> are connected to the input/output interface <NUM>.

The input unit <NUM> includes a keyboard, a mouse, a microphone, and the like. The output unit <NUM> includes a display, a speaker, and the like. The storage unit <NUM> includes a hard disk, a nonvolatile memory, and the like. The communication unit <NUM> includes a network interface and the like. The drive <NUM> drives a removable medium <NUM> such as a magnetic disk, an optical disk, a magneto-optical disk, or a semiconductor memory.

In the computer <NUM> configured as described above, the CPU <NUM> loads, for example, a program stored in the storage unit <NUM> to the RAM <NUM> through the input/output interface <NUM> and the bus <NUM>, and executes the program, such that the series of pieces of processing described above is performed.

The program executed by the computer <NUM> (CPU <NUM>) can be provided by being recorded in the removable medium <NUM> as a package medium or the like, for example. Furthermore, the program can be provided via a wired or wireless transmission medium such as a local area network, the Internet, or digital satellite broadcasting.

In the computer <NUM>, the program can be installed in the storage unit <NUM> via the input/output interface <NUM> by mounting the removable medium <NUM> on the drive <NUM>. Furthermore, the program can be received by the communication unit <NUM> via a wired or wireless transmission medium and installed in the storage unit <NUM>. In addition, the program can be installed in the ROM <NUM> or the storage unit <NUM> in advance.

Note that the program executed by the computer <NUM> may be a program by which the pieces of processing are performed in time series in the order described in the present specification, or may be a program by which the pieces of processing are performed in parallel or at a necessary timing such as when a call is performed or the like.

In addition, in the present specification, a system means a set of a plurality of components (devices, modules (parts), or the like), and it does not matter whether or not all the components are in the same housing. Therefore, a plurality of devices housed in separate housings and connected via a network and one device in which a plurality of modules is housed in one housing are both systems.

Note that the embodiment of the present technology is not limited to that described above, and may be variously changed without departing from the gist of the present technology.

For example, the present technology can have a configuration of cloud computing in which one function is performed in cooperation by a plurality of devices via a network.

Furthermore, each step described in the above-described flowchart can be performed by one device or can be performed in a distributed manner by a plurality of devices.

Moreover, in a case where a plurality of pieces of processing is included in one step, the plurality of pieces of processing included in the one step can be performed by one device or can be performed in a distributed manner by a plurality of devices.

Note that the present technology can also have the following configuration, which is merely illustrative and not limiting for the invention as defined by the claims.

Claim 1:
An imaging control device (<NUM>) comprising:
a rotation axis setting unit (<NUM>) that sets a rotation axis (A2) on a basis of a pivot point (P1) and a point of interest (P2, P11) on a subject (<NUM>, <NUM>, <NUM>), the pivot point (P1) serving as a fulcrum of a rod-shaped optical member (<NUM>) whose observation optical axis (A1) is inclined with respect to a central axis and being not positioned on the observation optical axis (A1); and
a posture control unit (<NUM>) that rotates the optical member (<NUM>) around the rotation axis (A2).