Patent Description:
<CIT> describes a system for providing a laparoscopic vision or view extended with X-ray vision. The System comprises a medical imaging device for acquiring an image of an anatomy and a laparoscope for generating a laparoscopic image of the interior of the anatomy. A processing device of the system is configured for receiving a signal representing a position and orientation of the laparoscope, and for combining the acquired image and the laparoscopic image into a combined view, which is then displayed.

Minimal invasive surgery (MIS) or laparoscopic surgery has become popular because of its advantages over the conventional open surgery. In MIS, surgeons treat a lesion inside the human body through small incisions, which cause less pain and allow faster recovery. During surgery, an assistant is usually required to hold the endoscope/laparoscope in order to adapt its field of view to the anatomical and interventional situation, which may have certain drawbacks, e. trembling of the hand of the assistant cause image blurring resulting in eyestrain and concentration problems for the surgeons and/or costs of a second physician.

A passive holder or robotic holder of the laparoscope could overcome some of this problem. However in a (cost efficient) scenario where only one physician is required for the entire procedure (a so called "solo-surgery" scenario) using a passive holder, the "solo" surgeon is required to release his hands from his surgical instruments in order to adjust the passive holder for each change of the laparoscopic viewing position. Depending on the surgical procedure, this may be difficult or even impossible.

A robotic system may overcome this problem, when the surgeon is able to provide positioning commands to the robotic holder without using its hands. The communication between the robotic system and the surgeons is however a sensible part of a such as system since it needs to be intuitive, fast, reliable and very safe given the sensible environment such a robotic device is working in.

Passive laparoscopic holders are widely using in surgery and are commercially available (e.g. ENDOCRANE from Karl Storz).

Robotic holders that provide human-machine interfaces that are suitable for solo surgery are also commercially available or have been presented in literature. Often they require additional hardware to establish a reliable human machine interface to the robot, e.g. food pedal, voice control (e.g. <NPL>. ), or tracking systems.

The publication <CIT> discloses rendering tool information as graphic overlays on displayed images of tools.

The publication <CIT> discloses a device for assisting laparoscopic surgery.

The international publication <CIT> discloses a robotic control of imaging devices with optical shape sensing.

The US publication <CIT> discloses an adaptive imaging and frame rate optimizing based on real-time shaped sensing of medical instruments.

The underlying technical problem of the invention is to facilitate an improved controlling of a system comprising an imaging modality.

This problem is solved by the device of claim <NUM> and by the system of claim <NUM>. Further dependent claims are related to further aspects of the invention.

A method for controlling a system comprising an imaging modality is disclosed in this application, but not claimed. However, the description of the method and their embodiments allow a better understanding of the device and the system. Whenever the formulation "This disclosure relates" is mentioned, the disclosure shall facilitate the understanding without claiming the method.

This disclosure relates in one aspect to a method comprising:.

This disclosure relates in one aspect to a method, wherein the predefined portion of the field of view is selected from a plurality of predefined portions of the field of view, wherein each portion of the plurality of predefined portions of the field of view is assigned to one of a plurality of different predefined actions.

This disclosure relates in one aspect to a method, the method comprising:.

This disclosure relates in one aspect to a method, said part of the instrument comprising a plurality of components, wherein each state of the plurality of predefined first states is assigned to one of a plurality of different positions and/or orientations of the components relative to each other and/or wherein each movement of the plurality of predefined movements is assigned to one of a plurality of different movements of the components relative to each other.

This disclosure relates in one aspect to a method, the instrument comprising a grasper and/or a scissor, the plurality of predefined first states comprising an open state of the grasper and/or the scissor and a closed state of the grasper and/or the scissor and/or the plurality of predefined movements comprising an opening movement of the grasper and/or the scissor and a closing movement of the grasper and/or the scissor.

This disclosure relates in one aspect to a method, wherein the predefined action includes changing a condition that is related to an acquisition of the image.

This disclosure relates in one aspect to a method, wherein the predefined action includes changing a condition that is selected from the group consisting of the field of view of the image, an angle of view of the image, a zoom of the image, a resolution of the image, a focus of the image, a position of an imaging data acquisition component of the imaging modality relative to said portion of said patient and/or relative to said part of said instrument, an orientation of the imaging data acquisition component of the imaging modality relative to said portion of said patient and/or relative to said part of said instrument and combinations thereof.

This disclosure relates in one aspect to a method, said action unit comprising the imaging modality and/or a positioning device for positioning and/or orienting an imaging data acquisition component of the imaging modality relative to said portion of said patient and/or relative to said part of said instrument.

This disclosure relates in one aspect to a method, said part of the instrument representing a pointer element of the graphical user interface.

This disclosure relates in one aspect to a method, wherein the imaging modality is selected from the group consisting of an optical imaging modality, a laparoscopic imaging modality, an endoscopic imaging modality and an x-ray imaging modality and/or wherein the imaging data acquisition component is selected from the group consisting of a camera, a laparoscope, an endoscope and an x-ray detector.

This disclosure relates in one aspect to a method, the method comprising.

In one aspect of the invention the instrument is a surgical instrument. In one aspect of the invention said part of the instrument is a tip and/or head of the instrument.

This disclosure relates in one aspect to a method, the method comprising determining whether said part of the instrument executes a given movement selected from a plurality of predefined movements throughout the series of the images.

This disclosure relates in one aspect to a method, the method comprising generating a control command for performing a predefined action of an action unit of the system depending on whether said part of the instrument executes the given movement selected from the plurality of predefined movements throughout the series of the images.

The invention relates in one aspect to a device comprising:.

The invention relates in one aspect to a device for executing the method according to any one of the aspects of the invention described herein.

The invention relates in one aspect to a system comprising.

The invention relates in one aspect to a system, said action unit comprising the imaging modality and/or a positioning device for positioning and/or orienting an imaging data acquisition component of the imaging modality relative to said portion of said patient and/or relative to said part of said instrument.

In one aspect of the invention the imaging modality and/or the positioning device perform the predefined action.

The invention relates in one aspect to a system for executing the method according to any one of the aspects of the invention described herein.

This disclosure relates in one aspect to a computer program product comprising a computer program, the computer program being loadable directly into a memory unit of a control unit of the system according to any one of the aspects of the invention described herein, including program code sections to make the system execute the method according to any one of the aspects of the invention described herein when the computer program is executed in said control unit.

This disclosure relates in one aspect to a machine-readable medium, on which program code sections of a computer program are saved, said program code sections being loadable directly into and/or executable in a processing unit of a control unit of the system according to any one of the aspects of the invention described herein to make the system execute the method according to any one of the aspects of the invention described herein when the program code sections are executed in said processing unit.

In one aspect of the invention the positioning device comprises a robot and/or a robotic laparoscopic holder.

In one aspect of the invention the positioning device is a robot and/or a robotic laparoscopic holder.

In one aspect of the invention, the inventors propose a way of providing positioning commands to a robotic laparoscopic holder using a specific propriety of an instrument, wherein at least a part of the instrument is covered by the field of view of the laparoscopic image. Therefore a surgical instrument already introduced in the human body cavity can be used to provide an intuitive human-machine interface to the physician. No additional hardware is necessary.

In many cases, the area of focus of the surgeon is around the location of the head of the surgical instruments. Therefore, the tip/head of the surgical instrument could act like a computer mouse and/or a pointer element of a graphical user interface. Based on its position and/or gestures, control commands can be generated for performing a predefined action of an action unit of the system. For example Opening/Closing of surgical graspers and/or scissors are among two of such gestures.

In another aspect of the invention, the inventors propose to use a surgical instrument as a pointer element of a graphical user interface with the following steps:.

In another aspect of the invention, the instrument or industrial tool can only be used as a pointer element, when the tool is in a specific state, e.g. the grasper of a surgical tool is open.

In one aspect of the invention the graphical user interface is a control interface of the action unit and/or a virtual control interface of the action unit.

In another aspect of the invention, at least one step of the method or the entire process can be activated or deactivated by the surgeon using the operating element. The operating element can be for example a foot switch and/or a safety switch.

In one aspect, the invention enables the utilization of a surgical instrument as a pointer element in conjunction with a graphical user interface superimposed on a laparoscopic image, e. of the inside of the human body cavity, to control the movement of a robotic laparoscopic holder.

In one aspect, the invention enables using existing mechanically modifiable, e. from outside the human body cavity, states, e.g. grasper open or closed states, of a surgical instruments to trigger, alter, active or deactivate a control element of the graphical user interface and/or a predefined action of an action unit.

Example embodiments of the invention will be illustrated below with reference to the accompanying figures. The illustration in the figures is schematic and highly simplified and not necessarily to scale.

<FIG> shows a diagram illustrating a method with the following steps:.

<FIG> shows a diagram illustrating a method with the steps RI, DP and GC and the following steps:.

<FIG> shows a device <NUM> comprising the receiving unit <NUM>, first determining unit <NUM>, the generating unit <NUM>, the device <NUM> further comprising:.

<FIG> shows a flow diagram illustrating a method according to one aspect of this disclosure.

Step <NUM> comprises inserting the laparoscope into the patient. Step <NUM> comprises inserting the instrument into the patient. Step <NUM> comprises determining DO whether the operating element <NUM> of the system <NUM> is in a given state selected from a plurality of predefined second states. If the operating element <NUM> is not in the given state selected from the plurality of predefined second states, the corresponding path labeled with "N" is followed returning to step <NUM>. If the operating element <NUM> is in the given state selected from the plurality of predefined second states, the path labeled with "Y" is followed.

Step <NUM> comprises determining DP whether said part <NUM> of the instrument <NUM> is located within a predefined portion <NUM> of the field of view. If said part <NUM> of the instrument <NUM> is not located within the predefined portion <NUM> of the field of view, the corresponding path labeled with "N" is followed returning to step <NUM>. If said part <NUM> of the instrument <NUM> is located within the predefined portion <NUM> of the field of view, the path leading to step <NUM> is followed. Step <NUM> comprises determining DP whether said part <NUM> of the instrument <NUM> is in a given state selected from a plurality of predefined first states.

If said part <NUM> of the instrument <NUM> is in a first given state selected from a plurality of predefined first states, the path <NUM> is followed. If said part <NUM> of the instrument <NUM> is in a second given state selected from a plurality of predefined first states, the path <NUM> is followed. Step <NUM> comprises locating and/or determining a position and/or a displacement of said part <NUM> of the instrument <NUM>. Step <NUM> comprises tracking of said part <NUM> of the instrument <NUM>. In step <NUM> the positioning device follows the position and/or the displacement of said part <NUM> of the instrument <NUM>.

Step <NUM> comprises determining DP whether said part <NUM> of the instrument <NUM> is located within a predefined portion <NUM>, <NUM> of the field of view. If said part <NUM> of the instrument <NUM> is located within portion <NUM> of the field of view, the path <NUM> is followed. If said part <NUM> of the instrument <NUM> is located within portion <NUM> of the field of view, the path <NUM> is followed. Else path <NUM> is followed.

According to an example embodiment of the invention, Step <NUM> comprises generating a control command for performing a first predefined action of the action unit <NUM>, <NUM>, <NUM> of the system <NUM>, wherein the portion <NUM> is assigned to the first predefined action.

According to an example embodiment of the invention, Step <NUM> comprises generating a control command for performing a second predefined action of the action unit <NUM>, <NUM>, <NUM> of the system <NUM>, wherein the portion <NUM> is assigned to the second predefined action.

After executing at least one of the steps <NUM>, <NUM>, and <NUM> a path returning to step <NUM> is followed.

The first predefined action comprises zooming in and/or moving the imaging data acquisition component towards the instrument. The second predefined action comprises zooming out and/or moving the imaging data acquisition component away from said part <NUM> of said instrument <NUM>.

<FIG> shows a system according to one aspect of the invention, the system <NUM> comprising.

The system further comprises the control unit <NUM>, the display device <NUM> and an input device <NUM> for control inputs. The control unit <NUM> comprises a processing unit <NUM>. The device <NUM> and/or one or many or all components of the device <NUM> are implemented in a processor system of the processing unit <NUM>. The processor system comprises one or several microprocessors.

The imaging modality <NUM>, <NUM> comprises an imaging data acquisition component <NUM>. The imaging modality <NUM>, <NUM> comprises an imaging modality control unit <NUM> for controlling the imaging modality <NUM>, <NUM> and/or the imaging data acquisition component <NUM>.

The arrows <NUM>, <NUM> and <NUM> indicate degrees of freedom of a position of the imaging data acquisition component <NUM> and/or an orientation of the imaging data acquisition component <NUM>. The degree of freedom indicated by the arrow <NUM> is controlled by an aspect of the present invention. In the example of <FIG>, the degrees of freedom indicated by the arrows <NUM> and <NUM> are controlled by visual servoing based on tool tracking.

The point <NUM> is a trocar pivot point of the instrument <NUM>. The point <NUM> is a trocar pivot point of the laparoscope <NUM>.

In the example of <FIG>, a part of the robot motion, i.e. laparoscopic robot motion, (<NUM> translational, arrow <NUM> in <FIG>) is controlled by the above mentioned approach of a surgical instrument as a pointer element.

Another part of the robot motion (<NUM> rotational degrees of freedom, arrows <NUM> and <NUM> of <FIG>) is controlled by simple tool tracking. The surgical tool used in this example is a surgical grasper.

In one aspect of the invention, the operating element is the "Instrument Track enable" switch. In one aspect of the invention, the instrument's head position is tracked when the "Instrument Track enable" switch is enabled by the surgeon (e.g. a foot switch). This "Instrument Track enable" switch enhances safety by preventing inadvertent harmful events such as tissue perforation or other collisions. In one aspect of the invention, the entire process described in the following is only active when this "Instrument Track enable" switch is pressed. Provided, that the "Instrument Track enable" switch is enabled, the tracking would be triggered based on the status of the grasper of the instrument.

<FIG>, <FIG> show and image <NUM> of a minimally invasive surgery scenario and a graphical user interface <NUM> of the action unit <NUM>, <NUM>, <NUM> superimposed on the image <NUM> according to one aspect of the invention. The portion <NUM> comprises the portion <NUM> and the portion <NUM>.

The part <NUM> of the instrument <NUM> is the head of the instrument <NUM> and a grasper <NUM> or, equivalent, a gripper <NUM>. In <FIG>, the grasper is in the closed state. The part <NUM> of the instrument <NUM> would be tracked in 2D when the part <NUM> of the instrument <NUM> is in the closed state. This results in <NUM> rotational movements of the robot. As long as the grasper <NUM> is closed, the robot <NUM> follows movement of the grasper <NUM> with <NUM> rotational degrees if freedom <NUM>, <NUM>. The part <NUM> of the instrument <NUM> would not be tracked in 2D, if the grasper is in the open state.

According to an example embodiment of the invention, if the part <NUM> of the instrument <NUM> is present in the portions <NUM> and <NUM> and the gripper <NUM> is in the closed state, no predefined action, to which at least one of portion <NUM> and <NUM> is assigned, would be performed.

According to an example embodiment of the invention, if the part <NUM> of the instrument <NUM> is present in the portion <NUM> and the gripper <NUM> is in the closed state, the first predefined action would not be performed.

According to an example embodiment of the invention, if the part <NUM> of the instrument <NUM> is present in the portion <NUM> and the gripper <NUM> is in the closed state, the second predefined action would not be performed.

the robot <NUM> will follow the movement of the part <NUM> of the instrument <NUM> using <NUM> rotational degrees of freedom as indicated by arrows <NUM> and <NUM> in <FIG>. No zooming in or zooming out (i.e. Robot in/out motion) will be performed if the grasper <NUM> is in the closed state and inside one of portions <NUM> and portion <NUM>.

For zooming in or out, which in one aspect of the invention corresponds to kinematically moving the camera <NUM> towards or away from the part <NUM> of the instrument <NUM> and/or the part of the patient <NUM> with <NUM> degree of freedom as indicated by the arrow <NUM> in <FIG>, there are predefined portions <NUM>, <NUM> in the image <NUM> which are assigned to a zooming in action and a zooming out action, respectively. If the instrument gripper <NUM> is in the open state and the instrument head is present in these zones, the robot <NUM> would move in and out of the patient.

Zooming in is performed if instrument head <NUM> is in open state and inside <NUM>. In this case the robot <NUM> moves the laparoscope <NUM> towards the part <NUM> of the instrument <NUM> and/or the part of the patient <NUM>.

Claim 1:
A device (<NUM>) comprising:
- a receiving unit (<NUM>) for receiving (RI) an image (<NUM>) and/or a series of images (<NUM>) of a portion of a patient (<NUM>) from an imaging modality (<NUM>, <NUM>), the field of view of the image (<NUM>) and/or of the series of images (<NUM>) covering at least a part (<NUM>) of an instrument (<NUM>),
- a first determining unit (<NUM>) for determining (DP) whether said part (<NUM>) of the instrument (<NUM>) is located within a predefined portion (<NUM>, <NUM>, <NUM>) of the field of view and whether said part (<NUM>) of the instrument (<NUM>) is in a given state selected from a plurality of predefined first states,
- a generating unit (<NUM>) for generating (GC) a control command for performing a predefined action of an action unit (<NUM>, <NUM>, <NUM>) of a system (<NUM>) comprising the device (<NUM>) depending on whether said part (<NUM>) of the instrument (<NUM>) is located within the predefined portion (<NUM>, <NUM>, <NUM>) of the field of view and whether said part (<NUM>) of the instrument (<NUM>) is in the given state selected from the plurality of predefined first states,
characterized in that
- the predefined portion (<NUM>, <NUM>, <NUM>) of the field of view is highlighted (HP) and a graphical user interface (<NUM>) of the action unit (<NUM>, <NUM>, <NUM>) is superimposed (SUI) on the image (<NUM>), the highlighted predefined portion (<NUM>, <NUM>, <NUM>) of the field of view representing a control element of the graphical user interface (<NUM>) regarding the predefined action, said part (<NUM>) of the instrument (<NUM>) representing a pointer element of the graphical user interface (<NUM>).