Patent Description:
Surgeons may require a microscope in order to perform a surgery at the bottom of deep wounds, i.e. for wounds that form a deep cavity within human tissue. For example, in brain surgeries the wound is typically deep in order to be able to access the tissue within the brain that is to be operated on. The tissue to be operated on is typically at the bottom of the cavity and a microscope is used to enable the surgeon to monitor the tools used at the bottom of the cavity.

The following publications relate to the determination of <NUM> Dimensional Models of wounds: <NPL>.

<CIT> relates to a system and method for adaptively and interoperatively configuring an automated arm used during a medical procedure.

An embodiment relates to a controller for a microscope such as a surgical microscope. The controller is configured according to the features defined by independent claim <NUM> or <NUM>.

According to an embodiment, the controller is configured to determine the line of sight based on an axis within the cavity that fulfills a predetermined criterion. Providing the possibility to define a predetermined criterion may provide the flexibility to adapt the determination of the line of sight to the individual surgery.

According to an embodiment, the controller is configured to determine the line of sight based on a cavity centroid axis of the cavity. Using the cavity centroid axis may determine the line of sight such that an average distance to the surrounding walls of the cavity along the line of sight is maximized. This may provide a line of sight with maximum safety margins in the event that the flexible sidewalls of the cavity of the wound move or deform during the course of the surgery.

According to an embodiment, the controller is configured to determine the line of sight based on an axis that maximizes the visible area of a bottom of the cavity. Maximizing the visible area of the bottom of the cavity may enable the surgeon to easily navigate at the bottom of the wound.

According to an embodiment, the controller is configured to determine a further line of sight to the bottom of the wound based on further image data generated using the line of sight. If the controller is enabled to update the line of sight and the alignment of the microscope by determining a further line of sight, the alignment may be continuously re-adjusted during the ongoing surgery to enable the surgeon to concentrate only on the surgery itself, which may, on the one hand, decrease the time for the surgery and, the other hand, increase the quality of the result.

An embodiment of a microscope system comprises a controller according to claim <NUM> or <NUM> and a microscope configured to align its optical axis with the line of sight based on the control signal. An embodiment of a microscope system may avoid manual adjustment of the microscope by the surgeon which may be slower and less reliable than an automated adjustment. A surgeon may so only need to concentrate on the surgery itself, which may result in a better result of the surgery.

According to an embodiment, the microscope system further comprises an imaging device configured to generate the image data. A microscope system according to such an embodiment may be used to autonomously determine the required image data and adjust the field of view of the microscope.

According to an embodiment, the imaging device is one of a time-of-flight camera, a stereo camera, and a three-dimensional camera. Equipping a microscope system with such an imaging device may enable the controller to determine a three-dimensional model of a cavity of the wound based on image data generated by means of the microscope system itself.

An embodiment of a method for controlling a microscope comprises the steps defined by independent claim <NUM> or <NUM>.

According to an embodiment of the method, the line of sight is determined based on a cavity centroid axis of the cavity.

According to an embodiment of the method, the line of sight is determined based on an axis that maximizes the visible area of a bottom of the cavity.

An embodiment of a computer program has a program code for performing a method for controlling a microscope according to claim <NUM> or <NUM> when the program is executed on processor. An embodiment of the computer program may so be used to supplement or upgrade an existing microscope system with the capability of automatically align an optical axis of the microscope to a desirable line of sight.

Accordingly, while further examples are capable of various modifications and alternative forms, some particular examples thereof are shown in the figures and will subsequently be described in detail. However, this detailed description does not limit further examples to the particular forms described. Further examples may cover all modifications, equivalents, and alternatives falling within the scope of the disclosure. Same or like numbers refer to like or similar elements throughout the description of the figures, which may be implemented identically or in modified form when compared to one another while providing for the same or a similar functionality. In any case, the scope of the present invention is defined by the appended claims.

It will be understood that when an element is referred to as being "connected" or "coupled" to another element, the elements may be directly connected or coupled or via one or more intervening elements. If two elements A and B are combined using an "or", this is to be understood to disclose all possible combinations, i.e. only A, only B as well as A and B, if not explicitly or implicitly defined otherwise. An alternative wording for the same combinations is "at least one of A and B" or "A and/or B". The same applies, mutatis mutandis, for combinations of more than two Elements.

<FIG> illustrates an embodiment of a controller <NUM> for a microscope <NUM> such as a surgical microscope <NUM>. Microscope <NUM> having an optical axis <NUM> is illustrated in <FIG> to supplement the description of the controller. A controller <NUM> and a microscope <NUM> together constitute an embodiment of a microscope system. In <FIG>, optional components are illustrated by dashed lines.

The controller <NUM> is configured to receive image data representing a wound and to determine a line of sight to a bottom of the wound using the image data. Further, the controller <NUM> is configured to output a control signal for the microscope <NUM>, the control signal instructing the microscope <NUM> to align its optical axis <NUM> with the line of sight. Using such a controller <NUM> for a microscope <NUM> may enable an automatic adjustment of the microscope's orientation such that the tissue to be operated on is visible for the surgeon while avoiding a manual adjustment by the surgeon which may be slower than automated adjustment. A surgeon may so save time to a patient's benefit. Further, a surgeon may only need to concentrate on the surgery itself without any interruption to adjust the microscope, which may result in a better result of the surgery. Aligning the optical axis <NUM> of the microscope with a line of sight may, for example, be performed such that the optical axis <NUM> is parallel to the line of sight or that it is equal to the line of sight.

Aligning the optical axis <NUM> of the microscope may comprise changing the orientation of the microscope <NUM> or moving the microscope. Embodiments of controllers may be used with different types of microscopes. For example, microscopes may be used together with the controller <NUM> which already comprise motorized adjustments for its orientation. The orientation may, for example, be changed by means of a robotic, or generally a motorized arm of the (surgical) microscope <NUM>. In the event of such a microscope, the control signal generated by the controller <NUM> may instruct the already present motors within the microscope <NUM> to result in an aligned optical axis <NUM> of the microscope <NUM>. However, further embodiments of controllers may also be used as an add on to existing microscopes which are only partially motorized, or which are not motorized at all. In this event, motors can be added, for example to the mechanics adjusting the field of view of the microscope. In such a setup, the control signal of the controller instructs the added motors to perform the actions required to adjust the zoom and/or the orientation of the mechanics of the microscope.

In order to receive the image data, the controller <NUM> may furthermore optionally comprise an input interface <NUM>. For outputting the control signal to the microscope <NUM>, the controller <NUM> may optionally comprise an output interface <NUM>. Depending on the circumstances and the capabilities of the microscope <NUM>, the image data may be generated by and received from the microscope <NUM> or by external imaging means connected to the input interface <NUM>. Any interconnect between the microscope <NUM> and the controller <NUM> maybe wired or wireless, the interconnect may establish a point-to-point connection, or it may be routed through an intermediate network, established over a bus system, or the like. In some embodiments, the controller <NUM> may furthermore optionally comprise a processor <NUM> to perform the calculations required to determine the line if sight using the image data.

In order to discuss different possibilities to identify a line of sight, <FIG> schematically illustrates a profile of a wound <NUM> forming a deep cavity in human tissue <NUM>. The wound <NUM> has an irregular shape and extends into the human tissue <NUM> to form a cavity therein. A deep wound may be characterized in that its extension at the surface of the wound is smaller than the depth of the cavity. In the exemplary coordinate system illustrated in <FIG>, the surface of the wound <NUM> is essentially parallel to the x-y plane, while the cavity extends parallel to the z-direction. <FIG> illustrates a typical example for a surgery, where the wound <NUM> and the surrounding tissue <NUM> is covered by a sheet <NUM> which may, for example, be made of green or blue cloth. The sheet <NUM> and its circular opening <NUM> define the area directly accessible by the surgeon. When deep wounds are required, the surgery is often performed at a bottom <NUM> of the wound. In this event it may be required that the microscope <NUM> aligns its optical axis <NUM> to a line of sight that enables the surgeon to see the bottom <NUM> of the wound <NUM>. Just as an example, <FIG> illustrates two possible lines of sight <NUM> and <NUM> that enable a view on the bottom <NUM> of the wound <NUM>. The two lines of sight may, however, be determined using different criteria.

Line of sight <NUM> is approximately perpendicular to the surface of the wound <NUM> and extends approximately along the z-direction. Line of sight <NUM> may, for example, be generated using multiple images of the wound <NUM> taken along different directions. The multiple images may be taken by means of an image sensor of the microscope <NUM> or, alternatively or additionally by a further imaging device added to the microscope <NUM>. <FIG> schematically illustrates an optional imaging device <NUM>, which may be used for this purpose. In order to determine the line of sight <NUM>, the controller is configured to calculate an area of the wound <NUM> within each of the images. In every image, the calculated area of the wound <NUM> corresponds to a projection of the real area of the wound onto a plane which is perpendicular to the direction from which the image has been taken. If the direction under which the image is taken is perpendicular to the surface of the wound <NUM>, the calculated area of the wound <NUM> becomes maximum. Therefore, the corresponding direction may be used as a line of sight <NUM> to which the optical axis can be aligned to guarantee a view on the bottom <NUM> of the wound <NUM>. The line of sight <NUM> may so be determined using only the imaging equipment of the microscope itself.

To calculate the area of the wound <NUM> within the individual images, the position of the wound is identified within the image data. There are multiple possible ways to identify a wound within various types of image data. Multiple possibilities to identify a wound within different type of image data are described in European patent application titled "A controller for a microscope, a corresponding method and a microscope system" and filed with the European patent office by the applicant on February <NUM>, <NUM>, receiving application number <CIT>. The teaching as to how a wound within image data can identified, for example by means of a controller, as well as the teaching regarding the image data that can be used is herewith incorporated herein by reference to said application.

The controller <NUM> may additionally or alternatively also be configured to determine a three-dimensional (3D) model of the cavity of the wound <NUM> based on the image data. A three-dimensional model may allow to determine a line of sight using arbitrary criteria to perform the alignment with high flexibility and the possibility to adapt the determination of the line of sight to the individual surgery or to the preferences of an individual surgeon. In other words, a <NUM>-D model may provide for the possibility to determine the line of sight based on an axis within the cavity that fulfills an arbitrary predetermined criterion.

For example, the controller may be configured to determine the line of sight based on an axis that maximizes the visible area of a bottom of the cavity. Maximizing the visible area of the bottom <NUM> of the cavity may enable the surgeon to easily navigate within the wound. Maximization may be performed using an arbitrary search algorithm. Determining the line of sight based on the criterion that the visible area of a bottom <NUM> of the cavity is maximum may also serve to avoid obstacles, such as for example obstacle <NUM> schematically illustrated by dashed lines in <FIG>. In the event that obstacle <NUM> is present, be it as part of tissue or by incorporation of external matter into the wound (for example cotton batting to absorb blood), determining the line of sight such that the visible area at the bottom <NUM> of the wound <NUM> is maximum will result in the bottom being still visible, although the obstacle <NUM> may block other possible lines of sight to the bottom <NUM> of the wound, such as for example line of sight <NUM>.

The controller may also be configured to determine the line of sight based on a cavity centroid axis of the cavity. Using the cavity centroid axis may determine the line of sight such that an average distance to the surrounding walls of the cavity along the line of sight is maximized. This may provide a line of sight with maximum safety margins in the event that the flexible sidewalls of the cavity of the wound <NUM> move or deform during the course of the surgery. The cavity centroid axis may be computed like a principal axis of inertia for a body having a mass. To this end, the volume of the cavity may be viewed as a volume with uniform mass density to compute the principal axes of inertia solving the corresponding well-established equations. The principal axis of inertia extending in the z-direction may then be used as the line of sight, for example.

After having determined a first line of sight, the controller may also be configured to determine a further line of sight to the bottom of the wound based on further image data generated using the line of sight. If the controller is enabled to update the line of sight and the alignment of the microscope by determining a further line of sight, the alignment may be continuously re-adjusted during the ongoing surgery to enable the surgeon to concentrate only on the surgery itself, which may, on the one hand, decrease the time for the surgery and, the other hand, increase the quality of the result. Further, obstacles <NUM> entering the wound may not permanently block a surgeon's view on the bottom of the wound since a re-determined further line of sight may be chosen such that the obstacle <NUM> will not block the view on the bottom along the further line of sight.

As illustrated as an optional feature in <FIG>, some embodiments of a microscope system, which comprises a microscope <NUM> and a controller <NUM>, may optionally further be equipped with an imaging device <NUM> configured to generate the image data.

For example, some embodiments are equipped with a time-of-flight camera, a stereo camera, or a <NUM>-D camera in order to enable the controller to generate a <NUM>-D model of the cavity based on the image data generated by the imaging device chosen.

<FIG> illustrates a flowchart of an embodiment of a method for controlling a microscope. The method comprises receiving image data <NUM> representing a wound and determining a line of sight <NUM> to a bottom of the wound using the image data. Further, the method comprises instructing a microscope <NUM> to align its optical axis with the line of sight.

As further illustrated in <FIG>, receiving image data <NUM>, determining a line of sight <NUM> and instructing a microscope <NUM> to align its optical axis with the line of sight may optionally be performed in a loop <NUM> according to some embodiments. Continuously re-aligning the optical axis of a microscope may automatically consider obstacles within the cavity of a wound, such as for example tools used during the surgery. A permanent re-alignment may enable the surgeon to concentrate on the surgery and to avoid situations where the surgeon is no longer able to see the tissue he is operating on.

In other words, some previously described embodiments propose an alignment of a microscope in order to visualize the bottom of a narrow surgical cavity, which is a process which can be tedious and time consuming. For example, one may mimic the alignment as it would be done by humans, i.e. perceive the long axis of the cavity and align accordingly. There are different technical approaches. One may, for example, use the 2D camera of the microscope, and via image processing determine the rim of the surgical cavity. One may then change the imaging angle and calculate for each angle the area included in the rim. The vertical position is when the area is maximized. Artificial Intelligence could help to further improve this approach. For example, one may perform a 3D scan of the surgical cavity, and then calculate the optimal imaging axis as the geometrical axis of the cavity space. 3D scanning could be performed with different ways such as: stereo camera, 3D camera (e.g. time of flight (TOF), pattern projection), or scanning of working distance and use contract mapping.

As opposed to conventional approaches, where a dedicated set of tools (e.g. a tubular retractor with attached 3D-position star) is used to provide guidance to a robotic arm which automatically performs alignment, the embodiments described before may achieve the same result or a better result without the use of special retractors and/or other 3D-position tools/markers.

Claim 1:
A controller (<NUM>) for a microscope (<NUM>), the controller being configured to:
receive image data representing a wound (<NUM>), the image data comprising multiple images of the wound (<NUM>) taken along different directions;
determine a line of sight (<NUM>, <NUM>) towards a bottom (<NUM>) of the wound (<NUM>) using the image data comprising;
calculating an area of the wound (<NUM>) within each image; and
determining the line of sight (<NUM>, <NUM>) based on the direction corresponding to the image having the wound (<NUM>) with the greatest area; and
output a control signal for the microscope (<NUM>), the control signal instructing the microscope (<NUM>) to align its optical axis (<NUM>) with the line of sight (<NUM>, <NUM>).