Patent Description:
In microscopy, a subject to be viewed with the microscope is illuminated, typically by means of a specific illumination source. Document <CIT> describes a microscopy arrangement in which a light detector is used to measure the intensity of the illumination of the subject. Illumination intensity is then controlled depending on the working distance of the subject with respect to the microscope. In this way, intensity of the illumination of the subject can be reduced when the working distance becomes smaller in order to prevent damage or injury of the subject due to high (or too high) illumination intensity. A similar approach is disclosed in <CIT>. Document <CIT> describes adapting illumination intensity based on a user's pupil activity in relation to a robot operator, and <CIT> describes a microscope with adaptable illumination.

According to the invention, a control system configured for use with a microscopy arrangemen, a microscopy arrangement and a method for adapting illumination intensity with the features of the independent claims are proposed. Advantageous further developments form the subject matter of the dependent claims and of the subsequent description.

The present invention relates to a control system according to claim <NUM>. Preferably, such microscope is a surgical microscope. In this case, the subject typically comprises or is human tissue or parts of the human body. The microscope might also be a microscope which is used for observing samples (as the subject) which could suffer from light irradiation which is higher than needed to acquire the expected information from the sample (e.g. a sample which tends to bleaching while fluorescence excitation). The control system is configured to receive information relating to an actual value of a pupil diameter of an eye of a user of the microscope, and to determine, based on that received information, a control value for the illumination intensity. According to the invention, the control system is also configured to adapt, by means of controlling an illumination source, the illumination intensity based on the control value.

The information relating to an actual value of the pupil diameter can be acquired, for example, by means of a pupil capturing system for acquiring such information. In particular, an eye tracking or eye gaze system might be used for this. In general, however, any imaging system for imaging an eye of the use might be used. From an image of the eye - which includes the pupil - the actual value of the diameter of the pupil can be determined, e.g., using image processing or image analyzing methods. Such pupil capturing system can be arranged at an appropriate position at the microscope, as will be discussed in more detail later.

Considering the actual value of the diameter of a user's eye allows a very simple but effective adjustment of the illumination intensity. This idea is based on the fact that the pupil diameter of a person's eye (normally) depends on the light or light intensity reaching the eye (e.g. on environmental brightness). With the intensity (or brightness) increasing, the pupil diameter decreases and vice versa. Of course, minimum and maximum pupil diameter values of a human eye have to be considered. Typical values are a minimum of <NUM> and a maximum of <NUM>, depending, however, on the specific person. Thus, if an actual value of the pupil diameter of a user's eye looking to an eyepiece of the microscope, for example, is relatively small, the illumination intensity can be reduced. The pupil diameter will then increase but the user will still experience a proper illumination of the subject The main effect is that the energy impact on the subject (or tissue) is reduced and, thus, a possible harm to the subject can be prevented.

A further effect is that the user's eye can more easily adapt to the environment which typically is relatively dark with respect to the illuminated subject Thus, if the user changes his view from the eyepiece to the environment and back to the eyepiece, his eyes do not have to adapt much. This also helps to reduce potential fatigue of the user's eyes which is of particular relevance in surgeries using a surgical microscope.

Preferably, the control system is further configured to determine the control value for the illumination intensity such that the actual value of the pupil diameter is maximized. In this way, the illumination intensity and thus the energy impact to the subject can be reduced without affecting the experience of the user. For example, the control system can be configured to control the pupil diameter within a closed loop control. This helps to find the lowest possible illumination intensity without any major effect to the user experience.

It is also of advantage, if the control system is configured to determine the control value for the illumination intensity such that the actual value for the illumination intensity does not fall below a minimum value. In this way a sufficient illumination can be achieved such that, e.g., every detail in the subject still can be recognized. Note that, depending on the specific minimum value, this can result in that the maximum pupil diameter will not be reached. In other words, this can (but does not need to) mean that maximizing the pupil diameter results in choosing the minimum value for illumination intensity. The eventual behavior will depend on the specific circumstances.

Further, the control system is, preferably, configured to determine the control value for the illumination intensity such that a condition for at least one parameter of an imaging system, configured for imaging the subject to be viewed, is fulfilled. Such imaging system might be a camera used with or integrated into the microscope. Then, a display means might be provided for a user to view the subject, imaged by the imaging system. Such imaging system and display means can be provided instead of or in addition to eyepieces as will be explained in more detail later. The imaging system (camera) typically requires specific parameters to be fulfilled in order to acquire a proper image (e.g., a live image) and/or to provide a proper view of the subject These parameters are, for example, an integration time, a frame rate, a signal to noise ratio, a gain, and a gamma value. It is to be noted that this requirement typically results in a minimum value for the illumination intensity. This minimum value can (but does not need to) coincide with the minimum value mentioned before.

The invention also relates to a microscopy arrangement, comprising a microscope (preferably, a surgical microscope), an illumination source, a pupil capturing system for acquiring information relating to a pupil diameter of a user's eye, and a control system as described before. As already mentioned the microscope, preferably, includes an eyepiece configured for viewing the subject Then, the pupil capturing system is arranged at the eyepiece and configured for acquiring information relating to a pupil diameter of the user's eye, when the user is looking into the eyepiece. Also, the pupil capturing system can be located (or arranged) inside the optical system of the microscope, i.e., the pupil capturing system would be integrated in the optical system. Depending on the microscope or the use it is intended for, the eyepiece can be configured for a single eye or for both eyes of a user. The latter is typical for a surgical microscope because it allows a stereo view of the subject The specific position of the pupil capturing system should be chosen such that the eye of the user can be imaged or captured at the time the user is looking into the eyepiece. For example, a camera included in the pupil capturing system can be integrated into a rim of the eyepiece or next to the center opening of the eyepiece.

Further, the microscopy arrangement, preferably, includes an imaging system (like a camera) configured for imaging the subject, and display means configured for displaying an image of the subject (which is provided by the imaging system; the display means can include display means integrated into the microscope). Then, the pupil capturing system is arranged at the display means and configured for acquiring information relating to a pupil diameter of the user's eye, when the user is looking at the display means. Although the pupil diameter is not or not directly associated with the actual illumination intensity of the subject, this configuration also allows reducing the illumination intensity in order to minimize the energy impact to the subject. Reducing the illumination intensity will not or not much influence the brightness of the display means the user is looking at Thus, the pupil diameter of the user's eye will not be increased, which in turn results in further decrease of the illumination intensity. This, eventually, results in reaching the minimum value of the illumination intensity such that the parameters of the imaging system are still fulfilled as mentioned before.

A further control (e.g., also by means of the control system) might be provided for controlling and/or adjusting the brightness of an environmental illumination (e.g. in a surgery room) and/or for controlling the brightness of the illumination being used for the display means. Adapting brightness of the environmental illumination and/or brightness of a display mounted in the (operation) room and/or brightness of the display in the microscope, in order to keep the pupil diameter at the (ideally) same (large) diameter, helps to reduce eye fatigue while changing between the different views.

It is to be noted that the display means can comprise display means integrated into the microscope. In this case, arrangement of the pupil capturing system at the display means can mean that the pupil capturing system is arranged at the microscope or - if an eyepiece is present - at the eyepiece. In case of a so-called hybrid microscope, including an eyepiece and integrated display means, which can be looked at via the eyepiece, it might be sufficient to have only one pupil capturing system, e.g., being arranged at the eyepiece.

Such minimum illumination intensity results in the mentioned reduced energy impact to the subject (or tissue) and, in addition, in a reduced brightness in the environment As mentioned before, the environment typically is relatively dark. Major sources of illumination are, typically, only the illumination source for illuminating the subject and the display means. Such situation might occur, e.g., in a surgical room with a surgeon using a surgical microscope. Such reduced illumination or brightness deviations between surgical field, display means and environment result in reduced fatigue of not only the user's (or surgeon's) eyes but also all persons located in the operation room, which change their view from the illuminated area to the environment.

This effect is of particular relevance, if the microscope includes an eyepiece and has integrated display means (so-called hybrid microscope) as mentioned before. When using such microscope, the user typically switches between a real view of the subject using the eyepiece and a virtual view via the display means (which might provide a different view of the subject than the real view does). Depending on the specific configuration of the microscope, such display means might be viewed via the eyepiece, i.e., there can be provided a mechanism which allows exchanging the direct visual view to the subject with the view to the display means. A further option is to provide an overlay of the imaged subject being captured by the imaging system (and, for example, digitally modified) with the direct visual view. In this specific option, the brightness ratio of these two images is important to achieve an acceptable performance. The display means' brightness is typically lower than the direct visual brightness. The reduction of the direct visual brightness is increasing the performance, i.e. the eventual quality of the image, of this option.

Preferably, the display means comprise at least one of: a display means being integrated into the microscope (as mentioned before), a display means for displaying two- and/or three-dimensional image information, a stereoscopic display means, an auto-stereoscopic display means, a multi-view display means, a holographic display means, and display means including a display for each of a user's eyes. It is to be noted that integrated display means can be embodied as any of the other types of display means mentioned.

A further aspect to be mentioned is that a microscope has a specific exit pupil diameter, its value typically being between <NUM> and <NUM>. With the pupil diameter of the user's eye being maximized (or, at least, being brought to a value larger than the exit pupil diameter of the microscope) the position and/or orientation of the user or his eye (or eyes) with respect to the eyepiece of the microscope might vary within specific limits without any major effect on the view through the microscope; the light rays emerging from the eyepiece still lie within the border of the pupil of the user's eye. For a surgeon for example, this means that he does not have to stay in a rigid or inflexible position when looking into the eyepiece. In other words, this effect enhances the ergonomics for a surgeon using a surgical microscope.

The invention also relates to a method for adapting an illumination intensity for illuminating a subject to be viewed with a microscope, comprising the following steps: determining information relating to an actual value of a pupil diameter of an eye of a user of the microscope; determining, based on the information relating to the actual value of the pupil diameter, a control value for the illumination intensity; and adapting the illumination intensity according to the control value. Preferably, the control value for the illumination intensity is determined such that the actual value of the pupil diameter is maximized, preferably within a closed loop control for controlling the pupil diameter.

With respect to further preferred embodiments or details and advantages of the microscopy arrangement and the method, it is also referred to the remarks for the control system above, which apply here correspondingly.

The invention also relates to a computer program with a program code for performing a method according to the invention when the computer program is run on a processor or on a control system according to the invention.

Further advantages and embodiments of the invention will become apparent from the description and the appended figures.

It should be noted that the previously mentioned features and the features to be further described in the following are usable not only in the respectively indicated combination, but also in further combinations or taken alone, without departing from the scope of the present invention.

In <FIG>, a very schematic overview of a microscopy arrangement <NUM> according to the invention in a preferred embodiment is shown. The microscopy arrangement <NUM> comprises a microscope <NUM>, e.g., a surgical microscope, with an eyepiece <NUM> for a user to look at the subject <NUM> to be viewed, an objective <NUM>, and an illumination source <NUM> for illuminating the subject <NUM>.

Further, the microscopy arrangement <NUM> includes an imaging system <NUM>, e.g., a camera, for imaging the subject <NUM> and display means <NUM> configured for displaying an image (this might be a live or real-time image), based on image data provided by the imaging system <NUM>. Both, the imaging system <NUM> and the display means <NUM> are, preferably, integrated into the microscope <NUM>. Thus, the microscope <NUM> can be called a hybrid microscope. It is to be noted that in <FIG> the display means <NUM> are also shown in an enlarged view at the right side, showing a front side at which a user is looking at.

Further, the microscopy arrangement <NUM> comprises a pupil capturing system <NUM>, e.g., an eye tracking system, arranged at the eyepiece <NUM>, and a control system <NUM>. By means of the pupil capturing system <NUM>, information <NUM> relating to an actual value of a pupil diameter d of an eye <NUM> of a user of the microscope <NUM> (the user looking into the eyepiece <NUM>) can be determined and transmitted (as a signal) to the control system <NUM>. By means of the control system <NUM>, a control value <NUM> for the illumination intensity can be determined or calculated, based on the information <NUM>. Further, the illumination intensity of the illumination source <NUM> can be adapted according to the control value <NUM>.

In <FIG> a flow diagram of a method according to the invention in preferred embodiment is shown schematically. The method can be performed, e.g., with or by means of the control system <NUM> or the microscopy arrangement <NUM> shown in <FIG>.

In step <NUM>, information <NUM> relating to the actual value of a pupil diameter d of the eye of the user of the microscope <NUM> is determined. For example, the actual value can be directly measured by means of the pupil capturing system <NUM>. Also, an image of the eye or the pupil can be acquired and used as the information <NUM>. In step <NUM>, information <NUM> relating to an actual value (with might be a fixed value) of the exit pupil diameter of the microscope is determined or read out In step <NUM>, parameters <NUM> of the imaging system <NUM> (or camera) are determined or read out. These parameters <NUM> can include, e.g., an integration time and a frame rate of the imaging system <NUM>.

This information <NUM>, <NUM> and parameters <NUM> (or their respective values) are received by the control system <NUM> and, in step <NUM>, the information and parameters are processed and/or compared in order to determine a control value <NUM> for the illumination intensity. As mentioned before, the illumination intensity should be chosen such that the pupil diameter d of the user's eye is maximized, however, considering the parameters <NUM> of the imaging system to be fulfilled. For example, specific minimum illumination intensity is required for achieving a sufficient integration time. In step <NUM>, the illumination intensity, provided by the illumination source, is adapted or set according to the control value <NUM>. In <FIG>, pupil diameters d of eyes <NUM> for different illumination intensities for a subject <NUM> are shown in order to further illustrate an idea of the invention. In the upper part of <FIG>, images of both eyes (each indicated by reference numeral <NUM>) of a user looking into an eyepiece (intended for use with both eyes, i.e., a binocular eyepiece) are shown. The pupil diameters d of the eyes (their actual values should be equally sized in both eyes) are indicated with a circle. This shows, for example, how a pupil capturing system might acquire images of the eyes and how the pupil diameters can be identified and determined (e.g., by means of image processing, identifying circularly shaped black regions).

In the lower part of <FIG>, a subject <NUM> (it is, by means of example, a graphic including different numbers as used for eye sight tests) is illuminated with relatively high illumination intensity. As a consequence of the high illumination intensity, the values of the pupil diameters are relatively small.

In <FIG>, the same situation as in <FIG> is shown, with the illumination intensity for the subject <NUM>, however, being less than in <FIG>. Accordingly, the values of the pupil diameters of the eyes are larger than in <FIG>. This shows that the pupil diameter or its actual value is indicative of the illumination intensity and, thus, can be used in order to adapt - in particular, reduce - the illumination intensity for the subject in order to minimize energy impact to it.

Some embodiments relate to a microscope comprising a system as described in connection with one or more of the <FIG>. Alternatively, a microscope may be part of or connected to a system as described in connection with one or more of the <FIG>.

<NUM> shows a schematic illustration of a microscopy arrangement <NUM> configured to perform a method described herein. The microscopy arrangement <NUM> comprises a microscope <NUM> and a computer or control system <NUM>. The microscope <NUM> is configured to take images and is connected to the computer system <NUM>. The computer system <NUM> is configured to execute at least a part of a method described herein. The computer system <NUM> may be configured to execute a machine learning algorithm. The computer system <NUM> and microscope <NUM> may be separate entities but can also be integrated together in one common housing. The computer system <NUM> may be part of a central processing system of the microscope <NUM> and/or the computer system <NUM> may be part of a subcomponent of the microscope <NUM>, such as a sensor, an actor, a camera or an illumination unit, etc. of the microscope <NUM>.

The computer system <NUM> may be a local computer device (e.g. personal computer, laptop, tablet computer or mobile phone) with one or more processors and one or more storage devices or may be a distributed computer system (e.g. a cloud computing system with one or more processors and one or more storage devices distributed at various locations, for example, at a local client and/or one or more remote server farms and/or data centers). The computer system <NUM> may comprise any circuit or combination of circuits. In one embodiment, the computer system <NUM> may include one or more processors which can be of any type. As used herein, processor may mean any type of computational circuit, such as but not limited to a microprocessor, a microcontroller, a complex instruction set computing (CISC) microprocessor, a reduced instruction set computing (RISC) microprocessor, a very long instruction word (VLIW) microprocessor, a graphics processor, a digital signal processor (DSP), multiple core processor, a field programmable gate array (FPGA), for example, of a microscope or a microscope component (e.g. camera) or any other type of processor or processing circuit. Other types of circuits that may be included in the computer system <NUM> may be a custom circuit, an application-specific integrated circuit (ASIC), or the like, such as, for example, one or more circuits (such as a communication circuit) for use in wireless devices like mobile telephones, tablet computers, laptop computers, two-way radios, and similar electronic systems. The computer system <NUM> may include one or more storage devices, which may include one or more memory elements suitable to the particular application, such as a main memory in the form of random access memory (RAM), one or more hard drives, and/or one or more drives that handle removable media such as compact disks (CD), flash memory cards, digital video disk (DVD), and the like. The computer system <NUM> may also include a display device, one or more speakers, and a keyboard and/or controller, which can include a mouse, trackball, touch screen, voice-recognition device, or any other device that permits a system user to input information into and receive information from the computer system <NUM>.

Claim 1:
A control system (<NUM>) configured for use with a microscopy arrangement (<NUM>) for adapting an illumination intensity for illuminating a subject (<NUM>) to be viewed with a microscope,
the microscopy arrangement (<NUM>) comprising the microscope (<NUM>), an illumination source (<NUM>) for illuminating the subject (<NUM>), and a pupil capturing system (<NUM>) for acquiring information (<NUM>) relating to a pupil diameter (d) of a user's eye,
the control system (<NUM>) being configured to:
receive information (<NUM>) relating to an actual value of the pupil diameter (d) of the eye (<NUM>) of the user of the microscope (<NUM>),
determine, based on the information (<NUM>) relating to the actual value of the pupil diameter (d), a control value (<NUM>) for the illumination intensity, and
to adapt, by means of controlling the illumination source (<NUM>), the illumination intensity based on the control value (<NUM>).