Patent Description:
In <CIT>, a method for the merged display of first image information captured using a first imaging device with second information captured using a second imaging device is shown, whereby said image information relates to the same area of examination in a hollow organ such as a prostate gland, bowel, gullet, bronchial tubes or vascular system. First, two-dimensional images are obtained from the area under examination using a continuous controlled movement of a medical instrument such as a catheter, an endoscope, a laparoscope or a colonoscope. The individual images are compiled into a three-dimensional fluorescence data record. The fluorescence data record is then registered with a three-dimensional data record, for example, taken by an MR or CT, 3D angiographic or 3D ultrasound system.

In <CIT>, an apparatus is described, which acquires tomographic images at depths from zero to several millimeters from a surface of a region. A probe is inserted through a forceps channel of an endoscope and rotated along its longitudinal axis. Thus, tomographic images are obtained, again from a hollow organ. The probe allows the acquisition of optical tomographic images generated using optical coherence tomography, ultrasound-modulated-light tomography and ultrasound-modulated-fluorescence tomography as well as ultrasonic tomography.

<CIT> discloses a tomographic fluorescent imaging device for imaging fluorophores in biological tissues has a scanned laser for scanning the tissue and a camera for receiving light from the biological tissue at an angle to the beam at a second wavelength ten or more nanometers greater in wavelength than the wavelength of the laser. Images are obtained at each of several positions of the beam. An image processing system receives the series of images, models a path of the beam through the tissue, and determines depth of fluorophore in tissue from intersections of the modeled path of the beam and the path of the received light.

It is known in the prior art to inject a fluorophore into tissue that is observed by an optical imaging device such as a microscope or an endoscope. The fluorophore is configured to mark only specific tissue. For example, the fluorophore may be adapted to remain in the blood stream only during the observation time so that only blood vessels are marked by the fluorophore. Another fluorophore may be provided in the form of a precursor material, which reacts only with tumor cells to produce a fluorophore which only marks cancer. Thus, fluorescence imaging can provide useful diagnostic information. However, fluorescence typically does not provide anatomical information, i.e. tissue color appearance. Conversely, visible-light image provides anatomical information but not fluorescence diagnostic information. Pseudocolor is a convenient way to present in a single image the information of those two different image types: visible-light and fluorescence. It is known in the prior art to superpose or merge the visible-light images and the fluorescence images.

The use of such an image superposition device and method greatly facilitates diagnosis and surgery. However, there is a need for further improving the technology thus to further facilitate diagnosis and surgery.

It is therefore the object of the present invention to provide a device and method, which further improves the existing devices and methods.

This object is solved by the subject-matter of the independent claims.

This goal is achieved by a microscope according to claim <NUM> which comprises, inter alia, an input interface and an output interface, the input interface being configured to receive subsequent sets of input image data and subsequent sets of auxiliary input data, the output interface being configured to output subsequent sets of output image data at an output data rate, the image processor being connected to the input interface for receiving the sets of input image data and auxiliary input data and to the output interface for outputting the output image data, the microscope further comprising an image processor wherein the image processor is connected to the input interface for receiving the subsequent sets of the input image data and the auxiliary input data, and to the output interface for outputting the subsequent sets of the output image data , the image processor further being configured to generate, in real time, subsequent sets of pseudo-color image data depending on a content of at least one of the auxiliary input data and the input image data, wherein the output image data are three-dimensional and wherein the image processor is configured to generate pseudo-color image data at different depth layers of the three-dimensional output image data depending on the content of the auxiliary input data.

Another device not forming part of the invention is a medical observation device which comprises an input interface, an image processor and an output interface, the input interface being configured to receive subsequent sets of input image data and subsequent sets of auxiliary input data, the output interface being configured to output subsequent sets of output image data at an output data rate, the image processor being connected to the input interface for receiving the sets of input image data and auxiliary input data and to the output interface for outputting the output image data, the image processor being further configured to generate subsequent sets of pseudo-color image data depending on a content of at least one of the auxiliary input data and the input image data and to generate at least one synthetic pseudo-color pattern field within the pseudo-color image data depending on the content of at least one of the input image data, the auxiliary input data and the pseudo-color image data, the pseudo-color pattern field comprising at least one of a temporally and a spatially changing pattern, and the image processor being configured to merge the pseudo-color image data and at least one section of the input image data to generate the output image data containing the pseudo-color pattern field.

The objective of the invention is also achieved by a method for displaying microscope images according to claim <NUM>, comprising, inter alia, the steps of receiving subsequent sets of input image data, receiving subsequent sets of auxiliary input data, generating subsequent sets of pseudo-color image data in real time depending on the content of at least one of the auxiliary input data and the input image data, merging the pseudo-color image data with the input image data to obtain three-dimensional output image data, wherein the pseudo-color image data are generated at different depth layers of the three-dimensional output image data depending on the content of the auxiliary input data, and displaying the output image data at an output data rate.

Another method not forming part of the invetion for displaying medical images comprises the steps of receiving subsequent sets of input image data, receiving subsequent sets of auxiliary input data, generating subsequent sets of pseudo-color image data depending on the content of at least one of the auxiliary input data and the input image data, generating at least one coherent pattern field within the pseudo-color image data depending on the content of at least one of the input image data, the auxiliary input data and the pseudo-color image data, the pattern field having at least one of a spatial and a temporal modulation, merging the pseudo-color image data with the input image data to obtain output image data, and displaying the output image data with the pseudo-color pattern field.

In the following, further aspects of the invention are described. Each of the further aspects described herein below, has its own advantage and technical effect.

According to one aspect, the image processor may be configured to generate a synthetic pseudo-color pattern field within the pseudo-color image data depending on the content of at least one of the input image data, the auxiliary input data and the pseudo-color image data, the pseudo-color pattern field comprising at least one of a temporally and/or spatially changing pseudo-color pattern, and the image processor being configured to merge the pseudo-color image data and at least one section of the input image data to generate the output image data containing the pseudo-color pattern field.

The use of the temporally and/or spatially modulated pseudo-color pattern allows the merging of further data in particular representing additional data sources to the output image. Further, the use of a pseudo-color pattern allows visualizing data in the output image data that would be otherwise be imperceptible in the pseudo-color image.

As the pattern field is both of pseudo-color and synthetic, it is immediately distinct from the input image, which is preferably used as the background in the output image data and based on visible-light image data.

In the context of this invention, a pattern designates a discernible regularity, in which elements repeat in a predictable manner. The regularity and predictability means that the pattern is not reliant on human cognitive processes. Rather, the pattern can be automatically discerned and discovered using pattern recognition algorithms. A pseudo-color is a color which does not occur naturally in that context, such as a neon color in biological tissues. A color in the context of this disclosure is marked as a particular combination of wavelengths of light and thus independent of human cognitive processes.

For example, the pattern field may be filled with a spatial pattern such as a hatching or with a regular repetition in time and/or space of a template. A template may, in a pattern having spatial modulation, consist of geometric forms such as dots, tiles, symbols and/or pictures. In a pattern having temporal modulation, the pattern consists of a regular repetition of a set of different images over time. A typical example of such a pattern is a hatching or a bitmap template of simple geometric forms such as circles, polygons and/or waves.

In a pattern having both temporal and spatial modulation, different patterns having spatial modulation are repeatedly displayed. The subsequent patterns in a temporally modulated pattern may be similar to each other, e.g. spatially displaced relative to each other but otherwise identical, to represent a direction of motion. For example, subsequent hatchings of a temporally modulated pattern within one cycle of the variation rate may all be shifted geometrically with respect to each other by the same amount. This can be used to produce a temporally modulated pattern which indicates a motion direction. A propagating wave pattern may, e.g. be used to indicate the direction of a fluid flow, such as a blood or lymph flow.

The pseudo-color pattern preferably extends over at least one coherent or contiguous section of the pseudo-color image data and thus forms a pseudo-color pattern field. Within a pseudo-color pattern field, the content of at least one of the image input data and the auxiliary input data satisfies preferably the same condition. For example, if the content of the auxiliary input data and/or the pseudo-color image data exceeds an upper threshold, this may represent saturation in the data. These data may then be represented by the pattern field so that they are marked as being unreliable. The same approach can be used in connection with a lower threshold, which may represent data which are below a sensitivity threshold and thus may be considered unreliable. An example of a temporal modulated pattern may be a pseudo-color image or a coherent part thereof, which is filled with a pseudo-color and in which the pseudo-color is switched on and off, or changes at least one of brightness, hue and saturation according to the variation rate.

A temporally modulated pattern may have a variation rate in which a cyclic repetition of alternating patterns occurs. In order to be clearly visible, the variation rate should be smaller than the flicker fusion rate, i.e. in particular be smaller than <NUM>.

According to the invention, the image processor is configured to determine the variation rate depending on the content of at least one of the input image data and the auxiliary input data. The variation rate may be automatically varied by the image processor over time. In particular, the variation over time may be dependent on the content of at least one of the input image data, the auxiliary input data and the pseudo-color image data. If, for example, the auxiliary data are ultrasound data, the variation rate may depend on the momentary flow velocity, or on a frequency and/or amplitude of velocity changes, e.g. caused by a pulse in a blood flow.

If the pseudo-color pattern field is a temporal pattern, wherein the image processor is configured to generate the temporal pattern with a variation rate, it is of advantage if the output data rate is larger than the variation rate. Thus, the temporal variation of the pattern can be smoothly displayed. The output data rate usually corresponds to a frame rate, and thus should be higher than the flicker fusion rate in order to produce a smooth sequence of images. In contrast, the variation rate is configured to produce a perceptible motion.

The pseudo-color image data may also comprise symbols, such as arrows, letters and numerals.

The input interface may, according to another aspect of the invention, comprise an input section configured to receive at least one of ultrasound data and fluorescence image data as auxiliary input data. The ultrasound data may be a one-dimensional data field as generated by the recordings of an ultrasound microphone. The ultrasound data may in addition or alternatively be multi-dimensional such as produced by an ultrasound sensor head or scanner. In particular, the ultrasound data may be two-dimensional or three-dimensional (in spatial dimensions) and contain, at each data point representing a spatial location additional data, such as a velocity and direction of flow or movement.

The image processor may comprise spatial alignment module, which is adapted to spatially align the input image data and any auxiliary input image data so that spatial features in both data are spatially congruent.

If for example the input image data are image data received from a visible-light camera and the auxiliary input data are fluorescent-light image data received from a fluorescence-camera, where both the visible-light camera and the fluorescence camera have overlapping field of views, the spatial alignment module rectifies at least one of the images so that the spatial features are of the same size and orientation in both image data. This allows exact merging of the various image data in the output image data. The auxiliary input data in such a case may alternatively or additionally comprise data from sensors detecting other physical, preferably non-visible, quantities. These data may be rectified in the same way using the spatial alignment module as the fluorescence image data mentioned above.

If the fluorescence takes place in the NIR range, the fluorescence camera may be a NIR camera and the fluorescence image data may be NIR image data.

If the auxiliary data are image data from a fluorescence camera, in particular fluorescent-light image data, the temporal and/or spatial pattern may depend on the saturation and/or brightness of the fluorescence image data. The pseudo-color pattern field may further be generated where two different colors or wavelengths in the fluorescence-image data overlap and, as an optional further criterion, each color or wavelength exceeds a minimum intensity.

If the auxiliary input image data comprise a three-dimensional data set, the temporal and/or spatial pattern may depend on the location, in particular depth, at which the pre-determined condition for switching this pattern on is detected. This allows including data in the output image data which represent structural and/or physical features in a plane that is below what is represented in the input image data, such as a blood vessel or a bone beneath a tumor marked by a fluorophore.

According to another aspect, the image processor may be configured to generate subsequent sets of pseudo-color image data according to a temporal modulation scheme. The temporal modulation scheme is preferably stored in the image processor. The modulation scheme may comprise at least one of a time-varying alteration between sets of different pseudo-color image data representing different pseudo-colors, pseudo-color image data of different saturation and/or brightness, different pseudo-color image data representing cyclically changing patterns.

The modulation scheme may further comprise a propagating wave pattern, wherein, in the propagating wave pattern, at least one of speed, direction and wavelength depends on the content of the auxiliary input data.

The aspect of the output image data comprising three-dimensional image data and of the image processor being configured to generate and/or to process three-dimensional output image data, in which pseudo-color image data are generated at different depths or planes of the three-dimensional image data depending on the content of the auxiliary data may be realized independent of the generation of a pseudo-color pattern field.

Three-dimensional input image data may be generated by Z-stacking using either a visible-light camera or a fluorescence camera. The pseudo-color image data may thus be used to present three-dimensional features. The three-dimensionality of the image data may, as has already been explained above, also result from a three-dimensional ultrasound image.

The output image data may be, according to a further aspect of the invention, two-dimensional image data and at least one of the input image data and auxiliary input data may comprise three-dimensional data. The image processor is adapted to compute the two-dimensional output image data from the three-dimensional input data using pseudo-colors for features which are not in the plane of the two-dimensional output image data. The pseudo-color image data, in particular a pseudo-color pattern field thereof, may be used to display data content of a part of the three-dimensional input image and/or the three-dimensional auxiliary input data that is not within the plane rendered in the two-dimensional output data.

The use of a pseudo-color image in connection with three-dimensional input image or auxiliary input data is advantageous on its own, i.e. without using pseudo-color pattern fields.

The medical observation device may comprise at least one visible-light camera which is connected to the input interface for providing the input image data. The visible-light camera may provide three-dimensional input image data or two-dimensional input image data. The medical observation device may further comprise at least one sensor device configured to sense a physical quantity which is not electromagnetic radiation in the visible wavelength range, such as at least one fluorescence camera, at least one ultrasound sensor, at least one microradiation camera, at least one gamma ray camera, at least one Roentgen camera and/or at least one thermographic camera. These devices may be connected to the input interface for providing the auxiliary input data. The auxiliary input data may be one-dimensional, two-dimensional, three-dimensional or provide sets of auxiliary input data of different dimensionalities.

The medical observation device is preferably configured to discern, process and display more than one fluorophore, i.e. more than one different fluorescent spectra. For this, at least one of a fluorescence camera, a light source and a visible-light camera may be provided with a filter arrangement for selectively blocking and/or transmitting the fluorescent spectra. The pseudo-color image comprises in such a case preferably at least two different pseudo-colors, wherein each pseudo-color is assigned to a different fluorescence spectrum, i.e. to a different fluorophore.

The image processor and its constituents may be hardware devices or may be, at least partly, represented by software modules executed in a multi-purpose computer or processor.

The invention may employ the merging of the pseudo-color image data with the input image data as described in the parallel application <CIT>.

In the following, the invention is described exemplarily with reference to the accompanying figures. In the figures, elements which correspond to each other with respect to at least one of function and design are assigned identical reference numerals.

The combination of features shown in the figures and described below is only an example. Individual features can be added or omitted if the technical effect of that particular feature is described above is needed or not necessary for a particular application.

First, the structure of a medical observation device <NUM> which includes the invention is described with reference to <FIG>.

Just by way of example, the medical observation device <NUM> is shown in <FIG> to be a microscope. The medical observation device <NUM> may also be an endoscope.

The medical observation device <NUM> is used to inspect an object <NUM>, such as live tissue <NUM>, for diagnostic or surgery purposes. In the object <NUM>, one or more fluorophores may be present.

Upon excitation by certain wavelengths of a light source <NUM>, the fluorophores emit fluorescence light at different peak emission wavelengths, e.g. in the NIR range. If two or more fluorophores are used, they may emit light of different peak emission wavelengths, or just have different spectra. Different fluorophores may be used to mark different types <NUM>, <NUM> of live tissue <NUM>, so that these types <NUM>, <NUM> may be discerned by the different fluorescent wavelengths they emit. The light source <NUM> may also serve as illumination for observation of the object in the visible-light range.

For example, a first fluorophore <NUM>, which is schematically represented in <FIG> by one type of hatching, may be used for marking specifically one type <NUM> of live tissue <NUM>, for example blood vessels. A second fluorophore <NUM>, represented in <FIG> by another type of hatching, may e.g. mark exclusively another type <NUM> of live tissue <NUM>, for example tumor cells.

The medical observation device <NUM> may comprise an optical system <NUM>, such as a zoom magnifying lens. In the case of an endoscope, the optical system <NUM> may also comprise fiber optics. The optical system <NUM> is, in operation of the medical observation device <NUM> and its medical image superposition device <NUM>, directed onto at least a section of the live tissue <NUM>. The visible section of the object <NUM> is determined by the field of view <NUM> of the optical system <NUM>. Light reflected off the live tissue <NUM> together with any fluorescent light emitted from the at least one fluorophore <NUM>, <NUM> is collected by the optical system <NUM> and directed to one or more beam splitter system <NUM> which contains at least one beam splitter.

A part <NUM> of the light <NUM> is directed to a visible-light camera <NUM> by the at least one beam splitter system <NUM>. Another part <NUM> of the light <NUM> may be directed to a fluorescence camera <NUM>. Another part <NUM> of the light <NUM> may be directed to an ocular <NUM> which may be monocular or binocular.

The ocular <NUM> may be directed onto a transmissive display <NUM>, which is arranged between the ocular <NUM> and the at least one beam splitter system <NUM>. The transmissive display <NUM> allows the part <NUM> of the light <NUM> to pass and to superpose a picture, which is currently displayed on the display <NUM>, onto the image provided by the optical system <NUM>.

Alternatively, the display <NUM> may not be transmissive. In this case, the display <NUM> may render a live view of the object <NUM> with additional information or display any other information that an operator of the medical image superposition device <NUM> or the medical observation device <NUM> requests. In particular, the background image, which in the case of a transmissive display <NUM> would be provided directly by the optical system <NUM>, may be instead provided in real time by the visible-light camera <NUM>.

The medical observation device <NUM> may further comprise at least one sensor <NUM>, e.g. an ultrasound or blood-flow sensor, which may be in contact with the object <NUM>.

In addition to the at least one sensor <NUM> at east one further sensor device <NUM> may be provided. The sensor device <NUM> is preferably a non-optical sensing device that is configured to sense non-visible physical quantities such as microradio, gamma, Roentgen, infrared, or sound data. The non-optical sensing device <NUM> provides two- or three-dimensional data. Examples for the non-optical sensing device <NUM> are microradiography cameras, ultrasound sensor heads, thermographic cameras, gamma ray cameras and Roentgen cameras.

The non-optical sensing device preferably has a sensor field <NUM>, which overlaps the field of view <NUM> so that the data captured by the optical system <NUM> and the non-optical sensing device <NUM> represent different physical quantities from the same area of the object <NUM>.

In order to separate the light in the visible-light range from the light in the fluorescence wavelengths and in order to avoid reflections, a filter arrangement <NUM> may be used preferably immediately in front of at least one of the light source <NUM>, the beam splitter system <NUM>, the visible-light camera <NUM>, the fluorescence camera <NUM>, the ocular <NUM> and the display <NUM>.

The visible-light camera <NUM> is connected via a data connection <NUM> to an input interface <NUM> of a medical image superposition device <NUM>. The medical superposition device may be realized by hardware dedicated to carry out specific functions, such as an ASIC, by software executed by a general-purpose computing device, or a combination of both.

The input interface <NUM> receives, in operation of the medical image superposition device <NUM> and the medical observation device <NUM> subsequent sets <NUM> of input image data <NUM>. The input image data <NUM> is two-dimensional and organized to contain pixels <NUM>. In the embodiment of <FIG>, the subsequent sets <NUM> of the input image data <NUM> correspond to the sequence of visible-light image data <NUM> provided by the camera <NUM>.

The input interface <NUM> is further configured to receive auxiliary input data <NUM>, which may be one-dimensional auxiliary input data <NUM>, sets <NUM> of two-dimensional auxiliary input data <NUM> or three-dimensional auxiliary input data <NUM> consisting of several planes <NUM> of two-dimensional data. Of course, the auxiliary input date <NUM> may also comprise data of a higher dimensionality, e.g. if hyperspectral cameras are used or ultrasound sensor heads, which output three-dimensional pictures, which contain additional physical data such as blood flow velocity and direction for each pixel or voxel.

The fluorescence camera <NUM> may transmit, via the data connection <NUM>, subsequent sets <NUM> of two-dimensionally auxiliary input data <NUM> which may in particular represent fluorescence image data <NUM> from the fluorescence camera <NUM>. The fluorescence image data <NUM> may have the same format as the input image data <NUM> but at a different resolution.

The sensor <NUM> may provide one-dimensional auxiliary input data e.g. in the form of a time sequence blood flow velocity <NUM> in the blood vessel <NUM>.

The sensing device <NUM> may provide subsequent sets <NUM> of two- or three-dimensional auxiliary input data <NUM> e.g. in planes <NUM> parallel to the depth direction <NUM> of the object <NUM>, i.e. to the viewing direction <NUM> of the medical observation device <NUM>.

The sensing device <NUM> may be connected to the input interface <NUM> of the medical superposition device <NUM> by a data connection <NUM>. Another data connection, not shown for simplicity's sake, may exist between the ultrasound sensor <NUM> and the input interface <NUM>. All data connections may comprise wired and/or wireless sections.

It is to be understood that auxiliary input data <NUM> can be received by the medical image superposition device <NUM> from any one of the described devices, e.g. only from the fluorescence camera <NUM> or only from the ultrasound sensor <NUM>, as well as any combination of such devices. Moreover, other devices may also be employed to provide auxiliary input data <NUM>, although not shown in <FIG>. For example, a thermographic camera, a microradiography camera, a Roentgen camera and/or a gamma ray camera may be used instead of or in addition to an ultrasound sensor and/or the fluorescence camera.

The medical image superposition device <NUM> comprises an image processor <NUM> which is configured to blend the input image data <NUM> from the visible-light camera <NUM> with the auxiliary input data <NUM> and provide output image data <NUM> which contain both at least parts of the input image data <NUM> and at least parts of the auxiliary input data <NUM>. The input image data <NUM> are used as background data whereas the auxiliary input data <NUM> are assigned at least one pseudo-color and/or a pseudo-color pattern before they are merged with the input image data <NUM>. The pseudo-color is manually or automatically by the image processor <NUM> selected from a list of colors which is not present in the image input date <NUM> and which preferably does not exist in tissue <NUM>, such as neon colors. Preferably, each different physical quantity is assigned a different pseudo-color and/or pseudo-color pattern.

For this, the image processor <NUM> comprises a pseudo-color image generator <NUM>. The pseudocolor image generator <NUM> assigns a pseudo-color to the auxiliary input data <NUM> depending on the content and/or source of the auxiliary input image data <NUM>. Preferably for each set <NUM> of image data <NUM>, the pseudo-color image generator <NUM> is configured to generate a corresponding set <NUM> of pseudo-color image data <NUM> in real time. The pseudo-color image data <NUM> are then blended with the input image data <NUM>.

The image processor <NUM>, in particular the pseudo-color image generator <NUM> is further configured to generate at least one pseudo-color pattern <NUM> within the pseudo-color image data <NUM>. The pattern <NUM> extends over a coherent section of the pseudo-color image data <NUM> and thus forms a pseudo-color pattern field <NUM>. The pattern <NUM> may comprise a temporal modulation and/or a spatial modulation of zones of pseudo-color within the pseudo-color image data <NUM>. A spatially modulated pattern has regularity in space, a temporally modulated pattern has regularity over time. Regularity means that the pattern repeats in a predictable way.

If the pattern <NUM> has temporal modification, it is generated by the pseudo-color image generator <NUM> to have a variation rate, which represents the time between successive repetitions of the pattern <NUM>.

The pattern <NUM> is generated in dependence of the content of the auxiliary input data <NUM>. For example, a pattern <NUM> may be generated instead of a solid pseudo-color by the pseudo-color image generator <NUM> in an area of the auxiliary input data <NUM>, in particular the fluorescent-light image data <NUM>, where a threshold, for example a brightness threshold, is not met. Different patterns <NUM> may be generated depending on how far the content is below the threshold. This is exemplarily described in the following with reference to <FIG>and <FIG> shows a pseudo-color pattern <NUM>' having spatial modulation, <FIG> show a pattern <NUM>" having temporal modulation, and <FIG> show a pattern <NUM>‴ having both temporal and spatial modulation, the temporal modulation being based on a temporal modulation scheme 87ʺʺ.

In <FIG>, a schematic representation of one set of pseudo-color image data <NUM> is given. The pseudo-color image data <NUM> contains at least one pseudo-color <NUM>, in this example two different pseudo-colors 89a and 89b, and two different pseudo-color pattern fields <NUM> containing different pseudo-color patterns <NUM>, having in this example only spatial modulation. One pseudo-color pattern field 88a for examples is built up of a repeating wave-form template and can be used to designate an area in the auxiliary input data <NUM>, e.g. from an ultrasound sensor, in which an increased liquid content has been discovered by automatically comparing the auxiliary input data <NUM> with a data library. Another pseudo-color pattem field 88b can be used to designate an area of the auxiliary input data <NUM>, in which the intensity is below or above a threshold. As the patterns <NUM> have only spatial modulation, they do not change over time. The extent of the pseudocolor pattern field <NUM>, however may change over time, as it depends on the content of the input image data <NUM> and/or the auxiliary input data <NUM>.

<FIG> show a pseudo-color pattern <NUM> with temporal modification. In this example, the temporal modification is a simple blinking operation over time. In <FIG>, the pseudo-color pattern field <NUM> contains a first pseudo-color 89a. In <FIG>, the same pseudo-color pattern field <NUM> is shown to contain a second pseudo-color 89b (or no pseudo-color) at a later point of time, which has replaced the first pseudo-color 89a. At again a later point in time, shown in <FIG>, the pseudo-color pattern field <NUM> is restored to the state shown in <FIG>. Thus, a repetitive pattern is displayed over time. The time-period between successive repetitions is determined by the variation rate of the temporally modulated pseudo-color pattern <NUM>. The variation rate is determined in the pseudo-color image generator <NUM> depending on the content of the input image data <NUM> and/or the auxiliary input data <NUM>. In order to have a clearly visible temporal modulation, the variation rate is smaller than the flicker fusion rate, in particular less than half the flicker fusion rate.

The temporal modulation shown in <FIG> may for example be assigned to auxiliary input data, in which the intensity is above a threshold, and which if assigned to a static pseudo-color, would, not be discernible in the pseudo-color image data <NUM>, e.g. because saturation has been reached. In such a case, instead of showing a second pseudo-color 98b, the pseudo-color can be switched of in the pseudo-color image data <NUM>.

Another example, where the temporal modulation shown in <FIG> may be used is an area, where otherwise two static pseudo-colors would have been assigned. Using the temporal modification, it can be clearly indicated that the pseudo-color pattern field <NUM> results from auxiliary input data of which the content satisfies the criteria for the assignment of more than one pseudo-color <NUM>. Of course, the temporal modulation shown in <FIG> can be extended to contain more than two pseudo-colors 89a, 89b in sequence.

The pseudo-color pattern field <NUM> may comprise a pseudo-color pattern field <NUM> which has both temporal and spatial modulation. This is shown in <FIG>, which depict a pattern <NUM> comprising at least one solid pseudo-color pattern area 87a which assumes a different location in subsequent sets <NUM> of pseudo-color image data <NUM> to produce a progressing wave pattern 87b progressing in the direction of the arrow 87c. Each of the <FIG> shows one of the subsequent sets <NUM> during one variation period. At the end of the variation period, in <FIG>, the cycle is repeated by starting with the pattern of <FIG>.

The pseudo-color pattern field <NUM> shows regularity in its change both over time and space in that the at least one solid pseudo-color pattern area 87a alternates regularly in space with a transparent area or an area filled with another pseudo-color, and in that the same spatial pattern is contained in the pseudo-color pattern field <NUM> after each passing of the variation rate. It is to note, that the variation rate itself may change over time depending on the content of the auxiliary input data <NUM>.

The pseudo-color pattern <NUM> shown in <FIG> may e.g. used for auxiliary input data <NUM> which represent a velocity, flow rate and/or direction, such as blood flow data. The velocity represented in the auxiliary input data <NUM> may be represented by the variation rate, the flow rate by the spatial variation rate or the intensity of the pseudo-color <NUM> in the pattern <NUM>. The extent of the pseudo-color pattern field <NUM> is determined automatically from the auxiliary input data <NUM> for each set <NUM>.

The image processor <NUM> may further comprise a spatial alignment module <NUM>. The spatial alignment module <NUM> is configured to rectify the input image data <NUM> and the auxiliary input data <NUM> spatially so that each set <NUM>, <NUM> is spatially congruent to each other. This can for example be done by algorithmically correlating spatial features which are present both in the input image data <NUM> and the auxiliary input image data <NUM>. The spatial alignment module <NUM> is configured to rectify at least one of the image input data <NUM> and the spatial input data <NUM> by morphing the image so that correlating structures are aligned. The morphing operation may comprise stretching, rotating, adjusting resolution, and/or warping of at least one of the input image data and the auxiliary input data.

For example, fluorescence image data from the fluorescence camera <NUM> may be slightly rotated, warped and displaced relative to the visible-light image data from the visible-light camera <NUM>. Further, two-dimensional data <NUM> from the ultrasound sensor <NUM> may have a different scale and resolution, and be warped and rotated with respect to the input image data <NUM> and/or the fluorescent-light image data <NUM> from the fluorescence camera. In order to correctly superpose these data, the spatial alignment module <NUM> is configured to execute a correlation algorithm to identify common structures in the respective data <NUM>, <NUM> and compute their respective orientation and scale. The spatial rectification is performed depending on the result of this algorithm for each synchronized set of input image data and auxiliary input data.

Further, the spatial alignment module <NUM> may comprise transfer functions of the devices which generate the auxiliary input data <NUM>, which transfer functions have been obtained by initial calibration processes. The transfer functions may be used to correct errors in the optical system such as vignetting, distortion and/or aberration.

The medical image superposition device <NUM> further comprises an output interface <NUM>, which is configured to output subsequent sets <NUM> of output image data <NUM>. The output image data <NUM> result from the merger of the pseudo-color image data <NUM> and the input image data <NUM>.

An example for such a merger is given in applicant's application <CIT>. Although the merging is described in this reference in relation to fluorescent-light image data only, the merging can be used for any other type of two-dimensional auxiliary input data <NUM>, <NUM>, <NUM> without any further change.

The output image data <NUM> are output at an output data rate which is preferably larger than the highest variation rate of a temporarily modulated pattern <NUM> in the output image data <NUM> by an order of magnitude. In particular, the output data rate is higher than the flicker fusion rate, in particular higher than <NUM>. The variation rate in contrast is smaller than the flicker fusion rate, so that the regular variation of the pseudo-color pattern <NUM> is clearly visible.

It is to be understood, that recognition of the pattern <NUM> does not depend on cognitive processes of the human mind but that the pattern <NUM> can be recognized by any automated pattern recognition process due to its regularity in at least one of the spatial and temporal domain.

The display <NUM> is connected via a data connection <NUM> to the output interface <NUM>. In the display <NUM>, the output image data <NUM> are displayed. If the display <NUM> is transmittive, the input image data <NUM> may be omitted from the output image data <NUM>, and only the pseudo-color image data <NUM> may be displayed. The input image data <NUM> in this case are used only for aligning the pseudo-color image data <NUM> and the pattern <NUM> with the input image data <NUM>. As the input image data <NUM> give an accurate rendition of what is seen through the display <NUM>, the effect is the same as using a non-transmittive display and displaying the merged input image data <NUM> and the pseudo-color image <NUM>.

A schematic example of what is seen through the ocular <NUM> is presented in detail I of <FIG>.

The fluorophores <NUM>, <NUM> are each assigned a different pseudo-color by the pseudo-color image generator <NUM>. The output image data <NUM> comprise at least one pseudo-color pattern field <NUM>, which is generated in real time by the pseudo-color image generator <NUM> and which has a temporal and/or spatial modulation. The pattern field <NUM> comprises the pseudo-color pattern <NUM> and is generated depending on the content of the input image data <NUM> and/or the auxiliary input data <NUM>.

For example if, in the fluorescent-light image data <NUM> of the fluorescence camera <NUM>, there are areas within the fluorescent-light image data <NUM> having a fluorescence intensity below a predetermined but preferably adjustable threshold, they may be marked with a pseudo-color pattern <NUM> in which the pseudo-color <NUM> may have at least half of their full brightness, so that in spite of the low intensity in the fluorescent-light image data <NUM>, this area is still visible to an observer. Alternatively, or additionally, a different pattern <NUM> may be used in an area where the fluorescent-light image data have an intensity which exceeds a predetermined but preferably adjustable threshold. This may indicate to an observer that these data are unreliable or that he has to adjust the camera sensitivity.

The medical image superposition device <NUM> and the medical observation device <NUM> may provide two- and/or three-dimensional images in the display <NUM> and, accordingly, the display <NUM> may be a 2D- or 3D-display.

If auxiliary input data <NUM> are used which comprise data from locations which are situated in the depth direction <NUM> beneath the plane of the field of view <NUM> of the optical system <NUM> and thus are not contained in the visible-light image data <NUM> from the visible-light camera <NUM> or in the fluorescent-light image data <NUM> from the fluorescence camera <NUM>, they may be added to the pseudo-color image data <NUM> using a pseudo-color pattern <NUM>. For example, if the ultrasound sensor <NUM> or <NUM> has detected a coherent structure <NUM> in the object <NUM>, such as a large blood vessel or a nerve beneath the field of view <NUM>, the structure <NUM> may be rendered by using a pseudo-color pattern field <NUM> in the pseudo-color image data and ultimately in the output image data.

Various auxiliary input data <NUM> from different devices may be simultaneously displayed by using different pseudo-colors and different patterns <NUM>.

Next, the process of generating output image data from the input image data and auxiliary input data via pseudo-color image data is explained with reference to <FIG>. In <FIG> optional steps are indicated by dashed lines.

<FIG> shows that the input image data <NUM> are pre-processed, in a first step <NUM>, which may be carried out by the medical image superposing device <NUM>. The pre-processing step <NUM> may include any one or any combination of histogram equalization, automatic adjustment of contrasts, compensation of aberration, vignetting and distortion errors caused by the optical system <NUM> as well as noise reduction, but may not be limited to these.

The auxiliary input data <NUM>, such as the fluorescent-light image data <NUM> or other and further auxiliary input data <NUM>, such as ultrasound data, microradiography image data, thermographic image data and/or Roentgen image data may also undergo a pre-processing step <NUM> to compensate at least one error, noise and distortions. The pre-processing algorithms carried out in step <NUM> may in general be the same as for the input image data <NUM> in step <NUM>, using however, different parameters to account for the different physical parameters and systems. For example, the compensation of errors introduced by the optical system <NUM> may be different for the fluorescent-light image data <NUM> than for the visible-light image data <NUM>.

In particular for the auxiliary input data <NUM>, the pre-processing <NUM> may also comprise a threshold comparison, in which those auxiliary input data with a content below a certain sensitivity threshold are for example blanked or deleted so that they will not be contained in the output image data. For example, pixels may be blanked or set to zero or set to transparent in auxiliary input data <NUM>, if the intensity of that pixel is below a sensitivity threshold. The sensitivity threshold may be determined by a user of the medical image superposing device <NUM> or medical observation device <NUM>. Thus, pixels which represent only a very low signal strength, i.e. fluorescence level or ultrasound reflectivity, may thus not be considered in the pseudo-color image generation.

The next step <NUM> comprises spatial adjustment of at least one of the input image data <NUM> and the auxiliary input data <NUM>, in particular of two-dimensional and/or three-dimensional auxiliary input data so that spatial features are located at the same location and in the same size and orientation in both the input image data <NUM> and the auxiliary input data <NUM>. At the end of spatial adjustment <NUM>, the input image data and the auxiliary input data are at least substantially spatially congruent. Preferably, the input image data <NUM> are used for reference in the spatial alignment step <NUM>. The auxiliary input data <NUM> are thus modified in the spatial alignment step <NUM>, whereas the input image data <NUM> are left unchanged. This is of particular advantage if the auxiliary input data <NUM> contain less data than the input image data, e.g. because pixels have been blanked and/or because the resolution measured in pixel of the fluorescence camera <NUM> and/or a microradiographic, thermographic or gamma-ray camera and/or ultrasound sensor is smaller than the resolution of the visible-light camera <NUM>. However, leaving the input image data unchanged will maintain the field of view <NUM> of the optical system <NUM> and the visible-light camera <NUM> and thus will render faithfully the filed of view <NUM> on the display <NUM>.

In the spatial alignment step <NUM> the auxiliary input data <NUM> of a set <NUM> or a plane <NUM> may be distorted, rotated and changed in its resolution to spatially correlate it to the input image data <NUM>. The input image data <NUM> and the auxiliary input data <NUM> may be time-stamped so that a set of input image data <NUM> is processed together with a set of auxiliary input data <NUM> that has been sampled at the same time as the input image data.

Next, in step <NUM>, pseudo-color image data are generated from the auxiliary input data <NUM>.

For example, the different fluorescence colors in a fluorescent-light image data <NUM>, which are in the NIR range and thus invisible to humans, may be assigned different pseudo-colors in the visible range. Further, auxiliary input data <NUM> representing invisible physical quantities, i.e. physical quantities not being electromagnetic waves in the visible light range, such as ultrasound, microradiation, gamma-ray or thermographic sensing, may also be assigned pseudo-colors. Preferably, each sensing modality is assigned a different pseudo-color and/or a different pattern.

Using many different types of auxiliary input data <NUM> in a single set of output image data may result in the use of too many pseudo-colors. Thus, a pattern using the same pseudo-color may be used to keep the numbers of different pseudo-colors small. Further, use of a pattern <NUM> such as a hatching only obscures part of the underlying input image data, and allows having a better view of the underlying input image data.

Further, if the content of the auxiliary input data <NUM> designates a low intensity of the recorded signal in the auxiliary input data <NUM>, they would be visualized with only subtle pseudo-coloring. Such a subtle pseudo-coloring may be insufficiently set apart from non-pseudo-colored areas. Thus, these areas should be not filled with a solid pseudo-color but with a pattern in which the pseudo-color has a high brightness, e.g. of at least <NUM> % of the maximum brightness value.

The different patterns and pseudo-colors are preferably selected automatically from a storage <NUM> maintained in the medical image superposing device <NUM>. The user may be able to preset certain assignments, i.e. to assign e.g. a particular pseudo-color and/or pattern to specific data sources, to specific content of data from a data source, such as to specific fluorophores, to pixel intensities, or to alteration patterns of data, such as rates of change.

In step <NUM>, subsequent sets <NUM> of pseudo-color image data <NUM> are obtained, in which at least one pseudo-color pattern field <NUM> is filled with a pattern <NUM> depending on the content of at least one of the input image data <NUM> and the auxiliary input data <NUM>.

In the next step <NUM>, the pseudo-color image and the input image are merged. It is to be understood that this merging occurs for each data set in the subsequent sets of input image data <NUM> and auxiliary input data <NUM> to allow a real-time processing.

The merging process is described in detail in the parallel application <CIT>. As a result of the image merging step <NUM>, the output image data <NUM> are obtained and finally displayed on the display <NUM>.

In <FIG>, the process depicted in <FIG> is further explained.

Image input data <NUM> represent visible-light image data <NUM> from the visible-light camera <NUM>. Fluorescent-light image data <NUM> are obtained as auxiliary input data <NUM> from an fluorescence camera <NUM> and contain the emission spectra of at least two fluorophores <NUM>, <NUM> which are assigned different pseudo-colors designated FL800 and FL400 in <FIG>.

At least one pseudo-color pattern field <NUM> has been generated in the pseudo-color image data <NUM> depending on the content of the fluorescence image data <NUM>. The pseudo-color pattern field 88a designates an area in which the two fluorophores <NUM>, <NUM> overlap. The pseudo-color pattern field 88a may contain one of or both the pseudo-colors which have been assigned to the respective fluorophores <NUM>, <NUM> or the corresponding emission spectra, or a different pseudo-color. Another synthetically generated pseudo-color pattern field 88c designates an area in which one of the pseudo-colors has a very low intensity. This pseudo-color pattern field <NUM> may use the same pseudo-color assigned to the respective fluorophore or fluorescence emission spectrum.

Two-dimensional auxiliary input data <NUM> from the ultrasound sensor <NUM> are also transformed to pseudo-color image data <NUM>. A pattern field <NUM> is synthetically generated in areas where the pixel intensity in the auxiliary input data <NUM> from the ultrasound is above a threshold. The pseudo-color images are then merged with the image input data <NUM> to obtain output image data <NUM> containing at least one pseudo-color pattern field <NUM>.

In <FIG>, a set <NUM> of output image data <NUM> has been generated by merging fluorescent-light image data <NUM> containing a fluorescent light from a tumor-marking fluorophore <NUM> with ultrasound image data and visible-light image data <NUM>.

The at least one pseudo-color pattern field <NUM> in the output image data <NUM> may have a temporal modulation as indicated by arrow <NUM>. The area occupied by the pseudo-color pattern field <NUM> corresponds to areas which were automatically recognized as blood vessels having a blood flow velocity and direction in each subsequent set. The pseudo-color pattern field <NUM> is a wave pattern which is modified in subsequent sets of output image data <NUM> to produce a wave pattern 87b which propagates in the direction of arrow <NUM>.

Claim 1:
Microscope, comprising an input interface (<NUM>) and an output interface (<NUM>),
the input interface (<NUM>) being configured to receive subsequent sets (<NUM>) of input image data (<NUM>) and auxiliary input data (<NUM>),
the output interface (<NUM>) being configured to output subsequent sets (<NUM>) of output image data (<NUM>) at an output data rate,
the microscope further comprising an image processor (<NUM>),
wherein the image processor is connected to the input interface (<NUM>) for receiving the subsequent sets (<NUM>) of the input image data (<NUM>) and the auxiliary input data (<NUM>), and to the output interface (<NUM>) for outputting the subsequent sets (<NUM>) of the output image data (<NUM>),
the image processor further being configured to generate, in real time, subsequent sets (<NUM>) of pseudo-color image data (<NUM>) depending on a content of at least one of the auxiliary input data (<NUM>) and the input image data (<NUM>),
wherein the output image data (<NUM>) are three-dimensional and
wherein the image processor (<NUM>) is configured to generate the pseudo-color image data (<NUM>) at different depth layers of the three-dimensional output image data depending on the content of the auxiliary input data (<NUM>),
wherein the image processor (<NUM>) is further configured to generate a synthetic pseudo-color pattern field (<NUM>) within the pseudo-color image data (<NUM>) depending on the content of at least one of the input image data (<NUM>), the auxiliary input data (<NUM>) and the pseudo-color image data (<NUM>), the pseudo-color pattern field (<NUM>) comprising a temporally modulated pattern (<NUM>", <NUM>‴) having a variation rate, and
wherein the image processor is configured to compute the variation rate of the temporally modulated pattern (<NUM>", <NUM>‴) depending on the content of at least one of the input image data (<NUM>) and the auxiliary input data (<NUM>), wherein the image processor (<NUM>) is configured to merge the pseudo-color image data (<NUM>) and at least one section of the input image data (<NUM>) to generate the output image data (<NUM>) containing the pseudo-color pattern field (<NUM>).