Patent ID: 12201273

DETAILED DESCRIPTION

An endoscopic video system10—herein also video endoscopy system—typically comprises an endoscope12and a camera head14. The endoscope12is connected to the camera head14by means of a detachable coupling16.

The camera head14has a lens18that serves to map the images supplied by the endoscope12to an image sensor.

In the example shown inFIG.1, the camera head14has two image sensors, namely an image sensor20for visible light and an image sensor22for infrared fluorescent images. In order to record single images in visible light with the image sensor20as well as fluorescent images with the image sensor22, a beam splitter24is provided that causes the lens18to map the optical images on both the image sensor20for visible light and on the image sensor22for fluorescent images. In order to optically separate the two image channels created this way, a blocking filter26for infrared light is provided in front of the image sensor20for visible light, which means a blocking filter that blocks infrared light and is transparent for visible light. Accordingly, a blocking filter28is provided for the fluorescent images in front of the image sensor22that blocks visible light and fluorescence-stimulating light.

In order to illuminate an object to be viewed—which can be located in a body cavity, for example—using the endoscope12, a light source30is provided that feeds light into the endoscope12via a light guide32, so that this light can come out at the distal end of the endoscope12. The light source30provides both visible light and light with a wavelength that is suitable for stimulating fluorescence.

During operation, two image sequences are preferably recorded simultaneously by the two image sensors20and22. The image sensor20for visible light records a first image sequence that is comprised of single images, and the image sensor22for fluorescent images records a second image sequence that is comprised of fluorescent images. In this case, each single image recorded with the image sensor20is associated with a fluorescent image simultaneously recorded with the image sensor22.

The signals representing the image sequences are supplied to an image processing device34that processes the single and fluorescent images as described in the following. The enhanced images of the image sequences generated by the image processing device34can then be displayed on a monitor36and, due to the image processing described below, in such a way that single images and fluorescent images can be superimposed so that the fluorescence is visible on the monitor36. In addition, the fluorescent glow can be electronically colored, for example, and added to the color image recorded in reflected light.

FIG.2Ais an abstract and schematic depiction of three consecutive single images100of a first image sequence that were recorded while the object was illuminated with visible light. What is shown is a current single image100.1at time t as well as a preceding single image100.2at time t-1, as well as a further preceding single image100.3at time t-2. As can be seen inFIG.3A, the single images100.1,100.2and100.3differ at the various points in time in that the object102shown in the single images100is displaced. Using easily identifiable object features (e.g. the structural branching104), the displacement106or108by which the two consecutive single images100differ can be determined. The displacement106or108identified this way results in a transformation function that can be used to precisely map a single image100at one point in time to a single image100at a different (earlier or later) point in time.

FIG.2Bshows three fluorescent images110in the second image sequence that were captured while being irradiated with fluorescence-stimulating radiation. The fluorescence112only occurs locally and is only faintly visible in a single image. An improved—in particular improved with regard to the signal-to-noise ratio—fluorescent image could be created by superimposing several fluorescent images110(which is also roughly equivalent to a longer exposure time). However, the problem is that the fluorescing object112is not always located at the same image location in the consecutive fluorescent images of a second image sequence; seeFIG.9.

It is therefore intended to first transform consecutive fluorescent images110of the second image sequence so that they can subsequently be superimposed. This is shown schematically inFIGS.3A,3B,4A and4B. The transformation functions required for transforming the fluorescent images are obtained from the consecutive single images100of the first image sequence, for example using procedures known per se, such as SIFT or SURF.FIG.3Ashows how a characteristic object feature104can be identified in each single image of the first image sequence. This way, a displacement vector106or108for a linear displacement can be determined for each transition from one single image100to the next single image100as a simple transformation function. The displacement vectors that define the transformation function are indicated as arrows inFIG.3A.

FIG.3Bshows that the same displacements106and108also apply to the corresponding fluorescent images110.

Therefore, fluorescent images110.2and110.3at times t-1and t-2that precede a current fluorescent image110.1at time t can be transformed in such a way that their image features are position-enhanced and at least approximately congruent as a result; seeFIG.4A. The transformed fluorescent images110.2′ and110.3′ can then be superimposed with the respectively current fluorescent image110.1to achieve an enhanced current fluorescent image114with improved signal-to-noise ratio; seeFIG.4B.

FIG.5shows a schematic depiction of a single fluorescent image110with a small structure112that has a weak glow in the fluorescent light with ideal imaging without image noise.

FIG.6shows a schematic depiction of a single image100of the first image sequence with a feature102in reflected light with ideal imaging without image noise

FIG.7shows a schematic depiction of the superimposing of the reflecting structure102fromFIG.5and fluorescing structures112with ideal imaging without image noise.

FIG.8shows a schematic depiction of a single fluorescent image110, like inFIG.5, as it is recorded by an image sensor. The fluorescing structure112is not visible because of the image noise of the image sensor.

FIG.9shows a schematic depiction of an addition of two superimposed fluorescent images with ideal imaging without image noise. The fluorescing structure is shown twice, i.e. the simple addition of two fluorescent images results in image artifacts instead of the desired amplification of the weak fluorescence.

FIG.10shows a schematic depiction of an addition of two superimposed fluorescent images as they are recorded by a sensor with image noise. The noise of the image is reduced. However, the fluorescing structure is still not visible because of the artifacts described in connection withFIG.9.

FIG.11shows a schematic depiction of a positionally correct addition performed with the method according to the invention of two superimposed fluorescent images. The noise of the image is reduced, and the fluorescing structure is visible.

FIG.12shows a schematic depiction of the superimposing of a single image of the first image sequence, that shows reflecting structures, with a single fluorescent image, that shows fluorescing structures, under the influence of the image noise of the sensor. The fluorescing structure is not visible because of the image noise.

FIG.13shows an image superimposing of the reflecting and fluorescing structures performed with the invention. The image noise is reduced, and the fluorescing structure is visible.

The process of a method according to the invention is as follows (seeFIG.14):

Initially, images of a first image sequence are captured (200), and images of a second image sequence (202) are captured simultaneously or alternately. The images of the first image sequence are single images (reflection images) generated by the light that is reflected by an endoscopically viewed object. The images of the second image sequence are fluorescent images that show fluorescence when an object is irradiated with fluorescence-stimulating radiation.

Subsequently, object or image features are detected (204) in the images of the first image sequence, and the feature positions of the detected object or image features are recorded (206).

The detection of image and/or object features can optionally comprise a rectification of the single images—i.e. a compensation of the optical distortion of the endoscope (208). In addition, an interpolation of the feature positions (210) of the detected object and image features is preferably performed between the feature positions detected in the single images in order to determine interpolated feature positions for points in time at which fluorescent images are captured.

Position changes are determined based on the feature positions recorded in different single images of the first image sequence of a respective detected object or image feature (212). The determined position changes are then used to form geometric transformations (transformation functions) that correspond to the position changes (214).

The transformation functions formed from the single images of the first image sequence are finally applied to fluorescent images of the second image sequence (216) in order to obtain transformed fluorescent images in which fluorescing structures are respectively located at the same image location.

The fluorescent images transformed this way are finally superimposed upon each other (218) in order to obtain an improved fluorescent image.

These steps are carried out for each current fluorescent image to obtain a sequence of improved fluorescent images that can ultimately be displayed (220).

Optionally, each improved fluorescent image can be superimposed on the associated single image of the first image sequence so that a video sequence of superimposed single and fluorescent images is created (22).

LIST OF REFERENCE SIGNS

10Endoscopic video system12Endoscope14Camera head16Detachable coupling18Lens20Image sensor for visible light22Image sensor for fluorescent images24Beam splitter26Beam splitter, blocks infrared light28Beam splitter, blocks visible light and stimulating light30Light source32Light guide34Image processing device36Monitor100,100.1,100.2,100.3Single image102Object104Structural branching106Displacement108Displacement110,110.1,110.2,110.3Fluorescent image112Fluorescence, fluorescing object114Improved current fluorescent image200Capturing a first image sequence in reflected light202Capturing a second image sequence in fluorescent light204Detection of object or image features206Determining the position of object or image features208Rectification of the single images210Interpolation of feature positions212Determination of position changes214Formation of transformation functions216Application of transformation functions218Superimposing of fluorescent images220Creating a sequence of fluorescent images222Superimposing of single and fluorescent images