Abstract:
A system is provided to improve retinal camera picture quality by providing a user-variable transfer function for each pixel that results in redistributing grayscale values to solve the problem of saturation caused by highly reflective retinal objects. The result is the ability to capture both optic nerve and retina detail in a single picture. The darker retina is brightened to permit observing retinal detail using the redistributed grayscale values, while preserving optic nerve detail. Those pixels experiencing high-intensity reflections are properly exposed to prevent saturation, while outputs of low-intensity pixels associated with the darker regions are intensified, in one embodiment in accordance with an adjustable Bezier curve. The result is that one can obtain retinal details previously flooded out by the reflections from the optic nerve while at the same time offering optic nerve detail. In one embodiment the redistributed grayscale values are optimized for each color plane to provide color-corrected images matching those associated with film cameras.

Description:
FIELD OF THE INVENTION  
       [0001]     This invention relates to retinal cameras and more particularly to a system for eliminating the effects of a highly reflective optic nerve so that detail of both the optic nerve and detail of the surrounding retinal material are viewable.  
       BACKGROUND OF THE INVENTION  
       [0002]     In retinal imaging, one seeks to obtain photographs of the detail of the optic nerve and the surrounding retinal material that heretofore has been captured on film.  
         [0003]     A constant problem in retinal imaging is the fact that the optic nerve, which is basically where all of the nerve bundles go back to the brain, is very reflective. When one fires a strobe into the eye to illuminate the retina, the optic nerve tends to throw large amounts of light back at the camera, which shows up as a bright blotch at the position where the optic nerve is attached to the retina. The rest of the detail of the retina that one wants to see, especially at the periphery, is very dark. The result of photographing the retina utilizing high-power strobe pulse illumination is a very high-contrast image where the darker regions are drowned out by the high reflectivity of the optic nerve.  
         [0004]     More importantly, the optic nerves of the darker-skinned races, including the Negroid and Hispanic races, are in general more highly reflective than white or Caucasian races, making retinal imaging even more difficult. It is noted that people of darker complexion have darker retinas because the pigment of the retina is darker.  
         [0005]     When taking retinal images, one has to inject enough light to illuminate the dark areas. However, if one increases the output of the xenon strobe normally used, one simply drowns out the optic nerve detail because of its high reflectivity.  
         [0006]     In the past, in order to obtain images of the detail of the optic nerve as well as images of the retina, one had to take numbers of photographs, each with different light outputs or different F stops on the camera. With multiple pictures one needs multiple strobe pulses, with each strobe pulse injecting energy into the eye. This causes pain and is very uncomfortable for the patient. What this means is that a patient may have to endure a number of 100 watt-second pulses discharged into the eye. It is therefore desirable to be able to eliminate the requirement for multiple exposures.  
         [0007]     However, the problem is not so much seeing the remainder of the optic nerve but dealing with the high reflections where the optic nerve is attached to the retina that visually resembles a hole in the back of the eye. As will be appreciated, the optic nerve is always on the nasal side of the eye and appears as an offset bright hole. On the other hand, the retina has blood vessels and arteries that stretch out across the eye including smaller capillaries that branch off. In general, the vascular structure forms a circular pattern. At the center of vision, which is called the macula, there are no blood vessels and therefore it is completely devoid of blood vessel structure.  
         [0008]     It is noted that in addition to the optic nerve, pathology can be highly reflective as well. High reflections can come from a scar, tumor or growth, the reflections from which will saturate the camera with the introduction of a pulse from the xenon strobe.  
         [0009]     As mentioned hereinabove, one technique to eliminate the problems of being able to view the structure of the eye is to have multiple photographs, one to expose the optic nerve and the other to expose the rest of the retina. In order to get the detail of the optic nerve, one could either reduce the output of the flash lamp or stop down the camera so that just this area is properly exposed to be able to see all the detail. However, by cutting down the flash power to be able to observe the optic nerve detail, one has insufficient light to be able to view the remainder of the retina. Note that with too high a strobe output the strobe saturates the camera due to the reflectivity of the optic nerve and all detail is gone.  
         [0010]     As will be apparent, by raising the flash lamp power output one would simply see hotspots in the image for which detail is completely lacking.  
         [0011]     In the past, in order to be able to view the detail of the retina away from the optic nerve, the typical practice was to slightly increase the flash lamp output which, while causing hotspots near the optic nerve or the pathology, it was possible to discern the detail of the darkened portion of the retina.  
         [0012]     There is therefore a necessity for eliminating the requirement for multiple photographs, both from a patient comfort point of view and to be able to view all of the retina in a single image or photograph.  
         [0013]     With the advent of digital cameras, usually having CCD sensor arrays of a 1- to 11-megapixel variety, it is possible to get real-time images of the retina. However, the problem of multiple pictures and flash lamp intensity versus optic nerve reflectivity has not as yet been resolved for the above reasons.  
       SUMMARY OF INVENTION  
       [0014]     Rather than taking multiple exposures to obtain retinal and optic nerve detail, in the subject invention one exposure is used and a single image carries both optic nerve and dark retina detail. To make this possible, flash lamp intensity is lowered to avoid saturation and for obtaining optic nerve detail; and pixels viewing the low-intensity, now further darkened retinal regions have their outputs amplified so that the dark regions are brightened to reveal the detail. Note that the ability to see all aspects of the eye in one image or photograph aids diagnosis.  
         [0015]     How this is accomplished is now described.  
         [0016]     It will be appreciated that the pixels in a digital camera have outputs that are ascertainable. The dynamic range of each output is characterized by a grayscale having a range from 0 to 255, such that one can obtain 256 shades of any one color or gray. The grayscale in essence defines the dynamic range of the camera and to a certain extent the colors of the observed image.  
         [0017]     In general, the outputs of each of the pixels of the array can be characterized by a transfer function that is linear, meaning a linear relationship between the input and the output. Normally this relationship of input to output is fixed and is dependent upon the characteristics of the digital camera.  
         [0018]     Since each of the outputs of the CCD array of the digital camera is addressable, one can arrange to weight the individual outputs of individual pixels of the CCD array so as to increase the transfer function between input and output for those input levels or intensities that are relatively low.  
         [0019]     Thus as one part of the subject invention, those pixels from the dark retina having a relatively low intensity have their outputs amplified to brighten those areas so that detail is visible. As a result, one can ascertain which of the pixels have relatively low outputs and multiply their inputs with a weight determined from a lookup table that will increase the output while not affecting the transfer functions for the pixels having higher outputs.  
         [0020]     By increasing the outputs of the pixels having low-intensity inputs, one can obtain detail of the dark retinal area.  
         [0021]     To obtain detail of the optic nerve and other highly reflective retinal objects, one first reduces flash lamp output to prevent saturation caused by reflections from the optic nerve or other highly reflective objects. Once having reduced the flash lamp output to avoid saturation one can view detail of the highly reflective retinal objects such as the optic nerve. However, reducing the flash lamp output further darkens the retina. With the subject technique the further darkened portions of the retina are brightened by the increased outputs for the low-output pixels. Thus both the optic nerve detail and the detail of the remainder of the retina are simultaneously viewable with one exposure in one picture. This solves the problems associated with multiple exposures.  
         [0022]     The weighting function used, rather than being a linear weighting function, is alterable in one embodiment by utilizing a Bezier curve, the curvature of which is determined by four points, with two points being fixed and two points being variable. Given a Bezier curve to define the grayscale distribution, if one lowers the curve at the center portion to provide a belly, generally in the 220 to 240 grayscale range, then pixel outputs that are the result of the darker regions are increased.  
         [0023]     In one embodiment the weighting system is user variable so that the user can move the belly of the curve up and down under the operator&#39;s control. This means that the operator can control the transfer function for all of the pixels, most notably the ones in the mid grayscale ranges corresponding to dark areas, simply by moving the variable points of the Bezier curve.  
         [0024]     Thus what is done in the subject invention is to redistribute the grayscale values in a non-linear fashion along a curve so that the dark areas get bright while at the same time not significantly amplifying the outputs of those pixels that are detecting the high-reflectance optic nerve. The result is to be able to view the detail of the highly reflective materials in the eye while at the same time viewing the detail of the dark retina, and to do so with one exposure taking one picture.  
         [0025]     The optimal nonlinear grayscale distribution takes the darker areas and makes the subtle details more exaggerated, while at the same time muting the changes in the lighter areas to produce a flattening effect.  
         [0026]     Note that for the high-intensity pixels the transfer function mutes the amplification, thereby to mute the output. In short, for brighter light areas one is not transferring as much gain to the output for the particular pixel, whereas for the dark areas one is providing gain, with the weights specified by a look up table.  
         [0027]     In one operating scenario, the first thing that one wants to do is to reduce the power of the flash lamp. The reason that one wants to reduce the power of the flash lamp is to limit the amount of reflection from the optic nerve. Thus one wants to get away from the situation where the reflection from the optic nerve saturates everything.  
         [0028]     Then one adjusts the grayscale distribution to correct for the effect of the lowered flash lamp intensity on the darker regions of the retina by amplifying the output for low-intensity pixels, while at the same time leaving the transfer functions for the bright area pixels alone so that the transfer function for these pixels remains the same.  
         [0029]     In summary, one cuts down the output of the flash lamp to eliminate gross saturation and then increases brightness of the dark regions or the low-intensity areas while at the same time keeping the others at the same level. This pops out the detail in both the darker regions and the bright regions.  
         [0030]     It is possible to decrease the amplification for those pixels viewing darker regions to make the darker regions darker; but in so doing one would have a very high contrast effect, which is undesirable. The idea in color retinal imaging is not to create contrast but to balance it. Contrast is basically taking the dark regions and making them darker and taking the light regions and making them lighter.  
         [0031]     In order to obtain a realistic view of the retina in terms of an image, one is trying to do just the opposite, namely trying to take everything and flatten the response out so that the image looks natural. This aids in the interpretation of the pathology of the retina so that what an ophthalmologist is looking at corresponds as nearly as possible to that which exists in nature. One in short does not want to create artificial conditions or artifacts that could in some sense make a diagnosis more difficult.  
         [0032]     There is, however, another aspect to the use of the non-uniform grayscale distribution and that is to make the image available from the digital camera correspond to the images available from film. The reason that this is important is because many doctors are used to viewing film images in order to make diagnosis and would like to have the images that are available from the digital camera more closely correspond to what they are used to looking at.  
         [0033]     By using the Bezier curve, which provides a polynomial fit between four points, and by adjusting the transfer function of each of the pixels based on the curve, one can adjust the curve to not only fix the problem of hotspots versus dark areas of a retina but also to correct the color response of the digital camera.  
         [0034]     For each retinal camera, a model is generated to create what are called color planes or curves. These curves in essence describe the transfer function for each of the pixels in the camera. Since the color distribution curve is the composite of the red, green and blue response of the camera, by adjusting these curves one can make adjustments for each color. This is done by generating red, green and blue curves, altering them and then forming a color composite curve.  
         [0035]     This is important to help compensate for flash temperatures. In general the output of a xenon flash strobe looks a little blue. One can correct to a realistic view by compensating for the blue illumination through adjusting the red and green response of the digital camera.  
         [0036]     It is noted that film images tend to be very yellow when the images are obtained by illuminating the retina with a blue-shifted xenon output. This is because film in general is somewhat blue-muted. Thus when one takes photographs of the retina on film, they tend to have a slight yellow-orange look to them that might not necessarily be real but that which doctors are generally used to seeing.  
         [0037]     In order to adjust the output of the CCD digital camera to provide yellow-orange, one wishes to make the output of these cameras look like what would be seen when using a film camera. One therefore actually seeks to mute the blue channel for the digital camera and can do so by generating weights from a nonlinear grayscale distribution.  
         [0038]     Thus by using the Bezier curve and plugging in the red, green and blue characteristics of each color, one can make the images from the digital camera approximate that which would be seen utilizing a film camera and yet still be able to pop out the dark areas of the retina and the detail of the highly reflective retinal objects.  
         [0039]     In summary, a system is provided to improve retinal camera picture quality by providing a user-variable transfer function for each pixel that results in redistributing grayscale values to solve the problem of saturation caused by highly reflective retinal objects. The result is the ability to capture both optic nerve and retina detail in a single picture. The darker retina is brightened using the redistributed grayscale values to permit observing retinal detail, while preserving optic nerve detail. The optic nerve and other highly reflective retinal objects are properly exposed by reducing flash lamp output to prevent saturation, while outputs of low-intensity pixels associated with the darker regions are intensified, in one embodiment in accordance with an adjustable Bezier curve. The result is that one can obtain retinal details previously flooded out by the reflections from the optic nerve while at the same time offering optic nerve detail. In one embodiment the redistributed grayscale values are optimized for each color plane to provide color-corrected images matching those associated with film cameras. 
     
    
     BRIEF DESCRIPTION OF THE DRAWINGS  
       [0040]     These and other features of the subject invention will be better understood in connection with a Detailed Description, in conjunction with the Drawings, of which:  
         [0041]      FIG. 1  is a diagrammatic illustration of a retinal camera, including a film camera, a digital camera, and drive for driving a strobe lamp so as to illuminate the retina of an eye;  
         [0042]      FIG. 2  is a diagrammatic illustration of a computer system used to weight the outputs of the CCD array in the digital camera of  FIG. 1  so as to be able to output a single digital image containing detail of the retina and the optic nerve;  
         [0043]      FIG. 3  is a series of images created by the digital camera of  FIG. 1 , illustrating an image of the retina showing a darkened image the result of reducing flash lamp output to eliminate saturation; an image using normal flash lamp outputs illustrating the saturation due to the reflections from the highly reflective optic nerve and other pathology; and am image showing the result of applying a nonlinear grayscale function to produce a compensated image in which retinal and optic nerve detail are obtained in one exposure;  
         [0044]      FIG. 4  is a series of graphs illustrating the control of the weights used in generating the images of  FIG. 3 , showing linear distribution for the dark and normal images and a Bezier curve defining the weights for the compensated image of  FIG. 3 ;  
         [0045]      FIG. 5  is a diagrammatic illustration of the specification of weights in a look up table through manipulation of a Bezier curve, with the specification of the weights dependent upon the pixel output level;  
         [0046]      FIG. 6  is a graph showing that for a dimly illuminated pixel, moving the Bezier curve of  FIG. 5  downwardly results in a  28 % increase in the output, thus generating a weight of  1 . 28  by which the output of the associated pixel is multiplied;  
         [0047]      FIG. 7  is a graph of the Bezier curve of  FIGS. 4, 5  and  6 , illustrating that the Bezier curve defines pixel output level based on input level, with the Bezier curve defining the transfer function for the particular pixel;  
         [0048]      FIG. 8  is a flow chart showing the operation of the subject system from image capture through the production of a corrected image;  
         [0049]      FIG. 9  is a diagrammatic illustration of the use of a non-linear grayscale distribution function to color-compensate a digital camera; and,  FIG. 10  is a graph showing the Bezier curves for blue, green and red compensation and the resultant composite Bezier curve. 
     
    
     DETAILED DESCRIPTION  
       [0050]     Referring now to  FIG. 1 , prior to discussing the subject retinal image optimization system, the operation of a typical retinal camera is discussed. Here a retinal imaging camera  10  includes a film camera  12  and a digital camera  14  mounted on a stand  16  such that an imaging system  18  images the retina  20  of eye  22  onto the focal planes of cameras  12  and  14 . In order to illuminate retina  20 , a xenon strobe lamp  24  has its output redirected by mirrors  26  and  28  out through imaging system  18  so that the output of xenon strobe  24  illuminates retina  20 . Note that an eyepiece  30  is used for focusing both the digital and film camera as well as directing the optics to the appropriate portion of the eye.  
         [0051]     A precise maximum strobe output includes the use of drive  32  that incorporates a power supply for delivering several hundred volts to the xenon strobe. As illustrated, this is accomplished by delivery of a several hundred-volt pulse  35 , with a pulse width of between 5 and 10 milliseconds. In one embodiment the strobe is activated by a foot switch  34 .  
         [0052]     It will be appreciated that foot switch  34  is also used to control camera  10  over line  36  to take the pictures such that any shuttering and exposure for either the film camera or the digital camera is controlled responsive to foot switch  34 ; or is actuated automatically if desired.  
         [0053]     As illustrated in  FIG. 2 , a monitor  46  is used to display the retinal image as well as to display the aforementioned Bezier curves. Monitor  40  is coupled to a computer  42 , with mouse  44  being used to specify the variable points of the Bezier curve. Note a keyboard  46  is used as a further input device.  
         [0054]     Referring now to  FIG. 3 , activation of the xenon flash lamp of  FIG. 1  results in an image of the retina along with the optic nerve attached. Note in the bottom photograph the retina is dark due to the lowering of the flash lamp output to eliminate saturation. While detail of some of the retina and the optic nerve can be seen, in general one must lighten up the dark retinal material in order to observe its features, and do so without causing saturation. The middle picture shows the result of using maximum flash lamp power. Here it can be seen that there are saturated areas that are completely whited out, thus destroying detail.  
         [0055]     The upper picture shows a compensated image in which not only is the darker retinal area lightened to make retinal detail visible, the optic nerve detail is also visible.  
         [0056]     The upper image is the result of applying a pixel weighting function. The weighting function affects the pixel transfer function by selectively amplifying the outputs of the low-intensity pixels. Here it can be seen that not only is the detail of the optic nerve observable, so also is the detail of the remainder of the retina, including all of the vascularization. It will be appreciated by the decreasing the flash lamp intensity to eliminate saturation coupled with the nonlinear grayscale weighting system that one can observe both the optic nerve detail and the detail of the darker surrounding retinal material in a single image. This aids diagnosis. An additional advantage is that only one photograph or one exposure per image need be made to obtain sufficient detail of all areas of the retina, thus limiting the pain associated with multiple exposures.  
         [0057]     Referring to  FIG. 4 , adjacent each of the dark, normal and compensated images is the corresponding curve that defines the weighting system used to weigh the outputs of the individual pixels. The lower and middle curves correspond to a linear distribution is used, meaning that for each pixel in the CCD array, its output is a fixed percentage of the input. This transfer function is the characteristic of the camera and is not altered.  
         [0058]     However, the top graph shows a Bezier curve that defines the weights to be applied to the pixels.  
         [0059]     Referring now to  FIG. 5 , how an individual pixel output is weighted is now described. Here it can be seen that the weights applied to a pixel are derived from a look up table  50  coupled to a computer  52 . Look up table  50  is arranged to output a specified weight to be multiplied by the output of an addressed pixel, with the weight stored in the look up table being determined from the Bezier curve calculated by the computer.  
         [0060]     The computer generates the Bezier curve on display  54 , which for each of the  256  grayscale input levels determines an output level. Thus for a CCD array  56 , a pixel  58 , defined as pixel X m Y n , has its output amplified at  60 , after which a weight is applied to its output by a weighting circuit  62 .  
         [0061]     The output of amplifier  60  is coupled to look up table  50  so that the initial level of the pixel can be ascertained. The look up table ascertains the grayscale input level for this pixel and ascertains the weight to be applied to the pixel output based on its input level. This weight is coupled over line  66  to unit  62  to apply a predetermined multiplication factor or weight to the output of amplifier  60 . Alternatively, the table originally has values corresponding to a linear curve. The weighting is accomplished by reassigning the red value with the new y-intercept point on the curve.  
         [0062]     As will be described, mouse  70  controls the curve  72  displayed at display  54  by in effect moving variable points  74  and  76 , with points  78  and  80  being fixed. The line between the four points is generated using a Bernstein polynomial fit program such that the weights specified by look up table  50  can be controlled by user interface  80  comprised of computer  52 , mouse  70  and display  54 .  
         [0063]     In the illustrated embodiment, an input level Ix m y n  is illustrated by dotted line  82 , whereas the associated output level for such an input level is indicated by dotted line  84 .  
         [0064]     Referring to  FIG. 6 , dotted line  82  intercepts Bezier curve  72  at point  86 , which as illustrated by arrow  88  specifies a 28% increase in output over that which would have occurred if curve  72  were linear as illustrated at  90 . Thus curve  72  specifies for an input illustrated by line  82  that there should be a 28% increase in the output for this particular pixel over that associated with a linear grayscale function.  
         [0065]     Referring to  FIG. 7 , the graph shows the intersection with Bezier curve  72  of a number of different grayscale input levels illustrated by lines  82 . In one embodiment, the grayscale is divided up into 256 levels. For each grayscale level there is an associated output. As can be seen from the low input levels at the mid range of the graph as illustrated at  82 ′, the output at  84 ′ is amplified over that specified by a linear relationship between input and output. Thus for the lower input levels the output associated with the particular pixel is highly amplified. However, for the higher input levels it will be seen that with the input level just below saturation as shown at  82 ″, the output level is not significantly amplified. At this point the Bezier curve approximates a linear curve. How much the output for a given pixel input level is varied is therefore determined by the intersection of the input level with the Bezier curve.  
         [0066]     How this is accomplished is illustrated by the flowchart of  FIG. 8  in which the image is captured as illustrated at  90 . The capture is accomplished with high-bit definition as illustrated at  92  that involves 12 bits or 4,096 levels. This resolution is reduced as illustrated at  94  in one embodiment by conversion to an 8-bit system with 256 values. The resultant 8-bit values are passed through the Bezier curve look up table at  96  to produce image  98 . This image is the corrected image, with the values for each pixel being multiplied by a weight determined by the look up table.  
         [0067]     As can be seen, the look up table values can be changed as described above and as illustrated at  100 , with the new values loaded into look up table  102 .  
         [0068]     Referring now to  FIG. 9  in which like elements between  FIGS. 5 and 9  have like reference characters, it is possible to color-correct the digital camera using the subject system by adjusting the initial red, blue and green Bezier curves for blue color correction. The composite grayscale curve, being made up of the red, green and blue components, determines the color output of the camera and can be used to correct for the normal blue shift associated with xenon flash tubes.  
         [0069]     In order to provide initial color correction, the weights generated by unit  62  include the color correction weights for each pixel, here Wix m y n . This refers to the initial color correction weights.  
         [0070]     It will be appreciated that individual weights can be assigned to individual pixels to provide overall color correction. The composite Bezier curve permits tailoring or tweaking of the individual pixel outputs so as to provide improved color correction prior to correcting the overall image for brightness. Here it can be seen that the brightness correction weights, Wx m y n , are added to the color correction weights, Wix m y n , so that the weight generated for a given pixel is both the color corrected output for the pixel and the brightness correction output for the pixel.  
         [0071]     Referring to  FIG. 10 , it will be appreciated that for a given color camera one can generate blue, green and red Bezier curves which, when combined into a composite Bezier curve, weight the output of the pixels of the digital camera based on pixel intensity levels.  
         [0072]     Put another way, these curves, generally defining so-called color planes, define the initial transfer function for each pixel based on input level. The color plane curves correct the image for the slightly blue tint of the xenon flash lamp.  
         [0073]     The brightness compensation curve is applied after initial compensation to provide for the subject brightness control.  
         [0074]     Also shown in this figure is the use of two variable Bezier points to generate the blue curve. Thus it can be seen that the three curves can be specified by two fixed points and two variable points, although more flexible points can be added if desired.  
         [0075]     Thus one can weight the outputs of the pixels based on input levels to be able to select the lower illuminated pixels and to heavily amplify their outputs while only slightly amplifying high-intensity pixels. The result is a flattening that permits viewing detail not only in the dark retinal areas, but also detail in the highly reflective regions; and to do so with only one exposure in a single image or picture.  
         [0076]     A program listing in C follows describing the generation of the Bezier curve, the operation of the look up table and the generation of the weights required to provide the corrected image.  
         [0077]     While the present invention has been described in connection with the preferred embodiments of the various figures, it is to be understood that other similar embodiments may be used or modifications or additions may be made to the described embodiment for performing the same function of the present invention without deviating therefrom. Therefore, the present invention should not be limited to any single embodiment, but rather construed in breadth and scope in accordance with the recitation of the appended claims.