System and method for enhancing an image

A system and method for enhancing the contrast within an image. An enhanced image can be generated in a real-time or substantially real-time manner from an initial image. The pixel values of the initial image can be used to populate a histogram or otherwise serve as the basis for subsequent processing. A valley can be identified within the range of pixel values for use as a stretch metric used by a stretch heuristic to expand the contrast of the pixel values in the initial image by expanding the range of pixel values associated with the pixels in the histogram. In some embodiments, the initial image is first divided into image regions that are each associated with individualized processing. A bilinear interpolation step can then be performed to smooth the integrated image after the individualized processing is used to stretch the pixels within the individual image regions.

BACKGROUND OF THE INVENTION

The invention relates generally to systems and methods for enhancing images (collectively the “system”). More specifically, the invention relates to systems for enhancing the contrast within an image.

There are many contexts in which human beings and/or machines are impeded by low-contrast images.

The ability of a human being to safely operate a vehicle or other type of machine can be substantially impeded by a relatively dark environment. Night time driving is substantially more difficult and dangerous than day time driving. Only 28% of driving occurs at night. However, night time driving is responsible for over 62% of pedestrian fatalities. Standard headlights on a “low beam” typically illuminate approximately 60 meters ahead of the vehicle. At a speed of 60 miles per hour, a driver has approximately three seconds to react to a visible obstacle. Further complicating the visibility of the driver is the glare of headlights from one or more vehicles traveling in one or more directions.

There are many other contexts in which human beings are impeded by poor visibility in dark (e.g. low contrast) environments. Whether the task is navigation, the operation of equipment, or simply observation, human beings can be adversely impacted by an inability to view “contrast” in their surroundings.

Difficulties associated with low contrast environments are not limited to human beings. Sensors and the various machines that rely on those sensors can also be negatively impacted by relatively dark environments. There are numerous examples in the known art where automated processing is performed in response to information obtained from various image-based sensors. Security applications, industrial robots, automated vehicles, military devices, and other applications involving automated machines utilizing images captured from various sensors are often impeded by low contrast/poor visibility environments.

SUMMARY OF THE INVENTION

The system uses a computer to create an enhanced image from an initial image.

In some embodiments, a contrast metric can be generated using the pixel values from the initial image to determine whether or not enhancement would be desirable with respect to a particular image.

In some embodiments, the different pixel values of the initial image are placed on a histogram to determine whether or not a particular pixel value should be changed to a brighter value, changed to a darker value, or left unchanged.

The present invention will be more fully understood upon reading the following detailed description in conjunction with the accompanying drawings.

DETAILED DESCRIPTION

The invention relates generally to systems and methods for enhancing images (collectively “enhancement systems” or simply the “system”). More specifically, the invention relates to systems for enhancing the contrast within an image. The system can be used in a wide variety of different contexts and operating environments. The system can be embodied in a wide variety of different components and architectures.

The system can be used to facilitate, support, or otherwise assist in the operation of:a vehicle by a human being;an automated vehicle guided by an image captured from a sensor;night vision goggles, cameras, and other “night vision” devices;security cameras;motion detectors;vehicle navigation and operation applications;camping equipment;security applications; andany other application or function in which an enhanced image is useful.

A variety of ancillary devices can be used to support the functionality of the system. For example, in the context of a system providing “night vision” to the driver of a vehicle, it might be desirable to use a CMOS camera along with an infrared illuminator to achieve high night time visibility as far as 150 m ahead of the vehicle. Different embodiments of the system can use different sensors for capturing images. Some embodiments of the system may use multiple sensors to capture multiple images in a simultaneous or substantially simultaneous manner. Some embodiments of the system will incorporate illumination assistance devices such as an infrared illuminator, while other embodiments will not.

II. Introduction of Elements

A. Initial Image

An initial image22can also be referred to as a raw image, a non-enhanced image, an un-enhanced image, or a captured image. The initial image22is the image that would need to be relied upon if the system20were not present to enhance the contrast within the initial image22. In a night time driving embodiment of the system20, the initial image22is the driver's unaided view of the road as captured by a camera or other form of sensor in or on the vehicle.

In many embodiments, the initial image22will be a digital image captured by a sensor in a real-time or substantially real-time manner. In other embodiments, the initial image22can be in an analog format, subject to a conversion process in which a digital version of the initial image22is created for use by the system20.

A computer23is any device or combination of devices capable of implementing the processing requirements of the system20to create enhanced images24from initial images22. In a night time driving embodiment of the system20, the computer23will typically be embedded into the vehicle. In a night vision goggle embodiment of the system20, the computer23could be located within the goggles themselves, or the computer23could be configured to communicate with the goggles through a wire or some type of wireless connection. With respect to embodiments of the system20that involve automated devices utilizing sensor images, the computer23could be located within the sensor, within the automated device, or at some third location, depending on the desired operating environment and processing goals of the system20.

In many embodiments of the system20, the system20is configured to create enhanced images24from initial images22in a real-time or substantially real-time manner. In other embodiments, the speed of the system20is not as important, and processing can be performed in a batch mode.

C. Enhanced Image

An enhanced image24is an initial image22that is enhanced by the operations of the computer23. The system20can be used to enhance images in one or more ways. Certain aspects or portions of the image (e.g. pixels) can be brightened to enhance the contrast within the image. Certain aspects or portions of the image (e.g. pixels) can be darkened to reduce the glare within the image.

The enhanced image24is typically a digital image, although in certain embodiments it could be an analog image. The format of the enhanced image24and other attributes of the enhanced image24can be influenced by the ways in which the enhanced image24is to be used by the application invoking the functionality of the system20.

D. Contrast Enhancement Heuristic

A contrast enhancement heuristic28is the set of instructions implemented by the computer23to obtain the functionality of the system20. The contrast enhancement heuristic28can also be referred to as an enhancement heuristic. The contrast enhancement heuristic28is typically implemented in the format of software for the computer23, but the contrast enhancement heuristic28can also be “programmed” or “hard coded” into the computer23exclusively through hardware components and configurations.

A contrast metric30is a metric that relates to the element of “contrast” within the initial image22. In some embodiments, the contrast metric30is used to determine the desirability of the system20performing the contrast enhancement heuristic28to create an enhanced image24from an initial image22. For example, the system20can be configured to selectively invoke the contrast enhancement heuristic28depending on the contrast metric30. In some embodiments of the system20, the contrast metric30is compared to a contrast threshold32(discussed below) to determine whether or not the contrast enhancement heuristic28should be invoked. Some embodiments of the system20will not use a contrast metric30.

An example of a mathematical equation that can be used to calculate a contrast metric30is provided inFIG. 3and discussed in greater detail below. Different embodiments of the system20can invoke different contrast metrics30. As illustrated inFIG. 1, the contrast metric30can be a comparative measure.

A contrast threshold32is a value used by the system20for the purposes of comparison with the contrast threshold32. Some embodiments of the system20will not include a contrast threshold32. In some embodiments of the system20, the contrast threshold32can be adjusted by the user of the system20. For example, a driver utilizing the system20to provide “night vision” could be allowed to adjust the desired contrast in the image outputted by the system20. Allowing the driver to modify the contrast threshold32could achieve such flexibility on the part of the system20.

The appropriate contrast threshold32for a particular embodiment of the system20can be influenced by the availability of an illumination assistance device, such as an infrared illuminator as well as the type of sensor or camera used to capture the initial image22. In some embodiments of the system20, different contrast thresholds32can be applied in different circumstances. For example, different contrast thresholds32could be applied to interior and exterior images. In some embodiments of the system, different contrast thresholds32can be applied to different regions of the image. For example, different contrast thresholds32could be applied to the center portion of the image compared to the perimeter image regions. There can be many contrast thresholds32for a single image

G. Interim Images

The system20can utilize one or more interim images34in the process of creating an enhanced image24from the initial image22. Different embodiments of the system20can involve a wide variety of different interim image processing steps. Interim images34can also be referred to as “work in process” images34.

H. Pixels and Pixel Values

Initial images22, interim images34, and enhanced images24processed by the system20can be divided up into individual pixels36. The number of pixels36in an image determines the potential quality and level of detail that the image can convey. For example, an image that is 1000 pixels high and 800 pixels wide is capable of greater resolution than an image that is only 500 pixels high and 400 pixels wide. The sensor used to capture the initial image22will typically be the limiting factor with respect to the number of pixels within the initial image22, and ultimately, the number of pixels within the enhanced image24. For example, the type of camera used to capture the initial image22can determine whether the initial image22will have a resolution of VGA (or higher).

Each pixel36can be associated with a pixel value38. A typical pixel value38for an 8-bit image sensor is a number between 0 and 255, with 0 representing the lowest level of brightness and 255 representing the highest measure of brightness. Different ranges of pixel values38can be incorporated into different embodiments of the system20. For example, a 12-bit image sensor will have pixel values38between 0 and 4095.

A histogram40is a type of graph or diagram. With respect to the functionality of the system20, the histogram40identifies the number of pixels36associated with a particular pixel value38. A wide variety of different histograms40can be used to facilitate the pixel-related processing of the contrast enhancement heuristic28.

The system20does not need to create or process histograms for the system20to function. Histograms40are merely one way in which to express relationships and processing based on pixel values38. The system20can apply a wide variety of statistical processes and techniques to the pixel values36used by the system20. Some of those statistical metrics and heuristics can be characterized as histogram-based or histogram-related, while others may best be characterized as other statistic processes and techniques known in the art.

J. Peaks and Valleys

Histograms40used by the system20can include potentially any attribute that is associated with histograms40in the existing art. Peaks42and valleys44are examples of generally known attributes of histograms40that can be associated with specific meanings and impacts within the contrast enhancement heuristics28that can be performed by the system20. Peaks42and valleys44are discussed in greater detail below.

K Stretch Heuristics and Stretch Metrics

A stretch heuristic46is the process by which the system20manipulates, influences, or “stretches” a range of pixel values38to enhance the “contrast” of those pixels36, enhancing the contrast in the image as a whole. The system20can include a wide variety of different stretch heuristics46, sometimes applying multiple stretch heuristics46to a single image. One category of examples is illustrated inFIG. 1. Stretch heuristics are discussed in greater detail below.

A single stretch heuristic46will often generate multiple stretch metrics48. Each pixel36subjected to the processing of the stretch heuristic46can be associated with a stretch metric48that is associated with the particular pixel36. The stretch metric48can also be referred to as an “intensity” value.

Dark (e.g. low-contrast) images are typically made up of pixels36with pixel values38of a relatively small range (e.g. 0 through 50 even though pixel values38can range from 0 through 255). The stretch heuristic46is the process by which the range of pixel values38are stretched out to cover the entire range of potential pixel values38. The stretch metric48is the measure by which the range of initial pixels36are stretched.

I. Suppression Heuristics and Suppression Metrics

In addition to enhancing the differences between pixels36(e.g. increasing or “stretching” the contrast between pixel values38), some embodiments of the system20can also be configured to suppress or reduce the contrast between certain pixel values38. This is done to reduce the glare within the image. For example, in a night time driving embodiment of the system20, the glare from the headlights can manifest itself as extremely high pixel values38relative to the other pixels36in the initial image22. A suppression heuristic50can be used to reduce the contrast of those pixel values38.

A suppression metric51represents the degree of magnitude by which a particular pixel value38is suppressed. In some respects, the suppression heuristics50/metrics51can be thought of as mirror images of the stretch heuristics46/metrics48.FIG. 1illustrates one category of examples relating to suppression heuristics50and metrics51. Suppression heuristics50and suppression metrics51are discussed in greater detail below.

J. Identification Heuristics and Image Regions

In some embodiments of the system20, the initial image22is “broken down” into image regions52before any enhancement processing is performed on the image. In many region-based embodiments, each image region52is processed independently of the other regions52. For example, with respect to the histogram40processing discussed below, each image region52could have its own histogram40.

The system20can use a variety of region identification heuristics54known in the art to identify image regions52within the initial image22. Depending on the purpose and operating environment of the application being supported by the system20, the system20can be configured to generate relatively more or relatively fewer image regions52from the initial image22. Multi-region processing is discussed in greater detail below.

A bilinear interpolation heuristic56is a process that can be used in multi-region embodiments of the system20to “smooth” the enhanced image after the various enhanced image regions52are reassembled into a single image. The bilinear interpolation heuristic56is not necessary in all embodiments of the system20. The bilinear interpolation heuristic56is discussed in greater detail below.

III. Activation of the Enhancement Process

FIG. 2is a flow chart diagram illustrating an example of an enhancement invocation heuristic69, the process by which a system20can activate the process of enhancing initial images22. The invocation heuristic69can also be referred to as an activation heuristic or an initiation heuristic. In some embodiments of the system20, the enhancement heuristic28is automatically invoked on a continuous or substantially continuous basis. In other embodiments, the enhancement heuristic28is only invoked when the initial image22is not “satisfactory” to the system20, one or more applications making use of the system20, or to potentially one or more end users of the system20.

At70, the contrast metric30is generated by the system20. One example of a process for generating a contrast metric30is displayed inFIG. 3. InFIG. 3, an example of a global (e.g. image-wide) contrast equation80is displayed. G(x,y) is the pixel value38associated with each pixel36in the initial image22, e.g. all (x,y) values in R, the range of (x,y) values in the initial image22. The variable m represents the mean pixel value38. The variable N represents the total number of pixels36in the initial image22. The equation80inFIG. 3is used to estimate or influence the normalized standard deviation of gray or intensity levels.

Returning to72ofFIG. 2, the contrast metric30is compared to the contrast threshold32. In many embodiments, the end user of the system20can adjust the contrast threshold32of the system20. In other embodiments, the contrast threshold32is predefined before the system20is implemented, although the contrast threshold32can still be changed by non-end users after implementation if such adjustments are desired. In driver assistance embodiments of the system20with predefined contrast thresholds32, a contrast threshold32of60(assuming a pixel value38range between 0 and 255 gray or intensity levels) can be desirable. Any potential pixel value38(typically a value between 0 and 255 gray or intensity levels) can potentially serve as a useful contrast threshold32.

In some embodiments of the system20, the contrast threshold32must exceed the contrast metric30to activate the contrast enhancement heuristic28. In other embodiments, the contrast enhancement heuristic28is activated so long as the contrast metric30is less than or equal to the contrast threshold32.

The processing loop illustrated inFIG. 2can be configured to continuously repeat, or it can be configured to be triggered by specific triggering events.

By allowing the enhancement heuristic28to be turned on and off, the system20can be used to automatically transition from the display of images captured in low-contrast environments to the display of images captured in high-contrast environments.

IV. Process Flow View of an Activated Enhancement Heuristic

The enhancement heuristic28can vary widely in different embodiments of the system20.FIG. 4provides an example of an activated contrast enhancement heuristic99that can enhance images in a real-time or substantially real-time manner. The activated contrast heuristic99displayed inFIG. 4presumes that night vision and other low contrast initial images22are associated with bimodal histograms (e.g. bimodal histogram embodiments). Bimodal histograms are discussed in greater detail below. The system20is not limited to the use of bimodal histograms44. The image regions52may also be associated with different bimodal histogram embodiments. Different types of histogram-based processing and presumptions can be incorporated into the system20.

A histogram computation is performed at100.

FIG. 5is a histogram diagram illustrating an example of an initial histogram110populated with pixel values38from an initial image22. The horizontal (x) axis represents the pixel values38associated with pixels36in the initial image22. The vertical (y) axis represents the number of pixels36that are associated with a particular pixel value38. As is illustrated inFIG. 5, pixel values38range from 0 through 255 gray or intensity levels. A first peak112in the histogram110can be seen inFIG. 5.

FIG. 6is a zoomed, close-up, or amplified view (collectively amplified view120) of the histogram diagram110inFIG. 5. In the amplified view, a second peak124is visible. In the amplified view, a valley (e.g. first valley)122between the first peak112and second peak124is visible in the diagram. The structure of an initial histogram with two peaks42surrounding a valley44is a presumption of the system20in a bimodal embodiment. Other embodiments of the system20may make different assumptions and/or presumptions relating to the attributes of the initial histogram110.

The processes for identifying peaks42and valleys44are discussed below. The identification of the valley44can play an important part in identifying the magnitude by which pixel values38should be enhanced by the system20.

Returning toFIG. 4, the system20can perform a histogram smoothing heuristic at102. A variety of smoothing heuristics are known in the art. One example of a soothing heuristic is a one-dimensional Gaussian filter.

FIG. 7is a histogram diagram illustrating an example of an initial histogram130that has been subject to the smoothing heuristic of a one-dimensional Gaussian filter.

At104ofFIG. 4, the system20can identify the “zero-crossing” point of the histogram40. This process involves creating a histogram derivative from the smoothed histogram, and then creating a quantized derivative diagram from the histogram derivative.

A first order derivative can be computed from the smoothed histogram130ofFIG. 7. An example of a first order derivative132is displayed inFIG. 8. The system20can then quantize the derivatives into three levels of −1, 0, and 1, and detect the zero-crossing points where the derivative decreases from 1 to −1. As illustrated inFIG. 9, the segment between pixel value93and pixel value247represents a zero-crossing “point” or segment.

D. Valley Detection

Returning toFIG. 4, a valley detection heuristic can be performed at106. The zero-crossing point that has the maximum number of accumulating pixels36is identified as the valley44.

In the example ofFIG. 9, the zero-crossing point runs from the pixel value of 93 through the pixel value of 247. The valley threshold value is 93 because that is the lowest pixel value38in the valley.

E. Enhance Image

Returning toFIG. 4, the system20at108can create the enhanced image24.

The enhanced image24can be created through the invocation of the stretching heuristic46. The system20invokes the stretching heuristic to stretch out the narrow range of pixel values38in the initial image22(typically between 0 and approximately 50 gray or intensity level in a poor contrast environment) and “stretch” out the range of values to the full range of potential pixel values38(typically between 0 and 255 gray or intensity levels). The stretch metric48(which can also be referred to as an “intensity value”) determines the magnitude of the stretch processing.

In some embodiments of the system20, the stretch heuristic46is a linear stretch heuristic. In other embodiments, the stretch heuristic46can be a non-linear stretch heuristic.

One example of a linear stretch heuristic is provided in the equation below.
Stretch Metric=G(x,y)*maximum potential pixel value Valley threshold value

In the equation, G(x,y) represents the pixel value38of a particular pixel36. The valley threshold value is identified and discussed above. The maximum potential pixel value is the highest possible pixel value38in the range of potential pixel values (typically running from 0 through 255).

A stretch metric48can be calculated for each pixel36to be “stretched” by the stretch heuristic46.

The enhancement process at108can in some embodiments include the invocation of the suppression heuristic50that suppresses pixel values that are probably too high as the result of glare. For example, headlights on the highway can distort the appropriate contrast of the initial image22. The pixel values38that are above a particular value can be suppressed by a predefined amount. For example, any pixel value38that exceeds (1) the valley threshold value; (2) the stretch metric; or (3) any other point of reference, can be reduced by a predefined percentage, such as between 5% and 25%. In many vehicle operation environments, pixel values38that exceed the valley threshold value are reduced by 10%. Other suppression heuristic values can be used to calculate or influence a pre-determined threshold pixel value38.

The modified or “stretched” pixels36can then be reassembled into the enhanced image24. The process inFIG. 4can then repeat again for the next captured initial image22.

V. Image Region Embodiments

In some embodiments of the system20, the processing identified above with respect to the activated enhancement heuristic99(an example of which is provided inFIG. 4) is performed with respect to the individual image regions52within the initial image22instead of to the initial image22as a whole. In “image region” embodiments of the system20, the initial image22is divided into a variety of image regions52.

For example, the “global” contrast metric30would actually relate solely to a particular image region52. Thus, each image region52could have its own: initial histogram130, histogram derivative132, quantized derivative134, contrast metric30, stretch metric48, and suppression metric51.

For each image region52, the initial histogram130can be truncated according to a user supplied contrast factor and/or the average number of pixels36at each pixel value (e.g. gray or intensity level)38. The surplus pixels36can be redistributed iteratively among all of the pixel values38(e.g. typically pixel values 0-255). This can allow the user to influence or even control how much the contrast within the initial image22is improved by the system20. A larger contrast factor will result in greater contrast enhancement. The bilinear interpolation heuristic56can be invoked to smooth the entire image after the image regions52are reassembled into a single interim image34.

In many embodiments of the system20, it can be beneficial to use predefined sizes for the image regions52and predefined contrast factors. For example, it can be useful to divide the initial image22evenly into 8×8 image regions52along x and y directions. It can also be useful to set the contrast factor at 6.0. The image region size can vary based on the application or other contrast threshold-related factors

A variety of different region identification heuristics54are known in the art, and can be used to divide up the initial image22into various image regions52. Some of those heuristics54involve a dynamic process in which the number of regions52and the boundaries of those regions52differ with each initial image22. In other embodiments, the number of regions52and the boundaries of those regions52are static and predefined.

VI. Process Flow View of Implementing the System

FIG. 10is a flow chart diagram illustrating an example of an implementation heuristic140for implementing the system20.

At142, a user control is configured to modify a contrast threshold32. In image region embodiments of the system20, the user control could be used to modify the contrast factor and/or the contrast threshold32. In certain embodiments of the system20, the ability of an end user to modify the contrast threshold32and/or the contrast factor is constrained to a predefined range of values that are embedded into the system20in accordance with the design specification of the application utilizing the system20. In other embodiments, no limits are placed on range of values from which the user can choose from.

At144, an enhancement heuristic28is embedded into a device that is influenced by the user control device configured at142. For example, the user controller and the computer23housing the enhancement heuristic28could be embedded into a vehicle. The system20could then be activated by an end user to obtain the functionality of image enhancement. In a night vision embodiment of the system20for use in a vehicle, the system20can function effectively with a sensor speed of approximately 30 frames per second on a computer with a processing speed of 3 GHz.

In accordance with the provisions of the patent statutes, the principles and modes of operation of this invention have been explained and illustrated in preferred embodiments. However, it must be understood that this invention may be practiced otherwise than is specifically explained and illustrated without departing from its spirit or scope.