# Patent

## Title
System and method for adaptively sharpening digital images

## Abstract

Embodiments of the invention provide a technique to sharpen luminance data associated with each pixel of a digital image without introducing certain sharpening artifacts. An alpha mask is generated based on first, second, and third alpha submasks that indicate how strongly overshoot, color fringe, and dark regions, respectively, are detected at each pixel location. The alpha mask is used to combine luminance data derived from the digital image with sharpened luminance data that is generated by applying a sharpening filter to the luminance data.

## Background
BACKGROUND OF THE INVENTION
1. Field of the Invention
The invention relates generally to image processing and, more specifically, to a system and method for adaptively sharpening digital images.
2. Description of the Related Art
Digital camera technology has developed rapidly over the past decade. A broad variety of digital cameras are now available to meet the diverse needs of a wide spectrum of consumers. Digital camera technology provides a user with the ability to capture digital images and/or digital video, and also provides the ability to transfer the recorded digital images and/or digital video to a computer-readable medium, such as a Digital Video Disc (DVD) or a computer memory, or to a physical medium, such as photographic paper.
Additionally, once image data and/or video data is captured by the digital camera, the data may be transferred to a computer system via a cord or connector that couples the digital camera to the computer system. Common types of connectors provided with digital cameras are universal serial bus (USB) connectors, firewire connectors, High-Definition Multimedia Interface (HDMI) connectors, proprietary connectors, or other types of connectors that may be used to transfer data. Some digital cameras may include a cord that connects the digital cameras to the computer system; whereas, other digital cameras may include a connector that protrudes from the digital camera and can be plugged directly into the computer system.
As is known, video data is comprised of a series of digital images. Each digital image is represented by a two-dimensional (2D) grid of pixels that may be displayed on a screen, such as a television set or a computer monitor, or printed on high-quality paper to provide a photographic still image. The color and brightness of the pixels can be represented using different formats. For example, each pixel may be defined by three values that define the brightness of red, green, and blue (RGB) color components associated with the pixel. The brightness value of each color typically falls between 0 and 1, where 0 represents a lack of the color, and 1 represents the maximum brightness of the color. Digital images that are represented as RGB values are referred to herein as “RGB images.” Each pixel of the digital image may also be defined by a luminance component (Y) that defines the brightness of the pixel, and two chrominance components (U and V) that define the color of the pixel. Digital images composed of pixels that are represented with YUV components are referred to herein as a “YUV images.”
RGB images and/or YUV images may be filtered to produce higher-quality digital images. For example, a digital image of either format may be sharpened using a sharpening filter to produce data that, when displayed on a screen, appears clearer. A typical sharpening filter amplifies the differences in brightness from pixel-to-pixel in a digital image. For example, a sharpening filter may modify the luminance component of a YUV image to provide a sharper YUV image. One problem with this approach is that typical sharpening filters often introduce a variety of distortions into the digital image. These distortions are known as “sharpening artifacts” that may be introduced into the digital image under several different circumstances.
For example, sharpening artifacts known as “bright overshoot” and “dark undershoot,” known collectively as “overshoot,” may be introduced into the digital image by the sharpening filter within regions of the digital image that rapidly transition from a bright area to a dark area. Such a region may be indicative of an edge between different objects represented by the digital image. This effect is illustrated in FIG. 9. As shown, FIG. 9 includes graph 900 of horizontal pixel position versus luminance based on a YUV image, according to the prior art. Original edge 910, represented by a solid line, indicates that the luminance of the YUV image decreases from a first value to a second value across the horizontal line of pixels. The region of the YUV image where this decrease occurs is shown on graph 900 as extended high gradient region 930.
After the YUV image is sharpened, the luminance value decreases more rapidly from the first value to the second value across extended high gradient region 930. However, the luminance value is also increased at the left-most extremity of the extended high gradient region 930, resulting in bright overshoot, and decreased at the right-most extremity of the high gradient region 930, resulting in dark undershoot. Bright overshoot causes excessively bright pixels to appear within a bright region that is near a boundary with a dark region, while dark undershoot causes excessively dark pixels to appear within a dark region that is near a boundary with a bright region. These excessively bright and dark pixels may disrupt the overall quality of the image and may be visually displeasing.
Another type of artifact is known as “color fringe.” When the RGB values of the RGB image are in focus, as shown in FIG. 10A, the RGB brightness values each change at the same rate across a boundary between a bright region and a dark region. However, when the RGB values are not in focus, as shown in FIG. 10B, these brightness values may change with different rates across such a boundary, resulting in a fringe near the boundary. Within this fringe, one color appears brighter than the other colors. This may be caused by, among other things, flaws in the lens of the DVC. The brightness of the fringe may be amplified when the image is sharpened due to the effects of overshoot described above, further exacerbating the color fringe and reducing the overall quality of the image.
A third type of artifact, referred to herein as a “dark artifact,” occurs when a dark region of a digital image is sharpened. Dark regions of a digital image may include random noise that, when sharpened, appears more pronounced. This amplified noise disrupts the continuity of the dark region and decreases the overall clarity of the digital image.
Accordingly, there remains a need in the art for a technique to sharpen digital images without introducing sharpening artifacts.
SUMMARY OF THE INVENTION
One embodiment of the invention sets forth a computer-implemented method for sharpening a digital image that is H pixels in height and W pixels in width. The method includes the steps of generating a plurality of luminance values for the digital image, where each luminance value corresponds to a different pixel location in the digital image, generating a plurality of sharpened luminance values for the digital image by applying a sharpening filter to one or more of the luminance values, where each sharpened luminance value corresponds to a different pixel location in the digital image, and generating a sharpened digital image for output to a display or for storage in a memory by selecting, for each pixel location in the digital image, either the luminance value in the plurality of luminance values corresponding to the pixel location or the sharpened luminance value in the plurality of sharpened luminance values corresponding to the pixel location based on a value in an alpha mask corresponding to the pixel location.
In various embodiments, the alpha mask is derived from at least one of an overshoot alpha submask that includes, for each pixel location in the digital image, an overshoot value that is indicative of how strongly an overshoot artifact is detected at the pixel location by the sharpening filter, a fringe alpha submask that includes, for each pixel location in the digital image, a fringe value that is indicative of how strongly a fringe artifact is detected at the pixel location by the sharpening filter, and a dark alpha submask that includes, for each pixel location in the digital image, a dark value that is indicative of whether a dark artifact was introduced at the pixel location by the sharpening filter.
Other embodiments of the invention set forth computer-readable media that store instructions that can configure a processing unit to generate the overshoot alpha submask and the fringe alpha submask.
Yet other embodiments of the invention include computer-implemented methods directed to one or more variations of the functionalities that can be implemented via the above computer-readable media as well as electronic devices configured to perform one or more variations of the functionalities that can be implemented via the above computer-readable media.
One advantage of the disclosed techniques is that they provide an adaptive sharpening filter that improves overall image sharpness, but reduces the incidence of sharpening artifacts relative to prior art image sharpening techniques.



BRIEF DESCRIPTION OF THE DRAWINGS
So that the manner in which the above recited features of the invention can be understood in detail, a more particular description of the invention, briefly summarized above, may be had by reference to embodiments, some of which are illustrated in the appended drawings. It is to be noted, however, that the appended drawings illustrate only typical embodiments of this invention and are therefore not to be considered limiting of its scope, for the invention may admit to other equally effective embodiments.
FIG. 1 is a block diagram of a hand-held device configured to implement one or more aspects of the invention;
FIG. 2 is a conceptual diagram illustrating an adaptive sharpening filter, according to one embodiment of the invention;
FIG. 3 is a flowchart of method steps for adaptively sharpening a digital image, according to one embodiment of the invention;
FIG. 4 is a conceptual diagram illustrating an overshoot detector, according to one embodiment of the invention;
FIG. 5 is a flowchart of method steps for generating an overshoot alpha submask, according to one embodiment of the invention;
FIG. 6 is a conceptual diagram illustrating a color fringe detector, according to one embodiment of the invention;
FIG. 7 is a flowchart of method steps for generating a color fringe alpha submask, according to one embodiment of the invention;
FIG. 8 is a conceptual diagram illustrating a dark artifact detector, according to one embodiment of the invention;
FIG. 9 is a graph representing an overshoot artifact, according to the prior art;
FIG. 10A is a graph representing focused red, green and blue (RGB) intensity values, according to the prior art; and
FIG. 10B is a graph representing a color fringe artifact, according to the prior art.



DETAILED DESCRIPTION
In the following description, numerous specific details are set forth to provide a more thorough understanding of the invention. However, it will be apparent to one of skill in the art that the invention may be practiced without one or more of these specific details. In other instances, well-known features have not been described in order to avoid obscuring the invention.
FIG. 1 is a block diagram of a system 100 configured to implement one or more aspects of the invention. As shown, the system 100 includes a digital camera 102 and a computer system 160. The digital camera 102 includes, without limitation, a data connector 108, a speaker 104, a microphone 106, a power supply 110, optical components 112, a digital image sensor 114, a central processing unit (CPU) 116, a preview screen 118, interface buttons 120, and an internal memory 138. In one embodiment, the digital camera 102 is a one-time use and/or disposable digital camera. In alternative embodiments, the digital camera 102 is a reusable digital camera. In still further embodiments, the digital camera 102 is a digital video camera.
The data connector 108 is an integrated mechanism that allows the digital camera 102 to be coupled to the computer system 160 or to a separate TV, and to transfer data to and from the computer system 160 and/or output video and audio to the TV. The data connector 108 may be a universal serial bus (USB) connector, a firewire connector, a HDMI connector, a serial connector, or another type of connector that is capable of connecting the digital camera 102 with the computer system 160 or the TV. In alternative embodiments, the data connector 108 is not an integrated mechanism and may include a cable connector.
In one embodiment, the digital camera 102 is configured to capture still image data of a particular scene through the optical components 112 and the digital sensor 114. In alternative embodiments, the digital camera 102 may be configured to capture video data through the optical components 112 and the digital sensor 114 and to capture audio of the particular scene through the microphone 106. When the digital camera 102 is in an image capture mode or video capture mode, the preview screen 118 displays a preview of the image to be captured. When in a playback mode, the digital camera 102 is configured to show previously-captured digital images and/or digital videos (comprised of captured video data and audio data) that are stored in the internal memory 138. When the preview screen is displaying previously-captured digital video, the audio data associated with the digital video is output through the speaker 104. In alternative embodiments, the previously-captured digital images and/or digital videos may be output to the TV or to the computer system 160 for playback.
The power supply 110 provides power to the digital camera 102. The power may be provided by a battery or an external power source (e.g., an AC outlet). In one embodiment, the battery is a rechargeable battery that may or may not be removable from the digital camera 102.
The interface buttons 120 include a power button and a capture button. The power button is configured to turn the digital camera ON and OFF. When the digital camera 102 is in the image capture mode, the capture button is configured to initiate the capture of a still image; whereas, when the digital camera 102 is in the video capture mode, the capture button is configured to initiate the capture of a video sequence. In some embodiments, the other interface buttons, which may include a left button, a right button, an increase button, a decrease button, a display button, and/or a delete button, may be implemented as capacitive-touch buttons. In alternative embodiments, these other interface buttons may be implemented as mechanical buttons, induction buttons, analog-resistive buttons, or any other technically feasible button type or combination of button types.
The CPU 116 communicates with the various components within the digital camera 102 to control the operations of the digital camera 102. The CPU may be implemented as a single chip or as a combination of multiple chips. The CPU 116 also processes inputs from the interface buttons 120. For example, when the digital camera 102 is in the image capture mode or the video capture mode, the CPU 116 transmits the data received from the digital sensor 114 to the preview screen 118 for display. Similarly, the CPU 116 may transmit the data to the internal memory 138 for storage. The internal memory 138 also stores firmware that is executed by the CPU 116 to control the operations of the digital camera 102.
The CPU 116 may include a digital signal processor (DSP) 117 and/or a microprocessor (MP) core 115. The DSP 117 includes logic optimized to efficiently perform digital signal processing tasks, such as mathematically intensive computations. In one embodiment, the DSP 117 comprises a co-processor configured to execute instructions independently from the MP core 115. The MP core 115 may be configured to execute a control application 142 stored in the internal memory 138, and the DSP 117 may be configured to execute a DSP application 144, also stored in the internal memory 138. Persons skilled in the art will recognize that the control application 142 and the DSP application 144 may be loaded into the CPU 116 for execution. In one embodiment, the DSP application may reside in embedded memory within the CPU 116.
The internal memory 138 comprises either volatile memory, such as dynamic random access memory (DRAM), or non-volatile memory, such as a hard disk or a flash memory module, or a combination of both volatile and non-volatile memory. The internal memory 138 also stores a software driver 140 implemented as a set of program instructions configured to coordinate operation between the interface buttons 120 and the other components of the digital camera 102.
In various embodiments, the digital camera 102 may be coupled to the computer system 160 via the data connector 108. The computer system 160 may be any type of computing device, such as a laptop or a desktop computer, a server computer, or a video processing kiosk, and includes a memory 162 and a processor 164. Other components included in the computer system 106, such as a display device, a disk drive, user input devices, and the like, are omitted to avoid obscuring embodiments of the invention.
As set forth in greater detail herein, embodiments of the present invention involve techniques for processing image data. The image data may be of any origin. For example, the image data may be associated with one or more still images captured by the digital camera 102 or may be associated with one or more still images derived from video data captured by the digital camera 102. The image data also may be associated with one or more still images stored on the computer system 160 or downloaded to the computer system 160 or may be associated with one or more still images derived from video data stored on the computer system 160 or downloaded to the computer system 160. The image processing techniques disclosed herein may be implemented solely by the processing components within the digital camera 102, when the digital camera 102 is either coupled to or decoupled from the computer system 160, or may be implemented solely by the processing components within the computer system 160, when the digital camera 102 is either coupled to or decoupled from the computer system 160. In other embodiments, the image processing techniques may be implemented in part by the processing components within the digital camera 102 and in part by the processing components within the computer system 160 while the digital camera 102 is coupled to the computer system 160.
FIG. 2 is a conceptual diagram illustrating an adaptive sharpening filter 200, according to one embodiment of the invention. As shown, the adaptive sharpening filter 200 includes an RGB to YUV converter 202, a sharpening filter 204, a YUV to RGB converter 206, a sharpening artifact detector 208, an alpha mask unit 210, and a YUV to RGB converter 212. The adaptive sharpening filter 200 is configured to receive a digital image that includes a red, green and blue (RGB) color value for each pixel of the digital image, and to generate a sharpened digital image.
The RGB to YUV converter 202 is configured to convert digital RGB data to YUV data. The RGB to YUV converter 202 receives the RBG data, ARGB, that is input to the adaptive sharpening filter 200. ARGB is H pixels in height and W pixels in width. Each pixel of ARGB includes an red, a green, and a blue color value that is a decimal and ranges from 0 to 1. In one embodiment, each pixel of ARGB is represented by an 8-bit integer value that ranges from 0 to 255. The RGB to YUV converter 202 generates YUV data, AYUV, based on the received digital RGB data. Each pixel of AYUV is defined by a luminance value that defines the brightness of the pixel, and two chrominance values that define the color of the pixel. The luminance values associated with AYUV are included in a luminance array, AY, while the chrominance components of AYUV are included within separate chrominance arrays, AU, and AV. Each of AY, AU, and AV, is an H×W array of decimal values ranging from 0 to 1. The RGB to YUV converter 202 transmits AY to the sharpening filter 204, to the sharpening artifact detector 208, and to the alpha mask unit 210. The RGB to YUV converter 202 also transmits AU and AV to the YUV to RGB converter 206 and to the YUV to RGB converter 212.
The sharpening filter 204 is a linear filter that is configured to perform a sharpening operation. The sharpening filter 204 receives AY from the RGB to YUV converter 202 and sharpens the received data to produce sharpened luminance array h(AY), where h( ) is a sharpening function. h(AY) includes a sharpened luminance value for each pixel location of the digital image, where each sharpened luminance value is a decimal that ranges from 0 to 1. Those skilled in the art will recognize that a variety of different types of linear filters are readily available and may be implemented, both in software, hardware, or a combination of hardware and software, to implement the sharpening filter 204. The sharpening filter 204 transmits h(AY) to the YUV to RGB converter 206, sharpening artifact detector 208, and alpha mask unit 210.
The YUV to RGB converter 206 is configured to convert YUV data to digital RGB data. The YUV to RGB converter 206 receives h(AY) from the sharpening filter 204, as well as AU and AV from the RGB to YUV converter 202. The YUV to RGB converter 206 then converts the received YUV data into digital RGB data, BRGB. BRGB is a sharpened version of ARGB that is the same size as ARGB and includes sharpened RGB color values. The YUV to RGB converter 206 transmits BRGB to the sharpening artifact detector 208.
The sharpening artifact detector 208 is configured to generate an alpha mask α. Alpha mask α is an H×W array of decimal values that range from 0 to 1 and that each correspond to a different pixel location in the digital image. Alpha mask α indicates regions of h(AY) where different sharpening artifacts have been detected. For a given pixel location, an alpha mask α value of 0 at that pixel location indicates that a sharpening artifact is strongly detected in h(AY). Conversely, an alpha mask α value of 1 at a given pixel location indicates that a sharpening artifact is not strongly detected in h(AY).
The sharpening artifact detector 208 receives the original digital RGB data, ARGB, as well as AY, h(AY), and BRGB and generates three alpha submasks that each indicate the likelihood of a different sharpening artifact. Each alpha submask is equal in size and data range to alpha mask α, and each alpha submask includes a value for each pixel of the digital image. The alpha submasks are combined to produce alpha mask α. The first alpha submask, αovershoot, is generated from AY and h(AY) and identifies regions of h(AY) where an overshoot artifact is detected. αovershoot is described in further detail in FIGS. 4-7. The second alpha submask, αfringe, is generated from BRGB and identifies regions of h(AY) where a color fringe artifact is detected. αfringe is described in further detail in FIGS. 6-9. The third alpha submask, αdark, is generated from AY and identifies dark regions of AY where dark artifacts are detected. αdark is described in further detail in FIG. 10.
The sharpening artifact detector 208 is configured to combine αovershoot, αfringe, and αdark to produce alpha mask α. In one embodiment, alpha mask α is generated according to Equation 1, where the function min(A, B) selects the minimum value of A and B:

α=(1−min(1, αovershoot+αfringe))*(1−αdark)   (1)

Those skilled in the art will understand that subtracting min(1, αovershoot+αfringe) from 1 and subtracting αfringe from 1, as shown in Equation 1, is performed by subtracting each value of min(1, αovershoot+αfringe) and αfringe from 1. In one embodiment, each value of αovershoot is added to the corresponding value of αfringe to generate a submask sum, and the function min(1, αovershoot+αfringe) generates a limited submask sum. In further embodiments, each value of the limited submask sum is subtracted from 1 to generate a reverse submask, and each value of αfringe is subtracted from 1 to generate a reverse dark alpha submask.
Alpha mask α is transmitted to the alpha mask unit 210. The alpha mask unit 210 is configured to receive alpha mask α, as well as AY and h(AY). Alpha mask unit 210 is further configured to combine AY and h(AY) using alpha mask α to generate an array of adaptively sharpened luminance values, CY. CY is an H×W array of decimal values that range from 0 to 1. In one embodiment, the alpha mask unit 210 combines AY and h(AY) to generate CY according to Equation 2:

CY=((1−α)*AY)+(α*h(AY))   (2)

Those skilled in the art will understand that CY is calculated by executing Equation 2 with each value of α and the corresponding value of AY. In one embodiment, α is subtracted from 1 to generate a reverse alpha mask, each value of the reverse alpha mask is multiplied by the corresponding value of AY to generate a first luminance portion, and each value of α is multiplied by the corresponding value of AY to generate a second luminance portion.
According to Equation 2, for each pixel value of h(AY), the corresponding value of alpha mask cc determines the fraction of that pixel value that contributes to CY. In addition, for each pixel value of AY, 1 minus the corresponding value of alpha mask α determines the fraction of that pixel value that is contributed to CY. The sum of these contributions determines each pixel value of CY. Those skilled in the art will recognize that other equations may be used to determine CY based on alpha mask α, AY, and h(AY). Once CY has been calculated, the alpha mask unit 210 transmits CY to the YUV to RGB converter 212.
The YUV to RGB converter 212 converts YUV data to RGB data. Accordingly, the output of the YUV to RGB converter 212 is adaptively sharpened RGB data, DRGB. The YUV to RGB converter 212 is configured to receive CY, AU and AV. The YUV to RGB converter 212 is further configured to generate DRGB based on the received data. DRGB is an array of RGB color values that is based on the original chrominance data, AU and AV, as well as the combined luminance data, CY, representative of a weighted combination of the original luminance data AY and the sharpened luminance data h(AY).
Once the adaptive sharpening filter 200 has generated DRGB from ARGB, DRGB may be output to a display screen, such as a cathode-ray tube (CRT) monitor, a liquid crystal display (LCD), a television (TV) set, or other type of display device. DRGB may also be stored in a memory unit or transmitted to a peripheral device such as a printer or copier and printed on a physical medium, such as photo paper. In one embodiment, adaptive sharpening filter 200 is used to process a sequence of digital images to produce adaptively sharpened video data.
In one embodiment, each component of the adaptive sharpening filter 200 described above may be implemented as a hardware component, such as an application-specific integrated circuit (ASIC), a digital signal processor (DSP), or a field-programmable gate array (FPGA), among others. In alternative embodiments, each component of the adaptive sharpening filter 200 may be implemented as a software program stored on a computer-readable medium that, when executed by a processor, performs the specific functionality associated with that component, as described above. In further alternative embodiments, the adaptive sharpening filter 200 may include any combination of hardware and/or software components. As described above in FIG. 1, the adaptive sharpening filter 200 may be executed by the CPU 116 included within the digital camera, or by the processor 164 included in the computer system 160.
FIG. 3 is a flowchart of method steps for adaptively sharpening a digital image, according to one embodiment of the invention. Persons skilled in the art will understand that, even though the method 300 is described in conjunction with the system of FIG. 1-2, any system configured to perform the method steps, in any order, is within the scope of the present invention.
As shown, the method 300 begins at step 302, where the adaptive sharpening filter 200 receives RGB data, ARGB. ARGB is an H×W array of RGB color values that each correspond to a different pixel location in the digital image. At step 304, the adaptive sharpening filter 200 converts ARGB to YUV data to produce AYUV. AYUV includes a luminance array AY, and two chrominance arrays AU, and AV, where AY includes the luminance values of AYUV, and AU and AV include the chrominance values of AYUV. Each of AY, AU, and AV is an H×W array of decimal values ranging from 0 to 1. In one embodiment, the RGB to YUV converter 202 included within the adaptive sharpening filter 200 performs steps 302 and 304.
At step 306, the adaptive sharpening filter 200 performs a sharpening operation using AY as input to generate sharpened luminance data, h(AY). h( ) is a sharpening function that implements a linear filter to provide sharpened data. Those skilled in the art will recognize that a wide variety of functions and/or linear filters may be implemented to perform the sharpening operation described herein. In one embodiment, the sharpening filter 204 included within the adaptive sharpening filter 200 performs step 306.
At step 308, the adaptive sharpening filter 200 converts h(AY), AU, and AV into RGB data to produce BRGB. BRGB represents a sharpened version of ARGB that is equal in size and data range to ARGB. In one embodiment, the YUV to RGB converter 206 included within the adaptive sharpening filter 200 performs step 308.
At step 310, the adaptive sharpening filter 200 identifies regions of h(AY) that are likely to include an overshoot artifact. These regions include, for example, a border between a bright area and a dark area. In one embodiment, the adaptive sharpening filter 200 only identifies regions where bright overshoot occurs. In another embodiment, the adaptive sharpening filter 200 only identifies regions where dark undershoot occurs. At step 312, the adaptive sharpening filter 200 generates an alpha submask, αovershoot, that is indicative of the regions where an overshoot artifact has been detected. αovershoot is an H×W array of decimal values that range from 0 to 1, where each value corresponds to a pixel location of h(AY) and represents the likelihood that an overshoot artifact exists at that pixel location. Those skilled in the art will recognize that a wide variety of techniques may be used to identify regions where overshoot artifacts are likely to occur. One such technique is described in further detail below in FIGS. 4-7.
At step 314, the adaptive sharpening filter 200 identifies regions of h(AY) that are likely to include a color fringe artifact. These regions include, for example, a border between a bright and a dark area of ARGB where the R, G, and B color components are focused with different sharpnesses. At step 316, the adaptive sharpening filter 200 generates an alpha submask αfringe that is indicative of the regions where a color fringe artifact is strongly detected. αfringe is an H×W array of decimal values that range from 0 to 1, where each value corresponds to a pixel location of h(AY) and represents how strongly a color fringe artifact is detected at that pixel location. Those skilled in the art will recognize that a wide variety of techniques may be used to identify regions where color fringe artifacts may occur. One such technique is described in further detail below in FIGS. 6-9.
At step 318, the adaptive sharpening filter 200 identifies the dark regions of AY. The dark regions of AY often include random noise that, when sharpened, becomes more noticeable. Those skilled in the art will recognize that a wide variety of techniques may be used to identify dark regions of a frame of luminance data. One such technique is described in further detail below in FIG. 10.
At step 320, the adaptive sharpening filter 200 generates an alpha submask αdark of size H×W that includes decimal values at each location that range from 0 to 1. Each value of αdark corresponds to a pixel location of h(AY) and represents whether the pixel location falls within a dark region of h(AY).
At step 322, the adaptive sharpening filter 200 combines αovershoot, αfringe, and αdark to produce alpha mask α. In one embodiment, the adaptive sharpening filter 200 generates alpha mask α using Equation 1, as previously described. In another embodiment, the sharpening artifact detector 208 included within the adaptive sharpening filter 200 performs steps 310, 312, 314, 316, 318, 320 and 322.
At step 324, the adaptive sharpening filter 200 combines AY and h(AY) using alpha mask α to produce adaptively sharpened luminance component CY. CY is an H×W array of decimal values ranging from 0 to 1. The values of pixels in CY that are located within regions where sharpening artifacts have been strongly detected are calculated based primarily on AY. The values of pixels in CY that are located within regions where sharpening artifacts have not been strongly detected are calculated based primarily on h(AY). The adaptive sharpening filter 200 may generate CY using Equation 2, as previously described herein. In one embodiment, the alpha mask unit 210 included within the adaptive sharpening filter 200 performs step 324.
At step 326, the adaptive sharpening filter 200 converts CY, AU, and AV into adaptively sharpened RGB data DRGB. DRGB includes color data derived from the original ARGB data. The luminance component CY associated with DRGB is based on a combination of the original luminance AY and the sharpened luminance h(AY). In one embodiment, the YUV to RGB converter 212 included within the adaptive sharpening filter 200 performs step 326.
DRGB may be output to a display, such as a CRT, LCD or TV screen, or printed on a physical medium, such as photo paper, using a printing device. In one embodiment, the adaptive sharpening filter 200 adaptively sharpens a single frame of RGB data. In alternative embodiments, the adaptive sharpening filter 200 sharpens a sequence of frames of RGB data to provide adaptively-sharpened video data.
As described above, the adaptive sharpening filter 200 generates αovershoot, αfringe, and αdark to produce alpha mask α. In alternative embodiments, alpha mask a is based on any combination of αovershoot, αfringe, or αdark. In the descriptions of FIGS. 4-10 that follow, embodiments of the invention are disclosed that provide techniques to generate αovershoot, αfringe, and αdark.
FIG. 4 is a conceptual diagram illustrating an overshoot detector 400, according to one embodiment of the invention. As shown, the overshoot detector 400 includes a vertical sobel filter 402, a horizontal sobel filter 404, square functions 406 and 408, a sum 410, a square root 412, a normalize function 414, a subtractor 416, dilators 418 and 420, a multiply 422, a normalize function 424, a dark mask unit 426, and a normalized sobel gradient magnitude 450. The overshoot detector 400 is configured to receive luminance data AY, sharpened luminance data h(AY) and to generate αovershoot.
The overshoot detector 400 receives AY into the normalized sobel gradient magnitude 450. The normalized sobel gradient magnitude 450 includes the vertical sobel filter 402, the horizontal sobel filter 404, the square functions 406 and 408, the sum 410, the square root function 412, and the normalize function 414. The normalized sobel gradient magnitude 450 receives AY and calculates the normalized sobel gradient magnitude of AY to generate a set of NSGM values included in an H×W array of NSGM values, denoted as ANSGM.
The normalized sobel gradient magnitude 450 receives AY into the vertical sobel filter 402 and into the horizontal sobel filter 404. As is well known, a sobel filter calculates the gradient of pixel values across a horizontal or a vertical line of pixels within a frame of data in order to detect edges within the frame of data. Those skilled in the art will recognize that, in addition to the sobel filter, other techniques may also be used to detect edges within a frame of data. The vertical sobel filter 402 calculates the gradient of AY across a vertical line of pixels to generate a vertical sobel gradient (VSG) for each pixel location of the digital image. Each VSG value is included in an H×W array of VSG values, denoted as AVSG. The vertical sobel filter 402 transmits the AVSG to the square function 406. Square function 406 squares each value of AVSG and then transmits AVSG2 to sum 410.
The horizontal sobel filter 404 calculates the gradient of AY across a horizontal line of pixels to generate a horizontal sobel gradient (HSG). Each HSG value is included in an H×W array of HSG values, denoted as AHSG. The horizontal sobel filter 404 transmits AHSG to the square function 408. The square function 408 squares each value of AHSG and then transmits AHSG2 to sum 410.
Sum 410 adds each pixel value of AVSG2 to the corresponding pixel value of AHSG2 to generate a combined gradient denoted Gsum. Sum 410 then transmits Gsum to the square root function 412. The square root function 412 calculates the square root of each value of Gsum to generate a sobel gradient magnitude (SGM) for each pixel location of the digital image. Each SGM value is included in an H×W array of SGM values denoted as ASGM. The square root function 412 transmits ASGM to the normalize function 414. The normalize function 414 identifies the maximum value of ASGM and then divides every value of ASGM by the maximum value. Those skilled in the art will understand that in order to determine the maximum pixel value, each pixel value is analyzed and the maximum pixel value is determined as the pixel value that exceeds all of the other pixel values. The result of this operation is an array of NSGM values corresponding to each pixel location of the digital image. Each NSGM value is included in an H×W array of NSGM values denoted as ANSGM 
Each NSGM value output by the normalized sobel gradient magnitude 450 corresponds to a value of AY and indicates whether a pixel within a given region of AY is located on a border between a high-luminance region and a low-luminance region of AY (i.e., a region where the magnitude of the gradient of AY is high). When a pixel is located on such a border, the NSGM value at the pixel location associated with the pixel is close to 1. When a pixel is located within a region of AY where the luminance value is relatively constant over a broad area, the NSGM value corresponding to that pixel location will be close to zero. Typically, a border between a high-luminance region and a low-luminance region within AY is associated with one or more edges of objects that may be included within the original RGB data received by the adaptive sharpening filter 200. Accordingly, normalized sobel gradient magnitude 450 may be used to identify the pixel locations associated with AY where edges are likely to occur.
Those skilled in the art will recognize that the processing operations performed by the vertical sobel filter 402, the horizontal sobel filter 404, the square function 406 and 408, the sum 410, the square root function 412, and the normalize function 412 to generate ANSGM, as described above, represent one possible technique that may be used to generate a normalized sobel gradient magnitude. In other embodiments, the normalized sobel gradient magnitude 450 generates each NSGM value using a different sequence of processing operations.
Those skilled in the art will further recognize that normalized sobel gradient magnitude 450 represents just one technique for identifying pixel locations in a digital image where edges are detected and, further, that alternative techniques are also available that may produce results similar to those produced by normalized sobel gradient magnitude 450. Accordingly, normalized sobel gradient magnitude 450 may be replaced by an alternative technique capable of identifying pixel locations associated with AY where edges are detected without substantially altering the overall functionality of the overshoot detector 400.
The normalized sobel gradient magnitude 450 transmits ANSGM to the subtractor 416 and to the dilator 418. The subtractor 416 is configured to subtract each value of ANSGM from 1 to generate a reverse NSGM for each pixel location of the digital image. Each reverse NSGM is included in a reverse NSGM array denoted as RVNSGM, which is an array that is the same size and data range as ANSGM (i.e., an H×W array of decimal values that range from 0 to 1). RVNSGM represents regions where edges are not strongly detected. Accordingly, when a pixel is located within a region of AY where the luminance is relatively constant over a broad area, the RVNSGM value corresponding to that pixel location will be close to 1. The subtractor 416 transmits RVNSGM to the dilator 420.
Dilators 418 and 420 receive an H×W array of data, where each value in the array corresponds to a pixel location. For each pixel location associated with the received array, dilators 418 and 420 determine the maximum pixel value within a certain neighborhood of pixels surrounding the pixel location. Those skilled in the art will understand that in order to determine the maximum pixel value within a certain neighborhood of pixel values, each pixel value in the neighborhood is analyzed and the maximum pixel value is determined as the pixel value that exceeds all of the other pixel values in the neighborhood. Dilators 418 and 420 generate an H×W dilated version of the received array, where each pixel value of the dilated version of the received array corresponds to the maximum value determined from the neighborhood associated with the pixel location in the received array.
Dilator 418 receives ANSGM from the normalized sobel gradient magnitude 450 and performs the dilation operation described above. In one embodiment, the neighborhood of pixels surrounding each pixel location is a P×Q rectangular neighborhood, where P and Q are integers. Those skilled in the art will recognize that neighborhoods of other sizes and shapes are also possible. Dilator 418 dilates ANSGM to generate a dilated version of ANSGM denoted as d(ANSGM). d(ANSGM) indicates regions of AY that are near pixels having a high gradient. For example, if AY includes a distinct edge where a high-luminance region borders a low-luminance region, pixels located along the distinct edge have a high gradient. The corresponding locations in d(ANSGM), as well as the nearby locations, will have a value that is close to 1. Dilator 418 transmits d(ANSGM) to multiply 422.
Dilator 420 receives RVNSGM from subtractor 416 and performs the dilation operation described above. In one embodiment, the neighborhood of pixels is equivalent to the neighborhood implemented by dilator 418. Those skilled in the art will recognize that neighborhoods of other sizes and shapes are possible. Dilator 420 dilates ANSGM to generate a dilated reverse NSGM denoted as d(RNSGM). d(RVNSGM) indicates regions of AY that are near pixels that have a low gradient. For example, if AY includes a broad, low-luminance region or a broad, high-luminance region, pixels within that region have a low gradient. The corresponding locations in d(RVNSGM), as well as the nearby locations, then have a value that is close to 1. Dilator 420 transmits d(RVNSGM) to multiply 422.
The overshoot detector 400 also receives h(AY) into the normalize function 424. The normalize function 424 performs the same operation as the normalize function 414. The normalize function 424 generates a normalized luminance value for each pixel location of the digital image. Each normalized luminance value is included in a normalized luminance array, denoted as Nsharp, which is an H×W array of decimal values ranging from 0 to 1, based on h(AY). The normalize function 424 transmits Nsharp to the dark mask unit 426.
The dark mask unit 426 identifies dark regions of h(AY). The dark mask unit 426 generates dark mask κ, which is an H×W array of decimal values that range from 0 to 1. Each value of dark mask κ represents a measure of the darkness of the corresponding pixel within h(AY). In one embodiment, the dark mask κ is generated according to Equation 3:




  
    
      
        (
        
          k
          =
          
            max
            ⁡
            
              (
              
                0
                ,
                
                  {
                  
                    
                      
                        
                          1
                          -
                          
                            h
                            ⁡
                            
                              (
                              
                                A
                                Y
                              
                              )
                            
                          
                        
                      
                      
                        
                          
                            h
                            ⁡
                            
                              (
                              
                                A
                                Y
                              
                              )
                            
                          
                          <
                          t
                        
                      
                    
                    
                      
                        
                          1
                          -
                          t
                          -
                          
                            a
                            ⁡
                            
                              (
                              
                                
                                  h
                                  ⁡
                                  
                                    (
                                    
                                      A
                                      Y
                                    
                                    )
                                  
                                
                                -
                                t
                              
                              )
                            
                          
                        
                      
                      
                        otherwise
                      
                    
                  
                  )
                
              
              )
            
          
        
      
    
    
      
        (
        3
        )
      
    
  




Those skilled in the art will understand that the two equations shown above, representing different components of Equation 3, are calculated using each value of h(AY). Those skilled in the art will also recognize that other techniques for detecting dark regions of h(AY) are possible. In one embodiment, t=0.05 and a=15.
The dark mask unit 426 transmits dark mask κ to multiply 422. Multiply 422 receives dark mask κ, d(RVNSGM), and d(ANSGM) and multiplies each corresponding element of dark mask κ, d(RVNSGM), and d(ANSGM) together to generate αovershoot. αovershoot indicates the regions of h(AY) where a high gradient region borders a low gradient region and, additionally, h(AY) is dark. Referring to FIG. 9, αcovershoot indicates regions of h(AY) where extended high gradient region 930 overlaps extended low gradient region 950. Within this region, overshoot is likely to occur.
In one embodiment, multiply 422 multiplies each value of d(RVNSGM) to a corresponding value of and d(ANSGM) to generate a boundary mask that indicates regions of the digital image that fall on the edge of a boundary between a high-gradient region and a low-gradient region. Referring again to FIG. 9, such a region is represented by the overlap between extended high gradient region 930 and extended low gradient region 940, and also by the overlap between extended high gradient region 930 and extended low gradient region 950.
In one embodiment, each component of the overshoot detector 400 described above may be implemented as a hardware component, such as an application-specific integrated circuit (ASIC), a digital signal processor (DSP), or a field-programmable gate array (FPGA), among others. In alternative embodiments, each component of the overshoot detector 400 may be implemented as a software program stored on a computer-readable medium that, when executed by a processor, performs the specific functionality associated with that component, as described above. In further alternative embodiments, the overshoot detector 400 may include any combination of hardware and/or software components.
FIG. 5 is a flowchart of method steps for generating an overshoot alpha submask, according to one embodiment of the invention. Persons skilled in the art will understand that, even though the method 500 is described in conjunction with the system of FIGS. 1-2 and 4, any system configured to perform the method steps, in any order, is within the scope of the present invention.
As shown, the method 500 begins at step 502, where the overshoot detector 400 receives an array of luminance data, AY, derived from the RGB data input to the adaptive sharpening filter 200. At step 504, the overshoot detector 400 executes a vertical sobel filter using AY as input to generate a vertical sobel gradient (VSG) for each pixel location of the digital image. The VSG is indicative of edges in the digital image. In one embodiment, an alternative type of filter is used to detect edges in the digital image. Each VSG value is included in an H×W array of decimal values denoted as AVSG. In one embodiment, the vertical sobel filter 402 included with the overshoot detector 400 receives AY and performs step 504. At step 506, the overshoot detector 400 squares each value of AVSG to generate AVSG2. AVSG2 has the same dimensions as AVSG. In one embodiment, the square function 406 included within the overshoot detector 400 performs step 506.
At step 508, the overshoot detector 400 executes a horizontal sobel filter using AY as input to generate horizontal sobel gradient (HSG) for each pixel location of the digital image. The HSG is indicative of edges in the digital image. In one embodiment, an alternative type of filter is used to detect edges in the digital image. Each HSG value is included in an H×W array of decimal values denoted as AHSG. In one embodiment, the horizontal sobel filter 404 included with the overshoot detector 400 receives AY and performs step 508. At step 510, the overshoot detector 400 squares each value of AHSG to generate AHSG2. AHSG2 has the same dimensions as AHSG2. In one embodiment, the square function 408 included within the overshoot detector 400 performs step 510.
At step 512, the overshoot detector 400 adds each value of AVSG2 to the corresponding value of AHSG2 to generate a combined gradient denoted as GSUM. GSUM has the same dimensions as both AVSG2 and AHSG2. In one embodiment, the sum 410 included within the overshoot detector 400 performs step 512. At step 514, the overshoot detector 400 calculates the square root of each value of GSUM to generate a sobel gradient magnitude (SGM) for each pixel location of the digital image. Each SGM value is included within an H×W array of decimal values denoted as ASGM. In one embodiment, the square root function 412 included within the overshoot detector 400 performs step 514.
At step 516, the overshoot detector 400 normalizes ASGM by dividing each value of ASGM by the maximum value of ASGM to generate ANSGM. ANSGM is an H×W array of decimal values that range from 0 to 1. ANSGM indicates high-gradient regions of AY, such as sharp boundaries between dark regions and bright regions within AY. In one embodiment, the normalize function 414 included within the overshoot detector 400 performs step 516. In another embodiment, the normalized sobel gradient magnitude 450 performs steps 504, 506, 508, 510, 512, 514, and 516.
At step 518, the overshoot detector 400 performs a dilation operation with ANSGM to generate a dilated NSGM for each pixel value in the digital image. Each dilated NSGM is included in an H×W array denoted as d(ANSGM). d(ANSGM) is generated as follows: For each pixel location within ANSGM, the overshoot detector 400 determines the maximum pixel value of ANSGM within a certain neighborhood of the pixel location. The overshoot detector 400 then generates d(ANSGM) comprising the maximum values corresponding to each pixel location. In one embodiment, the neighborhood is a P×Q rectangle, where P and Q are integers. In another embodiment, the dilator 418 included within the overshoot detector 400 performs step 518.
At step 520, the overshoot detector 400 subtracts each value of ANSGM from 1 to generate a reverse NSGM for each pixel location in the digital image. Each reverse NSGM is included in an H×W array denoted as RVNSGM. RVNSGM indicates low-gradient regions of AY, such as broad, low-luminance areas or broad, high-luminance areas. In one embodiment, subtractor 416 included within the overshoot detector 400 performs step 520. At step 522, the overshoot detector 400 performs a dilation operation with RVNSGM to generate a dilated reverse NSGM for each pixel location in the digital image. Each dilated reverse NSGM is included in an H×W array denoted as RVNSGM. The dilation operation performed by the overshoot detector 400 is substantially similar to the dilation operation described in step 518. In one embodiment, a different neighborhood is used than the neighborhood described in step 518. In another embodiment, the dilator 420 included within the overshoot detector 400 performs step 522.
At step 524, the overshoot detector 400 receives sharpened luminance array h(AY). At step 526, the overshoot detector 400 normalizes h(AY) by dividing each value of h(AY) by the maximum value of h(AY). The result of this operation is Nsharp, which is an H×W array of normalized sharpened luminance values. In one embodiment, the normalize function 424 included within the overshoot detector 400 performs step 526. At step 528, the overshoot detector 400 generates an dark mask K based on Nsharp. Dark mask κ is an H×W array of decimal values that indicates low-luminance regions of h(AY). In one embodiment, dark mask κ indicates high-luminance regions of h(AY). In another embodiment, the dark mask unit 426 included within the overshoot detector 400 performs step 528.
At step 530, the overshoot detector 400 multiplies each corresponding value of d(RVNSGM), d(ANSGM), and dark mask κ to generate αovershoot. αovershoot indicates regions of h(AY) where overshoot artifacts are are strongly detected. In one embodiment, the overshoot detector 400 multiplies the value of d(RVNSGM) at each pixel location by the value of d(ANSGM) at the pixel location to generate a boundary mask. The overshoot detector 400 then multiplies the value of the boundary mask at each pixel location by the value of the dark mask at the pixel location.
In another embodiment, αovershoot is used to combine AY with h(AY) to generate composite luminance data such that each pixel value of the composite luminance data is determined based on a weighted sum of pixel values drawn from AY and h(AY), where the weight values for AY and h(AY) are 1−αovershoot and αovershoot, respectively. RGB data may then be generated using the composite luminance. Embodiments of the invention thus provide a technique to reduce the effects of overshoot artifacts in a sharpened frame of RGB data.
FIG. 6 is a conceptual diagram illustrating a color fringe detector 600, according to one embodiment of the invention. As shown, the color fringe artifact detector 600 includes NSGM 602, a dilator 604, a subtractor 606, NSGM 608, a dilator 610, a subtractor 612, a multiply 614, a multiply 616, a normalize function 618, a normalize function 620, a gain factor 624, a limit 626, a multiply 628, a sum 630, an absolute value 632, a divider 634, a normalize function 636, a gain factor 638, a limit 640, and a dilator 642. The color fringe artifact detector 600 is configured to receive AR, AG, and AB color values associated with the RGB data received by adaptive sharpening filter 200 and to generate αfringe. AR, AG, and AB are each an H×W array of decimal values that range from 0 to 1 that represent the red, green and blue color values, respectively, at each pixel location.
The color fringe artifact detector 600 receives AR into normalized sobel gradient magnitude (NSGM) 602, subtractor 606, multiply 614, and sum 630. NSGM 602 performs a substantially similar operation as normalized sobel gradient magnitude 450, described above in FIG. 4, to generate a red NSGM for each pixel location of the digital image. Each red NSGM is included in an H×W array denoted as DNSGM. DNSGM has the same dimensions as AR and indicates regions of AR where the luminance of AR changes rapidly. NSGM 602 transmits DNSGM to dilator 604 and multiply 616.
Dilator 604 receives DNSGM from NSGM 602 and performs a dilation operation to generate a dilated red NSGM for each pixel location of the digital image. Each dilated red NSGM is included in an H×W array denoted as d(DNSGM). The dilation operation performed by dilator 604 is substantially similar to the dilation operations performed by dilators 418 and 420, described in FIG. 4. In one embodiment, the neighborhood implemented by dilator 604 to perform the dilation operation is different than the neighborhood implemented by the dilators 418 and 420. d(DNSGM) has the same dimensions as DNSGM and indicates regions of DNSGM near pixel locations that have a high gradient. Dilator 604 transmits d(DNSGM) to multiply 614.
Subtractor 606 receives AR and performs a substantially similar operation as the operation performed by subtractor 416 described in FIG. 4 in order to generate a reverse red color value for each pixel location in the digital image. Each reverse red color value is included in an H×W array denoted as RVred. Similar to how AR represents the red color value at each pixel location, RVred represents a lack of red color value at each pixel location. For example, a pixel value of 1 at a particular location in RVred would indicate that the red color value at the corresponding pixel location in AR equals zero. Subtractor 606 transmits RVred to multiply 616.
The color fringe artifact detector 600 receives AB into NSGM 608, subtractor 612, multiply 616, and sum 630. NSGM 608 performs a substantially similar operation as normalized sobel gradient magnitude 450, described above in FIG. 4, to generate a blue NSGM for each pixel location in the digital image. Each blue NSGM is included in an H×W array denoted as BNSGM. BNSGM has the same dimensions as AB and indicates regions of AB where the luminance of AB changes rapidly. NSGM 608 transmits BNSGM to dilator 610 and to multiply 614.
Dilator 610 receives BNSGM from NSGM 608 and performs a dilation operation to generate a dilated blue NSGM for each pixel location in the digital image. Each dilated blue NSGM is included in an H×W array denoted as d(BNSGM). The dilation operation performed by dilator 608 is substantially similar to the dilation operations performed by dilators 418 and 420, described in FIG. 4. In one embodiment, the neighborhood implemented by dilator 608 to perform the dilation operation is different than the neighborhood implemented by the dilators 418 and 420. d(BNSGM) has the same dimensions as d(DNSGM) and indicates regions of BNSGM that surround pixel locations that have a high gradient. Dilator 604 transmits d(BNSGM) to multiply 616.
Subtractor 612 receives AB and performs a substantially similar operation using AB as input as the operation performed by subtractor 416 described in FIG. 4 in order to generate a reverse blue color value for each pixel location in the digital image. Each reverse blue color value is included in an H×W array denoted as RVblue. Whereas AB represents the blue color value at each pixel location, RVblue represents a lack of blue color value at each pixel location. For example, a pixel value of 1 at a particular location in RVblue would indicate that the blue color value at the corresponding pixel location in AB equals zero. Subtractor 612 transmits RVblue to multiply 614.
Multiply 614 receives d(DNSGM), AR, RVblue, and BNSGM and multiplies each corresponding element together to generate a red edge value for each pixel location in the digital image. Each red edge value is included in an H×W array denoted as Redge. Redge indicates regions of ARGB that have a high red color value, a low blue color value, a high blue gradient, and are nearby regions that have a high red gradient (e.g., regions near an edge having a high red color value). Within such regions, the red color value decreases less rapidly than the blue color value across a border between a high-luminance region and low-luminance region, indicating that the red color values may be focused less sharply than the blue color values. Multiply 614 transmits Redge to normalize function 618. Normalize function 618 performs a normalization function that is substantially similar to that performed by normalize functions 414 and 424 described in FIG. 4 to generate a red fringe value for each pixel location in the digital image. Each red fringe value is included in a red fringe mask, which is an H×W array denoted as Rfringe. Rfringe indicates regions of ARGB where a red color fringe is likely to occur. Normalize function 618 transmits Rfringe to sum 622.
Multiply 616 receives d(BNSGM), AB, RVblue, and DNSGM and multiplies each corresponding element together to generate a blue edge value for each pixel location in the digital image. Each blue edge value is included in an H×W array denoted as Bedge that indicates regions of AB that have a high blue color value, a low red color value, a high blue gradient, and are nearby regions that have a high blue gradient (e.g., regions near an edge having a high blue color value). Within such regions, the blue color value decreases less rapidly than the red color value across a border between a high-luminance region and low-luminance region, indicating that the blue color values may be focused less sharply than the red color values. Multiply 616 transmits Bedge to normalize function 620. Normalize function 620 performs a normalization function that is substantially similar to that performed by normalize functions 414 and 424 described in FIG. 4 to generate a blue fringe value for each pixel location in the digital image. Each blue fringe value is included in a blue fringe mask, which is an H×W array denoted as Bfringe. Bfringe indicates regions of ARGB where a blue color fringe is likely to occur. Normalize function 620 transmits Bfringe to sum 622.
Sum 622 adds each element of Rfringe to the corresponding element of Bfringe to generate a red-blue mask sum denoted as RBsum. RBsum is an H×W array of values. RBsum has the same dimensions as ARGB and indicates regions of ARGB where the red and blue color components may be out of focus. Sum 622 transmits RBsum to the gain factor 624. The gain factor 624 multiplies each element of F by a gain factor X. Gain factor X may be an integer or a decimal value, and may take on a wide range of different values depending on various configurations of the color fringe detector 600. The gain factor 624 transmits X*RBsum to limit 626. Limit 626 sets all values of X*RBsum that are greater than 1 to be equal to 1 to generate RBfringe. Limit 626 transmits RBfringe to multiply 628.
The color fringe detector 600 also receives both AR and AB into sum 630 and subtracts each pixel value of AB from the corresponding pixel value of AR to generate a red-blue difference for each pixel location in the digital image. Each red-blue difference is included in an H×W array denoted as RBdiff. The absolute value 632 computes the absolute value of each pixel value of RBdiff and transmits the result of this operation to the divider 634. The divider receives abs(RBdiff B) as well as AG and divides abs(RBdiff) by AG to generate RBD. In one embodiment, the divider 634 adds a small increment ε to AG so that the value of abs(RBdiff) divided by AG is defined when AG is equal to 0. RBD indicates regions of ARGB where the green color value is low and the red color value is much higher than the blue color value or, alternatively, regions of ARGB where the green color value is low and the blue color value is much higher than the red color value. The divider 634 transmits RBD to the normalize function 636.
The normalize function 636 receives RBD as input and performs a substantially similar normalization operation as the normalization operation performed by normalize functions 414 and 424, as described in FIG. 4, to generate a normalized red-blue difference for each pixel location of the digital image. Each red-blue difference is included in an H×W array denoted as RBN. The normalize function 636 transmits RBN to the gain factor 638. Gain factor 638 multiplies each pixel value of RBN by a gain factor Z. Z may be an integer or a decimal value. Gain factor 638 transmits Z*RBN to limit 640. Limit 640 limits the values of Z*RBN that are greater than 1 to be equal to 1 in a similar fashion as limit 626. The result of this operation is a white rejection value for each pixel location in the digital image. Each white rejection value is included in a white rejection mask, which is an H×W array denoted as Wreject. Wreject indicates regions of ARGB where a red color fringe or a blue color fringe may occur. Normalize function 618 transmits Wreject to sum 622. Limit 640 transmits Wreject to multiply 628.
Multiply 628 receives RBfringe and Wreject and multiplies each pixel value of RBfringe by the corresponding pixel value of Wreject to generate a combined mask value for each pixel location of the digital image. Each of the combined mask values is included in an H×W array denoted as Afringe. Multiply 628 transmits Afringe to the dilator 642. The dilator 642 performs a substantially similar dilation operation as the dilation operation performed by the dilators 418 and 420 using Afringe as input to generate a dilated Afringe value for each pixel location of the digital image. In one embodiment, the dilator 642 uses a different neighborhood to perform the dilation operation than the neighborhood used by dilators 418 and 420. The dilator 642 dilates Afringe to generate αfringe.
αfringe is an H×W array of decimal values that range from 0 to 1. Each pixel value of αfringe corresponds to a pixel location in the digital image and is indicative of how strongly a color fringe artifact is detected at the pixel location within a sharpened version of ARGB. The color fringe artifact could be a red color fringe or a blue color fringe.
In one embodiment, each component of the color fringe detector 600 described above may be implemented as a hardware component, such as an application-specific integrated circuit (ASIC), a digital signal processor (DSP), or a field-programmable gate array (FPGA), among others. In alternative embodiments, each component of the overshoot detector 400 may be implemented as a software program stored on a computer-readable medium that, when executed by a processor, performs the specific functionality associated with that component, as described above. In further alternative embodiments, the color fringe detector 600 may include any combination of hardware and/or software components.
FIG. 7 is a flowchart of method steps for generating a color fringe alpha submask, according to one embodiment of the invention. Persons skilled in the art will understand that, even though the method 600 is described in conjunction with the system of FIGS. 1-2, 4 and 6, any system configured to perform the method steps, in any order, is within the scope of the present invention.
As shown, the method 700 begins at step 702, where the color fringe detector 600 receives a frame of ARGB data comprised of AR, AG, and AB color components. Each of AR, AG, and AB is an H×W array of decimal values. In one embodiment, the values of AR, AG, and AB are normalized to one prior to being input to the color fringe detector 600, so that each pixel value ranges from 0 to 1. Each value of AR, AG, and AB represents the red, green and blue color values, respectively, at the pixel location.
In one embodiment, the color fringe detector 600 receives AR into the NSGM 602, the multiply 614, the subtractor 606, and the sum 630, each of which is included within the color fringe detector 600, while performing step 702. In a further embodiment, the color fringe detector 600 receives AB into the NSGM 608, the multiply 616, the subtractor 612, and the sum 630, each of which is included within the color fringe detector 600, while performing step 702. In yet a further embodiment, the color fringe detector 600 receives AG into the divider 634 included within the color fringe detector 600 while performing step 702.
At step 704, the color fringe detector 600 calculates a red NSGM value for each pixel location in the digital image using AR as input to generate DNSGM. In one embodiment, the NSGM 602 performs step 702. At step 704, the color fringe detector 600 dilates DNSGM to generate d(DNSGM). In one embodiment, the dilator 604 performs step 706.
At step 708, the color fringe detector 600 calculates a blue NSGM for each pixel location in the digital image using AB as input to generate BNSGM. In one embodiment, the NSGM 608 performs step 708. At step 710, the color fringe detector 600 dilates BNSGM to generate d(BNSGM). In one embodiment, the dilator 608 performs step 710.
At step 712, the color fringe detector 600 subtracts each value of AR from 1 to generate RVred. In one embodiment, subtractor 612 performs step 712. At step 714, the color fringe detector 600 subtracts each value of AB from 1 to generate RVblue. In one embodiment, subtractor 606 performs step 714.
At step 716, the color fringe detector 600 multiplies each corresponding element of d(DNSGM), AR, RVblue, and BNSGM to generate an array of red edge values denoted as Redge. Each red edge value corresponds to a pixel location of the digital image and indicates regions of ARGB that have a high red color value, a low blue color value, a high blue gradient, and are nearby regions that have a high red gradient (e.g., regions near an edge having a high red color value). Within such regions, the red color value decreases less rapidly than the blue color value across a border between a light region and dark region, indicating that the red color values may be focused less sharply than the blue color values. In one embodiment, multiply 614 performs step 716. At step 718, the color fringe detector 600 performs a normalize function using Redge as input to generate red fringe mask Rfringe.
At step 720, the color fringe detector 600 multiplies each corresponding element of d(BNSGM), AB, RVblue, and DNSGM to generate an array of blue edge values denoted as Bedge. Each blue edge value corresponds to a pixel location of the digital image and indicates regions of ARGB that have a high blue color value, a low red color value, a high red gradient, and are nearby regions that have a high blue gradient (e.g., regions near an edge having a high blue color value). Within such regions, the blue color value decreases less rapidly than the red color value across a border between a light region and dark region, indicating that the blue color values may be focused less sharply than the red color values. In one embodiment, multiply 616 performs step 720. At step 722, the color fringe detector 600 performs a normalize function using Bedge as input to generate red fringe mask Bfringe.
At step 724, the color fringe detector 600 adds each pixel value of Rfringe to the corresponding pixel value of Bfringe to generate a red-blue mask sum denoted as RBsum. RBsum indicates regions where the red and blue color components may be out of focus. In one embodiment, the multiply 622 included within the color fringe detector 600 performs step 724. At step 726, the color fringe detector 600 multiplies each element of RBsum by a gain factor Z. In one embodiment, the gain factor 624 included within the color fringe detector 600 performs step 726. At step 728, the color fringe detector 600 limits the values of Z*RBsum to one by setting each values of Z*RBsum that exceeds 1 to be equal to 1. The result of this operation is a red-blue fringe mask denoted as RBfringe. In one embodiment, the limit 626 included within the color fringe detector 600 performs step 728.
At step 730, the color fringe detector 600 subtracts AB from AR to generate an array of red-blue differences that each correspond to a pixel location in the digital image. The array of red-blue difference is denoted as RBdiff. In one embodiment, the sum 630 performs step 730. At step 732, the color fringe detector 600 calculates the absolute value of RBdiff for each pixel location. In one embodiment, the absolute value 632 included within the color fringe detector 600 performs step 732. At step 734, the color fringe detector 600 divides each value of abs(RBdiff) by a corresponding value of AG to generate RBD. RBD is an array of values that each correspond to a pixel location of the digital image. RBD indicates regions of ARGB where the green color value is low and the red color value is much higher than the blue color value or, alternatively, regions of ARGB where the green color value is low and the blue color value is much higher than the red color value. In one embodiment, the divider 634 performs step 734.
At step 736, the color fringe detector 600 normalizes RBD by determining the maximum value of RBD and dividing each element of RBD by the maximum value. The result of this operation is a normalized red-blue difference value for each pixel location of the digital image that indicates regions of the digital image that are not excessively green or white. Each normalized red-blue difference value is included within a normalized red-blue difference mask denoted as RBN, which is an H×W array of decimal values that range from 0 to 1. In one embodiment, the normalize function 636 included within the color fringe detector 600 performs step 736.
At step 738, the color fringe detector 600 multiplies each value of RBN by a gain factor X. In one embodiment, the gain factor 638 included within the color fringe detector 600 performs step 638. At step 740, the color fringe detector 600 limits the values of X*RBN to one by setting each value of X*RBN that exceeds 1 to be equal to 1. The result of this operation is Wreject. In one embodiment, the limit 640 included within the color fringe detector 600 performs step 640.
At step 742, the color fringe detector 600 multiplies each element of RBfringe with the corresponding element of Wreject to generate a value at each pixel location of the digital image. Each of these values is included in a mask denoted as Afringe. In one embodiment, multiply 628 included within the color fringe detector 600 performs step 742. At step 744, the color fringe detector 600 performs a dilation operation with Afringe to generate αfringe. The method then terminates.
αfringe is an H×W array of decimal values that range from 0 to 1. Each value of αfringe corresponds to a different pixel location in the digital image and represents how strongly a color fringe artifact is detected at the pixel location within a sharpened version of ARGB. The color fringe artifact could be a red color fringe or a blue color fringe.
In one embodiment of the invention, αfringe is used to combine AY with h(AY) to generate composite luminance data such that each pixel value of the composite luminance data is determined based on a weighted sum of pixel values drawn from AY and h(AY), where the weight values for AY and h(AY) are 1−αfringe and αfringe, respectively. RGB data may then be generated using the composite luminance. Embodiments of the invention thus provide a technique to reduce the effects of color fringe artifacts in a sharpened frame of RGB data.
FIG. 8 is a conceptual diagram illustrating a dark artifact detector 800, according to one embodiment of the invention. As shown, the dark artifact detector 800 includes dark mask unit 802 and limit 804. The dark artifact detector 800 generates αdark, which is an H×W array of decimal values that range from 0 to 1 that each correspond to a different pixel location of the digital image. Each value of αdark indicates how strongly a dark artifact is detected in sharpened luminance data h(AY).
Dark artifact detector 1000 receives AY into the dark mask unit 802. The dark mask unit 802 generates dark mask δ that indicates pixel values that are above a certain threshold. In one embodiment, dark mask δ is generated using equation 4, where a and T are parameters:




  
    
      
        δ
        =
        
          
            a
            *
          
          ⁡
          
            (
            
              1
              -
              
                
                  A
                  Y
                
                T
              
            
            )
          
        
      
    
    
      
        (
        4
        )
      
    
  




Those skilled in the art will understand that Equation 4 is calculated using each value of AY. Dark mask unit 802 transmits dark mask δ to limit 804. Limit 804 limits the values of dark mask δ to fall between 0 and 1 by setting the values of dark mask δ that are less than 0 to be equal to 0, and by setting the values of dark mask δ that are greater than 1 to be equal to 1. The output of limit 804 is αdark. αdark indicates regions of AY that are too dark to sharpen without introducing a noticeable increase in noise.
In one embodiment of the invention, αdark is used to combine AY with h(AY) to generate composite luminance data such that each pixel value of the composite luminance data is determined based on a weighted sum of pixel values drawn from AY and h(AY), where the weight values for AY and h(AY) are 1−αdark and αdark, respectively. RGB data may then be generated using the composite luminance. Embodiments of the invention thus provide a technique to decrease noise in dark areas of a sharpened frame of RGB data.
Although embodiments of the invention described herein are directed towards various operations that may be performed using an H×W array of values as input, where each of those values corresponds to a different pixel in an H×W digital image, persons skilled in the art will recognize that the disclosed operations are equally applicable to a subset of the H×W array of values that corresponds to a subset of the H×W digital image. Any such subset constitutes a “portion” of the digital image, and, in some embodiments, a “portion” may constitute every pixel of the H×W digital image.
In sum, the adaptive sharpening filter generates an alpha mask that indicates how strongly a sharpening artifact is detected at each pixel location. The alpha mask value at each pixel location falls within a range from 0 to 1, where 0 indicates that a sharpening artifact is strongly detected, and 1 indicates that a sharpening artifact is not strongly detected. The alpha mask is used to combine luminance values associated with each pixel location of the original image with sharpened luminance values associated with each pixel location of the sharpened image to produce an adaptively sharpened image.
Each pixel of the adaptively sharpened image is determined by multiplying the luminance value of the sharpened image at the pixel location by the alpha mask value at the pixel location to produce a first product, multiplying the luminance value of the original image at the pixel location by 1 minus the alpha mask value at the pixel location to generate a second product, and then summing the two products. The alpha mask thus determines, for each pixel location of the adaptively sharpened image, a percentage of the adaptively sharpened luminance value contributed by the sharpened image, and a percentage of the adaptively sharpened luminance value contributed by the original image. These percentages are summed to provide the adaptively sharpened luminance value at the pixel location.
The adaptive sharpening filter generates the alpha mask by combining an overshoot alpha submask, a color fringe alpha submask, and a dark region alpha submask. The overshoot alpha submask detects regions of the digital image where dark undershoot is strongly detected. Specifically, the overshoot alpha submask indicates regions of the digital image on a low-luminance side of a border between a high-luminance region and a low-luminance region. The color fringe alpha submask indicates regions of the digital image where the average green color value is low, the average red color value is higher than the average blue color value, and the red color value decreases rapidly across sequential pixel locations. Additionally, the color fringe alpha submask indicates regions of the digital image where the average green color value is low, the average blue color value is higher than the average red color value, and the blue color value decreases rapidly across sequential pixel locations. Color fringe artifacts are strongly detected in these regions. Finally, the dark region alpha submask indicates regions of the digital image where the sharpened luminance is low. These regions may include noise that could be amplified by the sharpening filter.
The adaptive sharpening filter combines the three alpha submasks to generate the alpha mask. Then, the digital image is combined with the sharpened image to generate an adaptively sharpened image.
One advantage of the disclosed methods is that they provide an adaptive sharpening filter that improves overall image sharpness, but reduces the incidence of sharpening artifacts relative to prior art image sharpening techniques.
While the forgoing is directed to embodiments of the present invention, other and further embodiments of the invention may be devised without departing from the basic scope thereof. For example, aspects of the present invention may be implemented in hardware or software or in a combination of hardware and software. One embodiment of the invention may be implemented as a program product for use with a computer system. The program(s) of the program product define functions of the embodiments (including the methods described herein) and can be contained on a variety of computer-readable storage media. Illustrative computer-readable storage media include, but are not limited to: (i) non-writable storage media (e.g., read-only memory devices within a computer such as CD-ROM disks readable by a CD-ROM drive, flash memory, ROM chips or any type of solid-state non-volatile semiconductor memory) on which information is permanently stored; and (ii) writable storage media (e.g., floppy disks within a diskette drive or hard-disk drive or any type of solid-state random-access semiconductor memory) on which alterable information is stored. Such computer-readable storage media, when carrying computer-readable instructions that direct the functions of the present invention, are embodiments of the present invention. Therefore, the scope of the present invention is determined by the claims that follow.

## Claims

1. A non-transitory computer-readable medium including instructions that, when executed by a processing unit, cause the processing unit to sharpen a digital image that is H pixels in height and W pixels in width, by performing the steps of:
generating a plurality of luminance values for a portion of the digital image, wherein each luminance value corresponds to a different pixel location in the portion of the digital image;
generating a plurality of sharpened luminance values for the portion of the digital image by applying a sharpening filter to one or more of the luminance values, wherein each sharpened luminance value corresponds to a different pixel location in the portion of the digital image; and
generating a sharpened digital image for output to a display or for storage in a memory by selecting, for each pixel location in the portion of the digital image, either the luminance value in the plurality of luminance values corresponding to the pixel location or the sharpened luminance value in the plurality of sharpened luminance values corresponding to the pixel location based on a value in an alpha mask corresponding to the pixel location, wherein the alpha mask is derived from at least one of, an overshoot alpha submask, a fringe alpha submask, and a dark alpha submask.

2. The non-transitory computer-readable medium of claim 1, wherein the overshoot alpha submask includes, for each pixel location in the portion of the digital image, an overshoot value that is indicative of how strongly an overshoot artifact is detected at the pixel location.

3. The non-transitory computer-readable medium of claim 1, wherein the fringe alpha submask includes, for each pixel location in the portion of the digital image, a fringe value that is indicative of how strongly a fringe artifact is detected at the pixel location.

4. The non-transitory computer-readable medium of claim 1, wherein the dark alpha submask includes, for each pixel location in the portion of the digital image, a dark value that is indicative of how strongly a dark artifact is detected at the pixel location.

5. The non-transitory computer-readable medium of claim 1, wherein the portion of the digital image comprises every pixel in the digital image.

6. The non-transitory computer-readable medium of claim 1, wherein the alpha mask is derived from at least two of the overshoot alpha submask, the fringe alpha submask, and the dark alpha submask.

7. The non-transitory computer-readable medium of claim 1, wherein the alpha mask is derived from the overshoot alpha submask, the fringe alpha submask, and the dark alpha submask.

8. The non-transitory computer-readable medium of claim 7, further comprising the step of generating the alpha mask by:
generating a submask sum for each pixel location in the portion of the digital image by adding the overshoot value from the overshoot alpha submask at the pixel location to the fringe value from the fringe alpha submask at the pixel location;
generating a limited submask sum for each pixel location in the portion of the digital image by limiting each value in the submask sum to a value less than 1;
generating a reverse submask by subtracting the value of the limited submask sum at each pixel location in the portion of the digital image from 1;
generating a reverse dark alpha submask by subtracting the dark value of the dark alpha submask at each pixel location in the portion of the digital image from 1; and
for each pixel location in the portion of the digital image, multiplying the value of the reverse submask at each pixel location with the corresponding value of the reverse dark alpha submask at the pixel location.

9. The non-transitory computer-readable medium of claim 8, wherein the step of generating the sharpened digital image comprises:
generating a reverse alpha mask by subtracting the value of the alpha mask at each pixel location in the portion of the digital image from 1;
generating a first luminance portion for each pixel location in the portion of the digital image by multiplying the sharpened luminance value at the pixel location by the value of the reverse alpha mask at the pixel location;
generating a second luminance portion for each pixel location in the portion of the digital image by multiplying the luminance value at the pixel location by the value of the alpha mask at the pixel location; and
generating an adaptively sharpened luminance value for each pixel location in the portion of the digital image by adding the value of the first luminance portion at the pixel location to the value of the second luminance portion at the pixel location.

10. The non-transitory computer-readable medium of claim 1, wherein each pixel included in the portion of the digital image comprises red, green and blue (RGB) color values, and further comprising the steps of:
converting the portion of the digital image to a luminance and chrominance (YUV) format prior to the step of generating the plurality of sharpened luminance values; and
converting the generated sharpened digital image from a YUV format to an RGB format.

11. The non-transitory computer-readable medium of claim 1, wherein a first overshoot artifact is formed when a first region of pixels in the portion of the digital image resides in between second and third regions of pixels in the portion of the digital image, and the average of the sharpened luminance values associated with the first region of pixels in the portion of the digital image is less than the average of the sharpened luminance values associated with the second and third regions of pixels in the portion of the digital image, and the average of the sharpened luminance values associated with the second region of pixels in the portion of the digital image is different than the average of the sharpened luminance values associated with the third region of pixels in the portion of the digital image.

12. The non-transitory computer-readable medium of claim 1, wherein each pixel included in the portion of the digital image comprises red, green and blue (RGB) color values, and a first fringe artifact is formed when a first region of pixels in the portion of the digital image resides between second and third regions of pixels in the portion of the digital image, and the average of the red color values associated with the first region of pixels in the portion of the digital image is greater than the average of the blue color values associated with the first region of pixels in the portion of the digital image and is greater than the average of the green color values associated with the first region of pixels in the portion of the digital image and is greater than the average of the red color values associated with either the second region of pixels or the third region of pixels in the portion of the digital image, and the average of the red color values associated with the second region of pixels in the portion of the digital image is different than the average of the red color values associated with the third region of pixels in the portion of the digital image.

13. The non-transitory computer-readable medium of claim 1, further comprising the step of generating the dark alpha submask by:
deriving a first set of dark values from the plurality of luminance values, wherein each dark value in the first set of dark values corresponds to a pixel location in the portion of the digital image having a luminance value that is less than or equal to a threshold;
deriving a second set of dark values from the plurality of luminance values, wherein each dark value in the second set of dark values corresponds to a pixel location in the portion of the digital image having a luminance value that is greater than the threshold; and
combining the first set of dark values and the second set of dark values to create the dark alpha submask.