Patent Description:
The present invention relates to systems and methods for single-frame based super resolution interpolation for digital cameras.

Digital cameras use various systems to enable photographers to capture images of objects at long distances. Optical zoom systems use one or more zoom lenses that may be adjusted to narrow the field of visible radiation incident on the photo-detectors of the digital camera. The narrower field of visible radiation incident on the photo-detectors magnifies the captured image, albeit with a narrower field of view, without the introduction of significant image aberrations. In contrast, digital zoom systems process the image, or a subset of the image, to increase its resolution to create an effect similar to optical zoom (i.e., a magnified narrower field of view). Digital zoom systems, however, generally produce significant undesirable image aberrations relative to optical zoom systems. For example, digital zoom systems may introduce aliasing (i.e., jagged diagonal edges), blurring, and/or haloing into the image. The aberrations introduced during digital zoom process occur primarily at or around the edges of objects in the image.

Document <CIT> discloses an image enhancement system and method for generating high-resolution bitmaps from low-resolution bitmaps. An original, low-resolution bitmap is magnified to form a magnified image and high contrast edges are extracted from the magnified image. The magnified image is transformed into overlapping image patches which are analyzed by performing connected components analysis to determine foreground and background regions using said extracted high contrast edges located by the edge detector. Once the foreground and background regions have been determined, the contrast of the center pixel in each of the plurality of image patches is enhanced based on whether the region is a foreground or a background region. Finally, the image enhancement system and method of the invention combines the luminance of the enhanced output pixels with the color values produced by the magnification algorithm used to generate the magnified image resulting in a high-resolution bitmap from the contrast-enhanced pixels.

Document <CIT> discloses an apparatus which obtains a high-resolution image, including: a scaler to scale an input image and generate a first output image, a high frequency component extraction unit to extract a first high frequency component of the first output image from the input image, a high frequency component generation unit to generate a second high frequency component of the first output image based on the first high frequency component, and an image synthesis unit to synthesize the first output image and the second high frequency component and generate a second output image.

Document <CIT> discloses an image processing device for super-resolution processing wherein an input low-resolution image is first processed for edge enhancement and then super-resolution processing is applied to generate an output high-resolution image.

The invention is defined in the appended independent claims, to which attention is directed.

Various aspects of at least one example are discussed below with reference to the accompanying figures, which are not intended to be drawn to scale. The figures are included to provide an illustration and a further understanding of the various aspects and examples, and are incorporated in and constitute a part of this specification, but are not intended as a definition of the limits of the embodiments disclosed herein. The drawings, together with the remainder of the specification, serve to explain principles and operations of the described and claimed aspects and examples. In the figures:.

It is to be appreciated that examples of the methods and apparatuses discussed herein are not limited in application to the details of construction and the arrangement of components set forth in the following description or illustrated in the accompanying drawings. The methods and apparatuses are capable of implementation in other examples and of being practiced or of being carried out in various ways. Examples of specific implementations are provided herein for illustrative purposes only and are not intended to be limiting. In particular, acts, elements and features discussed in connection with any one or more examples are not intended to be excluded from a similar role in any other examples.

Any references to examples or elements or acts of the systems and methods herein referred to in the singular may also embrace examples including a plurality of these elements, and any references in plural to any example or element or act herein may also embrace examples including only a single element. References in the singular or plural form are not intended to limit the presently disclosed systems or methods, their components, acts, or elements. The use herein of "including," "comprising," "having," "containing," "involving," and variations thereof is meant to encompass the items listed thereafter and equivalents thereof as well as additional items. References to "or" may be construed as inclusive so that any terms described using "or" may indicate any of a single, more than one, and all of the described terms.

Embodiments of the present invention relate to single-frame based super resolution interpolation for digital cameras. It should be appreciated that the term "digital camera" used herein includes, but is not limited to, dedicated cameras as well as camera functionality performed by any electronic device (e.g., mobile phones, personal digital assistants, etc.). In addition, the methods and systems described herein may be applied to a plurality of images arranged in a time sequence (e.g., a video stream). The use herein of the term "module" is interchangeable with the term "block" and is meant to include implementations of processes and/or functions in software, hardware, or a combination of software and hardware. In one embodiment, the single-frame based super resolution interpolation system and method digitally zoom an image, or any portion of the image, without producing significant image aberrations in the output digitally zoomed image. Digitally zooming an image causes significant aberrations particularly to the edges of objects in the image. For example, the edges in digitally zoomed images may appear jagged from aliasing. Accordingly, in some embodiments, the systems and methods provided improve the quality of the edges in the super resolution process to produce a high quality digitally zoomed image.

<FIG> illustrates an example system for super resolution image processing. As shown, the super resolution image processing system <NUM> of <FIG> includes a super resolution engine <NUM> that receives an image <NUM> and performs image processing on the received image to produce a digitally zoomed image <NUM>. The super resolution engine <NUM> includes multiple functional blocks or modules including an image region selection block <NUM> that selects at least a portion of the image <NUM> and provides an image region <NUM>, a resolution enhancement block <NUM> that enhances the resolution of the portion of the image and provides a resolution enhanced image region <NUM>, and an edge enhancement block <NUM> that enhances edges in the enhanced image region and provides the digitally zoomed image <NUM>.

The input to the super resolution engine <NUM> is an image <NUM>. In one embodiment, the image comprises a plurality of pixels represented by one or more two dimensional matrices. Each element in the matrices represents information regarding a specific pixel location in the image. For example, the location (<NUM>, <NUM>) in a first, second, and third two dimensional matrix may represent data associated with a pixel at the (<NUM>,<NUM>) location in the image. Each of the one or more matrices may represent a set of intensities associated with the image in a given dimension. In one embodiment, each pixel in the image is represented by a three-dimensional vector. In this embodiment, the image is represented by a first two dimensional matrix for the first dimension, a second two dimensional matrix for the second dimension, and a third two dimensional matrix for the third dimension. It is appreciated that each pixel may be represented by any number of dimensions or any number of matrices.

In some embodiments, the image is represented by three two dimensional matrices consistent with a YUV color space. The YUV color space is composed of three distinct components Y, U, and V where each two dimensional matrix represents a specific component of the YUV space. The Y component is the luminance component that relates to the brightness of the image. The U and V components are chrominance components that relate to the specific color of the image. Each pixel in an image is represented by a vector in the YUV color space (i.e., some combination of the Y, U, and V components). In some embodiments, the image is represented by three two dimensional matrices consistent with a RGB color space. The RGB color space is also composed of three distinct components R, G, and B where each two dimensional matrix represents a specific component of the RGB space. All three of the distinct components (i.e., R, G, and B components) are all chrominance components that relate to the specific color of the image. It is appreciated that the image may be represented in any color space and is not limited to the YUV or RGB color spaces.

The image <NUM> is input into the super resolution engine <NUM>. The super resolution engine <NUM> executes a variety of processes to output a digitally zoomed image <NUM>. In one embodiment, the digitally zoomed image is a magnified image region <NUM>. The image region <NUM> may include the entire image or any subset of the image <NUM>. It is appreciated that the processes performed by the super resolution engine <NUM> do not need to be performed on all of the dimensions of each pixel of the image region <NUM>. In some embodiments, the pixels in the image region <NUM> are represented by a vector in the YUV color space. In these embodiments, the processes of the super resolution engine <NUM> may be performed on any combination dimensions in the YUV color space. For example, in one embodiment the processes of the super resolution engine may be performed only on the Y component (i.e., the luminance component) of the image region <NUM> pixels. In this embodiment, interpolation processes (e.g., bicubic interpolation) may be performed on the U and V components (i.e., the chrominance components) to obtain a digitally zoomed image <NUM>.

The various processes performed by the super resolution engine <NUM> to generate the digitally zoomed image <NUM> include the processes performed by the image region selection block <NUM>, the resolution enhancement block <NUM>, and the edge enhancement block <NUM>. In various embodiments, the image region selection block <NUM> crops the image as appropriate to select an image region <NUM> of the input image <NUM> to be digitally zoomed. It is appreciated that the image region selection block <NUM> is an optional block. For example, the super resolution engine <NUM> may digitally zoom the entire image <NUM>. In other embodiments, the super resolution engine <NUM> self-selects the entire image <NUM> or any subset of the image <NUM> to digitally zoom.

The resolution enhancement block <NUM> increases the resolution of the image region <NUM> to produce an enhanced resolution image region <NUM>. The resolution of the image region <NUM> is increased through interpolation of the image region <NUM>. Interpolation is a method of creating new data points between a set of known discrete data points. In this embodiment, the known data points are the known pixel values (e.g., the Y, U, and/or V channel intensity values for a given pixel). The new data points, in this embodiment, are the pixels created through interpolation. It is appreciated that the interpolation is preceded by one or more acts to smooth the edges of the image region <NUM> to improve the quality of the edges in the resolution enhanced image region <NUM>. The edge smoothing sharpens the image by shortening the transitions between color and/or brightness changes (i.e., object edges). A visual representation of an edge smoothing process is illustrated in the edge smoothing process diagrams <NUM> with reference to <FIG>. The super resolution engine <NUM> may proceed to the edge enhancement block <NUM>.

The edge enhancement block <NUM> enhances the edges of objects in the image. The interpolation performed in the resolution enhancement block <NUM> may create a smoothed image that lacks sharp details. Accordingly, the edge enhancement block <NUM> is capable of extracting and enhancing the edges of the resolution enhanced image region <NUM>. The edge enhancement block <NUM> extracts the edges of the resolution enhanced image region <NUM>. The extracted edges are then added to the resolution enhanced image region <NUM> to increase the edge detail.

It is appreciated that any combination of the super resolution image processing functional blocks (e.g., the image region selection block <NUM>, the resolution enhancement block <NUM>, and the edge enhancement block <NUM>) may perform operations in parallel. In one embodiment, the input image <NUM> may be divided into a plurality of subsections. Each subsection of the image <NUM> may be processed individually in parallel. The processed subsections may be combined to form the digitally zoomed image <NUM>. Implementing the super resolution image processing in parallel increases the speed in which the digitally zoomed image <NUM> can be provided.

It is appreciated that the super resolution engine <NUM> may take a variety of forms dependent upon the specific application. In some embodiments, the super resolution engine <NUM> comprises a series of instructions performed by an image processing system. In other embodiments, one or more of the blocks or modules <NUM>, <NUM>, and <NUM> may be implemented in hardware, such as an application-specific integrated circuit (ASIC), system on a chip (SoC), state machine, or a dedicated processor.

<FIG> illustrates an example image processing system <NUM> in which super resolution processing can be performed. As shown, the image processing system <NUM> of <FIG> includes a digital camera <NUM> that is configured to provide a digital image of a scene <NUM>. The digital camera <NUM> includes an image processor <NUM>, an analog-to-digital converter <NUM>, a memory <NUM>, an interface <NUM>, storage <NUM>, an image sensor <NUM>, and a lens <NUM>.

As illustrated in <FIG>, the digital camera <NUM> includes the image processor <NUM> to implement at least some of the aspects, functions and processes disclosed herein. The image processor <NUM> performs a series of instructions that result in manipulated data. The image processor <NUM> may be any type of processor, multiprocessor or controller. Some exemplary processors include processors with ARM11 or ARM9 architectures. The image processor <NUM> is connected to other system components, including one or more memory devices <NUM>.

The memory <NUM> stores programs and data during operation of the digital camera <NUM>. Thus, the memory <NUM> may be a relatively high performance, volatile, random access memory such as a dynamic random access memory ("DRAM") or static memory ("SRAM"). However, the memory <NUM> may include any device for storing data, such as a flash memory or other non-volatile storage devices. Various examples may organize the memory <NUM> into particularized and, in some cases, unique structures to perform the functions disclosed herein. These data structures may be sized and organized to store values for particular data and types of data.

The data storage element <NUM> includes a writeable nonvolatile, or nontransitory, data storage medium in which instructions are stored that define a program or other object that is executed by the image processor <NUM>. The data storage element <NUM> also may include information that is recorded, on or in, the medium, and that is processed by the image processor <NUM> during execution of the program. More specifically, the information may be stored in one or more data structures specifically configured to conserve storage space or increase data exchange performance. The instructions may be persistently stored as encoded signals, and the instructions may cause the image processor <NUM> to perform any of the functions described herein. The medium may, for example, be optical disk, magnetic disk or flash memory, among others. In operation, the image processor <NUM> or some other controller causes data to be read from the nonvolatile recording medium into another memory, such as the memory <NUM>, that allows for faster access to the information by the image processor <NUM> than does the storage medium included in the data storage element <NUM>. The memory may be located in the data storage element <NUM> or in the memory <NUM>, however, the image processor <NUM> manipulates the data within the memory, and then copies the data to the storage medium associated with the data storage element <NUM> after processing is completed. A variety of components may manage data movement between the storage medium and other memory elements and examples are not limited to particular data management components. Further, examples are not limited to a particular memory system or data storage system.

The digital camera <NUM> also includes one or more interface devices <NUM> such as input devices, output devices and combination input/output devices. Interface devices may receive input or provide output. More particularly, output devices may render information for external presentation. Input devices may accept information from external sources. Examples of interface devices include microphones, touch screens, display screens, speakers, buttons, etc. Interface devices allow the digital camera <NUM> to exchange information and to communicate with external entities, such as users and other systems.

Although the digital camera <NUM> is shown by way of example as one type of digital camera upon which various aspects and functions may be practiced, aspects and functions are not limited to being implemented on the digital camera <NUM> as shown in <FIG>. Various aspects and functions may be practiced on digital cameras having a different architectures or components than that shown in <FIG>. For instance, the digital camera <NUM> may include specially programmed, special-purpose hardware, such as an application-specific integrated circuit ("ASIC") or a system on a chip ("SoC") tailored to perform a particular operation disclosed herein. It is appreciated that the digital camera <NUM> may be incorporated into another electronic device (e.g., mobile phone, personal digital assistant etc.) and is not limited to dedicated digital cameras.

The lens <NUM> includes one or more lenses that focus the visible radiation on the image sensor <NUM>. It is appreciated that the lens <NUM> is not limited to a single physical lens as illustrated in <FIG>. In some embodiments, the lens <NUM> includes a plurality of zoom lenses that enable optical zoom. Optical zoom may be accomplished by narrowing the field of view of the visible radiation incident on the image sensor <NUM>.

The image sensor <NUM> may include a two dimensional area of sensors (e.g., photodetectors) that are sensitive to light. In some embodiments, the photo-detectors of the image sensor <NUM>, in some embodiments, can detect the intensity of the visible radiation in one of two or more individual color and/or brightness components. For example, the output of the photo-detectors may include values consistent with a YUV or RGB color space. It is appreciated that other color spaces may be employed by the image sensor <NUM> to represent the captured image.

In various embodiments, the image sensor <NUM> outputs an analog signal proportional to the intensity and/or color of visible radiation striking the photo-detectors of the image sensor <NUM>. The analog signal output by the image sensor <NUM> may be converted to digital data by the analog-to-digital converter <NUM> for processing by the image processor <NUM>. In some embodiments, the functionality of the analog-to-digital converter <NUM> is integrated with the image sensor <NUM>. The image processor <NUM> may perform variety of processes to the captured image. These processes may include, but are not limited to, one or more super resolution processes to digitally zoom the captured image.

<FIG> illustrates a more detailed functional block diagram of the super resolution image processing system <NUM> including an image region selection block <NUM>, a resolution enhancement block <NUM>, and an edge enhancement block <NUM> in accordance with <FIG>. As shown in <FIG>, the super resolution image processing system <NUM> of <FIG> includes a crop image block <NUM>, an edge smoothing block <NUM>, an interpolate block <NUM>, an edge extraction block <NUM>, a fine edge-preserving filter block <NUM>, a course edge-preserving filter block <NUM>, an edge processing function block <NUM>, a white noise block <NUM>, a difference block <NUM>, and a sum block <NUM>. It is appreciated that any block drawn in dashed lines is an optional block. In addition, large boxes drawn in dotted lines show the relation between the super resolution image processing system <NUM> including the image region selection block <NUM>, the resolution enhancement block <NUM>, and the edge enhancement block <NUM> described with reference to <FIG> and the functional blocks shown in <FIG>.

The crop image block <NUM> illustrates one possible implementation of the image region selection block <NUM>. The crop image block <NUM> crops the image as appropriate to feed into the later functional blocks of the super resolution image processing system <NUM>. In one embodiment, the crop image block <NUM> may crop the image responsive to input received through an interface from an external entity (e.g., interface <NUM> of the digital camera <NUM>). In another embodiment, the crop image block <NUM> self-selects an image region <NUM> and crops the image <NUM> as appropriate. It is appreciated that the crop image block <NUM> is an optional block.

The optional image region selection block <NUM> is followed by the resolution enhancement block <NUM>. The edge smoothing block <NUM> and the interpolate block <NUM> illustrate one possible implementation of the resolution enhancement block <NUM>. In one embodiment, the edge smoothing block <NUM> smoothes the edges of objects in the image region <NUM>. The edge of an object in an image may be characterized by a set of contiguous pixels where an abrupt change of intensity values occurs.

A visual representation of an edge smoothing process is illustrated by the edge smoothing process diagrams <NUM> with reference to <FIG>. As shown in <FIG>, the edge smoothing process diagrams <NUM> of <FIG> includes an object <NUM> within the field of view of a pixel matrix <NUM> that proceeds through a pre-captured image state <NUM>, a captured image state <NUM>, and an edge smoothed image state <NUM>.

The pre-captured image state <NUM> illustrates an object <NUM> in the field of view of a pixel matrix <NUM>. The edges of the object <NUM> cross through a plurality of pixels in the pixel matrix. The captured image state <NUM> of the pixel matrix <NUM> illustrates the effect of capturing an image of an object with edges crossing through multiple pixels. The third row of pixels from the top illustrates the edge blurring effect that occurs when an image is captured by a digital camera (e.g., digital camera <NUM>). The pixels in the third row from the top get darker moving from left to right proportional to the area of the pixel that was covered by the object. The edge smoothing process that occurs between the image captured state <NUM> and the edge smoothed image state <NUM> sharpens the image by eliminating some or all of the transition pixels. In this illustration, the elimination of transition pixels is shown in the third row of the pixel matrix <NUM> in the edge smoothed image state <NUM>. The degree of the transition (i.e., the amount of transition pixels remaining in the image region) may be controlled with a strength parameter associated with the edge smoothing filter applied.

Referring back to <FIG>, the edge smoothing block <NUM> may be any pertinent edge smoothing or edge-preserving filter. In one embodiment, the edge smoothing filter is the edge preserving noise reduction filter described in co-pending <CIT> (hereinafter the "'<NUM> application"). It should be appreciated that embodiments of the present invention are not limited to any particular type of edge smoothing or edge preserving filter, as various types of filters may be used.

The edge smoothing block is followed by the interpolate block <NUM>. The interpolate block may compute a new pixel matrix at a higher resolution based on the input pixel matrix. In one embodiment, the interpolation block analyzes a closest set of pixels from the input image pixel matrix to determine corresponding new pixel values in the new pixel matrix. The specific process that is used to perform the interpolation may be consistent with a variety of methods including, but not limited to, bicubic interpolation, bilinear interpolation, nearest neighbor interpolation, spline interpolation, or sinc interpolation.

The resolution enhancement block <NUM> is followed by the edge enhancement block <NUM>. The edge enhancement block <NUM>, in some embodiments, is illustrated by the edge extraction block <NUM>, the fine edge-preserving filter <NUM>, the course edge-preserving filter <NUM>, the edge processing function block <NUM>, the difference block <NUM>, and the sum block <NUM>. In one embodiment, the fine edge-preserving filter <NUM> and the course edge-preserving filter <NUM> are edge sensitive filters of different strengths. In this embodiment, the fine edge preserving filter <NUM> is a weaker edge filter than the course edge-preserving filter <NUM>. It is appreciated that the fine edge-preserving filter <NUM> may or may not be the same type of edge preserving filter as the course edge-preserving filter <NUM>. The output of the course edge preserving filter <NUM> is subtracted from the output of the fine edge preserving filter <NUM> in difference block <NUM>. The output of the difference block <NUM> contains only the edges of the resolution enhanced image region <NUM>. The combination of the fine edge-preserving filter <NUM>, the course edge-preserving filter <NUM> and the difference block <NUM> may thus form the edge extraction block <NUM>. It is appreciated that the specific method to obtain the edges of the resolution enhanced image region <NUM> may be altered. For example, a plurality of edge preserving filters of various strengths may be combined in a linear fashion to generate a similar resultant output (i.e., the edges of the image). In addition, a single filter that targets the edges of the images may also be used. In other embodiments, the fine edge-preserving filter <NUM> and the course edge-preserving filter <NUM> can be the edge preserving noise reduction filters described in the '<NUM> application. It is appreciated that any combination of the edge-preserving filters may be selectable, for example, based on input or preferences received from a user of the system.

The image edges from the difference block <NUM> may then be processed by the edge processing function block <NUM>. The edge processing function block may be any function, linear or otherwise, that enhances the resolution enhanced image <NUM> edges. The edge processing function block may be implemented, for example, with a lookup table (LUT). In one embodiment, the edge processing function block <NUM> is a multiplier that amplifies or attenuates the edges. In this embodiment, the edge processing function block <NUM> may receive input from a user through a user interface. For example, the user interface may have configurable settings regarding the level of detail in the digitally zoomed image <NUM>. The edge processing function block <NUM> may increase the multiplier responsive to a higher detail level setting or decrease the multiplier responsive to a lower detail level setting. It is appreciated that the edge processing function block <NUM> is an optional function. The output of the difference block <NUM> may be directly connected to the summation block <NUM>.

The summation block <NUM> combines the resolution enhanced image region <NUM> with the edges of the resolution enhanced image region <NUM> to form a digitally zoomed image <NUM>. The addition of the edges of the resolution enhanced image region <NUM> to the resolution enhanced image region <NUM> increases the detail of the edges that may have undergone some level of distortion in the resolution enhancement block <NUM>.

The summation block <NUM> may also combine white noise from the white noise block <NUM> with the digitally zoomed image <NUM>. The addition of white noise to the digitally zoomed image <NUM> gives the appearance of high resolution noise in the digitally zoomed image <NUM> rather than amplified low resolution noise. The intensity of white noise added may vary. For example, in one embodiment the amount of white noise added to the image is computed responsive to the camera settings when the image was taken (e.g., ISO, aperture, or shutter speed). It is appreciated that the addition of white noise, and consequently the white noise block <NUM>, is optional. In addition, the white noise from the white noise module may be combined with the resolution enhanced image region <NUM>.

Having described at least one implementation of the super resolution image processing system, it is appreciated that the specific implementations described may be altered to generate the same result (i.e., a digitally zoomed image without significant image aberrations). Other example implementations of the super resolution image processing systems are illustrated by functional block diagrams of super resolution image processing systems <NUM> and <NUM> with reference to <FIG> and <FIG> respectively.

<FIG> illustrates an alternative functional block diagram of the super resolution image processing system <NUM> including an image region selection block <NUM>, a resolution enhancement block <NUM>, and an edge enhancement block <NUM> in accordance with <FIG>. As shown in <FIG>, the super resolution image processing system <NUM> of <FIG> includes a crop image block <NUM>, edge smoothing blocks <NUM> and <NUM>, an interpolation block <NUM>, an edge extraction block <NUM>, a fine edge-preserving filter block <NUM>, a course edge-preserving filter block <NUM>, an edge processing function block <NUM>, a white noise block <NUM>, a difference block <NUM>, and a sum block <NUM>. It is appreciated that any block drawn in dashed lines is an optional block. In addition, large boxes drawn in dotted lines show the relation between the processes described with reference to <FIG> and the functional blocks shown in <FIG>.

The image region selection block <NUM> and the edge enhancement block <NUM> implemented by the crop image block <NUM>, the edge extraction block <NUM>, the fine edge preserving filter <NUM>, the course edge-preserving filter <NUM>, the edge processing function block <NUM>, the difference block <NUM>, and the sum block <NUM> are similar to that described with reference <FIG> above. The resolution enhancement block <NUM> that is implemented in the embodiment depicted in <FIG> by the edge smoothing blocks <NUM> and <NUM> in addition to the interpolate block <NUM> is a variation of the resolution enhancement block <NUM> described with reference to <FIG>. The additional edge smoothing block <NUM> improves the quality of the edges in the resolution enhanced image <NUM> and consequently the final digitally zoomed image <NUM>. The additional cost of the additional edge smoothing block <NUM> is the additional computation required to perform the additional edge smoothing. It is appreciated that the filter used in the edge smoothing block <NUM> may, or may not, be constructed to be identical to the edge filter in the edge smoothing block <NUM>. In addition, the summation block <NUM> may also combine white noise from the white noise block <NUM> with the digitally zoomed image <NUM> as described with reference to <FIG>.

<FIG> illustrates a further alternative functional block diagram of the super resolution image processing system <NUM> including an image region selection block <NUM>, are solution enhancement block <NUM>, and an edge enhancement block <NUM> in accordance with <FIG>. As shown in <FIG>, the super resolution image processing system <NUM> of <FIG> includes a crop image block <NUM>, edge smoothing blocks <NUM> and <NUM>, interpolation blocks <NUM> and <NUM>, edge extraction blocks <NUM> and <NUM>, fine edge-preserving filter blocks <NUM> and <NUM>, course edge-preserving filter blocks <NUM> and <NUM>, edge processing function blocks <NUM> and <NUM>, white noise block <NUM>, difference blocks <NUM> and <NUM>, and sum blocks <NUM> and <NUM>. It is appreciated that any block drawn in dashed lines is an optional block. In addition, large boxes drawn in dotted lines show the relation between the processes described with reference to <FIG> and the functional blocks shown in <FIG>.

The image region selection block <NUM>, resolution enhancement block <NUM>, and the edge enhancement block <NUM> are similar to that described with reference <FIG> above. The implementation of the super resolution image processing system in <FIG> shows multiple iterations of the resolution enhancement block <NUM> and the edge enhancement block <NUM>. Multiple iterations of the resolution enhancement block <NUM> and the edge enhancement block <NUM> processes increase the detail in the digitally zoomed image <NUM> at the cost of additional computation. It is appreciated that the resolution enhancement block <NUM> and the edge enhancement block <NUM> may be repeated any number of times to achieve the desired level of detail in the digitally zoomed image <NUM>. In one embodiment, the image <NUM> may need to be digitally zoomed substantially. In this embodiment, the first iteration of the resolution enhancement block <NUM> and the edge enhancement block <NUM> may only digitally zoom the image <NUM> a fraction of the required total digital zoom. The second iteration of the resolution enhancement block <NUM> and the edge enhancement block <NUM> may increase the resolution to the desired final digitally zoomed image <NUM>. It should be appreciated that an additional edge smoothing block (e.g., edge smoothing block <NUM> in <FIG>) can be added to the first iteration, the second iteration, or both. In addition, the summation block <NUM> may also combine white noise from the white noise block <NUM> with the digitally zoomed image <NUM> after both iterations of the super resolution processes.

It is appreciated that the super resolution image processing system <NUM> including the image region selection block <NUM>, the resolution enhancement block <NUM>, and the edge enhancement block <NUM> may be performed on any combination of the chrominance and/or luminance channels of the image <NUM>. For example, in one embodiment the image <NUM> is in the YUV color space and the super resolution image processing system <NUM> including the image region selection block <NUM>, the resolution enhancement block <NUM>, and the edge enhancement block <NUM> are only applied to the luminance channel (i.e., Y channel) of the image <NUM>. In this embodiment, the U and V channels undergo only interpolation (e.g., bicubic interpolation). This embodiment is of lower computational complexity when compared to performing the super resolution processes on all three channels. The loss of quality is minimal, in this embodiment, because the human eye is substantially more sensitive to brightness than color. The sensitivity of the photoreceptors that perceive brightness (i.e., rods of the human eye) are more numerous and more sensitive than photoreceptors that perceive color (i.e., cones of the human eye).

Claim 1:
A digital camera system for super resolution image processing, comprising:
a resolution enhancement module (<NUM>) configured to receive at least a portion of an image, to smooth (<NUM>) at least one edge of the portion by shortening the transitions between color and/or brightness changes by reducing the amount of transition pixels, to increase the resolution of the smoothed portion via interpolation, and to output a resolution enhanced image;
an edge enhancement module (<NUM>) configured to receive the resolution enhanced image and comprising an edge extraction module (<NUM>) configured to receive the resolution enhanced image, to extract at least one edge of the resolution enhanced image, and to output the extracted at least one edge of the resolution enhanced image; and
wherein the edge enhancement module (<NUM>) is configured to combine the extracted at least one edge with the resolution enhanced image.