Patent ID: 12236555

DETAILED DESCRIPTION

The input data of a conventional image signal processing (ISP) pipeline are frames of color filtered image data comprising pixels. Typical components of an ISP pipeline include color interpolation (i.e., interpolates red (R), green (G) and blue (B) values for each pixel), color correction (i.e., correct color values), gamma correction (i.e., change from linear to non-linear space) and color space conversion (e.g., transform from a RGB color format to a YUV format). More complex ISP pipelines also include noise reduction, lens shading compensation, using a 3 dimensional look up table (3DLUT) for mapping color spaces, image sharpening, image cropping and image scaling. The YUV pixel data can be then compressed, transmitted, decoded and displayed, depending on the applications.

The image sensor typically includes a Bayer color filter array. Correspondingly, the output of image sensors of the Bayer color filter array is color filtered data. For an ISP pipeline, the raw image color filtered data is then converted to a full-color image by a particular interpolation technique tailored to the pattern of the Bayer color filter array.

A Bayer image (color filtered image resulting from a Bayer color filter array) represents pixels using three color components (e.g., R, G and B). Each pixel of a frame is represented by one of the three color components, while the other two color components are missing from each corresponding pixel. The ISP pipeline includes a demosaicing process, which interpolates the missing colors and converts the Bayer image to a full resolution color image in 3 color planes for display.

Conventional demosaicing algorithms and ISP processing devices that are used to interpolate the pixels of a Bayer image (image produced by a Bayer color filter array) often result in output color images or video frames which suffer from loss of fine structures (e.g., blurry edges), zippering (e.g., alternating patterns) and high frequency contents (pixel values that rapidly change in space), which introduce artifacts in the image.

For example, conventional demosaicing (i.e., interpolation) techniques include using pre-designed filters which are applied to a sliding window of a Bayer image. Depending on the output of the color filter array, each pixel is classified into several classifications, such as a flat area, a textured area and an edge area. Then, different demosaicing algorithms (i.e., interpolation algorithms) are employed to reconstruct the missing color components according to the result of classifications. However, these techniques are inefficient because the filters must be well designed, the classifiers need fine-tuned parameters in various scenarios to distinguish different areas properly, and interpolation methods at different areas require careful design.

In addition, because one color component (typically the G component) is sampled for the Bayer image more than the other two color components, (e.g., the R component and the B component) and provides more precise gradients than the R and B components, conventional demosaicing algorithms first interpolate the missing G components and then use the full-resolution G component to interpolate the R and B components, requiring additional clock cycles and more power consumption to complete.

Features of the present disclosure include processing devices and methods of demosaicing a Bayer image using a one direction linear model and directional weighting fusion.

An image processing device is provided which comprises memory and a processor. The processor is configured to, for a pixel of a Bayer image which filters an acquired image using three color components, determine directional color difference weightings in a horizontal direction and a vertical direction, determine a color difference between a first color component and a second color component and a color difference between a second color component and a third color component based on the directional color difference weightings, interpolate a color value of the pixel from the one color component and the color differences and provide a color image for display.

A method of image demosaicing is provided which comprises, for a pixel of a Bayer image, determining directional color difference weightings in a horizontal direction and a vertical direction, determining a color difference between a first color component and a second color component and a color difference between the second color component and the third color component based on the directional color difference weightings, interpolating a color value of the pixel from the one color component and the color differences and providing a color image for display.

An image processing device is provided which comprises an image capturing device configured to capture an image and an image sensor comprising a Bayer color filter array configured to color filter the image according to a first color component, a second color component and a third color component. The image processing device also comprises a processor configured to, for a pixel of the Bayer image represented as one of the first color component, the second color component and the third color component, determine directional color difference weightings in a horizontal direction and a vertical direction, determine a color difference between the first color component and the second color component and a color difference between the second color component and the third color component based on the directional color difference weightings and interpolate a color value of the pixel from the one color component and the color differences

FIG.1is a block diagram of an example device100in which one or more features of the disclosure can be implemented. The device100can include, for example, a computer, a gaming device, a handheld device, a set-top box, a television, a mobile phone, or a tablet computer. The device100includes a processor102, a memory104, a storage106, one or more input devices108, and one or more output devices110. The device100can also optionally include an input driver112and an output driver114. It is understood that the device100can include additional components not shown inFIG.1.

In various alternatives, the processor102includes a central processing unit (CPU), a graphics processing unit (GPU), a CPU and GPU located on the same die, or one or more processor cores, wherein each processor core can be a CPU or a GPU. In various alternatives, the memory104is located on the same die as the processor102, or is located separately from the processor102. The memory104includes a volatile or non-volatile memory, for example, random access memory (RAM), dynamic RAM, or a cache.

The storage106includes a fixed or removable storage, for example, a hard disk drive, a solid state drive, an optical disk, or a flash drive. The input devices108include, without limitation, a keyboard, a keypad, a touch screen, a touch pad, a detector, a microphone, an accelerometer, a gyroscope, a biometric scanner, or a network connection (e.g., a wireless local area network card for transmission and/or reception of wireless IEEE 802 signals). The output devices110include, without limitation, a display, a speaker, a printer, a haptic feedback device, one or more lights, an antenna, or a network connection (e.g., a wireless local area network card for transmission and/or reception of wireless IEEE 802 signals).

The input driver112communicates with the processor102and the input devices108, and permits the processor102to receive input from the input devices108. The output driver114communicates with the processor102and the output devices110, and permits the processor102to send output to the output devices110. It is noted that the input driver112and the output driver114are optional components, and that the device100will operate in the same manner if the input driver112and the output driver114are not present. The output driver116includes an accelerated processing device (“APD”)116which is coupled to a display device118. The APD accepts compute commands and graphics rendering commands from processor102, processes those compute and graphics rendering commands, and provides pixel output to display device118for display. As described in further detail below, the APD116includes one or more parallel processing units to perform computations in accordance with a single-instruction-multiple-data (“SIMD”) paradigm. Thus, although various functionality is described herein as being performed by or in conjunction with the APD116, in various alternatives, the functionality described as being performed by the APD116is additionally or alternatively performed by other computing devices having similar capabilities that are not driven by a host processor (e.g., processor102) and provides graphical output to a display device118. For example, it is contemplated that any processing system that performs processing tasks in accordance with a SIMD paradigm may perform the functionality described herein. Alternatively, it is contemplated that computing systems that do not perform processing tasks in accordance with a SIMD paradigm performs the functionality described herein.

FIG.2is a block diagram of the device100, illustrating additional details related to execution of processing tasks on the APD116. The processor102maintains, in system memory104, one or more control logic modules for execution by the processor102. The control logic modules include an operating system120, a kernel mode driver122, and applications126. These control logic modules control various features of the operation of the processor102and the APD116. For example, the operating system120directly communicates with hardware and provides an interface to the hardware for other software executing on the processor102. The kernel mode driver122controls operation of the APD116by, for example, providing an application programming interface (“API”) to software (e.g., applications126) executing on the processor102to access various functionality of the APD116. The kernel mode driver122also includes a just-in-time compiler that compiles programs for execution by processing components (such as the SIMD units138discussed in further detail below) of the APD116.

The APD116executes commands and programs for selected functions, such as ISP operations and graphics operations that may be suited for parallel processing. The APD116can be used for executing ISP pipeline operations such as pixel operations (e.g., channel resampling and interpolation), geometric computations, and rendering an image to display device118based on commands received from the processor102. The APD116also executes compute processing operations that are not directly related to ISP and graphics operations, such as operations related to physics simulations, computational fluid dynamics, or other tasks, based on commands received from the processor102.

The APD116includes compute units132that include one or more SIMD units138that perform operations at the request of the processor102in a parallel manner according to a SIMD paradigm. The SIMD paradigm is one in which multiple processing elements share a single program control flow unit and program counter and thus execute the same program but are able to execute that program with different data. In one example, each SIMD unit138includes sixteen lanes, where each lane executes the same instruction at the same time as the other lanes in the SIMD unit138but can execute that instruction with different data. Lanes can be switched off with predication if not all lanes need to execute a given instruction. Predication can also be used to execute programs with divergent control flow. More specifically, for programs with conditional branches or other instructions where control flow is based on calculations performed by an individual lane, predication of lanes corresponding to control flow paths not currently being executed, and serial execution of different control flow paths allows for arbitrary control flow.

The basic unit of execution in compute units132is a work-item. Each work-item represents a single instantiation of a program that is to be executed in parallel in a particular lane. Work-items can be executed simultaneously as a “wavefront” on a single SIMD processing unit138. One or more wavefronts are included in a “work group,” which includes a collection of work-items designated to execute the same program. A work group can be executed by executing each of the wavefronts that make up the work group. In alternatives, the wavefronts are executed sequentially on a single SIMD unit138or partially or fully in parallel on different SIMD units138. Wavefronts can be thought of as the largest collection of work-items that can be executed simultaneously on a single SIMD unit138. Thus, if commands received from the processor102indicate that a particular program is to be parallelized to such a degree that the program cannot execute on a single SIMD unit138simultaneously, then that program is broken up into wavefronts which are parallelized on two or more SIMD units138or serialized on the same SIMD unit138(or both parallelized and serialized as needed). A scheduler136performs operations related to scheduling various wavefronts on different compute units132and SIMD units138.

The parallelism afforded by the compute units132is suitable for ISP and graphics related operations such as pixel value calculations, pixel value interpolation, vertex transformations, and other ISP and graphics operations. Thus in some instances, an ISP pipeline134, which accepts ISP processing commands from the processor102, provides computation tasks to the compute units132for execution in parallel.

The compute units132are also used to perform computation tasks not related to ISP and graphics or not performed as part of the “normal” operation of an ISP pipeline134(e.g., custom operations performed to supplement processing performed for operation of the ISP pipeline134). An application126or other software executing on the processor102transmits programs that define such computation tasks to the APD116for execution.

FIG.3is a block diagram illustrating example components of a processing device300in which one or more features of the disclosure can be implemented. As shown inFIG.3, processing device300includes processor302, memory304and camera306. Camera306includes lens308, image sensor310and image signal processor318. Image sensor310incudes of sensor pixels312, read-out circuits314(e.g., including an analog to digital converter (ADC) circuit) and Bayer color filter array316. In some examples, image processor318is integrated as part of processor302, instead of camera306.

Lens308includes a single lens or an assembly of lenses which collects light reflected from objects and/or light directly transmitted from illuminants. Image sensor310is, for example, a complementary metal-oxide-semiconductor (CMOS) based image sensor, which includes an array of cells, each corresponding to a pixel of an image (i.e., frame). Image sensor310is configured to expose the pixels to light passing through lens308. The light passed through the corresponding color filter array316at each cell and captured at each cell is transformed into electrons having a value (i.e., an accumulated charge). The charge (i.e., analog value) of each cell is read out, during a read-out time, and sent to read out circuits314, which converts the analog values into digital values.

Image processor318controls the exposure timing of the image sensor310(e.g., the delay time period between the exposures of each frame or between lines of a frame) such that time difference between the start of each readout time period is at least sufficient for the readout circuit to read out each line. The frame rate of the video is also controlled by controlling the exposure timing of the lines of each frame.

Processor302is configured to control both the exposure timing of the image sensor310(e.g., via image processor318) and image processor318. Processor302is also configured to perform a plurality of functions as described herein. For example, processor302is configured to interpolate missing pixel color values of a Bayer image using a one direction linear model and directional weighting fusion.

Processor302is also in communication with display device118(e.g., in communication with a display controller (not shown) which controls the operation of the display device118) for displaying images captured by camera306.

FIG.4is a block diagram illustrating an example flow of processing images via an ISP pipeline134according to features of the present disclosure. As shown at block402ofFIG.4, raw image data is received at the pipeline134. The raw image data is color filtered data resulting from the image sensor310.

The raw image data (e.g., non-processed data) is resampled, at block404, and Bayer filtered, by the Bayer color filter array316, according to a Bayer HDR format (i.e., HDR fused) at block406to produce a Bayer image.

FIG.5is an illustration of an example portion of an image sensor500comprising a Bayer color filter array502(i.e., a Bayer color filter mosaic) and a pixel sensor array504. The Bayer color filter array502(i.e., front portion of image sensor500shown inFIG.5) includes a plurality of color filters502adefining a color filter pattern. The pixel sensor array504(i.e., back portion of image sensor500shown inFIG.5) includes a plurality of or photosensors or pixel sensors (i.e., pixels)504a. Each color filter502ais disposed over a pixel504aof the pixel sensor array504to capture color information for each corresponding pixel504ain the Bayer image. The portion of the Bayer color filter array502shown inFIG.5includes an array of 5 columns and 5 rows of color filters (i.e., a 5×5 block). The size of the portion of the Bayer color filter array502shown inFIG.5is merely an example used for simplified explanation.

The color filters502afilter light, sensed by the pixel sensors504a, by wavelength range, such that the separate filtered intensities include information about the color of light. The Bayer color filter array502provides information about the intensity of the light in R, G, and B wavelength regions. That is, the light for each pixel504aof an image is filtered to record one of three different color components (i.e., a first color component, a second color component and a third color component shown inFIG.5).

The three different color components include an R component, a G component and a B component. Typically, the second color component is selected as the G component, which is the dominant part of luminance, while the first color component and the third color component correspond to the R component and B component, respectively. Accordingly, as shown in the color filter pattern atFIG.5, for each 2×2 block506, the second color component (e.g., G component) is sampled twice as much as the first component (e.g., R component) and the second color component (e.g., B component). The resulting raw color filtered data is referred to as the Bayer pattern image (i.e., Bayer image). Because each pixel504ais filtered to record only one of three colors, the color information of each pixel504ain the Bayer image cannot represent the R, G and B color values of the pixel502a. Therefore, the pixels504aof the Bayer image are color interpolated, at block408, by using demosaicing algorithms which estimate the color values for each pixel from surrounding pixels of corresponding colors.

After the Bayer image is color interpolated, the pixels504aare processed according to other components of the ISP pipeline410,412,414and416shown inFIG.4. That is, the pixels are color corrected, at block410and gamma corrected at block412. Color space conversion is then performed at block414to convert the pixels504afrom RGB color space to a YUV color space at block416. In more sophisticated ISP pipeline, additional image processing blocks can be added to the ISP pipeline134(e.g., added between any two of the blocks:402,404,406,408,410,412,414, and416in ISP pipeline134). The images are then transmitted, decoded and displayed on a display device, such as display device118shown inFIG.1.

As described above, conventional demosaicing techniques (i.e., color interpolation at block408inFIG.4) used to interpolate pixel colors of the pixels in the Bayer image include using pre-designed filters which are applied to a sliding window on a Bayer image. Depending on the output of the Bayer color filter array502, each pixel504ais classified into several classifications, such as for example a flat area, a textured area and an edge area. Then, different demosaicing algorithms (i.e., interpolation algorithms) are employed to reconstruct the missing color components according to the result of classifications. However, these techniques are inefficient because the filters must be well designed, the classifiers need fine-tuned parameters in various scenarios to distinguish different areas properly, and interpolation methods at different areas require careful design.

In addition, the second color components (e.g., G component) are sampled more than the first color component (e.g., R component) and the third color component (e.g., B component) and provide more precise gradients than the first and third color components. Accordingly, conventional demosaicing algorithms first interpolate the missing second color components (e.g., G component) and then use the full-resolution second color component to interpolate the first and third color components, requiring additional clock cycles and more power consumption to complete.

As described in more detail below with reference toFIGS.6to9B, features of the present disclosure include processing devices and methods of demosaicing a Bayer image using a one-direction linear model and directional weighting fusion.

FIG.6is a flow diagram illustrating an example method of image demosaicing according to features of the present disclosure;

As shown at block602, the method600includes obtaining a Bayer image. For example, a Bayer image resulting from the Bayer color filter array500shown inFIG.5is received by a processor (e.g., processor302inFIG.3).

Blocks604-618are performed for each pixel position, at block603, of the Bayer image. As shown at block604, the method600includes, determining a horizontal slope and offset and a vertical slope and offset using a linear model.

FIGS.7and8are used together to describe determining the horizontal and vertical slope and offset of the Bayer image shown at block604.FIG.7is a flow diagram illustrating an example method700of determining the horizontal and vertical slope and offset shown at block604inFIG.6. As described in more detail below, the color differences are determined in method700using a linear model, in which linear interpolation coefficients k (for slope) and b (for offset) are adaptively determined using a local variance and a local co-variance.

FIG.8is an illustration of an example portion of a Bayer image800used for implementing features of the present disclosure. The color component pattern shown inFIG.8is the same as the pattern shown inFIG.5.

Each pixel504ain the pattern inFIG.8corresponds to a different pixel position in the Bayer image800. The pattern includes an array of 5 columns of pixels504aand 5 rows of pixels504a(i.e., a 5×5 array). The size of the array shown inFIG.8is merely an example used for simplified explanation. Features of the present disclosure can be implemented for arrays having a size different from the size of the array shown inFIG.8.

As shown inFIG.8, the first row (i.e., top row) of pixels504ais denoted as row j−2, the second row of pixels504ais denoted as row j−1, the third row (i.e., middle row) of pixels504ais denoted as row j, the fourth row of pixels504ais denoted as row j+1 and the fifth row (i.e., bottom row) of pixels504ais denoted as row j+2, The first column (i.e., left column) of pixels504ais denoted as column i−2, the second column of pixels504ais denoted as column i−1, the third column (i.e., middle column) of pixels504ais denoted as column i, the fourth column of pixels504ais denoted as column i+1 and the fifth column (i.e., right column) of pixels504ais denoted as column i+2,

Blocks704-712are performed for each pixel504ain the Bayer image. However, for simplification purposes, the method700is described for predicting the color for a current pixel P1, at block702, corresponding to the middle pixel position (j, i) of the array shown inFIG.8. As described above, because each pixel504ais filtered to record only one of three colors, the color information of each pixel504ain the Bayer image cannot represent each of the R, G and B color values of each pixel502a. Accordingly, the color represented in a Bayer image varies between pixels (i.e., color is location variant). Therefore, the horizontally adjacent color sample sets P1hand P2h(i.e., color sample sets of the pixels to the left and right of pixel P1) are determined at block704as:
P1h={P1h(j,i−1),P1(j,i),P1h(j,i+1)}
P2h={P2(j,i−1),P2h(j,i),P2(j,i+1)

where h is horizontal, P2is the color of the horizontally adjacent pixels, and
P1h(j,i−1)=(P1(j,i−2)+P1(j,i))/2
P1h(j,i+1)=(P1(j,i+2)+P1(j,i))/2
P2h(j,i)=(P2(j,i−1)+P1(j,i+1))/2

That is, when the center pixel is the current pixel P1, the color of the pixel to the left of pixel P1(i.e., P1h(j,i−1)) is the average of the color value of the pixel to the left of pixel P1(j,i−1) and the color value of the center pixel P1(j,i)). The color of the pixel to the right of pixel P1(i.e., P1h(j,i+1)) is the average of the color value of the pixel to the right of pixel P1(j,i+1) and the color value of the center pixel P1(j,i). Then, P2h(j,i) is determined as the average of P2(j,i−1) and P1h(j,i+1).

Similarly, the vertically adjacent color sample sets P1vand P2v(i.e., colors of the pixels above and below pixel P1) are determined at block706as:
P1v={P1h(j−1,i),P1(j,i),P1v(j+1,i)}
P3v={P3(j−1,i),P3v(j,i),P3(j+1,i)}

where v is vertical, P3is the color of the vertically adjacent pixels, and
P1v(j−1,i)=(P1(j−2,i)+P1(j,i))/2
P1v(j+1,i)=(P1(j+2,i)+P1(j,i))/2
P3v(j,i)=(P3(j−1,i)+P3(j+1,i))/2

That is, when the center pixel is the current pixel P1, the color of the pixel above the center pixel P1(i.e., P1v(j−1,i)) is the average of the color value of the pixel above the pixel P1(j,i−1) and the color value of the center pixel P1(j,i)). Also, the color of the pixel below pixel P1(i.e., P1v(j+1,i)) is the average of the color value of the pixel below pixel P1(j,i+1) and the color value of the center pixel P1(j,i). Then, P3v(j,i) is determined as the average of P3(j−1,i) and P3(j+1,i).

The example described above uses three samples. However, a color set can be extended from 2 or more samples.

As shown at block708, covariances, variances and mean values are then determined from the adjacent color sample sets P1hand P2h. That is, the horizontal covariance (cov {P1h,P2h}), the horizontal variance (var {P1h}), and the horizontal mean values (mean {P1h}) and mean {P2h}) are determined and the vertical covariance (cov {P1v,P2v}), the vertical variance (var {P1v}), and the vertical mean values (mean {P1v}) and (mean {P2v}) are determined.

The horizontal (H) and vertical (V) slope k and offset b of the linear model are determined at block710. The slope k and offset b of the linear model are determined as follows:

kh=cov⁡({P1h},{P2h})var⁢{P1h},bh=P2h_-kh×P1h_kv=cov⁡({P1v},{P3v})var⁢{P1v},bv=P3v_-kv×P1v_

The horizontal linear model gives the horizontal prediction P2hof color P2at position (j,i) as: P2h(j,i)=kh×P1(j,i)+bh. The vertical linear model gives the vertical prediction P3vof color P3atFIG.8, position (j,i) as: P3v(j,i)=kv×P1(j,i)+bv.

The missing colors are then predicted at block712. The colors P1, P2and P3are variant depending on location. For example, with reference to the Bayer image800inFIG.8, at position (j,i), P1=R, P2=G and P3=G. This position is referred to as R position. At position (j,i+1), P1=G, P2=R and P3=B. This position is referred to as GR position (i.e., G at R row). At position (j+1,i), P1=G, P2=B, P3=R. This position is referred to as GB position, (i.e., G at B row). At position (j+1,i+1), P1=B, P2=G, P3=G. This position is referred to as B position.

Based on the horizontal and vertical color differences determined at block606, the color difference gradients are determined in 4 directions at block608. The color difference gradient in the east direction ΔE(i.e., right direction) is determined as follows:
ΔE=(η0×|ξh(j−1,i)−ξh(j−1,i+1)|+η1×[ξh(j−1,i+1)−ξh(j−1,i+1)]+2η0×|ξh(j,i)−ξh(j,i+1)+|+2η1×[ξh(j,i+1)−ξh(j,i+1)]+η0×|ξh(j+1,i)−ξh(j+1,i+1)|+η1×[ξh(j+1,i+1)−ξh(j+1,i+1)])/(2η0+2η1)

The color difference gradients are similarly determined in the three remaining directions processor is configured to: (i.e., the west, north and south directions. For example, the color difference gradients in the west direction ΔW(i.e., the left direction), the north direction ΔN(i.e., up direction) and the south direction ΔS(i.e., down direction) are determined as follows:
ΔW=(η0×|ξh(j−1,i)−ξh(j−1,i−1)|+η1×[ξh(j−1,i−1)−ξh(j−1,i−2)]+2η0×|ξh(j,i)−ξh(j,i−1)|+2η1×[ξh(j,i−1)−ξh(j,i−2)]+η0×|ξh(j+1,1)−ξh(j+1,i−1)|+n1×[ξh(j+1,i−1)−ξh(j+1,i−2)])/(2η0+2η1)
ΔN=(η0×|ξh(j,i−1)−ξh(j−1,i−1)|+η1×[ξh(j−1,i−1)−ξh(j−2,i−1)]+2η0×|ξh(j,i)−ξh(j−1,i)|+2η1×[ξh(j−1,i)−ξh(j−2,i)]+η0×|ξh(j,i+1)−ξh(j−1,i+1)|+η1×[ξh(j−1,i+1)−ξh(j−2,i+1)])/(2η0+2η1)
ΔS=(η0×|ξh(j,i−1)−ξh(j+1,i−1)|+η1×[ξh(j+1,i−1)−ξh(j+2,i−1)]+2η0×|ξh(j,i)−ξh(j+1,i)|+2η1×[ξh(j+1,i)−ξh(j+2,i)]+η0×|ξh(j,i+1)−ξh(j+1,i+1)|+η1×[ξh(j+1,i+1)−ξh(j+2,i+1)])/(2η0+2η1)

where η0and η1both have a default value=1, but the values of η0and η1can be changed for different tuning parameters.

Referring back toFIG.6, the horizontal and vertical color differences are determined at block606as follows:

ξh(j,i)={P1(j,i)-Pˆ2h(j,i)if⁢P1=greenPˆ2h(j,i)-P1(j,i)if⁢P2=greenξv(j,i)={P1(j,i)-Pˆ3v(j,i)if⁢P1=greenPˆ3v(j,i)-P1(j,i)if⁢P3=green

where ξh(j,i) denotes the horizontal color difference at (j,i) and ξv(j,i) denotes the vertical color difference at (j,i).

FIG.9Aillustrates the horizontal color differences ξhat the pixel positions of the portion of the Bayer image800.FIG.9Billustrates the vertical color differences ξvat the pixel positions of the portion of the Bayer image800.

The pixel differences in the horizontal and vertical directions are determined at block610as follows:
diffh=(|P(j−1,i−2)−P(j−1,i−1)|+|P(j−1,i−1)−P(j−1,i)|+|P(j−1,i)−P(j−1,i+1)|+|P(j−1,i+1)−P(j−1,i+2)|+2×[|P(j,i−2)−P(j,i−1)|+|P(j,i−1)−P(j,i)|+|P(j,i)−P(j,i+1)|+|P(j,i+1)−P(j,i+2)|]+|P(j+1,i−2)−P(j+1,i−1)|+|P(j+1,i−1)−P(j+1,i)|+|P(j+1,i)−P(j+1,i+1)|+|P(j+1,i+1)−P(j+2,i+1)|)/2
diffv=(|P(j−2,i−1)−P(j−1,i−1)|+|P(j−1,i−1)−P(j,i−1)|+|P(j,i−1)−P(j+1,i−1)|+|P(j+1,i−1)−P(j+2,i−1)|+2×[|P(j−2,i)−P(j−1,i)|+|P(j−1,i)−P(j,i)|+|P(j,i)−P(j+1,i)|+|P(j+1,i)−P(j+2,i)|]+|P(j−2,i+1)−P(j−1,i+1)|+|P(j−1,i+1)−P(j,i+1)|+|P(j,i+1)−P(j+1,i+1)|+|P(j+1,i+1)−P(j+2,i+1)|)/2

The N-direction weightings α′ are then determined at block612. In this example, 4 directions are used. However, features of the present disclosure can be implemented by determining weightings in more than 4 directions. In this example, the 4 directional weightings at (j,i) and their sum are determined as follows:

αE′=ΔE-2×(diffh)-2(diffh)-2+(diffv)-2αW′=ΔW-2×(diffh)-2(diffh)-2+(diffv)-2αN′=ΔN-2×(diffv)-2(diffh)-2+(diffv)-2αS′=ΔS2×(diffv)-2(diffh)-2+(diffv)-2Sum=αE′+αW′+αN′+αS′

Color differences for the current pixel P1are determined at block614. That is, the color difference between the second color component (C2) and the first color component (C1)) (i.e., ξEG-R) for the current pixel P1as well as the color difference between the second color component (C2) and the third color component (C3) (i.e., ξG-B(j, i)) are determined at block614, as follows:
ξG-R(j,i)=ξEG-R(j,i)×αE+ξWG-R(j,i)×αW+ξNG-R(j,i)×αN+ξSG-R(j,i)×αS
ξG-B(j,i)=ξEG-B(j,i)×αE+ξWG-B(j,i)×αW+ξNG-B(j,i)×αN+ξSG-B(j,i)×αS
where,

ξEG-R(j,i)={λ0×ξh(j,i)+λ1×ξh(j,i+1)+λ2×ξh(j,i+2)λ0+λ1+λ2if⁢P1=red⁢or⁢P2=redξh(j-1,i)+ξh(j+1,i)2if⁢P1=blue⁢or⁢P2=blueξWG-R(j,i)={λ0×ξh(j,i)+λ1×ξh(j,i+1)+λ2×ξh(j,i-2)λ0+λ1+λ2if⁢P1=R⁢or⁢P2=Rξh(j-1,i)+ξh(j+1,i)2if⁢P1=B⁢or⁢P2=BξNG-R(j,i)={λ0×ξv(j,i)+λ1×ξv(j-1,i)+λ2×ξv(j-2,i)λ0+λ1+λ2if⁢P1=R⁢or⁢P3=Rξv(j,i-1)+ξv(j,i+1)2if⁢P1=B⁢or⁢P3=BξSG-R(j,i)={λ0×ξv(j,i)+λ1×ξv(j+1,i)+λ2×ξv(j+2,i)λ0+λ1+λ2if⁢P1=R⁢or⁢P3=Rξv(j,i-1)+ξv(j,i+1)2if⁢P1=B⁢or⁢P3=B

As described above, the light for each pixel of an image is Bayer filtered to record an R (red) component, a G (green) component or a B (blue) component. Accordingly, the raw data of the Bayer filtered current pixel P1to be interpolated corresponds to an R component, a G component or a B component. The RGB color value (e.g., full color value) of the current pixel P1is then interpolated by determining the missing color components (i.e., the other two color components not representing the current pixel P1from the raw data) for the current pixel P1are determined at block616as follows:

if the current pixel P1=R:
R=P(j,i)
G=P(j,i)+ξG-R(j,i)
B=P(j,i)+ξG-R(j,i)−ξG-B(j,i)

if the current pixel P1=G:
R=P(j,i)−ξG-R(j,i)
G=P(j,i)
B=P(j,i)−ξG-B(j,i)

if the current pixel P1=B:
R=P(j,i)+ξG-B(j,i)−ξG-R(j,i);
G=P(j,i)+ξG-B(j,i)
B=P(j,i)

Due to the features described above, the local structure similarity between color components is leveraged to calculate linear model parameters. Features of the present disclosure efficiently interpolate color coefficients while maintaining high image quality in regions having different image content.

In addition, directional fusion weightings are calculated on integrated gradients, resulting in more accurate directional weightings than conventional techniques.

Further, missing RGB color components of pixels of a Bayer image are efficiently reconstructed together to reproduce an RGB color image, in contrast to conventional methods which interpolate G components separate from (i.e., prior to) the R and B components.

It should be understood that many variations are possible based on the disclosure herein. Although features and elements are described above in particular combinations, each feature or element can be used alone without the other features and elements or in various combinations with or without other features and elements.

The various functional units illustrated in the figures and/or described herein (including, but not limited to, the processor102,302,318, the input driver112, the input devices108, the output driver114, the output devices110, the accelerated processing device116, the scheduler136, the graphics processing pipeline134, the compute units132, the SIMD units138, the camera306and image sensor310may be implemented as a general purpose computer, a processor, or a processor core, or as a program, software, or firmware, stored in a non-transitory computer readable medium or in another medium, executable by a general purpose computer, a processor, or a processor core. The methods provided can be implemented in a general purpose computer, a processor, or a processor core. Suitable processors include, by way of example, a general purpose processor, a special purpose processor, a conventional processor, a digital signal processor (DSP), a plurality of microprocessors, one or more microprocessors in association with a DSP core, a controller, a microcontroller, Application Specific Integrated Circuits (ASICs), Field Programmable Gate Arrays (FPGAs) circuits, any other type of integrated circuit (IC), and/or a state machine. Such processors can be manufactured by configuring a manufacturing process using the results of processed hardware description language (HDL) instructions and other intermediary data including netlists (such instructions capable of being stored on a computer readable media). The results of such processing can be maskworks that are then used in a semiconductor manufacturing process to manufacture a processor which implements features of the disclosure.

The methods or flow charts provided herein can be implemented in a computer program, software, or firmware incorporated in a non-transitory computer-readable storage medium for execution by a general purpose computer or a processor. Examples of non-transitory computer-readable storage mediums include a read only memory (ROM), a random access memory (RAM), a register, cache memory, semiconductor memory devices, magnetic media such as internal hard disks and removable disks, magneto-optical media, and optical media such as CD-ROM disks, and digital versatile disks (DVDs).