Patent Description:
In order to capture infrared (IR) images with a camera, an IR filter and sensor is often implemented along with a visible light (red/green/blue "RGB") filter and sensor that captures visible light images. The use of a dual band RGBIR filter allows a single camera to be utilized to capture both visible light images and IR images. By using a single camera, size and cost can be saved, as well as avoiding the need to do a geometry calibration that is required when two cameras are utilized.

However, the colors resident in the visible image can be contaminated by the IR energy passing through the IR band of the dual band filter, since RGB pixels also have a spectral response in the IR band. Additionally, the IR image can be contaminated by the visible light energy in a similar manner.

Accordingly, in a camera that has a dual band RGBIR filter, conventional techniques for acquiring a quality RGB image employ a unique image signal processor (ISP), since the pattern of a pixel array on an RGBIR sensor is different than that of a conventional RGB sensor, which produces a conventional RGB "Bayer" pattern. Using an RGBIR sensor, some pixels of a conventional Bayer pattern are replaced by IR pixels. A conventional ISP is not able to perform conventional processing on the resulting image that it receives from an RGBIR sensor. <CIT> and <NPL> disclose and RGB-IR image sensor with image signal decontamination based on cross-talk and providing RGB and IR images for further processing.

A more detailed understanding can be had from the following description, given by way of example in conjunction with the accompanying drawings.

Although a more detailed description follows, briefly a method according to claim <NUM>, an apparatus according to claim <NUM>, and a computer program according to claim <NUM> is described herein for receiving and processing an image that includes a visible light (red/green/blue "RGB") component and an infrared (IR) component. The method allows a conventional image signal processor (ISP) to perform conventional processing an RGBIR signal with a decontamination function being performed on the signal as well as an interpolation (e.g., red, green, blue light interpolation) being performed on the signal. The contamination in the signal can be due to a number of factors.

For example, some contamination in a received signal may be due to spectral crosstalk which occurs because of imperfect color filters passing through some amount of unwanted light of the colors being filtered out. Optical spatial crosstalk contamination is due to color filters being located at some distance from an image pixel surface due to metal and insulation layers. The light coming at angles other than orthogonal passes through a filter and can partially be absorbed by the adjacent pixel being generated rather than one below. Depending on the f-number of the lens, this portion of the light absorbed by a neighboring pixel can vary significantly and can be big enough for low f-numbers. Micro-lenses located on the top of color filters can reduce this component of crosstalk significantly when appropriate form of micro-lenses and optimum position of them are chosen. Electrical crosstalk contamination results from photo-generated carriers having the possibility to move to neighboring charge accumulation sites. Electrical crosstalk occurs in both monochrome and color image sensors. The quantity of the carriers that can be accumulated by the neighboring pixel and the corresponding crosstalk depends on the pixel structure, collection area, and distribution of sensitivity inside a pixel. Accordingly, infrared light can contaminate the visible component of a received RGBIR image, and visible light can contaminate the infrared component of the received RGBIR image.

A method of performing processing in an image capturing device is disclosed. The method includes receiving an image by the image capturing device that includes an infrared component and a visible component. A decontamination is performed to generate a decontaminated infrared component and a decontaminated visible component. An interpolation is performed on the decontaminated visible component to generate an interpolated visible component, and the decontaminated infrared component and the decontaminated interpolated visible component are provided to an image signal processor (ISP) for further processing. An image capturing device is disclosed, The image capturing device includes a camera module, a processor operatively coupled to the camera module, and an image signal processor (ISP) operatively coupled with the processor. The camera module receives an image that includes an infrared component and a visible component and forwards the received image to the processor. The processor performs a decontamination to generate a decontaminated infrared component and a decontaminated visible component. The processor performs an interpolation on the decontaminated visible component to generate an interpolated visible component, and forwards the decontaminated infrared component and the decontaminated interpolated visible component to the ISP for further processing. A non-transitory computer readable medium having instructions recorded thereon, that when executed by a computing device, cause the computing device to perform operations is disclosed. The operations include receiving an image that includes an infrared component and a visible component. A decontamination is performed to generate a decontaminated infrared component and a decontaminated visible component. An interpolation is performed on the decontaminated visible component to generate an interpolated visible component, and the decontaminated infrared component and the decontaminated interpolated visible component are provided to an image signal processor (ISP) for further processing. <FIG> is a block diagram of an example system <NUM> in which one or more disclosed embodiments may be implemented. The example system <NUM> includes an image capturing device, (e.g., camera), <NUM> that includes at least a lens <NUM> for capturing an image. For purposes of example, the image shown in <FIG> is a person. The image that is captured by the lens <NUM> includes both visible (RGB) light and IR light. It should be noted that additional types of light can be captured by the lens <NUM>, but for purposes of example, visible and IR are discussed herein.

<FIG> is a block diagram of one of the example devices <NUM> depicted in <FIG> in which one or more features of the disclosure can be implemented. More detail regarding the implementation of a method of seam finding as it relates to <FIG> is described in further detail below. Although the device <NUM> has been described as an image capturing device, (e.g., camera), it is noted that the device <NUM> can include, for example, a computer, a gaming device, a handheld device, a set-top box, a television, a mobile phone, or a tablet computer. For purposes of example herein, the example device <NUM> is described as an image capturing device <NUM>, which for example is a camera.

Accordingly, the device <NUM> includes a processor <NUM>, a memory <NUM>, a storage <NUM>, one or more input devices <NUM>, a camera module <NUM>, and one or more output devices <NUM>. The device <NUM> can also optionally include an input driver <NUM> and an output driver <NUM>. It is understood that the device <NUM> can include additional components not shown in <FIG>.

The camera module <NUM> includes the lens <NUM> described above, a filter <NUM>, and a sensor <NUM>. The filter <NUM> and sensor <NUM> are a RGBIR filter/sensor pair that perform filtering for the RGB and IR light bands and generation of an RGBIR pattern, described in more detail below.

In various alternatives, the processor <NUM> includes a central processing unit (CPU), a graphics processing unit (GPU), a CPU and GPU located on the same die, or one or more processor cores, wherein each processor core can be a CPU or a GPU. In various alternatives, the memory <NUM> is be located on the same die as the processor <NUM>, or is located separately from the processor <NUM>. The memory <NUM> includes a volatile or non-volatile memory, for example, random access memory (RAM), dynamic RAM, or a cache. A conventional image signal processor (ISP) can be included in the processor <NUM> to perform RGBIR/Bayer image signal processing described in more detail below. Alternatively, the ISP can be included in the APD <NUM> or as a separate processing unit (not shown). That is, although the location of the conventional ISP is not specifically shown, it can reside separately from, or be integrated within the processor <NUM> or APD <NUM>. The storage <NUM> includes a fixed or removable storage, for example, a hard disk drive, a solid state drive, an optical disk, or a flash drive. The input devices <NUM> include, without limitation, a keyboard, a keypad, a touch screen, a touch pad, a detector, a microphone, an accelerometer, a gyroscope, a biometric scanner, or a network connection (e.g., a wireless local area network card for transmission and/or reception of wireless IEEE <NUM> signals). The output devices <NUM> include, without limitation, a display, a speaker, a printer, a haptic feedback device, one or more lights, an antenna, or a network connection (e.g., a wireless local area network card for transmission and/or reception of wireless IEEE <NUM> signals).

The input driver <NUM> communicates with the processor <NUM> the input devices <NUM>, and the lens <NUM>, and permits the processor <NUM> to receive input from the input devices <NUM> and the lens <NUM>. The output driver <NUM> communicates with the processor <NUM> and the output devices <NUM>, and permits the processor <NUM> to send output to the output devices <NUM>. It is noted that the input driver <NUM> and the output driver <NUM> are optional components, and that the device <NUM> will operate in the same manner if the input driver <NUM> and the output driver <NUM> are not present. The output driver <NUM> includes an accelerated processing device ("APD") <NUM> which is coupled to a display device <NUM>. The APD is configured to accept compute commands and graphics rendering commands from processor <NUM>, to process those compute and graphics rendering commands, and to provide pixel output to display device <NUM> for display. As described in further detail below, the APD <NUM> includes one or more parallel processing units configured to perform computations in accordance with a single-instruction-multiple-data ("SIMD") paradigm. Thus, although various functionality is described herein as being performed by or in conjunction with the APD <NUM>, in various alternatives, the functionality described as being performed by the APD <NUM> is additionally or alternatively performed by other computing devices having similar capabilities that are not driven by a host processor (e.g., processor <NUM>) and configured to provide graphical output to a display device <NUM>. For example, it is contemplated that any processing system that performs processing tasks in accordance with a SIMD paradigm may be configured to perform the functionality described herein. Alternatively, it is contemplated that computing systems that do not perform processing tasks in accordance with a SIMD paradigm performs the functionality described herein. The image signal processing described herein can also be performed by the APD <NUM>.

<FIG> is a block diagram of an example system <NUM> in which one or more features of the disclosure can be implemented. The system <NUM> includes substantially similar components to the device <NUM>, except for the camera module <NUM>. The system <NUM> can include, for example, a computer, a gaming device, a handheld device, a set-top box, a television, a mobile phone, or a tablet computer. The system <NUM> can be in communication with the device <NUM> depicted in <FIG> and can provide control programming to the device <NUM> and receive data from the device <NUM>.

<FIG> is a block diagram of the device <NUM> and system <NUM>, illustrating additional details related to execution of processing tasks on the APD <NUM>. For example, the calibration, decontamination and interpolation operations described below can be performed where appropriate utilizing the parallel SIMD paradigm described herein. The processor <NUM> maintains, in system memory <NUM>, one or more control logic modules for execution by the processor <NUM>. The control logic modules include an operating system <NUM>, a kernel mode driver <NUM>, and applications <NUM>. These control logic modules control various features of the operation of the processor <NUM> and the APD <NUM>. For example, the operating system <NUM> directly communicates with hardware and provides an interface to the hardware for other software executing on the processor <NUM>. The kernel mode driver <NUM> controls operation of the APD <NUM> by, for example, providing an application programming interface ("API") to software (e.g., applications <NUM>) executing on the processor <NUM> to access various functionality of the APD <NUM>. The kernel mode driver <NUM> also includes a just-in-time compiler that compiles programs for execution by processing components (such as the SIMD units <NUM> discussed in further detail below) of the APD <NUM>.

The APD <NUM> executes commands and programs for selected functions, such as graphics operations and non-graphics operations that may be suited for parallel processing. The APD <NUM> can be used for executing graphics pipeline operations such as pixel operations, geometric computations, and rendering an image to display device <NUM> based on commands received from the processor <NUM>. The APD <NUM> also executes compute processing operations that are not directly related to graphics operations, such as operations related to video, physics simulations, computational fluid dynamics, or other tasks, based on commands received from the processor <NUM>.

The APD <NUM> includes compute units <NUM> that include one or more SIMD units <NUM> that are configured to perform operations at the request of the processor <NUM> in a parallel manner according to a SIMD paradigm. The SIMD paradigm is one in which multiple processing elements share a single program control flow unit and program counter and thus execute the same program but are able to execute that program with different data. In one example, each SIMD unit <NUM> includes sixteen lanes, where each lane executes the same instruction at the same time as the other lanes in the SIMD unit <NUM> but can execute that instruction with different data. Lanes can be switched off with predication if not all lanes need to execute a given instruction. Predication can also be used to execute programs with divergent control flow. More specifically, for programs with conditional branches or other instructions where control flow is based on calculations performed by an individual lane, predication of lanes corresponding to control flow paths not currently being executed, and serial execution of different control flow paths allows for arbitrary control flow.

The basic unit of execution in compute units <NUM> is a work-item. Each work-item represents a single instantiation of a program that is to be executed in parallel in a particular lane. Work-items can be executed simultaneously as a "wavefront" on a single SIMD processing unit <NUM>. One or more wavefronts are included in a "work group," which includes a collection of work-items designated to execute the same program. A work group can be executed by executing each of the wavefronts that make up the work group. In alternatives, the wavefronts are executed sequentially on a single SIMD unit <NUM> or partially or fully in parallel on different SIMD units <NUM>. Wavefronts can be thought of as the largest collection of work-items that can be executed simultaneously on a single SIMD unit <NUM>. Thus, if commands received from the processor <NUM> indicate that a particular program is to be parallelized to such a degree that the program cannot execute on a single SIMD unit <NUM> simultaneously, then that program is broken up into wavefronts which are parallelized on two or more SIMD units <NUM> or serialized on the same SIMD unit <NUM> (or both parallelized and serialized as needed). A scheduler <NUM> is configured to perform operations related to scheduling various wavefronts on different compute units <NUM> and SIMD units <NUM>.

The parallelism afforded by the compute units <NUM> is suitable for graphics related operations such as pixel value calculations, vertex transformations, and other graphics operations. Thus in some instances, a graphics pipeline <NUM>, which accepts graphics processing commands from the processor <NUM>, provides computation tasks to the compute units <NUM> for execution in parallel.

The compute units <NUM> are also used to perform computation tasks not related to graphics or not performed as part of the "normal" operation of a graphics pipeline <NUM> (e.g., custom operations performed to supplement processing performed for operation of the graphics pipeline <NUM>). An application <NUM> or other software executing on the processor <NUM> transmits programs that define such computation tasks to the APD <NUM> for execution. For purposes of example, the method described herein, (e.g., the decontamination and the interpolation described below), can be performed by sending the instructions to the compute units <NUM> of the APD <NUM> for execution.

<FIG> is a flow diagram of an example method <NUM> for performing processing in a camera. In step <NUM>, a captured image that includes an RGB and IR component, (e.g., captured by camera <NUM>) undergoes black level correction (BLC). BLC is a conventional process for removing the black level in the input RGBIR image. Once the BLC is performed in step <NUM>, defect pixel correction (DPC) is performed on the received RGBIR image (step <NUM>). The DPC operation is a conventional operation where a pixel is compared to surrounding pixels, and if the differences between the pixel and the surrounding pixels exceeds a pre-defined threshold or is below another pre-defined threshold, the pixel is replaced, (e.g., with a pixel that is derived from the average values of the surrounding pixels). These conventional operations, (e.g., steps <NUM> and <NUM>), can be performed by the processor <NUM> or APD <NUM> described above.

Decontamination is then performed (step <NUM>) to remove IR light energy from the visible component and RGB light energy from the infrared component. The decontamination operation can be performed by the processor <NUM> or the APD <NUM>. For example, the RGBIR image is transferred into memory, (e.g., memory <NUM>). The image is then fed back into the processor, (e.g., processor <NUM> or APD <NUM>), to be decontaminated for the generation of a still image or video for output. The decontaminated image includes a separate visible component and an infrared component. The decontamination may be modeled in accordance with the following. By taking the vector I = [R,G,B,IR]T for an observed signal level, and the vector <MAT> for the ideal non-contaminated value, where T is the transpose of the matrix, R,G,B and IR are observed values from the RGBIR sensor, and where R̂, Ĝ, B̂ and <MAT> are ideal non-contaminated values. The relationship between the observed and ideal non-contaminated value can characterized in advance, for example through the use of a calibration operation (described below), can be used to generate a matrix A to describe the relationship between observed and ideal non-contaminated values. The contamination model can therefore be described in accordance with the following equation: <MAT>.

As described above, a calibration process can be performed prior to decontamination to determine a contamination factor that defines how much contamination exists in the received signal, (e.g., the contamination matrix A) based upon non-contaminated values. This calibration can be performed by generating one uniform test chart, which is a diffusor backlighted by narrow spectrum band light, (Red, Green, Blue or NIR), and capturing a series of raw images with an RGBIR camera, (e.g., image device <NUM>), on the test chart. The raw data generated can be used to calculate the contamination factor for each channel, (e.g., R/G/B/IR), to form the matrix A for each pixel position.

Accordingly, matrix A is the contamination matrix representing the contamination that is applied to the ideal non-contaminated values to arrive at the observed, contaminated signal levels. The decontamination matrix is then computed as the inverse contamination matrix. That is: <MAT> where A-<NUM> is the inverse contamination matrix A, (i.e., decontamination matrix).

Once the decontamination matrix is determined, the values of the components in the matrix can be utilized to remove unwanted contamination, (e.g., RGB or IR energy), from the image to generate the decontaminated visible component and infrared component. However, because the decontamination step removes a portion of the energy from the decontaminated image, potentially producing some level of a distorted component, an adjustment can be performed on the resultant components to account for the energy loss. Therefore, once decontaminated in step <NUM>, an IR adjustment is performed in step <NUM>, as well as an RGB adjustment in step <NUM>. The IR adjustment (step <NUM>) is performed on the infrared component. Again, the adjustment operations, (e.g., steps <NUM> and <NUM>), can be performed by the processor <NUM> or the APD <NUM>. That is, an example <NUM> bit pixel has a value between <NUM>-<NUM>. In a highlighted, or saturated area, the value will usually approach the upper boundary <NUM>, therefore the energy removal by decontamination in such an area results in a highlight contour, (i.e., discontinuity), pattern. To counter this effect, the observed IR image is compared against a pre-defined threshold, and if the value is greater than the pre-defined threshold, the strength of decontamination is degraded gradually towards the upper boundary <NUM> so that the discontinuity will not happen. That is, less decontamination may be performed, or not performed at all. The threshold can be increased or decreased, (e.g., by a programmer), as desired according to the visual effect being rendered.

In step <NUM>, the decontaminated RGB component is adjusted, (e.g., by the processor <NUM> or the APD <NUM>). That is, when removing the IR energy from the RGBIR image, dark areas are left in the image. If the image includes a large percentage of IR energy that is removed, this can lead to a very noticeable difference in the RGB component where the dark regions occur. Accordingly, the output decontaminated RGB image is compared against a pre-defined threshold, and if the value is less than the pre-defined threshold, a portion of the non-decontaminated RGB image is added back to the output RGB component to degrade the strength of decontamination in that portion. This threshold can also be increased or decreased according to the visual effect being rendered. On the other hand, in a highlighted or saturated area in the RGB image, if the observed RGB value is saturated, the decontamination function is not performed at all since it can be difficult to estimate the value of decontamination to remove the possible added energy by the IR contamination to get correct color. That is, the decontamination on such an area can result in false color.

Once the decontamination and adjustment steps are performed (steps <NUM> and <NUM>), interpolation is performed on the adjusted RGB component (step <NUM>) by the processor <NUM> or APD <NUM>, to prepare the component for conventional processing by a conventional ISP. During interpolation, the IR components of the image that are removed get replaced with visible components, (e.g., R, G, B components). <FIG> shows an example RGBIR pattern <NUM>. As shown in pattern <NUM>, there are four <NUM> x <NUM> RGBIR patterns, each group including a group of B, G, IR, R components. One group is shown in the top left quadrant, one group in the top right quadrant, one group in the bottom left quadrant, and one group in the bottom right quadrant. During interpolation in step <NUM>, each IR component removed during decontamination is replaced with a G component. Therefore, <FIG> shows an example interpolated Bayer pattern 500B with IR contamination removed. The decontaminated and adjusted IR Image, and the decontaminated, adjusted and interpolated RGB image are both then provided to the conventional ISP for further conventional processing.

Thus, as shown in <FIG>, a G component exists in each quadrant where the IR component has been removed. However, it should be noted that an example <NUM> x <NUM> Bayer pattern can include four possible layouts, "RGGB", "BGGR", "GRBG" or "GBRG". The example BGGR layout depicted in <FIG> is one example layout interpolating from that in <FIG>.

The methods provided can be implemented in a general purpose computer, a processor, or a processor core. Suitable processors include, by way of example, a general purpose processor, a special purpose processor, a conventional processor, a digital signal processor (DSP), a plurality of microprocessors, graphics processor, one or more microprocessors in association with a DSP core, a controller, a microcontroller, Application Specific Integrated Circuits (ASICs), Field Programmable Gate Arrays (FPGAs) circuits, any other type of integrated circuit (IC), and/or a state machine. Such processors can be manufactured by configuring a manufacturing process using the results of processed hardware description language (HDL) instructions and other intermediary data including netlists (such instructions capable of being stored on a computer readable media). The results of such processing can be maskworks that are then used in a semiconductor manufacturing process to manufacture a processor which implements aspects of the embodiments. For example, the above methods can be implemented either in a hardware implementation of a system on chip (SoC), or in the programming of a GPU to compute normal RGB and IR images.

Claim 1:
A method of performing processing in an image capturing device, comprising:
receiving an image at the image capturing device (<NUM>), wherein each pixel of the image includes a value of an infrared component and a value of a visible component;
obtaining, by a first processor (<NUM>), a contamination matrix that defines relationships between ideal non-contaminated values and the value of the infrared component and the value of the visible component of each pixel;
generating, by the first processor (<NUM>), decontaminated values of the infrared component and decontaminated values of the visible component for each pixel by performing a decontamination process based on the contamination matrix;
performing, by a first processor (<NUM>), an adjustment process on the decontaminated values of the visible component, wherein the adjustment process includes adding a portion of non-decontaminated values to values of the visible component of a corresponding portion from the image when the one or more decontaminated values of the visible component are less than a threshold;
performing, by the first processor (<NUM>), an interpolation on the decontaminated values of the visible component to generate interpolated values of the visible component; and
providing, by the first processor (<NUM>), the decontaminated values of the infrared component and the decontaminated interpolated values of the visible component to an image signal processor (ISP) for further processing.