Signal processing apparatus and method, image sensing apparatus and system for correction of pixel signal output based on signal output of adjacent pixels and calculated correction coefficient

A signal processing apparatus performs predetermined signal processing on an image signal output from an image sensor having a pixel array in which a plurality of pixels are arrayed in a direction along a row and a direction along a column. The signal processing apparatus comprises: a storage unit that stores characteristic information indicating characteristics of signal component mix in each pixel from adjacent pixels according to the pixel position in the pixel array of the image sensor; and a correction unit that calculates a correction coefficient according to the position of a pixel for correction in the pixel array from the characteristic information, and corrects an output image signal of the pixel for correction based on an output image signal of adjacent pixels of the pixel for correction and the calculated correction coefficient.

BACKGROUND OF THE INVENTION

1. Field of the Invention

The present invention relates to a signal processing apparatus, image sensing apparatus, image sensing system, and signal processing method.

2. Description of the Related Art

In a conventional image sensing apparatus such as a digital camera or a digital video camera, a CCD image sensor, a CMOS image sensor, or the like is used as an image sensor.

An image sensor IS shown inFIG. 13is provided with a pixel array PA in which a plurality of pixels are arranged in a direction along a row and a direction along a column. Each of pixels P1to P5in the pixel array PA includes a photoelectric conversion unit PD, a color filter CF, and a microlens ML. In each of the pixels P1to P5, an open area OA between the color filter CF and microlens ML and the photoelectric conversion unit PD is defined by wiring layers WL. The photoelectric conversion units PD of adjacent pixels are electrically separated from each other by an element isolation region IR. The photoelectric conversion units PD and the element isolation regions IR are disposed within a semiconductor substrate SB.

A case is considered in which, inFIG. 13, the color filter CF of the pixel P1transmits light of a first color, and the color filter CF of the pixel P2transmits light of a second color. There may be an instance where first color light IL1′ among diagonally incident light IL1on the pixel P1passes through the open area OA of the pixel P1, and then a portion IL1″ of the first color light IL1′ is transmitted through the element isolation region IR and arrives at the photoelectric conversion unit PD of the adjacent pixel P2. In this case, although the photoelectric conversion unit PD of the pixel P2should properly receive the light of the second color, the photoelectric conversion unit PD further receives the first color light IL1″ mixed in from the adjacent pixel, and generates a signal corresponding to the first color light IL1″. That is, a so-called color mixture phenomenon may occur in which the signals of adjacent pixels interfere with each other.

The ease with which this color mixture occurs differs according to the F value (stop aperture diameter) of a shooting lens in the image sensing apparatus, as shown inFIGS. 14A and 14B. Compared to a case where the stop aperture diameter is small as shown inFIG. 14B, in a case where the stop aperture diameter is large as shown inFIG. 14A, diagonal incident light more easily mixes into an adjacent pixel. That is, as the F value of the shooting lens becomes smaller, the stop aperture diameter becomes larger, and there is a tendency for the amount of diagonal incident light that mixes into an adjacent pixel to increase.

Also, the ease with which color mixture occurs differs according to an exit pupil distance (distance from the image plane to the exit pupil position of the shooting lens) of the shooting lens in the image sensing apparatus, as shown inFIGS. 15A and 15B. Compared to a case where the exit pupil distance is long as shown inFIG. 15A, in a case where the exit pupil distance is short as shown inFIG. 15B, diagonal incident light more easily mixes into an adjacent pixel. That is, as the exit pupil distance becomes shorter, there is a tendency for the amount of diagonal incident light that mixes into an adjacent pixel to increase.

As shown inFIG. 16, the photoelectric conversion unit PD of the pixel P2receives second color light IL2′ that has been transmitted through the color filter CF of the pixel P2among the incident light IL2on the pixel P2, and generates charges (signal) according to the received second color light IL2′. Furthermore, the photoelectric conversion unit PD of the pixel P2, which has received the first color light IL1″ from the adjacent pixel P1for the reasons shown inFIGS. 14A,14B,15A, and15B, generates charges (signal) according to the first color light IL1″, as shown inFIG. 16. Thus, the photoelectric conversion unit PD of the pixel P2generates a signal according to the first color light IL1″ in addition to a signal according to the second color light IL2′, and thereby color mixture occurs.

Also, the ease with which color mixture occurs differs according to the color of light incident on the photoelectric conversion unit PD, as shown inFIGS. 16 and 17. This is because the depth from the surface of the semiconductor substrate SB at the position where light is converted to charges in the photoelectric conversion unit PD differs according to the wavelength of the light. That is, this is because, in comparison to light having a short wavelength, light having a long wavelength is photoelectrically converted at a deeper position in the photoelectric conversion unit PD.

Here, the color filter CF of the pixel P1shown inFIG. 16transmits red (R) light, the color filter CF of the pixel P2transmits green (G) light, and the color filter CF of a pixel P6shown inFIG. 17transmits blue (B) light.

As shown inFIG. 16, light that has passed through the red (R) light color filter CF, in comparison to light that has passed through the color filters CF of the other colors (G, B), is photoelectrically converted at a deeper position in the photoelectric conversion unit PD. Therefore, the red (R) light IL1′ passes through the photoelectric conversion unit PD of the pixel P1where that light should be incident, and a portion IL1″ of that light easily becomes incident on the photoelectric conversion unit PD of the adjacent pixel P2. The light IL1″ that is incident on the photoelectric conversion unit PD of the adjacent pixel P2is photoelectrically converted there, so without producing charges (signal) of the pixel P1where the light should be incident, a mixed color component for the signal of the adjacent pixel P2is generated.

On the other hand, as shown inFIG. 17, light IL6′ that has passed through the blue (B) color filter CF, in comparison to light IL1′ and IL2′ that has passed through the color filters CF of the other colors (R, G), is photoelectrically converted at a shallower position in the photoelectric conversion unit PD. Therefore, even if a light ray IL6is diagonally incident on the pixel P6, there is a tendency for the light ray to be photoelectrically converted in the photoelectric conversion unit PD of the pixel P6prior to arriving at the photoelectric conversion unit PD of an adjacent pixel P7. That is, because it is unlikely that the blue (B) light IL6′ will pass through the photoelectric conversion unit PD of the pixel P6where that light should be incident and arrive at the adjacent pixel P7, it is unlikely that a mixed color component for the signal of the adjacent pixel P7will be generated.

Also, as shown inFIG. 18, at a deep position in the semiconductor substrate SB, between the photoelectric conversion units PD of adjacent pixels, electrical separation by the element isolation region IR is inadequate. Therefore, charges (signal) that are stored at a deep position in the photoelectric conversion unit PD of the pixel P1are dispersed and mixed into the photoelectric conversion unit PD of the adjacent pixels P2and P4at a deep position in the semiconductor substrate SB. This crosstalk within the semiconductor substrate SB also causes color mixture.

The ease with which color mixture due to this crosstalk occurs differs according to the color light that is incident on the photoelectric conversion unit PD, as shown inFIGS. 19 and 20.

As shown inFIG. 19, light IL1′ that has passed through the red (R) color filter CF, in comparison to light that has passed through the color filters CF of the other colors (G, B), is photoelectrically converted and stored at a deeper position in the photoelectric conversion unit PD. Therefore, charges (signal) stored in the photoelectric conversion unit PD of the pixel P1according to the red (R) light IL1′ easily passes, at a deep position in the semiconductor substrate SB, through the area deeper than the element isolation region IR and is dispersed in the photoelectric conversion units PD of the adjacent pixels P2and P4. In the area deeper than the element isolation region IR, it is conceivable that electrical separation is inadequate between the photoelectric conversion units of adjacent pixels. Thus, charges (signal) dispersed in the photoelectric conversion units PD of the adjacent pixels P2and P4easily generate mixed color components for the signal of the adjacent pixels P2and P4, without becoming the charges (signal) of the pixel P1where the dispersed charges (signal) should be stored.

On the other hand, as shown inFIG. 20, light IL6′ that has passed through the blue (B) color filter CF, in comparison to light that has passed through the color filters CF of the other colors (R, G), is photoelectrically converted at a shallower position in the photoelectric conversion unit PD. Therefore, charges (signal) stored in the photoelectric conversion unit PD of the pixel P6according to the blue (B) light IL6′ is blocked, at a shallow position in the semiconductor substrate SB, by the element isolation region IR, and is unlikely to be dispersed into the photoelectric conversion unit PD of an adjacent pixel P8. The charges (signal) stored in the photoelectric conversion unit PD of the pixel P6according to the blue (B) light IL6′ are unlikely to generate mixed color components for the signal of the adjacent pixel P8.

Due to color mixture that occurs in this manner, the image signal that is output from the image sensor deteriorates, and thereby color reproducibility deteriorates.

Japanese Patent Laid-Open No. 2004-135206 describes that, in a CCD image sensing element having a color filter array according to a Bayer array, color mixture correction subtracts, from the signal of a designated color pixel, a fixed ratio calculated from the signal of the designated color pixel and the signal of an adjacent pixel of a color other than the designated color.

In this correction processing, it is assumed that color mixture occurs, relative to a pixel of interest, isotropically from a plurality of surrounding pixels that are adjacent to that pixel of interest, i.e., that a signal component is mixed in at the same ratio from a plurality of surrounding pixels. Under this assumption, a signal component of a fixed ratio is isotropically subtracted.

On the other hand, Japanese Patent Laid-Open No. 2007-142697 describes that, in an actual solid image sensing element, the light receiving face of a photoelectric conversion unit is disposed at an offset position within a pixel, depending on the wiring pattern and the layout of electrodes within the pixel or in the vicinity of the pixel. As a result, the physical center of the pixel and the optical center of the pixel do not match, and thereby color mixture from surrounding pixels relative to the pixel of interest can be made to occur with directionality.

To address this problem, Japanese Patent Laid-Open No. 2007-142697 proposes changing, independent of each other, correction parameters Ka, Kb, Kc, and Kd for correcting color mixture from surrounding pixels respectively at the upper left, upper right, lower left, and lower right. Thus, according to Japanese Patent Laid-Open No. 2007-142697, it is possible to realize correction processing of color mixture that is made to have directionality according to the amount of color mixture from the surrounding pixels.

In Japanese Patent Laid-Open No. 2007-142697, a correction circuit for performing color mixture correction processing receives a control signal of directionality selection supplied from outside via a communications I/F, and changes the correction parameters Ka, Kb, Kc, and Kd independent of each other according to the received directionality selection control signal. Specifically, when the directionality selection control signal that has been supplied from outside via the communications I/F is 0, Ka=Kb=K1, and Kc=Kd=K2 are set. When the directionality selection control signal is 1, Ka=Kc=K1, and Kb=Kd=K2 are set, and when the directionality selection control signal is 2, Ka=Kd=K1, and Kb=Kc=K2 are set. In the correction circuit described in Japanese Patent Laid-Open No. 2007-142697, the correction parameters Ka, Kb, Kc, and Kd used for the respective signals of a plurality of pixels disposed in the solid image sensing element have values that are common to each pixel. However, strictly speaking, in the actual image sensing apparatus, the angle of incident light rays on the pixels differs according to the arrangement of pixels in a sensor face. The amount of color mixture from an adjacent pixel for a pixel of interest differs according to the light ray incidence angle, i.e., the amount of color mixture in each pixel differs according to the pixel arrangement of respective pixels in the sensor face. In the case of such color mixture that occurs in a non-uniform manner in the sensor face, with the correction circuit described in Japanese Patent Laid-Open No. 2007-142697, there is a high possibility that the accuracy of color mixture correction processing will deteriorate according to the position of pixels in the sensor face (pixel array).

SUMMARY OF THE INVENTION

The present invention aims to improve the accuracy of color mixture correction processing for each pixel in a pixel array.

According to a first aspect of the present invention, there is provided a signal processing apparatus that performs predetermined signal processing on an image signal output from an image sensor having a pixel array in which a plurality of pixels are arrayed in a direction along a row and a direction along a column, the signal processing apparatus comprising: a storage unit that stores characteristic information indicating characteristics of signal component mix in each pixel from adjacent pixels according to the pixel position in the pixel array of the image sensor; and a correction unit that calculates a correction coefficient according to the position of a pixel for correction in the pixel array from the characteristic information, and corrects an output image signal of the pixel for correction based on an output image signal of adjacent pixels of the pixel for correction and the calculated correction coefficient.

According to a second aspect of the present invention, there is provided an image sensing apparatus, comprising: an image sensor that has a pixel array in which a plurality of pixels are arrayed in a direction along a row and a direction along a column, and a readout unit that reads out a signal from the pixel array; a storage unit that stores characteristic information indicating characteristics of signal component mix in each pixel from adjacent pixels according to the pixel position in the pixel array of the image sensor; and a correction unit that calculates a correction coefficient according to the position of a pixel for correction in the pixel array from the characteristic information, and corrects an output image signal that has been read out from the pixel for correction by the readout unit based on an output image signal that has been read out from adjacent pixels of the pixel for correction by the readout unit and the calculated correction coefficient.

According to a third aspect of the present invention, there is provided an image sensing system, comprising: an image sensing apparatus that generates image data by performing image sensing of an object; and a processing apparatus that receives the image data from the image sensing apparatus, and processes the received image data; the image sensing apparatus including: an image sensor having a pixel array in which a plurality of pixels are arrayed in a direction along a row and a direction along a column, and a readout unit that reads out a signal from the pixel array; a storage unit that stores characteristic information indicating characteristics of signal component mix in each pixel from adjacent pixels according to the pixel position in the pixel array of the image sensor; and a generation unit that generates the image data by attaching the characteristic information to an image signal that has been read out by the readout unit; the processing apparatus including a correction unit that calculates a correction coefficient according to the position of a pixel for correction from the characteristic information included in the image data, and corrects an output image signal of the pixel for correction in the image data based on an output image signal of adjacent pixels of the pixel for correction and the calculated correction coefficient in the image data.

According to a fourth aspect of the present invention, there is provided a method for signal processing of an image signal that is output from an image sensor in which a plurality of pixels are arrayed in a direction along a row and a direction along a column, the method comprising: a first step of calculating, from characteristic information indicating characteristics of signal component mix in each pixel from adjacent pixels according to the pixel position in a pixel array of the image sensor that has been stored in advance, a correction coefficient according to the position of a pixel for correction in the pixel array; and a second step of correcting an output image signal of the pixel for correction based on an output image signal of adjacent pixels of the pixel for correction and the calculated correction coefficient.

According to the present invention, it is possible to improve the accuracy of color mixture correction processing for each pixel in a pixel array.

DESCRIPTION OF THE EMBODIMENTS

In the present specification, light incidence angle means an angle that the center of gravity of light forms with the normal line of a light incidence face.

The present inventors focused their attention on the fact that, as shown inFIG. 21, the incidence angle of a light ray PL incident on each pixel of an image sensor IS100via an aperture portion of a stop ST differs according to the positional relationship between an optical center PC of a pixel array PA100and a pixel in the pixel array PA100. The present inventors thought that the ease with which color mixture occurs depends also on the difference in the incidence angle of the light ray on each pixel, i.e., the difference in the position of each pixel in the pixel array PA100.

However, as shown inFIG. 21, with the image sensor IS100having a large pixel size (pitch), a large difference in color mixture was not found between a pixel P1positioned at the optical center PC of the pixel array PA100and a pixel P3at a position shifted from the optical center PC.

Here, the pixel P1receives light having a center of gravity CG1that forms a small incidence angle (≈0°), and the pixel P3receives light having a center of gravity CG3that forms a large incidence angle. When the pixel size (pitch) is large, even when there is a light ray (center of gravity CG3) diagonally incident on the pixel P3, which is in a position shifted from the optical center PC, that light ray is unlikely to arrive at an adjacent pixel P8. In other words, because the difference in color mixture due to the difference in the position of each pixel in the pixel array PA100is sufficiently small relative to the absolute amount of color mixture in each pixel, there is a tendency that the difference in color mixture does not become large enough to cause a problem.

On the other hand, as shown inFIG. 22, with an image sensor IS200having a small pixel size (pitch), a large difference in color mixture was found between a pixel P1positioned at the optical center PC of a pixel array PA200and a pixel P13at a position shifted from the optical center PC.

Here, the pixel P1receives light having a center of gravity CG1that forms a small incidence angle, and the pixel P13receives light having a center of gravity CG3that forms a larger incidence angle. When the pixel size (pitch) is small, a light ray (center of gravity CG3) diagonally incident (at a large incidence angle) on the pixel P13, which is in a position shifted from the optical center PC, easily arrives at an adjacent pixel P18. In other words, because the difference in color mixture due to the difference in the position of each pixel in the pixel array PA200is large relative to the absolute amount of color mixture in each pixel, there is a tendency that the difference in color mixture becomes large enough to cause a problem so that the difference in color mixture cannot be ignored.

In particular, recently shrinkages in pixel size are advancing, and due to further reducing the size of pixels, as shown inFIG. 22, it is conceivable that the difference in color mixture due to the difference in the position of each pixel in the pixel array PA200will further increase. Therefore, it is conceivable that the difference in color mixture due to the difference in the position of each pixel in the pixel array PA200will cause a problem that cannot be ignored, and will have a large effect on color reproducibility.

Next is a description of an image sensing apparatus100according to a first embodiment of the present invention, with reference toFIG. 1.FIG. 1shows the configuration of the image sensing apparatus100according to the first embodiment of the present invention.

The image sensing apparatus100, for example, is a digital camera or a digital video camera. The image sensing apparatus100is provided with the following constituent elements.

An optical system1includes a lens1aand a stop1b. The lens1arefracts incident light to form an image of an object in a pixel array (image sensing face) of an image sensor3. The stop1bis provided between the lens1aand the image sensor3in an optical path, and adjusts the amount of light guided to the image sensor3after passing through the lens1a.

A mechanical shutter2is provided between the optical system1and the image sensor3in the optical path, and controls exposure of the image sensor3.

The image sensor3converts the image of the object that has been formed in a pixel array PA300to an image signal. That is, the image sensor3performs image sensing of the object. The image sensor3reads out and outputs that image signal from the pixel array PA300. The image sensor3, for example, is a CMOS image sensor or a CCD image sensor.

Specifically, as shown inFIG. 2, the image sensor3includes the pixel array PA300, a readout unit31, and an amplifier32.FIG. 2shows the configuration of the image sensor3in the first embodiment of the present invention.

In the pixel array PA300, a plurality of pixels are arrayed in a direction along a row and a direction along a column. Each pixel includes a photoelectric conversion unit PD (seeFIG. 13). The photoelectric conversion unit PD generates and stores charges (signal) according to light.

The readout unit31reads out a signal from the pixel array PA300. That is, the readout unit31reads out the charges (signal) stored by the photoelectric conversion unit PD of each pixel of the pixel array PA300, or a signal corresponding to the charges, from each pixel. The readout unit31supplies the signal that has been read out to the amplifier32. The readout unit31, for example, is a circuit that reads out a signal via a vertical signal line from each pixel in a CMOS image sensor, or is a vertical/horizontal transfer CCD in a CCD image sensor.

The amplifier32generates and outputs an image signal by amplifying the supplied signal. The amplifier32, for example, is an output amplifier that amplifies a signal received via a horizontal signal line from the readout unit31in a CMOS image sensor, or is a floating diffusion amplifier in a CCD image sensor.

An A/D converter4receives an image signal (analog signal) that has been output from the image sensor3. The A/D converter4converts the received image signal (analog signal) to an image signal (digital signal) and outputs the converted signal.

A timing signal generation circuit5generates a timing signal used as a reference and supplies that timing signal to a driving circuit6.

The driving circuit6, in synchronization with the supplied timing signal, respectively drives the optical system1, the mechanical shutter2, the image sensor3, and the A/D converter4.

A power switch (power SW)16receives an instruction to turn on power from a user. The power switch16supplies the received power switch on instruction to a system control unit13.

A first switch (SW1)17receives a first instruction from the user. The first instruction is, for example, an instruction for causing performance of focus adjustment processing, exposure control processing, and white balance processing. The first switch17supplies the received first instruction to the system control unit13.

A second switch (SW2)18receives a second instruction from the user. The second instruction is, for example, an instruction for causing execution of shooting processing. The second switch18supplies the received second instruction to the system control unit13.

The system control unit13performs overall control of each part of the image sensing apparatus100.

For example, the system control unit13starts up each part in response to a power on instruction received from the power switch16.

For example, the system control unit13controls each part so as to perform focus adjustment processing, exposure control processing, and white balance processing in response to a first instruction received from the first switch17.

For example, the system control unit13controls each part so as to perform shooting processing in response to a second instruction received from the second switch18.

A volatile memory (RAM)15temporarily stores predetermined data. For example, the volatile memory15is used as a work area of the system control unit13. That is, the system control unit13transfers programs stored in a non-volatile memory14, control data, and correction data to the volatile memory15and temporarily stores them there, and appropriately refers to the stored items when performing control of each part.

The non-volatile memory (ROM)14stores a program describing a control method executed by the system control unit13, and control data such as parameters and tables used when executing the program.

Also, the non-volatile memory14stores information used for color mixture correction processing, i.e., a first characteristic of color mixture information and a second characteristic of color mixture information.

The first characteristic of color mixture information is information that indicates characteristics related to a signal component that mixes in from an adjacent pixel to a pixel corresponding to a position in the pixel array PA300(seeFIG. 2). The first characteristic of color mixture information includes a first coefficient table. The first coefficient table is a table in which, for each pixel, a position in the pixel array PA300is associated with a first correction coefficient that has been determined in advance so as to correct a signal component that mixes into a pixel from adjacent pixels.

The second characteristic of color mixture information is information that indicates characteristics related to a signal component that leaks out to an adjacent pixel from a pixel corresponding to a position in the pixel array PA300(seeFIG. 2). The second characteristic of color mixture information includes a second coefficient table. The second coefficient table is a table in which, for each pixel, a position in the pixel array PA300is associated with a second correction coefficient that has been determined in advance so as to correct a signal component that leaks out from a pixel to adjacent pixels.

A signal processing circuit7receives an image signal (digital signal) that has been output from the A/D converter4. The signal processing circuit7performs predetermined signal processing on the received image signal (digital signal).

For example, the signal processing circuit (correction unit)7performs color mixture correction processing that corrects a signal that has been read out from a pixel for correction in the pixel array PA300by the readout unit31(seeFIG. 2).

Specifically, the signal processing circuit7calculates the first correction coefficient for correcting a signal component that mixes into the pixel for correction from its adjacent pixel according to the position of the pixel for correction in the pixel array PA300and the first coefficient table included in the first characteristic of color mixture information.

The signal processing circuit7calculates the second correction coefficient for correcting a signal component that leaks out from the pixel for correction to its adjacent pixel according to the position of the pixel for correction in the pixel array PA300and the second coefficient table included in the second characteristic of color mixture information (first step).

The signal processing circuit7uses a signal read out from adjacent pixels of the pixel for correction by the readout unit31(seeFIG. 2), the first correction coefficient, and the second correction coefficient to correct the signal that has been read out from the pixel for correction by the readout unit31(second step).

The signal processing circuit7generates image data by performing this signal processing. The signal processing circuit7supplies the generated image data to an image memory8or the system control unit13. Alternatively, the signal processing circuit7converts the generated image data to compressed image data for recording, and supplies the converted compressed image data to a recording circuit10. Alternatively, the signal processing circuit7converts the generated image data to an image signal for display, and supplies the converted image signal for display to a display circuit12.

The image memory8temporarily stores the image data that has been supplied from the signal processing circuit7.

A recording medium9is detachably connected to the recording circuit10. The recording circuit10records the compressed image data for recording that has been supplied from the signal processing circuit7to the connected recording medium9.

The display circuit12displays an image corresponding to the image signal for display supplied from the signal processing circuit7in an image display device11.

In this way, the signal processing circuit7calculates each of the first correction coefficient and the second correction coefficient according to the position of the pixel for correction in the pixel array PA300(seeFIG. 2), so it is possible to appropriately set each of the first correction coefficient and the second correction coefficient according to the position of the pixel for correction.

Also, the signal processing circuit7uses a signal read out from adjacent pixels of the pixel for correction, the first correction coefficient, and the second correction coefficient to correct the signal that has been read out from the pixel for correction. Thus, it is possible to perform appropriate correction according to the position of the pixel for correction in the pixel array PA300(seeFIG. 2).

Accordingly, it is possible to reduce the effect of a difference in color mixture due to a difference in the position of each pixel in the pixel array PA300on the color reproducibility of an image corresponding to the image signal, so it is possible to improve the accuracy of color mixture correction processing for each pixel in the pixel array.

Next is a description of operation in shooting processing using the mechanical shutter2in the image sensing apparatus100. The shooting processing, as described later, includes exposure processing, development processing, and recording processing.

Prior to operations (processings) in the shooting processing, when starting operation of the system control unit13, such as when powering on the image sensing apparatus100, necessary programs, control data, and correction data are transferred from the non-volatile memory14to the volatile memory15, and stored there.

Then, the shooting processing starts. With the start of shooting processing, the system control unit13uses the various programs and data by, as necessary, transferring the various programs and data from the non-volatile memory14to the volatile memory15, or directly reading out the various programs and data from the non-volatile memory14.

The system control unit13controls exposure processing in the shooting processing. With a control signal from the system control unit13, the optical system1is driven for the stop1band the lens1ato form an image of an object that has been set to an appropriate brightness on the image sensor3.

With a control signal from the system control unit13, the mechanical shutter2is driven so as to expose the image sensor3during a necessary exposure time in accordance with operation of the image sensor3. Here, when the image sensor3has an electronic shutter function, this function may be used together with the mechanical shutter2to secure the necessary exposure time.

The image sensor3is driven by a driving pulse generated by the driving circuit6based on an operation pulse generated by the timing signal generation circuit5controlled by the system control unit13, and photoelectrically converts the object image to an electrical signal, and outputs that signal as an analog image signal.

The analog image signal output from the image sensor3is converted to a digital image signal by the A/D converter4, by the driving pulse generated by the driving circuit6based on the operation pulse generated by the timing signal generation circuit5controlled by the system control unit13.

The system control unit13controls development processing in the shooting processing. Thus, the signal processing circuit7generates image data by performing image processing such as various correction including color mixture correction, color conversion, white balance, and gamma correction, resolution conversion processing, image compression processing, and so forth with respect to the digital image signal.

The image memory8is used in order to temporarily store a digital image signal during signal processing, and to store image data that is the signal-processed digital image signal.

The system control unit13controls recording processing in the shooting processing. Thus, the image data that has been signal-processed by the signal processing circuit7and the image data that has been stored in the image memory8, in the recording circuit10, is converted to compressed image data appropriate to the recording medium9(for example, compressed data of a file system having a hierarchical structure), and recorded to the recording medium9.

Also, after resolution conversion processing by the signal processing circuit7is performed on the image data that has been converted to a digital image signal by the A/D converter4, that processed image data is converted to a signal appropriate to the image display device11(for example, such as an NTSC-format analog signal) in the display circuit12. Then, that converted signal is displayed in the image display device11.

Here, in the signal processing circuit7, without performing signal processing by a control signal from the system control unit13, the digital image signal may be used as-is as image data, and output to the image memory8or the recording circuit10.

Also, when there has been a request from the system control unit13, the signal processing circuit7outputs information of the digital image signal or image data produced in the course of signal processing, or information extracted from such digital image signal or image data, to the system control unit13. Information of the digital image signal or image data may be, for example, information such as an image spatial frequency, the mean value of a designated region, the amount of compressed image data, and so forth.

When there has been a request from the system control unit13, the recording circuit10outputs information such as the type and available space of the recording medium9to the system control unit13.

Next is a description of playback operation in the image sensing apparatus100when image data has been recorded to the recording medium9.

The system control unit13receives an instruction to play back the image data recorded to the recording medium9from the first switch17and/or the second switch18, or from another switch (not shown). The system control unit13controls the recording circuit10according to the received playback instruction.

By a control signal from the system control unit13, the recording circuit10reads out image data from the recording medium9.

By a control signal from the system control unit13, the signal processing circuit7, when the image data is a compressed image, performs image decompression processing, and then stores the image data in the image memory8. After resolution conversion processing by the signal processing circuit7is performed on the image data stored in the image memory8, that processed image data is converted to a signal appropriate to the image display device11in the display circuit12, and displayed in the image display device11.

Next is a description of color mixture that occurs between a pixel for correction and pixels adjacent to that pixel, with reference toFIG. 3.FIG. 3schematically shows color mixture that occurs between a pixel for correction and pixels adjacent to that pixel.

A case is considered where attention is focused on one specific pixel (referred to as pixel X). In comparison to an ideal signal SigX′, the level of a signal SigX that is read out from the pixel X is reduced by signal components SC11to SC14that leak out to adjacent pixels, and the level of that signal SigX is increased by signal components SC1to SC4that mix in from adjacent pixels.

Here, the signal component SC11is a signal component that has leaked out from the pixel X to a pixel L that is adjacent in a first direction (for example, left) relative to the pixel X. The signal component SC12is a signal component that has leaked out from the pixel X to a pixel R that is adjacent in a second direction (for example, right) relative to the pixel X. The signal component SC13is a signal component that has leaked out from the pixel X to a pixel U that is adjacent in a third direction (for example, up) relative to the pixel X. The signal component SC14is a signal component that has leaked out from the pixel X to a pixel D that is adjacent in a fourth direction (for example, down) relative to the pixel X.

The signal component SC1is a signal component that has mixed into the pixel X from the pixel L that is adjacent in the first direction (for example, left) relative to the pixel X. The signal component SC2is a signal component that has mixed into the pixel X from the pixel R that is adjacent in the second direction (for example, right) relative to the pixel X. The signal component SC3is a signal component that has mixed into the pixel X from the pixel U that is adjacent in the third direction (for example, up) relative to the pixel X. The signal component SC4is a signal component that has mixed into the pixel X from the pixel D that is adjacent in the fourth direction (for example, down) relative to the pixel X.

The amount of the signal that has leaked out in a predetermined direction from the pixel X has a fixed ratio relative to the ideal signal SigX′, but here, for the sake of simplification, the amount of that signal is considered to have a fixed ratio relative to the signal SigX. The signal amount of the components that have leaked out to adjacent pixels from the pixel X can be calculated by multiplying SigX by a coefficient that expresses the ratio of the signal that leaks out to the adjacent pixel in each direction from the pixel X relative to the signal SigX that has been read out from the pixel X.

Likewise, the signal amount of the components that have mixed into the pixel X from the adjacent pixels can be calculated by multiplying the signal that has been read out from each adjacent pixel by a coefficient that expresses the ratio of the signal that mixes into the pixel X from each adjacent pixel relative to the signal that has been read out from each adjacent pixel.

Here, a signal that has been read out from the pixel X is called SigX, and a signal that has been read out from the pixel L that is adjacent in the first direction (for example, left) relative to the pixel X is called SigL. A signal that has been read out from the pixel R that is adjacent in the second direction (for example, right) relative to the pixel X is called SigR, and a signal that has been read out from the pixel U that is adjacent in the third direction (for example, up) relative to the pixel X is called SigU. A signal that has been read out from the pixel D that is adjacent in the fourth direction (for example, down) relative to the pixel X is called SigD.

Also, a coefficient for correcting the signal component that leaks out from the pixel X to the pixel L is called [x1], and a coefficient for correcting the signal component that mixes into the pixel X from the pixel L is called [l2]. A coefficient for correcting the signal component that leaks out from the pixel X to the pixel R is called [x2], and a coefficient for correcting the signal component that mixes into the pixel X from the pixel R is called [r1]. A coefficient for correcting the signal component that leaks out from the pixel X to the pixel U is called [x3], and a coefficient for correcting the signal component that mixes into the pixel X from the pixel U is called [u4]. A coefficient for correcting the signal component that leaks out from the pixel X to the pixel D is called [x4], and a coefficient for correcting the signal component that mixes into the pixel X from the pixel D is called [d3].

Here, the ideal signal SigX′, in which the color mixture component included in the signal SigX of the pixel X has been corrected, is calculated by the following Formula 1.
SigX′=SigX+SigX*([x1]+[x2]+[x3]+[x4])−SigL*[l2]−SigR*[r1]SigU*[u4]−SigD*[d3]  Formula 1

As indicated in Formula 1, in order to correct pixel color mixture, the first characteristic of color mixture information may be set by associating the four color mixture correction coefficients [l2], [r1], [u4], and [d3] with the position (coordinates) of each pixel. Also, the second characteristic of color mixture information may be set by associating the four color mixture correction coefficients [x1], [x2], [x3], and [x4] with the position (coordinates) of each pixel.

Alternatively, from Formula 1, it is possible to gather the coefficients for correcting the signal components that leak out from the pixel X. Therefore,
[x]=[x1]+[x2]+[x3]+[x4]  Formula 2
may be adopted, and the second characteristic of color mixture information set by associating the one color mixture correction coefficient [x] with the position of each pixel.

Alternatively, SigX*[x1] is the signal component that leaks out to the pixel L that is adjacent in the first direction when viewed from the pixel X, and is also the signal component that mixes in from the pixel X when viewed from the pixel L. Both signal components are theoretically equivalent. Therefore, the second characteristic of color mixture information may be set by associating the four color mixture correction coefficients [x1], [x2], [x3], and [x4] with the position of each pixel, and the first characteristic of color mixture information may be derived from the second characteristic of color mixture information. The signal processing circuit7, when performing color mixture correction processing, for the position of each pixel, associates the four color mixture correction coefficients [l2], [r1], [u4], and [d3] with the four color mixture correction coefficients [x1], [x2], [x3], and [x4] of the four adjacent pixels, related to each of those four pixels, and uses these color mixture correction coefficients [l2], [r1], [u4], and [d3] for the first characteristic of color mixture information. That is, a coefficient table in which the four color mixture correction coefficients [x1], [x2], [x3], and [x4] are associated with the position of each pixel may be used as the second coefficient table, and the first coefficient table may be derived from the second coefficient table.

Alternatively, conversely, the first characteristic of color mixture information may be set by associating the four color mixture correction coefficients [l2], [r1], [u4], and [d3] with the position of each pixel, and the second characteristic of color mixture information may be derived from the first characteristic of color mixture information. The signal processing circuit7, when performing color mixture correction processing, for the position of each pixel, associates the four color mixture correction coefficients [x1], [x2], [x3], and [x4] with the four color mixture correction coefficients [l2], [r1], [u4], and [d3] of the four adjacent pixels, related to each of those four pixels, and uses these color mixture correction coefficients [x1], [x2], [x3], and [x4] for the second characteristic of color mixture information. That is, a coefficient table in which the four color mixture correction coefficients [l2], [r1], [u4], and [d3] are associated with the position of each pixel may be used as the first coefficient table, and the second coefficient table may be derived from the first coefficient table.

This color mixture correction coefficient can be calculated by measuring, in advance, the signal output of the pixel of interest and pixels adjacent to that pixel, using a light source with a very constricted (small) irradiation angle to irradiate light on a single pixel of interest of the image sensor. Here, the ratio of signal leak out from the pixel X or pixels adjacent to that pixel changes depending on the incidence angle component of the light ray on each pixel, so when measuring, it is necessary to measure at a plurality of irradiation angles while changing the angle of irradiated light relative to the single pixel of interest.

In addition to the relationship of the incidence angle of light relative to the pixel X and the respective signal amounts of the pixel X and the pixels adjacent to that pixel, the color mixture correction coefficient is calculated also based on the angle of a light ray incident on the image sensor from the optical system of the image sensing apparatus and the positional relationship of each pixel on the image sensor with the optical center of the image sensing apparatus. That is, it is possible to create a coefficient table in which the position of each pixel in the pixel array is associated with the 8 color mixture correction coefficients [x1], [x2], [x3], [x4], [l2], [r1], [u4], and [d3].

Alternatively, the color mixture correction coefficient may be determined based on a theoretical value that has been calculated by a simulation. A theoretical color mixture correction coefficient may be calculated by a simulation, from the positional relationship of each pixel on the image sensor with the optical center of the image sensing apparatus, the incidence angle of a light ray on each pixel, pixel cell size, pixel pitch, color filter height, the internal structure of the image sensor, and so forth.

Next is a description of the relationship of the position of pixels in the pixel array and the color mixture correction coefficient, with reference toFIGS. 2 and 4.FIG. 2shows the position of each pixel in the pixel array, including pixels Xc and X1to X4and pixels adjacent thereto.FIG. 4shows light respectively incident on pixels X1and X2and adjacent pixels in a direction along that row.

As shown inFIGS. 2 and 4, in the pixel X1disposed on the left side relative to the optical center PC in the pixel array PA300, in comparison to a pixel Xc positioned at the optical center PC, the coefficients [x1] and [r1] are large, and conversely the coefficients [x2] and [l2] are small. On the other hand, in the pixel X1, the coefficients [x3], [x4], [u4], and [d3] are the same as in the pixel Xc positioned at the optical center PC.

As shown inFIGS. 2 and 4, in the pixel X2disposed on the right side relative to the optical center PC in the pixel array PA300, in comparison to the pixel Xc positioned at the optical center PC, the coefficients [x2] and [l2] are large, and conversely the coefficients [x1] and [r1] are small. On the other hand, in the pixel X2, the coefficients [x3], [x4], [u4], and [d3] are the same as in the pixel Xc positioned at the optical center PC.

As shown inFIG. 2, in the pixel X3disposed above the optical center PC in the pixel array PA300, in comparison to the pixel Xc positioned at the optical center PC, the coefficients [x3] and [d3] are large, and conversely the coefficients [x4] and [u4] are small. On the other hand, in the pixel X3, the coefficients [x1], [x2], [l2], and [r1] are the same as in the pixel Xc positioned at the optical center PC.

As shown inFIG. 2, in the pixel X4disposed below relative to the optical center PC in the pixel array PA300, in comparison to the pixel Xc positioned at the optical center PC, the coefficients [x4] and [u4] are large, and conversely the coefficients [x3] and [d3] are small. On the other hand, in the pixel X4, the coefficients [x1], [x2], [l2], and [r1] are the same as in the pixel Xc positioned at the optical center PC.

In this way, the color mixture correction coefficient depends on the positional relationship of each pixel in the pixel array PA300with the optical center PC, and has a different value for each pixel. Therefore, in order to increase the accuracy of color mixture correction of the image signal throughout the entire screen, it is possible to adopt a coefficient table in which, for all pixels, the 8 coefficients [x1], [x2], [x3], [x4], [l2], [r1], [u4], and [d3] are associated with the position of each pixel. Alternatively, a coefficient table may be adopted in which the 5 coefficients [x], [l2], [r1], [u4], and [d3] are associated with the position of each pixel. Alternatively, a coefficient table may be adopted in which the 4 coefficients [x1], [x2], [x3], and [x4] are associated with the position of each pixel. Alternatively, a coefficient table may be adopted in which the 4 coefficients [l2], [r1], [u4], and [d3] are associated with the position of each pixel.

In the present embodiment, an image sensor employing a Bayer array is described, but when carrying out the present invention, the color filter array in pixels is not limited to being a Bayer array. Also, in the present embodiment, each pixel is made to have respective correction coefficients for color mixture that occurs between each pixel and pixels that are adjacent in the up, down, left, and right directions, but each pixel may be made to have a correction coefficient for pixels that are adjacent in a diagonal direction. Furthermore, each pixel may be made to have a correction coefficient not only for correcting color mixture that occurs between each pixel and adjacent pixels, but also for correcting color mixture that occurs between each pixel and pixels that are separated by a predetermined number of pixels from a pixel of interest.

Next is a description of an image sensing apparatus100iaccording to a second embodiment of the present invention. Below, mainly portions that differ from the first embodiment will be described.

As shown inFIG. 5, a non-volatile memory14iof the image sensing apparatus100istores, as the first characteristic of color mixture information, instead of the first coefficient table, a first row correction data RD1and a first column correction data CD1. The first row correction data RD1is data in which a position in the direction along a row (horizontal direction) in the pixel array PA300is associated with a coefficient that has been determined in advance so as to correct a signal component that mixes into a pixel in a row that includes the optical center PC of the pixel array PA300from an adjacent pixel (a pixel adjacent in the direction along a column to the pixel for correction). The first column correction data CD1is data in which a position in the direction along a column (vertical direction) in the pixel array PA300is associated with a coefficient that has been determined in advance so as to correct a signal component that mixes into a pixel in a column that includes the optical center PC of the pixel array PA300from an adjacent pixel (a pixel adjacent in the direction along a row to the pixel for correction).FIG. 5shows the first characteristic of color mixture information and the second characteristic of color mixture information in the second embodiment of the present invention.

Also, as shown inFIG. 5, the non-volatile memory14istores, as the second characteristic of color mixture information, instead of the second coefficient table, a second row correction data RD2and a second column correction data CD2. The second row correction data RD2is data in which a position in the direction along a row (horizontal direction) in the pixel array PA300is associated with a coefficient that has been determined in advance so as to correct a signal component that leaks out from a pixel in a row that includes the optical center PC of the pixel array PA300to an adjacent pixel (a pixel adjacent in the direction along a column to the pixel for correction). The second column correction data CD2is data in which a position in the direction along a column (vertical direction) in the pixel array PA300is associated with a coefficient that has been determined in advance so as to correct a signal component that leaks out from a pixel in a column that includes the optical center PC of the pixel array PA300to an adjacent pixels (a pixel adjacent in the direction along a row to the pixel for correction).

A signal processing circuit7iof the image sensing apparatus100icalculates the first correction coefficients [r1] and [l2] for the direction along a row, according to the position in the direction along a row of the pixel for correction in the pixel array PA300and the first row correction data RD1. The signal processing circuit7icalculates the first correction coefficients [u4] and [d3] for the direction along a column, according to the position in the direction along a column of the pixel for correction in the pixel array PA300and the first column correction data CD1.

Also, the signal processing circuit7iof the image sensing apparatus100icalculates the second correction coefficients [x1] and [x2] for the direction along a row, according to the position in the direction along a row of the pixel for correction in the pixel array PA300and the second row correction data RD2. The signal processing circuit7icalculates the second correction coefficients [x4] and [x3] for the direction along a column, according to the position in the direction along a column of the pixel for correction in the pixel array PA300and the second column correction data CD2.

In this way, the non-volatile memory14istores, as the first characteristic of color mixture information, instead of the first coefficient table which is two-dimensional data, the first row correction data RD1and the first column correction data CD1, which are each one-dimensional data. Also, the non-volatile memory14istores, as the second characteristic of color mixture information, instead of the second coefficient table which is two-dimensional data, the second row correction data RD2and the second column correction data CD2, which are each one-dimensional data. Thus, it is possible to reduce the data amount of the first characteristic of color mixture information and the data amount of the second characteristic of color mixture information.

For example, when determining the color mixture correction coefficient, the aperture diameter of the stop of the lens in the optical system of the image sensing apparatus is a factor affecting the angle of a light ray incident on the image sensor from the optical system of the image sensing apparatus. Therefore, in order to improve the accuracy of color mixture correction of the image signal even when the optical system changes, it is necessary to change the color mixture correction coefficient for each F value of the stop of the lens. Here, the image sensing apparatus stores, for each lens stop F value, for all pixels of the image sensor, characteristic of color mixture information that includes a plurality of color mixture correction coefficients that differ by pixel, and thus the data amount of the characteristic of color mixture information greatly increases.

On the contrary, in the present embodiment, attention is focused on the fact that the amount of color mixture is determined by the distance from the optical center in the pixel array. Based on the color mixture correction coefficients for all pixels, the respective color mixture correction coefficients [x1], [x2], [l2], and [r1] of each pixel in a pixel row to which a pixel disposed in the optical center in the pixel array belongs are converted to one-dimensional data in the horizontal direction that corresponds to [x1], [x2], [l2], and [r1]. Also, the respective color mixture correction coefficients [x3], [x4], [u4], and [d3] of each pixel in a pixel column to which a pixel disposed in the optical center in the pixel array belongs are converted to one-dimensional data in the vertical direction that corresponds to [x3], [x4], [u4], and [d3]. A configuration may be adopted in which the non-volatile memory14istores color mixture correction data (one-dimensional data) instead of the color mixture correction coefficients, and the signal processing circuit7iperforms correction by deriving color mixture correction coefficients corresponding to each pixel based on that data.

As shown inFIG. 6, the non-volatile memory14istores a table of one-dimensional data and stop F values. From this table, the signal processing circuit7iselects color mixture correction data (one-dimensional data) that corresponds to the F value that matches the shooting condition, obtains color mixture correction coefficients that correspond to the pixel for correction, and thus can perform signal correction.FIG. 6shows a table of one-dimensional data and stop F values in the second embodiment of the present invention.

Alternatively, for example, when determining the color mixture correction coefficient, the exit pupil distance of the lens is a factor affecting the angle of a light ray incident on the image sensor from the optical system of the image sensing apparatus. Therefore, in order to favorably correct color mixture of the image signal even when the optical system changes, it is necessary to change the color mixture correction coefficient for each exit pupil distance of the lens. Here, the image sensing apparatus has, for each exit pupil distance of the lens, for all pixels of the image sensor, a plurality of color mixture correction coefficients that differ by pixel, and thus the data amount of the characteristic of color mixture information greatly increases.

On the contrary, in the present embodiment, as shown inFIG. 7, the non-volatile memory14istores a table of one-dimensional data and exit pupil distances. From this table, the signal processing circuit7iselects color mixture correction data (one-dimensional data) that corresponds to the exit pupil distance that matches the shooting condition, obtains color mixture correction coefficients that correspond to the pixel for correction, and thus can perform signal correction.FIG. 7shows a table of one-dimensional data and exit pupil distances in the second embodiment of the present invention.

Alternatively, for example, when determining the color mixture correction coefficient, when the optical system further includes a zoom lens in addition to the lens1a(seeFIG. 1), the zoom position of the zoom lens is a factor affecting the angle of a light ray incident on the image sensor from the optical system of the image sensing apparatus. The stop and exit pupil distance of the lens1a(seeFIG. 1) change due to changing the zoom position of the zoom lens. Therefore, in order to improve the accuracy of color mixture correction of the image signal even when the operating state of the optical system changes, it is necessary to change the color mixture correction coefficient for each zoom position of the lens. Here, the image sensing apparatus has, for each zoom position of the lens, for all pixels of the image sensor, a plurality of color mixture correction coefficients that differ by pixel, and thus the data amount of the characteristic of color mixture information greatly increases.

On the contrary, in the present embodiment, the signal processing circuit7icalculates the stop F value and exit pupil distance that match the zoom position of the lens. The signal processing circuit7iselects color mixture correction data (one-dimensional data) that corresponds to the calculated F value from the table shown inFIG. 6, and selects color mixture correction data (one-dimensional data) that corresponds to the calculated exit pupil distance from the table shown inFIG. 7. The signal processing circuit7iadds the correction coefficient for that stop and the correction coefficient for that exit pupil distance, and then can perform correction of the signal of the pixel for correction.

The non-volatile memory14imay store a table (not shown) of one-dimensional data and zoom positions. In this case, from this table, the signal processing circuit7iselects color mixture correction data (one-dimensional data) that corresponds to the zoom position that matches the shooting condition, obtains color mixture correction coefficients that correspond to the pixel for correction, and thus can perform signal correction.

Alternatively, for example, when determining the color mixture correction coefficient, when the present invention is applied to a lens-swappable image sensing apparatus or the like, the type of lens that can be mounted is a factor affecting the angle of a light ray incident on the image sensor from the optical system of the image sensing apparatus. The stop and exit pupil distance of the lens1a(seeFIG. 1) change due to changing the type of lens that is mounted. Therefore, in order to improve the accuracy of color mixture correction of the image signal even when the optical system changes, it is necessary to change the color mixture correction coefficient for each type of lens that is mounted. Here, the image sensing apparatus has, for each type of lens that is mounted, for all pixels of the image sensor, a plurality of color mixture correction coefficients that differ by pixel, and thus the data amount of the characteristic of color mixture information greatly increases.

On the contrary, in the present embodiment, the signal processing circuit7icalculates the stop F value and exit pupil distance that match the type of lens that is actually mounted. The signal processing circuit7iselects color mixture correction data (one-dimensional data) that corresponds to the calculated F value from the table shown inFIG. 6, and selects color mixture correction data (one-dimensional data) that corresponds to the calculated exit pupil distance from the table shown inFIG. 7. The signal processing circuit7iadds the correction coefficient for that stop and the correction coefficient for that exit pupil distance, and then can perform correction of the signal of the pixel for correction.

The non-volatile memory14imay store a table (not shown) of one-dimensional data and lens types. In this case, from this table, the signal processing circuit7iselects color mixture correction data (one-dimensional data) that corresponds to the type of lens that is actually mounted, obtains color mixture correction coefficients that correspond to the pixel for correction, and thus can perform signal correction.

Alternatively, for example, when determining the color mixture correction coefficient, the color of light that is incident on the photoelectric conversion unit is a factor affecting the angle of a light ray incident on the image sensor from the optical system of the image sensing apparatus. Therefore, in order to improve the accuracy of color mixture correction of the image signal even when the optical system changes, it is necessary to change the color mixture correction coefficient for each color of light that is incident on the photoelectric conversion unit. Here, the image sensing apparatus has, for each color of light that is incident on the photoelectric conversion unit, for all pixels of the image sensor, a plurality of color mixture correction coefficients that differ by pixel, and thus the data amount of the characteristic of color mixture information greatly increases.

On the contrary, in the present embodiment, the non-volatile memory14istores a table of one-dimensional data and light colors. From this table, the signal processing circuit7iselects correction data (row correction data or column correction data that is one-dimensional data) as shown inFIG. 8that corresponds to the color of the color filter of the pixel for correction. For example, the signal processing circuit7iselects correction data ‘i’ shown inFIG. 8when the pixel for correction is an R pixel (when the color of the color filter is red). For example, the signal processing circuit7iselects correction data shown inFIG. 8when the pixel for correction is a G pixel (when the color of the color filter is green). For example, the signal processing circuit7iselects correction data ‘iii’ shown inFIG. 8when the pixel for correction is a B pixel (when the color of the color filter is blue). Thus, the signal processing circuit7iobtains the color mixture correction coefficients that correspond to the pixel for correction, and can perform signal correction.FIG. 8shows a correction data of each color in the second embodiment of the present invention.

Next is a description of an image sensing apparatus according to a third embodiment of the present invention. Below, portions that differ from the first embodiment will be mainly described.

The non-volatile memory of the image sensing apparatus according to the present embodiment, although not shown, as first characteristic of color mixture information, stores a first relational expression instead of a first coefficient table. The first relational expression is a formula that expresses the relationship between the position in the pixel array and the first correction coefficient that has been predetermined so as to correct a signal component that mixes into a pixel from an adjacent pixel.

Also, the non-volatile memory, although not shown, as second characteristic of color mixture information, stores a second relational expression instead of a second coefficient table. The second relational expression is a formula that expresses the relationship between the position in the pixel array and the second correction coefficient that has been predetermined so as to correct a signal component that leaks out from a pixel to an adjacent pixel.

The signal processing circuit of the image sensing apparatus calculates first correction coefficients [r1], [l2], [u4], and [d3] according to the position of the pixel for correction in the pixel array and the first relational expression.

Also, the signal processing circuit of the image sensing apparatus calculates second correction coefficients [x1], [x2], [x3], and [x4] according to the position of the pixel for correction in the pixel array and the second relational expression.

In this way, the non-volatile memory stores, as the first characteristic of color mixture information, instead of the first coefficient table which is 2-dimensional data, the first relational expression which is 0-dimensional data. Also, the non-volatile memory stores, as the second characteristic of color mixture information, instead of the second coefficient table which is 2-dimensional data, the second relational expression which is 0-dimensional data. Thus, it is possible to further reduce the amount of data of the first characteristic of color mixture information and the amount of data of the second characteristic of color mixture information.

The non-volatile memory of the image sensing apparatus may store, as the first characteristic of color mixture information, instead of the first relational expression, a first row relational expression and a first column relational expression. The first row relational expression is a formula that expresses the relationship of a position in the direction along a row (horizontal direction) in the pixel array with a coefficient that has been determined in advance so as to correct a signal component that mixes into a pixel in a row that includes the optical center of the pixel array from adjacent pixels (pixels adjacent in the direction along a column). The first column relational expression is a formula that expresses the relationship of a position in the direction along a column (vertical direction) in the pixel array with a coefficient that has been determined in advance so as to correct a signal component that mixes into a pixel in a column that includes the optical center of the pixel array from adjacent pixels (pixels adjacent in the direction along a row).

Also, the non-volatile memory stores, as the second characteristic of color mixture information, instead of the second relational expression, a second row relational expression and a second column relational expression. The second row relational expression is a formula that expresses the relationship of a position in the direction along a row (horizontal direction) in the pixel array with a coefficient that has been determined in advance so as to correct a signal component that leaks out from a pixel in a row that includes the optical center of the pixel array to an adjacent pixel (a pixel adjacent in the direction along a column to the pixel for correction). The second column relational expression is a formula that expresses the relationship of a position in the direction along a column (vertical direction) in the pixel array with a coefficient that has been determined in advance so as to correct a signal component that leaks out from a pixel in a column that includes the optical center of the pixel array to an adjacent pixel (a pixel adjacent in the direction along a row to the pixel for correction).

The signal processing circuit of the image sensing apparatus calculates the first correction coefficients [r1] and [l2] for the direction along a row, according to the position in the direction along a row of the pixel for correction in the pixel array and the first row relational expression. The signal processing circuit calculates the first correction coefficients [u4] and [d3] for the direction along a column, according to the position in the direction along a column of the pixel for correction in the pixel array and the first column relational expression.

Also, the signal processing circuit of the image sensing apparatus calculates the second correction coefficients [x1] and [x2] for the direction along a row, according to the position in the direction along a row of the pixel for correction in the pixel array and the second row relational expression. The signal processing circuit calculates the second correction coefficients [x3] and [x4] for the direction along a column, according to the position in the direction along a column of the pixel for correction in the pixel array and the second column relational expression.

Next is a description of an image sensing apparatus100jaccording to a fourth embodiment of the present invention. Below, portions that differ from the first embodiment will be mainly described.

The image sensing apparatus100jis provided with an image sensor3jand a signal processing circuit7j.

As shown inFIG. 9, the image sensor3jincludes a readout unit31j. The readout unit31j, in a first mode (whole-screen mode), reads out a signal from all of the pixels in the pixel array PA300, and in a second mode (sub-sampling mode), reads out a signal from a portion of the pixels in the pixel array PA300. For example, the readout unit31j, in the second mode (sub-sampling mode), reads out a signal from pixels indicated in black inFIG. 9.FIG. 9shows the configuration of the image sensor3jin the fourth embodiment of the present invention.

In this way, in the sub-sampling mode in which pixels are sub-sampled from the pixel array PA300and an image signal is only read from necessary predetermined pixels, a signal is not read from adjacent pixels in the pixel array PA300, so it is not possible to correct the signal of adjacent pixels. Therefore, in the second mode (sub-sampling mode), correction is performed in the following manner, while considering the output level of a signal that is adjacent in the read out image signal to be approximately the same as the output level of the signal of adjacent pixels that was not read out.

The signal processing circuit7j, in the second mode, using pixels adjacent to the pixel for correction in an image signal of one frame that has been read out by the readout unit31jas the adjacent pixels of the pixel for correction in the pixel array PA300, corrects the signal that has been read out from the pixel for correction. For example, the signal processing circuit7j, when performing color mixture correction processing of a pixel X, corrects the signal of the pixel X using pixels LL, RR, UU, and DD as adjacent pixels of the pixel X.

Specifically, the following correction is performed. A signal that is read out from the pixel X is called SigX, and a signal that is read out from the pixel LL is called SigLL. A signal that is read out from the pixel RR is called SigRR, and a signal that is read out from the pixel UU is called SigUU. A signal that is read out from the pixel DD is called SigDD. As stated in the first embodiment, as color mixture correction coefficients that correspond to the pixel X, coefficients [x1], [x2], [x3], and [x4] that correct components that leak out from the pixel X, and coefficients [l2], [r1], [u4], and [d3] that correct components that leak in from pixels adjacent to the pixel X, are used. The signal processing circuit7jperforms correction by the calculation expressed in Formula 3, thus obtaining a corrected signal SigX′.
SigX′=SigX+SigX*([x1]+[x2]+[x3]+[x4])−SigLL*[l2]−SigRR*[r1]−SigUU*[u4]−SigDD*[d3]  Formula 3

Next is a description of an image sensing apparatus100kaccording to a fifth embodiment of the present invention. Below, portions that differ from the first embodiment will be mainly described.

The image sensing apparatus100kis provided with an image sensor3kand a signal processing circuit7k.

As shown inFIG. 10A, the image sensor3kincludes a readout unit31k. The readout unit31k, in a first mode (whole screen mode), reads out a signal from all of the pixels in the pixel array PA300, and in a third mode (addition mode), performs readout by adding together signals for each instance of at least two pixels of the same color that are positioned near each other in the pixel array PA300. For example, the readout unit31k, in the third mode (addition mode), performs readout by adding together signals from pixels of the same color indicated in black inFIG. 10A.FIG. 10Ashows the configuration and operation of the image sensor3kin the fifth embodiment of the present invention.

Prior to performing color mixture correction in this way, for the pixel signal generated by each pixel of the image sensor, in the addition mode in which an image signal is generated after performing addition for each set of a predetermined number of pixels, the signals of a predetermined number of pixels having the same color on the image sensor are added together and the result is read out. Therefore, the signal of pixels adjacent to the pixel for color mixture correction cannot be used as-is for correction. Consequently, in the third mode (addition mode), correction is performed in the following manner, while considering that the amount of signal leakout to the respective adjacent pixels from the added pixels is about the same as the amount of leakout to adjacent pixels from a pixel positioned at the center of gravity of the added pixels.

The signal processing circuit7k, in the third mode, corrects the signal that has been read out from the pixel for correction by the readout unit31kusing the position of the center of gravity of at least two pixels that are added together as the position of the pixel for correction in the pixel array PA300. For example, when the signal processing circuit7ksums and averages the signals of the pixels indicated in black inFIG. 10Ato obtain a signal SigXX shown inFIG. 10B, the signal SigXX of the pixel for correction is corrected using the position of the pixel X, which is the position of the center of gravity of the pixels indicated in black, as the position of the pixel for correction.

Specifically, the following correction is performed. The signals of the pixels indicated in black, including the pixel X, inFIG. 10Aare summed and averaged to obtain the signal SigXX inFIG. 10B. The signals of the pixels indicated by the bold diagonal pattern, including the pixel L, inFIG. 10Aare summed and averaged to obtain the signal SigLL inFIG. 10B. The signals of the pixels indicated by the diagonal line pattern, including the pixel R, in FIG.10A are summed and averaged to obtain the signal SigRR inFIG. 10B. The signals of the pixels indicated by the horizontal line pattern, including the pixel U, inFIG. 10Aare summed and averaged to obtain the signal SigUU inFIG. 10B. The signals of the pixels indicated by the lattice pattern, including the pixel D, inFIG. 10Aare summed and averaged to obtain the signal SigDD inFIG. 10B. As stated in the first embodiment, as color mixture correction coefficients that correspond to the pixel X, coefficients [x1], [x2], [x3], and [x4] that correct components that leak out from the pixel X, and coefficients [l2], [r1], [u4], and [d3] that correct components that leak in from pixels adjacent to the pixel X, are used. The signal processing circuit7kperforms correction by the calculation expressed in Formula 4, thus obtaining a corrected signal SigXX′.
SigXX′=SigXX+SigXX*([x1]+[x2]+[x3]+[x4])−SigLL*[l2]−SigRR*[r1]−SigUU*[u4]−SigDD*[d3]  Formula 4

Next is a description of an image sensing system400that includes an image sensing apparatus100paccording to a sixth embodiment of the present invention, with reference toFIG. 11.FIG. 11shows the configuration of the image sensing system400including the image sensing apparatus100paccording to the sixth embodiment of the present invention. Below, portions that differ from the first embodiment will be mainly described.

The image sensing system400includes the image sensing apparatus100pand a processing apparatus200. The image sensing apparatus100pis connected to the processing apparatus200via a communications line300so as to be capable of communications. The communications line300, for example, is a wired communications line capable of serial communications such as a serial cable, or a wireless communications line employing Bluetooth or the like. The processing apparatus200, for example, is a personal computer. The image sensing apparatus100psupplies image data to the processing apparatus200via the communications line300. The processing apparatus200receives image data from the image sensing apparatus100p, and processes the received image data.

The image sensing apparatus100pincludes a signal processing circuit (generation unit)7pand a communication interface (I/F)19p.

In the present embodiment, the color mixture correction processing is performed not within the image sensing apparatus, but outside of the image sensing apparatus. When the color mixture correction processing is performed externally, it is necessary to output information that is necessary for performing correction along with the image data.

Therefore, the signal processing circuit7pgenerates image data by associating the position in the pixel array with the image signal of one frame that has been read out by the readout circuit, and also attaching the first characteristic of color mixture information and the second characteristic of color mixture information (seeFIG. 12). The signal processing circuit7psupplies the generated image data to the communication interface19p.

The signal processing circuit7pmay attach to the image signal (image information) color mixture correction coefficients (or, color mixture correction data or a color mixture correction format) that match shooting conditions such as the lens F value, exit pupil distance, or zoom position. Alternatively, the signal processing circuit7pmay attach to the image signal (image information) information related to shooting conditions such as the lens F value, exit pupil distance, or zoom position.

The communication interface19psends the supplied image data to the processing apparatus200via the communications line300.

The processing apparatus200is provided with a communication interface (I/F)202and a correction unit201.

The communication interface202receives image data from the image sensing apparatus100pvia the communications line300. The communication interface202supplies the received image data to the correction unit201.

As shown inFIG. 12, the correction unit201receives image data from the communication interface202. The correction unit201calculates the first correction coefficient for correcting a signal component that mixes into the pixel for correction from adjacent pixels according to the position of the pixel for correction in the pixel array indicated by the image data and the first characteristic of color mixture information. The correction unit201calculates the second correction coefficient for correcting a signal component that leaks out from the pixel for correction to an adjacent pixel according to the position of the pixel for correction in the pixel array indicated by the image data and the second characteristic of color mixture information. The correction unit201corrects the signal of the pixel for correction in the image data using the signals of pixels adjacent to the pixel for correction in the image data, the first correction coefficient, and the second correction coefficient. As shown inFIG. 12, the correction unit201outputs the corrected image signal to a later stage (for example, a recording medium such as a hard disk or a memory card).FIG. 12is a dataflow diagram for the correction unit201in the sixth embodiment of the present invention.

This application claims the benefit of Japanese Patent Application No. 2008-305626, filed Nov. 28, 2008, which is hereby incorporated by reference herein in its entirety.