Patent Description:
Imaging apparatuses that capture images using image sensors are conventionally known. An image obtained by such an imaging apparatus may contain noise depending on the physical characteristics of the image sensor. In view of this, there is conventionally disclosed an electronic camera that, after performing an imaging operation, operates a shutter and performs a dummy imaging operation in a state in which the light receiving surface of an image sensor is shielded from light, and corrects captured image data using dummy imaging data obtained by the dummy imaging operation (see Patent Literature (PTL) <NUM>).

With the method in PTL <NUM>, however, noise cannot be appropriately obtained depending on the noise type.

The present disclosure accordingly relates to an image generation method, an imaging apparatus, and a program that can obtain noise more appropriately than conventional techniques.

To achieve the object stated above, there is provided an image generation method according to a first aspect of the present invention, comprising an image generation method in an imaging apparatus that includes a plurality of pixels, the image generation method including: performing a first imaging operation of capturing an image when each of the plurality of pixels is shielded from light, in a state in which a first reference signal level in the first imaging operation is set to a first offset value, the first offset value corresponding to a first reset voltage, VRST1; and generating first image data based on a first pixel signal obtained by the first imaging operation, wherein the first offset value is set higher than a second offset value, the second offset value corresponding to a second reset voltage, VRST2, that is a second reference signal level in a second imaging operation of capturing an image in a state in which light is incident on each of the plurality of pixels, and performing the second imaging operation; generating second image data based on a second pixel signal obtained by the second imaging operation; and generating third image data by subtracting the first image data from the second image data.

To achieve the object stated above, there is provided an imaging apparatus according to a second aspect of the present invention, said imaging apparatus including: a plurality of pixels; a controller configured to perform control to perform a first imaging operation of capturing an image when each of the plurality of pixels is shielded from light, in a state in which a first reference signal level in the first imaging operation is set to a first offset value, the first offset value corresponding to a first reset voltage, VRST1; and a generator configured to generate image data based on a pixel signal obtained by the first imaging operation, wherein the controller is further configured to control the first offset value to be higher than a second offset value, the second offset value corresponding to a second reset voltage, VRST2, that is a second reference signal level in a second imaging operation of capturing an image in a state in which light is incident on each of the plurality of pixels, and, is further configured to perform the second imaging operation; and the generator is further configured to generate second image data based on a second pixel signal obtained by the second imaging operation; and the imaging apparatus further comprises a signal processor configured to generate third image data by subtracting the first image data from the second image data.

To achieve the object stated above, there is provided a program according to a third aspect of the present invention, said program causing a computer to execute the above-described image generation method.

These general and specific aspects may be implemented using a system, a method, an integrated circuit, a computer program, or a computer-readable non-transitory recording medium such as CD-ROM, or any combination of a system, a method, an integrated circuit, a computer program, and a recording medium. The program may be stored in the recording medium beforehand, or supplied to the recording medium via a wide area communication network such as the Internet.

With the image generation method, etc. according to an aspect of the present disclosure, noise can be obtained more appropriately than conventional techniques.

Operation when performing correction in an imaging apparatus according to a comparative example will be described below, with reference to <FIG> and <FIG>. <FIG> is a first schematic diagram explaining operation when performing correction in the imaging apparatus according to the comparative example. <FIG> illustrates the case where, from among white flaws and black flaws, only white flaws occur in imaging by long exposure.

In the specification, a white flaw is noise in which a signal level higher than an actual level is detected due to a pixel defect and that grows (increases) with exposure time. In other words, a white flaw is noise of a positive value in which a signal level higher than an actual level is detected when the exposure time is longer. A black flaw is noise in which a signal level lower than an actual level is detected due to a pixel defect and that grows (increases) with exposure time. In other words, a black flaw is noise of a negative value in which a signal level lower than an actual level is detected when the exposure time is longer.

The imaging apparatus according to the comparative example is capable of executing a first imaging operation of performing imaging in a state in which a shutter is closed and a second imaging operation of performing imaging in a state in which the shutter is open. The first imaging operation is, for example, an imaging operation performed to generate light shielding data for removing noise contained in image data generated by the second imaging operation. For example, the first imaging operation is performed following the second imaging operation. The first imaging operation is performed under the same exposure condition (for example, exposure time) as the second imaging operation. In the first imaging operation, light shielding exposure that involves exposure (exposure operation) in a light shielding state is performed. In the first imaging operation image data (light shielding data) including only noise of the imaging apparatus is obtained.

The second imaging operation is an imaging operation performed to capture an image of a subject and generate image data. For example, the second imaging operation is performed using long exposure, i.e. exposure for a time longer than a predetermined time, and is performed with the shutter being open for <NUM> sec or more. In addition to this, the imaging apparatus according to the comparative example may perform normal exposure, i.e. exposure for the predetermined time, in a state in which the shutter is open. The normal exposure is exposure for capturing an image of the subject and generating image data, for example, exposure performed with the shutter being open for a period shorter than <NUM> sec. The second imaging operation may be performed using the normal exposure.

In <FIG>, the vertical axis represents the signal level (pixel value), and the horizontal axis represents the pixel position. For example, the respective signal levels of a plurality of pixels arranged in one line are illustrated in <FIG>. The signal illustrated in <FIG> is image data obtained by AD conversion of an analog signal from the plurality of pixels.

(a) in <FIG> illustrates image data (long-exposure still image data) obtained by the second imaging operation (for example, long exposure). As illustrated in (a) in <FIG>, the image data obtained by the long exposure contains signal components according to the brightness of the subject and noise components due to white flaws according to the exposure time. In the long exposure, such white flaws are noticeable. Although the image is a still image in this example, the image may be a moving image.

(b) in <FIG> illustrates image data (long light-shielding exposure data) obtained by the first imaging operation. As illustrated in (b) in <FIG>, the image data obtained by the first imaging operation contains white flaws according to the exposure time.

(c) in <FIG> illustrates image data (still image data after noise subtraction) obtained by a process of removing noise from the image data illustrated in (a) in <FIG>. Specifically, the image data illustrated in (c) in <FIG> is image data obtained by subtracting the image data illustrated in (b) in <FIG> from the image data illustrated in (a) in <FIG> and adding an offset value to the subtraction outcome. As illustrated in (c) in <FIG>, the white flaws are removed as a result of this process. The offset value is set, for example, in order to remove noise such as dark noise (for example, thermal noise). For example, the offset value is a reference signal level in the second imaging operation. The offset value is, for example, a value (signal level) when subtracting a certain offset signal component from image data after digital conversion.

Here, depending on the physical characteristics and the like of a photoelectric conversion element included in the imaging apparatus, both white flaws and black flaws may occur. The following will describe the case of performing the same noise removal as above in the imaging apparatus including such a photoelectric conversion element, with reference to <FIG> is a second schematic diagram explaining operation when performing correction in the imaging apparatus according to the comparative example. In <FIG>, the vertical axis represents the signal level, and the horizontal axis represents the pixel position.

(a) in <FIG> illustrates image data obtained by the second imaging operation. As illustrated in (a) in <FIG>, the image data obtained by the second imaging operation contains signal components according to the brightness of the subject and noise components due to white flaws and black flaws according to the exposure time.

(b) in <FIG> illustrates image data obtained by the first imaging operation. As illustrated in (b) in <FIG>, the image data obtained by the first imaging operation contains white flaws and black flaws according to the exposure time. There is, however, an undetected component in the black flaws. That is, the imaging apparatus according to the comparative example cannot appropriately obtain noise (the black flaws in the example in (b) in <FIG>). This phenomenon occurs because the signal level of the offset value is smaller than the signal level of the black flaw (i.e. the absolute value of the black flaw). Although an example in which the image data obtained by the first imaging operation from among the respective image data obtained by the first imaging operation and the second imaging operation contains an undetected component is illustrated in <FIG>, an undetected component may also be contained in the image data obtained by the second imaging operation.

(c) in <FIG> illustrates image data obtained by a process of removing noise from the image data illustrated in (a) in <FIG>. Specifically, the image data illustrated in (c) in <FIG> is image data obtained by subtracting the image data illustrated in (b) in <FIG> from the image data illustrated in (a) in <FIG> and adding an offset value to the subtraction outcome. As illustrated in (c) in <FIG>, the white flaws are removed, but part of the black flaws is unable to be removed.

Thus, with the noise removal method according to the comparative example, in the case where not only white flaws but also black flaws occur, the black flaws may not be appropriately removed. The present inventors accordingly conducted intensive study on how, in the case where both white flaws and black flaws occur, to appropriately obtain both the white flaws and the black flaws. The present inventors then found out that the problem can be solved by adjusting the offset value in the first imaging operation. This will be described in detail below.

An image generation method, an imaging apparatus, and a program according to the present disclosure will be described in detail below, with reference to drawings. The embodiments described below each show a preferable specific example of the present disclosure. The numerical values, shapes, materials, structural elements, the arrangement and connection of the structural elements, steps, the processing order of the steps etc. shown in the following embodiments are mere examples.

The accompanying drawings and the following description are provided to help a person skilled in the art to fully understand the present disclosure. Each drawing is a schematic and does not necessarily provide precise depiction.

In the specification, the terms indicating the relationships between elements, such as "equal", the numerical values, and the numerical ranges are not expressions of strict meanings only, but are expressions of meanings including substantially equivalent ranges, for example, a difference of about several percent.

An embodiment will be described below, with reference to <FIG>.

The structures of imaging apparatus <NUM> and camera <NUM> including imaging apparatus <NUM> according to this embodiment will be described below, with reference to <FIG> and <FIG>. <FIG> is a block diagram illustrating the structure of camera <NUM> including imaging apparatus <NUM> according to this embodiment. <FIG> is a block diagram illustrating the structure of imaging apparatus <NUM> according to this embodiment. <FIG> is a block diagram illustrating the structure of solid-state imaging device <NUM> in detail. Shutter <NUM> is not illustrated in <FIG>.

As illustrated in <FIG>, camera <NUM> according to this embodiment includes imaging apparatus <NUM>, lens <NUM>, display <NUM>, and operation portion <NUM>.

Imaging apparatus <NUM> captures an image of a subject according to operation (input) by a user, and performs predetermined signal processing on the captured image data. Imaging apparatus <NUM> performs long exposure, light shielding exposure, and normal exposure. The long exposure and the normal exposure are each exposure performed in a state in which a subject image is formed on an imaging element (the below-described organic photoelectric conversion element in this embodiment) by lens <NUM> or the like. The long exposure and the normal exposure are performed to obtain image data of a captured image of the subject. The long exposure has a longer exposure time than the normal exposure, and is performed, for example, when capturing an image of a nightscape, night sky, and the like. The normal exposure has a shorter exposure time than the long exposure, i.e. has a normal exposure time, and is performed, for example, when performing normal imaging. In the long exposure and the normal exposure, a reference signal level in imaging is set to the below-described second offset value.

The light shielding exposure is exposure performed in a state in which the formation of a subject image is blocked by shutter <NUM> or the like. The light shielding exposure is performed to remove noise (noise including white flaws and black flaws) from image data obtained by the long exposure or the like, or obtain image data for identifying a defective pixel. For example, in the case of removing noise from image data obtained by the long exposure, the light shielding exposure is performed for the same exposure time as the long exposure. In the light shielding exposure, a reference signal level in imaging is set to the below-described first offset value.

The light shielding exposure may be executed only in the case where the long exposure is performed from among the long exposure and the normal exposure. A long exposure mode in which the long exposure is performed and a normal exposure mode in which the normal exposure is performed are switchable by operation portion <NUM>. The long exposure and the normal exposure are an example of transmissive exposure. An example in which the transmissive exposure is the long exposure will be described below.

Imaging apparatus <NUM> includes solid-state imaging device <NUM>, signal processor <NUM>, shutter <NUM>, and controller <NUM>. As illustrated in <FIG>, solid-state imaging device <NUM> further includes pixel array portion <NUM>, column AD converter <NUM>, row scanner <NUM>, column scanner <NUM>, and drive controller <NUM>. In pixel array portion <NUM> and its surrounding region, column signal line <NUM> is provided for each pixel column, and scan line <NUM> is provided for each pixel row.

Pixel array portion <NUM> is an imaging portion in which a plurality of pixels <NUM> are arranged in a matrix.

Column AD converter (analog-digital converter) <NUM> is a converter that digitally converts a signal (analog pixel signal) input from each column signal line <NUM> to obtain, hold, and output a digital value (digital pixel signal) corresponding to the amount of light received by pixel <NUM>.

Row scanner <NUM> controls reset operation, charge accumulation operation, and read operation for pixels <NUM> in units of rows.

Column scanner <NUM> sequentially outputs the digital values of one row held in column AD converter <NUM> to row signal line <NUM>, to output the digital values to signal processor <NUM>.

Drive controller <NUM> supplies various control signals to row scanner <NUM> and column scanner <NUM>, to control these units. For example, drive controller <NUM> supplies various control signals to row scanner <NUM> and column scanner <NUM>, based on control signals from controller <NUM>.

Imaging apparatus <NUM> according to this embodiment is, for example, an imaging apparatus for capturing a still image. Alternatively, imaging apparatus <NUM> may be an imaging apparatus for capturing a moving image.

Imaging apparatus <NUM> may include an interface (not illustrated) for communication between external circuitry and at least one of solid-state imaging device <NUM>, signal processor <NUM>, and controller <NUM>. The interface is, for example, a communication port composed of a semiconductor integrated circuit.

Referring back to <FIG>, lens <NUM> includes a lens system that can be driven in an optical axis direction. As a result of the lens system being driven in the optical axis direction, light from outside of imaging apparatus <NUM> is focused onto pixel array portion <NUM>.

Display <NUM> is a display device capable of displaying images generated by signal processor <NUM>. An example of display <NUM> is a liquid crystal monitor. Display <NUM> is also capable of displaying various configuration information in the camera. For example, display <NUM> can display imaging conditions (aperture, ISO sensitivity, etc.) during imaging.

Operation portion <NUM> is an input unit that receives input from the user. Examples of operation portion <NUM> include a release button and a touch panel. For example, the touch panel is bonded to the liquid crystal monitor. Operation portion <NUM> receives an imaging instruction, a change in imaging conditions, and the like from the user. Operation portion <NUM> may obtain input from the user by voice, gesture, etc..

The structure of solid-state imaging device <NUM> will be described in more detail below, with reference to <FIG> is a diagram illustrating an example of the circuit structure of pixel <NUM> according to this embodiment.

As illustrated in <FIG>, pixel <NUM> includes photoelectric conversion element <NUM>, reset transistor <NUM>, amplification transistor <NUM>, selection transistor <NUM>, and charge accumulator <NUM>.

Photoelectric conversion element <NUM> is a photoelectric converter that photoelectrically converts received light into a signal charge (pixel charge). Specifically, photoelectric conversion element <NUM> is composed of upper electrode 211a, lower electrode 211b, and photoelectric conversion film 211c sandwiched between the two electrodes. Photoelectric conversion film 211c is a film made of a photoelectric conversion material that generates a charge according to received light. In this embodiment, photoelectric conversion film 211c is an organic photoelectric conversion film containing organic molecules having high light absorption function. In other words, in this embodiment, photoelectric conversion element <NUM> is an organic photoelectric conversion element including an organic photoelectric conversion film, and solid-state imaging device <NUM> is an organic sensor using the organic photoelectric conversion element. The organic photoelectric conversion film is formed across the plurality of pixels <NUM>. Each of the plurality of pixels <NUM> includes the organic photoelectric conversion film.

The organic photoelectric conversion film is varied in light transmittance, as a result of the voltage applied to the organic photoelectric conversion film being made variable. That is, a shutter function can be realized by adjusting the voltage applied to the organic photoelectric conversion film. Hence, all of the plurality of pixels <NUM> including the organic photoelectric conversion film can be put in a light shielding state substantially simultaneously. A global shutter can thus be realized without adding elements such as memory. Therefore, it is possible to reduce distortion (rolling distortion) caused by reading by a rolling shutter.

The thickness of photoelectric conversion film 211c is, for example, approximately <NUM>. Photoelectric conversion film 211c is formed using, for example, a vacuum vapor deposition method. The organic molecules have a high light absorption function over an entire visible light range of approximately <NUM> to <NUM> in wavelength.

Photoelectric conversion element <NUM> included in pixel <NUM> according to this embodiment is not limited to being formed by the foregoing organic photoelectric conversion film, and may, for example, be a photodiode made of an inorganic material.

Upper electrode 211a is an electrode opposite to lower electrode 211b, and is formed on photoelectric conversion film 211c so as to cover photoelectric conversion film 211c. That is, upper electrode 211a is formed across the plurality of pixels <NUM>. Upper electrode 211a is made of a transparent conductive material (for example, indium tin oxide (ITO)) in order to allow light to enter photoelectric conversion film 211c.

Lower electrode 211b is an electrode for extracting electrons or holes generated in photoelectric conversion film 211c between lower electrode 211b and upper electrode 211a arranged opposite to each other. Lower electrode 211b is formed for each pixel <NUM>. Lower electrode 211b is made of, for example, Ti, TiN, Ta, or Mo.

Charge accumulator <NUM> is connected to photoelectric conversion element <NUM>, and accumulates a signal charge extracted via lower electrode 211b.

Reset transistor <NUM> has its drain supplied with reset voltage VRST and its source connected to charge accumulator <NUM>, and resets (initializes) the potential of charge accumulator <NUM>. Specifically, as a result of the gate of reset transistor <NUM> being supplied with a predetermined voltage from row scanner <NUM> through reset scan line 170A (i.e. turned on), reset transistor <NUM> resets the potential of charge accumulator <NUM>. As a result of the supply of the predetermined voltage being stopped, the signal charge is accumulated in charge accumulator <NUM> (i.e. exposure is started).

In this embodiment, second reset voltage VRST2 supplied to the drain of reset transistor <NUM> before the long exposure starts and first reset voltage VRST1 supplied to the drain of reset transistor <NUM> before the light shielding exposure starts are different voltages.

Amplification transistor <NUM> has its gate connected to charge accumulator <NUM> and its drain supplied with power voltage VDD, and outputs an analog pixel signal corresponding to the amount of signal charge accumulated in charge accumulator <NUM>.

Selection transistor <NUM> has its drain connected to the source of amplification transistor <NUM> and its source connected to column signal line <NUM>, and determines the timing of outputting the analog pixel signal from amplification transistor <NUM>. Specifically, as a result of a predetermined voltage being supplied from row scanner <NUM> to the gate of selection transistor <NUM> through selection scan line 170B, the analog pixel signal is output from amplification transistor <NUM>.

Pixel <NUM> having the structure described above can be read non-destructively. Herein, "non-destructive reading" denotes reading, without destroying the charge (signal charge) accumulated in charge accumulator <NUM>, an analog pixel signal corresponding to the amount of charge during exposure. Herein, "during exposure" denotes any timing in an exposure period.

Column AD converter <NUM> is composed of AD converters <NUM> provided for respective column signal lines <NUM>. Each AD converter <NUM> is, for example, a <NUM>-bit AD converter. For example, each AD converter <NUM> digitally converts an analog pixel signal output from pixel <NUM> by ramp method, to output a digital value corresponding to the amount of light received by pixel <NUM>. AD converter <NUM> includes a comparator and an up/down counter (not illustrated).

Herein, "AD conversion by ramp method" is AD conversion using a ramp wave, i.e. a method whereby, when an analog pixel signal (input signal) is input, a ramp wave whose voltage increases in a certain slope is caused to rise, the time from the point of rise to when both signals (input signal and ramp wave) match in voltage is measured, and the measured time is output as a digital value. The comparator compares the voltage of the analog pixel signal (input signal) and the voltage of the reference signal input as the ramp wave, and outputs a signal indicating the timing at which the voltage of the reference signal matches the voltage of the column signal.

The up/down counter counts down (or counts up) in the period from when the reference signal is input to the comparator to when the reference signal reaches the voltage of the analog pixel signal indicating a base component, and then counts up (or counts down) in the period from when the reference signal is input to the comparator to when the reference signal reaches the voltage of the analog pixel signal indicating a signal component. In this way, the up/down counter eventually holds a digital pixel signal corresponding to the difference obtained by subtracting the base component from the signal component of the analog pixel signal. The analog pixel signal indicating the base component is a pixel signal output from pixel <NUM> to AD converter <NUM> at a reset level (for example, second offset value). The analog pixel signal indicating the signal component is a pixel signal output from pixel <NUM> to AD converter <NUM> when a charge is accumulated as a result of exposure operation.

The digital values held in the respective up/down counters are sequentially output to row signal line <NUM>, and output to signal processor <NUM> via an output circuit (such as an output buffer (not illustrated)).

Drive controller <NUM> controls row scanner <NUM> and column scanner <NUM>, to control reset operation, charge accumulation operation, and read operation in each pixel <NUM> or digital pixel signal output operation from each AD converter <NUM> to signal processor <NUM>.

For example, upon receiving a read instruction from controller <NUM>, drive controller <NUM> controls row scanner <NUM> to apply a predetermined voltage sequentially to selection scan lines 170B to output analog pixel signals to AD converter <NUM>. Drive controller <NUM> also controls column scanner <NUM> to sequentially output digital pixel signals held in AD converters <NUM> to signal processor <NUM>.

Signal processor <NUM> will be described below, with reference to <FIG>.

Signal processor <NUM> performs a process of subjecting each digital pixel signal obtained from solid-state imaging device <NUM> to predetermined signal processing to generate image data and storing and outputting the generated image data. For example, signal processor <NUM> outputs the generated image data to display <NUM>. For example, signal processor <NUM> stores the image data in storage <NUM> or an external storage device (for example, USB memory).

As illustrated in <FIG>, signal processor <NUM> includes generator <NUM>, first determiner <NUM>, first corrector <NUM>, second corrector <NUM>, second determiner <NUM>, and storage <NUM>. First determiner <NUM> and first corrector <NUM> are provided to execute a process of removing white flaws and black flaws from image data obtained by the second imaging operation. Second corrector <NUM> and second determiner <NUM> are provided to execute a process (pixel refresh) of detecting defective pixels. The term "defective pixel" denotes, for example, a pixel having white flaws or black flaws mentioned above.

Generator <NUM> is a processor that performs predetermined signal processing on each digital pixel signal obtained from solid-state imaging device <NUM> to generate image data. For example, generator <NUM> generates image data (an example of first image data) based on a digital pixel signal (an example of a first pixel signal) obtained by the first imaging operation, and generates image data (an example of second image data) based on a digital pixel signal (an example of a second pixel signal) obtained by the second imaging operation.

First determiner <NUM> is a processor that determines whether the second image data generated by generator <NUM> contains any underexposed or overexposed pixel. In this embodiment, first determiner <NUM> performs the determination only for defective pixels from among pixels <NUM> of defective pixels and non-defective pixels.

First corrector <NUM> is a processor that performs a noise removal process of correcting the second image data to remove white flaws and black flaws. First corrector <NUM> generates third image data from which white flaws and black flaws have been removed, based on the second image data and the first image data. Specifically, based on the first image data and the second image data obtained from generator <NUM> and the determination result obtained from first determiner <NUM>, first corrector <NUM> subtracts the first image data from the second image data to generate the third image data.

Second corrector <NUM> is a processor that performs, on the first image data obtained from generator <NUM>, a correction process according to the shading characteristics of each of the plurality of pixels <NUM>. For example, second corrector <NUM> includes one or more filters. In this embodiment, second corrector <NUM> includes infinite impulse response (IIR) filter <NUM>. For example, in the case where each of the plurality of pixels <NUM> has shading characteristics, IIR filter <NUM> is used to obtain the shading amount (for example, local shading amount) of each of the plurality of pixels <NUM>.

Second corrector <NUM> may include filters other than IIR filter <NUM>. From the viewpoint of obtaining a more accurate local shading amount, however, second corrector <NUM> preferably includes IIR filter <NUM>. Since IIR filter <NUM> can remove, from the first image data containing components of white flaws and black flaws, the components of white flaws and black flaws (described in detail alter), a more accurate local shading amount can be obtained (see <FIG>).

Second determiner <NUM> is a processor that performs a process of identifying any defective pixel from among the plurality of pixels <NUM> based on the corrected first image data obtained from second corrector <NUM>. For example, second determiner <NUM> determines, for each of the plurality of pixels <NUM> in the first image data, whether the pixel value of pixel <NUM> is outside a predetermined range, thus identifying any defective pixel. Second determiner <NUM> stores position information indicating the position of pixel <NUM> that is a defective pixel, in storage <NUM>.

Storage <NUM> is a storage device that stores programs executed by the processers in signal processor <NUM>, information necessary to execute the programs, and the like. Storage <NUM> is, for example, semiconductor memory. Storage <NUM> may be, for example, dynamic random access memory (DRAM) or ferroelectric memory. Storage <NUM> may be not included in signal processor <NUM>, as long as it is included in imaging apparatus <NUM>. Storage <NUM> also functions as work memory for the processers in signal processor <NUM>.

Shutter <NUM> controls the time for the light flux from lens <NUM> to reach pixel array portion <NUM>, and is a mechanical shutter configured to travel a shutter curtain, such as a focal plane shutter. The opening/closing operation of shutter <NUM> is controlled by controller <NUM>.

Controller <NUM> controls various structural elements in imaging apparatus <NUM>. Controller <NUM> controls solid-state imaging device <NUM>, signal processor <NUM>, and shutter <NUM>, based on input from operation portion <NUM>. For example, controller <NUM> drives solid-state imaging device <NUM>, to output each digital pixel signal from solid-state imaging device <NUM> to signal processor <NUM>. Controller <NUM> may also adjust the voltage applied to the organic photoelectric conversion film, to control exposure start and exposure end. For example, controller <NUM> applies a predetermined voltage to the organic photoelectric conversion film to set a transmission state, and stops applying the voltage to the organic photoelectric conversion film to set a light shielding state.

Moreover, controller <NUM> controls signal processor <NUM> to execute predetermined signal processing. Controller <NUM> controls solid-state imaging device <NUM> and the opening/closing operation of shutter <NUM>, to switch between the first imaging operation and the second imaging operation.

In this embodiment, controller <NUM> further controls solid-state imaging device <NUM> so that reset voltage VRST supplied to the drain of reset transistor <NUM> will be different between the first imaging operation and the second imaging operation. Specifically, controller <NUM> controls a power source (not illustrated) for supplying reset voltage VRST so that an offset value (an example of a first offset value) in the first imaging operation is higher than an offset value (an example of a second offset value) in the second imaging operation.

When operation portion <NUM> receives an imaging instruction from the user, controller <NUM> may control lens <NUM> (specifically, a motor for controlling the position of lens <NUM>), to adjust the degree of focus of external light and the like.

For example, controller <NUM> executes this process by reading a program from memory (not illustrated) and executing the read program.

The processes executed by imaging apparatus <NUM> will be described below, with reference to <FIG>. First, the process of removing white flaws and black flaws contained in image data obtained by imaging will be described below, with reference to <FIG>. <FIG> is a flowchart illustrating operation when performing correction in imaging apparatus <NUM> according to this embodiment. <FIG> is a schematic diagram explaining operation when performing correction in imaging apparatus <NUM> according to this embodiment. In each of (a) to (f) in <FIG>, the vertical axis represents the signal level, and the horizontal axis represents the pixel position.

As illustrated in <FIG>, imaging apparatus <NUM> obtains image data (an example of second image data) by the second imaging operation (S10). For example, the second imaging operation is performed using the long exposure, where exposure is performed for a first period (for example, about <NUM> sec to <NUM> sec) in a state in which shutter <NUM> is open (transmission state) and charge accumulator <NUM> accumulates a charge. In other words, charge accumulation in the first period is executed in Step S10.

Specifically, in Step S10, controller <NUM> turns on reset transistor <NUM> in a state in which shutter <NUM> is closed (light shielding state), to reset the potential of charge accumulator <NUM>. Here, second reset voltage VRST2 is a voltage value (second reset voltage) corresponding to the second offset value. Controller <NUM> then controls shutter <NUM> to open so that photoelectric conversion element <NUM> can receive light, and turns off reset transistor <NUM>. Consequently, a charge corresponding to the amount of light received by photoelectric conversion element <NUM> is accumulated in charge accumulator <NUM>.

When the exposure in the first period ends, controller <NUM> controls drive controller <NUM> to sequentially output the digital pixel signals corresponding to the accumulated charges to signal processor <NUM>.

Signal processor <NUM> thus obtains image data containing white flaws and black flaws, for example as illustrated in (a) in <FIG>. Generator <NUM> performs a predetermined process on the obtained image data. For example, generator <NUM> executes offset subtraction on the obtained image data (specifically, subtracts the second offset value from the image data), to generate offset-subtracted image data (an example of second image data) as illustrated in (b) in <FIG>. The offset subtraction is a process of subtracting a certain offset signal component from image data. In the offset subtraction in this embodiment, the second offset value is subtracted from the image data. The second offset value is a reference signal level in the second imaging operation.

Generator <NUM> outputs the generated second image data to first determiner <NUM> and first corrector <NUM>. Step S10 is an example of a second imaging step and a second generation step.

Next, imaging apparatus <NUM> obtains image data (an example of first image data) by the first imaging operation (S20). The first imaging operation is exposure performed for the first period in a state in which shutter <NUM> is closed (light shielding state) and charge accumulator <NUM> accumulates a charge. For example, the exposure time in Step S10 and the exposure time in Step S20 are the same. The exposure time in Step S20 is a period during which a charge is accumulated in the light shielding state.

The process of obtaining image data by the first imaging operation will be described below, with reference to <FIG> is a flowchart illustrating operation when obtaining the first image data in imaging apparatus <NUM> according to this embodiment.

As illustrated in <FIG>, controller <NUM> sets the first offset value which is a reference signal level in the first imaging operation, to a value higher than the second offset value (S21). Specifically, controller <NUM> sets the first offset value to a value higher than the second offset value, by controlling the voltage value of first reset voltage VRST1. In other words, controller <NUM> controls the voltage value of first reset voltage VRST1 so that the signal level reference value in the first imaging operation will be a signal level corresponding to the first offset value. The second offset value is set before the first offset value. The second offset value is set beforehand according to the exposure time and the like, and may be stored in storage <NUM>.

Thus, controller <NUM> sets the offset value when accumulating a charge while shielding light (i.e. when obtaining light shielding data), to a value higher than the offset value when accumulating a charge without shielding light (i.e. when obtaining image data).

Controller <NUM> may set the first offset value so that there will be no undetected component for both white flaws and black flaws. Controller <NUM> may set the first offset value to a signal level between <NUM>% and <NUM>% of the maximum value of the signal level (pixel value) after digital conversion. Controller <NUM> more preferably sets the first offset value to a signal level between <NUM>% and <NUM>% of the maximum value of the signal level (pixel value), and further preferably sets the first offset value to a signal level between <NUM>% and <NUM>% of the maximum value of the signal level (pixel value). In the case where the white flaws and the black flaws are approximately equal in signal level, controller <NUM> may set the second offset value to approximately <NUM>% of the maximum value of the signal level (pixel value). In the case where the signal level (pixel value) after digital conversion is expressed in <NUM> bits (<NUM> to <NUM>), controller <NUM> may set the first offset value to a pixel value of about <NUM>.

In Step S21, first, controller <NUM> turns on reset transistor <NUM> in a state in which shutter <NUM> is closed (light shielding state), to reset the potential of charge accumulator <NUM>. Here, first reset voltage VRST1 is a voltage value corresponding to the first offset value. The first reset voltage is a voltage different from the second reset voltage. The first reset voltage is, for example, a voltage higher than the second reset voltage.

Next, controller <NUM> performs control to execute the first imaging operation (S22). Controller <NUM> sets the offset value to the first offset value, and causes charge accumulation in the light shielding state. Specifically, controller <NUM> turns off reset transistor <NUM> while maintaining the state in which shutter <NUM> is closed (light shielding state), to cause charge accumulation in a second period. Consequently, a charge corresponding to pixel defects is accumulated in charge accumulator <NUM>. In charge accumulator <NUM> of pixel <NUM> having no defects, substantially no charge is accumulated. In charge accumulator <NUM> of pixel <NUM> having no defects, the charge corresponding to the second reset voltage remains accumulated. Step S22 is an example of a first imaging step.

For example, the length of the first period and the length of the second period are equal. This makes the signal strength of white flaws and black flaws contained in the generated first image data equal to the signal strength of white flaws and black flaws contained in the second image data, so that white flaws and black flaws can be removed effectively by the below-described process. The signal strength of a white flaw contained in the first image data denotes the difference between the signal level of pixel <NUM> in which the white flaw occurs and the signal level in the case where the white flaw does not occur in pixel <NUM>. The signal strength of a black flaw contained in the first image data denotes the difference between the signal level of pixel <NUM> in which the black flaw occurs and the signal level in the case where the black flaw does not occur in pixel <NUM>. The first period and the second period are not limited to being equal.

Next, after the charge accumulation in the first period ends, generator <NUM> obtains each pixel signal (first pixel signal) from solid-state imaging device <NUM> (S23). Specifically, controller <NUM> controls drive controller <NUM> to sequentially output the digital pixel signals corresponding to the accumulated charges to signal processor <NUM> (for example, generator <NUM>). Signal processor <NUM> thus obtains image data containing white flaws and black flaws, for example as illustrated in (c) in <FIG>.

Next, generator <NUM> performs a predetermined process on the obtained image data to generate first image data (S24). For example, generator <NUM> performs offset subtraction on the obtained image data (specifically, subtracts the first offset value from the image data) to obtain offset-subtracted image data (an example of first image data), for example as illustrated in (d) in <FIG>. Step S24 is an example of a first generation step.

Generator <NUM> outputs the generated first image data to first corrector <NUM>. Generator <NUM> may further output the generated first image data to second corrector <NUM>. This makes it possible to execute the below-described process of identifying a defective pixel using the first image data generated in the subject imaging process.

Referring back to <FIG>, signal processor <NUM> then subtracts the first image data from the second image data to generate third image data (S30). For example, signal processor <NUM> subtracts the image data (an example of first image data) illustrated in (d) in <FIG> from the image data (an example of second image data) illustrated in (b) in <FIG> to generate image data (still image data after dark subtraction) illustrated in (e) in <FIG>, and adds the second offset value to the image data to generate image data (still image data after offset addition) illustrated in (f) in <FIG>. The image data illustrated in (f) in <FIG> is an example of third image data. Step S30 is an example of a third generation step.

The process of generating the third image data will be described below, with reference to <FIG> is a flowchart illustrating operation when generating the third image data in imaging apparatus <NUM> according to this embodiment.

As illustrated in <FIG>, first, first determiner <NUM> obtains position information indicating the position of each defective pixel (S31). For example, first determiner <NUM> reads the position information of each defective pixel stored in storage <NUM>.

Next, first determiner <NUM> determines, for each pixel, whether the pixel (target pixel subjected to determination) is a defective pixel (S32). First determiner <NUM> performs the determination, for example, based on the position information of each defective pixel.

Next, in the case where the target pixel is a defective pixel (S32: Yes), first determiner <NUM> determines whether the image data (for example, the image indicated by the image data) is underexposed (S33). First determiner <NUM> determines whether the image data is underexposed, based on the pixel value of the defective pixel. For example, underexposure means that the pixel value of the defective pixel is less than or equal to the second offset value.

Specifically, first determiner <NUM> determines whether the pixel value of the defective pixel identified by the obtained position information is less than or equal to a first threshold. In the case where the pixel value of the defective pixel is less than or equal to the first threshold, first determiner <NUM> determines that the defective pixel is underexposed. The first threshold is set to, for example, a value with which whether the defective pixel is underexposed can be determined. For example, the first threshold is a value equal in signal level (pixel value) to the second offset value or slightly higher in signal level (pixel value) than the second offset value (for example, a value of about <NUM> times the signal level of the second offset value).

<FIG> illustrates an example of underexposed image data. In the case where the pixel value of the defective pixel is less than or equal to the first threshold as illustrated in <FIG>, first determiner <NUM> determines that the defective pixel is underexposed.

In Step S33, first determiner <NUM> may further determine whether pixels <NUM> neighboring (i.e. around) the underexposed defective pixel are underexposed. For example, first determiner <NUM> may determine, based on the pixel value of each pixel <NUM> (normal pixel) neighboring the defective pixel and the first threshold, whether pixel <NUM> neighboring the defective pixel is underexposed.

For example, in the case where the pixel value of at least one pixel <NUM> of pixels <NUM> neighboring the defective pixel is less than or equal to the first threshold, first determiner <NUM> determines that pixel <NUM> neighboring the defective pixel is underexposed. For example, <FIG> illustrates the case where pixel <NUM> neighboring the defective pixel is not underexposed.

In the case where first determiner <NUM> determines that the defective pixel is underexposed (S33: Yes), first determiner <NUM> outputs the determination result to first corrector <NUM> (S34). For example, first determiner <NUM> outputs information for identifying the underexposed defective pixel, to first corrector <NUM>. The process then advances to Step S35.

In the case where first determiner <NUM> determines that the image data (for example, the image indicated by the image data) is not underexposed (S33: No), first determiner <NUM> further determines whether the image data (for example, the image indicated by the image data) is overexposed (S36). For example, overexposure means that the pixel value (signal level) of the defective pixel is saturated.

Specifically, for example, first determiner <NUM> determines whether the pixel value of the defective pixel determined as No in Step S33 is greater than or equal to a second threshold. In the case where the pixel value of the defective pixel is greater than or equal to the second threshold, first determiner <NUM> determines that the image data (for example, the image indicated by the image data) is overexposed. The second threshold is set to, for example, a value with which whether the defective pixel is overexposed can be determined. For example, the second threshold is a value higher in signal level (pixel value) than the first offset value. The second threshold may be set to, for example, the maximum pixel value which imaging apparatus <NUM> can take.

<FIG> illustrates an example of overexposed image data. In the case where the pixel value of the defective pixel is greater than or equal to the second threshold as illustrated in <FIG>, first determiner <NUM> determines that the defective pixel is overexposed.

In Step S36, first determiner <NUM> may further determine whether pixels <NUM> neighboring the overexposed defective pixel are overexposed. For example, first determiner <NUM> may determine, based on the pixel value of each pixel <NUM> (normal pixel) neighboring the defective pixel and the first threshold, whether pixel <NUM> neighboring the defective pixel is overexposed.

For example, in the case where the pixel value of at least one pixel <NUM> of pixels <NUM> neighboring the defective pixel is greater than or equal to the second threshold, first determiner <NUM> determines that pixel <NUM> neighboring the defective pixel is overexposed. For example, <FIG> illustrates the case where pixel <NUM> neighboring the defective pixel is not overexposed (i.e. pixel <NUM> neighboring the defective pixel is saturated).

In the case where first determiner <NUM> determines that the image data (for example, the image indicated by the image data) is overexposed (S36: Yes), first determiner <NUM> outputs the determination result to first corrector <NUM> (S37). For example, first determiner <NUM> outputs information for identifying the overexposed defective pixel, to first corrector <NUM>. The process then advances to Step S35.

The first threshold and the second threshold are, for example, stored in storage <NUM> beforehand.

In the case where the pixel is underexposed or overexposed, there is an undetected component of a black flaw or a white flaw in the long exposure. In the example in <FIG>, a black flaw component less than or equal to the first threshold is an undetected component. In the example in <FIG>, a white flaw component greater than or equal to the second threshold is an undetected component. In the case where the subtraction process is performed when there is an undetected component of a black flaw or a white flaw, the second image data cannot be appropriately corrected (i.e. noise cannot be removed). Hence, first corrector <NUM> does not perform the subtraction process for such a defective pixel.

Next, first corrector <NUM> determines the pixel value of the defective pixel (the pixel) determined as underexposed or overexposed by first determiner <NUM>, based on the pixel values of pixels <NUM> (normal pixels) neighboring the defective pixel (S35). For example, first corrector <NUM> determines the average value of the pixel values of neighboring pixels <NUM>, as the pixel value of the defective pixel. This process is executed, for example, using a filter such as a median filter (not illustrated) included in first corrector <NUM>. First corrector <NUM> is not limited to determining the average value of the pixel values of neighboring pixels <NUM> as the pixel value of the defective pixel, and may determine the maximum value, the minimum value, the median value, the mode value, or the like of the pixel values of neighboring pixels <NUM> as the pixel value of the defective pixel.

Thus, first corrector <NUM> does not perform the subtraction process on the defective pixel (the pixel) determined as underexposed or overexposed by first determiner <NUM>.

In the case where pixel <NUM> neighboring the defective pixel is underexposed or overexposed, first corrector <NUM> need not perform the process in Step S35 for the defective pixel. This is because the pixel value of pixel <NUM> neighboring the defective pixel for complementing the pixel value of the defective pixel is not an accurate value.

In the case where first determiner <NUM> determines that the defective pixel is neither underexposed nor overexposed (S33 and S36: No), first corrector <NUM> executes the subtraction process on the defective pixel (the pixel) (S38).

In the case where first determiner <NUM> determines that the target pixel is not a defective pixel (S32: No), first corrector <NUM> executes the subtraction process on the pixel (normal pixel) (S38).

After the process in Step S35 or S38 ends, first corrector <NUM> determines whether the determination in Step S32 has been performed for all pixels (S39). In the case where the determination in Step S32 has ended for all pixels <NUM> (S39: Yes), the process in Step S30 ends. In the case where the determination in Step S32 has not ended for all pixels (S39: No), the process returns to Step S32 and remaining pixels <NUM> are subjected to the process.

Although <FIG> illustrates an example in which first corrector <NUM> does not execute the subtraction process for each defective pixel determined as underexposed or overexposed by first determiner <NUM>, the present disclosure is not limited to such. First corrector <NUM> may execute the subtraction process for each defective pixel and, after executing the subtraction process, execute the process in Step S35 for any underexposed or overexposed defective pixel.

Although first determiner <NUM> determines whether the defective pixel is underexposed and overexposed in <FIG>, the determination may not necessarily be performed. First corrector <NUM> may obtain the position information of each defective pixel from storage <NUM>, and execute the process in Step S35 for each defective pixel identified by the obtained position information. Thus, in the case where the defective pixel is identifiable, the pixel value of the defective pixel may be substituted by the average value of the pixel values of pixels <NUM> neighboring the defective pixel using a median filter or the like. In this case, the process of subtracting the first image data from the second image data may be omitted.

Referring back to <FIG>, first corrector <NUM> then outputs the third image data (S40). First corrector <NUM> may output the third image data to display <NUM> to display an image corresponding to the third image data on display <NUM>, or output the third image data to storage <NUM> to store the third image data. In the case where imaging apparatus <NUM> includes a wireless communication module, first corrector <NUM> may transmit the third image data via the wireless communication module.

The process illustrated in <FIG> may be executed each time imaging by the long exposure is performed, or executed periodically. The obtainment of the first image data in Step S20 may be performed by obtaining the first image data obtained when imaging apparatus <NUM> performs imaging operation before executing Step S10. In other words, in Step S20, the first imaging operation need not be performed. For example, generator <NUM> stores the first image data in storage <NUM>. In Step S20, first determiner <NUM> may obtain first image data by reading, from storage <NUM>, first image data obtained before the current imaging and stored in storage <NUM>. In the case where a plurality of items of first image data are stored in storage <NUM>, first determiner <NUM> may read first image data stored most recently in storage <NUM>, or read first image data whose first period is close to the second period in current Step S20.

Specifically, in the case where first image data captured under the same exposure condition as the exposure condition in the second imaging operation in Step S10 is stored in storage <NUM>, first determiner <NUM> may determine to read the first image data. In the case where there is no change between the exposure condition in the previous second imaging operation and the exposure condition in the current second imaging operation, first determiner <NUM> may determine to read first image data. In the case where there is no change between the exposure condition in the previous second imaging operation and the exposure condition in the current second imaging operation and the elapsed time from the previous second imaging operation is within a predetermined time (for example, <NUM>), first determiner <NUM> may determine to read first image data. The elapsed time from the previous second imaging operation is, for example, the time from when image data is obtained by the previous second imaging operation to when image data is obtained by the current second imaging operation.

In the case where there is no change between the exposure condition in the previous second imaging operation and the exposure condition in the current second imaging operation and the temperature of solid-state imaging device <NUM> (for example, the temperature of photoelectric conversion element <NUM>) is the same in the previous second imaging operation and the current second imaging operation, first determiner <NUM> may determine to read first image data. For example, in the case where the difference of the temperature in solid-state imaging device <NUM> or imaging apparatus <NUM> between the previous second imaging operation and the current second imaging operation is within a predetermined range (for example, ±<NUM> degree), first determiner <NUM> may determine to read first image data. In this case, imaging apparatus <NUM> includes a temperature sensor that measures the temperature of solid-state imaging device <NUM> or the like. For example, the temperature sensor measures the temperature in imaging apparatus <NUM>. The exposure condition includes at least one of aperture, shutter speed, and ISO sensitivity.

In the case where first determiner <NUM> determines to read first image data in Step S20, first image data may be obtained by reading first image data from storage <NUM>. In the case where a plurality of items of first image data are stored in storage <NUM>, if at least one item of first image data can be subjected to the read determination, first determiner <NUM> may perform the read determination.

The process of identifying pixel <NUM> (defective pixel) having white flaws or black flaws will be described below, with reference to <FIG>. <FIG> is a flowchart illustrating operation when obtaining the position information of each defective pixel in imaging apparatus <NUM> according to this embodiment. The process illustrated in <FIG> is executed when camera <NUM> is shipped or when the user performs pixel refresh.

As illustrated in <FIG>, first, image data (an example of second image data) is obtained by the second imaging operation (S110). The process in Step S110 is the same as that in Step S10, and its description is omitted. When performing detection operation of detecting a defective pixel, the difference between the second image data and the first image data is not calculated, and accordingly Step S110 may be omitted.

Next, imaging apparatus <NUM> obtains image data (an example of first image data) by the first imaging operation (S120). The process in Step S120 is the same as that in Step S20, and its description is omitted.

Next, imaging apparatus <NUM> executes a shading correction process for the first image data (S130). Specifically, second corrector <NUM> executes the shading correction process using IIR filter <NUM>. Step S130 is an example of a correction step.

<FIG> is a diagram illustrating an example of shading characteristics in imaging apparatus <NUM> according to this embodiment. In <FIG>, the vertical axis represents the signal level, and the horizontal axis represents the pixel position. The upper limit threshold and the lower limit threshold are thresholds for determining whether a pixel is a defective pixel. Specifically, the upper limit threshold is a pixel value for determining whether a white flaw occurs. The lower limit threshold is a pixel value for determining whether a black flaw occurs. The alternate long and short dash line indicates black level when there is no shading (which is a reference signal level, for example, a signal level based on the first offset value). The shading characteristics tend to vary among the plurality of pixels <NUM>. Information of the black level when there is no shading is, for example, stored in storage <NUM>.

In <FIG>, white flaws w1 and w2 and black flaws b1 to b3 occur. If there is no shading, white flaws w1 and w2 are each greater than or equal to the upper limit threshold, and pixel <NUM> is determined as a defective pixel. If there is no shading, black flaws b1 to b3 are each less than or equal to the lower limit threshold, and pixel <NUM> is determined as a defective pixel. Since there is shading, however, the pixel value of pixel <NUM> having white flaw w1 and the pixel value of pixel <NUM> having black flaw b3 are each located between the upper limit threshold and the lower limit threshold, as illustrated in <FIG>. Thus, in the case where there is shading, second determiner <NUM> cannot determine that pixel <NUM> having white flaw w1 and pixel <NUM> having black flaw b3 are defective pixels. In view of this, second corrector <NUM> executes the shading correction process of correcting the shading characteristics as mentioned above.

<FIG> is a diagram explaining the shading correction process in imaging apparatus <NUM> according to this embodiment. Specifically, <FIG> is a diagram explaining the process of calculating the local shading amount in the shading correction process. (a) in <FIG> corresponds to dashed-line region R in <FIG>, and is an enlarged view of dashed-line region R. In (a) in <FIG>, pixel <NUM> having white flaw w1 is indicated as pixel p6, and its neighboring pixels <NUM> are indicated as pixels p1 to p5 and p7 to p11. (b) in <FIG> illustrates the local shading amount in each pixel in dashed-line region R illustrated in (c) in <FIG>.

(a) in <FIG> illustrates image data input to IIR filter <NUM>, for example, first image data (long-exposure light shielding data) output from generator <NUM>. (b) in <FIG> illustrates data output from IIR filter <NUM>, indicating the shading amount (local shading amount). The term "local shading amount" denotes, for example, the shading amount for each pixel <NUM>.

As illustrated in (a) in <FIG>, the signal level (pixel value) varies due to the shading characteristics of each pixel. For example, second corrector <NUM> executes the local shading amount calculation process from one end to the other in the direction in which pixels p1 to p11 are arranged. In this embodiment, second corrector <NUM> executes the local shading amount calculation process in the direction from pixel p1 to pixel p11.

As illustrated in (a) and (b) in <FIG>, second corrector <NUM> sets the pixel value of the output (hereafter also referred to as "output pixel value") of pixel p1 as the pixel value of the input (hereafter also referred to as "input pixel value") of pixel p1. Second corrector <NUM> determines the output pixel value of pixel p2 based on the difference between the input pixel value of pixel p2 and the output pixel value of pixel p1. Specifically, in the case where the difference is less than or equal to a predetermined threshold (an example of a third threshold), for example, in the case where the absolute value of the difference is less than or equal to the predetermined threshold, second corrector <NUM> determines that pixel p2 has no white flaw or black flaw. This means that pixel p2 is a normal pixel. Second corrector <NUM> then sets the output pixel value of pixel p2 to an average value (an example of a second average value) based on the input pixel value of pixel p2 and the output pixel value of pixel p1 (an example of a first average value). Second corrector <NUM> may calculate the average value by equivalent average or weighted average.

Next, second corrector <NUM> determines the output pixel value of pixel p3 based on the difference between the input pixel value of pixel p3 and the output pixel value of pixel p2. Specifically, in the case where the difference is less than or equal to the predetermined threshold (an example of a third threshold), second corrector <NUM> determines that pixel p3 has no white flaw or black flaw. This means that pixel p3 is a normal pixel. Second corrector <NUM> then sets the output pixel value of pixel p3 to an average value (an example of a second average value) based on the input pixel value of pixel p3 and the output pixel value of pixel p2 (an example of a first average value). Thus, second corrector <NUM> performs a process of averaging the pixel value of pixel <NUM> having no white flaw or black flaw.

Second corrector <NUM> sequentially executes the local shading amount calculation process in this way.

The local shading amount calculation process for a defective pixel having a white flaw or a black flaw will be described below. Second corrector <NUM> determines the output pixel value of pixel p6 based on the difference between the input pixel value of pixel p6 and the output pixel value of pixel p5. Specifically, in the case where the difference is greater than the predetermined threshold (an example of a third threshold), for example, in the case where the absolute value of the difference is greater than the predetermined threshold, second corrector <NUM> determines that pixel p6 has a white flaw or a black flaw. This means that pixel p6 is a defective pixel. Second corrector <NUM> then sets the output pixel value of pixel p6 to the output pixel value of pixel p5 (an example of a first average value).

Therefore, even when a pixel has a white flaw or a black flaw, the output pixel value of the pixel can be set to an average value (local shading amount) eliminating the influence of the white flaw or the black flaw. Second corrector <NUM> thus performs a process of substituting the pixel value of pixel <NUM> having a white flaw or a black flaw by the average pixel value of one or more pixels <NUM> located in one direction (leftward in the example in <FIG>) with respect to pixel <NUM>.

Second corrector <NUM> executes this process for each of the plurality of pixels <NUM>. For example, second corrector <NUM> executes the process using IIR filter <NUM>. This makes it possible to calculate the local shading amount in each of the plurality of pixels <NUM> while suppressing the influence of white flaws or black flaws. That is, IIR filter <NUM> can remove, from first image data having components of white flaws and black flaws, the components of white flaws and black flaws, so that a more accurate local shading amount can be calculated.

After executing the foregoing process for each of the plurality of pixels <NUM>, second corrector <NUM> subtracts, for each of the plurality of pixels <NUM>, a correction value based on the local shading amount from the first image data as input. For example, second corrector <NUM> subtracts the correction value of each of the plurality of pixels <NUM> illustrated in (c) in <FIG> from the first image data. Second corrector <NUM> calculates the correction value illustrated in (c) in <FIG>, based on the black level (signal level) when there is no shading and the local shading amount illustrated in (b) in <FIG>. For example, second corrector <NUM> calculates the difference between the black level (signal level) when there is no shading and the local shading amount illustrated in (b) in <FIG> in each of the plurality of pixels <NUM>, as the correction value.

(c) in <FIG> illustrates the correction value of pixel <NUM> corresponding to pixel <NUM> illustrated in <FIG>. The correction value in dashed-line region R in (c) in <FIG> indicates the correction value based on the local shading amount in each of pixels p1 to p11 illustrated in (b) in <FIG>. The five dashed-line circles in (c) in <FIG> indicate the correction values corresponding to the defective pixel having a black flaw or a white flaw and the pixels neighboring the defective pixel illustrated in <FIG>. Specifically, the dashed-line circles indicate the correction values including the defective pixels having white flaw w1, black flaw b1, black flaw b2, black flaw b3, and white flaw w2 in order from the left in the drawing.

The correction value of a defective pixel having a white flaw or a black flaw is equal to the correction value of a pixel adjacent to the defective pixel. Take the defective pixel having white flaw w1 as an example. The correction value of pixel p6 having white flaw w1 included in dashed-line region R is equal to the correction value of pixel <NUM> (left-adjacent pixel <NUM> in (c) in <FIG>) adjacent to pixel p6. The same applies to the correction values of the defective pixels having black flaw b1, black flaw b2, black flaw b3, and white flaw w2 (the correction values in the other dashed-line circles in (c) in <FIG>).

Thus, second corrector <NUM> can generate the first image data from which the influence of shading characteristics has been removed. Second corrector <NUM> outputs the first image data that has undergone the shading correction process, to second determiner <NUM>. The correction value based on the local shading amount may be calculated by performing a predetermined operation other than the above on the local shading amount, or may be the local shading amount itself.

<FIG> is a diagram illustrating first image data that has undergone the shading correction process in imaging apparatus <NUM> according to this embodiment. Specifically, <FIG> illustrates first image data obtained as a result of performing the shading correction process on the first image data illustrated in <FIG>.

Referring back to <FIG>, second determiner <NUM> determines, for each of the plurality of pixels <NUM>, whether the pixel value in the first image data as a result of the shading correction process is outside a predetermined range (S140). In other words, second determiner <NUM> determines, for each of the plurality of pixels <NUM>, whether a white flaw or a black flaw occurs, based on the first image data.

Here, second determiner <NUM> can perform the determination more accurately by using the first image data from which the influence of shading has been removed, as illustrated in <FIG>. For example, second determiner <NUM> can perform the determination using thresholds defined by certain values such as an upper limit threshold and a lower limit threshold. Step S140 is an example of a determination step.

In the case where second determiner <NUM> determines that the pixel value is outside the predetermined range (S140: Yes), second determiner <NUM> stores position information of pixel <NUM> in storage <NUM> (S150). The position information is information indicating the position of pixel <NUM>. Step S150 is an example of a storage step.

In the case where second determiner <NUM> determines that the pixel value is not outside the predetermined range (S140: No), the process advances to Step S160. That is, in the case where the pixel value is not outside the predetermined range, second determiner <NUM> does not store the position information of pixel <NUM> in storage <NUM>.

Next, second determiner <NUM> determines whether the foregoing determination has been performed for all of the plurality of pixels <NUM> (S160). In the case where the determination in Step S140 has been performed for all pixels <NUM> (S160: Yes), second determiner <NUM> ends the process. In the case where the determination in Step S140 has not been performed for all pixels <NUM> (S160: No), the process returns to Step S140 and the process is performed for remaining pixels <NUM>.

Examples of camera <NUM> including imaging apparatus <NUM> described above include digital still camera 1A illustrated in (a) in <FIG> and digital video camera 1B illustrated in (b) in <FIG>. For example, as a result of imaging apparatus <NUM> according to this embodiment being included in the camera illustrated in (a) or (b) in <FIG>, even in the case where black flaws and white flaws occur, both of the black flaws and the white flaws can be appropriately detected and also appropriately removed, as described above.

Camera <NUM> may be included in a mobile terminal such as a smartphone or a tablet terminal, a game machine, or the like.

As described above, the image generation method according to this embodiment is an image generation method in imaging apparatus <NUM> that includes a plurality of pixels <NUM>, the image generation method including: a first imaging step of performing a first imaging operation of capturing an image when each of the plurality of pixels <NUM> is shielded from light, in a state in which a reference signal level in the first imaging operation is set to a first offset value (S22); and a first generation step of generating first image data based on a pixel signal (an example of a first pixel signal) obtained by the first imaging operation (S24). The first offset value is higher than a second offset value that is a reference signal level in a second imaging operation of capturing an image in a state in which light is incident on each of the plurality of pixels <NUM>.

Thus, in the case where a black flaw occurs, underexposure due to the black flaw can be suppressed as compared with the case where the offset value in the first imaging operation is the second offset value, so that the black flaw can be accurately detected. Therefore, with the image generation method according to this embodiment, noise can be obtained more appropriately than conventional techniques. That is, with the image generation method according to this embodiment, noise can be obtained more accurately than conventional techniques.

For example, in the case where the imaging apparatus corrects noise (white flaws and black flaws) using the first image data obtained by the foregoing method, more accurate correction than conventional techniques is possible. In particular, more accurate correction than conventional techniques can be made for black flaws.

The image generation method further includes: a second imaging step of performing the second imaging operation (S10); a second generation step of generating second image data based on a pixel signal (an example of a second pixel signal) obtained by the second imaging operation (S10); and a third generation step of generating third image data by subtracting the first image data from the second image data (S30).

Thus, the third image data is image data from which noise has been removed than conventional. That is, image data from which noise, in particular black flaws, has been removed than conventional techniques can be obtained.

The image generation method further includes: a determination step of determining, for each of the plurality of pixels <NUM>, whether a pixel value of pixel <NUM> in the first image data is outside a predetermined range, based on the first image data (S140); and a storage step of storing position information indicating a position of a defective pixel whose pixel value is determined to be outside the predetermined range in the determination step (S150).

Thus, in the determination step, the defective pixel can be determined more accurately. For example, a defective pixel having a black flaw can be detected more accurately than conventional techniques.

The image generation method further includes: a correction step of correcting, according to shading characteristics of each of the plurality of pixels <NUM>, the pixel value of the pixel in the first image data generated in the first generation step (S130). In the determination step, the determination is performed on the first image data corrected in the correction step.

Thus, whether the pixel is a defective pixel can be determined without being influenced by the shading characteristics. This further improves the determination accuracy in the determination step.

In the correction step, the shading characteristics of each of the plurality of pixels <NUM> are obtained by: when a difference between the pixel value of pixel <NUM> and a first average value of respective one or more pixel values of one or more pixels <NUM> located in one direction with respect to pixel <NUM> is less than or equal to a third threshold, setting the pixel value of pixel <NUM> to a second average value that is an average value of the pixel value of pixel <NUM> and the first average value; and when the difference is greater than the third threshold, setting the pixel value of pixel <NUM> to the first average value.

Thus, the shading characteristics (for example, local shading amount) can be obtained without being influenced by white flaws and black flaws. Therefore, even in the case where white flaws and black flaws occur, correction according to the shading characteristics can be performed more accurately.

In the third generation step, position information of a defective pixel whose noise component increases with exposure time is obtained, and the subtracting in the defective pixel is not performed when a pixel value of the defective pixel in the second image data based on the position information obtained is less than or equal to a first threshold that is a signal level lower than the second offset value.

Thus, the subtraction process is not performed for an underexposed defective pixel, i.e. a defective pixel having an undetected noise component in a black flaw. By not performing the subtraction process for a defective pixel whose black flaw cannot be corrected appropriately, inappropriate correction (for example, excessive subtraction) can be prevented.

In the third generation step, position information of a defective pixel whose noise component increases with exposure time is obtained, and the subtracting in the defective pixel is not performed when a pixel value of the defective pixel in the second image data based on the position information obtained is greater than or equal to a second threshold that is a signal level higher than the second offset value.

Thus, the subtraction process is not performed for an overexposed defective pixel, i.e. a defective pixel having an undetected noise component in a white flaw. By not performing the subtraction process for a defective pixel whose white flaw cannot be corrected appropriately, inappropriate correction (for example, excessive subtraction) can be prevented.

In the third generation step, the pixel value of the defective pixel for which the subtracting is not performed is determined based on a pixel value of at least one pixel <NUM> neighboring the defective pixel.

Thus, the pixel value of the underexposed or overexposed defective pixel can be complemented based on the pixel value of neighboring pixel <NUM>. Therefore, even in the case where appropriate correction of white flaws or black flaws is not possible, the pixel value of the defective pixel can be determined appropriately.

An exposure time in the second imaging operation and an exposure time in the first imaging operation are equal.

Thus, in each defective pixel, the signal level of white flaws and black flaws contained in the first image data and the signal level of white flaws and black flaws contained in the second image data are equal. Accordingly, white flaws and black flaws can be effectively removed from the second image data.

Imaging apparatus <NUM> is capable of executing normal exposure in which exposure is performed for a predetermined time and long exposure in which exposure is performed longer than the predetermined time. The second imaging operation is performed using the long exposure, and the second offset value is same as a third offset value that is a reference signal level in the normal exposure.

Thus, in the long exposure in which white flaws and black flaws tend to be noticeable, the white flaws and the black flaws can be appropriately removed.

Imaging apparatus <NUM> according to this embodiment includes: a plurality of pixels <NUM>; controller <NUM> that performs control to perform a first imaging operation of capturing an image when each of the plurality of pixels <NUM> is shielded from light, in a state in which a reference signal level in the first imaging operation is set to a first offset value; and generator <NUM> that generates image data based on a pixel signal obtained by the first imaging operation. Controller <NUM> controls the first offset value to be higher than a second offset value that is a reference signal level in a second imaging operation of capturing an image in a state in which light is incident on each of the plurality of pixels <NUM>. A program according to this embodiment is a program for causing a computer to execute the image generation method described above.

This achieves the same effects as the foregoing image generation method.

The foregoing embodiment has been described to illustrate the disclosed technology, through the detailed description and the accompanying drawings.

The structural elements in the detailed description and the accompanying drawings may include not only the structural elements essential for the solution of the problem but also the structural elements not essential for the solution of the problem, to illustrate the disclosed technology. The inclusion of such optional structural elements in the detailed description and the accompanying drawings therefore does not mean that these optional structural elements are essential structural elements.

For example, although the foregoing embodiment describes an example in which the image generation method, etc. are executed in the case where image data contains both white flaws and black flaws, the present disclosure is not limited to such. The image generation method, etc. may be executed in the case where image data contains only black flaws from among white flaws and black flaws. Hence, black flaws can be obtained appropriately.

Although the foregoing embodiment describes an example in which first determiner <NUM> determines whether a defective pixel is underexposed in Step S32 and whether the defective pixel is overexposed in Step S33, the present disclosure is not limited to such. First determiner <NUM> may also determine whether a normal pixel is underexposed and whether the normal pixel is overexposed.

Second corrector <NUM> in the foregoing embodiment may include a low-pass filter, a minimum value filter, or the like, instead of or together with IIR filter <NUM>.

First corrector <NUM> in the foregoing embodiment may further perform the shading correction process. For example, first corrector <NUM> may include an IIR filter, and perform the shading correction process on at least one of the first image data and the second image data or on the third image data.

Although the foregoing embodiment describes an example in which second corrector <NUM> corrects the first image data using the calculated local shading amount, the present disclosure is not limited to such. For example, second corrector <NUM> may correct the predetermined third threshold and fourth threshold using the calculated local shading amount. Second corrector <NUM> outputs the third threshold and the fourth threshold corrected according to the shading amount, to second determiner <NUM>. Second determiner <NUM> may then perform the determination in Step S140 based on the first image data having shading and the third threshold and the fourth threshold corrected according to the shading amount. Thus, second corrector <NUM> may set the threshold (for example, the third threshold and the fourth threshold) according to the local shading amount, at each pixel position.

The structural elements (functional blocks) in imaging apparatus <NUM> may be individually formed into one chip or part or all thereof may be included in one chip, using a semiconductor device such as IC (Integrated Circuit) or LSI (Large Scale Integration). The circuit integration technique is not limited to LSIs, and dedicated circuits or general-purpose processors may be used to achieve the same. A field programmable gate array (FPGA) which can be programmed after manufacturing the LSI, or a reconfigurable processor where circuit cell connections and settings within the LSI can be reconfigured, may be used. Further, in the event of the advent of an integrated circuit technology which would replace LSIs by advance of semiconductor technology or a separate technology derived therefrom, such a technology may be used for integration of the functional blocks, as a matter of course. Application of biotechnology is a possibility.

The whole or part of the foregoing processes may be realized by hardware such as an electronic circuit, or realized by software. A process by software is realized by a processor in imaging apparatus <NUM> executing a program stored in memory. The program may be recorded in a recording medium and distributed or circulated. For example, by installing the distributed program in another apparatus including a processor and causing the processor to execute the program, the processes can be performed by the apparatus.

The structural elements forming the foregoing processors such as controller <NUM>, generator <NUM>, first determiner <NUM>, first corrector <NUM>, second corrector <NUM>, and second determiner <NUM> may be a single element that performs centralized control, or may be a plurality of elements that perform distributed control in cooperation with each other. The software program may be provided as an application by communication via a communication network such as the Internet, communication according to a mobile communication standard, or the like.

The division of the functional blocks in each block diagram is an example, and a plurality of functional blocks may be realized as one functional block, one functional block may be divided into a plurality of functional blocks, or part of functions may be transferred to another functional block. Moreover, functions of a plurality of functional blocks having similar functions may be realized by single hardware or software in parallel or in a time-sharing manner.

The order in which the steps are performed in each flowchart is an example provided for specifically describing the presently disclosed techniques, and order other than the above may be used. For example, Steps S10 and S20 in <FIG> may be performed in reverse order. Part of the steps may be performed simultaneously (in parallel) with other steps.

Any embodiment obtained by combining the structural elements and functions in the foregoing embodiments is also included in the scope of the present disclosure.

Claim 1:
An image generation method for an imaging apparatus (<NUM>) that includes a plurality of pixels (<NUM>), the image generation method comprising:
performing a first imaging operation (S20) of capturing an image when each of the plurality of pixels (<NUM>) is shielded from light, in a state in which a first reference signal level in the first imaging operation is set to a first offset value, the first offset value corresponding to a first reset voltage, VRST1, said reset voltage resetting the potential of a charge accumulator (<NUM>) of said pixels (<NUM>); and
generating first image data (S24) based on a first pixel signal obtained (S23) by the first imaging operation (S22),
wherein the first offset value is set (S21) higher than a second offset value, the second offset value corresponding to a second reset voltage, VRST2, that is a second reference signal level in a second imaging operation of capturing an image in a state in which light is incident on each of the plurality of pixels (<NUM>),
performing the second imaging operation (S10);
generating second image data based on a second pixel signal obtained by the second imaging operation (S10); and
generating third image data by subtracting the first image data from the second image data (S30).