Patent Publication Number: US-10791290-B2

Title: Image processing apparatus, image processing method, and non-transitory computer readable recording medium

Description:
CROSS-REFERENCE TO RELATED APPLICATION 
     This application is a continuation of International Application No. PCT/JP2016/072442, filed on Jul. 29, 2016, the entire contents of which are incorporated herein by reference. 
    
    
     BACKGROUND 
     The present disclosure relates to an image processing apparatus, an image processing method, and a non-transitory computer readable recording medium. 
     Generally, when an image sensor used in an imaging apparatus, a circuit constituting a pixel may become defective due to manufacturing variation or degradation over time. A pixel value read out from such a pixel tends to be greater or smaller compared with pixels at neighboring portions (hereinafter referred to as “neighboring pixels”). As a technique for correcting the pixel value of a pixel having such a defect (hereinafter referred to as a “defective pixel”), there is a known technique that calculates pixel value variance when the pixel values of the neighboring pixels of a pixel of interest are defined as a group, and that calculates excessiveness indicating a degree of excessiveness of the pixel value of the pixel of interest with respect to the neighboring pixels obtained by normalizing a difference value between the pixel value of the pixel of interest and the pixel values of the neighboring pixels on the basis of the calculated variance (Refer to JP 4453332 B2). In this technique, the degree of excessiveness of the pixel value of the pixel of interest is compared with the degree of excessiveness of the pixel value of a different color filter at the same position as the pixel of interest to determine whether the pixel is a defective pixel, and when the pixel of interest is a defective pixel, the pixel value of the pixel of interest is corrected by using the pixel values of the neighboring pixels. 
     Furthermore, the image sensor reads out a pixel value corresponding to the electrical charge generated in accordance with the exposure amount. The maximum charge accumulation amount (saturation charge amount) in each of pixels of the image sensor varies from pixel to pixel due to manufacturing errors of the image sensor, or the like. For example, in the imaging of a flat subject with no contrast and high luminance, it is preferable that the read pixel values are uniformly saturated, and all pixels exhibit equivalent pixel values. However, the pixel having a low saturation charge amount due to variation in individual pixels is saturated at a pixel value lower than that of other pixels, leading to the pixel values not exhibiting the same pixel value. In a known technique (refer to WO 2013/154105 A), the pixel value of a pixel saturated with a lower pixel value (hereinafter referred to as “low saturation pixel”) compared with the pixel values of the other pixels in this manner is replaced by an average value of pixel values of neighboring pixels so as to correct the pixel value of the pixel of interest. 
     SUMMARY 
     The present disclosure is directed to an improvement of an image processing apparatus and an image processing method. 
     According to a first aspect of the present disclosure, an image processing apparatus is provided which processes noise resulting from a defective pixel different from a low saturation pixel, included in image data generated by an image sensor including a plurality of pixels two-dimensionally arranged to receive light from outside and generate a signal according to an amount of received light and including a plurality of read-out circuits that read out the signal as a pixel value. The image processing apparatus includes an acquisition unit that obtains the image data generated by the image sensor and a saturation level of each of the plurality of pixels; and a noise processing unit that processes the noise on the basis of the saturation level obtained by the acquisition unit. 
     According to a second aspect of the present disclosure, an image processing method is provided which executed by an image processing apparatus that corrects noise resulting from a defective pixel different from a low saturation pixel, included in image data generated by an image sensor including a plurality of pixels two-dimensionally arranged to receive light from outside and generate a signal according to an amount of received light and including a plurality of read-out circuits that read out the signal as a pixel value. The method includes obtaining the image data generated by the image sensor and a saturation level of each of the plurality of pixels; and processing the noise on the basis of the saturation level obtained in the acquisition step. 
     According to a third aspect of the present disclosure, a non-transitory computer readable recording medium is provided which stores a program that causes an image processing apparatus that corrects noise resulting from a defective pixel different from a low saturation pixel, included in image data generated by an image sensor including a plurality of pixels two-dimensionally arranged to receive light from outside and generate a signal according to an amount of received light and including a plurality of read-out circuits that read out the signal as a pixel value. The program includes obtaining the image data generated by the image sensor and a saturation level of each of the plurality of pixels; and processing the noise on the basis of the saturation level obtained in the acquisition step. 
     The above and other features, advantages and technical and industrial significance of this disclosure will be better understood by reading the following detailed description of presently preferred embodiments of the disclosure, when considered in connection with the accompanying drawings. 
    
    
     
       BRIEF DESCRIPTION OF THE DRAWINGS 
         FIG. 1  is a block diagram schematically illustrating a configuration of an imaging system according to a first embodiment of the present disclosure; 
         FIG. 2  is a diagram schematically illustrating a configuration of main portions of an image sensor according to the first embodiment of the present disclosure; 
         FIG. 3  is a diagram schematically illustrating an example of a defective pixel and a low saturation pixel in the image sensor according to the first embodiment of the present disclosure; 
         FIG. 4  is a diagram schematically illustrating an example of low saturation pixel information recorded in a low saturation pixel information recording unit according to the first embodiment of the present disclosure; 
         FIG. 5  is a diagram schematically illustrating another example of the low saturation pixel information recorded in the low saturation pixel information recording unit according to the first embodiment of the present disclosure; 
         FIG. 6  is a flowchart illustrating an outline of a noise process executed by an image processing apparatus according to the first embodiment of the present disclosure; 
         FIG. 7  is a flowchart illustrating an outline of a defective pixel detection process in  FIG. 6 ; 
         FIG. 8  is a flowchart illustrating an outline of a defective pixel correction process in  FIG. 6 ; 
         FIG. 9  is a flowchart illustrating an outline of a direction discrimination process in  FIG. 8 ; 
         FIG. 10A  is a diagram schematically illustrating an example of the direction discrimination according to the first embodiment of the present disclosure; 
         FIG. 10B  is a diagram schematically illustrating an example of the direction discrimination according to the first embodiment of the present disclosure; 
         FIG. 10C  is a diagram schematically illustrating an example of the direction discrimination according to the first embodiment of the present disclosure; 
         FIG. 10D  is a diagram schematically illustrating an example of the direction discrimination according to the first embodiment of the present disclosure; 
         FIG. 10E  is a diagram schematically illustrating an example in which a defective pixel exists around the pixel of interest in the direction discrimination according to the first embodiment of the present disclosure; 
         FIG. 11  is a flowchart illustrating an outline of a low saturation pixel correction process in  FIG. 6 ; 
         FIG. 12A  is a diagram schematically illustrating an outline of an interpolation A determination performed by a low saturation pixel correction unit according to the first embodiment of the present disclosure; 
         FIG. 12B  is a diagram schematically illustrating an outline of the interpolation A determination performed by the low saturation pixel correction unit according to the first embodiment of the present disclosure; 
         FIG. 12C  is a diagram schematically illustrating an outline of the interpolation A determination performed by the low saturation pixel correction unit according to the first embodiment of the present disclosure; 
         FIG. 13  is a flowchart illustrating an outline of the defective pixel correction process executed by an image processing apparatus according to a second embodiment of the present disclosure; 
         FIG. 14  is a diagram illustrating an example of a reference mask referred to by a defective pixel correction unit according to the second embodiment of the present disclosure; 
         FIG. 15  is a flowchart illustrating an outline of saturation level calculation process executed by the image processing apparatus according to the second embodiment of the present disclosure; and 
         FIG. 16  is a block diagram schematically illustrating a configuration of an imaging system according to a third embodiment of the present disclosure. 
     
    
    
     DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS 
     Hereinafter, embodiments of the present disclosure (hereinafter, referred to as embodiment(s)) will be described with reference to the drawings. Note that the present disclosure is not limited by the following embodiments. In the description of the drawings, the same portions are given the same reference numerals. 
     First Embodiment 
     Configuration of Imaging System 
       FIG. 1  is a block diagram schematically illustrating a configuration of an imaging system according to a first embodiment of the present disclosure. An imaging system  1  illustrated in  FIG. 1  includes an imaging apparatus  10 , an image processing apparatus  30 , and a display device  40 . 
     Configuration of Imaging Apparatus 
     First, a configuration of the imaging apparatus  10  will be described. As illustrated in  FIG. 1 , the imaging apparatus  10  includes an optical system  101 , a diaphragm  102 , a shutter  103 , a driver  104 , an image sensor  105 , an analog processing unit  106 , an analog-to-digital (A/D) converter  107 , an operation unit  108 , a memory interface (I/F) unit  109 , a recording medium  110 , a volatile memory  111 , a non-volatile memory  112 , a bus  113 , an imaging controller  114 , and a first external I/F unit  115 . 
     The optical system  101  includes one or more lenses. The optical system  101  includes a focus lens and a zoom lens, for example. 
     The diaphragm  102  adjusts exposure by limiting an amount of incident light collected by the optical system  101 . Under the control of the imaging controller  114  described below, the diaphragm  102  limits the incident amount of the light collected by the optical system  101 . Note that the incident amount of light may be limited by using the shutter  103  and an electronic shutter in the image sensor  105  without using the diaphragm  102 , in other embodiments. The optical system  101  and the diaphragm  102  may be configured to be detachable from the imaging apparatus  10 . 
     The shutter  103  sets the state of the image sensor  105  to an exposure state or a light-shielding state. The shutter  103  includes a focal plane shutter, for example. Incidentally, an electronic shutter may be used in the image sensor  105  instead of the shutter  103 . 
     Under the control of the imaging controller  114  described below, the driver  104  drives the optical system  101 , the diaphragm  102 , and the shutter  103 . For example, the driver  104  moves the optical system  101  along an optical axis O 1  to perform zoom magnification change or focusing position adjustment for the imaging apparatus  10 . 
     Under the control of the imaging controller  114  described below, the image sensor  105  receives the light collected by the optical system  101 , converts the received light into image data (electrical signal), and outputs the image data. The image sensor  105  includes a complementary metal oxide semiconductor (CMOS) including a plurality of two-dimensionally arranged pixels. An RGB filter arranged in a Bayer array is disposed on a front surface of each of the pixels. Note that the image sensor  105  is not limited to the Bayer array but may be in a stacked form such as Fovion&#39;s, for example. Moreover, not only RGB filter but also any other filters such as a complementary color filter may be employed. Alternatively, a light source may be used which is capable of emitting light beams of different colors on a time division basis without disposing a filter on the image sensor  105 , and to form a color image using sequentially captured images while changing the colors of the beams to be emitted. Moreover, the image sensor  105  has an electronic shutter function capable of electronically controlling the amount of received light. 
     Now, a configuration of the image sensor  105  will be described in detail.  FIG. 2  is a diagram schematically illustrating a configuration of main portions of the image sensor  105 . Note that the image sensor  105  in  FIG. 2  is an exemplary case where a read-out circuit is shared by a plurality of pixels in order to enhance sensitivity by increasing the aperture ratio of the pixel. Note that the image sensor  105  illustrated in  FIG. 2  includes one read-out circuit arranged for eight pixels, that is, two pixels in the horizontal direction (lateral direction)×four pixels in the vertical direction (longitudinal direction). Note that, while  FIG. 2  illustrates an exemplary case where one read-out circuit is arranged for eight pixels, that is, two pixels in the horizontal direction (lateral direction)×four pixels in the vertical direction (longitudinal direction), the image sensor  105  according to the first embodiment is assumed to have a configuration in which the above-described pixels and the read-out circuit are arranged side by side in horizontal and vertical directions. 
     As illustrated in  FIG. 2 , the image sensor  105  includes a plurality of pixels (photodiodes)  105   a , a first switch  105   b , a transfer line  105   c , a floating diffusion (FD) unit  105   d , an amplifier unit  105   e , a second switch  105   f , a control line  105   g , and a transfer line  105   h . Each of the plurality of pixels  105   a  receives light by exposure, and performs photoelectric conversion, thereby generating electrical charges in accordance with an exposure amount. The first switch  105   b  is provided on each of the plurality of pixels  105   a , and opens or closes in accordance with the control of the imaging controller  114 . The transfer line  105   c  transfers signals (electrical charges) output from each of the plurality of pixels  105   a . The floating diffusion (FD) unit  105   d  stores the signals output from the plurality of pixels  105   a . The amplifier unit  105   e  amplifies the signals output from the FD unit  105   d . The second switch  105   f  opens or closes according to the control of the imaging controller  114 . The control line  105   g  controls the second switch  105   f . The transfer line  105   h  transfers the electrical signal amplified by the amplifier unit  105   e.    
     When the above-configured image sensor  105  reads out a signal that corresponds to the exposure amount on pixels  105   a ( 1 ) to  105   a ( 8 ) as a pixel value, the electrical charge generated on the pixel  105   a ( 1 ) is transferred to the FD unit  105   d  by first resetting the FD unit  105   d  and by turning on solely a first switch  105   b ( 1 ) by the imaging controller  114 . Thereafter, the imaging controller  114  turns on the second switch  105   f , whereby the image sensor  105  causes the amplifier unit  105   e  to amplify the electrical charges stored in the FD unit  105   d  and reads out (outputs) the electrical charge as a pixel value. Next, the image sensor  105  resets the FD unit  105   d  and the imaging controller  114  turns on solely a first switch  105   b ( 2 ), whereby the image sensor  105  transfers the electrical charges generated at the pixel  105   a ( 2 ) to the FD unit  105   d . Thereafter, the imaging controller  114  turns on the second switch  105   f , whereby the image sensor  105  causes the amplifier unit  105   e  to amplify the electrical charges stored in the FD unit  105   d  and reads out the electrical charges as a pixel value. By sequentially performing such read-out operation, the image sensor  105  can sequentially output the signal that corresponds to the exposure amount on each of the pixels  105   a ( 1 ) to  105   a ( 8 ), as a pixel value. Note that in the first embodiment, the FD unit  105   d , the amplifier unit  105   e , and the second switch  105   f  serve as a read-out circuit that reads out electrical charges from each of the plurality of pixels  105   a.    
     Returning to  FIG. 1 , description of the configuration of the imaging apparatus  10  follows below. 
     The analog processing unit  106  performs predetermined analog process onto an analog signal input from the image sensor  105  and outputs the processed signal to the A/D converter  107 . Specifically, the analog processing unit  106  performs noise reduction process, gain-up process, or the like, on the analog signal input from the image sensor  105 . For example, the analog processing unit  106  performs, onto the analog signal, reduction of reset noise and so on, and waveform shaping, and then, further performs gain-up so as to achieve intended brightness. The A/D converter  107  generates digital image data (hereinafter, referred to as “RAW image data”) by performing A/D conversion onto the analog signal input from the analog processing unit  106 , and outputs the generated data to the volatile memory  111  via the bus  113 . Note that the A/D converter  107  may directly output the RAW image data to predetermined parts of the imaging apparatus  10  described below. Note that the analog processing unit  106  and the A/D converter  107  may be provided in the image sensor  105 , and the image sensor  105  directly may output digital RAW image data. 
     The operation unit  108  issues various instructions to each part of the imaging apparatus  10 . Specifically, the operation unit  108  includes a power switch that switches the power supply states of the imaging apparatus  10  between an on-state and an off-state, a release switch that issues an instruction of still image shooting, an operation switch that switches various settings of the imaging apparatus  10 , and a moving image switch that issues an instruction of moving image shooting. 
     The recording medium  110  includes a memory card attached from outside of the imaging apparatus  10 , and is removably attached onto the imaging apparatus  10  via the memory I/F unit  109 . Moreover, the recording medium  110  may output programs and various types of information to the non-volatile memory  112  via the memory I/F unit  109  under the control of the imaging controller  114 . 
     The volatile memory  111  temporarily records image data input from the A/D converter  107  via the bus  113 . For example, the volatile memory  111  temporarily stores image data sequentially output from the image sensor  105  frame by frame, via the analog processing unit  106 , the A/D converter  107 , and the bus  113 . The volatile memory  111  includes a synchronous dynamic random access memory (SDRAM). 
     The non-volatile memory  112  being constituted with a flash memory, or the like, records various programs needed to operate the imaging apparatus  10  and various types of data used in execution of the program. The non-volatile memory  112  further includes: a program recording unit  112   a ; a defective pixel information recording unit  112   b  that records position information (position information includes one or both of position information of the read-out circuit that reads out the pixel value (position information of the amplifier unit  105   e ) and position information of a pixel in which a defective pixel occurs) of a defective pixel corresponding to the position of a pixel in the image sensor  105 ; and a low saturation pixel information recording unit  112   c  that records position information (position information includes one or both of position information of the read-out circuit that reads out the pixel value (position information of the amplifier unit  105   e ) and position information of a pixel in which a low saturation pixel occurs) of a low saturation pixel corresponding to the position of a pixel in the image sensor  105 . Here, the defective pixel may include defective pixels known as a white spot, a black spot, and a blinking defect that causes blinking defect noise. 
     The bus  113  includes a transmission line that connects individual parts of the imaging apparatus  10  with each other, and transfers various types of data generated inside the imaging apparatus  10  to each of the individual parts of the imaging apparatus  10 . 
     The imaging controller  114  may be configured of a central processing unit (CPU) or the like, and integrally controls operations of the imaging apparatus  10  by providing instruction and transferring data to individual part of the imaging apparatus  10  in response to an instruction signal and a release signal from the operation unit  108 . For example, when a second release signal has been input from the operation unit  108 , the imaging controller  114  starts control of shooting operation on the imaging apparatus  10 . Here, the shooting operation in the imaging apparatus  10  is operation in which the analog processing unit  106  and the A/D converter  107  perform predetermined process on the exposure timing of the image sensor  105 , the output timing of the analog signal, and the analog signal output from the image sensor  105 . The image data processed in this manner are recorded in the recording medium  110  via the bus  113  and the memory I/F unit  109  under the control of the imaging controller  114 . 
     The first external I/F unit  115  outputs information input from external apparatuses via the bus  113 , to the non-volatile memory  112  or the volatile memory  111 , and together with this, outputs, to external apparatuses via the bus  113 , information recorded in the volatile memory  111 , information recorded in the non-volatile memory  112 , and the image data generated by the image sensor  105 . Specifically, the first external I/F unit  115  outputs image data, defective pixel information, and low saturation pixel information generated by the image sensor  105  to the image processing apparatus  30  via the bus  113 . 
     Configuration of Image Processing Apparatus 
     Next, a configuration of the image processing apparatus  30  will be described. The image processing apparatus  30  includes a third external I/F unit  31 , a noise processing unit  32 , an image processing unit  33 , an operation unit  34 , a recording unit  35 , and an image processing controller  36 . 
     The third external I/F unit  31  operates as an acquisition unit that obtains image data generated by the image sensor  105  via the first external I/F unit  115  of the imaging apparatus  10 , and obtains defective pixel information recorded in the defective pixel information recording unit  112   b  and low saturation pixel information recorded in the low saturation pixel information recording unit  112   c  within the non-volatile memory  112 , and outputs the obtained image data, defective pixel information and low saturation pixel information to the noise processing unit  32  and the recording unit  35 . The third external I/F unit  31  and the first external I/F unit  115  are mutually connected via a control cable, wireless communication, or the like, capable of bidirectionally exchanging information, for example. The third external I/F unit  31  functions as an acquisition unit in the present embodiment. 
     The noise processing unit  32  processes the noise resulted from the defective pixel in the RAW image data input from the third external I/F unit  31  on the basis of the saturation level, and outputs the corrected pixel value to the image processing unit  33 . The noise processing unit  32  includes a defective pixel detection unit  51 , a defective pixel correction unit  52 , and a low saturation pixel correction unit  53 . 
     The defective pixel detection unit  51  detects a defective pixel and a level of the defective pixel by using the RAW image data. Specifically, the defective pixel detection unit  51  detects a defective pixel on the basis of the saturation level. The level of the defective pixel (hereinafter referred to as “defective pixel level”) varies depending on the state (for example, temperature, voltage, and so on.) of the image sensor  105 . Therefore, in the first embodiment, the defective pixel detection unit  51  detects the defective pixel level by using at least the RAW image data. Note that the defective pixel detection unit  51  may input a position as defective pixel information and detect a defective pixel level, or may detect both the position and the level of the defective pixel. Furthermore, the defective pixel detection unit  51  may simply detect the position of the defective pixel and may use the neighboring pixels of the low saturation pixel to perform correction, without using (that is, excluding) the defective pixel in the low saturation pixel correction unit  53  as described below. 
     The defective pixel correction unit  52  corrects a defective pixel in the RAW image data on the basis of the saturation level. Specifically, the defective pixel correction unit  52  corrects the defective pixel on the basis of the defective pixel level detected by the defective pixel detection unit  51 . Note that the defective pixel correction unit  52  may use the position of the defective pixel alone to perform correction by using the neighboring pixels of the defective pixel. 
     The low saturation pixel correction unit  53  corrects the low saturation pixel in the RAW image data. Specifically, the low saturation pixel correction unit  53  performs correction on the basis of the low saturation pixel information (including information on the position of low saturation pixel and the saturation level in the low saturation pixel other than the defective pixel, and information on the position of the defective pixel and the saturation level in the defective pixel) such that correction is made so as to achieve a pixel value similar to the neighboring pixel(s) of the low saturation pixel when the low saturation pixel is saturated. On the other hand, the pixel value of the pixel is output as it is when the pixel is not saturated. 
     The image processing unit  33  applies predetermined image processes on the image data in which noise has been corrected by the noise processing unit  32 , and outputs the processed data to the display device  40 . The predetermined image processes may include basic image process including at least optical black subtraction process, white balance adjustment process, and including synchronization process of the image data, color matrix calculation process, y correction process, color reproduction process, and edge enhancement process when the image sensor  105  is arranged in a Bayer array. Moreover, the image processing unit  33  performs image process of reproducing a natural image on the basis of individual image processing parameters that have been set beforehand. The parameters of image process are values of contrast, sharpness, saturation, white balance, and gradation. 
     The operation unit  34  receives inputs of various operation signals related to the image processing apparatus  30 . The operation unit  34  is configured by using, for example, a cross button, a push button, a touch panel, and the like. 
     The recording unit  35  is configured by using a volatile memory or a non-volatile memory, and records defective pixel information and low saturation pixel information output from the third external I/F unit  31 . The recording unit  35  may record individual information of the position of the defective pixel, the defective pixel level, and the saturation level, each of which are detected by the noise processing unit  32 . 
     The image processing controller  36  integrally controls individual components of the image processing apparatus  30 . The image processing controller  36  is formed of a CPU or the like. The image processing controller  36  controls instructions for individual parts of the image processing apparatus  30 , data transfer, or the like. 
     Configuration of Display Device 
     Next, a configuration of the display device  40  will be described. The display device  40  displays an image that corresponds to the image data input from the image processing apparatus  30 . The display device  40  includes a display panel of liquid crystal, organic electroluminescence (EL), or the like. 
     In the imaging system  1  having the above configuration, the image processing apparatus  30  corrects a defective pixel and/or a low saturation pixel in the image sensor  105 , and the display device  40  displays an image that corresponds to the image data that has undergone image process performed by the image processing apparatus  30 . 
     Low Saturation Pixel Information 
     Next, the low saturation pixel information recorded in the low saturation pixel information recording unit  112   c  will be described.  FIG. 3  is a diagram schematically illustrating an example of a defective pixel and a low saturation pixel in the image sensor.  FIG. 4  is a diagram schematically illustrating an example of low saturation pixel information recorded in the low saturation pixel information recording unit  112   c .  FIG. 5  is a diagram schematically illustrating another example of the low saturation pixel information recorded in the low saturation pixel information recording unit  112   c . In  FIG. 3 , a pixel P 1  indicates a normal pixel (no hatch), a pixel P 2  (with hatch) indicates a defective pixel including a white spot defect, a black spot defect, or a blinking defective, and a pixel P 3  (with hatch) indicates a low saturation pixel. Moreover, in  FIG. 4 , pixels P 10  and P 11  indicate low saturation pixels. 
     As illustrated in  FIGS. 3 to 5 , the low saturation pixel information includes position information for each of pixels and a saturation level detected beforehand by a detection device or the like in association with each other. As a recording method, there are a method (refer to  FIG. 4 ) of obtaining a saturation level map in which a saturation level is set for each of a plurality of pixels in RAW image data generated by the image sensor  105 , and a method (refer to  FIG. 5 ) of obtaining coordinates (address) and the saturation level of the low saturation pixel, in association with each other. When implementing the method of obtaining the coordinates (address) of the low saturation pixel in association with the saturation level, a coordinate and a corresponding saturation level are recorded in association with each other for the low saturation pixel and only the saturation level is recorded for a pixel other than the low saturation pixel. When the coordinates in association with the saturation level are recorded, the corresponding saturation level is going to be used. When the coordinates are not recorded, the saturation level of pixels other than the low saturation pixel is going to be used. However, regarding a pixel that is not a low saturation pixel, when it is sufficiently saturated, a maximum value of the pixel value (for example, 4095 in the case of 12 bits) may be set as the saturation level. 
     Incidentally, in  FIGS. 5 and 6 , the saturation level is recorded so as to correspond to the position of the pixel. However, when the saturation level is lowered due to the read-out circuit, the saturation level may be recorded to correspond to the position of the read-out circuit (of course, the saturation level may be recorded by the method illustrated in  FIGS. 5 and 6 ). In this case, the information converted to the saturation level on a pixel-by-pixel basis may be set as the low saturation pixel information, such as setting the same saturation level to the pixels sharing the read-out circuit after reading out the recorded information. 
     Furthermore, the saturation level of each of pixels may desirably be determined in consideration of the linearity of the pixel value and random noise, or the like. For example, a value obtained by reducing a value based on the random noise amount of the luminance from the pixel value of an image obtained by exposure under a condition achieving complete saturation may be set as the saturation level of the pixel. Alternatively, the pixel value at which the linearity is lost may be set as the saturation level. The saturation level may of course be set in consideration of both. 
     Process in Image Processing Apparatus 
     Next, process to be executed by the image processing apparatus  30  will be described.  FIG. 6  is a flowchart illustrating an outline of noise process executed by the image processing apparatus  30 , that is, a flowchart of a main routine executed by the image processing apparatus  30 . As illustrated in  FIG. 6 , first, the defective pixel detection unit  51  executes defective pixel detection process of detecting a defective pixel in the RAW image input from the third external I/F unit  31  (Step S 101 ). Details of the defective pixel detection process will be described later. 
     Subsequently, the defective pixel correction unit  52  executes a defective pixel correction process of correcting the defective pixel in the RAW image (Step S 102 ). Details of the defective pixel correction processing will be described below. 
     Thereafter, the low saturation pixel correction unit  53  executes the low saturation pixel correction process of correcting the low saturation pixel in the RAW image (Step S 103 ). Details of the low saturation pixel correction process will be described later. After Step S 103 , the image processing apparatus  30  finishes the current process. 
     Outline of Defective Pixel Detection Processing 
     Next, details of the defective pixel detection processing described in Step S 101  in  FIG. 6  will be described.  FIG. 7  is a flowchart illustrating an outline of defective pixel detection processing, that is, a flowchart of a subroutine executed by the image processing apparatus  30 . 
     As illustrated in  FIG. 7 , first, the defective pixel detection unit  51  resets a counter y (to y=0) indicating the position of the RAW image in the vertical direction (Step S 201 ), and resets a counter x (to x=0) indicating the position of the RAW image in the horizontal direction (Step S 202 ). Incidentally, an upper end of the RAW image is set to 0 in the counter y indicating the position of the RAW image in the vertical direction, and a left end of the RAW image is set to 0 in the counter x indicating the position of the RAW image in the horizontal direction. 
     Subsequently, the defective pixel detection unit  51  determines whether the pixel value of the pixel of interest is lower than the saturation level of the pixel of interest (Step S 203 ). When the pixel value of the pixel of interest is lower than the saturation level of the pixel of interest (Step S 203 : Yes), the process proceeds to Step S 204  described below. When the pixel value of the pixel of interest is not lower than the saturation level of the pixel of interest (Step S 203 : No), the process proceeds to Step S 213  described below. Furthermore, the defective pixel detection unit  51  may determine whether all or a part of the pixel of interest and the vicinity of the pixel of interest (for example, the same color pixels (reference pixels) in a range of 5×5) are lower than the saturation level of each of the pixels. In this case, when all or a part of the pixel of interest and the vicinity of the pixel of interest (for example, the same color pixels in a range of 5×5) (including a case where it is determined whether all pixels in the vicinity of the pixel of interest are non-saturated pixels (normal pixels) of which level is lower than the saturation level, and determined that a predetermined number of pixels or more (for example, two or more) have levels lower than the saturation levels of each of the pixels) are lower than the saturation level of each of the pixels, the defective pixel detection unit  51  may advance the process to Step S 204  described below. On the contrary, when all or a part of the pixel of interest and the vicinity of the pixel of interest are not lower than the saturation level of each of the pixels, the defective pixel detection unit  51  may advance the process to Step S 213  described below. 
     In Step S 204 , the defective pixel detection unit  51  calculates a variation degree on the basis of the pixel value of each of the pixel of interest and the vicinity of the pixel of interest (for example, the same color pixel in the range of 5×5). Here, the variation degree is a degree calculated by various statistical methods such as standard deviation, interquartile range, distribution range, or the like. In this case, when the neighboring pixels of the pixel of interest are saturated, the defective pixel detection unit  51  may calculate the variation degree by excluding the pixel value of the saturated pixel. Furthermore, when the neighboring pixels are saturated, the defective pixel detection unit  51  may calculate the variation degree by reducing the weight of the pixels. For example, when calculating the variance (standard deviation) as the variation degree, the defective pixel detection unit  51  may calculate a weighted average for the square of the difference from the average. 
     Subsequently, the defective pixel detection unit  51  determines whether the variation degree is less than a threshold (Step S 205 ). When the variation degree is less than the threshold (Step S 205 : Yes), the defective pixel detection unit  51  advances the process to Step S 206 . When the variation degree is not less than the threshold (Step S 205 : No), the processing advances the process to Step S 213  described below. 
     In Step S 206 , the defective pixel detection unit  51  calculates an average value on the basis of the pixel values of each of the pixel of interest and the vicinity of the pixel of interest (for example, the same color pixel in the range of 5×5). In this case, the defective pixel detection unit  51  may calculate the average value by excluding saturated pixels. Alternatively, the defective pixel detection unit  51  may calculate the weighted average by reducing the weight of the saturated pixels. Furthermore, instead of the average value, the defective pixel detection unit  51  may use another statistic value by which an effect of reducing noise is expected, such as median or an addition average. Furthermore, when the image sensor has a color filter, the defective pixel detection unit  51  may calculate an average value of the pixel value of the pixel of interest and the pixel values of peripheral pixels of the same color as the pixel of interest. 
     Subsequently, the defective pixel detection unit  51  calculates a difference between the pixel value of the pixel of interest and the average value (Step S 207 ). 
     Thereafter, the defective pixel detection unit  51  determines whether the absolute value of the difference between the pixel value of the pixel of interest and the average value is less than a threshold (Step S 208 ). When the absolute value of the difference between the pixel value of the pixel of interest and the average value is less than the threshold (Step S 208 : Yes), the process proceeds to Step S 209  described below. When the absolute value of the difference between the pixel value of the pixel of interest and the average value is not less than the threshold (Step S 208 : No), the process proceeds to Step S 210  described below. 
     In Step S 209 , when the address information of the pixel of interest has been registered as defective pixel information, the defective pixel detection unit  51  cancels the registration of the address information of the pixel of interest. After Step S 209 , the image processing apparatus  30  advances the process to Step S 211  described below. 
     In Step S 210 , when the address information of the pixel of interest has not been registered as defective pixel information, the defective pixel detection unit  51  registers the address information of the pixel of interest. After Step S 210 , the image processing apparatus  30  advances the process to Step S 211  described below. 
     In Step S 211 , the defective pixel detection unit  51  determines whether the address information of the pixel of interest has been registered as defective pixel information. When the address information of the pixel of interest has been registered as defective pixel information (Step S 211 : Yes), the process proceeds to Step S 212  described below. When the address information of the pixel of interest has not been registered as the defective pixel information (Step S 211 : No), the process proceeds to Step S 213  described below. 
     In Step S 212 , the defective pixel detection unit  51  updates the defective pixel level corresponding to the pixel of interest in the defective pixel information by using the difference between the pixel value of the pixel of interest and the average value (here, a value to be used is a value obtained by subtracting the average value from the pixel value of the pixel of interest). In this case, because the difference between the pixel value of the pixel of interest and the average value might may vary due to random noise, a subject, or the like, the defective pixel detection unit  51  may update the defective pixel level while suppressing the variation in the defective pixel level by using a known method. 
     Subsequently, the defective pixel detection unit  51  increments the counter x (x=×+1) indicating the position of the RAW image in the horizontal direction (Step S 213 ), and determines whether the counter x is smaller than the image width of the RAW image (Step S 214 ). When the counter x is smaller than the image width of the RAW image (Step S 214 : Yes), the defective pixel detection unit  51  returns the process to the above-described Step S 203 . When the counter x is not smaller than the image width of the RAW image (Step S 214 : No), the defective pixel detection unit  51  advances the process to Step S 215 . 
     Subsequently, the defective pixel detection unit  51  increments the counter y (y=y+1) indicating the position of the RAW image in the vertical direction (Step S 215 ), and judges whether the counter y is smaller than the image height of the RAW image (Step S 216 ). In a case where the counter y is smaller than the image height of the RAW image (Step S 216 : Yes), the defective pixel detection unit  51  returns to the above-described Step S 202 . In a case where the counter y is not smaller than the image height of the RAW image (Step S 216 : No), the defective pixel detection unit  51  finishes the current process (subroutine) and returns the process to the main routine of  FIG. 6 . 
     In this manner, the defective pixel detection unit  51  performs the defective pixel detection process, and when there is a low saturation pixel among the pixel of interest or neighboring pixels and the neighboring pixel is saturated, the low saturation pixel is excluded by the defective pixel detection unit  51 . Accordingly, it is possible to avoid false detection of a defective pixel, erroneous calculation of the defective pixel level, or a failure in updating the defective pixel level, enabling accurate detection of defective pixel and defective pixel level even when a low saturation pixel exists in the vicinity of the defective pixel. 
     In addition, although the defective pixel detection unit  51  performs cancellation and registration of defective pixels in the defective pixel detection processing of  FIG. 7  (refer to Steps S 209  and S 210 ), each of steps of cancellation and registration of defective pixels may be omitted and update of the defective pixel level may be performed simply for the previously detected defective pixel. Furthermore, because determination of whether the pixel is a defective pixel frequently may switch due to random noise or the like, the defective pixel detection unit  51  may use a known method to suppress the frequent switching by applying hysteresis control or the like. 
     Outline of Defective Pixel Correction Processing 
     Next, details of the defective pixel correction processing described in Step S 102  in  FIG. 6  will be described.  FIG. 8  is a flowchart illustrating an outline of a defective pixel correction process, that is, a flowchart of a subroutine executed by the image processing apparatus  30 . 
     As illustrated in  FIG. 8 , first, the defective pixel correction unit  52  resets a counter y (to y=0) indicating the position of the RAW image in the vertical direction (Step S 301 ) and resets a counter x (to x=0) indicating the position of the RAW image in the horizontal direction (Step S 302 ). Hereinafter, an upper end of the RAW image is set to 0 in the counter y indicating the position of the RAW image in the vertical direction, while a left end of the RAW image is set to 0 in the counter x indicating the position of the RAW image in the horizontal direction. 
     Subsequently, the defective pixel correction unit  52  determines whether the pixel of interest is a defective pixel (Step S 303 ). When the pixel of interest is a defective pixel (Step S 303 : Yes), the process proceeds to Step S 304  described below. When the pixel of interest is not a defective pixel (Step S 303 : No), the process proceeds to Step S 310  described below. 
     In Step S 304 , the defective pixel correction unit  52  subtracts the defective pixel level from the pixel value of the pixel of interest and calculates a correction candidate value. 
     Subsequently, the defective pixel correction unit  52  executes a direction discrimination process of discriminating a direction having a high correlation with the pixel of interest (Step S 305 ). After Step S 305 , the defective pixel correction unit  52  advances the process to Step S 306  described below. 
     Outline of Direction Discrimination Process 
       FIG. 9  is a flowchart illustrating an outline of the direction discrimination process in Step S 304  in  FIG. 8 . As illustrated in  FIG. 9 , the defective pixel correction unit  52  first resets the index i (i=0) indicating the direction (Step S 401 ) and then calculates the correlation value in the i direction (Step S 402 ). 
       FIG. 10A  is a diagram schematically illustrating an example of direction discrimination according to the first embodiment of the present disclosure, in which the direction discrimination direction is a direction 0.  FIG. 10B  is a diagram schematically illustrating an example of direction discrimination according to the first embodiment of the present disclosure, in which the direction discrimination direction is a direction 1.  FIG. 10C  is a diagram schematically illustrating an example of direction discrimination according to the first embodiment of the present disclosure, in which the direction discrimination direction is a direction 2.  FIG. 10D  is a diagram schematically illustrating an example of direction discrimination according to the first embodiment of the present disclosure, in which the direction discrimination direction is a direction 3. 
     As illustrated in  FIGS. 10A to 10D , the defective pixel correction unit  52  calculates, in a range of 9×9, an absolute difference between adjacent pixels (arrows A 1  in  FIGS. 10A to 10D ) of the same color as a pixel of interest P 100  (R pixel) so as to calculate an average of the calculated absolute differences, as a correlation value. In this case, the smaller the correlation value is, the higher the correlation value becomes in that direction. Furthermore, as illustrated in  FIG. 10E , the defective pixel correction unit  52  would not calculate the difference between the adjacent pixels when one of the adjacent pixels is a defective pixel (refer to a pixel P 101  in  FIG. 10E ). Furthermore, in consideration of the saturation level, the defective pixel correction unit  52  may calculate the absolute difference when the adjacent pixels are pixels below the saturation level and when neither is the defective pixel. In  FIGS. 10A to 10D , although the defective pixel correction unit  52  uses the absolute difference between some adjacent pixels in the 9×9 pixels, the absolute differences between all the adjacent pixels may be used. For example, while the defective pixel correction unit  52  uses R pixels in three lines in the case of the horizontal direction in the direction 2 of  FIG. 10C , the R pixels in five lines may be used in the case of  FIG. 10C . Moreover, when the number of calculation of the absolute difference is zero (all are defective pixels etc.), the correlation value in that direction would be the maximum value (value with the lowest correlation) that can be taken as the average value of absolute differences. 
     Thereafter, the defective pixel correction unit  52  increments an index i (i=i+1) indicating the direction (Step S 403 ), determines whether the index i indicating the direction is less than the number of directions (Step S 404 ). When the index i indicating the direction is less than the number of directions (index i is less than four in the exemplary case illustrated in  FIGS. 10A to 10D ) (Step S 404 : Yes), the defective pixel correction unit  52  returns the process back to the above-described Step S 402 . When the index i indicating the direction is not less than the number of directions (Step S 404 : No), the defective pixel correction unit  52  proceeds to Step S 405  described below. 
     In Step S 405 , the defective pixel correction unit  52  finalizes the direction having the highest correlation as a direction discrimination result for the pixel of interest. At this time, when the correlation is the same in all the directions, the defective pixel correction unit  52  finalizes a predetermined direction (for example, a direction in which i=0) as the direction discrimination result. After Step S 405 , the image processing apparatus  30  returns the process back to the subroutine in  FIG. 8 . 
     In this manner, by performing the direction discrimination process, the defective pixel correction unit  52  can determine the direction having high correlation with the pixel of interest even when there is a defective pixel around the pixel of interest. Note that in the case of performing the direction discrimination process, the defective pixel correction unit  52  may perform the direction discrimination process using pixels that are not saturated in consideration of low saturation pixels. With this, even when the low saturation pixel is located around the pixel of interest, direction discrimination may be accurately performed. Furthermore, the defective pixel correction unit  52  needs not necessarily perform the direction discrimination process as long as the correlation between the pixel of interest and the neighboring pixels can be determined by using a known technique, for example, block matching or the like. 
     Returning to  FIG. 8 , Step S 306  and subsequent process will be described. 
     In Step S 306 , the defective pixel correction unit  52  calculates a neighboring interpolation value using pixel values of pixels that are not defective pixels in the direction of high correlation, on the basis of the direction discrimination result. Here, the neighboring interpolation value is defined, for example, as an average value of pixels which are not defective pixels and which are not saturated pixels in a direction of high correlation within a range of 5×5 around the pixel of interest. The neighboring interpolation value may be a weighted average value obtained by using a pixel other than the defective pixel and reducing the weight of the saturated pixel. Alternatively, when there is no unsaturated pixel and non-defective pixel in a direction of high correlation within a range of 5×5 around the pixel of interest, the defective pixel correction unit  52  may use a statistic value of the neighboring pixels, such as median within a range of 5×5 around the pixel of interest, as the surrounding interpolation value. Moreover, when the image sensor has a color filter, the defective pixel correction unit  52  calculates the surrounding interpolation value by using pixel values of the pixel of interest and pixels of the same color as the pixel of interest. 
     Subsequently, the defective pixel correction unit  52  determines whether the correction candidate value is smaller than the neighboring interpolation value (Step S 307 ). When the correction candidate value is smaller than the surrounding interpolation value (Step S 307 : Yes), the defective pixel correction unit  52  outputs the surrounding interpolation value as a pixel value of the pixel of interest (Step S 308 ). More specifically, when the correction candidate value is smaller than the surrounding interpolation value, the defective pixel correction unit  52  outputs the surrounding interpolation value as a correction result because there is a possibility of overcorrection. After Step S 308 , the defective pixel correction unit  52  advances the process to Step S 311  described below. At this time, it is allowable to determine Yes when the difference between the correction candidate value and the neighboring interpolation value is a predetermined value or more, and to determine No when the difference is less than the predetermined value. This makes it possible to prevent overcorrection of a white spot defect and unfinished correction of a black spot defect. That is, overcorrection and unfinished correction can be prevented by comparing the differences. 
     In Step S 307 , when the correction candidate value is not smaller than the neighboring interpolation value (Step S 307 : No), the defective pixel correction unit  52  outputs the correction candidate value as the pixel value of the pixel of interest (Step S 309 ). After Step S 309 , the defective pixel correction unit  52  advances the process to Step S 311  described below. 
     In Step S 310 , when the pixel of interest is not a defective pixel (Step S 310 : No), the defective pixel correction unit  52  outputs the pixel value of the pixel of interest (S 310 ) as it is because correction is unnecessary. After Step S 310 , the defective pixel correction unit  52  advances the process to Step S 311  described below. 
     Subsequently, the defective pixel correction unit  52  increments the counter x (x=x+1) indicating the position of the RAW image in the horizontal direction (Step S 311 ), and determines whether the counter x is smaller than an image width of the RAW image (Step S 312 ). When the counter x is smaller than the image width of the RAW image (Step S 312 : Yes), the defective pixel correction unit  52  returns the process back to the above-described Step S 303 . When the counter x is not smaller than the image width of the RAW image (Step S 312 : No), the defective pixel correction unit  52  advances the process to Step S 313 . 
     Subsequently, the defective pixel correction unit  52  increments the counter y (y=y+1) indicating the position of the RAW image in the vertical direction (Step S 313 ), and determines whether the counter y is smaller than an image height of the RAW image (Step S 314 ). When the counter y is smaller than the image height of the RAW image (Step S 314 : Yes), the defective pixel correction unit  52  returns the process back to the above-described Step S 302 . When the counter y is not smaller than the image height of the RAW image (Step S 314 : No), the defective pixel correction unit  52  finishes the current process (the subroutine) and returns the process to the main routine of  FIG. 6 . 
     In this manner, the defective pixel correction unit  52  can correct defective pixels with high accuracy by performing the above-described process. 
     Outline of Low Saturation Pixel Correction Process 
     Next, details of the low saturation pixel correction process described in Step S 103  in  FIG. 6  will be described in detail.  FIG. 11  is a flowchart illustrating an outline of low saturation pixel correction process, that is, a flowchart of a subroutine executed by the image processing apparatus  30 . 
     As illustrated in  FIG. 11 , first, the low saturation pixel correction unit  53  resets a counter y (to y=0) indicating the position of the RAW image in the vertical direction (Step S 501 ), and resets a counter x (to x=0) indicating the position of the RAW image in the horizontal direction (Step S 502 ). Incidentally, an upper end of the RAW image is set to 0 in the counter y indicating the position of the RAW image in the vertical direction, and a left end of the RAW image is set to 0 in the counter x indicating the position of the RAW image in the horizontal direction. 
     Subsequently, the low saturation pixel correction unit  53  performs interpolation A determination (Step S 503 ). Specifically, the low saturation pixel correction unit  53  counts the number of unsaturated pixels (pixel of interest and the pixel of the same color as the pixel of interest when the image sensor has a color filter) among the pixel values of neighboring pixels of the pixel of interest and determines whether the count value is a threshold or less. When the count value is the threshold or less, the low saturation pixel correction unit  53  sets an interpolation A flag. 
       FIG. 12A  is a diagram schematically illustrating a pixel to be referred to in the interpolation A determination performed by the low saturation pixel correction unit  53 , in which the pixel of interest is a G pixel. In addition, in  FIG. 12A , the pixel at the position described as “ 1 ” around the pixel of interest as the center is defined as a same-color reference pixel when the pixel of interest is a G pixel.  FIG. 12B  is a diagram schematically illustrating a pixel to be referred to in the interpolation A determination performed by the low saturation pixel correction unit  53 , in which the pixel of interest is a R pixel or a B pixel. In addition, in  FIG. 12B , the pixel at the position described as “ 1 ” around the pixel of interest as the center is defined as a same-color reference pixel when the pixel of interest is a R pixel or a B pixel.  FIG. 12C  is a diagram schematically illustrating pixels referred to in the interpolation A determination performed by the low saturation pixel correction unit  53 , in which the image sensor  105  is a monochrome sensor (sequential lighting method using an endoscope). In addition, in  FIG. 12C , pixels around the pixel of interest as a center and described with “ 1 ” are defined as the reference pixels. 
     As illustrated in  FIG. 12A , when the pixel of interest is a G pixel, the low saturation pixel correction unit  53  counts unsaturated pixels among the reference G pixels of in surrounding range of 5×5 and determines whether the count value is a threshold or less. When the count value is the threshold or less, the low saturation pixel correction unit  53  sets the interpolation A flag. In addition, as illustrated in  FIG. 12B , when the pixel of interest is a R pixel or a B pixel, the low saturation pixel correction unit  53  counts unsaturated pixels among the reference pixels having same color as the pixel of interest in the neighboring range of 5×5 and determines whether the count value is a threshold or less. When the count value is the threshold or less, the low saturation pixel correction unit  53  sets the interpolation A flag. Furthermore, as illustrated in  FIG. 12C , when the image sensor  105  is a monochrome sensor, the low saturation pixel correction unit  53  counts the unsaturated pixels among the reference pixels of the eight pixels excluding the pixel of interest in a neighboring range of 3×3, and determines whether the count value is a threshold or less. When the count value is the threshold or less, the low saturation pixel correction unit  53  sets the interpolation A flag. 
     Returning to  FIG. 11 , Step S 504  and subsequent process will be described. 
     In Step S 504 , the low saturation pixel correction unit  53  calculates a median (or an average value) of the saturated same-color reference pixels among the pixel values of neighboring pixels of the pixel of interest and defines the calculated value as an interpolation A interpolation value. Specifically, the low saturation pixel correction unit  53  calculates the median (or the average value) of the saturated same-color reference pixels among the same-color reference pixels (for example, pixels denoted as “ 1 ”) having the same color as the pixel of interest referred to in the above-described interpolation A determination of Step S 503 , as the interpolation A interpolation value. 
     Subsequently, the low saturation pixel correction unit  53  performs interpolation B determination and calculates an interpolation value (Step S 505 ). Specifically, first, the low saturation pixel correction unit  53  calculates a median of non-low saturation pixels (pixels having saturation level higher than a predetermined value) out of the same-color reference pixels. More specifically, similarly to the interpolation A determination in Step S 503  described above, the low saturation pixel correction unit  53  calculates the median of the non-low saturation pixels (pixels having a saturation level higher than a predetermined value) among the same-color reference pixels (for example, pixels denoted as “ 1 ”) having the same color as the referenced pixel of interest. Next, when the pixel of interest is a low saturation pixel, the pixel of interest is saturated, and the pixel of interest is the median or less, the low saturation pixel correction unit  53  sets the interpolation B flag and sets the median to the interpolation B interpolation value. 
     Thereafter, the low saturation pixel correction unit  53  determines whether the interpolation B flag is set to the pixel of interest and whether the pixel value of the pixel of interest is less than the interpolation B interpolation value (Step S 506 ). When the interpolation B flag is set to the pixel of interest and the pixel value of the pixel of interest is less than the interpolation B interpolation value (Step S 506 : Yes), the low saturation pixel correction unit  53  advances the process to Step S 507  described below. When the interpolation B flag is set to the pixel of interest and the pixel value of the pixel of interest is not less than the interpolation B interpolation value (Step S 506 : No), the low saturation pixel correction unit  53  advances the process to Step S 508  described below. 
     In Step S 507 , the low saturation pixel correction unit  53  outputs the pixel value of the pixel of interest as the interpolation B interpolation value. After Step S 507 , the low saturation pixel correction unit  53  advances the process to Step S 511  described below. 
     The low saturation pixel correction unit  53  determines in Step S 508  whether the interpolation A flag is set for the pixel of interest and the pixel value of the pixel of interest is less than the interpolation A interpolation value (Step S 508 ). When the interpolation A flag is set and the pixel value of the pixel of interest is less than the interpolation A interpolation value (Step S 508 : Yes), the low saturation pixel correction unit  53  outputs the pixel value of the pixel of interest as the interpolation A interpolation value (Step S 509 ). After Step S 509 , the low saturation pixel correction unit  53  proceeds to Step S 511  described below. 
     In Step S 508 , when the interpolation A flag is set for the pixel of interest and the pixel value of the pixel of interest is not less than the interpolation A interpolation value (Step S 508 : No), the low saturation pixel correction unit  53  outputs the pixel value of the pixel of interest as it is as an input pixel value (Step S 510 ). After Step S 510 , the low saturation pixel correction unit  53  advances the process to Step S 511  described below. 
     In Step S 511 , the low saturation pixel correction unit  53  increments the counter x (x=x+1) indicating the position of the RAW image in the horizontal direction (Step S 511 ), and judges whether the counter x is smaller than the image width of the RAW image (Step S 512 ). When the counter x is smaller than the image width of the RAW image (Step S 512 : Yes), the low saturation pixel correction unit  53  returns to the above-described Step S 503 . When the counter x is not smaller than the image width of the RAW image (Step S 512 : No), the low saturation pixel correction unit  53  proceeds to Step S 513 . 
     Subsequently, the low saturation pixel correction unit  53  increments the counter y (y=y+1) indicating the position of the RAW image in the vertical direction (Step S 513 ), and determines whether the counter y is smaller than the image height of the RAW image (Step S 514 ). When the counter y is smaller than the image height of the RAW image (Step S 514 : Yes), the low saturation pixel correction unit  53  returns the process to the above-described Step S 502 . When the counter y is not smaller than the image height of the RAW image (Step S 514 : No), the low saturation pixel correction unit  53  finishes the current process (the subroutine) and returns the process to the main routine of  FIG. 6 . 
     In this manner, by performing the low saturation pixel correction process, the low saturation pixel correction unit  53  can achieve sufficient image quality even under the situation where there is a low saturation pixel. 
     According to the first embodiment of the present disclosure described above, it is possible to process the defective pixel even when there is a low saturation pixel around the pixel of interest. 
     In the first embodiment of the present disclosure, the defective pixel detection unit  51  does not update the defective pixel level when the neighboring pixels of the pixel of interest are saturated. However, the defective pixel detection unit  51  may use the defective pixel level detected in the image data of an immediate preceding frame to update the defective pixel level in the image data of the current frame. In this case, the defective pixel level detected in the image data of an immediate succeeding frame may be used to update the defective pixel level in the image data of the current frame. 
     Modification of First Embodiment 
     Next, a modification of the first embodiment of the present disclosure will be described. A modification of the first embodiment of the present disclosure has a configuration similar to that of the imaging system  1  according to the above-described first embodiment, with a difference in the defective pixel detection processing executed by the defective pixel detection unit  51 . Specifically, in the modification of the first embodiment of the present disclosure, the calculation method for calculating the average value is different. Hereinafter, defective pixel detection processing executed by a defective pixel detection unit according to the modification of the first embodiment of the present disclosure will be described. A same reference sign will be given to the portions identical to the configuration of the imaging system  1  according to the above-described first embodiment, and description for this will be omitted. 
       FIG. 13  is a flowchart illustrating an outline of a defective pixel detection process executed by the defective pixel detection unit  51  according to the modification of the first embodiment of the present disclosure. Steps S 601  to S 605  in  FIG. 13  respectively correspond to Steps S 201  to S 205  described above. 
     In Step S 606 , the defective pixel detection unit  51  calculates an average value on the basis of the pixel value of each of the pixel of interest and the vicinity of the pixel of interest (pixel value of the same color pixel when the image sensor has a color filter). 
       FIG. 14  is a diagram schematically illustrating a method of calculating the average value by the defective pixel detection unit  51 . In  FIG. 14 , each of a pixel of interest P 100 , a defective pixel or saturated pixel P 200 , and a pixel P 201  is represented by hatching. Additionally, a pixel P 301  positioned point-symmetrically opposite to the defective pixel or saturated pixel P 200  across the pixel of interest P 100  and a pixel P 300  positioned point-symmetrically opposite to the pixel P 201  across the pixel of interest P 100  are represented by hatching. 
     The defective pixel detection unit  51  calculates an average value by excluding the defective pixel or saturated pixel P 200  and additionally excluding the pixel P 301  positioned point-symmetrically opposite to the pixel P 200  across the pixel of interest P 100  illustrated in  FIG. 14 . When excluding part of pixels leads to crossover in an edge, the edge may be influential. However, by excluding the pixel positioned point-symmetrically, it is possible to eliminate the effect. Here, in the case of excluding the pixel P 201  illustrated in  FIG. 14 , the defective pixel detection unit  51  also excludes the pixel P 300  positioned point-symmetrically opposite to the pixel  201  across the pixel of interest P 100  in a similar manner, similarly to the case of the pixel P 200 . Note that, instead of the average value, the defective pixel detection unit  51  may use another statistic value by which an effect of reducing noise is expected, such as a median or a weighted average value. With this operation, it is possible to calculate an appropriate average value in the vicinity of the edge while suppressing the influence of the edge. 
     Steps S 607  to S 616  respectively correspond to Steps S 207  to S 216  described above. 
     According to the modification of the first embodiment of the present disclosure described above, the defective pixel detection unit  51  excludes the defective pixel or the pixel positioned point-symmetrically opposite to the saturated pixel across the pixel of interest, and calculates the average value. Accordingly, it is possible to calculate an appropriate average value while suppressing the influence of the edge in the vicinity of the edge. 
     Second Embodiment 
     Next, a second embodiment of the present disclosure will be described. An imaging system according to the second embodiment has a configuration different from that in the imaging system according to the first embodiment, and executes a process different from that executed by the imaging system  1  according to the first embodiment. Hereinafter, the configuration of the imaging system according to the second embodiment will be described, and thereafter a process executed by the image processing apparatus according to the second embodiment will be described. A same reference sign will be given to the configuration identical to the configuration of the imaging system according to the above-described first embodiment, and description for this will be omitted. 
     Configuration of Imaging System 
       FIG. 15  is a block diagram schematically illustrating a configuration of an imaging system according to the second embodiment of the present disclosure. An imaging system  1   a  illustrated in  FIG. 15  includes an image processing apparatus  30   a  in place of the image processing apparatus  30  of the imaging system  1  according to the first embodiment described above. The image processing apparatus  30   a  includes a noise processing unit  32   a  in place of the noise processing unit  32  of the image processing apparatus  30  according to the first embodiment described above. The noise processing unit  32   a  includes a defective pixel detection unit  51  and a defective pixel correction unit  52 . 
     Process in Image Processing Apparatus 
     Next, a process executed by the image processing apparatus  30   a  will be described.  FIG. 16  is a flowchart illustrating an outline of a noise process executed by the image processing apparatus  30   a , that is, a flowchart of a main routine executed by the image processing apparatus  30   a . Steps S 701  and S 702  in  FIG. 16  correspond to Steps S 101  and S 10  in above-described  FIG. 6 , respectively. 
     According to the second embodiment of the present disclosure described above, it is possible to process a defective pixel even when there is a low saturation pixel around the pixel of interest. 
     Other Embodiments 
     The present disclosure is not limited to the above-described embodiments, but various modifications and further applications are available within the scope of the present disclosure. For example, besides the imaging apparatus used in the description of the present disclosure, the present disclosure can be applied to any apparatus capable of shooting an image of a subject such as a mobile apparatus having an image sensor of a mobile phone or a smartphone or an imaging apparatus that images the subject by using an optical device such as a video camera, an endoscope, a surveillance camera, or a microscope. 
     Furthermore, in the present disclosure, the defective pixel information recording unit and the low saturation pixel information recording unit are provided in the imaging apparatus. Alternatively, however, these units may be provided in the above-described image processing apparatus, the lens unit, the endoscope main body, or the like. Moreover, the defective pixel information recording unit and the low saturation pixel information recording unit may be provided in a server capable of bidirectional communication via a network, thereby obtaining Random Telegraph Signal (RTS) noise information via the network. 
     Furthermore, in the present disclosure, it is also allowable to provide the image processing apparatus of the present disclosure in a control apparatus (processor) connected with an endoscope including an imaging apparatus at a distal end portion of an insertion portion insertable into a subject. 
     Furthermore, in the present disclosure, each of the defective pixel detection unit, the defective pixel correction unit, and the low saturation pixel correction unit is provided in the image processing apparatus. However, it is sufficient that at least one of these is provided. 
     Moreover, the present disclosure is applicable to image data other than the image data used for display and recording. For example, the present disclosure is applicable to the image data in an OB area, and image data in an area outside the image circle, with no optical design assurance. 
     Moreover, in the description of the flowcharts for the operations described above in the present specification, terms such as “first”, “next”, “subsequently”, and “thereafter” are used to describe operation for convenience. These do not denote, however, that the operations need to be performed in this order. 
     Moreover, the processes performed by the image processing apparatus in the above-described embodiments, that is, any of the processes illustrated in the flowcharts may be recorded as a program that can be executed by a control unit such as a CPU. In addition, it is possible to distribute by storing in a recording medium of the external recording device, such as memory cards (ROM card, RAM card, etc.), a magnetic disk (floppy disk (registered trademark), hard disk, etc.), an optical disc (CD-ROM, DVD, etc.), or a semiconductor memory. The control unit such as a CPU reads the program recorded in the recording medium of the external recording device and controls the operation by the read program to execute the above-described process. 
     Moreover, note that the present disclosure is not limited to the above-described embodiments and modifications just as they are, but can be embodied by modifying the components without departing from the scope of the disclosure at a stage of implementation of the disclosure. Furthermore, a plurality of components disclosed in the above-described embodiments may be appropriately combined to form various embodiments. For example, some components may be omitted from the all the components described in the embodiments and the modifications. Furthermore, the components described in each of the embodiments and modifications may be appropriately combined with each other. 
     Moreover, a term used at least once in the specification or the drawings, associated with another term having a broader or similar meaning, may be substituted by the another term anywhere in the specification and the drawings. In this manner, various modifications and further application may be implemented within a scope that does not depart from the present disclosure. 
     According to the present disclosure, it is possible to accurately process a defective pixel even when there is a low saturation pixel around the pixel of interest. 
     Additional advantages and modifications will readily occur to those skilled in the art. Therefore, the disclosure in its broader aspects is not limited to the specific details and representative embodiments shown and described herein. Accordingly, various modifications may be made without departing from the spirit or scope of the general inventive concept as defined by the appended claims and their equivalents.