Abstract:
Disclosed is an image processor that reduces or eliminates the effects of a flash band regardless of performing a shutter operation or not. An imaging device includes a flash band detector configured to determine whether each image of the consecutive images shown by image signals output from an imager is a flash band image a part of which is affected by a flash, and a flash band compensator configured to, if a first image is determined as the flash band image, generate a third image which is supposed where the third image is fully affected by a flash by using a second image which is not determined as the flash band image, and output the third image instead of the first image.

Description:
CROSS-REFERENCE TO RELATED APPLICATION 
     This application claims priority to Japanese Patent Applications No. 2013-030717 filed on Feb. 20, 2013, and No. 2013-078655 filed on Apr. 4, 2013, the entire disclosure of which is incorporated by reference herein. 
     BACKGROUND 
     The present disclosure relates to image processors represented by imaging devices such as digital cameras and video cameras (camcorders) for capturing consecutive images. 
     Many complementary metal oxide semiconductor (CMOS) imaging elements use a rolling shutter technique for allowing exposure to sequentially start for each horizontal line (hereinafter referred to as “line”), and sequentially reading image signals of respective lines to form a frame. In the imaging element using the rolling shutter technique, the starting time and ending time of the exposure period differ for each line. Therefore, if an external flash etc. is used during a period that is shorter than one frame period, a band of high luminance (a flash band) may appear on an output image of the imaging element. 
     International Patent Publication No. WO 2010/100866 discloses a technique for compensating for a flash band by adding image signals of two consecutive frames affected by flash. 
     SUMMARY 
     The present disclosure provides an image processor that can reduce an adverse effect due to flash regardless of performing a shutter operation or not. 
     An image processor of the present disclosure processes consecutive images, and includes: a determiner configured to determine whether each image of the consecutive images is a flash band image a part of which is affected by a flash; and a compensator configured to, if a first image is determined as the flash band image by the determiner, generate a third image which is supposed where the third image is fully affected by a flash by using a second image which is not determined as the flash band image, and output the third image instead of the first image. 
     The image processor of the present disclosure can reduce or eliminate the effects of a flash band regardless of performing a shutter operation or not. 
    
    
     
       BRIEF DESCRIPTION OF THE DRAWINGS 
         FIG. 1  illustrates a general configuration of an imaging device that is an example of an image processor according to a first embodiment. 
         FIG. 2  illustrates a detailed configuration of a flash band compensator of  FIG. 1 . 
         FIG. 3  illustrates a timing diagram of an operation example 1 of the imaging device in  FIG. 1  where a flash band occurs in only one frame in an image signal In of  FIG. 1  when a shutter operation is performed. 
         FIGS. 4A ,  4 B,  4 C, and  4 D illustrate the operation of the flash band compensator of  FIG. 1  in the operation example 1. 
         FIG. 5  illustrates a timing diagram of an operation example 2 of the imaging device in  FIG. 1  where a flash band occurs in each of two consecutive frames in the image signal In of  FIG. 1  when a shutter operation is not performed. 
         FIG. 6  illustrates another timing diagram of the operation example 2 of the imaging device in  FIG. 1  where a flash band occurs in each of two consecutive frames in the image signal In of  FIG. 1  when a shutter operation is performed. 
         FIGS. 7A ,  7 B,  7 C, and  7 D illustrate the operation of the flash band compensator of  FIG. 1  in the operation example 2. 
         FIGS. 8A ,  8 B,  8 C, and  8 D illustrate the operation of the flash band compensator of  FIG. 1  in an operation example 3 where a frame to which full-frame flash is emitted is provided between two frames each including a flash band in the image signal In of  FIG. 1 . 
     
    
    
     DETAILED DESCRIPTION 
     Embodiments are described in detail below with reference to the attached drawings. However, unnecessarily detailed description may be omitted. For example, detailed description of well-known techniques or description of the substantially same elements may be omitted. Such omission is intended to prevent the following description from being unnecessarily redundant and to help those skilled in the art easily understand it. 
     Inventor(s) provides the following description and the attached drawings to enable those skilled in the art to fully understand the present disclosure. Thus, the description and the drawings are not intended to limit the scope of the subject matter defined in the claims. 
     First Embodiment 
     1-1. Configuration 
     1-1-1. General Configuration of Imaging Device 
     First, an imaging device in a first embodiment will be described. A technique for compensating for a flash band in the present disclosure is applied to an image processor, and in this embodiment, the imaging device will be described as one example of the image processor. 
       FIG. 1  illustrates the configuration of an imaging device  100  in the first embodiment. In  FIG. 1 , the imaging device  100  includes an imager  110  configured to image an object to generate an image signal, a flash band detector  120  configured to detect presence or absence of a flash band, and a flash band compensator  130  configured to correct the flash band based on flash band detection information. 
     The imager  110  has a CMOS image sensor (imaging elements), a sensor drive circuit, and an analog/digital converter. The imager  110  converts light from an object into electric signals by photoelectric conversion, and supplies the flash band detector  120  and the flash band compensator  130  with an image signal In as a digital image signal. 
     The flash band detector  120  detects presence or absence of a flash band based on the image signal In input from the imager  110 , and outputs flash band determination information J showing presence or absence of a flash band, and a flash band level L to the flash band compensator  130 . The flash band level L means luminance of a flash band. In this embodiment, if there is a flash band, the flash band determination information J is expressed as follows: J=1. If there is no flash band, the flash band determination information J is expressed as follows: J=0. The flash band detector  120  sets a threshold value and detects presence or absence of a flash band, and a thin flash band can be neglected and the flash band determination information J can be considered as 0. 
     The flash band compensator  130  outputs an images Out depending on presence or absence of a flash band by utilizing the image signal In of the imager  110 , the flash band determination information J and the flash band level L of the flash band detector  120 . 
     1-1-2. Configuration of Flash Band Compensator 
       FIG. 2  illustrates a detailed configuration of the flash band compensator  130 . The flash band compensator  130  includes a memory  131  and a pseudo-flash generator  132 . 
     The memory  131  refers to flash band determination information J output from the flash band detector  120 , and stores and outputs an image signal M of the latest frame, in which no flash band occurs, of previous frames. 
     The pseudo-flash generator  132  outputs an image Out in which a flash band is compensated for based on the image signal M, output from the memory  131 , of the frame in which no flash band occurs, and the flash band determination information J and the flash band level L output from the flash band detector  120 . 
     If the flash band determination information J is 0, the pseudo-flash generator  132  does not process anything, and outputs the image signal M output from the memory  131  without any change. 
     If the flash band determination information J is 1, the pseudo-flash generator  132  generates pseudo-flash for the output M output from the memory  131 . 
     In order to generate a pseudo-flash image, the pseudo-flash generator  132  performs an offset process and a gain process for the output M from the memory  131 . 
     A gain value to be multiplied and an offset value to be added are changed depending on luminance of an input image M. In view of the properties that when an object colored with black or white is lighted, the black object is less likely to reflect light while the white object is more likely to reflect light, the gain value to be multiplied is reduced if the luminance is smaller, while the gain value to be multiplied is increased if the luminance is larger. The gain value corresponding to a level of the luminance signal is multiplied for each pixel. The gain value is also changed depending on a flash band level L output from the flash band detector  120 . The output value of each pixel can be expressed as follows:
 
 YO=YI×GY×GL+OY  
 
     where YI is the luminance level of each input pixel, YO is the luminance level of each output pixel, GY is the gain value corresponding to luminance, GL is the gain value corresponding to the flash band level L, and OY is the offset value corresponding to the luminance level. 
     1-2. Operation of Imaging Device 
     Next, examples of the operation of the imaging device will be described. 
     1-2-1. Operation Example 1 
       FIG. 3  shows a case where a flash band occurs in only one frame in an image signal In of  FIG. 1  when a shutter operation is performed. As shown in the figure, in imaging devices using a rolling shutter technique, the larger the line number is, the wider the shift of the exposure period in the line from the exposure period in line  1  is. Hatching inclined downward to the left in  FIG. 3  shows a non-exposure period due to a shutter operation. When flash is emitted under such a condition at the timing shown in  FIG. 3 , a flash band occurs in only the lower half of one frame image, as shown by hatching inclined downward to the right in the one frame image. 
     The technique for adding image signals of two consecutive frames in International Patent Publication No. WO 2010/100866 cannot correct such a flash band occurring in only one frame. 
       FIGS. 4A-4D  shows specific examples of a flash band compensation in the present disclosure when a flash band occurs in only one frame. The numbers shown in  FIG. 4A  indicate a frame number of an input signal. 
     The flash band detector  120  outputs portions of flash band determination information J 1 -J 4  with respect to In 1 -In 4  output from the imager  110 . In the case of  FIG. 4B , a flash band occurs in only the frame number 2, J 2  is 1, and J 1 , J 3 , and J 4  are 0. The flash band compensator  130  refers to flash band determination information J output from the flash band detector  120  in the memory  131 , and outputs an image M including no flash band. In the case of  FIG. 4C , J 2  is 1, and therefore, M 2  is In 1 . J 1 , J 3 , and J 4  are 0, and therefore, M 1  is In 1 , M 3  is In 3 , and M 4  is In 4 . 
     The image M including no flash band output from the memory  131  is input into the pseudo-flash generator  132 . In the case of  FIG. 4D , J 2  is 1, and J 1 , J 3 , and J 4  are 0, and therefore, a pseudo-flash process is performed with respect to only M 2 , and Out 2  (M 1 ′) is output. The prime symbol (′) indicates that the pseudo-flash process is performed. M 1 , M 3 , and M 4  are output as Out 1 , Out 3 , and Out 4 , respectively, without any change. 
     As a result, as shown in the outputs of  FIG. 4D , an image on which no flash band appears can be obtained. 
     1-2-2. Operation Example 2 
       FIG. 5  shows a case where a flash band occurs in each of two consecutive frames in the image signal In of  FIG. 1  when a shutter operation is not performed. A sensor reading timing and a shutter timing coincides with each other, and the sensor can fully expose flashed images to light, and flash bands are vertically complementary. 
       FIG. 6  shows a case where a flash band occurs in each of two consecutive frames in the image signal In of  FIG. 1  when a shutter operation is performed. In this case, there is a time at which flashed images are not exposed to the sensor, and as a result, non-complementary flash bands occur in a vertical direction. 
     The technique for adding image signals of two consecutive frames in International Patent Publication No. WO 2010/100866 can correct the complementary flash bands as shown in  FIG. 5 . However, such a technique cannot correct the non-complementary flash bands as shown in  FIG. 6 . In other words, when a shutter operation is performed, flash bands are not necessarily complementary, and in such a technique, if the image signals of the frames are added, a part of the image is disadvantageously dark. 
     The present disclosure can solve this problem as described below. 
       FIGS. 7A-7D  show an example of a flash band compensation in the present disclosure when a flash band occurs in each of two frames. 
     The flash band detector  120  outputs portions of flash band determination information J 1 -J 4  with respect to In 1 -In 4  ( FIG. 7A ) output from the imager  110 . In the case of  FIG. 7B , a flash band occurs in each of frame numbers 2, and 3, and therefore, J 2  is 1, J 3  is 1, and J 1  and J 4  are 0. 
     The flash band compensator  130  refers to flash band determination information J output from the flash band detector  120  in the memory  131 , and outputs an image M including no flash band. In the case of  FIG. 7C , J 2  and J 3  are 1, and therefore, M 2  and M 3  are In 1 . J 1  and J 4  are 0, and therefore, M 1  is In 1 , and M 4  is In 4 . 
     The image M including no flash band and output from the memory  131  is input into the pseudo-flash generator  132 . In the case of  FIG. 7D , J 2  and J 3  are 1, and J 1  and J 4  are 0, and therefore, a pseudo-flash process is performed with respect to M 2  and M 3 , and Out 2  (M 1 ′) and Out 3  (M 1 ′) are output. M 1  are M 4  are output as Out 1  and Out 4 , respectively, without any change. 
     As a result, as shown in the outputs of  FIG. 7D , an image on which no flash band appears can be obtained regardless of performing a shutter operation or not. 
     1-2-3. Operation Example 3 
       FIGS. 8A-8D  show a case where a frame to which full-frame flash is emitted is provided between two frames each including a flash band. The full-frame flash means that the effect of the flash appears on the entire one frame. This case may occur in, e.g., a press conference in which many reporters emit flash in a scene prior to or after a specific scene. 
     The flash band detector  120  outputs portions of flash band determination information J 1 -J 4  with respect to In 1 -In 4  ( FIG. 8A ) output from the imager  110 . In the case of  FIG. 8B , a flash band occurs in each of frame numbers 2, and 4, and therefore, J 2  is 1, and J 4  is 1, and J 1  and J 3  are 0. 
     The flash band compensator  130  refers to flash band determination information J output from the flash band detector  120  in the memory  131 , and outputs an image M including no flash band. In the case of  FIG. 8C , J 2  and J 4  are 1, and therefore, M 2  is In 1  and M 4  is In 3 . J 1  and J 3  are 0, and therefore, M 1  is In 1 , and M 3  is In 3 . 
     The image M including no flash band output from the memory  131  is input into the pseudo-flash generator  132 . In the case of  FIG. 8D , J 2  and J 4  are 1, and J 1  and J 3  are 0, and therefore, a pseudo-flash process is performed with respect to M 2  and M 4 , and Out 2  (M 1 ′) and Out 4  (M 3 ′) are output. M 1  are M 3  are output as Out 1  and Out 3 , respectively, without any change. 
     As a result, as shown in the output of  FIG. 8D , an image on which no flash band appears can be obtained. 
     1-3. Advantages 
     The imaging device  100  in this embodiment can correct flash bands even if the flash bands are non-complementary flash bands occurring under an environment with a shutter operation. 
     Other Embodiments 
     As described above, the first embodiment has been described as example techniques disclosed in the present application. However, the techniques according to the present disclosure are not limited to this embodiment, but are also applicable to embodiments where modifications, substitutions, additions, and omissions are made as appropriate. In addition, elements described in the first embodiment may be combined to provide a different embodiment. 
     Here, other embodiments will be now described. 
     In the first embodiment, the example has been described where the flash band compensation technique of the present disclosure is applied to an imaging device as one example of an image processor. However, the flash band compensation technique of the present disclosure is also applicable to image processors such as computers. 
     In the first embodiment, the case has been described where the gain value corresponding to luminance and the flash band level L is multiplied or the offset value corresponding to luminance and the flash band level L is added. However, the gain value or the offset value may be constant. A flash image may be generated by using a lookup table, not the addition or multiplication. GY has been described as the gain value whose level corresponds to a level of a luminance signal. However, GY may be a gain value whose level corresponds to a level of, e.g., color signals such as RGB. 
     In the first embodiment, the case has been described where the flash band detector  120  does not detect full-frame flash. If the full-frame flash can be detected, a pseudo-flash process may not be performed when an output image M output from the memory  131  corresponds to a frame to which full-frame flash is emitted. 
     In the first embodiment, the flash band compensator  130  performs a pseudo-flash process with respect to all the frames in which J is 1. However, the pseudo-flash process does not have to be performed with respect to all the frames in which J is 1. For example, if three or more frames in which J is 1 are consecutively provided, no pseudo-flash process is performed after the third frame, and image signals of frames after the third frame may be output without any change in view of reduction of uncomfortable viewing due to a temporary halt of the consecutive images. 
     The number of the frames used for the pseudo-flash process is optional. It is possible to use a frame posterior to the frame that is to be compensated for. However, it is convenient to use a frame prior to the frame that is to be compensated for rather than the posterior frame. 
     Various embodiments have been described above as example techniques of the present disclosure, in which the attached drawings and the detailed description are provided. 
     As such, elements illustrated in the attached drawings or the detailed description may include not only essential elements for solving the problem, but also non-essential elements for solving the problem in order to illustrate such techniques. Thus, the mere fact that those non-essential elements are shown in the attached drawings or the detailed description should not be interpreted as requiring that such elements be essential. 
     Since the embodiment described above is intended to illustrate the techniques in the present disclosure, it is intended by the following claims to claim any and all modifications, substitutions, additions, and omissions that fall within the proper scope of the claims appropriately interpreted in accordance with the doctrine of equivalents and other applicable judicial doctrines. 
     The present disclosure provides a pseudo-flash image that can be replaced for a flash band image, and therefore, is applicable to image processors that output images in which realistic flash remains in news scene etc. The present disclosure is also applicable to image processors that correct a flash band even if a shutter is provided to reduce flicker.