Abstract:
The disclosure provides for globally detecting and correcting unwanted sensor artifacts arising from processing techniques combining multiple visual images into an output image while possessing information relating to a localized area of the image that can be accomplished during scanning of the image and independent of the number of images or the computer processing method used to acquire them.

Description:
CROSS-REFERENCE TO RELATED APPLICATIONS 
       [0001]    This application claims the benefit under 35 U.S.C. §119(e) to U.S. Provisional Patent Application No. 62/362,766, filed on Jul. 15, 2016 in the United States Patent &amp; Trademark Office, the disclosure of which is incorporated by reference herein in its entirety. 
     
    
     TECHNICAL FIELD 
       [0002]    This disclosure is in the technical field of visual image processing. 
       DISCUSSION OF RELATED ART 
       [0003]    Visual image correction techniques (including de-ghosting) can be used with high dynamic range (HDR) images and other scenes generated from multiple visual images with different image sensor exposure times. The most common way to create an HDR scene is to capture (and then combine) multiple individual images depicting a visual scene which are created using different exposure times. Movement of an object in the scene occurring between different captures of an image (or movement of the camera sensor itself) may cause unwanted visual artifacts (such as ghosting or thin line) which should not be present but which nonetheless appear in the resulting visual image. Motion-related artifacts may also occur in de-noising algorithms (or other techniques) requiring a combination of multiple renderings of a scene to form a single visual image. 
         [0004]    Motion compensation uses precise detection of the motion and consistent correction over the image when combining multiple visual images into a single resulting image. Detecting unwanted visual artifacts based solely on light radiation intensity differences may allow false negatives and/or false positives; e.g., motion-related artifacts with similar intensities might remain in the resulting image despite computer processing designed to remove them and/or newly-appearing objects (such as those created by a noisy camera sensor environment) might be detected incorrectly as motion. 
         [0005]    Many techniques used for visual image correction utilize storing of the full image in computer buffer memory prior to removing such artifacts and can be performed only during post-image capture processing. Other real-time visual image processing techniques use special computer hardware to accomplish correction and sacrifice optimum image resolution. An alternate technique assigns to a visual image location (e.g., pixel) the light radiation intensity value(s) for which a majority of the individual image(s) are in agreement but this method uses a large number of images with at least some level of redundancy between them. 
       SUMMARY OF THE INVENTION 
       [0006]    Exemplary embodiments of the invention as described herein generally provide for detecting and correcting unwanted sensor artifacts arising from processing techniques utilizing multiple visual images while possessing information relating to a localized area of the image that can be accomplished during scanning of the image and independent of the number of images or the computer processing method used to acquire them. 
         [0007]    According to an exemplary aspect of the invention, a set of two or more visually inconsistent images are combined as an input from an image sensor and a single corrected image is output to eliminate any artifact(s) appearing in one (or more) of the input image(s) as follows:
       For different images of a visual scene a local area is examined surrounding a reference pixel found substantially within the center of an array based at an identified image location to detect whether an artifact is present.   Correction is performed to remove an artifact that is detected based upon evaluating a statistical (e.g., aggregation and/or variance) analysis of light radiation intensity value difference(s) between different image combination(s) at the reference pixel location.   The image is optionally processed using signal-to-noise ratio reduction techniques to output a single corrected visual image that eliminates any one or more of the artifact(s) appearing in a detected region of at least one (or more) of the input image(s).       
 
         [0011]    According to a further exemplary aspect of the invention, the detection and correction steps are divided into sub-steps relating to discovery of a new object and/or simple motion. If a new object is detected then a new object correction sub-step can find a stable image to use while ignoring others. If simple motion is detected then a simple motion correction sub-step can extend use of at least one previous image selection decision to maintain consistency. 
         [0012]    According a to further exemplary aspect of the invention, there is provided a computer device containing software program instruction(s) executable by the computer to perform at least the foregoing operation(s) for detecting and correcting unwanted artifact(s) arising from technique(s) utilizing multiple visual image(s) input from a sensor and processed into an output image depicting a scene. 
         [0013]    The subject matter which is regarded as the invention is particularly pointed out and distinctly claimed in the concluding portion of the specification, as set forth in claims which are to be interpreted as broadly as the law permits to cover the full scope of the invention, including all equivalents thereto. 
     
    
     
       BRIEF DESCRIPTION OF DRAWINGS 
         [0014]      FIG. 1 —Photographic illustrations of false negatives and/or false positives appearing in HDR digital visual images as a result of motion artifacts. 
           [0015]      FIG. 2 —Photographic illustrations of input visual images (A+B) that are processed to form an output map (C) using new object detection. 
           [0016]      FIG. 3 —Photographic illustrations of input visual images (A+B) that are processed to form an output map (C) using simple motion detection. 
           [0017]      FIG. 4 —Illustrations of images photographed from a moving car and combined to form a resulting HDR image with (and without) motion correction. 
           [0018]      FIG. 5 —Illustrations of two blurry images photographed with different exposure times and shown after correction. 
           [0019]      FIG. 6 —A schematic diagram of a general-purpose computer configured to execute software instructions programmed to process data for detecting and correcting at least one artifact arising from processing multiple visual images depicting a scene. 
           [0020]      FIGS. 7 &amp; 8 —Flowcharts depicting a general process for detecting and correcting at least one artifact arising from processing multiple visual images depicting a scene. 
       
    
    
     DETAILED DESCRIPTION OF EXEMPLARY EMBODIMENTS 
       [0021]      FIG. 1  shows photographic illustrations of false negatives that might appear in high dynamic range (HDR) visual images as a result of motion artifacts (such as ghosting or thin line) along with false positives taking the form of newly-appearing objects such as those created by a camera (or other image sensor) noise environment which appear in the resulting image. As shown with reference to  FIGS. 2 &amp; 3 , a visual image  10  is comprised of an array  20  of pixel location(s)  30  that may be subjected to varying light radiation intensity (S) value(s) created through use of different image sensor exposure (E) time(s). 
         [0022]    An image detection and correction method according to an exemplary embodiment disclosed herein assumes N&gt;1 images S 1  . . . S N  taken from the same visual scene with different exposure times E 1  . . . E N  (respectively) and for an image sensor signal (S) there is assigned (as S x,y ) the light radiation intensity value of the pixel at location (x, y) in the subject image array. For simplicity in the examples below the following assumptions are made:
       The images are sorted from the image with the shortest exposure time (S 1 ) to the one with the longest exposure time (S N ) such that for i&gt;j then E i &gt;E j .   An image is normalized according to its exposure time; e.g., the image (S i ) is captured as (Ŝ i  ) and its overall intensity value converted to       
 
         [0000]    
       
         
           
             ( 
             
               
                 S 
                 i 
               
               = 
               
                 
                   
                     S 
                     i 
                   
                   ^ 
                 
                 · 
                 
                   C 
                   
                     E 
                     i 
                   
                 
               
             
             ) 
           
         
       
     
         [0000]    for some constant C&gt;0. 
         [0025]    In the examples below, the visual images  10  are digitally scanned in a row-by-row (e.g., raster) arrayed order where the steps of the scan use only a small localized exposure area (e.g., patch) of each image around the scanned pixel. The edge size of the patch (in pixels) is denoted as P (such that the whole pixel patch for example would be of array size P×P) where the reference pixel  30  around which the patch is defined is substantially in the center of the pixel array  20 . 
         [0026]    In the examples below, two steps are used to correct an unwanted artifact with the first being its detection and the second being correction of the visual image in the detected region as follows:
       For a given pixel position in an input image a local exposure patch (e.g., 7×7 pixels) is examined surrounding the subject reference pixel located centrally within the array to detect whether an artifact arising from an inconsistency between combined image(s) is present.   If an artifact is detected then correction is performed where an HDR algorithm can optionally be used to output a single resulting image.       
 
         [0029]    For example, the following HDR algorithm can optionally be used to minimize the output image signal-to-noise ratio (SNR) by examining as the output pixel at point (x,y) the value 
         [0000]    
       
         
           
             
               ( 
               
                 O 
                 
                   x 
                   , 
                   y 
                 
               
               ) 
             
             = 
             
               
                 ∑ 
                 
                   i 
                   = 
                   1 
                 
                 K 
               
                
               
                 
                   
                     E 
                     i 
                   
                   
                     
                       ∑ 
                       
                         j 
                         = 
                         1 
                       
                       K 
                     
                      
                     
                       E 
                       j 
                     
                   
                 
                  
                 
                   S 
                   i 
                 
               
             
           
         
       
     
         [0000]    where (K)=argmax k {S x,y   k  is not burnt} is the numeric index of an input image with a maximum exposure time (E) that yields a defined useful light radiation intensity value for (S) at pixel (x,y) and (i,j,k) are natural numeric counting value(s) defined within the range set by the equation and where the algorithm assumes depiction of the same visual scene in all input images without any motion artifacts occurring. 
         [0030]    Flowcharts depicting a general process for detecting and correcting at least one artifact arising from processing multiple visual images depicting a scene according to the exemplary embodiment(s) described herein are shown in  FIGS. 7 &amp; 8 . In the examples below, the detection step  70  and correction step  80  are both divided into sub-steps of new object detection/correction  73 / 83  and simple motion detection/correction  76 / 86  each having a different purpose in unwanted artifact resolution. The new object detection step  73  can identify initiation of a false positive difference between images and if a new object is detected then the new object correction step  83  can find a stable single image pixel patch to use in a corrected output image  90  while ignoring others. The simple motion detection step  76  can identify any difference between two image pixel patches (whether created due to motion or not) and if simple motion is detected then the simple motion correction step  86  can extend use of at least one of the previous image selection decisions to maintain consistency. The interplay between these steps allows artifact correction with differing levels of certainty. These techniques are described in the below examples with reference to two (2) input visual images  50 / 53  in a combination  60  but can be easily extended to use with more than two input images as will be further explained below. 
       New Object Detection 
       [0031]    Given two visually inconsistent images having light radiation intensity value(s) (S) and (R) a decision criterion for new object detection around reference pixel (i,j) within a pixel patch array of size (P×P) according to an exemplary embodiment includes: 
         [0000]    
       
         
           
             
               
                 D 
                 
                   S 
                   , 
                   R 
                 
                 o 
               
                
               
                 ( 
                 
                   i 
                   , 
                   j 
                 
                 ) 
               
             
             = 
             
               
                 Var 
                 ( 
                 
                   
                     { 
                     
                        
                       
                         
                           
                             1 
                             
                               w 
                               2 
                             
                           
                            
                           
                             
                               ∑ 
                               
                                 
                                   k 
                                   = 
                                   0 
                                 
                                 , 
                                 … 
                                  
                                 
                                     
                                 
                                 , 
                                 
                                   w 
                                   - 
                                   1 
                                 
                               
                               
                                   
                               
                             
                              
                             
                               
                                 ∑ 
                                 
                                   
                                     t 
                                     = 
                                     0 
                                   
                                   , 
                                   … 
                                    
                                   
                                       
                                   
                                   , 
                                   
                                     w 
                                     - 
                                     1 
                                   
                                 
                                 
                                     
                                 
                               
                                
                               
                                 S 
                                 
                                   
                                     m 
                                     + 
                                     k 
                                   
                                   , 
                                   
                                     n 
                                     + 
                                     t 
                                   
                                 
                               
                             
                           
                         
                         - 
                         
                           R 
                           
                             
                               m 
                               + 
                               k 
                             
                             , 
                             
                               n 
                               + 
                               t 
                             
                           
                         
                       
                        
                     
                     } 
                   
                   
                     
                       m 
                       = 
                       
                         i 
                         - 
                         
                           ⌊ 
                           
                             P 
                             2 
                           
                           ⌋ 
                         
                       
                     
                     , 
                     … 
                      
                     
                         
                     
                     , 
                     
                       i 
                       + 
                       
                         ⌊ 
                         
                           P 
                           2 
                         
                         ⌋ 
                       
                       - 
                       w 
                       + 
                       1 
                     
                     , 
                     
                       n 
                       = 
                       
                         j 
                         - 
                         
                           ⌊ 
                           
                             P 
                             2 
                           
                           ⌋ 
                         
                       
                     
                     , 
                     … 
                      
                     
                         
                     
                     , 
                     
                       j 
                       + 
                       
                         ⌊ 
                         
                           P 
                           2 
                         
                         ⌋ 
                       
                       - 
                       w 
                       + 
                       1 
                     
                   
                 
                 ) 
               
               &gt; 
               
                 T 
                 o 
               
             
           
         
       
     
         [0032]    In this equation T o  is a pre-defined light radiation intensity variance threshold and the noise reduction parameter w can be used to reduce the effect of camera noise for some for some 0≦w≦P (in experiments 
         [0000]    
       
         
           
             
               w 
               = 
               
                 ⌈ 
                 
                   P 
                   2 
                 
                 ⌉ 
               
             
             ) 
           
         
       
     
         [0000]    within a pixel patch of array size (P×P); e.g., calculating the variance is based upon averaging light radiation intensity difference value(s) for the pixel(s) in the array patch surrounding reference pixel (i,j) when D o (i,j) is calculated over natural numeric counting value(s) (m, k, n, t) that are defined (in terms of w and P) within the range set by the equation. In extending the decision to more than two images, the value D o   S,R  (i,j) can be calculated for different combined pair(s) of images (S) and (R) and an aggregation function (such as max or average or weighted average value) can be performed over substantially all possible image combination(s). 
         [0033]      FIG. 2  shows photographic illustrations of input visual images (A+B) that are processed to form an output map (C) using the new object detection technique in which whitened shading identifies area(s) where value(s) for D o    A,B  were found to be greater than the threshold T o  for a new object artifact that has been detected. This decision criterion can detect changes in edges and corners of a visual image allowing other artifacts (in addition to new objects) to be traced in that manner. 
       Simple Motion Detection 
       [0034]    Given two visually inconsistent images having light radiation intensity value(s) (S) and (R) a decision criterion for simple motion detection around reference pixel (i,j) within a pixel patch array of size (P×P) according to an exemplary embodiment includes: 
         [0000]    
       
         
           
             
               
                 D 
                 
                   S 
                   , 
                   R 
                 
                 s 
               
                
               
                 ( 
                 
                   i 
                   , 
                   j 
                 
                 ) 
               
             
             = 
             
               
                 
                   ∑ 
                   
                     
                       m 
                       = 
                       
                         i 
                         - 
                         
                           ⌊ 
                           
                             P 
                             2 
                           
                           ⌋ 
                         
                       
                     
                     , 
                     … 
                      
                     
                         
                     
                     , 
                     
                       i 
                       + 
                       
                         ⌊ 
                         
                           P 
                           2 
                         
                         ⌋ 
                       
                       - 
                       w 
                       + 
                       1 
                     
                   
                   
                       
                   
                 
                  
                 
                   
                     ∑ 
                     
                       
                         n 
                         = 
                         
                           j 
                           - 
                           
                             ⌊ 
                             
                               P 
                               2 
                             
                             ⌋ 
                           
                         
                       
                       , 
                       … 
                        
                       
                           
                       
                       , 
                       
                         i 
                         + 
                         
                           ⌊ 
                           
                             P 
                             2 
                           
                           ⌋ 
                         
                         - 
                         w 
                         + 
                         1 
                       
                     
                     
                         
                     
                   
                    
                   
                      
                     
                       
                         
                           ∑ 
                           
                             
                               k 
                               = 
                               0 
                             
                             , 
                             … 
                              
                             
                                 
                             
                             , 
                             
                               w 
                               - 
                               1 
                             
                           
                           
                               
                           
                         
                          
                         
                           
                             ∑ 
                             
                               
                                 t 
                                 = 
                                 0 
                               
                               , 
                               … 
                                
                               
                                   
                               
                               , 
                               
                                 w 
                                 - 
                                 1 
                               
                             
                             
                                 
                             
                           
                            
                           
                             S 
                             
                               
                                 m 
                                 + 
                                 k 
                               
                               , 
                               
                                 n 
                                 + 
                                 t 
                               
                             
                           
                         
                       
                       - 
                       
                         R 
                         
                           
                             m 
                             + 
                             k 
                           
                           , 
                           
                             n 
                             + 
                             t 
                           
                         
                       
                     
                      
                   
                 
               
               &gt; 
               
                 T 
                 s 
               
             
           
         
       
     
         [0035]    In this equation T S  is a pre-defined light radiation intensity aggregation threshold and the noise reduction parameter w can again be used to reduce the effect of camera noise for some 0≦w≦P (in experiments w=P) within a pixel patch of array size (P×P) when Ds(i,j) is calculated for reference pixel (i,j) over natural numeric counting value(s) (m, k, n, t) that are defined (in terms of w and P) within the range set by the equation. Arriving at a single value for D s  (i,j) in extending the decision to more than two images again involves use of any known aggregation function (such as max or average or weighted average value) that can be performed over substantially all possible image combination(s).  FIG. 3  shows photographic illustrations of input visual images (A+B) that are processed to form an output map (C) using the simple motion detection technique in which whitened shading identifies area(s) where value(s) for D s    A,B  were found to be greater than the threshold T s  for a simple motion artifact that has been detected. 
         [0036]    As mentioned above and as shown in  FIGS. 7 &amp; 8 , the image correction process is divided into sub-steps of new object correction and simple motion correction that depend on the outcome of the detection step; e.g., if both D s    A,B &gt;T s  and D o    A,B &gt;T o  or if only the latter condition is true then new object correction is used (otherwise) if only the former condition is true then simple motion correction is used. 
       New Object Correction 
       [0037]    In the new object correction mode for an image (S i ) a threshold (U i ) is set to indicate a maximum permissible normalized overall intensity value for S i ; e.g., above this threshold the image is “burnt” (from lengthy exposure time) and cannot be chosen. A lower threshold (L i ) is also set below which the image is too “noisy” (from insufficient exposure time) and cannot be chosen. Thus given a set of images S 1  . . . S N  at a reference pixel position (i,j) the value chosen for the output image according to an exemplary embodiment includes:
       IF L 1 &lt;neighborhood ij (S 1 )&lt;U 1  THEN choose S ij   1      ELSE IF L 2 &lt;neighborhood ij (S 2 )&lt;U 2  THEN choose middle pixel of S ij   2      . . .   ELSE IF L N &lt;neighborhood ij (S N )&lt;U N  THEN choose middle pixel of S ij   N      ELSE choose S ij   1          
 
         [0043]    In these equation(s) neighborhood ij (S) is an averaging of the light radiation intensity value(s) of the pixel patch array surrounding reference pixel (i,j) which is done to reduce noise artifacts in sensor image signal (S). 
       Simple Motion Correction 
       [0044]    The simple motion correction step can be used to handle a case where artifacts are suspected but not confirmed to exist. This step may be also referred to as a propagation step to extend use of at least one previous image selection decision from previously defined pixel(s); e.g., a decision on the light radiation intensity value of the pixel at position (x,y) can be determined by previously known value(s) of pixel(s) at position(s) (x−1,y),(x,y−1) and (x−1, y−1) for example. 
         [0045]    To propagate a previous decision to a pixel (i,j) first note that simple motion detection for a pixel at position (x,y) such that x&lt;i,y&lt;j was already determined by some policy 
         [0000]    
       
         
           
             ( 
             
               
                 p 
                 xy 
               
               = 
               
                 
                   1 
                   N 
                 
                  
                 
                   
                     ∑ 
                     
                       
                         k 
                         = 
                         1 
                       
                       , 
                       … 
                        
                       
                           
                       
                       , 
                       N 
                     
                     
                         
                     
                   
                    
                   
                     
                       W 
                       k 
                     
                      
                     
                       S 
                       k 
                     
                   
                 
               
             
             ) 
           
         
       
     
         [0000]    for (N) examined image(s) having corresponding light radiation intensity value(s) (S 1  . . . S N ) and for some weighting function (W 1  . . . W N ) such that (Σ k  W k =1) when (p xy ) is calculated over natural numeric counting value (k) that is defined (in terms of N) within the range set by the equation. The propagation policy at the reference pixel (i,j) according to an exemplary embodiment then includes: 
         [0000]    
       
         
           
             
               p 
               ij 
             
             = 
             
               
                 
                   ∑ 
                   
                     
                       
                         i 
                         - 
                         q 
                       
                       ≤ 
                       x 
                       &lt; 
                       i 
                     
                     , 
                     
                       
                         j 
                         - 
                         q 
                       
                       ≤ 
                       y 
                       &lt; 
                       j 
                     
                   
                   
                       
                   
                 
                  
                 
                   
                     w 
                     xy 
                     ij 
                   
                    
                   
                     p 
                     xy 
                   
                 
               
               
                 
                   ∑ 
                   
                     
                       
                         i 
                         - 
                         q 
                       
                       ≤ 
                       x 
                       &lt; 
                       i 
                     
                     , 
                     
                       
                         j 
                         - 
                         q 
                       
                       ≤ 
                       y 
                       &lt; 
                       j 
                     
                   
                   
                       
                   
                 
                  
                 
                   w 
                   xy 
                   ij 
                 
               
             
           
         
       
     
         [0046]    In these equation(s), parameter (q≧1) defines the number of previously examined pixel(s) at reference position (i, j) and can be set to a small value, e.g., 3 and (w xy   ij ≧0) is another weighting function between the pixel at position (x, y) and position (i,j): for fast approximate results this value could be simply set to 1 whereas for more accurate results this value can represent the distance between the two pixels (smaller distance−higher value) and/or the light radiation intensity difference(s) (e.g., if |S xy −S ij |≦|S ab − ij | then w xy   ij &gt;w ab   ij ) for pixel indice(s) x, aα&lt;i and y, b&lt;j. 
         [0047]      FIG. 4  shows illustrations of images photographed from a moving car and combined to form a resulting HDR image viewed with (and without) motion detection and correction. This example combines three (3) inconsistent input images photographed from a moving car showing several ghosting artifacts (such as the traffic light shadow and traffic polls appearing as new objects in inconsistent image locations) whereas those artifacts disappear after correction.  FIG. 5  shows illustrations of two output HDR images with (and without) motion compensation. The input images are created using two different exposure times and with motion blur. This example combines two images by choosing only the image with the shortest exposure time in the blurry region(s) which is less likely to be blurred. In this way unwanted motion artifacts appearing in visual images with differing exposure times (such as an HDR image) can be eliminated in a fast and local manner (e.g., during real-time scanning of the image using a minimum amount of computer memory storage space) to eliminate ghosting (and other types of defects) appearing in such an image in a way that can be applied to movies as well as to RGB and/or grayscale and/or Mosaic (e.g., BAYER filter format) images for example. 
         [0048]    As shown with reference to  FIG. 6 , it is to be understood that the inventive concept(s) described herein include element(s) that can be implemented on at least one general-purpose computer  131 ; including a signal source  138  and/or processor(s)  132 / 139  and/or memory  133 / 137  along with input/output device(s)  135 / 136  operatively coupled with each other via circuitry  134  which can be implemented on at least one integrated circuit and configured to operate by execution of software program instruction(s) to process data according to at least one or more exemplary embodiment(s) as described above. Thus, it is to be understood by one skilled in the art that these inventive concept(s) can be implemented using conventional computer hardware, software or a combination of both. 
         [0049]    It will be understood by one skilled in the art that the present inventive concept(s) are only by way of example described and illustrated by reference to the foregoing description taken in conjunction with the accompanying drawings; and that the described feature(s), structure(s) and/or characteristic(s) may be combined and arranged and designed in different ways and that modification(s) and/or change(s) can be made to include device(s), system(s) and/or processe(s) consistent with the inventive concept(s) as embodied in the following claims, which are to be interpreted as broadly as the law permits to cover the full scope of the invention, including all equivalents thereto.