Abstract:
A user provides a plurality of digital images of a single subject all taken from a single location and with common camera settings. This set of digital images is parsed one pixel location at a time. For each pixel location, a set of pixel colors is extracted from each of the plurality of digital images. For each of these pixel colors, a relationship parameter to each of the other pixel colors in the set is calculated. Using this relationship parameter information, the final desired pixel color is chosen and applied to the pixel location in a final image.

Description:
BACKGROUND OF THE INVENTION  
       [0001]     Many photographers go to extreme lengths, such as waking up early in the morning, in an attempt to photograph public locations without having people walk through their photo. Historically, some photographers (with a lot of time on their hands) have utilized very dark neutral density filters over their lens, allowing very long exposure times in the hope that people in the way of their desired subject will keep moving, and their contribution to the resulting photo will be so blurred as to be undetectable. Other photographers have used large numbers of multiple exposures as people move around in front of their subject in the hope that each person will be in only one of the multiple exposures and their “ghostly” representation in the final image will be so faint as to be undetectable. These processes have worked well in certain situations, however, problems remain. First of all these exposures take a very long time to capture and extreme care must be taken in calculating exposure times. Secondly, even though the unwanted people are blurred or faint, there is still contribution to the final exposure by these unwanted subjects.  
       SUMMARY OF THE INVENTION  
       [0002]     A user provides a plurality of digital images of a single subject all taken from a single location and with common camera settings. This set of digital images is parsed one pixel location at a time. For each pixel location, a set of pixel colors is extracted from each of the plurality of digital images. For each of these pixel colors, a relationship parameter to each of the other pixel colors in the set is calculated. Using this relationship parameter information, the final desired pixel color is chosen and applied to the pixel location in a final image.  
         [0003]     Other aspects and advantages of the present invention will become apparent from the following detailed description, taken in conjunction with the accompanying drawings, illustrating by way of example the principles of the invention. 
     
    
     BRIEF DESCRIPTION OF THE DRAWINGS  
       [0004]      FIG. 1  is an example series of four digital images, all of the same scene taken with similar camera settings.  
         [0005]      FIG. 2  is an example final image generated from the images of  FIG. 1  with all of the moving objects removed from the final image according to an example embodiment of the present invention.  
         [0006]      FIG. 3  is an illustration of the red-green-blue (RGB) color space.  
         [0007]      FIG. 4  is an illustration of the RGB color space from  FIG. 3  with the color location of four pixel colors plotted in the color space according to an example embodiment of the present invention.  
         [0008]      FIG. 5  is a flowchart of an example embodiment of a method of removing moving objects from a series of digital images according to the present invention.  
         [0009]      FIG. 6A  is a front view of an example embodiment of a device configured to remove moving objects from a series of digital images according to the present invention.  
         [0010]      FIG. 6B  is a rear view of an example embodiment of a device configured to remove moving objects from a series of digital images according to the present invention.  
     
    
     DETAILED DESCRIPTION  
       [0011]     This description of the preferred embodiments is intended to be read in connection with the accompanying drawings, which are to be considered part of the entire written description of this invention. In the description, relative terms such as “lower,” “upper,” “horizontal,” “vertical,” “up,” “down,” “top,” “bottom,” “left,” and “right” as well as derivatives thereof (e.g., “horizontally,” “downwardly,” “upwardly,” etc.) should be construed to refer to the orientation as then described or as shown in the drawing figure under discussion. These relative terms are for convenience of description and normally are not intended to require a particular orientation. Terms concerning attachments, coupling and the like, such as “connected,” “coupled,” and “interconnected,” refer to a relationship wherein structures are secured or attached to one another either directly or indirectly through intervening structures, as well as both movable or rigid attachments or relationships, unless expressly described otherwise.  
         [0012]      FIG. 1  is an example series of four digital images, all of the same scene taken with similar camera settings. In this example, four digital images of the same scene are taken by a user with similar camera settings. For best results, a camera support of some sort, such as a tripod, will be used so that the non-moving portions of the images are as identical as possible. In this example embodiment of the present invention a mountain scene with a tumbleweed rolling through the scene is to be processed to remove the moving elements (in this case, the tumble weed). A selected pixel location  108  is highlighted within each of the images. In the first image  100 , the tumbleweed is located at the selected pixel location  108 , and the color of this pixel location  108  in this first image  100  will be that of the tumbleweed. In the second image  102 , the tumbleweed has now moved to the right of the selected pixel location  108  and the color of this pixel location  108  in this second image  102  will be that of the background. In the third image  104 , the tumbleweed has now moved further to the right of the selected pixel location  108  and the color of this pixel location  108  in this third image  104  will be that of the background. In the fourth image  106 , the tumbleweed has now moved still further to the right of the selected pixel location  108  and the color of this pixel location  108  in this fourth image  106  will be that of the background. Those of skill in the art will recognize that the results of the process of removing moving objects according to the present invention will be improved with the availability of more initial images, and by the use of a support to stabilize the camera. Any number of initial images may be used within the scope of the present invention. If necessary, pre-processing of the set of initial images may be applied to align the features within the set of initial images such that individual pixel locations within the set of initial image all represent the same portion of the captured scene. In other words, it is desired that each pixel location within the set of images corresponds to the same small (pixel sized) area of the image. Those of skill in the art will recognize that there are a wide variety of methods that may be used to align digital images all within the scope of the present invention. These methods are commonly used when a series of digital images is stitched together to form a panoramic image.  
         [0013]      FIG. 2  is an example final image generated from the images of  FIG. 1  with all of the moving objects removed from the final image according to an example embodiment of the present invention. This example embodiment of the present invention shows the results of the use of the method of the present invention to remove moving objects from images. The four images of  FIG. 1  have been processed to remove the moving tumbleweed from the image, and this final image  200  shows the scene from the images of  FIG. 1  without the moving tumbleweed. Obviously, this method may be used to remove a very wide variety of moving objects from a series of images.  
         [0014]      FIG. 3  is an illustration of the red-green-blue (RGB) color space. In this example embodiment of the present invention a RBG color space  300  is used for calculation of relationship parameters between pixel colors. Those of skill in the art will recognize that there are a very wide variety of possible color spaces that may be used all within the scope of the present invention, and the RGB color space is simply an example of one of the possible color spaces. In the RGB color space  300 , the color black is located at the point (0, 0, 0)  302 , and white is located at the point (Max, Max, Max)  316 , where Max is set to the maximum possible value of the chosen color space. For example, using 8-bits to represent each of the red, green, and blue colors, Max would be the hexadecimal value of FF or 255 in decimal. Other embodiments of the present invention may use more or less bits to represent each of the axes in the chosen color space, all within the scope of the present invention. In the RGB color space  300 , the color red is located at the point (Max, 0, 0)  308 , green is located at the point (0, Max, 0)  304 , blue is located at the point (0, 0, Max)  306 , magenta is located at the point (Max, 0, Max)  310 , cyan is located at the point (0, Max, Max)  312 , and yellow is located at the point (Max, Max, 0)  314 .  
         [0015]      FIG. 4  is an illustration of the RGB color space from  FIG. 3  with the color location of four pixel colors plotted in the color space according to an example embodiment of the present invention. In this example embodiment of the present invention, the colors of the selected pixel location  108  from the series of four digital images  100 ,  102 ,  104 , and  106  from  FIG. 1  have been plotted in the RGB color space from  FIG. 3 . In this example, the color of pixel location  108  from the first image  100  is located at a first point  400  near the green corner of the RGB color space, and represents the color of the tumbleweed from  FIG. 1 . The color of pixel location  108  from the second image  102  is located at a second point  402  near the magenta corner of the RGB color space, and represents the color of the background from  FIG. 1 . The color of pixel location  108  from the third image  104  is located at a third point  404  near the magenta corner of the RGB color space, and represents the color of the background from  FIG. 1 . The color of pixel location  108  from the fourth image  106  is located at a fourth point  406  near the magenta corner of the RGB color space, and represents the color of the background from  FIG. 1 .  
         [0016]     From this color data a relationship parameter is calculate between each pair of colors. For example, starting with the first point  400 , a first relationship parameter is calculated between this point and the second, third, and fourth points  402 ,  404 , and  406 . In some example embodiments of the present invention this relationship parameter is the distance in the color space between the points. When the distance between points is used as the relationship parameter on this set of data, the relationship parameter will be small between the second, third, and fourth points  402 ,  404 , and  406 , and large between any of these three points and the first point  400 . Thus the set of relationship parameters for the first point  400  will contain three large numbers representing the distance to each of the three other points. For the second point  402 , the set of relationship parameters will contain one large number representing the distance to the first point  400 , and two small numbers representing the distance to the third and fourth points  404 , and  406 . Similarly, the set of relationship parameters for the third and fourth points  404 , and  406  will contain two small numbers and one large number. By examining these sets of data points and selecting the set with the smallest sum of relationship parameters, in this example embodiment of the present invention, either the second  402 , third  404 , or fourth point  406  will be selected as the pixel color for the current pixel location  108  of the final image and this pixel will be the color of the background, not the tumbleweed. This system of “pixel voting” may be used with formulas for calculating relationship parameters other than the distance formula shown in this example embodiment of the present invention.  
         [0017]     In some cases involving three or more different colors among the set of pixel colors for the current pixel location including a pixel color near the center (gray) of the color space, the distance function may actually select the incorrect color. It may be desirable for some sets of images to use a different formula for calculating the relationship parameters. For example, using the sum of the square roots of the distances in color space may, in some cases, give more accurate results than using the sum of the distances themselves. Other functions such as a log 2 of the distances may be used in other embodiments of the present invention. Functions with a positive first derivative, but a negative second derivative result in relationship parameters such that close distances between pixel colors are the deciding factor in the selection of the final pixel color, and distant pixel colors that are not in the neighborhood of a cluster of pixel colors have such large relationship parameters that they have a very small chance of being selected as the final pixel color.  
         [0018]      FIG. 5  is a flowchart of an example embodiment of a method of removing moving objects from a series of digital images according to the present invention. In an optional step  500 , a series of digital images are aligned with each other. In a step  502  a pixel location within a series of digital images is selected. In a step  504 , a set of pixel colors comprising the color of the current pixel location in each of the series of digital images is created. In a step  506 , a relationship parameter is computed in a color space for each pixel color to every other pixel color in the set of pixel colors. In a step  508 , a final pixel color for the current pixel location is selected from the set of pixel colors using the relationship parameters. In a step  510  the color of the current pixel location in a final image is set to the selected pixel color. In a decision step  512  if all the pixel locations have been processed, control is passed to a step  514  and the final image is stored in a memory. If more pixel locations are remaining to be processed, control is passed to a step  502  where another pixel location is selected.  
         [0019]      FIG. 6A  is a front view of an example embodiment of a device configured to remove moving objects from a series of digital images according to the present invention. In this example embodiment of the present invention, a digital camera is designed including a body  600 , a lens  602 , an image sensor  604  configure to sense an image projected onto it by the lens  602 , a memory  606 , electrically coupled with the image sensor  604  configure to store digital image data. The digital camera also includes a processor  614  configured to process a series of digital images, remove moving objects from the digital images, and store a final image in a memory  606 . This example digital camera also includes a shutter button  608 , a viewfinder  610 , and a flash  612 . In some example embodiments of the present invention, some or all of the memory  606  may be non-volatile memory. Those of skill in the art will recognize that the method of the present invention may be performed by a wide variety of electronic devices all within the scope of the present invention. While this example embodiment shows the method being performed in a digital camera, the method of the present invention also lends itself well to post-processing of digital images on a computer, printer, or other device all within the scope of the present invention.  
         [0020]      FIG. 6B  is a rear view of an example embodiment of a device configured to remove moving objects from a series of digital images according to the present invention. This example digital camera also includes a display  616  (such as an LCD) electrically coupled to the memory  606  configured to display the series of captured images and to display the final image with all of the moving objects removed from the image.  
         [0021]     The foregoing description of the present invention has been presented for purposes of illustration and description. It is not intended to be exhaustive or to limit the invention to the precise form disclosed, and other modifications and variations may be possible in light of the above teachings. The embodiments were chosen and described in order to best explain the principles of the invention and its practical application to thereby enable others skilled in the art to best utilize the invention in various embodiments and various modifications as are suited to the particular use contemplated. It is intended that the appended claims be construed to include other alternative embodiments of the invention except insofar as limited by the prior art.