Abstract:
A method of processing a digital image to animate a figure in the image. The method includes the steps of providing the digital still image; identifying the figure in the digital still image to be animated; generating an animation model of the figure; substituting the animation model for the figure in the digital still image; detecting an area of missing detail in the digital still image resulting from the substitution; identifying information in the digital still image adjacent the area of missing detail; replicating the identified information; and inserting the replicated identified information into the area of missing detail to generate an animated image.

Description:
FIELD OF THE INVENTION  
         [0001]    The present invention relates to image processing, and more particularly to a method of image processing an animated figure.  
         BACKGROUND OF THE INVENTION  
         [0002]    The animation of figures, objects or characters has been the subject and attention of artists for many years. Ranging from Nickelodeon machines of the late 19 th  century that flipped pages of paper creating the illusion of movement, to the sophisticated cartoons of the 20 th  century, there has been tremendous interest in the process of animation.  
           [0003]    Techniques have been developed for creating photorealistic 3D facial models and animations from photographs of a human subject. (Refer to Synthesizing Realistic Facial Expressions from Photographs” by Pighin et al, SIGGRAPH 98, Computer Graphics Proceedings, Annual Conference Series, 1998, pages 75-83.) An animation can encompass any change, for example, a movement of the head or a change in facial expression.  
           [0004]    It may be desired to select a figure in a photograph, animate the figure, and create a new photograph of the animated figure. That is, it may be desirable to “cut” a figure from an image, animate the figure, and “insert” the animated figure back into the image. However, in inserting the animated figure into the image, details of the image may be missing. Such missing details can result, for example, if the animated figure has been moved or repositioned within the image. For example, if the image is of a person looking over his left shoulder, and the animated figure is of the person looking over his right shoulder, details of the image about the person&#39;s left shoulder may be missing when the animated figure is inserted back into the image.  
           [0005]    U.S. Pat. No. 5,914,748 (Parulski, et al) discloses an imaging system useful for compositing images including a first image of a subject plus a background and a second image of the background only. The first and second images are subtracted to generate a difference image, and the difference image is processed to generate a mask image. The mask image is then applied to the first image to select an area in which the subject is located, whereby the selected subject area is used in the compositing of a new image. While this method may have achieved certain degrees of success in particular applications, particularly where a photographer can physically move or manipulate a subject, it is not suitable when the subject and background are physically combined, as in an existing or legacy photograph.  
           [0006]    Accordingly, a need continues to exist for a method for processing an image wherein a figure is animated and overcomes the problems noted above.  
         SUMMARY OF THE INVENTION  
         [0007]    An object of the present invention is to provide a method for processing an image wherein a figure is animated.  
           [0008]    Another object of the invention is to provide such a method which detects areas of missing detail in the image and corrects for these areas.  
           [0009]    Still another object of the invention is to provide such a method which enhances the animated image.  
           [0010]    These objects are given only by way of illustrative example. Thus, other desirable objectives and advantages inherently achieved by the disclosed invention may occur or become apparent to those skilled in the art. The invention is defined by the appended claims.  
           [0011]    According to one aspect of the invention, there is provided a method of image processing a digital still image. The method comprises the steps of providing the digital still image; identifying the figure in the digital still image to be animated; generating an animation model of the figure; substituting the animation model for the figure in the digital still image; detecting an area of missing detail in the digital still image resulting from the substitution; identifying information in the digital still image adjacent the area of missing detail; replicating the identified information; and inserting the replicated identified information into the area of missing detail to generate an animated image.  
           [0012]    The present invention provides a method for processing an image wherein a figure is animated. The method detects areas of missing detail, and corrects for these areas. 
       
    
    
     BRIEF DESCRIPTION OF THE DRAWINGS  
       [0013]    The foregoing and other objects, features, and advantages of the invention will be apparent from the following more particular description of the preferred embodiments of the invention, as illustrated in the accompanying drawings.  
         [0014]    [0014]FIG. 1 shows a first digital still image comprising a figure to be animated.  
         [0015]    [0015]FIG. 2 shows a second image which comprises an animation of the first image of FIG. 1.  
         [0016]    [0016]FIG. 3 is a flow diagram of a method in accordance with the present invention.  
         [0017]    [0017]FIG. 4 shows the figure of the first image of FIG. 1 outlined in dashed lines.  
         [0018]    [0018]FIG. 5 shows the figure of the first image of FIG. 1 extracted from the first image.  
         [0019]    [0019]FIG. 6 shows an animation model of the first image of FIG. 1.  
         [0020]    [0020]FIG. 7 shows the substitution of the animation model for the figure in the first image.  
         [0021]    [0021]FIG. 8 shows a mask area outlining the difference area of the substitution of the animation model for the figure.  
         [0022]    [0022]FIG. 9 shows, in exaggerated enlargement, a portion of FIG. 8 illustrating the pixel arrangement of the mask area.  
         [0023]    [0023]FIG. 10 shows, in exaggerated enlargement, a portion of FIG. 9 illustrating the pixel arrangement of the mask area.  
         [0024]    [0024]FIG. 11( a )-( c ) shows a series of animations to be image processed using the method in accordance with the present invention.  
         [0025]    [0025]FIG. 12. is a flow diagram of a method in accordance with the present invention for a series of animations.  
     
    
     DETAILED DESCRIPTION OF THE INVENTION  
       [0026]    The following is a detailed description of the preferred embodiments of the invention, reference being made to the drawings in which the same reference numerals identify the same elements of structure in each of the several figures.  
         [0027]    [0027]FIG. 1 shows a first image  10  which is a digital still image captured by an image capture device such as a digital camera or camcorder and stored in digital form, such as on a floppy disk, memory card, or CD. Those skilled in the art will recognize that a photograph taken using an image capture device such as a photographic film camera, can be converted to a digital still image using a scanner or other device for converting images for storage in digital form.  
         [0028]    First image  10  comprises a first FIG. 12 and a background  14 . First FIG. 12 is shown as a front view of a human subject, though first FIG. 12 may be an object or character. For ease of discussion, background  14  is shown as a plurality of lines, though background  14  may comprise any figure/object/character or combination thereof.  
         [0029]    [0029]FIG. 2 shows a second image  20  comprising a second FIG. 22 and background  14 . Second FIG. 22 is an animation of first FIG. 12 which has been generated using known animation techniques. As illustrated, second FIG. 22 is a right side view of the human subject.  
         [0030]    The method of the present invention is shown in a flow diagram illustrated in FIG. 3. The method in accordance with the present invention processes first image  10  to generate second image  20 .  
         [0031]    First, an image is provided (step  100 ), and a figure disposed with the image is identified to be animated (step  102 ). Once identified, the figure is extracted from the image (step  104 ). Accordingly, the figure is isolated for extraction from the image. Once extracted, an animation model of the figure is generated (step  106 ). The animation model is substituted for the figure in the image (step  108 ) so that an area of missing detail can be detected (step  110 ). The information adjacent to the area of missing information is identified (step  112 ) and replicated (step  114 ), and then inserted (step  116 ) into the area of missing detail to generate an animated image.  
         [0032]    The method of the present invention is now more particularly described with regard to FIGS.  4 - 10  wherein the figure to be animated is selected as first FIG. 12 of first image  10 , the animated model is second FIG. 22; and the resulting animated image is second FIG. 20.  
         [0033]    As indicated above, first FIG. 12 of first image  10  is identified as the figure to be animated at step  102 . As such, first FIG. 12 must be isolated so that it can be cut/extracted from first image  10 . FIG. 4 shows a dashed lines  30  around first FIG. 12, providing an outline of first FIG. 12 so as to indicate the figure to be extracted for animation.  
         [0034]    The step of identifying the figure to be animated can be accomplished manually by an operator or automatically by a software program. The outline might be generated using a light pen, such as available from Inkwell Systems of Talent, Oregon. Other means are known to those skilled in the art are available for isolating and extracting a figure from an image. For example, ADOBE PHOTOSHOP™ is a commercially available software package which may be suitable.  
         [0035]    The area outlined by dashed lines  30  defines an extraction area  32  as shown in FIG. 5. Accordingly, when first FIG. 12 is extracted from first image  10  at step  104 , extraction area  32  is viewed in first image  10  as a blank space.  
         [0036]    The animation model of first FIG. 12 may be generated at step  106  by means known to those skilled in the art. As indicated above, Pighin et al. have disclosed techniques for creating photorealistic 3D facial models and animations from photographs of a human subject. LifeFX Networks, Inc., at www.lifefx.com, indicate the development of technology to replicate and portray human facial movements and expressions. For the instant example, FIG. 6 shows second FIG. 22 which is the animation model of first FIG. 12 generated using the known means.  
         [0037]    At step  108 , the animation model is substituted for the figure in the image. That is, second FIG. 22 is substituted for first FIG. 12 in first image  10 . As such, second FIG. 22 is disposed in first image  10  and overlaid on extraction area  32 . An area of missing detail results from the substitution of the animation model for the image. That is, since second FIG. 22 is of a different size and shape than extraction area  32 , portions of extraction area  32  are not replaced/covered by second FIG. 22. This area of missing detail is detected at step  110 . FIG. 7 shows the area of missing detail as difference area  34  when second FIG. 12 is substituted for first FIG. 12 in first image  10 . In FIG. 7, difference area is shaded for clearer illustration. As illustrated, for the particular example, difference area  34  is comprised of two sections.  
         [0038]    The detection of the area of missing detail can be accomplished by methods known to those skilled in the art. For example, using ADOBE PHOTOSHOP™.  
         [0039]    To complete the area of missing detail, image information is developed and inserted into difference area  34 . A technique for developing and inserting information in the area of missing detail is shown in FIGS.  8 - 10 .  
         [0040]    Difference area  34  is comprised of a plurality of pixels  41 . Any pixel which is partially or fully disposed within difference area  34  is assigned a value associated with a blue screen or other specialty colored background. Then, any pixel having a value corresponding to a blue screen value is detected as having blue screen information. Each pixel of the image is scanned to determine if the pixel comprises a portion of or all of blue screen information. A collection of the pixels having blue screen information defines a mask area  42  having a profile as shown in FIG. 8. As such, mask area  42  results from identifying each pixel that comprises blue screen information, including each pixel which intersects dashed lines  30  representing difference area  34 . For the particular example, mask area  42  comprises two sections.  
         [0041]    [0041]FIG. 9 illustrates an exaggerated enlargement of mask area  42  and difference area  34  to more particularly illustrate the pixel arrangement relative to the profile of mask area  42 . Each pixel within mask area  42  is identified, and an adjacent neighboring pixel is identified (step  12 ). The adjacent neighboring pixel is replicated (step  114 ) and inserted into the area of missing detail (step  116 ). Preferably, the nearest neighbor pixel is used for replacement.  
         [0042]    Referring now to FIG. 10, FIG. 10 illustrates an exaggerated enlargement of a portion of mask area  42  wherein the pixels are numbered for ease of discussion. Pixel  1  would be replaced with pixel  1 ′; pixel  2  would be replaced with pixel  2 ′; pixel  3  would be replaced with pixel  3 ′; etc. Depending on the arrangement of the pixel within mask area  42 , several pixel-replacement options may be possible. For example, pixel  3  could be replaced by pixel  3 ′, pixel  4 ′, or pixel  5 ′. Similarly, pixel  6  could be replaced with pixel  4 ′, or pixels  3  or  7  (once pixels  3  or  7  have themselves been replaced). Further, since a small portion of pixel  4  is disposed within difference area  34 , pixel  4  might be replaced with pixel  1  rather than pixel  5 , since the portion of pixel  4  within difference area  34  is adjacent the edge of pixel  1 . Those skilled in the art will recognize that only the pixels disposed within or having a portion disposed within difference area  34  need to be replaced.  
         [0043]    Blurring techniques might be employed to enhance the image. Such blurring techniques are known to those skilled in the art, such as the use of a blurring filter from ADOBE PHOTOSHOP™. On a simple level, combinations of the pixels might be employed. For example, a combination of pixels  3 ′,  4 ′, and  5 ′might be used to replace pixel  3 .  
         [0044]    A method illustrated in FIGS.  11 ( a )-( c ) and  12  is employed if there is a series of animations. For example, assume that a series of three animated figures is desired, as shown in FIGS.  11 ( a )- 11 ( c ), wherein FIG. 11( a ) shows a digital still image of a front view figure which is desired to be animated to a side view figure (shown in FIG. 11( b )) and then back to a front view figure (shown in FIG. 11( c )).  
         [0045]    A flow diagram of a method in accordance with the present invention is illustrated in FIG. 12. A counter n is set to 0 (step  200 ) and incremented to 1 (step  202 ). An image I( 1 ) is provided (step  204 ); image I( 1 ) is the digital still image which is to be animated. This image is shown in FIG. 11( a ), and is the same as first image  10  illustrated in FIG. 1. The figure to be extracted from image I( 1 ) is identified (step  206 ). As with first image  10 , the figure to be extracted from image I( 1 ) is FIG. 12, which is now identified as F( 1 ) in FIG. 11( a ). FIG. F( 1 ) is extracted from image I( 1 ) (step  208 ) as described above. An animation model of FIG. F( 1 ) is generated (step  212 ) and defined as FIG. F( 2 ). FIG. F( 2 ) is substituted for FIG. ( 1 ) in image I( 1 ) (step  212 ), and the area of missing detail is detected (step  214 ). This area of missing detail is difference area  34 , and for ease of discussion is defined as difference area D(F( 1 );F( 2 )). Step  216  involved identifying information in image I( 1 ) adjacent difference area D(F( 1 );F( 2 )). This information is replicated (step  218 ), and inserted into difference area D(F( 1 );F( 2 )) (step  222 ) to generate image I( 2 ) shown in FIG. 11( b ).  
         [0046]    If there is no further animation to be generated, the animation is complete (step  224 ).  
         [0047]    However, if another animated is to be generated, the steps of  202 - 220  are repeated. For the particular example, counter n is incremented to  2  (step  202 ). An image I( 2 ) is provided (step  204 ); image I( 2 ) is the image generated when counter n was 1, that is, FIG. F( 2 ). This image is shown in FIG. 11( b ), and is the same as second image  20  illustrated in FIG. 2. The figure to be extracted from image I( 2 ) is identified (step  206 ). As with second image  20 , the figure to be extracted from image I( 2 ) is FIG. 22, which is now identified as F( 2 ) in FIG. 11( b ). FIG. F( 2 ) is extracted from image I( 2 ) (step  208 ) as described above. An animation model of FIG. F( 2 ) is generated (step  212 ) and defined as FIG. F( 3 ). FIG. F( 3 ) is substituted for FIG. ( 2 ) in image I( 2 ) (step  212 ), and the area of missing detail is detected (step  214 ). This area of missing detail is difference area D(F( 2 );F( 3 )). Step  216  involved identifying information in image I( 2 ) adjacent difference area D(F( 2 );F( 3 )). This information is replicated (step  218 ), and inserted into difference area D(F( 2 );F( 3 )) (step  222 ) to generate image I( 3 ) shown in FIG. 11( c ).  
         [0048]    The invention has been described in detail with particular reference to a presently preferred embodiment, but it will be understood that variations and modifications can be effected within the spirit and scope of the invention. The presently disclosed embodiments are therefore considered in all respects to be illustrative and not restrictive. The scope of the invention is indicated by the appended claims, and all changes that come within the meaning and range of equivalents thereof are intended to be embraced therein.  
       PARTS LIST  
       [0049]    10 first image  
         [0050]    12 first figure  
         [0051]    14 background  
         [0052]    20 second image  
         [0053]    22 second figure  
         [0054]    30 dashed lines; outline  
         [0055]    32 extraction area  
         [0056]    34 difference area  
         [0057]    40 monitor  
         [0058]    41 pixels  
         [0059]    42 mask area