Patent ID: 6249613
Filing Date: 2001-06-19
Classification: H04N

Abstract:
A method for automatically segmenting foreground and background objects in images, comprising:encoding and decoding a first image at a first time reference;extracting macroblocks from a second image at a second time reference;mapping the macroblocks with corresponding arrays in the decoded first image according to a macroblock local vector;deriving frame residuals between the macroblocks and the corresponding arrays;mapping macroblocks to the first image according to a global motion model;deriving global residuals between the macroblocks and the corresponding global motion compensated array in the first image;tagging the macroblocks as a frame prediction type based on one local motion vector;classifying the macroblocks as foreground or background by comparing the global residuals with the derived frame residuals;classifying the macroblocks as foreground when the global residuals are greater than a function of the frame residuals;classifying the macroblocks as background when the frame residuals are greater than some function of the global residuals;updating the mosaic with macroblocks tagged as background;creating a segmentation map that identifies the macroblocks in the second image as either foreground or background;smoothing the segmentation map to remove extraneous foreground and background macroblocks in the segmentation map, wherein smoothing the segmentation map includes:taking macroblock neighbors around a target macroblock in the segmentation map;taking the segmentation map values for each of the macroblock neighbors and the target macroblock;ranking the segmentation map values in increasing order; andselecting the output of the target macroblock as the value of the ranked neighbor at a selected threshold;updating the mosaic with the identified background macroblocks in the smoothed segmentation map;forward warping the mosaic but not changing the contents of the mosaic when all of the following conditions occur:pixel sample values in the macroblocks belong to a decoded video object plane;the mosaic content is already determined at the pixel locations; andthe macroblock is labeled as foreground;forward warping the mosaic and blending the pixel sample values into the mosaic when all of the following conditions occur:pixels in the macroblock belong to the decoded video object plane;the mosaic content is already determined at the pixel locations; andthe macroblock is labeled as background;forward warping and updating the mosaic content with content of the pixel sample values in the decoded second image when all of the following conditions occur:pixels in the macroblock belong to the decoded second image;the mosaic content is undetermined at the pixel locations; andthe macroblock is labeled as background; andinitializing the mosaic by either inserting the first decoded image into the mosaic and then updating the mosaic in time only with macroblocks classified as background or setting a mosaic buffer to zero everywhere and then incrementally updating the mosaic in time only with macroblocks classified as background.