A computer system programmed for performing color change on an object within a video in real time, comprising a memory storing a program for execution by a processor, the program including instructions for: receiving a selection of object in the video to be recolored; utilizing a predictive location and predictive geometrical shape of the object to select pixels to be ignored when searching for pixels belonging to the object and thereby reduce search space of pixels that should be included in the object; performing element selection to obtain collection of pixels that belong to the selected object; performing a check of neighboring pixels to confirm the collection of pixels; grouping pixels as belonging to the object using information from previous frames; performing surface continuity check to eliminate mislabeled pixels due to reflections or shadowing; generating a plurality of masks, each mask storing different property values of the collection of pixels, wherein the plurality of masks comprise: a black and white mask segmenting out the object, a grayscale mask storing grayscale values of pixels belonging to the object, red color mask representing red areas of the object, green color mask representing green areas of the object, a blue color mask representing blue areas of the object, brightness mask of intensity map of all of the pixels belonging to the object, and a light reflection mask storing reflection values of pixels belonging to the object; selecting a new color; color replacement of the object through mask manipulation, wherein the mask manipulation includes replacing one of the plurality of masks with a new color mask, and/or multiplication or adding a new color mask to the plurality of masks and/or subtraction and division of a new color mask from the plurality of mask; applying the manipulated mask/masks on the original video to generate a new video; and, displaying the new video. receiving a selection of object in the video to be recolored; utilizing a predictive location and predictive geometrical shape of the object to select pixels to be ignored when searching for pixels belonging to the object and thereby reduce search space of pixels that should be included in the object; performing element selection to obtain collection of pixels that belong to the selected object; performing a check of neighboring pixels to confirm the collection of pixels; grouping pixels as belonging to the object using information from previous frames; performing surface continuity check to eliminate mislabeled pixels due to reflections or shadowing; generating a plurality of masks, each mask storing different property values of the collection of pixels, wherein the plurality of masks comprise: a black and white mask segmenting out the object, a grayscale mask storing grayscale values of pixels belonging to the object, red color mask representing red areas of the object, green color mask representing green areas of the object, a blue color mask representing blue areas of the object, brightness mask of intensity map of all of the pixels belonging to the object, and a light reflection mask storing reflection values of pixels belonging to the object; selecting a new color; color replacement of the object through mask manipulation, wherein the mask manipulation includes replacing one of the plurality of masks with a new color mask, and/or multiplication or adding a new color mask to the plurality of masks and/or subtraction and division of a new color mask from the plurality of mask; applying the manipulated mask/masks on the original video to generate a new video; and, displaying the new video. <EOS>