Video object segmentation from long term trajectories

In this thesis we explore the problem of video object segmentation and also propose techniques that have better performances compared to other state-of-the -art techniques presented to date. We define video object (VO) segmentation as the task of labelling the various regions in an image sequence as belonging to a particular object. The main focus in this thesis is on two categories of VO segmentation techniques we define as sparse trajectory segmentation techniques and dense pixel segmentation techniques. A sparse trajectory segmentation approach labels a sparse set of pixel points tracked over a sequence. Here tracking of sparse pixel locations from frame to frame in a sequence generates feature point trajectories. Unlike a sparse segmentation where only trajectories are labelled, a dense pixel segmentation process labels all the pixels in a sequence as corresponding to the objects of interest in a sequence. The major challenge in dense pixel segmentation is providing temporal and spatial consistency from frame to frame in the segmentation produced for a sequence. The majority of previous dense segmentation techniques fail to provide a robust solution to this problem. Hence we propose a dense segmentation Bayesian framework that utilizes long term trajectory (temporal) information in order to introduce spatiotemporal consistency. Our technique first classifies the sparse trajectories into sparsely defined objects in the sparse trajectory segmentation step. Then the sparse object trajectories together with motion model side information are used to generate a dense segmentation of each video frame. Unlike previous work, we do not use the sparse trajectories only to propose motion models, but instead use their position and motion throughout the sequence as part of the classification of pixels in the dense segmentation. Furthermore, we introduce novel colour and motion priors that employ the sparse trajectories to make explicit the spatiotemporal smoothness constraints. We are mainly motivated to produce techniques that can be used in cinema post-production applications. Semi-automated object segmentation is an important step in the cinema postproduction workflow. Hence, we have two adaptations of our general segmentation framework that produce semi-automatic and automatic segmentations respectively. In the semi-automatic segmentation process a user guides the segmentation process, while in the automatic process no user information is required. The long term tracking of sparse features in an image is important for many applications other that VO segmentation, such as motion estimation and people surveillance. The majority of existing tracking frameworks are based on some kind of prediction/correction idea. However, given a careful selection of interest points throughout the sequence, the problem of tracking can be solved with the Viterbi algorithm. We present in this thesis a novel approach to interest point selection for tracking using the Mean Shift algorithm over short time windows. The resulting points are then articulated within a Viterbi algorithm for creating very long term tracking data. The tracks are shown to be more accurate than traditional KLT implementations and also do not suffer from accumulation of error with time.