Flickr Distance: A Motion Prediction Approach for Visual Concepts

Image alignment has been studied in different areas of computer vision for hundreds of years, aligning images depicting different scenes remains a challenging problem. Variant to optical flow where an image is aligned to its temporally adjacent frame we propose SIFT flow (1), a method to align an image to its nearest neighbors in a large database containing huge number of scenes. The SIFT flow algorithm consists of matching densely sampled pixel-wise SIFT features between two images. The SIFT features allow unambiguous matching across different scene appearances. The proposed approach combines the concept of Flickr Distance along with the SIFT flow method by determining Flickr Distance between the image concepts. The Flickr distance between two concepts is defined as the Jensen-Shannon (J-S) divergence between their LTVLM. Based on SIFT flow, we propose an alignment based large database framework for image analysis and predicting the motion of images. Here the images are taken from the database and provided as nearest neighbors to a query image. This skeleton can be used in the applications such as motion field prediction from a single image, motion prediction satellite image registration and face recognition. Similarity measurement has been studied for decades and it remains a hot research topic particularly in multimedia literature. There are quite a few vital applications related to this, including indexing, positioning, clustering and annotation. Similarity measurement leads to the Conceptual correlation measurement which in turn calculates the semantic distance between the two image concepts and determines its Flickr Distance (5). Image alignment, registration and correspondence between images are major topics in computer vision. There are several levels of scenarios in which image alignment depends. The easiest stage, aligning different frames of the same scene, has been studied for the purpose of image stitching and stereo matching. The image alignment problem becomes more challenging for dynamic frames in video clippings, e.g. optical flow estimation (7). The similarities between two contiguous frames in a video are often formulated as an inference of a 2D flow field. In this work, we are paying attention in a new, higher level of image arrangement: aligning two images from dissimilar 3D scenes but sharing similar scene characteristics. Image alignment at the frame level is thus called scene association. The two images to match may contain object instances captured from different viewpoints, placed at different spatial locations, or imaged at different scales. The two images may also contain different levels of objects of the same class, and some objects present in one image might be mislaid in the other. Due to these issues the scene association problem is extremely exigent. Motivated by optical flow methods, which are able to produce dense, pixel-to-pixel association between two images, we propose SIFT flow (1) for adopting the computational framework of optical flow, but by matching SIFT features instead of raw pixels. In SIFT flow, a SIFT features (8) are carved out from each pixel to exemplify local image structures and predetermine contextual information. A distinct flow assessment algorithm is used to match the SIFT features between two images. The use of SIFT features allows vigorous matching across different frame/object appearances and the spatial replica allows matching of objects located at different parts of the frame. Moreover a coarse-to-fine matching scheme is considered to drastically step up the flow estimation process (1).