ARTEMIS at TRECVID 2013: Instance Search Task

This paper describes the approach proposed by the ARTEMIS team at TRECVID 2013, Instance Search (INS) task. The method is based on the Bag-of-Words representation obtained from uniform sampling of the frames of the videos. We propose two types of shot descriptors: one relying on a single representative frame for each video shot and another one collecting visual descriptors from multiple frames of the video clips. 1 Structured Abstract Briefly, what approach or combination of approaches did you test in each of your submitted runs? (please use the run id from the overall results table NIST returns) • all runs: Hessian Affine detectors and RootSIFT descriptor quantized into BoW feature vectors using a vocabulary of size 1M. • F_NO_ARTEMIS_1_1: BoW vectors generated at shot level on resized frames (384x288 surface). Single query BoW vector generated from the multiple example images. • F_NO_ARTEMIS_2_2: BoW vectors generated at shot level on resized frames (384x288 surface). A BoW vector is generated for each query image and queried on the entire dataset. The best score of the video clip among the different query runs is selected. • F_NO_ARTEMIS_3_3: BoW vectors generated from a single representative keyframe for each video shot. Single query BoW vector generated from the multiple example images. • F_NO_ARTEMIS_4_4: BoW vectors generated from a single representative keyframe for each video shot. A BoW vector is generated for each query image and queried on the entire dataset. What if any significant differences (in terms of what measures) did you find among the runs? Although the shot based descriptor has a richer representation, the resizing of the video resolutions has strongly affected its performances. Using one single frame for every video clip yielded better results. The use of a single query descriptor computed from all provided query images improves significantly the results and the retrieval time. Based on the results, can you estimate the relative contribution of each component of your system/approach to its effectiveness? The large size of the vocabulary has compensated partially the reduced number of detected interest points from the resized video frames. The aggregations of the results from multiple runs into a single result list can affect negatively the overall performance. The unified query descriptor gave better results, but makes it impossible to perform a spatial consistency check for re-ranking top results. Overall, what did you learn about runs/approaches and the research question(s) that motivated them? We have learned that the shot level BoW vectors provide a rich representation of the video shot and capture information that can be lost when selecting only a few of the frames of the video shot. However, the resizing of the video frames makes it difficult to retrieve small and tiny objects. 2 Instance Search Task description The Instance Search Task (INS) [1] tackles the problem of retrieving an object in a video dataset as quickly as possible by using a limited amount of data about the object (a couple of images). Given a collection of test video clips and a collection of queries that delimit a person, object or place entity in some example video, participant applications have to locate for each query up to the 1000 clips most likely to contain a recognizable instance of the entity. The video dataset proposed for the 2013 edition consisted of a 189 hours of video content from the BBC Eastenders TV series (programme material ©BBC) [2]. The videos feature a high quality and resolution (768 x 576) and are encoded in the MPEG-4 format. The corpus consists of approx. 470,000 short video clips of a couple of seconds for which the mastershot references are given. A set of frames from the BBC Eastenders dataset is illustrated in Figure 1. For testing, 30 query topics were considered, each consisting of a set of 4 example images drawn from test videos containing the item of interest. For each example frame, a binary mask of the region of interest was provided. In addition, each topic included information about the category of the current topic (e.g., person, object, location). A subset of topics and example images are depicted in Figure 2. Figure 1. Sample from the BBC Eastenders dataset.