The MediaMill at TRECVID 2013: : Searching concepts, Objects, Instances and events in video

In this paper we summarize our TRECVID 2013 [15] video retrieval experiments. The MediaMill team participated in four tasks: concept detection, object localization, instance search, and event recognition. For all tasks the starting point is our top-performing bag-of-words system of TRECVID 2008-2012, which uses color SIFT descriptors, average and difference coded into codebooks with spatial pyramids and kernel-based machine learning. New this year are concept detection with deep learning, concept detection without annotations, object localization using selective search, instance search by reranking, and event recognition based on concept vocabularies. Our experiments focus on establishing the video retrieval value of the innovations. The 2013 edition of the TRECVID benchmark has again been a fruitful participation for the MediaMill team, resulting in the best result for concept detection, concept detection without annotation, object localization, concept pair detection, and visual event recognition with few examples. 1 Task I: Concept Detection Our concept detection approach builds on previous editions of the MediaMill semantic video search engine [19,20]. New this year is our convolutional neural network and concept detection without annotation. Since deep learning is critically dependent on labeled examples and suffers from noisy and incomplete annotations, as common in TRECVID [1,18], we manually extended the collaborative annotations. Color Difference Coding Our baseline concept detection system uses a bag-of-words with color point descriptors only. For point sampling we rely on dense sampling, with an interval distance of six pixels and sampled at multiple scales. We used a spatial pyramid of 1x1 and 1x3 regions in our experiments. We used a mixture of SIFT, TSIFT, and C-SIFT descriptors [23]. We compute the descriptors around points ∗Visiting from INRIA, Rennes, France. obtained from dense sampling, and reduce the dimensionality with principal component analysis. We encode the color descriptors with the aid of difference coding using Fisher vectors with a Gaussian Mixture Model codebook [17]. For efficient storage we perform product quantization [6] on the features. The classifier is a linear SVM, which we apply on either the keyframe or on a maximum of six frames per shot. Convolutional Neural Network Our deep learning concept detection system is a convolutional neural network with 8 layers with weights [8]. The input is raw pixel data, the output are concept scores. The network is trained using error back propagation. However, in contrast to ImageNet, there are too few labeled examples in the TRECVID SIN 2013 set for deep learning to be effective. We studied how additional examples from ImageNet [2] can be exploited to better train our networks. To improve the results, we took a network that had already been trained on ImageNet and retrained it for the 60 TRECVID 2013 SIN concepts. Similar to our color difference coding baseline we apply the network on either the keyframe or on a maximum of six frames per shot. Learning from Social Media Learning video concept detectors from social-tagged media sources, such as Flickr images and YouTube videos, has the potential to address a wide variety of concept queries for video search. The focus of our TRECVID 2013 investigations for the no-annotation task are experiments with a video search engine which is capable of learning concept detectors from social media [7]. For each of the 60 concepts defined in the Semantic Indexing task we harvest positive examples from Flickr using two strategies (see runs below.) Our total pool is 1 million images for tag relevance training. The training set for concept detectors is 200k consisting of images tagged with the 60 concepts. We compute tag relevance on the 200k using the 1 million for neighbor voting [9]. We subselect relevant positive and negatives from the 200k to a maximum of 4,000 top-ranked examples [7]. The negatives were sampled from the other 59 categories of the 200k set using negative 0 0.1 0.2 0.3 0.4 0.5 0.6 0.7 0.8 0.9 1 Studio_With_Anchorperson Skating Quadruped Military_Airplane George_Bush Forest Flags Fields Door_Opening Baby Throwing Telephones Sitting_Down Singing Running People_Marching Old_People News_Studio Motorcycle Kitchen Instrumental_Musician Hand Government−Leader Girl Flowers Female−Human−Face−Closeup Explosion_Fire Dancing Computers Chair Bus Bridges Boy Boat_Ship Beach Animal Anchorperson Airplane Inferred Average Precision S e m a n ti c C o n c e p t TRECVID 2013 Semantic Indexing Task Benchmark Comparison