## 3 Object-of-Interest Annotation
### Video Pre-Processing  
As first step, all videos are pre-processed in an off-line process to enable an efficient annotation process. Keyframes and visual features are thereby extracted from the videos without any user interaction. We expect that a large number of videos has to be processed in this process and that new videos are frequently added. Thus, pre-processing can be performed in a distributed manner on multi-core and multi-machine architectures. The generated data is stored in the database schema described above.  
**Keyframe extraction:** Keyframes are a very common way to present videos and they are often used as input for content-based video analysis. The keyframe extraction approach of [48] and others is similar to the one of this work. A standard shot detection [95] is applied before an activity-based keyframe selection algorithm is performed for each shot. Shots with little activity are summarized with only one or a little keyframes while up to 10 keyframes are extracted for high-activity shots. We measure the activity with global SIFT descriptors similar to the work of [PERSON] and [PERSON] [77] that used global features to capture the gist of a scene. As shown in Algorithm 1, the Euclidian distance is used to compare these features and an adaptive threshold that is decreased until no more than 10 keyframes are selected for a shot. During analysis of the first few video shots an appropriate value is automatically selected for this threshold, and thus its start value (default 0.5) only affects these first shorts. Due to the simplicity of these global features, keyframe selection performs faster than real time.  
**Feature extraction:** The selected keyframes are then used to extract various visual features for object specification, recognition approach selection, and object annotation. Note that the proposed system is not restricted to the use of specific features and recognition approaches, but the following  
Figure 2: Annotation workflow (left) with the associated database schema (right).  
features are currently integrated to the system: SIFT [67], SURF [68], Gabor Wavelets [96], MPEG-7 DominantColor, ColorLayout, ColorStructure, and EdgeHistogram [97] features. These features can be extracted globally from the entire keyframe or locally from dense sampled regions [73], Difference of Gaussian points [67], MSER regions [70], Viola-Jones faces [98], histogram of gradients body regions [78], and from image segments using a mean-shift segmentation [75]. Furthermore, a set of image pre-processing, region filtering approaches and object models are used. As feature extraction takes time and feature storage takes memory, it is reasonable to select only a subset of these features for analysis of specific video domains. The used recognition infrastructure (see Section 4) enables the selection of such sub-sets even on a running system without high development and redeployment efforts.  
```
vector<keyframe>\(frames\) = every 10\({}^{th}\) frame of the shot float \(threshold=\) threshold of last shot output: vector<keyframe>\(keyframes\) repeat \(keyframes\).clear() for all\(f\) in \(frames\)do  boolean \(lowActivity\) = false for all\(k\) in \(keyframes\)do if EuclidianDistance( globalSIFT(\(f\)), globalSIFT(\(k\)) ) \(<threshold\)then \(lowActivity\) = true endif endfor if\(lowActivity\) == false then \(keyframes\).add(\(f\)) endif endfor if\(keyframes\).size() \(<=2\)then \(threshold=threshold\) * 1.25 else \(threshold=threshold\) * 0.95 endif until\(keyframes\).size() \(<=10\)
```  
**Algorithm 1** Keyframe Selection