## 3 Object-of-Interest Annotation
### Object Specification  
In the second annotation step, users can interactively select a few keyframes that contain the object-of-interest. These keyframes are used to select an appropriate recognition approach including region detector-descriptor chains, matching strategies and dissimilarity measures as well as the parameter settings of all components. Thus, it is important to collect examples of the object that represent the appearance of this object in all videos of the portal well with appropriate difficulties and levels of abstraction. If we want to annotate all cars independent of the used camera view, for example, then the initial object-set should not only include blue SUVs shown from a frontal view. The object specification is composed of following three steps. **Select keyframe:** First, a single keyframe of the object-of-interest has to be selected in one of the videos. This can be done when the user is browsing through the extracted keyframes, while she is watching a video, or in any other way that is provided by the surrounding video portal. The prototype that is presented in Section 5.1 allows keyframe selection in a simple interface where the thumbnails of an entire video are shown. The actual position of the object within a selected keyframe is automatically computed in a later processing step, described below. Generally, examples of an object might also be used from outside the video collection but this is not supported yet because we need to know from which video and keyframe the first object example stems. ```
vector\(<\)features\(>\)\(positives\) = features of the selected keyframe vector\(<\)features\(>\)\(negatives\) = features from keyframes without the object integer \(videoId\) = video id of the selected keyframe integer \(keyframeId\) = id of the selected keyframe
0 Output: vector\(<\)keyframe\(>\)\(suggested\) for\(i=1\) to numberOfKeyframes(\(videoId\) ) do \(features\) = loadFeatures(\(keyframeId\) + \(i\) ) \(distPos\) = nearestNeighbour(\(features\), \(positive\) ) \(distNeg\) = nearestNeighbour(\(features\), \(negative\) ) if\(distPos>thresholdA\) and \(distPos/distNeg<thresholdB\)then \(suggested\).add(\(keyframeId\) + \(i\) ) endif \(i=-i\) if\(i<0\)then \(i=i-1\) endif endfor
```  
**Algorithm 2** Suggest similar keyframes  
**Similar keyframes:** After the user has selected a keyframe that shows the object, we start to search for further examples of this object from nearby keyframes according to Algorithm 2. This search starts in the video of the selected keyframe but, if necessary, it can be expanded to related videos that stem from the same user or that have been labeled with similar tags. Due to the high run-time requirements of this process (the user is actively waiting for results to mark them as positive or negative examples) we use a fast bag-of-features matching (BoF) with dense sampled SIFT and ColorLayout features. Feature matching of an investigated keyframe \(K\) against all positive and all negative object instances \(O\) is thereby performed with a nearest neighbor strategy and the histogram intersection:  
\[histogramIntersection=\sum_{i=0}^{i<cbSize}\max{(K_{i},O_{i})}-O_{i} \tag{1}\]where \(i\) is the current codebook index of both BoFs. This histogram intersection decrease the importance of background objects that are shown in both images compared to traditional distance measures such as Minkowski distances and statistical measures [99]. Keyframes are suggested to the user if the distance of the current BoF to nearest positive neighbor is below a certain threshold and the ratio between the nearest positive neighbor and the nearest negative neighbor is above another threshold. The suggested keyframes are shown to the user as soon as they are available. Instantly after the user provides feedback (if a keyframe contains the object-of-interest or not), the visual features of this frame are also used for further keyframe suggestions. This process is terminated after a few object examples have been identified.