## 3 Object-of-Interest Annotation
### Recognition Approach Selection  
In the automatic approach selection, we compute the results for different features and recognition approaches, evaluate them, and combine the best ones. Optionally, we further present the results to the user to get a feedback. As shown in Figure 2, this process consists of the three steps: analysis, evaluation, and fusion. The initial object examples are used to compare different recognition approaches against each other and, if user feedback is given, these steps are iteratively executed to improve the achieved results. As specified in Algorithm 3, appropriate approaches are selected after the recognition results of all approaches have been individually generated and evaluated. This algorithm uses the popular leave-one-out strategy [71] to match all object examples except one against the keyframes of all videos  
Figure 3: Automatic detection of object regions from a few keyframes. and the left-out object in each iteration. The position of this left-out object within all results is then used for evaluation. The algorithm can be applied to analyze a few hundred hours of video at once. However, it is also reasonable to start the analysis on a single video. In each iteration, an ordered list with object instances is returned and one combined list per recognition approach is then generated using the same fusion that is later used to combine the selected recognition approaches for the investigated object. In the following, details about analysis, evaluation, and fusion are given. ```
vector<recognition approach>\(approaches\) = recognition approaches to investigate vector<instance>\(objects\) = the initially selected object instances vector<keyframe>\(collection\) = videos to annotate Output: vector<instance>\(results\) for all\(a\) in\(approaches\)do vector<features>\(resultsA\) for all\(leftOut\) in\(objects\)do if\(leftOut\).positiveExample() is false then continue endif \(query\) = \(objects\) - \(leftOut\) \(testset\) = \(collection\) + \(leftOut\) \(orderedResults\) = analysis(\(query\), \(dataset\) ) \(position\) = evaluate(\(orderedResults\), \(leftOut\) ) \(eval\).add(\(position\) ) \(resultsA\).add(\(orderedResults\) ) endfor \(result\).add(fusion(\(resultsA\) ) ) endfor vector<setup>\(bestApproaches\) = selectBest(\(eval\) ) \(results\) = fusion(\(results\), \(bestApproaches\) )
```  
**Algorithm 3** Recognition Approach Selection  
**Analysis:** In the analysis step, we try to recognize the selected object in all keyframes with a couple of different recognition approaches. Object instances are generated for each recognition approach and ordered according to their recognition probability. Instances with a high recognition probability are given on top of this list. As mentioned before, analysis is not restricted to specific recognition approaches, but the following approaches have been used in the experiments of this work with the features mentioned in Section 3.1. On the one hand, we have the matching strategies: nearest neighbor, k-nearest neighbor, nearest neighbor distance ratio [67], thresholding as well as their combinations. All of these strategies have been used with the dissimilarity measures Manhattan distance (L1), Euclidian distance (L2), Canberra metric, Jeffrey divergence, Psi Square, cosine function based similarity, and histogram intersection. Please refer to [99] for details about this dissimilarity measures. On the other hand, we used SVMs [71] with different kernels and applied a few post-processing steps, like geometric verification [67] and feature voting. Each approach further contains a number of parameters, such as the threshold of simple feature matching, that can be tuned to improve the recognition of specific objects. Further details about the actual recognition approaches are given in the related works section. **Evaluation:** If no region information is given in the initial object examples, a positive match is generated when the left-out keyframe is contained in the analysis results and we return the position of this match.