## 5 Experiments
### Trevid  
The main purpose of the presented framework is the annotation of video objects in order to improve the text-based video retrieval of online portals. Unfortunately, it is very difficult to compare the results of different annotation systems against each other because the output of these systems vary in several matters, such as the representation of objects. Furthermore, users can annotate different objects in the same video, give different labels to the same objects, and operate with a different annotation speed and accuracy. However, these limiting factors do not exist for example-based video object retrieval, and thus initiatives like the TRECVID instance search tasks emerged lately to provide benchmarks for retrieval systems. We took the opportunity to participate at this pilot task that exists since 2010 with the goal to retrieve video shots of specific objects, persons, or locations. As shown in Figure 13, the used object queries are similar to the initially selected objects of the case study in Section 5.2, though a perfect segmentation is given as binary mask. Each query contains between 2 and 6 keyframes of the object-of-interest, extracted from nearby video sequences.  
The National Institute of Standards and Technology (NIST) performs the ground-truth generation and the evaluation of this TRECVID task. In this ground-truth, 1830 video shots of 25 different objects (see the bottom of Figure 14) are annotated in about 136 h of video. The used BBC rushes video collection contains raw material from which programs and films are made in the editing room. These videos include material for several dramatic series as well as for travel programs. A shot detection was already performed by NIST to produce a collection of 20982 short clips and it was specified that these clips have to be treated independently during analysis. We extracted about 70000 keyframes from these shots using Algorithm 1 for further analysis. The output file of each video object query contains up to 1000 ordered shots with those shots on top that are most likely to contain the object-of-interest. In the evaluation, NIST computes the average precision for each query together with a combined mean average precision for all queries [63]. These measures indicate how much of the annotated video sequences are recognized with a higher weighting of the topmost list entries.  
Figure 13: Object examples of the instance search task.  
Figure 14 shows the achieved results for each object as black dots together with the median results of all 14 participating teams as dashed line. Our results are generally above this median except for a few objects where we achieved similar results. Overall a mean average precision of 0.170 was achieved and we retrieved up to 847 of the 1830 annotated ground-truth shots correctly. The left side of Figure 15 shows the interpolated recall-precision curve over all queries and, on the right side of this figure, the precision for the first \(n\) shots of the output list. Both diagrams indicate that the proposed framework works accurate to identify correct shots of a query object on top of the result list. About 15% of all objects instances are detected with a recall above 50%. These instances are good candidates for the automatic annotation of Section 3.4 where precision is more important than recall and where a few false annotations are acceptable. User feedback for the first 48 video suggestions (as done in the case study) would lead to an annotation of about 25% of all object instances in this task. On average this means that we would annotate about 12 instances per object query, which is almost the same result that we achieved in the case study of Section 5.2, compare Figure 11.  
Figure 14: Evaluation results for each query.  
Figure 15: Interpolated recall-precision curve (left) and precision at \(n\) shots (right).