## 2 Related Work
### Content-Based Annotation and Retrieval  
Several content-based video retrieval systems have been proposed in the literature, such as the QBIC system [21, 33], and the systems JACOB, VideoQ, Violence, Netra-V, VisualGREP, InsightVideo, ANVIL, and VARS that are reviewed in [22, 34]. Most of these systems stem from the 1990s and are focused on query by example techniques that have been adapted from image retrieval [25]. They usually extract low-level features (color, shape, texture) from keyframes and only a few of these systems incorporate motion [35]. Specific video domains, for instance broadcast news and sport, have been investigated in [36, 37] to annotate specifics like the sport type, playing fields, players and presenters, the audience and inserted text. Moreover, a few systems have been proposed that are specifically designed for large-scale video collections [38, 39, 40] that focus on scalability rather than most accurate analysis approaches. The  
Figure 1: A high-level perspective of the proposed annotation framework. predominant inconvenience of query by example systems is the query generation itself. In contrast to text queries, it is quite difficult to obtain content examples that fit to specific retrieval goals that a user might have in mind. Thus, recent research mainly focused on approaches for the generation of additional, richer metadata that is suitable for text-based video retrieval. In the simplest case, videos are thereby classified into a few predefined genres using text, audio, and visual cues [41]. Most approaches annotate videos with up to a few hundred concepts or labels. This task is also referred to as \"video concept detection\", \"video semantic analysis\", and \"high-level feature extraction\" [42]. Proposed techniques include support vector machines, Gaussian mixture models, maximum entropy methods, modified nearest-neighbour classifier, and multiple instance learning as described in the survey of [PERSON] _et al_. [43]. However, concepts consist of fuzzy semantics. Objects and object relations are given as concepts together with events and sometimes even with attributes like the size and speed of objects [44]. Concept annotation is usually performed globally for entire videos and without spatial information. A recent study about video indexing and retrieval [20] describes the state-of-the-art in video structure analysis, the used visual features, and analysis techniques. Another trend in the computer vision society is the use of external knowledge and existing annotations for the generation of new metadata. Similar images or videos are obtained by searching (starting from a few keywords) to collect further tags for the investigated content, see [45, 46]. Search-based approaches [47] are further used to collect a sufficiently large training set for machine learning environments, for instance in the video concept detection system TubeTagger [48]. [49] proposed an approach to employ the redundancy of online video portals in order to improve the annotation of videos. They performed near duplicate video detection is performed to detect identical and overlapping videos that have been uploaded to exchange their annotations. Traditional video retrieval is mainly performed on a global level for entire videos. This trend is fostered by video portals that only allow global annotations and that mainly consist of short, user-generated videos. However, some annotation and retrieval techniques are performed on scene and shot level. For instance, the ShotTagger [50] allows shot-based video tagging and the authors presented a user study that indicates that the resulting retrieval is more convenient for users than the standard approach. The ViGOR system [51] presents a video retrieval system for complex search tasks, like the query to retrieve shots where a person is walking or riding a bicycle.