## 1 Introduction  
Video sharing has become one of the most important activities in the internet and it accounts for a large share of today's internet traffic [1]. Professional videos and user-generated content of different genres are available on video portals like YouTube, YouKo, Vimeo, Hulu, and Metacafe [2; 3; 4; 5; 6]. Millions of users visit such portals every day [7] in order to watch videos on their computer screens and mobile phones. Despite this success, current video sharing and retrieval works quite differently from classical internet document retrieval for text documents, websites, and images. In order to understand the big picture of current video sharing and its development potentials it is necessary to get a deeper knowledge of the roles and aims of video portals, their users, and about the nature of video documents. Thus, we start with a brief overview of these topics before a flexible object-of-interest annotation framework is proposed for the assistance of various video portals. _Video documents_ consist of audio-visual content and they are accompanied by metadata. The entities of video documents are scenes, shots, and frames on a temporal level; objects on spatial level; and object related events on a semantic level [8]. Bibliographic metadata is used to describe general video attributes, such as the video title, its genre, owner, duration, and number of views on a video portal. Structural metadata includes the hierarchy of scenes and shots while content metadata describes information like the shown actors and audio transcripts of a video [9]. In the process of video _annotation_, metadata of all three types is generated using manual, interactive, or automatic annotation techniques. Most video _retrieval_ techniques solely operate on metadata although the video content itself can also be used by text queries and visual queries [10]. More than a decade ago, studies about the formulations of text queries have been made in [9], and [11]. They distinguished between (a) queries about the videos as a whole; (b) queries about the topic content; (c) queries about sensory content (e.g., the appearance or location of objects); and (d) queries about the data and metadata. In contrast to text queries, visual queries use images or videos as input to retrieve similar videos or videos that show specific objects. The main difficulty of these visual queries is the generation or retrieval of appropriate query content itself. _Video portals_ are one of the major success stories of the Web 2.0. Since YouTube went online in 2005 an immense number of videos made their way through the internet and more than 50% of all web users visit one of several hundred online portals on a daily basis [7]. In short, these portals try to provide as many videos as possible and to maximise the number of page visits and views. Complex server architectures and service models are used [1] to deal with the vast amount of data requests. User generated video portals like YouTube, and YouKo mainly rely on advertisement as their source of revenue while paid accounts are frequently used from domain specific portals such as Netflix where only professional content is shown [12]. Portals for user-generated content allow video uploads and users can annotate their own videos with free text tags and make comments on videos they have watched. Most portals can only be used to generate metadata that belongs to an entire video while a few systems such as Synvie [13] and the prototypical system of [14] enable users to make comments on video scenes and shots. The portals use own, proprietary metadata formats and differentiate between different video genres. Furthermore, there are often restrictions for the video content, codecs, and duration that is allowed for uploading. Beside video consumption on the websites or in external video players, each portal provides some video retrieval and browsing services. Videos are presented with one or multiple keyframes and metadata, such as its title, duration, number of views, and a general description.