## 4 Recognition Infrastructure
### Visual Feature Extraction  
The extraction of visual features from video frames presents the starting point of the proposed object annotation. This extraction consists of several modules for image loading and pre-processing (e.g., to scale and normalize the keyframes), or for region-of-interest detection and description, and these modules have to be executed in an appropriate order. The actual modules differ from task to task and Figure 6 shows the used modules of the first three annotation tasks in a visual feature extraction graph. Global SIFT features (_keyframe extraction_), a bag-of-features approach with dense sampled SIFT and ColorLayout features (_similar keyframes_), and local SIFT features from Difference of Gaussian (_DoG_) points (_object regions_) are used for these tasks. In this feature extraction graph, modules are shown as labeled rectangles and the arrows between them indicate the interconnections of their input and output. The output of each step (e.g., images from the image reader) can be used as input for several other modules and they can be combined to visual features as shown on the right side of the graph. Modules that are used for more than one task are shown in multiple colors in Figure 6. The repeated use of the same intermediate results accelerates the feature extraction significantly and reduces the memory requirements  
modules, _filters_ can be specified to use only inputs that fulfill certain criteria. For instance, it is possible to use only regions with a certain gray-level contrast in their neighborhood or to use the 300 regions with the highest contrast, as specified for DoG-SIFT features (Line 17). **Modules:** As shown in Figure 7, each module provides methods to explicitly state which input descriptors it expects, which output descriptors it will generate, and which parameters it holds. The internal state of a module instance can be changed using the configure method. The actual algorithms and functionality of a module are hidden in the generate-descriptors method. Each module further contains a description method to explain its functionality. New recognition approaches and algorithms can be added to CORI either as heavy-weight modules (that perform many processing steps internally) or as set of light-weight modules that contain smaller processing steps separately. Light-weight modules facilitate the reuse of intermediate results for succeeding modules. This improves computation and memory efficiency. **Descriptors:** Descriptors present all data of the recognition infrastructure in a type-safe way. They are implemented as smart pointers (Boost library) to prevent unnecessary copy operations. Furthermore, they provide binary serialization and de-serialization capabilities, see Figure 7. Descriptors of the same type can be stored in descriptor-sets together with the information from which modules they stem. Several descriptor-sets can be grouped together in a model where they are stored in a map according to their type. Models are used as IO-arguments for the generate-descriptors method of modules and visual feature extraction graphs and they provide methods to add, remove, and access descriptor-sets and single descriptors. Most visual features are described by vectors of basic data types, such as float-vectors. More complex features can be stored in user-defined descriptors or as combinations of simple descriptors in a model. User-defined descriptors can be added to CORI by inheritance from the descriptor base class and by overwriting the serialization methods. In addition to vectors of the basic data types, images, parameters and recognition results are the most important descriptors.