nils-herrmann/clean_scirepeval huggingface dataset:
Commercial video retrieval by induced semantics

Video information processing and retrieval is a key challenge for future multimedia technologies and applications. Commercial videos encode several planes of expression through a rich and dense use of colors, editing effects, viewpoints and rhythms, which are exploited together to attract potential purchasers. In this paper, previous research in the marketing and semiotics fields is translated into a multimedia engineering perspective, and a link is formalized between visual features of commercials at the perceptual level and the specific information that is being vehiculated to the audience. The link allows us to define higher level semantic features capturing the main narrative structures of the video, and embed them in a video retrieval system supporting access to a database of commercials based on four different semiotic categories.
Bloom Filters and Compact Hash Codes for Efficient and Distributed Image Retrieval

This paper presents a novel method for efficient image retrieval, based on a simple and effective hashing of CNN features and the use of an indexing structure based on Bloom filters. These filters are used as gatekeepers for the database of image features, allowing to avoid to perform a query if the query features are not stored in the database and speeding up the query process, without affecting retrieval performance. Thanks to the limited memory requirements the system is suitable for mobile applications and distributed databases, associating each filter to a distributed portion of the database (database shard), addressing large scale archives and allowing query parallelization. Experimental validation has been performed on three standard image retrieval datasets, outperforming state-of-the-art hashing methods in terms of precision, while the proposed indexing method obtains a 2x speedup.
3D content-based retrieval with spin images

Along with images and videos, 3D models have recently gained increasing attention for a number of reasons: advancements in 3D hardware and software technologies; their ever decreasing prices and increasing availability; affordable 3D authoring tools; the establishment of open standards for 3D data interchange. The ever increasing availability of 3D models demands tools to support their effective and efficient management. Among these tools, those enabling content-based retrieval play a key role. We present a novel approach to 3D content-based retrieval that is based on spin images. Spin images are used to derive a view-independent description of both database and query objects: a set of spin images is first created for each object; then, a descriptor is evaluated for each spin image in the set; clustering is performed on the set of image-based descriptors of each object to achieve a compact representation of the object, thus allowing for efficient indexing and matching. Experimental results are presented for a test database of about 300 models. These results indicate that spin images can be successfully exploited for content-based retrieval of 3D objects
Three-Dimensional Interfaces for Querying by Example in Content-Based Image Retrieval

Image databases are widely exploited in a number of different contexts, ranging from history of art, through medicine, to education. Existing querying paradigms are based either on the usage of textual strings, for high-level semantic queries or on 2D visual examples for the expression of perceptual queries. Semantic queries require manual annotation of the database images. Instead, perceptual queries only require that image analysis is performed on the database images in order to extract salient perceptual features that are matched with those of the example. However, usage of 2D examples is generally inadequate as effective authoring of query images, attaining a realistic reproduction of complex scenes, needs manual editing and sketching ability. Investigation of new querying paradigms is therefore an important-yet still marginally investigated-factor for the success of content-based image retrieval. In this paper, a novel querying paradigm is presented which is based on usage of 3D interfaces exploiting navigation and editing of 3D virtual environments. Query images are obtained by taking a snapshot of the framed environment and by using the snapshot as an example to retrieve similar database images. A comparative analysis is carried out between the usage of 3D and 2D interfaces and their related query paradigms. This analysis develops on a user test on retrieval efficiency and effectiveness, as well as on an evaluation of users' satisfaction.
Modified filtering algorithm (MFA) for backbone interconnection-hardware design and implementation issues

The problem of fast filtering in bridging Ethernet local-area-networks (LANs) through high speed fiber distributed data interface (FDDI) networks is addressed. A filtering algorithm, called MFA is introduced, which sets the minimum power requirements for a bridge connecting an Ethernet LAN to an FDDI backbone. A hardware solution based on the use of content addressable memories is described. The MFA tries to avoid packet broadcasting on the FDDI ring. This is done by keeping track of remote Ethernet addresses coupled with the FDDI address of the related bridge. With MFA, the bridge performs filtering and routing operations only on packets coming from the Ethernet side, while the filtering from FDDI side is directly performed by the FDDI media access control. As a result, MFA has only to cope with the maximum Ethernet transmission rate, slower by one order of magnitude, and thus requiring less processing power.<<ETX>>
Spatial arrangement of color flows for video retrieval

Retrieval of video based on high-level content requires that models are developed to map low level perceptual features into high level semantic concepts. Commercials are a video category where the link between low level perceptual features and high level semantics is stressed, since the way colors are chosen and modified throughout a spot create a large part of the message. In this paper, we propose a model for representation and comparison of video content based on the spatial arrangement of color flows. A model for representation and comparison of spatial relationships between extended sets of pixels in a 3D space is introduced by developingon the concept of weighted walkthroughs. Results of preliminary experiments are reported for a library of video commercials.
Object Tracking by Oversampling Local Features

In this paper, we present the ALIEN tracking method that exploits oversampling of local invariant representations to build a robust object/context discriminative classifier. To this end, we use multiple instances of scale invariant local features weakly aligned along the object template. This allows taking into account the 3D shape deviations from planarity and their interactions with shadows, occlusions, and sensor quantization for which no invariant representations can be defined. A non-parametric learning algorithm based on the transitive matching property discriminates the object from the context and prevents improper object template updating during occlusion. We show that our learning rule has asymptotic stability under mild conditions and confirms the drift-free capability of the method in long-term tracking. A real-time implementation of the ALIEN tracker has been evaluated in comparison with the state-of-the-art tracking systems on an extensive set of publicly available video sequences that represent most of the critical conditions occurring in real tracking environments. We have reported superior or equal performance in most of the cases and verified tracking with no drift in very long video sequences.
A Rao-Blackwellized particle filter for EigenTracking

Subspace representations have been a popular way to model appearance in computer vision. In Jepson and Black's influential paper on EigenTracking, they were successfully applied in tracking. For noisy targets, optimization-based algorithms (including EigenTracking) often fail catastrophically after losing track. Particle filters have recently emerged as a robust method for tracking in the presence of multi-modal distributions. To use subspace representations in a particle filter, the number of samples increases exponentially as the state vector includes the subspace coefficients. We introduce an efficient method for using subspace representations in a particle filter by applying Rao-Blackwellization to integrate out the subspace coefficients in the state vector. Fewer samples are needed since part of the posterior over the state vector is analytically calculated. We use probabilistic principal component analysis to obtain analytically tractable integrals. We show experimental results in a scenario in which we track a target in clutter.
Expressive semantics for automatic annotation and retrieval of video streams

For the practical use of video retrieval systems, access must be provided so as to bridge the semantic gap between the system and users. Semiotics, which is concerned with the production of sense and of the way in which it is received by humans appears as the formal background for this goal. According to semiotics, semantics can be extracted at different levels of signification through a suitable set of rules which combine visual and auditory signs. An intermediate semantic level, which has to do with the combination of low-level signs, such as color, motion, and shape and their changes through time, accounts for the video expressiveness and the emotions provoked. It can be obtained automatically and is often useful for retrieval by content and similarity-based classification of video streams. Applications to retrieval of commercials based on expressive semantics are discussed.
An evaluation of nearest-neighbor methods for tag refinement

The success of media sharing and social networks has led to the availability of extremely large quantities of images that are tagged by users. The need of methods to manage efficiently and effectively the combination of media and metadata poses significant challenges. In particular, automatic image annotation of social images has become an important research topic for the multimedia community. In this paper we propose and thoroughly evaluate the use of nearest-neighbor methods for tag refinement. Extensive and rigorous evaluation using two standard large-scale datasets shows that the performance of these methods is comparable with that of more complex and computationally intensive approaches and that, differently from these latter approaches, nearest-neighbor methods can be applied to `web-scale' data.
Issues and directions in visual information retrieval

Visual information retrieval is attracting an increasing number of researchers from disparate fields, like image analysis, computer vision, databases, knowledge representation, artificial intelligence, man-machine interaction. Although a number of prototype systems have been made available, nevertheless this discipline has not yet reached a mature stage, and overall has not yet been credited as of concrete use in practical applications. Among the very many different lines of development we focus particularly on the importance of bridging the semantic gap between the user and visual information retrieval systems. We focus on usage of semiotics as a framework for extraction of semantics, graphs and graph matching as the representation model and retrieval engine and visualization spaces to capture semantics during the interaction.
Learning Foveal Sensing Strategies in Unconstrained Surveillance Environments

In this paper we report on techniques for automatically learning foveal sensing strategies for an active pan-tiltzoom camera. The approach uses reinforcement learning to discover foveal actions maximizing the performance of visual detectors, that are in turn assumed to be highly correlated with the task at hand. In our case, the main goal is to recognize people, hence a frontal face detection module is employed. The system uses reinforcement learning to learn if, when and how to foveate on a subject, based on its previous experience in terms or successful actions in similar situations. An action is successful if it leads to a correct face detection in the high resolution images obtained when the subject is zoomed in. In contrast with existing methods, the proposed approach obviates the need for camera calibration and camera performance modeling. Also, the method does not rely on active tracking of targets. Experimental results show how the system is capable of learning foveation strategies without requiring extensive a priori information or environmental models. Results also illustrate how the system effectively learns a strategy that allows the camera to foveate only in situations where successful detection is highly likely.
Using multiple examples for content-based image retrieval

Query specification for content based image retrieval is typically accomplished through query-by-example paradigms, such as query-by-image and query-by-sketch. In some cases query-by-sketch can be difficult because of lack of sketching abilities, difficulty to detect distinguishing image features. Therefore, querying is performed through the query-by-image paradigm. However, this paradigm often fails since a single sample image rarely includes all and only the characterizing elements the user is looking for. The paper presents a system that supports query-by-image using multiple image examples, both positive and negative. The system also enables editing of examples so as to disregard those image features that are not relevant to the query.
Improving the robustness of particle filter-based visual trackers using online parameter adaptation

In particle filter-based visual trackers, dynamic velocity components are typically incorporated into the state update equations. In these cases, there is a risk that the uncertainty in the model update stage can become amplified in unexpected and undesirable ways, leading to erroneous behavior of the tracker. Moreover, the use of a weak appearance model can make the estimates provided by the particle filter inaccurate. To deal with this problem, we propose a continuously adaptive approach to estimating uncertainty in the particle filter, one that balances the uncertainty in its static and dynamic elements. We provide quantitative performance evaluation of the resulting particle filter tracker on a set of ten video sequences. Results are reported in terms of a metric that can be used to objectively evaluate the performance of visual trackers. This metric is used to compare our modified particle filter tracker and the continuously adaptive mean shift tracker. Results show that the performance of the particle filter is significantly improved through adaptive parameter estimation, particularly in cases of occlusion and erratic, nonlinear target motion.
Video retrieval based on dynamics of color flows

Content based video retrieval is particularly challenging because the huge amount of data associated with videos complicates the extraction of salient information content descriptors. Commercials are a video category where a large part of the content depends on low level perceptual features such as colors and color dynamics. These are related to the evolution-in terms of shrinking, growth and translation"of colored regions along consecutive frames. Each colored region, during its evolution, defines a 3D volume: a color flow. In the paper, a system is presented that supports description of color flows based on 3D wavelet decomposition and retrieval of commercials based on color flow similarity.
MORF: Multi-Objective Random Forests for face characteristic estimation

In this paper we describe a technique for joint estimation of head pose and multiple soft biometrics from faces (Age, Gender and Ethnicity). Our proposed Multi-Objective Random Forests (MORF) framework is a unified model for the joint estimation of multiple characteristics that automatically adapts the measure of information gain used for evaluating the quality of weak learners. Since facial characteristics are related in the feature space, estimating all of them jointly can be beneficial as trees can learn to condition the estimation of some characteristics on others. We reformulate the splitting criterion of random trees in our multi-objective formulation and evaluate it on publicly available face characteristic estimation imagery. These preliminary experiments show promising results.
Analysis of optical flow constraints

Different constraint equations have been proposed in the literature for the derivation of optical flow. Despite of the large number of papers dealing with computational techniques to estimate optical flow, only a few authors have investigated conditions under which these constraints exactly model the velocity field, that is, the perspective projection on the image plane of the true 3-D velocity. These conditions are analyzed under different hypotheses, and the departures of the constraint equations in modeling the velocity field are derived for different motion conditions. Experiments are also presented giving measures of these departures and of the induced errors in the estimation of the velocity field.
Semantics-based retrieval by content

For the practical use of visual information retrieval systems, access to visual data must be provided so as to bridge the semantic gap between the system and users. Although the problem seems to face only the level of representation, indeed the user interface and feature organization play equally essential roles in obtaining effective retrieval systems. Semiotics, which is concerned with the description of conditions for the production of sense and of the way in which it is received by humans appears as the formal background to casting the development of retrieval by semantic information.
Adaptive Video Compression for Video Surveillance Applications

This article describes an approach to adaptive video coding for video surveillance applications. Using a combination of low-level features with low computational cost, we show how it is possible to control the quality of video compression so that semantically meaningful elements of the scene are encoded with higher fidelity, while background elements are allocated fewer bits in the transmitted representation. Our approach is based on adaptive smoothing of individual video frames so that image features highly correlated to semantically interesting objects are preserved. Using only low-level image features on individual frames, this adaptive smoothing can be seamlessly inserted into a video coding pipeline as a pre-processing state. Experiments show that our technique is efficient, outperforms standard H.264 encoding at comparable bit rates, and preserves features critical for downstream detection and recognition.
Curvature maps for 3D CBR

Along with images and videos, 3D models have recently gained increasing attention for a number of reasons: advancements in 3D hardware and software technologies, their ever decreasing prices and increasing availability, affordable 3D authoring tools, and the establishment of open standards for 3D data interchange. In this paper we address the problem of content-based retrieval of 3D models. The solution we present here relies on the description of each model by means a curvature map: after an initial pre-processing of the model, differential properties of points on the surface of the 3D object are evaluated; the model surface is then warped into an ellipsoid, and is mapped onto a 2D image retaining curvature information of the original model. Matching is performed by comparing the 2D map of the query against the 2D maps of the database models. This method has been implemented in a prototype system supporting retrieval by content of 3D objects through a Web interface.
Compact hash codes and data structures for efficient mobile visual search

In this paper we present an efficient method for mobile visual search that exploits compact hash codes and data structures for visual features retrieval. The method has been tested on a large scale standard dataset of one million SIFT features, showing a retrieval performance comparable or superior to state-of-the-art methods, and a very high efficiency in terms of memory consumption and computational requirements. These characteristics make it suitable for application to mobile visual search, where devices have limited computational and memory capabilities.
Image retrieval by elastic matching of shapes and image patterns

In this paper, we present a system for interactive image database retrieval, based on elastic template matching. The template can be both a 1D template modeling the contour of an object, and a 2D template modeling a part of an image with a significant grey-level pattern. The retrieval process is obtained as a continuous interaction, by which the original user query can be refined or changed on the basis of the results provided by the system.
Annotation and retrieval of video documents: sports videos

Live logging and posterity logging are the two basic applications for video databases. The former aims at providing effective annotation of video in quasi-real time and supports extraction of meaningful clips from the live stream; the latter provides annotation for later reuse of video material and is the prerequisite for retrieval by content from video digital libraries. Both require that information is adequately structured with interchange format and that annotation is performed, at a great extent, automatically. Video information structure must encompass both low-intermediate level video organisation and event relationships that define specific highlights and situations. Analysis of the visual data of the video stream permits to extract hints, identify events and detect highlights. All of this must be supported by a-priori knowledge of the subject and effective reasoning engines capable to capture the inherent semantics of the visual events.
Unsupervised Scene Adaptation for Faster Multi-scale Pedestrian Detection

In this paper we describe an approach to automatically improving the efficiency of soft cascade-based person detectors. Our technique addresses the two fundamental bottlenecks in cascade detectors: the number of weak classifiers that need to be evaluated in each cascade, and the total number of detection windows to be evaluated. By simply observing a soft cascade operating on a scene, we learn scale specific linear approximations of cascade traces that allows us to eliminate a large fraction of the classifier evaluation. Independently, this time by observing regions of support in the soft cascade on a training set, we learn a coarse geometric model of the scene that allows our detector to propose candidate detection windows and significantly reduce the number of windows run through the cascade. Our approaches are unsupervised and require no additional labeled person images for learning. Our linear cascade approximation results in about 28% savings in detection, while our geometric model gives a saving of over 95%, without appreciable loss of accuracy.
Segmentation of 3D Objects Using Pulse-Coupled Oscillator Networks

Along with image and video libraries, archives of 3D models have recently gained increasing attention. Accordingly, there is an increasing demand for solutions enabling retrieval of 3D models based on global properties as well as properties of object parts. In particular, retrieval based on object parts relies on segmentation of 3D objects into their constituent parts. This is a challenging task, as the identification of object parts should conform to human perceptual judgement. Therefore, definition of models and solutions that enable decomposition of 3D objects into perceptually relevant parts is a fundamental step to enable effective retrieval based on object parts. However, a few approaches have been proposed to support segmentation of 3D meshes into perceptually relevant parts. In this paper, we propose a model based on pulse-coupled oscillator networks. Preliminary experiments are reported to demonstrate the validity and potential of the proposed solution