nils-herrmann/clean_scirepeval huggingface dataset:
Self-calibrating 3D context for retrieving people with luggage

We outline the retrieval of images from a network of security cameras by means of an attribute-based query. Our approach is based on detectors for several object classes which enable combined queries to retrieve people based on characteristic pieces of luggage. The approach works independently of camera recording frame rates since it does not rely on tracking or background assumptions, and it requires neither real training images nor manual annotations since it is entirely trained on synthetic data. By performing an approximate 3D auto-calibration for each camera from a few detected humans and exploiting object-level context in a 3D coordinate system, we can significantly improve the precision of otherwise weakly performing detectors for inconspicuous object classes. We evaluate our approach on data from an airport security camera network and demonstrate the system's ability to respond to combined appearance and 3D metric contextual attribute queries over multiple cameras.
Position calibration of microphones and loudspeakers in distributed computing platforms

We present a novel algorithm to automatically determine the relative three-dimensional (3-D) positions of audio sensors and actuators in an ad-hoc distributed network of heterogeneous general purpose computing platforms such as laptops, PDAs, and tablets. A closed form approximate solution is derived, which is further refined by minimizing a nonlinear error function. Our formulation and solution accounts for the lack of temporal synchronization among different platforms. We compare two different estimators, one based on the time of flight and the other based on time difference of flight. We also derive an approximate expression for the mean and covariance of the implicitly defined estimator using the implicit function theorem and approximate Taylors' series expansion. The theoretical performance limits for estimating the sensor 3-D positions are derived via the Crame/spl acute/r-Rao bound (CRB) and analyzed, with respect to the number of sensors and actuators, as well as their geometry. We report extensive simulation results and discuss the practical details of implementing our algorithms in a real-life system.
Precise Visibility Determination of Displays in Camera Images

In many novel application scenarios such as smart rooms or sensing rooms visual sensors (such as cameras) need to know which visual actuators (such as displays) are visible to them. Often only parts of a display are visible from a camera. Therefore, a novel algorithm for precise visibility determination is presented. The algorithm makes the assumption that the displays are active, i.e., they can be controlled by the application. Under these conditions the algorithm determines precisely where which parts of a display are imaged by a camera
Learning to Reassemble Shredded Documents

In this paper, we address the problem of automatically assembling shredded documents. We propose a two-step algorithmic framework. First, we digitize each fragment of a given document and extract shape- and content-based local features. Based on these multimodal features, we identify pairs of corresponding points on all pairs of fragments using an SVM classifier. Each pair is considered a point of attachment for aligning the respective fragments. In order to restore the layout of the document, we create a document graph in which nodes represent fragments and edges correspond to alignments. We assign weights to the edges by evaluating the alignments using a set of inter-fragment constraints which take into account shape- and content-based information. Finally, we use an iterative algorithm that chooses the edge having the highest weight during each iteration. However, since selecting edges corresponds to combining groups of fragments and thus provides new evidence, we reevaluate the edge weights after each iteration. We quantitatively evaluate the effectiveness of our approach by conducting experiments on a novel dataset. It comprises a total of 120 pages taken from two magazines which have been shredded and annotated manually. We thus provide the means for a quantitative evaluation of assembly algorithms which, to the best of our knowledge, has not been done before.
A fast method for training support vector machines with a very large set of linear features

Current systems for object detection often use support vector machines (SVM) as the basic classification algorithm. A rather common case is to compute a small set of linear features and then train the classifier on these features. We present a fast method to train and evaluate SVM with many linear features and show results for face detection using a set of 210400 features. The resulting classifier is both more accurate and faster than a classifier trained on raw pixel features, which total up only to 576 features in our case.
Learning an object class representation on a continuous viewsphere

We propose an approach to multi-view object class detection and approximate 3D pose estimation. It relies on CAD models as positive training examples and discriminatively learns photometric object parts such that an optimal coverage of intra-class and viewpoint variation is guaranteed. In contrast to previous work, the approach shows a significantly reduced training set dependency while avoiding any manual training supervision or annotation, since it is capable of deriving all relevant information exclusively from the provided set of 3D CAD models and an arbitrary set of 2D negative images. In entirely circumventing semantic or view-based representations, part symmetries and co-occurrences between viewpoints can be efficiently exploited. This, in turn, leads to a significantly lower complexity while still achieving state-of-the-art performance on two current benchmark data sets for two different object classes.
A graph algorithmic framework for the assembly of shredded documents

In this paper we propose a framework to address the reassembly of shredded documents. Inspired by the way humans approach this problem we introduce a novel algorithm that iteratively determines groups of fragments that fit together well. We identify such groups by evaluating a set of constraints that takes into account shape- and content-based information of each fragment. Accordingly, we choose the best matching groups of fragments during each iteration and implicitly determine a maximum spanning tree of a graph that represents alignments between the individual fragments. After each iteration we update the graph with respect to additional contextual knowledge. We evaluate the effectiveness of our approach on a dataset of 16 fragmented pages with strongly varying content. The robustness of the proposed algorithm is finally shown in situations in which material is lost.
PLSA on Large Scale Image Databases

The Web and image repositories such as Fickrtrade are the largest image databases in the world. There are billions of images on the web, and hundreds of million high-quality images in image repositories. Currently, these images are indexed based on manually-entered tags and individual and group usage patterns. In this work we a exploring a third information dimension: image features. We are exploring probabilistic latent semantic analysis in order to infer which visual patterns describe each object. We wish to build models that connect words and image features, and use content features and tags to better find similar images.
A detector tree of boosted classifiers for real-time object detection and tracking

This paper presents a novel tree classifier for complex object detection tasks together with a general framework for real-time object tracking in videos using the novel tree classifier. A boosted training algorithm with a clustering-and-splitting step is employed to construct branches in the nodes recursively, if and only if it improves the discriminative power compared to a single monolithic node classifier and has a lower computational complexity. A mouth tracking system that integrates the tree classifier under the proposed framework is built and tested on XM2FDB database. Experimental results show that the detection accuracy is equal or better than a single or multiple cascade classifier, while being computational less demanding.
Fast and Effective Features for Recognizing Recurring Video Clips in Very Large Databases

Three different frame features (color patches, color coherence vectors, and gradient histograms) are investigated for their suitability to recognize recurring video clips in very large databases. They are evaluated in a real-time processing and real-time recognition system. Real-time recognition means that each clip must be recognized one second after its start. As the experimental results show, only gradient histograms work satisfactorily across different video material with the same video domain independent parameter set. For instance, they are - in contrast to color features - not negatively affected by dark frame sequences in video clips and the live video stream. By means of pre- computation and subsequent table look-ups, gradient histograms can be implemented such that their computational costs come very close to that of color features.
Localizing and segmenting text in images and videos

Many images, especially those used for page design on Web pages, as well as videos contain visible text. If these text occurrences could be detected, segmented, and recognized automatically, they would be a valuable source of high-level semantics for indexing and retrieval. We propose a novel method for localizing and segmenting text in complex images and videos. Text lines are identified by using a complex-valued multilayer feed-forward network trained to detect text at a fixed scale and position. The network's output at all scales and positions is integrated into a single text-saliency map, serving as a starting point for candidate text lines. In the case of video, these candidate text lines are refined by exploiting the temporal redundancy of text in video. Localized text lines are then scaled to a fixed height of 100 pixels and segmented into a binary image with black characters on white background. For videos, temporal redundancy is exploited to improve segmentation performance. Input images and videos can be of any size due to a true multiresolution approach. Moreover, the system is not only able to locate and segment text occurrences into large binary images, but is also able to track each text line with sub-pixel accuracy over the entire occurrence in a video, so that one text bitmap is created for all instances of that text line. Therefore, our text segmentation results can also be used for object-based video encoding such as that enabled by MPEG-4.
Acoustic source localization in distributed sensor networks

This paper studies the problem of sound source localization in a distributed wireless sensor network formed by mobile general purpose computing and communication devices with audio I/O capabilities. In contrast to well understood localization methods based on dedicated microphone arrays, in our setting sound localization is performed using a sparse array of arbitrary placed sensors (in a typical scenario, localization is performed by several laptops/PDAs co-located in a room). Therefore any far-field assumptions are no longer valid in this situation. Additionally, localization algorithm's performance is affected by uncertainties in sensor position and errors in A/D synchronization. The proposed source localization algorithm consists of two steps. In the first step, time differences of arrivals (TDOAs) are estimated for the microphone pairs, and in the second step the maximum likelihood (ML) estimation for the source position is performed. We evaluate the Cramer-Rao bound (CRB) on the variance of the location estimation and compare it with simulations and experimental results. We also discuss the effects of distributed array geometry and errors in sensor positions on the performance of the localization algorithm. The performances of the system are likely to be limited by errors in sensor locations and increase when the microphones have a large aperture with respect to the source.
Approximating Optimal Visual Sensor Placement

Many novel multimedia applications use visual sensor arrays. In this paper we address the problem of optimally placing multiple visual sensors in a given space. Our linear programming approach determines the minimum number of cameras needed to cover the space completely at a given sampling frequency. Simultaneously it determines the optimal positions and poses of the visual sensors. We also show how to account for visual sensors with different properties and costs if more than one kind is available, and report performance results.
Synchronization of acoustic sensors for distributed ad-hoc audio networks and its use for blind source separation

Advanced computing devices equipped with various wired and wireless network capabilities, built-in microphones and audio capture devices are becoming increasingly popular. At the same time, sophisticated signal processing algorithms for hands-free acoustic human-machine interfaces are being developed. Those algorithms are currently restricted to dedicated audio hardware, in part because they require perfectly synchronized audio data. Naive attempts to use the available audio devices for microphone array processing in a distributed wireless setting fails due to the algorithms' sensitivity to deviations in the sampling rates of the distributed devices. We propose a synchronization scheme, which combines the microphones of different spatially distributed computing devices to an acoustic ad-hoc network. The proposed scheme is capable to significantly compensate the sampling rate deviations of the different audio capture devices and we show that, as an example, blind source separation performs well with the synchronized data of distributed acoustic sensors.
Improving VLAD: Hierarchical coding and a refined local coordinate system

The enormous growth of image databases calls for new techniques for fast and effective image search that scales with millions of images. Most importantly, the setting requires a compact but also descriptive image signature. Recently, the vector of aggregated local descriptors (VLAD)  has received much attention in large-scale image retrieval. In this paper we present two modifications for VLAD which improve the retrieval performance of the signature.
A system for effortless content annotation to unfold the semantics in videos

We propose and investigate a new but simple and natural extension of the way people record video. This extension allows one to unfold the semantics of video clips and thus enables a completely new set of applications for raw video footage. Two microphones are connected to a camcorder: a headworn speech input microphone and an environmental microphone. During recording the cameraman speaks out loud content-descriptive annotations and/or editing commands. Due to the two-microphones setup the sound of annotations and editing commands can be removed from the environmental audio by adaptive filtering enabling people to play back the video as if there had been no annotations. Simultaneously, these annotations are transcribed to ASCII by means of a standard speech recognition engine. The viability of this approach is demonstrated by means of an important application for video libraries: the automatic abstraction of raw video footage.
Filtering adult image content with topic models

Protecting children from exposure to adult content has become a serious problem in the real world. Current statistics show that, for instance, the average age of first Internet exposure to pornography is 11 years, that the largest consumer group of Internet pornography is the age group of 12-to-17- year-olds and that 90% of the 8-to-16-year-olds have viewed porn online. To protect our children, effective algorithms for detecting adult images are needed. In this research we evaluate the use of probabilistic Latent Semantic Analysis (pLSA) for this task. We will show that topic models based on pLSA can detect adult content with a correct positive rate of 92.7%, while only showing off a false positive rate of 1.9%. Even when using grayscale images only, a correct positive rate of 90.8% at a false positive rate of 2% can be achieved.
Sound texture modelling with linear prediction in both time and frequency domains

Sound textures - for instance, a crackling fire, running water, or applause - constitute a large and largely neglected class of audio signals. Whereas tonal sounds have been effectively and flexibly modelled with sinusoids, aperiodic energy is usually modelled as white noise filtered to match the approximate spectrum of the original over 10-30 ms windows, which fails to provide a perceptually satisfying reproduction of many real-world noisy sound textures. We attribute this failure to the loss of short-term temporal structure, and we introduce a second modelling stage in which the time envelope of the residual from conventional linear predictive modelling is itself modelled with linear prediction in the spectral domain. This cascade time- and frequency-domain linear prediction (CTFLP) leads to noise-excited resyntheses that have high perceptual fidelity. We perform a novel quantitative error analysis by measuring the proportional error within time-frequency cells across a range of timescales.
Spectral vs. spectro-temporal features for acoustic event detection

Automatic detection of different types of acoustic events is an interesting problem in soundtrack processing. Typical approaches to the problem use short-term spectral features to describe the audio signal, with additional modeling on top to take temporal context into account. We propose an approach to detecting and modeling acoustic events that directly describes temporal context, using convolutive non-negative matrix factorization (NMF). NMF is useful for finding parts-based decompositions of data; here it is used to discover a set of spectro-temporal patch bases that best describe the data, with the patches corresponding to event-like structures. We derive features from the activations of these patch bases, and perform event detection on a database consisting of 16 classes of meeting-room acoustic events. We compare our approach with a baseline using standard short-term mel frequency cepstal coefficient (MFCC) features. We demonstrate that the event-based system is more robust in the presence of added noise than the MFCC-based system, and that a combination of the two systems performs even better than either individually.
Multispeaker speech activity detection for the ICSI meeting recorder

As part of a project into speech recognition in meeting environments, we have collected a corpus of multichannel meeting recordings. We expected the identification of speaker activity to be straightforward given that the participants had individual microphones, but simple approaches yielded unacceptably erroneous labelings, mainly due to crosstalk between nearby speakers and wide variations in channel characteristics. Therefore, we have developed a more sophisticated approach for multichannel speech activity detection using a simple hidden Markov model (HMM). A baseline HMM speech activity detector has been extended to use mixtures of Gaussians to achieve robustness for different speakers under different conditions. Feature normalization and crosscorrelation processing are used to increase the channel independence and to detect crosstalk. The use of both energy normalization and crosscorrelation based postprocessing results in a 35% relative reduction of the frame error rate. Speech recognition experiments show that it is beneficial in this multispeaker setting to use the output of the speech activity detector for presegmenting the recognizer input, achieving word error rates within 10% of those achieved with manual turn labeling.
Soundtrack classification by transient events

We present a method for video classification based on information in the soundtrack. Unlike previous approaches which describe the audio via statistics of mel-frequency cepstral coefficient (MFCC) features calculated on uniformlyspaced frames, we investigate an approach to focusing our representation on audio transients corresponding to sound-track events. These event-related features can reflect the “foreground” of the soundtrack and capture its short-term temporal structure better than conventional frame-based statistics. We evaluate our method on a test set of 1873 YouTube videos labeled with 25 semantic concepts. Retrieval results based on transient features alone are comparable to an MFCC-based system, and fusing the two representations achieves a relative improvement of 7.5% in mean average precision (MAP).
Micbots: Collecting large realistic datasets for speech and audio research using mobile robots

Speech and audio signal processing research is a tale of data collection efforts and evaluation campaigns. Large benchmark datasets for automatic speech recognition (ASR) have been instrumental in the advancement of speech recognition technologies. However, when it comes to robust ASR, source separation, and localization, especially using microphone arrays, the perfect dataset is out of reach, and many different data collection efforts have each made different compromises between the conflicting factors in terms of realism, ground truth, and costs. Our goal here is to escape some of the most difficult trade-offs by proposing MICbots, a low-cost method of collecting large amounts of realistic data where annotations and ground truth are readily available. Our key idea is to use freely moving robots equiped with microphones and loudspeakers, playing recorded utterances from existing (already annotated) speech datasets. We give an overview of previous data collection efforts and the trade-offs they make, and describe the benefits of using our robot-based approach. We finally explain the use of this method to collect room impulse response measurement.
Audio Set: An ontology and human-labeled dataset for audio events

Audio event recognition, the human-like ability to identify and relate sounds from audio, is a nascent problem in machine perception. Comparable problems such as object detection in images have reaped enormous benefits from comprehensive datasets - principally ImageNet. This paper describes the creation of Audio Set, a large-scale dataset of manually-annotated audio events that endeavors to bridge the gap in data availability between image and audio research. Using a carefully structured hierarchical ontology of 632 audio classes guided by the literature and manual curation, we collect data from human labelers to probe the presence of specific audio classes in 10 second segments of YouTube videos. Segments are proposed for labeling using searches based on metadata, context (e.g., links), and content analysis. The result is a dataset of unprecedented breadth and size that will, we hope, substantially stimulate the development of high-performance audio event recognizers.
Evaluating Source Separation Algorithms With Reverberant Speech

This paper examines the performance of several source separation systems on a speech separation task for which human intelligibility has previously been measured. For anechoic mixtures, automatic speech recognition (ASR) performance on the separated signals is quite similar to human performance. In reverberation, however, while signal separation has some benefit for ASR, the results are still far below those of human listeners facing the same task. Performing this same experiment with a number of oracle masks created with a priori knowledge of the separated sources motivates a new objective measure of separation performance, the Direct-path, Early echo, and Reverberation, of the Target and Masker (DERTM), which is closely related to the ASR results. This measure indicates that while the non-oracle algorithms successfully reject the direct-path signal from the masking source, they reject less of its reverberation, explaining the disappointing ASR performance.