nils-herrmann/clean_scirepeval huggingface dataset:
Towards universal visual vocabularies

Many content-based image mining systems extract local features from images to obtain an image description based on discrete feature occurrences. Such applications require a visual vocabulary also known as visual codebook or visual dictionary to discretize the extracted high-dimensional features to visual words in an efficient yet accurate way. Once such an application operates on images of a very specific domain the question arises if a vocabulary built from those domain-specific images needs to be used or if a ”universal” visual vocabulary can be used instead. A universal visual vocabulary may be computed from images of a different domain once and then be re-used for various applications and other domains. We therefore evaluate several visual vocabularies from different image domains by determining their performance at pLSA-based image classification on several datasets. We empirically conclude that vocabularies suit our classification tasks equally well disregarding the image domain they were derived from.
Activity-Conditioned Continuous Human Pose Estimation for Performance Analysis of Athletes Using the Example of Swimming

In this paper we consider the problem of human pose estimation in real-world videos of swimmers. Swimming channels allow filming swimmers simultaneously above and below the water surface with a single stationary camera. These recordings can be used to quantitatively assess the athletes' performance. The quantitative evaluation, so far, requires manual annotations of body parts in each video frame. We therefore apply the concept of CNNs in order to automatically infer the required pose information. Starting with an off-the-shelf architecture, we develop extensions to leverage activity information – in our case the swimming style of an athlete – and the continuous nature of the video recordings. Our main contributions are threefold: (a) We apply and evaluate a fine-tuned Convolutional Pose Machine architecture as a baseline in our very challenging aquatic environment and discuss its error modes, (b) we propose an extension to input swimming style information into the fully convolutional architecture and (c) modify the architecture for continuous pose estimation in videos. With these additions we achieve reliable pose estimates with up to +16% more correct body joint detections compared to the baseline architecture.
Note onset detection for the transcription of polyphonic piano music

Transcription of music is the process of generating a symbolic representation such as a score sheet or a MIDI file from an audio recording of a piece of music. A statistical machine learning approach for detecting note onsets in polyphonic piano music is presented. An area from the spectrogram of the sound is concatenated into one feature vector. A cascade of boosted classifiers is used for dimensionality reduction and classification in an one-versus-all manner. The presented system achieves an accuracy of 87.4% in onset detection outperforming the best comparison system by 25.1 %.
Building a semantic part-based object class detector from synthetic 3D models

This paper presents a new approach for multi-view object class detection based on part models. While most existing approaches have in common that they use real images for training, our approach requires only a database of synthetic 3D models to represent both the appearance and the geometry of an object class. We use semantically equivalent object points on 3D models to build part models and encode the local appearance of the parts by a discriminative learning method that applies AdaBoost to histograms of gradients. The geometric configuration of the parts is represented by spatial distributions which are also directly derived from the 3D models. For recognizing an object in an image, our model provides object hypotheses which are re-ranked with global appearance models. The 2D localization is evaluated on the PASCAL 2006 data set for cars and bicycles, showing that its performance can compete with state-of-the-art detection results.
The impact of SMT/SMP designs on multimedia software engineering - a workload analysis study

This paper presents the study of running several core multimedia applications on a simultaneous multithreading (SMT) architecture and derives design principles for multimedia software engineering. The multimedia workloads range from memory to computational-bounded kernels. A performance metric to evaluate effective SMT performance gain is introduced, and compared to similar metrics on symmetric multiprocessor (SMP) systems. In addition, we analyze and compare SMT versus SMP systems, and highlight the advantages in the studied applications. The results indicate that sharing the cache in SMT processors can provide better cache locality and thus better performance although sharing the cache can introduce cache conflicts and reduce the actual cache size available for each logical processor. We also propose "mutually beneficial prefetching" - a technique to schedule threads so that they prefetch data for each other in order to reduce cache miss penalty.
Partial contour matching for document pieces with content-based prior

In this paper we present a method for aligning shredded document pieces based on outer contours and content-based prior information. Our approach relies on domain-specific knowledge that document pieces must complement each other when aligned correctly. Building on this intuition we propose a variant of MSAC (M-estimator SAmple Consensus) to estimate an hypothesis that recovers the spatial relationship between pairs of pieces. To do so we first approximate their boundaries by polygons from which we define consensus sets between fragments. Each consensus set provides multiple hypotheses for aligning one piece onto the other. An optimal hypothesis is identified by applying a two-stage procedure in which we discard locally inconsistent hypotheses before verifying the remainder for global consistency.
Self-aware distributed AV sensor and actuator networks for improved media adaptation

Most of the existing research work in the area of media adaptation is concentrated on content adaptation, transcoding and delivery mechanisms without addressing the actual input and output of multimedia data. However, it is the I/O stage of media processing that humans are concerned about. Up until now, most multimedia applications have relied on standalone I/O devices (microphone, headphones, monitor, camera) to capture or render multimedia data. This situation is about to change. Nowadays we are surrounded by a vast number of audio/video (AV) sensors and actuators. They are built into our cellular phones, PDAs, tablets, laptops, and surveillance systems. A natural idea that comes out of this fact is to combine multiple I/O devices into a distributed array of sensors and actuators. The paper shows the feasibility of this idea and shifts media adaptation research away from a single device/stream paradigm towards array multimedia processing. We demonstrate how to transform a network of off-the-shelf devices into a distributed I/O array by providing common time (with tens of microseconds precision) and 3D space coordinates (with a few centimetres precision). We also discuss the implications and potentials of self-calibrating distributed AV-sensor/actuator networks for improved media adaptation.
On the importance of exact synchronization for distributed audio signal processing

We propose a new paradigm for implementations of audio array processing algorithms on a network of distributed general-purpose computers. In contrast to currently existing DSP processor-based solutions, our approach offers new possibilities for advanced array signal processing by enabling the usage of general-purpose computing platforms with their superior computational and storage resources. We demonstrate that synchronization of sensors is essential for acoustic blind source separation (BSS) algorithms, and we propose a synchronization scheme that enables BSS on distributed, wirelessly networked computers and can easily be implemented on existing hardware.
Saliency-guided selective magnification for company logo detection

Fast R-CNN is a well-known approach to object detection which is generally reported to be robust to scale changes. In this paper we examine the influence of scale within the detection pipeline in the case of company logo detection. We demonstrate that Fast R-CNN encounters problems when handling objects which are significantly smaller than the receptive field of the utilized network. In order to overcome these difficulties, we propose a saliency-guided multiscale approach that does not rely on building a full image pyramid. We use the feature representation computed by Fast R-CNN to directly classify large objects while at the same time predicting salient regions which contain small objects with high probability. Only selected regions are magnified and a new feature representation for these enlarged regions is calculated. Feature representations from both scales are used for classification, improving the detection quality of small objects while keeping the computational overhead low. Compared to a naive magnification strategy we are able to retain 79% of the performance gain while only spending 36% of the computation time.
Fisher vector encoding of micro color features for (real world) jigsaw puzzles

In this work we propose a computationally efficient yet effective color encoding scheme for the purpose of jigsaw puzzle solving. Our contribution is twofold: (i) First of all, we show that a Fisher vector encoding gives superior performance over the commonly used compatibility descriptor of stacked color values. (ii) Furthermore, we show experimentally on synthetic and real-world data that our proposed color encoding is more robust in the presence of noise, as compared to three widely used features. Due to the robustness of our proposed descriptor we also anticipate its use to yield performance improvements in other applications, e.g., for the virtual reconstruction of hand-torn documents and archaeological findings.
Multimodal Image Captioning for Marketing Analysis

Automatically captioning images with natural language sentences is an important research topic. State of the art models are able to produce human-like sentences. These models typically describe the depicted scene as a whole and do not target specific objects of interest or emotional relationships between these objects in the image. However, marketing companies require to describe these important attributes of a given scene. In our case, objects of interest are consumer goods, which are usually identifiable by a product logo and are associated with certain brands. From a marketing point of view, it is desirable to also evaluate the emotional context of a trademarked product, i.e., whether it appears in a positive or a negative connotation. We address the problem of finding brands in images and deriving corresponding captions by introducing a modified image captioning network. We also add a third output modality, which simultaneously produces real-valued image ratings. Our network is trained using a classification-aware loss function in order to stimulate the generation of sentences with an emphasis on words identifying the brand of a product. We evaluate our model on a dataset of images depicting interactions between humans and branded products. The introduced network improves mean class accuracy by 24.5 percent. Thanks to adding the third output modality, it also considerably improves the quality of generated captions for images depicting branded products.
On the detection and recognition of television commercials

TV commercials are interesting in many respects: advertisers and psychologists are interested in their influence on human purchasing habits, while parents might be interested in shielding their children from their influence. In this paper, two methods for detecting and extracting commercials in digital videos are described. The first method is based on statistics of measurable features and enables the detection of commercial blocks within TV broadcasts. The second method performs detection and recognition of known commercials with high accuracy. Finally, we show how both approaches can be combined into a self-learning system. Our experimental results underline the practicality of the methods.
Exploring the use of Hyper-Threading technology for multimedia applications with Intel/spl reg/ OpenMP compiler

Processors with Hyper-Threading technology can improve the performance of applications by permitting a single processor to process data as if it were two processors by executing instructions from different threads in parallel rather than serially. However, the potential performance improvement can be only obtained if an application is multithreaded by parallelization techniques. This paper presents the threaded code generation and optimization techniques in the Intel/spl reg/ C++/Fortran compiler. We conduct the performance study of two multimedia applications parallelized with OpenMP pragmas and compiled with the Intel compiler on the Hyper-Threading technology (HT) enabled Intel single-processor and multi-processor systems. Our performance results show that the multithreaded code generated by the Intel compiler achieved up to 1.28x speedups on a HT-enabled single-CPU system and up to 2.23x speedup on a HT-enabled dual-CPU system. By measuring IPC (Instructions Per Cycle), UPC (Uops Per Cycle) and cache misses of both serial and multithreaded execution of each multimedia application, we conclude three key observations: (a) the multithreaded code generated by the Intel compiler yields a good performance gain with the parallelization guided by OpenMP pragmas or directives; (b) exploiting thread-level parallelism (TLP) causes inter-thread interference in caches, and places greater demands on the memory system. However, the Hyper-Threading technology hides the additional latency, so that there is a small impact on the whole program performance; (c) Hyper-Threading technology is effective on exploiting both task- and data-parallelism inherent in multimedia applications.
Improving Media Services on P2P Networks

The Media Accelerating Peer Services system extends P2P infrastructures to improve multimedia services across heterogeneous computing platforms. In this article, we present an architecture and resource management and adaptation framework that transcends existing infrastructures to accommodate and accelerate multimedia peer applications and services. We also propose key technology components that support seamless adaptation of resources to enhance quality of service and the building of better tools and applications that utilize the peer-computing network's underlying power.
Providing common I/O clock for wireless distributed platforms

We propose a novel synchronization scheme for distributed audio-video input and output on heterogeneous general purpose computers (GPC) such as laptops, tablets, PDA, smart telephones, audio recorders, and camcorders. These devices typically possess sensors such as microphones and possibly cameras, and actuators such as loudspeakers and displays. In order to combine them wirelessly into a distributed array signal processing system, it is necessary to provide relative time synchronization to sensors and actuators. In this work we propose a setup and an algorithm to synchronize input and output for a network of distributed multichannel audio sensors and actuators connected to GPC. An IEEE 802.11 wireless network is used to deliver the global clock to distributed GPC, while the interrupt mechanism is employed to distribute the clock between I/O devices. Experimental results demonstrate a precision in A/D D/A synchronization precision better than 50 /spl mu/s (a couple of samples at 48 kHz).
Monocular 3D human pose estimation by classification

We present a novel approach to 2D and 3D human pose estimation in monocular images by building on and improving recent advances in this field. We take the full body pose as a combination of a 3D pose and a viewpoint and in this way define classes that are then learned by a classifier. Compared to part based approaches, our approach does not suffer from self-occluded body parts since such occlusions are characteristic for certain classes and thus are captured during class definition. Moreover, we significantly relax the requirements posed on training data by the fact that we do neither require labeled viewpoints nor background subtracted images, and the carried out action does not need to be cyclic. By combining an efficient classifier with efficient image features, we present a generic and fast way to estimate human poses in images and achieve comparable results to state-of-the art approaches which we demonstrate on a public benchmark.
Fast Gabor Transformation For Processing High Quality Audio

The Gabor transformation with a Gaussian window has several advantages over classical short time transformations such as the windowed FFT and wavelets. It allows for perfect localization in time and frequency according to the absolute bound expressed by the Heisenberg uncertainty principle. Furthermore the time-frequency resolution can be chosen as desired. This is bought dearly by the necessity of oversampling and very large windows resulting in high computational and storage costs. To overcome this disadvantages the FFT can be used as an underlying technique to speed up the computation for some rare dedicated time-frequency resolutions. In this paper the use of the FFT is extended to allow choosing the time-frequency resolution arbitrarily, by introducing only a small computational overhead. With the same approach we show, how to use the FFT to compute the Gabor transformation on non-separable lattices. The speed-up factors over an optimized DFT approach range from 2.5 to 100
Automatic object annotation from weakly labeled data with latent structured SVM

In this paper we present an approach to automatic object annotation. We are given a set of positive images which all contain a certain object and our goal is to automatically determine the position of said object in each image. Our approach first applies a heuristic to identify initial bounding boxes based on color and gradient features. This heuristic is based on image and feature statistics. Then, the initial boxes are refined by a latent structured SVM training algorithm which is based on the CCCP training algorithm. We show that our approach outperforms previous work on multiple datasets.
Calibrating Visual Sensors and Actuators in Distributed Platforms

Many novel multimedia, home entertainment, visual surveillance and health applications use multiple audio-visual sensors and actuators. In this paper we present a novel approach for position and pose calibration of visual sensors and actuators, i.e. cameras and displays, in a distributed network of general purpose computing devices. It complements our work on position calibration of audio sensors and actuators in a distributed computing platform [14]. The approach is suitable for a wide range of possible - even mobile - setups since (a) synchronization is not required, (b) it works automatically, (c) only weak restrictions are imposed on the positions of the cameras and displays, and (d) no upper limit on the number of cameras and displays under calibration is imposed. Corresponding points across different camera images are established automatically and found with subpixel accuracy. Cameras do not have to share one common view. Only a reasonable overlap between camera subgroups is necessary. The method has been sucessfully tested in numerous multi-camera environments with a varying number of cameras and displays and has proven itself to work extremely accurate.
Methods and Applications for Distance Based ANN Training

Feature learning has the aim to take away the hassle of hand-designing features for machine learning tasks. Since the feature design process is tedious and requires a lot of experience, an automated solution is of great interest. However, an important problem in this field is that usually no objective values are available to fit a feature learning function to. Artificial Neural Networks are a sufficiently flexible tool for function approximation to be able to avoid this problem. We show how the error function of an ANN can be modified such that it works solely with objective distances instead of objective values. We derive the adjusted rules for back propagation through networks with arbitrary depths and include practical considerations that must be taken into account to apply difference based learning successfully. On all three benchmark datasets we use, linear SVMs trained on automatically learned ANN features outperform RBF kernel SVMs trained on the raw data. This can be achieved in a feature space with up to only a tenth of dimensions of the number of original data dimensions. We conclude our work with two experiments on distance based ANN training in two further fields: data visualization and outlier detection.
A closer look: Small object detection in faster R-CNN

Faster R-CNN is a well-known approach for object detection which combines the generation of region proposals and their classification into a single pipeline. In this paper we apply Faster R-CNN to the task of company logo detection. Motivated by the weak performance of Faster R-CNN on small object instances, we perform a detailed examination of both the proposal and the classification stage, examining their behavior for a wide range of object sizes. Additionally, we look at the influence of feature map resolution on the performance of those stages. We introduce an improved scheme for generating anchor proposals and propose a modification to Faster R-CNN which leverages higher-resolution feature maps for small objects. We evaluate our approach on the Flicker data set improving the detection performance on small object instances.
Evaluating and improving performance of multimedia applications on simultaneous multi-threading

This paper presents the study and results of running several core multimedia applications on a simultaneous multithreading (SMT) architecture, including some detailed analysis ranging from memory-bounded kernels to computational-bounded functions. A performance metric to evaluate effective SMT performance gain is introduced, and compared to similar metrics on symmetric multiprocessor (SMP) systems. In addition, we analyze and compare SMT versus SMP systems, and highlight the advantages in the studied applications. The results indicate that sharing the cache in SMT processors can provide better cache locality and thus better performance although sharing the cache can introduce cache conflicts and reduce the actual cache size available for each logical processor. We also propose "mutual prefetching" -a technique to schedule threads so that they prefetch data for each other in order to reduce cache miss penalty.
Norm-Induced Entropies for Decision Forests

The entropy measurement function is a central element of decision forest induction. The Shannon entropy and other generalized entropies such as the Renyi and Tsallis entropy are designed to fulfill the Khinchin-Shannon axioms. Whereas these axioms are appropriate for physical systems, they do not necessarily model well the artificial system of decision forest induction. In this paper, we show that when omitting two of the four axioms, every norm induces an entropy function. The remaining two axioms are sufficient to describe the requirements for an entropy function in the decision forest context. Furthermore, we introduce and analyze the p-norm-induced entropy, show relations to existing entropies and the relation to various heuristics that are commonly used for decision forest training. In experiments with classification, regression and the recently introduced Hough forests, we show how the discrete and differential form of the new entropy can be used for forest induction and how the functions can simply be fine tuned. The experiments indicate that the impact of the entropy function is limited, however can be a simple and useful post-processing step for optimizing decision forests for high performance applications.