nils-herrmann/clean_scirepeval huggingface dataset:
Automating multi-camera self-calibration

We demonstrate a convenient and accurate method for fully automatic camera calibration. The method needs at least two cameras and one projector to function, but the cameras need not to be synchronized. By projecting a predefined black and white sequence into the cameras' field of view a large number of individual points are tagged by a binary bit sequence over time. This solves the correspondence problem among the adjacent views and furthermore allows for Forward Error Correction (FEC) yielding a dense error free and subpixel accurate point cloud which is used for internal and external camera calibration. Experimental results and comparison with varying permutations of the projection sequence are given at the end of this paper. Finally, the method gives instant feedback to the user as the resulting calibration point cloud is in fact a 3D scan of the arbitrary calibration scene which can be easily visualized.
Real-time active multiview 3D reconstruction

We report on an active multi view system for realtime 3D reconstruction based on phase measuring triangulation. Our system overcomes one of the greatest drawbacks in active 3D reconstruction, namely occlusions due do shadowing of either the camera or the projector light source. Our system is comprised of two high speed cameras in conjunction with two projectors and is currently capable of capturing and rendering up to 5.2 million 3D vertices at 10 fps. Four additional color cameras provide for texturing the underlying 3D geometry, thus making the system suitable for real-time view synthesis on conventional, stereoscopic or novel autostereoscopic multiview displays.
Multiple cue indexing and summarization of surveillance video

In this paper we propose a system for the summarization of safety and security surveillance video. By combining the information provided by multiple analysis cues, we improve the quality of the information extracted out of the analyzed video sequences with respect to the state-of-the-art approaches, therefore, being able to generate summaries that better align with the content of the original video. The proposed system has been tested using an extensive set of surveillance sequences, showing compression ratios ranging from 11 to 114, depending on the video content and the configuration of the system.
Detecting People Carrying Objects Utilizing Lagrangian Dynamics

The availability of dense motion information in computer vision domain allows for the effective application of Lagrangian techniques that have their origin in fluid flow analysis and dynamical systems theory. A well established technique that has been proven to be useful in image-based crowd analysis are Finite Time Lyapunov Exponents (FTLE). Based on this, we present a method to detect people carrying object and describe a methodology how to apply established flow field methods onto the problem of describing individuals. Further, we reinterpret Lagrangian features in relation to the underlying motion process and show their applicability towards the appearance modeling of pedestrians. This definition allows to increase performance of state-of-the-art methods and is shown to be robust under varying parameter settings and different optical flow extraction approaches.
Dynamic Spectral Envelope Modeling for Timbre Analysis of Musical Instrument Sounds

We present a computational model of musical instrument sounds that focuses on capturing the dynamic behavior of the spectral envelope. A set of spectro-temporal envelopes belonging to different notes of each instrument are extracted by means of sinusoidal modeling and subsequent frequency interpolation, before being subjected to principal component analysis. The prototypical evolution of the envelopes in the obtained reduced-dimensional space is modeled as a nonstationary Gaussian Process. This results in a compact representation in the form of a set of prototype curves in feature space, or equivalently of prototype spectro-temporal envelopes in the time-frequency domain. Finally, the obtained models are successfully evaluated in the context of two music content analysis tasks: classification of instrument samples and detection of instruments in monaural polyphonic mixtures.
Feature-based global motion estimation using the Helmholtz principle

Global motion estimation is an important task for various video processing techniques. The estimation itself has to be robust in presence of arbitrarily moving foreground objects. For that task, two different kinds of estimation methods exist. On the one hand, pixel-based approaches deliver more precise results and work more robust on video sequences with foreground objects. On the other hand, when working on encoded video streams, block-based methods can be used for a much faster but often less precise estimation. We propose a two step estimation method based on the determination and tracking of feature points of video frames and robust motion model estimation using the Helmholtz principle. Therefore, good trackable features are detected and tracked in video sequences. Subsequently, a perspective motion model is derived from the resulting correspondencies by removing feature pairs not belonging to global motion.
Lossy image coding in the pixel domain using a sparse steering kernel synthesis approach

Kernel regression has been proven successful for image de-noising, deblocking and reconstruction. These techniques lay the foundation for new image coding opportunities. In this paper, we introduce a novel compression scheme: Sparse Steering Kernel Synthesis Coding (SSKSC). This pre- and postprocessor for JPEG performs non-uniform sampling based on the smoothness of an image, and reconstructs the missing pixels using adaptive kernel regression. At the same time, the kernel regression reduces the blocking artifacts from the JPEG coding. Crucial to this technique is that non-uniform sampling is performed while maintaining only a small overhead for signalization. Compared to JPEG, SSKSC achieves a compression gain for low bits-per-pixel regions of 50% or more for PSNR and SSIM. A PSNR gain is typically in the 0.0-0.5 bpp range, and an SSIM gain can mostly be achieved in the 0.0-1.0 bpp range.
Counting People in Crowded Environments by Fusion of Shape and Motion Information

Knowing the number of people in a crowded scene is of big interest in the surveillance scene. In the past, this problem has been tackled mostly in an indirect, statistical way. This paper presents a direct, counting by detection, method based on fusing spatial information received from an adapted Histogram of Oriented Gradientsalgorithm (HOG) with temporal information by exploiting distinctive motion characteristics of different human body parts. For that purpose, this paper defines a measure for uniformity of motion. Furthermore, the system performance is enhanced by validating the resulting human hypotheses by tracking and applying a coherent motion detection. The approach is illustrated with an experimental evaluation.
Real-time person counting by propagating networks flows

In this paper we present a system that tracks multiple persons by detection in real-time. We introduce a measure for similarity of detections which segments significant information from background clutter by using statistical information obtained during the learning phase of the detector. In order to track multiple persons we map the detections into flow networks utilizing this measure. A continuous real-time processing of video streams is accomplished by analyzing only small chunks of detections consecutively using different networks. By propagating the result of one network into the subsequent one a temporal consistent association is achieved. The system was evaluated using a standard video sequence containing a crowded scene and an own dataset with very long sequences. The results demonstrate that the system performs comparable to other systems while meeting real-time requirements.
Adpative dense vector field interpolation for temporal filtering

Inloop filters are a well known tool to improve the compression performance of hybrid video codecs. These filters generally work in the spatial domain. If temporal filters are used instead, their performance strongly depends on the available motion data. In this paper a new method to retrieve accurate motion information per pixel is introduced and evaluated within the HEVC test model HM 8.0. The interpolated motion vector field is significnatly closer to the real object motion than the block-based motion data. An average bit rate reduction of 0.4% is observed with a maximum reduction of 2.0%.
An Image-Based Rendering (IBR) Approach for Realistic Stereo View Synthesis of TV Broadcast Based on Structure from Motion

In the past years, the 3D display technology has become a booming branch of research with fast technical progress. Hence, the 3D conversion of already existing 2D video material increases more and more in popularity. In this paper, a new approach for realistic stereo view synthesis (RSVS) of existing 2D video material is presented. The intention of our work is not a real-time conversion of existing video material with a deduction in stereo perception, but rather a more realistic off-line conversion with high accuracy. Our approach is based on structure from motion techniques and uses image-based rendering to reconstruct the desired stereo views for each video frame. The algorithm is tested on several TV broadcast videos, as well as on sequences captured with a single handheld camera. Finally, some simulation results will show the remarkable performance of this approach.
Lossless image compression based on Kernel Least Mean Squares

This paper introduces a novel approach for coding luminance images using kernel-based adaptive filtering and context-adaptive arithmetic coding. This approach tackles the problem that is present in current image and video coders; these coders depend on assumptions of the image and are constrained by the linearity of their predictors. The efficacy of the predictors determines the compression gain. The goal is to create a generic image coder that learns and adapts to the characteristics of the signals and handles nonlinearity in the prediction. Results show that pixel luminance prediction using the Kernel Least Mean Squares (KLMS) yields a significant gain compared to the standard Least Mean Squares algorithm. By coding the residual using a Context-Adaptive Arithmetic Coder (CAAC), the codec is able to outperform the current industry standards of lossless image coding. An average bitrate reduction of more than 2.5% is found for the used test set.
Cross based robust local optical flow

In many computer vision applications local optical flow methods are still a widely used. Such methods, like the Pyramidal Lucas Kanade and the Robust Local Optical Flow, have to address the trade-off between run time and accuracy. In this work we propose an extension to these methods that improves the accuracy especially at object boundaries. This extension makes use of the cross based variable support region generation proposed in  accounting for local intensity discontinuities. In the evaluation using Middlebury data set we prove the ability of the proposed extension to increase the accuracy by a slight increase of run time.
Recursive decoder distortion estimation based on AR(1) source modeling for video

We introduce a recursive block based decoder distortion estimation technique for video and present results showing the accordance of the estimation results with simulation results. Each block in a frame, with all its corresponding blocks along the video sequence, are modeled as an AR(1) source where the correlation coefficient of the source depends on the loop filtering effects and the additional noise term on the motion compensated block difference as well as on the quantization distortion of the block. The distortion term for each block of each frame in the video sequence is calculated recursively depending on the packet loss rate of the channel.
Global motion estimation using variable block sizes and its application to object segmentation

Global motion is estimated either in the pixel domain or in block based domain. Until now, all the approaches regarding the latter are based on fixed sized blocks while the recent compression methods tend to use variable block sizes during motion estimation. In this paper we present a new procedure for global motion estimation based on a variable block size motion vector field. A block matching algorithm which is able to adapt the block size according to the motion complexity within the frame is used. The resulting motion vectors are employed for global motion estimation. Furthermore, binary foreground-background masks are created based on the frame-by-frame motion compensated differences by exploiting spatial conditions through anisotropic diffusion filtering. For global motion estimation the performance evaluation in terms of background PSNR shows an enhancement of more than 2.5 dB in the well-known “Stefan” sequence, compared to the conventional case of fixed block size, at a reasonable implementation complexity.
Steered mixture-of-experts for light field coding, depth estimation, and processing

The proposed framework, called Steered Mixture-of-Experts (SMoE), enables a multitude of processing tasks on light fields using a single unified Bayesian model. The underlying assumption is that light field rays are instantiations of a non-linear or non-stationary random process that can be modeled by piecewise stationary processes in the spatial domain. As such, it is modeled as a space-continuous Gaussian Mixture Model. Consequently, the model takes into account different regions of the scene, their edges, and their development along the spatial and disparity dimensions. Applications presented include light field coding, depth estimation, edge detection, segmentation, and view interpolation. The representation is compact, which allows for very efficient compression yielding state-of-the-art coding results for low bit-rates. Furthermore, due to the statistical representation, a vast amount of information can be queried from the model even without having to analyze the pixel values. This allows for “blind” light field processing and classification.
Real-time detection of sport in MPEG-2 sequences using high-level AV-descriptors and SVM

We present a new approach for classifying MPEG-2 video sequences as dasiasportpsila or dasianon-sportpsila by analyzing new high-level audiovisual features of consecutive frames in real-time. This is part of the well-known video-genre-classification problem, where popular TV-broadcast genres like cartoon, commercial, music video, news and sports are studied. Such applications have also been discussed in the context of MPEG-7. In our method the extracted features are logically combined by a support vector machine to produce a reliable detection. The results demonstrate a high identification rate of 98.5% based on a large balanced database of 100 representative video sequences gathered from free digital TV-broadcasting and world wide web.
Color prediction in image coding using Steered Mixture-of-Experts

We propose a novel approach for modeling and coding color in images and video. Luminance is linearly correlated with chrominance locally, as such we can predict color given the luma value. Using the Steered Mixture-of-Experts (SMoE) approach, the image is viewed as a stochastic process over 5 random variables including the 2-D pixel locations, 1 luminance and 2 chrominance values. We model this process as a continuous joint density function by fitting a K-modal 5-D Gaussian Mixture Model (GMM). As such, the chroma values are predicted as the expectation of the conditional density. To validate, the technique was integrated within JPEG showing PSNR gains in the lower bitrate regions. A deeper analysis of the tolerance of the activation function is given through recycling color models in video sequences, yielding a high quality reconstruction over a considerable range of frames.
Sequential sensor fusion combining probability hypothesis density and kernelized correlation filters for multi-object tracking in video data

This work applies the Gaussian Mixture Probability Hypothesis Density (GMPHD) Filter to multi-object tracking in video data. In order to take advantage of additional visual information, Kernelized Correlation Filters (KCF) are evaluated as a possible extension of the GMPHD tracking-by-detection scheme to enhance its performance. The baseline GMPHD filter and its extension are evaluated on the UA-DETRAC benchmark, showing that combining both methods leads to a higher recall and a better quality of object tracks to the cost of increased computational complexity and increased sensitivity to false-positives.
Multi-view video streaming over P2P networks with low start-up delay

We propose to stream multi-view video over a multi-tree peer- to-peer (P2P) network using the NUEPMuT protocol. Each view of the multi-view video is streamed over an independent P2P streaming tree and each peer only contributes upload capacity in a single tree, in order to limit the adverse effects of ungraceful peer departures. Additionally, we introduce a quick join procedure to reduce the start-up delay for the first data packet after a join request. Continuity index and decoded video quality performance for simulcast and MVC encoding in a large topology under different settings are reported, in addition to the improvements achieved by the quick join procedure.
Efficiency of shape-adaptive 2-D transforms for coding of arbitrarily shaped image segments

We introduce a formula to compute an optimum 2-D shape-adaptive Karhunen-Loeve transform (KLT) suitable for coding pels in arbitrarily-shaped image segments. The efficiency of the KLT on a 2-D AR(1) process is used to benchmark two other shape-adaptive transforms described in literature. It is shown that the optimum KLT significantly outperforms the well known shape-adaptive DCT method introduced by Gilge et al. (1989) for coding Segments of arbitrary shape in intraframe coding mode. A statistical transform gain close to the Gilge-method can be achieved with a shape-adaptive DCT algorithm introduced by Sikora and Makai (see , FRG, Nov. 1993) which is implemented with much lower complexity.
Image guided phase unwrapping for real-time 3D-scanning

3D-reconstructions produced by active 3D-scanning systems based on structured light can achieve high accuracy reconstructions of the scene surfaces. Structured light algorithms based on phase measuring triangulation (PMT) utilize phase-shifted sinusoidal patterns projected into the scene for a precise determination of correspondencies. The number of patterns used for that purpose may vary depending on the design of the algorithm. No matter how many patterns are required, all of these algorithms suffer from the acquisition time needed to record all patterns sequentially. In case of a dynamic scene the sequential acquisition of images lead to the capture of dynamic objects in different poses which in turn result in erroneous reconstructions depending on the object's velocity. Our goal is to achieve a more robust result during dynamic scene capture as well as better scene reconstruction rate. Two novel approaches are presented to reduce the amount of required patterns for a high-accuracy 3D-reconstruction. This is achieved by incorporating passive matching techniques in the phase-unwrapping stage of the algorithm, allowing to drop one half of the sinusoidal patterns.
The MPEG-7 visual standard for content description-an overview

The MPEG-7 visual standard under development specifies content-based descriptors that allow users or agents (or search engines) to measure similarity in images or video based on visual criteria, and can be used to efficiently identify, filter, or browse images or video based on visual content. More specifically, MPEG-7 specifies color, texture, object shape, global motion, or object motion features for this purpose. This paper outlines the aim, methodologies, and broad details of the MPEG-7 standard development for visual content description.
Trends and Perspectives in Image and Video Coding

The objective of the paper is to provide an overview on recent trends and future perspectives in image and video coding. Here, I review the rapid development in the field during the past 40 years and outline current state-of-the art strategies for coding images and videos. These and other coding algorithms are discussed in the context of international JPEG, JPEG 2000, MPEG-1/2/4, and H.261/3/4 standards. Novel techniques targeted at achieving higher compression gains, error robustness, and network/device adaptability are described and discussed.
Image Adaptation and Dynamic Browsing Based on Two-Layer Saliency Combination

In recent years, image adaptation has attracted more and more attention during the evolution of the integration of broadcasting, Internet and telecommunications. The diversity of display devices requires images to be resized for optimal display on different terminals. In this paper, an image adaptation scheme and a dynamic browsing strategy are proposed, which are based on the visual attention model (VAM) with two-layer saliency optimization. First, a VAM is constructed by optimizing the image saliency according to the global and local layers by simulated annealing and the saliency of focus of attention (FOA) can be ranked by the obtained saliency map. Then an image adaptation scheme is designed based on the obtained saliency map. In the proposed adaptation scheme, each predominant FOA is modeled as a rectangular region, and the image is adjusted corresponding to the display terminal and the rank of FOA. Based on the principle of global features precedence, a dynamic browsing strategy is developed for browsing large images on small display devices. Experiments on saliency map show that the cross-layer saliency detection algorithm has advantages on detecting salient regions accurately and reflecting the shift of FOA implicitly, which are useful for image adaptation. In addition, subjective evaluation experiments demonstrate the proposed image adaptation scheme and dynamic browsing strategy are feasible and have promising applications in practice.
Audio classification based on MPEG-7 spectral basis representations

In this paper, we present an MPEG-7-based audio classification and retrieval technique targeted for analysis of film material. The technique consists of low-level descriptors and high-level description schemes. For low-level descriptors, low-dimensional features such as audio spectrum projection based on audio spectrum basis descriptors is produced in order to find a balanced tradeoff between reducing dimensionality and retaining maximum information content. High-level description schemes are used to describe the modeling of reduced-dimension features, the procedure of audio classification, and retrieval. A classifier based on continuous hidden Markov models is applied. The sound model state path, which is selected according to the maximum-likelihood model, is stored in an MPEG-7 sound database and used as an index for query applications. Various experiments are presented where the speaker- and sound-recognition rates are compared for different feature extraction methods. Using independent component analysis, we achieved better results than normalized audio spectrum envelope and principal component analysis in a speaker recognition system. In audio classification experiments, audio sounds are classified into selected sound classes in real time with an accuracy of 96%.