nils-herrmann/clean_scirepeval huggingface dataset:
Image sequence analysis for emerging interactive multimedia services-the European COST 211 framework

Flexibility and efficiency of coding, content extraction, and content-based search are key research topics in the field of interactive multimedia. Ongoing ISO MPEG-4 and MPEG-7 activities are targeting standardization to facilitate such services. European COST Telecommunications activities provide a framework for research collaboration. At present a significant effort of the COST 211/sup ter/ group activities is dedicated toward image and video sequence analysis and segmentation-an important technological aspect for the success of emerging object-based MPEG-4 and MPEG-7 multimedia applications. The current work of COST 211 is centered around the test model, called the analysis model (AM). The essential feature of the AM is its ability to fuse information from different sources to achieve a high-quality object segmentation. The current information sources are the intermediate results from frame-based (still) color segmentation, motion vector based segmentation, and change-detection-based segmentation. Motion vectors, which form the basis for the motion vector based intermediate segmentation, are estimated from consecutive frames. A recursive shortest spanning tree (RSST) algorithm is used to obtain intermediate color and motion vector based segmentation results. A rule-based region processor fuses the intermediate results; a postprocessor further refines the final segmentation output. The results of the current AM are satisfactory.
Motion modeling for motion vector coding in HEVC

During the standardization of HEVC, new motion information coding and prediction schemes such as temporal motion vector prediction have been investigated to reduce the spatial redundancy of motion vector fields used for motion compensated inter prediction. In this paper a general motion model based vector coding scheme is introduced. This scheme includes estimation, coding and dynamic recombination of parametric motion models to generate vector predictors and merge candidates for all common HEVC inter coding settings. Bit rate reductions of up to 4.9% indicate that higher order motion models can increase the efficiency of motion information coding in modern hybrid video coding standards.
Temporal trajectory filtering for bi-directional predicted frames

In this work the application of a temporal in-loop filtering approach for B-frames in video compression based on the Temporal Trajectory Filter (TTF) is investigated. The TTF constructs temporal pixel trajectories for individual image points in the P-frames of a video sequence, which can be utilized to improve the quality of the reconstructed frames used for prediction. It is shown, how this concept can be adapted to B-frames despite the fact that these already use temporal motion information to a great extent through the flexible choice of reference frames and prediction modes. The proposed filter has been integrated into the H.264/AVC encoder using the extended profile with hierarchical B-frames and was tested on a wide range of sequences. The filter produces bit rate reductions of up to −4% with an average of −1.6% over all tested sequences while also improving the subjective quality of the decoded video.
Extending H.264/AVC with a background sprite prediction mode

The latest standardized hybrid video codec, H.264/AVC, significantly outperforms earlier video coding standards. Despite combining improved and new algorithms within this codec, it is still possible to find methods which lead to a higher coding efficiency. We tackle the prediction problem adding a new prediction mode to the codec. It has been shown that the generation of a background sprite image containing all the background information of a certain sequence is very useful e.g. for object-based video coding. We use a pre-generated background sprite image for creating a new prediction mode in the encoder loop. For the current frame to be compensated, blocks reconstructed from the background sprite are used beside the remaining modes to calculate the residual. The rate-distortion optimization decides which mode is taken. Experimental results show the improvement using the new sprite prediction (SP) mode with the considered test sequences.
Extracting High Level Semantics by Means of Speech, Audio, and Image Primitives in Surveillance Applications

Traditional surveillance systems are usually based on visual information only. With the emerging multimedia analysis techniques, interests are changing towards systems that incorporate multiple sensors and different modalities, which leads to new ways of analyzing this multimedia data and more sophisticated applications. This paper shortly reviews the ideas of traditional surveillance systems and explains actual research interests in this domain. Then, it focuses on the typical structure, goals, and applications of multimedia surveillance systems. These issues are supported by short descriptions of selected analysis steps of such a system currently under development. Some experimental results are given to illustrate the extracted semantics and to assess the performance of the individual steps.
Automatic topic detection strategy for information retrieval in spoken document

This paper suggests an alternative solution for the task of spoken document retrieval (SDR). The proposed system runs retrieval on multi-level transcriptions (word and phone) produced by word and phone recognizers respectively, and their outputs are combined. We propose to use latent Dirichlet allocation (LDA) model for capturing the semantic information on word transcription. The LDA model is employed for estimating topic distribution in queries and word transcribed spoken documents, and the matching is performed at the topic level. Acoustic matching between query words and phonetically transcribed spoken documents is performed using phone-based matching algorithm. The results of acoustic and topic level matching methods are compared and shown to be complementary.
Simulation and identification of the vascular system dynamics for adaptive TAH-controller design

A nonlinear mathematical model of the short-term behavior of the systemic vascular dynamics has been developed for the design of adaptive total artificial heart (TAH) controllers and their optimization. A three-step identification scheme was applied in connection with measurements observed on TAH calves to estimate the unknown model parameters. It was proved that all model parameters are identifiable. A good representation of the real system dynamics was obtained, and the robust identifiability of the total compliance of the system was established, which implies an estimate with a low standard deviation.<<ETX>>
Multiple Background Sprite Generation Using Camera Motion Characterization for Object-Based Video Coding

Recent work has shown that object-based video coding can provide higher coding gain than common H.264/AVC for single-view and the MVC standard based on H.264 for multi-view (MVC). The use of background sprites outperformes the AVC/MVC especially in sequences containing rotating camera motion and moving foreground objects. The coding performance strongly relies on the preprocessing steps, e.g. sprite generation and object segmentation. In this paper, we present an enhanced background sprite generation algorithm for object-based single- and multi-view video coding (OBVC/OBMVC). It is a new feature for our OBVC/OBMVC recently proposed. We produce multiple background sprites based on camera motion characterization and physical camera parameter estimation. Experimental results show how these multiple sprites increase the coding performance for single- and multi-view sequences.
A Parametric Merge Candidate for High Efficiency Video Coding

Block based motion compensated prediction still is the main technique used for temporal redundancy reduction in modern hybrid video codecs. However, the resulting motion vector fields are highly redundant as well. So, motion vector prediction and difference coding are used to compress such vector fields. A drawback of common motion vector prediction techniques is their inability to predict complex motion such as rotation and zoom in an efficient way. We present a novel Merge candidate for improving already existing vector prediction techniques based on higher order motion models to overcome this issue. To transmit the needed models, an efficient compression scheme is utilized. The improvement results in bit rate savings of 1.7% in average and up to 4% respectively.
Motion-aware video quality assessment

This work focuses on considering motion towards improving video quality assessment algorithms. The improvement refers to improving computational video quality assessment algorithms in order to be in closer agreement with the subjective evaluation of video quality. We propose a motion saliency model that exploits motion features on spatial level and also an approach for consideration of global motion in the temporal dimension, leading to further improvements in the accuracy of video quality assessment. We perform evaluation by integrating our approaches in existing objective quality models and also by comparing them to existing related state-of-the-art video quality assessment methods.
Adaptive parallax for 3D television

In this paper we present a concept for maintaining the three dimensional aspect ratio of a given 3D image or video upon resizing the viewing window or changing the display size altogether. A stereoscopic 3D scene that is resized horizontally and vertically will appear skewed along the z-axis. That is, the perceptual depth will grow aproportionally because the disparity grows linear with the resizing factor. The geometric model explained in this paper solves this problem by deriving a non-linear resize factor that non-linearly alters the depth dependent disparity of a stereo or multi-baseline camera rig, and thereby preservers proper aspect ratios in the third dimension on arbitrary displays and viewport sizes.
Hybrid recursive energy-based method for robust optical flow on large motion fields

We present a new reliable hybrid recursive method for optical flow estimation. The method efficiently combines the advantage of discrete motion estimation and optical flow estimation in a recursive block-to-pixel estimation scheme. Integrated local and global approaches using the robust statistic of anisotropic diffusion remove outliers from the estimated motion field. We separately describe the process with two frameworks i.e. an incremental updating framework and a robust energy minimization framework. With robust error norms of Perona and Marik anisotropic diffusion, the formulation usually leads to non-convex optimization problems. Thus, the solution has many local minima, and convergence to the global minima is not guaranteed. Our hybrid recursive energy-based method employs a hierarchical block-to-pixel estimation concept to prevent this problem. The experimental results prove the excellent performance on several large motion fields.
Video indexing and summarization as a tool for privacy protection

The ever increasing number of surveillance camera networks being deployed all over the world has resulted in a high interest in the development of algorithms to automatically analyze the video footage, but has also opened new questions as how to efficiently manage the vast amount of information generated and, more important, how to protect the privacy of the individuals being recorded in their daily life. In this paper, we present a survey on video summarization techniques developed in order to efficiently access to the points of interest in the video footage. Thereby, we emphasize on the links that these techniques show with the task of privacy protection and draw lines of future research directions to incorporate indexing and summarization as tools for privacy protection by design.
A Modular Scheme for 2D/3D Conversion of TV Broadcast

The 3D reconstruction from 2D broadcast video is a challenging problem with many potential applications, such as 3DTV, free-viewpoint video or augmented reality. In this paper, a modular system capable of efficiently reconstructing 3D scenes from broadcast video is proposed. The system consists of four constitutive modules: tracking and segmentation, self-calibration, sparse reconstruction and, finally, dense reconstruction. This paper also introduces some novel approaches for moving object segmentation and sparse and dense reconstruction problems. According to the simulations for both synthetic and real data, the system achieves a promising performance for typical TV content, indicating that it is a significant step towards the 3D reconstruction of scenes from broadcast video.
Creating Experts From the Crowd: Techniques for Finding Workers for Difficult Tasks

Crowdsourcing is currently used for a range of applications, either by exploiting unsolicited user-generated content, such as spontaneously annotated images, or by utilizing explicit crowdsourcing platforms such as Amazon Mechanical Turk to mass-outsource artificial-intelligence-type jobs. However, crowdsourcing is most often seen as the best option for tasks that do not require more of people than their uneducated intuition as a human being. This article describes our methods for identifying workers for crowdsourced tasks that are difficult for both machines and humans. It discusses the challenges we encountered in qualifying annotators and the steps we took to select the individuals most likely to do well at these tasks.
Boosting Multi-hypothesis Tracking by Means of Instance-Specific Models

In this paper we present a visual person tracking-by-detection system based on on-line-learned instance-specific information along with the kinematic relation of measurements provided by a generic person-category detector. The proposed system is able to initialize tracks on individual persons and start learning their appearance even in crowded situations and does not require that a person enters the scene separately. For that purpose we integrate the process of learning instance-specific models into a standard MHT-framework. The capability of the system to eliminate detections-to-object association ambiguities occurring from missed detections or false ones is demonstrated by experiments for counting and tracking applications using very long video sequences on challenging outdoor scenarios.
Robust Anisotropic Disparity Estimation with Perceptual Maximum Variation Modeling

We present a robust anisotropic dense disparity estimation algorithm which employs perceptual maximum variation modeling. Edge-preserving dense disparity vectors are estimated using a coarse-to-fine diffusive method on iteratively filtered images, i.e. the scale-space. While an energy-minimization framework adjusts local disparity, the edges are efficiently preserved by anisotropic disparity-field diffusion. However, the localization at weak image edges which have small brightness variations is fundamentally difficult. In this paper, perceptual maximum variation modeling prevents the delocalization flow over edges, e.g. over-diffusion and back-diffusion, computed by evaluating small variations. We perform disparity-field diffusion on a perceptually optimized color space, which combines the small differences in both brightness and chromaticity. Additionally a consistency constraint is employed in the modeling to avoid the influence of global color distributions and to enhance important edges as the human vision system does. The experimental results show the excellent localization performance preserving the disparity discontinuity of each object.
A novel Kernel PCA/KLT approach for transform coding of waveforms

A novel Kernel PCA/Kernel KLT transform (S-KPCA) is introduced which incorporates higher order statistics into the design of the transform matrix using a Reproducing Kernel Hilbert Space (RKHS) approach. The goal is to arrive at an orthonormal transform matrix E with column eigenvectors that allow reconstruction of an input vector with few coefficients and superior signal fidelity. In contrast to the well known Kernel PCA the number of the generated transform coefficients is not dependent on the size of the training set and the “pre-image problem” is avoided completely. Results indicate that the derived transform is more compact than the standard PCA/KLT in terms of fidelity measures in RKHS.
Incorporating prior knowledge on the digital media creation process into audio classifiers

In the process of music content creation, a wide range of typical audio effects such as reverberation, equalization or dynamic compression are very commonly used. Despite the fact that such effects have a clear impact on the audio features, they are rarely taken into account when building an automatic audio classifier. In this paper, it is shown that the incorporation of prior knowledge of the digital media creation chain can clearly improve the robustness of the audio classifiers, which is demonstrated on a task of musical instrument recognition. The proposed system is based on a robust feature selection strategy, on a novel use of the virtual support vector machines technique and a specific equalization used to normalize the signals to be classified. The robustness of the proposed system is experimentally evidenced using a rather large and varied sound database.
Lossy parametric motion model compression for global motion temporal filtering

It has been shown that techniques using higher-order motion parameters outperform common translational motion compensated prediction for hybrid video coders. A critical issue is the transmission of accurate higher-order motion parameters with as little additional bits as possible to maximize the compression gain of the whole system. For that, we propose a compression scheme for perspective motion models using transformation before quantization and temporal redundancy reduction and integrate this scheme into a video coding environment using adaptive global motion temporal filtering. Experimental results show that using the proposed compression scheme for the perspective motion models, the BD-rate can be improved up to 8.8% in average in the higher bit rate range and up to 7.7% in average in the lower bit rate range compared to the latest version of the HEVC test model HM 4.0.
Prioritized Sequential 3D Reconstruction in Video Sequences with Multiple Motions

In this study, an algorithm is proposed to solve the multi-frame structure from motion (MFSfM) problem for monocular video sequences in dynamic scenes. The algorithm uses the epipolar criterion to segment the features belonging to independently moving objects. Once the features are segmented, corresponding objects are reconstructed individually by using a sequential algorithm, which is also capable of prioritizing the frame pairs with respect to their reliability and information content, thus achieving a fast and accurate reconstruction through efficient processing of the available data. A tracker is utilized to increase the baseline distance between views and to improve the F-matrix estimation, which is beneficial to both the segmentation and the 3D structure estimation processes. The experimental results demonstrate that our approach has the potential to effectively deal with the multi-body MFSfM problem in a generic video sequence.
Theoretical Considerations Concerning Pixelwise Temporal Filtering

Temporal in loop filters present one possible way to reduce noise introduced in compressed video sequences at low bit rates. Some of these filtering approaches make use of the quantized and generally noisy motion information conveyed in the bit stream generated by the encoder. One key feature of such filters is an adaptive filter length depending on the image content and the quality of the motion field. This paper derives mathematical equations to model the behaviour of one such filter in the presence of noisy motion vectors. The predicted optimal filter lengths are demonstrated to have a global optimum. They also show strong correlation with a real-world implementation of the previously introduced Temporal Trajectory Filter based on the HEVC main profile.
The MPEG-4 standard and its applications in virtual 3D environments

The Moving Pictures Experts Group (SC11/WG29 of ISO/IEC) is currently developing the MPEG-4 standard aiming at interactive multimedia applications. Its first version was released in early 1999, a second one will follow early 2000. In addition to an increased compression efficiency for storage and transmission applications at a wide range of bit-rates, MPEG-4 will offer advanced content-based functionalities for interactive services, such as the possibility of accessing and manipulating natural or synthetic audiovisual objects in 2D or 3D scene representations. Against this background the paper first outlines the current status of MPEG-4 standardization and then concentrates on an example application if MPEG-4 to interactive services using virtual 3D environments in combination with streaming of real-time data.
On the detection and localization of facial occlusions and its use within different scenarios

Face analysis is a very active research field, due to its large variety of applications and the different challenges (illumination, pose, expressions or occlusions) the methods need to cope with. Facial occlusions are one of the biggest challenges since they are difficult to model and have a large influence on the performance of subsequent analysis modules. This paper describes a face detection/classification module that allows to detect and localize faces and present occlusions and discusses the use of this additional information within different application scenarios. The approach is evaluated on two databases with realistic occlusions and performs very well for the different detection/classification tasks. It achieves a f-measure of over 97% for face detection and around 86% for component detection. Regarding the occlusion detection, the proposed approach reaches a recognition rate above 91% for both faces and components.
Robust concealment for erroneous block bursts in stereoscopic images

With the increasing number of image communication applications especially in the low complexity domain, error concealment has become a very important field of research. Since many compression standards for images and videos are block-based a lot of methods were applied to conceal block losses in monocular images. The fast progress of capture, representation and display technologies for 3D image data advances the efforts on 3D concealment strategies. Because of their psycho-visual characteristics, stereoscopic images have to fulfill a very high quality demand. We propose an algorithm that makes use of the redundancies between two views of a stereo image pair. In many cases erroneous block bursts occur and can be highly disturbing, thus we mainly concentrate on these errors. In addition, we focused on the quality assessment of several error concealment strategies. Beside the objective evaluation measures, we carried out a subjective quality test following the DSCQS methodology as proposed by MPEG. The results of this test demonstrate the efficiency of the approach.
Comparison of MPEG-7 audio spectrum projection features and MFCC applied to speaker recognition, sound classification and audio segmentation

We evaluate the MPEG-7 audio spectrum projection (ASP) features for general sound recognition performance against the well established MFCC. The recognition tasks of interest are speaker recognition, sound classification, and segmentation of audio using sound/speaker identification. For sound classification we use three approaches: direct approach; hierarchical approach without hints; hierarchical approach with hints. For audio segmentation, the MPEG-7 ASP features and MFCCs are used to train hidden Markov models (HMM) for individual speakers and sounds. The trained sound/speaker models are then used to segment conversational speech involving a given subset of people in panel discussion television programs. Results show that the MFCC approach yields a sound/speaker recognition rate superior to MPEG-7 implementations.