nils-herrmann/clean_scirepeval huggingface dataset:
Experiments on far-field multichannel speech processing in smart homes

In this paper, we examine three problems that rise in the modern, challenging area of far-field speech processing. The developed methods for each problem, namely (a) multi-channel speech enhancement, (b) voice activity detection, and (c) speech recognition, are potentially applicable to a distant speech recognition system for voice-enabled smart home environments. The obtained results on real and simulated data, regarding the smart home speech applications, are quite promising due to the accomplished improvements made in the employed signal processing methods.
Morphological filters-Part I: Their set-theoretic analysis and relations to linear shift-invariant filters

This paper examines the set-theoretic interpretation of morphological filters in the framework of mathematical morphology and introduces the representation of classical linear filters in terms of morphological correlations, which involve supremum/infimum operations and additions. Binary signals are classified as sets, and multilevel signals as functions. Two set-theoretic representations of signals are reviewed. Filters are classified as set-processing (SP) or function-processing (FP). Conditions are provided for certain FP filters that pass binary signals to commute with signal thresholding because then they can be analyzed and implemented as SP filters. The basic morphological operations of set erosion, dilation, opening, and closing are related to Minkowski set operations and are used to construct FP morphological filters. Emphasis is then given to analytically and geometrically quantifying the similarities and differences between morphological filtering of signals by sets and functions; the latter case allows the definition of morphological convolutions and correlations. Toward this goal, various properties of FP morphological filters are also examined. Linear shift-invariant filters (due to their translation-invariance) are uniquely characterized by their kernel, which is a special collection of input signals. Increasing linear filters are represented as the supremum of erosions by their kernel functions. If the filters are also discrete and have a finite-extent impulse response, they can be represented as the supremum of erosions only by their minimal (with respect to a signal ordering) kernel functions. Stable linear filters can be represented as the sum of (at most) two weighted suprema of erosions. These results demonstrate the power of mathematical morphology as a unifying approach to both linear and nonlinear signal-shaping strategies.
Multimodal Fusion and Learning with Uncertain Features Applied to Audiovisual Speech Recognition

We study the effect of uncertain feature measurements and show how classification and learning rules should be adjusted to compensate for it. Our approach is particularly fruitful in multimodal fusion scenarios, such as audio-visual speech recognition, where multiple streams of complementary features whose reliability is time-varying are integrated. For such applications, by taking the measurement noise uncertainty of each feature stream into account, the proposed framework leads to highly adaptive multimodal fusion rules for classification and learning which are widely applicable and easy to implement. We further show that previous multimodal fusion methods relying on stream weights fall under our scheme under certain assumptions; this provides novel insights into their applicability for various tasks and suggests new practical ways for estimating the stream weights adaptively. The potential of our approach is demonstrated in audio-visual speech recognition experiments.
A fast multigrid implicit algorithm for the evolution of geodesic active contours

Active contour models are among the most popular PDE-based tools in computer vision. In this paper we present a new algorithm for the fast evolution of geodesic active contours and compare it with other established numerical schemes. The new algorithm employs a full time-implicit and unconditionally stable numerical scheme and applies multigrid methods for the efficient solution of the occurring sparse linear system. When we utilize very big time-steps for the numerical evolution of the front, the proposed scheme has increased accuracy and better rotational invariance properties compared with the alternative AOS scheme. This allows for the rapid evolution and convergence of the contour to its final configuration after only very few iterations. Standard pyramidal and/or narrowband techniques can be easily integrated into our algorithm and further accelerate the curve evolution. Experimental results in object boundary detection demonstrate the power of the method.
Multiband Modulation Energy Tracking for Noisy Speech Detection

The ability to accurately locate the boundaries of speech activity is an important attribute of any modern speech recognition, processing, or transmission system. The effort in this paper is the development of efficient, sophisticated features for speech detection in noisy environments, using ideas and techniques from recent advances in speech modeling and analysis, like presence of modulations in speech formants, energy separation and multiband filtering. First we present a method, conceptually based on a classic speech-silence discrimination procedure, that uses some newly developed, short-time signal analysis tools and provide for it a detection theoretic motivation. The new energy and spectral content representations are derived through filtering the signal in various frequency bands, estimating the Teager-Kaiser energy for each and demodulating the most active one in order to derive the signal's dominant AM-FM components. This modulation approach demonstrated an improved robustness in noise over the classic algorithm, reaching an average error reduction of 33.5% under 5-30-dB noise. Second, by incorporating alternative modulation energy features in voice activity detection, improvement in overall misclassification error of a high hit rate detector reached 7.5% and 9.5% on different benchmarks
Gestural teleoperation of a mobile robot based on visual recognition of sign language static handshapes

This paper presents results achieved in the frames of a national research project (titled “DIANOEMA”), where visual analysis and sign recognition techniques have been explored on Greek Sign Language (GSL) data. Besides GSL modelling, the aim was to develop a pilot application for teleoperating a mobile robot using natural hand signs. A small vocabulary of hand signs has been designed to enable desktopbased teleoperation at a high-level of supervisory telerobotic control. Real-time visual recognition of the hand images is performed by training a multi-layer perceptron (MLP) neural network. Various shape descriptors of the segmented hand posture images have been explored as inputs to the MLP network. These include Fourier shape descriptors on the contour of the segmented hand sign images, moments, compactness, eccentricity, and histogram of the curvature. We have examined which of these shape descriptors are best suited for real-time recognition of hand signs, in relation to the number and choice of hand postures, in order to achieve maximum recognition performance. The hand-sign recognizer has been integrated in a graphical user interface, and has been implemented with success on a pilot application for real-time desktop-based gestural teleoperation of a mobile robot vehicle.
Human action recognition using Histographic methods and hidden Markov models for visual martial arts applications

Human Action Recognition is being used with an increasing rate in applications designed to describe human activity in everyday life. However, some areas still remain far from the epicenter of scientific research, like dynamic problems of detection and classification of movements from visual Martial Arts. With this paper, we are proposing a novel recognition system focused on these types of action, based on the use of local spatio-temporal features from Histographic methods, as those extracted by Histograms of Oriented Gradient (HOG) and Histograms of Optical Flow (HOF), while we also pursue the reduction of the problem's dimensionality by applying on them Principal Components Analysis (PCA). In continuation, we combine these features with the use of Hidden Markov Models (HMM) in order to train models for each different movement. Our system is tested with very encouraging results upon a database comprising sequences of shotokan karate movements (katas), created by us for the needs of this research. In parallel, we additionally attach an educational character to our application, with the extraction of a score for the accuracy of execution of each movement based on the prototypes we have built.
Bayesian Inference on Multiscale Models for Poisson Intensity Estimation: Applications to Photon-Limited Image Denoising

We present an improved statistical model for analyzing Poisson processes, with applications to photon-limited imaging. We build on previous work, adopting a multiscale representation of the Poisson process in which the ratios of the underlying Poisson intensities (rates) in adjacent scales are modeled as mixtures of conjugate parametric distributions. Our main contributions include: 1) a rigorous and robust regularized expectation-maximization (EM) algorithm for maximum-likelihood estimation of the rate-ratio density parameters directly from the noisy observed Poisson data (counts); 2) extension of the method to work under a multiscale hidden Markov tree model (HMT) which couples the mixture label assignments in consecutive scales, thus modeling interscale coefficient dependencies in the vicinity of image edges; 3) exploration of a 2-D recursive quad-tree image representation, involving Dirichlet-mixture rate-ratio densities, instead of the conventional separable binary-tree image representation involving beta-mixture rate-ratio densities; and 4) a novel multiscale image representation, which we term Poisson-Haar decomposition, that better models the image edge structure, thus yielding improved performance. Experimental results on standard images with artificially simulated Poisson noise and on real photon-limited images demonstrate the effectiveness of the proposed techniques.
A cross-validatory statistical approach to scale selection for image denoising by nonlinear diffusion

Scale-spaces induced by diffusion processes play an important role in many computer vision tasks. Automatically selecting the most appropriate scale for a particular problem is a central issue for the practical applicability of such scale-space techniques. This paper concentrates on automatic scale selection when nonlinear diffusion scale-spaces are utilized for image denoising. The problem is studied in a statistical model selection framework and cross-validation techniques are utilized to address it in a principled way. The proposed novel algorithms do not require knowledge of the noise variance and have acceptable computational cost. Extensive experiments on natural images show that the proposed methodology leads to robust algorithms, which outperform existing techniques for a wide range of noise types and noise levels.
Multimodal human action recognition in assistive human-robot interaction

Within the context of assistive robotics we develop an intelligent interface that provides multimodal sensory processing capabilities for human action recognition. Human action is considered in multimodal terms, containing inputs such as audio from microphone arrays, and visual inputs from high definition and depth cameras. Exploring state-of-the-art approaches from automatic speech recognition, and visual action recognition, we multimodally recognize actions and commands. By fusing the unimodal information streams, we obtain the optimum multimodal hypothesis which is to be further exploited by the active mobility assistance robot in the framework of the MOBOT EU research project. Evidence from recognition experiments shows that by integrating multiple sensors and modalities, we increase multimodal recognition performance in the newly acquired challenging dataset involving elderly people while interacting with the assistive robot.
Advances on action recognition in videos using an interest point detector based on multiband spatio-temporal energies

This paper proposes a new visual framework for action recognition in videos, that consists of an energy detector coupled with a carefully designed multiband energy based filterbank. The tracking of video energy is performed using perceptually inspired 3D Gabor filters combined with ideas from Dominant Energy Analysis. Within this framework, we utilize different alternatives such as non-linear energy operators where actions are implicitly considered as manifestations of spatio-temporal oscillations in the dynamic visual stream. Texture and motion decomposition of actions through multiband filtering is the basis of our approach. This new energy-based saliency measure of action videos leads to the extraction of local spatio-temporal interest points that give promising results for the task of action recognition. Such interest points are processed further in order to formulate a robust representation of an action in a video. Theoretical formulation is supported by evaluation in two popular action databases, in which our method seems to outperform the state of the art.
Audiovisual-to-Articulatory Speech Inversion Using HMMs

We address the problem of audiovisual speech inversion, namely recovering the vocal tract's geometry from auditory and visual speech cues. We approach the problem in a statistical framework, combining ideas from multistream Hidden Markov Models and canonical correlation analysis, and demonstrate effective estimation of the trajectories followed by certain points of interest in the speech production system. Our experiments show that exploiting both audio and visual modalities clearly improves performance relative to either audio-only or visual-only estimation. We report experiments on the QSMT database which contains audio, video, and electromagnetic articulography data recorded in parallel.
Movie summarization based on audiovisual saliency detection

Based on perceptual and computational attention modeling studies, we formulate measures of saliency for an audiovisual stream. Audio saliency is captured by signal modulations and related multi-frequency band features, extracted through nonlinear operators and energy tracking. Visual saliency is measured by means of a spatiotemporal attention model driven by various feature cues (intensity, color, motion). Audio and video curves are integrated in a single attention curve, where events may be enhanced, suppressed or vanished. The presence of salient events is signified on this audiovisual curve by geometrical features such as local extrema, sharp transition points and level sets. An audiovisual saliency-based movie summarization algorithm is proposed and evaluated. The algorithm is shown to perform very well in terms of summary informativeness and enjoyability for movie clips of various genres.
Some experiments in ADPCM coding of images

This paper is a preliminary report on a study of the application of two-dimensional linear prediction in image quantization. The study has focused on three major concerns: implementation of an adaptive linear predictor, adaptive quantization of the prediction error signal, and the adaptive predictive coding of density (logarithm of intensity) images. The results of the study indicate that through the use of adaptive prediction and quantization, a high level of image fidelity can be obtained for both intensity and density images at information rates well below one bit/pixel.
Affine-invariant modeling of shape-appearance images applied on sign language handshape classification

We propose a novel affine-invariant modeling of hand shape-appearance images, which offers a compact and descriptive representation of the hand configurations. Our approach combines: 1) A hybrid representation of both shape and appearance of the hand that models the handshapes without any landmark points. 2) Modeling of the shape-appearance images with a linear combination of variation images that is followed by an affine transformation, which accounts for modest pose variation. 3) Finally, an optimization based fitting process that results on the estimated variation image coefficients that are further employed as features. The proposed modeling is applied on handshapes from Sign Language video data after segmentation and tracking. It is evaluated on extensive experiments of handshape classification, which investigate the effect of the involved parameters and moreover provide a variety of comparisons to base-line approaches found in the literature. The results of at least 10.5% absolute improvement indicate the effectiveness of our approach in the handshape classification problem.
Demonstration of an HMM-based photorealistic expressive audio-visual speech synthesis system

The usage of conversational agents is rapidly increasing in everyday life (cortana, siri, etc.). It has been shown that the inclusion of a talking face, increases the intelligibility of speech and the naturalness of human-computer interaction. Furthermore, an agent capable of expressing emotions has a stronger appeal to the human party and affects the interlocutor's emotional state. The proposed demonstration is a Hidden Markov Model (HMM) based photorealistic audio-visual speech synthesis system, capable of expressing emotions [1, 2]. The system is capable of generating a talking head speaking in three emotions: happiness, anger, and sadness, plus in neutral speaking style. Further capabilities of the system include 1) the usage of HMM interpolation  in order to generate speech with mixtures of the original emotions (e.g., both anger and happiness), and speech with different levels of expressiveness (by mixing with the neutral emotion), 2) the usage of HMM adaptation , in order to adapt to a target emotion using only a few number of sentences. Equipment In order to showcase our system we will use a laptop and speakers. The system will run fully on the laptop. Demonstration Experience During the demonstration, viewers will have the opportunity to: 1. Watch videos of the talking head speaking in 3 different emotions (plus neutral) and see how the expressive talking head feels more natural compared to the talking head speaking in neutral style. 2. Watch the talking head speaking in two or more emotions at the same time, and see how the weights assigned to each emotion affects the outcome. It will also be of great interest to see which emotion each viewer perceives. In addition, through interpolation with the neutral emotion, viewers will be able to watch the talking head speak in different expressiveness levels for each emotion. 3. See how the neutral talking head can be adapted to speak in another emotion using only a few sentences, and how the number of sentences used affects the expressiveness of the resulting talking head.
Modulation models for image processing and wavelet-based image demodulation

AM-FM modulation models for two- and higher-dimensional complex-valued nonstationary signals and images that are locally coherent but globally wideband are studied. Demodulation techniques are studied for demodulating both one-component and multicomponent signals. For the case of multicomponent signals immersed in noise, the individual components must be isolated by spectrally localized multiband filtering prior to demodulation. An algorithm for demodulating the filtered signal components is developed using a quasi-eigenfunction approximation, and bounds for the approximation error are given. After multiband filtering with Gabor wavelets, the filtered demodulation algorithm is used to estimate the dominant emergent frequencies of multicomponent images.<<ETX>>
Audiovisual-to-articulatory speech inversion using Active Appearance Models for the face and Hidden Markov Models for the dynamics

We are interested in recovering aspects of vocal tract's geometry and dynamics from auditory and visual speech cues. We approach the problem in a statistical framework based on Hidden Markov Models and demonstrate effective estimation of the trajectories followed by certain points of interest in the speech production system. Alternative fusion schemes are investigated to account for asynchrony between the modalities and allow independent modeling of the dynamics of the involved streams. Visual cues are extracted from the speaker's face by means of active appearance modeling. We report experiments on the QSMT database which contains audio, video, and electromagnetic articulography data recorded in parallel. The results show that exploiting both audio and visual modalities in a multistream HMM based scheme clearly improves performance relative to either audio or visual-only estimation.
Optimum design of chamfer distance transforms

The distance transform has found many applications in image analysis. Chamfer distance transforms are a class of discrete algorithms that offer a good approximation to the desired Euclidean distance transform at a lower computational cost. They can also give integer-valued distances that are more suitable for several digital image processing tasks. The local distances used to compute a chamfer distance transform are selected to minimize an approximation error. A new geometric approach is developed to find optimal local distances. This new approach is easier to visualize than the approaches found in previous work, and can be easily extended to chamfer metrics that use large neighborhoods. A new concept of critical local distances is presented which reduces the computational complexity of the chamfer distance transform without increasing the maximum approximation error.
Advances in texture analysis: energy dominant component & multiple hypothesis testing

Modelling textured images as AM-FM functions has been applied during the last years to texture analysis and segmentation tasks. In this paper we present some advances in two directions, namely the improvement of modulation feature extraction and texture vs. non texture discrimination. First we present a modified dominant AM-FM component analysis scheme based on non-linear energy operators. Subsequently we propose a novel approach to the discrimination between textured and non-textured image areas using multiband filter responses which is formulated as a statistical multiple hypothesis testing problem. The theoretical insights of this approach are supported by experimental validation using natural images.
Demodulation of images modeled by amplitude-frequency modulations using multidimensional energy separation

Locally narrowband images can be modeled as 2D spatial AM-FM signals with several applications in image texture analysis and computer vision. In this paper we formulate such an image demodulation problem, and present a solution based on the multidimensional energy operator /spl Phi/(f)=/spl par//spl nabla/f/spl par//sup 2/-f/spl nabla//sup 2/f. We discuss some interesting properties of this multidimensional operator and develop multidimensional energy separation algorithms to estimate the amplitude envelope and instantaneous frequencies of 2D spatially-varying AM-FM signals. Experiments are also presented on applying this 2D energy demodulation algorithm to estimate the instantaneous amplitude contrast and spatial frequencies of image textures bandpass filtered via Gabor filters. The attractive features of the multidimensional energy operator and the 2D energy separation algorithm are their simplicity, efficiency, and ability to track instantaneously-varying spatial modulation patterns.<<ETX>>