nils-herrmann/clean_scirepeval huggingface dataset:
Dominant spatio-temporal modulations and energy tracking in videos: Application to interest point detection for action recognition

The presence of multiband amplitude and frequency modulations (AM-FM) in wideband signals, such as textured images or speech, has led to the development of efficient multicomponent modulation models for low-level image and sound analysis. Moreover, compact yet descriptive representations have emerged by tracking, through non-linear energy operators, the dominant model components across time, space or frequency. In this paper, we propose a generalization of such approaches in the 3D spatio-temporal domain and explore the potential of incorporating the Dominant Component Analysis scheme for interest point detection and human action recognition in videos. Within this framework, actions are implicitly considered as manifestations of spatio-temporal oscillations in the dynamic visual stream. Multiband filtering and energy operators are applied to track the source energy in both spatial and temporal frequency bands. A new measure for extracting keypoint locations is formulated as the temporal dominant energy computed over the spatial dominant components, in terms of their modulation energy, of input video frames. Theoretical formulation is supported by evaluation and comparisons in human action classification, which demonstrate the potential of the proposed spatio-temporal detector.
Some advances in nonlinear speech modeling using modulations, fractals, and chaos

We briefly summarize our on-going work on modeling nonlinear structures in speech signals, caused by modulation and turbulence phenomena, using the theories of modulation, fractals, and chaos as well as suitable nonlinear signal analysis methods. Further, we focus on two advances: (i) AM-FM modeling of fricative sounds with random modulation signals of the 1/f-noise type; (ii) improved methods for speech analysis and prediction on reconstructed multidimensional attractors.
An integrated system for digital restoration of prehistoric Theran wall paintings

We present a computer vision system for robust restoration of prehistoric Theran wall paintings, replacing or just supporting the work of a specialist. In the case of significant information loss on some areas of murals, the local inpainting methods are not sufficient for satisfactory restoration. Our strategy is to detect an area of relevant semantics, geometry and color in another location of the wall paintings, which in turn is stitched into the missing area by applying a seamless image stitching algorithm. An important part of our digital restoration system is the damaged and missing areas detector. It is used in combination with total variation inpainting at first for the missing area extraction and repair, and secondly for the elimination of minor defects on the retrieved part in the non-local inpainting mechanism. We propose a morphological algorithm for rough detection and we improve upon this approach by incorporating edge information. For missing areas with complicated boundaries we enhance the detection by using iterated graph cuts.
Projective non-negative matrix factorization for unsupervised graph clustering

We develop an unsupervised graph clustering and image segmentation algorithm based on non-negative matrix factorization. We consider arbitrarily represented visual signals (in 2D or 3D) and use a graph embedding approach for image or point cloud segmentation. We extend a Projective Non-negative Matrix Factorization variant to include local spatial relationships over the image graph. By using properly defined region features, one can apply our method of unsupervised graph clustering for object and image segmentation. To demonstrate this, we apply our ideas on many graph based segmentation tasks such as 2D pixel and super-pixel segmentation and 3D point cloud segmentation. Finally, we show results comparable to those achieved by the only existing work in pixel based texture segmentation using Nonnegative Matrix Factorization, deploying a simple yet effective extension that is parameter free. We provide a detailed convergence proof of our spatially regularized method and various demonstrations as supplementary material. This novel work brings together graph clustering with image segmentation.
Texture modulation-constrained image decomposition

Texture modeling and separation of structure in images are treated in synergy. A variational image decomposition scheme is formulated using explicit texture reconstruction constraints from the outputs of linear filters tuned to different spatial frequencies and orientations. Relevant to the texture image part information is reconstructed using modulation modeling and component selection. The general formulation leads to a u + Kv model of K + 1 image components, with multiple texture subcomponents.
Unsupervised classification of extreme facial events using active appearance models tracking for sign language videos

We propose an Unsupervised method for Extreme States Classification (UnESC) on feature spaces of facial cues of interest. The method is built upon Active Appearance Models (AAM) face tracking and on feature extraction of Global and Local AAMs. UnESC is applied primarily on facial pose, but is shown to be extendable for the case of local models on the eyes and mouth. Given the importance of facial events in Sign Languages we apply the UnESC on videos from two sign language corpora, both American (ASL) and Greek (GSL) yielding promising qualitative and quantitative results. Apart from the detection of extreme facial states, the proposed Un-ESC also has impact for SL corpora lacking any facial annotations.
Generalized multiscale connected operators with applications to granulometric image analysis

In this paper, generalized granulometric size distributions and size histograms (a.k.a 'pattern spectra') are developed using generalized multiscale lattice operators of the opening and closing type. The generalized size histograms are applied to granulometric analysis of soil-section images. An interesting structure is obtained when the histogram is based on area openings. Furthermore, a fast implementation of the generalized size histograms is presented using threshold analysis-synthesis. Comparisons with size distributions based on conventional morphological operators indicate that the generalized histograms provide a more direct and informative description of the image content in objects with scale-dependent geometric attributes. Applications are also developed for studying the structure of soilsection images.
Modeling resonances with phase modulated self-similar processes [speech processing example]

In this paper, we propose a nonlinear model for time-varying random resonances where the instantaneous phase (and frequency) of a sinusoidal oscillation is allowed to vary proportionally to a random process that belongs to the class of /spl alpha/-stable self-similar stochastic processes. This is a general model that includes phase modulations by fractional Brownian motion or fractional stable Levy motion as special cases. We explore theoretically this random modulation model and derive analytically its autocorrelation and power spectrum. We also propose an algorithm to fit this model to arbitrary resonances with random phase modulation. Further, we apply the above ideas to some speech data and demonstrate that the model is suitable for fricative sounds.
Demodulation of discrete multicomponent AM-FM signals using periodic algebraic separation and energy demodulation

Existing multicomponent AM-FM demodulation algorithms either assume spectrally distinct components or components separable via linear filtering and break down when the components overlap spectrally or if one of the components is stronger than the other. In this paper, we present a nonlinear algorithm for multicomponent AM-FM demodulation which avoids the above shortcomings and works well even for extremely small spectral separation of the components. The proposed algorithm separates the multicomponent demodulation problem into two tasks: periodicity-based algebraic separation of the components and then monocomponent demodulation via energy-based methods.
Multichannel linear predictive coding of color images

This paper reports on a preliminary study of applying single-channel (scalar) and multichannel (vector) 2-D linear prediction to color image modeling and coding. Also, the novel idea of a multi-input single-output 2-D ADPCM coder is introduced. The results of this study indicate that texture information in multispectral images can be represented by linear prediction coefficients or matrices, where as the prediction error conveys edge-information. Moreover, by using a single-channel edge-information we obtained, from original color images of 24 bits/pixel, reconstructed images of good quality at information rates of 1 bit/pixel or less.
Geodesic active regions for segmentation and tracking of human gestures in sign language videos

Reliable segmentation and motion tracking algorithms are required to achieve gesture detection and tracking for human-machine interaction. In this paper we present an efficient method for detecting and tracking moving hands in sign language video frames. We make use of the geodesic active region framework in conjunction with new color and motion forces; color information is provided by a skin color model, while motion information is derived from the optical flow field. Extensive experimentation indicates that the proposed algorithm behaves sufficiently well for gesture detection and tracking.
Estimation of eye gaze direction angles based on active appearance models

In this paper we demonstrate efficient methods for continuous estimation of eye gaze angles with application to sign language videos. The difficulty of the task lies on the fact that those videos contain images with low face resolution since they are recorded from distance. First, we proceed to the modeling of face and eyes region by training and fitting Global and Local Active Appearance Models (LAAM). Next, we propose a system for eye gaze estimation based on a machine learning approach. In the first stage of our method, we classify gaze into discrete classes using GMMs that are based either on the parameters of the LAAM, or on HOG descriptors for the eyes region. We also propose a method for computing gaze direction angles from GMM log-likelihoods. We qualitatively and quantitatively evaluate our methods on two sign language databases and compare with a state of the art geometric model of the eye based on LAAM landmarks, which provides an estimate in direction angles. Finally, we further evaluate our framework by getting ground truth data from an eye tracking system Our proposed methods, and especially the GMMs using LAAM parameters, demonstrate high accuracy and robustness even in challenging tasks.
An inpainting system for automatic image structure - texture restoration with text removal

In this paper we deal with the inpainting problem and with the problem of finding text in images. We first review many of the methods used for structure and texture inpaintings. The novel contribution of the paper is the combination of the inpainting techniques with the techniques of finding text in images and a simple morphological algorithm that links them. This combination results in an automatic system for text removal and image restoration that requires no user interface at all. Examples on real images show very good performance of the proposed system and the importance of the new linking algorithm.
Kinect-based multimodal gesture recognition using a two-pass fusion scheme

We present a new framework for multimodal gesture recognition that is based on a two-pass fusion scheme. In this, we deal with a demanding Kinect-based multimodal dataset, which was introduced in a recent gesture recognition challenge. We employ multiple modalities, i.e., visual cues, such as colour and depth images, as well as audio, and we specifically extract feature descriptors of the hands' movement, handshape, and audio spectral properties. Based on these features, we statistically train separate unimodal gesture-word models, namely hidden Markov models, explicitly accounting for the dynamics of each modality. Multimodal recognition of unknown gesture sequences is achieved by combining these models in a late, two-pass fusion scheme that exploits a set of unimodally generated n-best recognition hypotheses. The proposed scheme achieves 88.2% gesture recognition accuracy in the Kinect-based multimodal dataset, outperforming all recently published approaches on the same challenging multimodal gesture recognition task.
Generalized Flooding and Multicue PDE-Based Image Segmentation

Image segmentation remains an important, but hard-to-solve, problem since it appears to be application dependent with usually no a priori information available regarding the image structure. Moreover, the increasing demands of image analysis tasks in terms of segmentation results' quality introduce the necessity of employing multiple cues for improving image segmentation results. In this paper, we attempt to incorporate cues such as intensity contrast, region size, and texture in the segmentation procedure and derive improved results compared to using individual cues separately. We emphasize on the overall segmentation procedure, and we propose efficient simplification operators and feature extraction schemes, capable of quantifying important characteristics, like geometrical complexity, rate of change in local contrast variations, and orientation, that eventually favor the final segmentation result. Based on the well-known morphological paradigm of watershed transform segmentation, which exploits intensity contrast and region size criteria, we investigate its partial differential equation (PDE) formulation, and we extend it in order to satisfy various flooding criteria, thus making it applicable to a wider range of images. Going a step further, we introduce a segmentation scheme that couples contrast criteria in flooding with texture information. The modeling of the proposed scheme is done via PDEs and the efficient incorporation of the available contrast and texture information, is done by selecting an appropriate cartoon-texture image decomposition scheme. The proposed coupled segmentation scheme is driven by two separate image components: artoon U (for contrast information) and texture component. The performance of the proposed segmentation scheme is demonstrated through a complete set of experimental results and substantiated using quantitative and qualitative criteria.
Estimating double support in pathological gaits using an HMM-based analyzer for an intelligent robotic walker

For a robotic walker designed to assist mobility constrained people, it is important to take into account the different spectrum of pathological walking patterns, which result into completely different needs to be covered for each specific user. For a deployable intelligent assistant robot it is necessary to have a precise gait analysis system, providing real-time monitoring of the user and extracting specific gait parameters, which are associated with the rehabilitation progress and the risk of fall. In this paper, we present a completely non-invasive framework for the on-line analysis of pathological human gait and the recognition of specific gait phases and events. The performance of this gait analysis system is assessed, in particular, as related to the estimation of double support phases, which are typically difficult to extract reliably, especially when applying non-wearable and non-intrusive technologies. Furthermore, the duration of double support phases constitutes an important gait parameter and a critical indicator in pathological gait patterns. The performance of this framework is assessed using real data collected from an ensemble of elderly persons with different pathologies. The estimated gait parameters are experimentally validated using ground truth data provided by a Motion Capture system. The results obtained and presented in this paper demonstrate that the proposed human data analysis (modeling, learning and inference) framework has the potential to support efficient detection and classification of specific walking pathologies, as needed to empower a cognitive robotic mobility-assistance device with user-adaptive and context-aware functionalities.
Conditions for positivity of an energy operator

The authors present necessary and sufficient conditions such that the output from the Teager-Kaiser (1989) energy operator /spl lsqb/s/spl dot/(t)=ds(t)/dt/spl rsqb/ /spl Psi//sub c//spl lsqb/s(t)/spl rsqb/=s/spl dot//sup 2/(t)/spl minus/s(t)s/spl uml/(t) for continuous-time signals s(t) and the output from the corresponding discrete-time energy operator /spl Psi//sub d//spl lsqb/s(n)/spl rsqb/=s/sup 2/(n)/spl minus/s(n+1)s(n/spl minus/1) be non-negative everywhere. These operators have been shown to be effective for AM and FM demodulation in several useful classes of signals, such as speech and image signals.
MRL-filters: a general class of nonlinear systems and their optimal design for image processing

A class of morphological/rank/linear (MRL)-filters is presented as a general nonlinear tool for image processing. They consist of a linear combination between a morphological/rank filter and a linear filter. A gradient steepest descent method is proposed to optimally design these filters, using the averaged least mean squares (LMS) algorithm. The filter design is viewed as a learning process, and convergence issues are theoretically and experimentally investigated. A systematic approach is proposed to overcome the problem of nondifferentiability of the nonlinear filter component and to improve the numerical robustness of the training algorithm, which results in simple training equations. Image processing applications in system identification and image restoration are also presented, illustrating the simplicity of training MRL-filters and their effectiveness for image/signal processing.
Time-frequency distributions for automatic speech recognition

The use of general time-frequency distributions as features for automatic speech recognition (ASR) is discussed in the context of hidden Markov classifiers. Short-time averages of quadratic operators, e.g., energy spectrum, generalized first spectral moments, and short-time averages of the instantaneous frequency, are compared to the standard front end features, and applied to ASR. Theoretical and experimental results indicate a close relationship among these feature sets.
Photorealistic adaptation and interpolation of facial expressions using HMMS and AAMS for audio-visual speech synthesis

In this paper, motivated by the continuously increasing presence of intelligent agents in everyday life, we address the problem of expressive photorealistic audio-visual speech synthesis, with a strong focus on the visual modality. Emotion constitutes one of the main driving factors of social life and it is expressed mainly through facial expressions. Synthesis of a talking head capable of expressive audio-visual speech is challenging due to the data overhead that arises when considering the vast number of emotions we would like the talking head to express. In order to tackle this challenge, we propose the usage of two methods, namely Hidden Markov Model (HMM) adaptation and interpolation, with HMMs modeling visual parameters via an Active Appearance Model (AAM) of the face. We show that through HMM adaptation we can successfully adapt a “neutral” talking head to a target emotion with a small amount of adaptation data, as well as that through HMM interpolation we can robustly achieve different levels of intensity for an emotion.
Model-level data-driven sub-units for signs in videos of continuous Sign Language

We investigate the issue of sign language automatic phonetic subunit modeling, that is completely data driven and without any prior phonetic information. A first step of visual processing leads to simple and effective region-based visual features. Prior to the sub-unit modeling we propose to employ a pronunciation clustering step with respect to each sign. Afterwards, for each sign and pronunciation group we find the time segmentation at the hidden Markov model (HMM) level. The models employed refer to movements as a sequence of dominant hand positions. The constructed segments are exploited explicitly at the model level via hierarchical clustering of HMMs and lead to the data-driven movement sub-unit construction. The constructed movement sub-units are evaluated in qualitative analysis experiments on data from the Boston University (BU)- 400 American Sign Language corpus showing promising results.
Online Wideband Spectrum Sensing Using Sparsity

Wideband spectrum sensing is an essential part of cognitive radio systems. Exact spectrum estimation is usually inefficient as it requires sampling rates at or above the Nyquist rate. Using prior information on the structure of the signal could allow near exact reconstruction at much lower sampling rates. Sparsity of the sampled signal in the frequency domain is one of the popular priors studied for cognitive radio applications. Reconstruction of signals under sparsity assumptions has been studied rigorously by researchers in the field of compressed sensing (CS). CS algorithms that operate on batches of samples are known to be robust but can be computationally costly, making them unsuitable for cheap low power cognitive radio devices that require spectrum sensing in real time. On the other hand, online algorithms that are based on variations of the least mean squares (LMS) algorithm have very simple updates so they are computationally efficient and can easily adapt in real time to changes of the underlying spectrum. In this paper, we will present two variations of the LMS algorithm that enforce sparsity in the estimated spectrum given an upper bound on the number of nonzero coefficients. Assuming that the number of nonzero elements in the spectrum is known, we show that under conditions the hard threshold operation can only reduce the error of our estimation. We will also show that we can estimate the number of nonzero elements of the spectrum at each iteration based on our online estimations. Finally, we numerically compare our algorithm with other online sparsity-inducing algorithms in the literature.
Soil image segmentation and texture analysis: a computer vision approach

Automated processing of digitized soilsection images reveals elements of soil structure and draws primary estimates of bioecological importance, like ground fertility and changes in terrestrial ecosystems. We examine a sophisticated integration of some modern methods from computer vision for image feature extraction, texture analysis, and segmentation into homogeneous regions, relevant to soil micromorphology. First, we propose the use of a morphological partial differential equation-based segmentation scheme based on seeded region-growing and level curve evolution with speed depending on image contrast. Second, we analyze surface texture information by modeling image variations as local modulation components and using multifrequency filtering and instantaneous nonlinear energy-tracking operators to estimate spatial modulation energy. By separately exploiting contrast and texture information, through multiscale image smoothing, we propose a joint image segmentation method for further interpretation of soil images and feature measurements. Our experimental results in images digitized under different specifications and scales demonstrate the efficacy of our proposed computational methods for soil structure analysis. We also briefly demonstrate their applicability to remote sensing images.