nils-herrmann/clean_scirepeval huggingface dataset:
Audio retrieval by latent perceptual indexing

We present a query-by-example audio retrieval framework by indexing audio clips in a generic database as points in a latent perceptual space. First, feature-vectors extracted from the clips in the database are grouped into reference clusters using an unsupervised clustering technique. An audio clip-to-cluster matrix is constructed by keeping count of the number of features that are quantized into each of the reference clusters. By singular-value decomposition of this matrix, each audio clip of the database is mapped into a a point in the latent perceptual space. This is used for indexing the retrieval system. Since each of the initial reference clusters represents a specific perceptual quality in a perceptual space (similar to words that represent specific concepts in the semantic space), querying-by-example results in clips that have similar perceptual qualities. Subjective human evaluation indicates about 75% retrieval performance. Evaluation on semantic categories reveals that the system performance is comparable to other proposed methods.
Analysis of interaction attitudes using data-driven hand gesture phrases

Hand gesture is one of the most expressive, natural and common types of body language for conveying attitudes and emotions in human interactions. In this paper, we study the role of hand gesture in expressing attitudes of friendliness or conflict towards the interlocutors during interactions. We first employ an unsupervised clustering method using a parallel HMM structure to extract recurring patterns of hand gesture (hand gesture phrases or primitives). We further investigate the validity of the derived hand gesture phrases by examining the correlation of dyad's hand gesture for different interaction types defined by the attitudes of interlocutors. Finally, we model the interaction attitudes with SVM using the dynamics of the derived hand gesture phrases over an interaction. The classification results are promising, suggesting the expressiveness of the derived hand gesture phrases for conveying attitudes and emotions.
Deep convolutional recurrent neural network with attention mechanism for robust speech emotion recognition

We present a deep convolutional recurrent neural network for speech emotion recognition based on the log-Mel filterbank energies, where the convolutional layers are responsible for the discriminative feature learning. Based on the hypothesis that a better understanding of the internal configuration within an utterance would help reduce misclassification, we further propose a convolutional attention mechanism to learn the utterance structure relevant to the task. In addition, we quantitatively measure the performance gain contributed by each module in our model in order to characterize the nature of emotion expressed in speech. The experimental results on the eNTERFACE'05 emotion database validate our hypothesis and also demonstrate an absolute improvement by 4.62% compared to the state-of-the-art approach.
Features for comparing tune similarity of songs across different languages

Finding tunes that are similar across languages and cultures offers new ways to study global musical influences and similarities. From a signal processing point of view, we find that the availability of vocal music tracks provides us a means for computing tune similarity even in the presence of language differences. While the different acoustic characteristics of each language add to the inherent ambiguity in these kind of problems, the guarantee that a vocal track exists can be a boon in disguise. For this purpose we use the Multi Band Autocorrelation Peak (MBAP) features, extracted in multiple bands providing complementary information which helps to improve the accuracy. Results obtained on a classification task suggest that these features can outperform traditional features like Chroma which capture information from the entire spectrum. Alignment cost using the dynamic time warping algorithm was used a classification metric on a dataset of songs obtained from Youtube.
Acoustic correlates of user response to error in human-computer dialogues

Using tagged data from the DARPA Communicator Project, we investigate acoustic features of user responses to system errors. We measure acoustic parameters such as energy, fundamental frequency, sub-band energy, ratios of voiced, unvoiced and silent regions of speech, fundamental frequency slope, spectral slope, and spectral center of gravity. We investigate different types of user responses to the errors, including frustration and various types of corrections. It is confirmed that the most prominent acoustic parameter for responses to the errors is fundamental frequency maximum and range, while other features are found to be salient for specific reaction types. More interestingly, acoustic characteristics of user responses to the errors are found to be different depending on whether the responses are the initial or continued responses to the errors. Similarly, normal user responses can differ acoustically depending on whether or not they were preceded by responses to error. We also present results on automatic classification of error response types using these features.
Accelerated 3D MRI of vocal tract shaping using compressed sensing and parallel imaging

3D MRI of the upper airway has provided valuable insights into vocal tract shaping and data for the modeling of speech production. Small movements of articulators can lead to large changes in the produced sound, therefore improving the resolution of these datasets, within the constraints of a sustained sound (6–12 seconds), is an important area for investigation. This paper provides the first application of compressed sensing (CS) with parallel imaging to high-resolution 3D upper airway MRI. We use spatial finite difference as the sparsifying transform, and investigate the use of high-resolution phase information as a constraint during CS reconstruction. In a retrospective subsampling experiment with no sound production, 5x undersampling produced acceptable image quality when using phase-constrained CS reconstruction. The prospective use of this accelerated acquisition enabled 3D vocal-tract MRI during sustained production of English /s/,/∫/,/i/,/r/ with 1.33×1.33×1.33-mm3 spatial resolution and 10-seconds of scan time.
An attribute-based approach to audio description applied to segmenting vocal sections in popular music songs

We present a descriptive approach for analyzing audio scenes that can comprise a mixture of audio sources. We apply this method to segment popular music songs into vocal and non-vocal sections. Unlike existing methods that directly rely on within-class feature similarities of acoustic sources, the proposed data-driven system is based on a training set where the acoustic sources are grouped by their perceptual or semantic attributes. Our audio analysis approach is based on a quantitative time-varying metric to measure the interaction between acoustic sources present in a scene developed using pattern recognition methods. Using the proposed system that is trained on a general sound effects library, we achieve less than ten percent vocal-section segmentation error and less than five percent false alarm rates when evaluated on a database of popular music recordings that spans four different genres (rock, hiphop, pop, and easy listening)
Improved HMM phone and triphone models for real-time ASR telephony applications

The development of human-machine dialog applications for messaging and information retrieval over the telephone poses stringent requirements on the accuracy and speed of the automatic speech recognition (ASR) system. The authors describe strategies for improved acoustic-phone modeling directed towards increasing recognition accuracy while keeping the number of phone units low. Specifically, the paper considers: (1) the development of an improved set of head-tail context-dependent (CD) triphones; (2) a novel criterion for better selection of the number of states assigned to each phone unit based on the coefficient of variation measure of feature components in HMM-Gaussians. The performance of the models is evaluated using data that represent real telephony applications.
Speech rate estimation via temporal correlation and selected sub-band correlation

In this paper, we propose a novel method for speech rate estimation without requiring automatic speech recognition. It extends the methods of spectral subband correlation by including temporal correlation and the use of selecting prominent spectral subbands for correlation. Furthermore, to address some of the practical issues in previously published methods, we introduce some novel components into the algorithm such as the use of pitch confidence, magnifying window, relative peak measure and relative threshold. By selecting the parameters and thresholds from realistic development sets, this method achieves a 0.972 correlation coefficient on syllable number estimation and a 0.706 correlation on speech rate estimation. This result is about 6.9% improvement over the current best single estimator and 3.5% improvement over the current multi-estimator, evaluated on the same switchboard database.
Nonproduct Data-Dependent Partitions for Mutual Information Estimation: Strong Consistency and Applications

A new framework for histogram-based mutual information estimation of probability distributions equipped with density functions in (Rd,B(Rd)) is presented in this work. A general histogram-based estimate is proposed, considering nonproduct data-dependent partitions, and sufficient conditions are stipulated to guarantee a strongly consistent estimate for mutual information. Two emblematic families of density-free strongly consistent estimates are derived from this result, one based on statistically equivalent blocks (the Gessaman's partition) and the other, on a tree-structured vector quantization scheme.
Unifying conversational multimedia interfaces for accessing network services across communication devices

This paper investigates architecture and interface design issues in engineering conversational multimedia interfaces for accessing network-based services. In particular, we focus on the problem of providing conversation-capable multimodal, multimedia services that can be accessed from a variety of communication devices and environments (telephones, PDAs, desktops etc) with a "consistent" user experience. The goals are two-fold: (i) provisioning a standards-compliant architecture and (ii) designing interfaces that are adaptive to the access device characteristics. We use a directory assistance application that can be accessed from a telephone, kiosk, desktop or a PDA to illustrate architecture and interface design issues.
Creating ensemble of diverse maximum entropy models

Diversity of a classifier ensemble has been shown to benefit overall classification performance. But most conventional methods of training ensembles offer no control on the extent of diversity and are meta-learners. We present a method for creating an ensemble of diverse maximum entropy (∂MaxEnt) models, which are popular in speech and language processing. We modify the objective function for conventional training of a MaxEnt model such that its output posterior distribution is diverse with respect to a reference model. Two diversity scores are explored - KL divergence and posterior cross-correlation. Experiments on the CoNLL-2003 Named Entity Recognition task and the IEMOCAP emotion recognition database show the benefits of a ∂MaxEnt ensemble.
Lattice-based lexical cues for word fragment detection in conversational speech

Previous approaches to the problem of word fragment detection in speech have focussed primarily on acoustic-prosodic features , . This paper proposes that the output of a continuous Automatic Speech Recognition (ASR) system can also be used to derive robust lexical features for the task. We hypothesize that the confusion in the word lattice generated by the ASR system can be exploited for detecting word fragments. Two sets of lexical features are proposed -one which is based on the word confusion, and the other based on the pronunciation confusion between the word hypotheses in the lattice. Classification experiments with a Support Vector Machine (SVM) classifier show that these lexical features perform better than the previously proposed acoustic-prosodic features by around 5.20% (relative) on a corpus chosen from the DARPA Transtac Iraqi-English (San Diego) corpus . A combination of both these feature sets improves the word fragment detection accuracy by 11.50% relative to using just the acoustic-prosodic features.
Robust Voice Activity Detection Using Long-Term Signal Variability

We propose a novel long-term signal variability (LTSV) measure, which describes the degree of nonstationarity of the signal. We analyze the LTSV measure both analytically and empirically for speech and various stationary and nonstationary noises. Based on the analysis, we find that the LTSV measure can be used to discriminate noise from noisy speech signal and, hence, can be used as a potential feature for voice activity detection (VAD). We describe an LTSV-based VAD scheme and evaluate its performance under eleven types of noises and five types of signal-to-noise ratio (SNR) conditions. Comparison with standard VAD schemes demonstrates that the accuracy of the LTSV-based VAD scheme averaged over all noises and all SNRs is ~6% (absolute) better than that obtained by the best among the considered VAD schemes, namely AMR-VAD2. We also find that, at -10 dB SNR, the accuracies of VAD obtained by the proposed LTSV-based scheme and the best considered VAD scheme are 88.49% and 79.30%, respectively. This improvement in the VAD accuracy indicates the robustness of the LTSV feature for VAD at low SNR condition for most of the noises considered.
Training ensemble of diverse classifiers on feature subsets

Ensembles of diverse classifiers often out-perform single classifiers as has been well-demonstrated across several applications. Existing training algorithms either learn a classifier ensemble on pre-defined feature sets or independently perform classifier training and feature selection. Neither of these schemes is optimal. We pose feature subset selection and training of diverse classifiers on selected subsets as a joint optimization problem. We propose a novel greedy algorithm to solve this problem. We sequentially learn an ensemble of classifiers where each subsequent classifier is encouraged to learn data instances misclassified by previous classifiers on a concurrently selected feature set. Our experiments on synthetic and real-world data sets show the effectiveness of our algorithm. We observe that ensembles trained by our algorithm performs better than both a single classifier and an ensemble of classifiers learnt on pre-defined feature sets. We also test our algorithm as a feature selector on a synthetic dataset to filter out irrelevant features.
Modeling Dynamics of Expressive Body Gestures In Dyadic Interactions

Body gestures are an important non-verbal expression channel during affective communication. They convey human attitudes and emotions as they dynamically unfold during an interpersonal interaction. Hence, it is highly desirable to understand the dynamics of body gestures associated with emotion expression in human interactions. We present a statistical framework for robustly modeling the dynamics of body gestures in dyadic interactions. Our framework is based on high-level semantic gesture patterns and consists of three components. First, we construct a universal background model (UBM) using Gaussian mixture modeling (GMM) to represent subject-independent gesture variability. Next, we describe each gesture sequence as a concatenation of semantic gesture patterns which are derived from a parallel HMM structure. Then, we probabilistically compare the segments of each gesture sequence extracted from the second step with the UBM obtained from the first step, in order to select highly probabilistic gesture patterns for the sequence. The dynamics of each gesture sequence are represented by a statistical variation profile computed from the selected patterns, and are further described in a well-defined kernel space. This framework is compared with three baseline models and is evaluated in emotion recognition experiments, i.e., recognizing the overall emotional state of a participant in a dyadic interaction from the gesture dynamics. The recognition performance demonstrates the superiority of the proposed framework over the baseline models. The analysis of the relationship between the emotion recognition performance and the number of the selected segments also indicates that a few local salient events, rather than the whole gesture sequence, are sufficiently informative to trigger the human summarization of their overall global emotion perception.
Interpreting ambiguous emotional expressions

Emotion expression is a complex process involving dependencies based on time, speaker, context, mood, personality, and culture. Emotion classification algorithms designed for real-world application must be able to interpret the emotional content of an utterance or dialog given the modulations resulting from these and other dependencies. Algorithmic development often rests on the assumption that the input emotions are uniformly recognized by a pool of evaluators. However, this style of consistent prototypical emotion expression often does not exist outside of a laboratory environment. This paper presents methods for interpreting the emotional content of non-prototypical utterances. These methods include modeling across multiple time-scales and modeling interaction dynamics between interlocutors. This paper recommends classifying emotions based on emotional profiles, or soft-labels, of emotion expression rather than relying on just raw acoustic features or categorical hard labels. Emotion expression is both interactive and dynamic. Consequently, to accurately recognize emotional content, these aspects must be incorporated during algorithmic design to improve classification performance.
Pathological Voice Assessment

While there are number of guidelines and methods used in practice, there is no standard universally agreed upon system for assessment of pathological voices. Pathological voices are primarily labeled based on the perceptual judgments of specialists, a process that may result in different label(s) being assigned to a given voice sample. This paper focuses on the recognition of five specific pathologies. The main goal is to compare two different classification methods. The first method considers single label classification by assigning a new label (single label) to the ensembles to which they most likely belong. The second method employs all labels originally assigned to the voice samples. Our results show that the pathological voice assessment performance in the second method is improved with respect to the first method
Statistical Modeling and Retrieval of Polyphonic Music

In this article, we propose a solution to the problem of query by example for polyphonic music audio. We first present a generic mid-level representation for audio queries. Unlike previous efforts in the literature, the proposed representation is not dependent on the different spectral characteristics of different musical instruments and the accurate location of note onsets and offsets. This is achieved by first mapping the short term frequency spectrum of consecutive audio frames to the musical space (the spiral array) and defining a tonal identity with respect to center of effect that is generated by the spectral weights of the musical notes. We then use the resulting single dimensional text representations of the audio to create a-gram statistical sequence models to track the tonal characteristics and the behavior of the pieces. After performing appropriate smoothing, we build a collection of melodic n-gram models for testing. Using perplexity-based scoring, we test the likelihood of a sequence of lexical chords (an audio query) given each model in the database collection. Initial results show that, some variations of the input piece appears in the top 5 results 81% of the time for whole melody inputs within a 500 polyphonic melody database. We also tested the retrieval engine for small audio clips. Using 25s segments, variations of the input piece are among the top 5 results 75% of the time.
Support Vector Regression for Automatic Recognition of Spontaneous Emotions in Speech

We present novel methods for estimating spontaneously expressed emotions in speech. Three continuous-valued emotion primitives are used to describe emotions, namely valence, activation, and dominance. For the estimation of these primitives, support vector machines (SVMs) are used in their application for regression (support vector regression, SVR). Feature selection and parameter optimization are studied. The data was recorded from 47 speakers in a German talk-show on TV. The results were compared to a rule-based fuzzy logic classifier and a fuzzy k-nearest neighbor classifier. SVR was found to give the best results and to be suited well for emotion estimation yielding small classification errors and high correlation between estimates and reference.
Multiple Instance Learning for Behavioral Coding

We propose a computational methodology for automatically estimating human behavioral patterns using the multiple instance learning (MIL) paradigm. We describe the incremental diverse density algorithm, a particular formulation of multiple instance learning, and discuss its suitability for behavioral coding. We use a rich multi-modal corpus comprised of chronically distressed married couples having problem-solving discussions as a case study to experimentally evaluate our approach. In the multiple instance learning framework, we treat each discussion as a collection of short-term behavioral expressions which are manifested in the acoustic, lexical, and visual channels. We experimentally demonstrate that this approach successfully learns representations that carry relevant information about the behavioral coding task. Furthermore, we employ this methodology to gain novel insights into human behavioral data, such as the local versus global nature of behavioral constructs as well as the level of ambiguity in the expression of behaviors through each respective modality. Finally, we assess the success of each modality for behavioral classification and compare schemes for multimodal fusion within the proposed framework.
Object classification in sidescan sonar images with sparse representation techniques

Most supervised classification approaches try to learn patterns in inter class variabilities using training samples. However in the real world, their discriminative power is often diminished, because data is seldom free from irregularities within a class. Apriori modeling of these intra class variabilities poses a challenge even in underwater sidescan sonar images that we consider for object classification in this work. Sparse representation techniques prove particularly useful in this regard because of their data driven approach to model these variabilities. Results on the NSWC sidescan sonar database suggest that sparse representation classifier with zernike magnitude features is significantly robust in the presence of these non-idealities.