nils-herrmann/clean_scirepeval huggingface dataset:
A study of generic models for unsupervised on-line speaker indexing

On-line speaker indexing sequentially detects the points where a speaker identity changes in a multi-speaker audio stream, and classifies each speaker segment. The paper addresses two challenges. The first relates to monitoring, which requires on-line processing. The second relates to the fact that the number/identity of the speakers is unknown. The indexing needs to be made in an unsupervised process. To address these issues, we apply a predetermined generic speaker-independent model set, sample speaker model (SSM). This set can be useful for more accurate speaker modeling and clustering without requiring training models on target speaker data. Once a speaker-independent model is selected from the sample models, it is adapted into a speaker-dependent model progressively. Experiments were performed with the speaker recognition benchmark NIST Speech (1999). Results showed that our new technique, simulated using the Markov chain Monte Carlo method, gave 92.47% indexing accuracy on telephone conversation data.
Classification of emotional content of sighs in dyadic human interactions

Emotions are an important part of human communication and are expressed both verbally and non-verbally. Common nonverbal vocalizations such as laughter, cries and sighs carry important emotional content in conversations. Sighs often are associated with negative emotion. In this work, we show that emotional sighs exist along both ends of the valence axis (positive-emotion vs. negative-emotion sighs) in spontaneous affective dialogs and that they have certain distinct multimodal characteristics. Classification results show that it is possible to differentiate between the two types of emotionally valenced sighs, using a combination of acoustic and gestural features with an overall unweighted accuracy of 58.26%.
Spatial and temporal alignment of multimodal human speech production data: Real time imaging, flesh point tracking and audio

In speech production research, the integration of articulatory data derived from multiple measurement modalities can provide rich description of vocal tract dynamics by overcoming the limited spatio-temporal representations offered by individual modalities. This paper presents a spatial and temporal alignment method between two promising modalities using a corpus of TIMIT sentences obtained from the same speaker: flesh point tracking from Electromagnetic Articulography (EMA) that offers high temporal resolution but sparse spatial information and real time Magnetic Resonance Imaging (MRI) that offers good spatial details but at lower temporal rates. Spatial alignment is done by using palate tracking of EMA, but distortion in MRI audio and articulatory data variability make temporal alignment challenging. This paper proposes a novel alignment technique using joint acoustic-articulatory features which combines dynamic time warping and automatic feature extraction from MRI images. Experimental results show that the temporal alignment obtained using this technique is better (12% relative) than that using acoustic feature only.
A Computational Study of Expressive Facial Dynamics in Children with Autism

Several studies have established that facial expressions of children with autism are often perceived as atypical, awkward or less engaging by typical adult observers. Despite this clear deficit in the quality of facial expression production, very little is understood about its underlying mechanisms and characteristics. This paper takes a computational approach to studying details of facial expressions of children with high functioning autism (HFA). The objective is to uncover those characteristics of facial expressions, notably distinct from those in typically developing children, and which are otherwise difficult to detect by visual inspection. We use motion capture data obtained from subjects with HFA and typically developing subjects while they produced various facial expressions. This data is analyzed to investigate how the overall and local facial dynamics of children with HFA differ from their typically developing peers. Our major observations include reduced complexity in the dynamic facial behavior of the HFA group arising primarily from the eye region.
Rapid Language Identification

A critical challenge to automatic language identification (LID) is achieving accurate performance with the shortest possible speech segment in a rapid fashion. The accuracy to correctly identify the spoken language is highly sensitive to the duration of speech and is bounded by the amount of information available. The proposed approach for rapid language identification transforms the utterances to a low dimensional i-vector representation upon which language classification methods are applied. In order to meet the challenges involved in rapidly making reliable decisions about the spoken language, a highly accurate and computationally efficient framework of i-vector extraction is proposed. The LID framework integrates the approach of universal background model (UBM) fused total variability modeling. UBM-fused modeling yields the estimation of a more discriminant, single i-vector space. This way, it is also a computationally more efficient alternative than system level fusion. A further reduction in equal error rate is achieved by training the i-vector model on long duration speech utterances and by the deployment of a robust feature extraction scheme that aims to capture the relevant language cues under various acoustic conditions. Evaluation results on the DARPA RATS data corpus suggest the potential of performing successful automated language identification at the level of one second of speech or even shorter duration.
Real-time Emotion Detection System using Speech: Multi-modal Fusion of Different Timescale Features

The goal of this work is to build a real-time emotion detection system which utilizes multi-modal fusion of different timescale features of speech. Conventional spectral and prosody features are used for intra-frame and supra-frame features respectively, and a new information fusion algorithm which takes care of the characteristics of each machine learning algorithm is introduced. In this framework, the proposed system can be associated with additional features, such as lexical or discourse information, in later steps. To verify the realtime system performance, binary decision tasks on angry and neutral emotion are performed using concatenated speech signal simulating realtime conditions.
Robust speaker clustering strategies to data source variation for improved speaker diarization

Agglomerative hierarchical clustering (AHC) has been widely used in speaker diarization systems to classify speech segments in a given data source by speaker identity, but is known to be not robust to data source variation. In this paper, we identify one of the key potential sources of this variability that negatively affects clustering error rate (CER), namely short speech segments, and propose three solutions to tackle this issue. Through experiments on various meeting conversation excerpts, the proposed methods are shown to outperform simple AHC in terms of relative CER improvements in the range of 17-32%.
A knowledge transfer and boosting approach to the prediction of affect in movies

Affect prediction is a classical problem and has recently garnered special interest in multimedia applications. Affect prediction in movies is one such domain, potentially aiding the design as well as the impact analysis of movies. Given the large diversity in movies (such as different genres and languages), obtaining a comprehensive movie dataset for modeling affect is challenging while models trained on smaller datasets may not generalize. In this paper, we address the problem of continuous affect ratings with the availability of limited in-domain data resources. We initially setup several baseline models trained on in-domain data, followed by a proposal of a Knowledge Transfer (KT) + Gradient Boosting (GB) approach. KT learns models on a larger (mismatched) data which are then adapted to make predictions on the data of interest. GB further updates these predictions based on models learnt from the in-domain data. We observe that the KT + GB models provide Concordance Correlation Coefficient values of 0.13 and 0.27 for valence and affect prediction on the continuous LIRIS ACCEDE dataset against best baseline prediction values of 0.12 and 0.11. Not only the KT + GB models improve the overall performance metrics, we also observe a more consistent model performance across movies of various genres.
Integration and Automation of Data Preparation and Data Mining

Data mining tasks typically require significant effort in data preparation to find, transform, integrate and prepare the data for the relevant data mining tools. In addition, the work performed in data preparation is often not recorded and is difficult to reproduce from the raw data. In this paper we present an integrated approach to data preparation and data mining that combines the two steps into a single integrated process and maintains detailed metadata about the data sources, the steps in the process, and the resulting learned classifier produced from data mining algorithms. We present results on an example scenario, which shows that our approach provides significant reduction in the time in takes to perform a data mining task.
Liquids in Tamil

Tamil is unusual among the world's languages in that some of its dialects have five liquids. The paper focuses on the articulatory characterization of these sounds, with the ultimate goal of modeling their production dynamics and articulatory acoustic mappings. Articulatory data were obtained using different techniques: palatography, magnetic resonance imaging (MRI), and magnetometer (EMMA). The study illustrates the use of multiple techniques for investigating both static and dynamic articulatory characteristics.
Optimal Arousal Identification and Classification for Affective Computing Using Physiological Signals: Virtual Reality Stroop Task

A closed-loop system that offers real-time assessment and manipulation of a user's affective and cognitive states is very useful in developing adaptive environments which respond in a rational and strategic fashion to real-time changes in user affect, cognition, and motivation. The goal is to progress the user from suboptimal cognitive and affective states toward an optimal state that enhances user performance. In order to achieve this, there is need for assessment of both 1) the optimal affective/cognitive state and 2) the observed user state. This paper presents approaches for assessing these two states. Arousal, an important dimension of affect, is focused upon because of its close relation to a user's cognitive performance, as indicated by the Yerkes-Dodson Law. Herein, we make use of a Virtual Reality Stroop Task (VRST) from the Virtual Reality Cognitive Performance Assessment Test (VRCPAT) to identify the optimal arousal level that can serve as the affective/cognitive state goal. Three stimuli presentations (with distinct arousal levels) in the VRST are selected. We demonstrate that when reaction time is used as the performance measure, one of the three stimuli presentations can elicit the optimal level of arousal for most subjects. Further, results suggest that high classification rates can be achieved when a support vector machine is used to classify the psychophysiological responses (skin conductance level, respiration, ECG, and EEG) in these three stimuli presentations into three arousal levels. This research reflects progress toward the implementation of a closed-loop affective computing system.
Speaker identification using supra-segmental pitch pattern dynamics

Most conventional speaker identification systems rely on short-time spectral envelope features. Recent efforts have yielded significant progress by capturing and modeling speaker-specific aspects of long-term information in the spoken language signal such as prosodic, syntactic and other conversational features. Although significant results have been reported, substantial improvements can be made by using detailed models better describing the specific behavior of each feature. We focus on modeling pitch pattern dynamics at different prosodic scale levels. Trends in pitch variation are believed to appear at different time-scales - such as microprosody, accent, phrase and discourse levels - making wavelet analysis of the f0 contour a suitable choice for investigating the corresponding pitch patterns. We then introduce a transform of the f0 contour wavelet coefficients that results in a compact representation and better reveals the spatio-temporal details in the coefficient sequences representation. In turn, the dynamics of the transformed sequence are modeled by a first order Markov chain, at each scale level. Classification is carried out at each level and the scores of the classifiers operating at the different supra-segmental levels are fused together. The proposed method achieves an EER of 4.8% on the NIST 2001 speaker ID extended data task using a 16-conversation subset, based solely on f0-based information.
Multimodal Meeting Monitoring: Improvements on Speaker Tracking and Segmentation through a Modified Mixture Particle Filter

In this paper we address improvements to our multimodal system for tracking of meeting participants and speaker segmentation with a focus on the microphone array modality. We propose an algorithm that uses Directions-of-Arrival estimated for each microphone pair as observations and performs tracking of an unknown number of acoustically-active meeting participants and subsequent speaker segmentation. We propose modified mixture particle fillter (mMPF) for tracking of acoustic sources in the track-before-detection (TbD) framework. Trajectories of sound sources are reconstructed by the optimal assignment of posterior mixture components produced by mMPF in consecutive frames. Further, we propose a sequential optimal change-point detection algorithm which discovers speech segments in the reconstructed trajectories i.e., performs speaker segmentation. The algorithm is tested on a multi-participant meeting dataset both separately and as a part of the multimodal system. On the task of speaker detection in the multimodal setup we report significant improvement over our previous state of the art implementation.
Speech emotion estimation in 3D space

Speech processing is an important element of affective computing. Most research in this direction has focused on classifying emotions into a small number of categories. However, numerical representations of emotions in a multi-dimensional space can be more appropriate to reflect the gradient nature of emotion expressions, and can be more convenient in the sense of dealing with a small set of emotion primitives. This paper presents three approaches (robust regression, support vector regression, and locally linear reconstruction) for emotion primitives estimation in 3D space (valence/activation/dominance), and two approaches (average fusion and locally weighted fusion) to fuse the three elementary estimators for better overall recognition accuracy. The three elementary estimators are diverse and complementary because they cover both linear and nonlinear models, and both global and local models. These five approaches are compared with the state-of-the-art estimator on the same spontaneously elicited emotion dataset. Our results show that all of our three elementary estimators are suitable for speech emotion estimation. Moreover, it is possible to boost the estimation performance by fusing them properly since they appear to leverage complementary speech features.
Enhanced Sparse Imputation Techniques for a Robust Speech Recognition Front-End

Missing data techniques (MDTs) have been widely employed and shown to improve speech recognition results under noisy conditions. This paper presents a new technique which improves upon previously proposed sparse imputation techniques relying on the least absolute shrinkage and selection operator (LASSO). LASSO is widely employed in compressive sensing problems. However, the problem with LASSO is that it does not satisfy oracle properties in the event of a highly collinear dictionary, which happens with features extracted from most speech corpora. When we say that a variable selection procedure satisfies the oracle properties, we mean that it enjoys the same performance as though the underlying true model is known. Through experiments on the Aurora 2.0 noisy spoken digits database, we demonstrate that the Least Angle Regression implementation of the Elastic Net (LARS-EN) algorithm is able to better exploit the properties of a collinear dictionary, and thus is significantly more robust in terms of basis selection when compared to LASSO on the continuous digit recognition task with estimated mask. In addition, we investigate the effects and benefits of a good measure of sparsity on speech recognition rates. In particular, we demonstrate that a good measure of sparsity greatly improves speech recognition rates, and that the LARS modification of LASSO and LARS-EN can be terminated early to achieve improved recognition results, even though the estimation error is increased.
An audio-visual approach to learning salient behaviors in couples' problem solving discussions

We present a method for characterizing salient behavioral events from audio-visual data of dyadic human interactions. This behavioral signal processing work is aimed at supporting observational analysis of domain experts such as psychologists and clinicians. We extract prosodic and spectral speech features as well as visual motion vector features on overlapping windows from a multimodal corpus. We then apply a technique called multiple instance learning to detect salient audio and visual instances for predicting human expert annotated behavior ratings. We demonstrate the performance gains achieved through multimodal fusion in characterizing complex behavior patterns of interest such as blame and acceptance in recordings of couples' problem solving discussions during marital therapy.
A near-optimal (minimax) tree-structured partition for mutual information estimation

A novel histogram-based mutual information estimator using data-driven tree-structured partitions (TSP) is presented in this work. The TSP is the solution of a complexity regularized empirical information maximization (EIM) criterion, with the objective to find a good tradeoff between the known estimation and approximation errors. We show that this solution is density-free strongly consistent and, furthermore, it provides a near-optimal balance between the mentioned variance-bias errors.
Experiments in Automatic Genre Classification of Full-length Music Tracks using Audio Activity Rate

The activity rate of an audio clip in terms of three defined attributes results in a generic, quantitative measure of various acoustic sources present in it. The objective of this work is to verify if the acoustic structure measured in terms of these three attributes can be used for genre classification of music tracks. For this, we experiment on classification of full-length music tracks by using a dynamic time warping approach for time-series similarity (derived from the activity rate measure) and also a Hidden Markov Model based classifier. The performance of directly using timbral (Mel-frequency Cepstral Coefficients) features is also presented. Using only the activity rate measure we obtain classification performance that is about 35% better than baseline chance and this compares well with other proposed systems that use musical information such as beat histogram or pitch based melody information.
Multiresolution spectral conversion for multichannel audio resynthesis

Multichannel audio is attracting rapidly increasing popularity in audio reproduction. In most cases, however, its transmission requirements are extremely demanding compared to the available bandwidth. One possible solution to this problem could be to transmit a reference channel and recreate the remaining channels at the receiving end. Such a method is proposed by taking advantage of spectral conversion techniques that have been successfully applied to speech processing. Applications of the proposed system include transmission of multichannel audio over the current Internet infrastructure and, as an extension of the methods proposed here, remastering of existing monophonic and stereophonic recordings for multichannel rendering.
Smooth Gmm Based Multi-Talker Spectral Conversion for Spectrally Degraded Speech

Because of the limited spectro-temporal resolution associated with the implant device, cochlear implant (CI) patients are more susceptible to talker variability than normal hearing (NH) listeners. In the present study, the effect of a smooth GMM based spectral conversion algorithm on multi-talker sentence recognition was tested in CI patients. In a model of CI speech processing (4-16 channels of spectrally degraded speech), talker distortion was significantly reduced with relatively few (~64) GMM components. CI patients' sentence recognition was measured for one male (M1) and one female (F1) talker, as well as for spectrally converted speech (from M1 to F1 and from F1 to M1). Overall, CI users were sensitive to talker differences; some subjects performed better with M1, others with F1. After converting the spectrum of the less-understood talker to that of the better-understood talker, recognition of the less-understood talker's speech was significantly improved. The results suggest that smooth GMM-based spectral conversion may improve CI patients' multi-talker speech recognition
Using naïve text queries for robust audio information retrieval

The goal of this work is to build an audio information retrieval system which provides users with flexibility in formulating their queries: from audio examples to naïve text. Specifically, the focus of this paper is on using naïve text to create input queries describing the desired information of the users. Using naïve text queries, however, raises interoperability issues between annotation and retrieval processes due to the wide variety of available audio descriptions. In this paper, we propose an intermediate audio description layer (iADL) to solve the interoperability issues between the annotation and retrieval processes. The iADL comprises two axes corresponding to semantic and onomatopoeic descriptions based on human-to-human communication experiments on how humans express sounds verbally. Various text modeling schemes, such as latent semantic analysis (LSA) and latent topic model, are utilized to transform the naïve text onto the proposd iADL.
Affective language model adaptation via corpus selection

Motivated by methods used in language modeling and grammar induction, we propose the use of pragmatic constraints and perplexity as criteria to filter the unlabeled data used to generate the semantic similarity model. We investigate unsupervised adaptation algorithms of the semantic-affective models proposed in [1, 2]. Affective ratings at the utterance level are generated based on an emotional lexicon, which in turn is created using a semantic (similarity) model estimated over raw, unlabeled text. The proposed adaptation method creates task-dependent semantic similarity models and task-dependent word/term affective ratings. The proposed adaptation algorithms are tested on anger/distress detection of transcribed speech data and sentiment analysis in tweets showing significant relative classification error reduction of up to 10%.