nils-herrmann/clean_scirepeval huggingface dataset:
Acoustic modeling and feature selection for speech recognition

The standard hidden Markov model (HMM) has been proved to be the most successful model for speech recognition. A most widely addressed problem of the HMM is the assumption of independent observations given the state sequence. In the past few years, a wide range of state-space models and graphical models, such as segmental models and switching linear dynamical systems, have been applied to the speech recognition task. An underlying difficulty of those proposed systems is the tradeoff of computational complexity and representation capability. This thesis presents some results and explorations which indicate that recognition performance can be improved by incorporating acoustic phonetic prior information into a nonlinear state-space model and by incorporating more discriminative measurements to a support vector machine (SVM) and HMM combined system. 
The investigation of the thesis can be divided into three parts. In the first part, a nonlinear dynamic system is proposed for formant tracking. Compared to previous formant trackers depending on least squares estimation of LPC coefficients, MUSIC (Multiple Signal Classification) and ESPRIT (Estimation of Signal Parameters via Rotational Invariance Techniques) are used to improve the accuracy of formant estimation. Furthermore, a mixture of nonlinear dynamic systems is developed to improve the performance of formant tracking. In the second part, the formant tracker system is extended to perform phoneme recognition. The results indicate that the incapability of estimating the system measurement error prevents the system from performing well in the phoneme recognition tasks. In the third part, an SVM and HMM combined system is used to prove that the formant information is indeed useful to distinguish different phonemes. And the result in this part suggests that the output of the SVM can be treated as a particular case of discriminant transformation of the original acoustic space and might be useful for speech recognition.
Prosodic parallelism as a cue to repetition disfluency

Repetition disfluencies are among the most frequent type of disfluency in conversational speech, accounting for over 20% of disfluencies, yet they do not generally lead to comprehension errors for human listeners. We propose that parallel prosodic features in the REP and ALT intervals of the repetition disfluency provide strong perceptual cues that signal the repetition to the listener. We report results from a transcription analysis of repetition disfluencies that classifies disfluent regions on the basis of prosodic factors, and preliminary evidence from F0 analysis to support our finding of prosodic parallelism. 1. Acoustic-prosodic correlates of disfluency Disfluency occurs in spontaneous speech at a rate of about one every 10-20 words, or 6% per word count [17], yet this interruption of fluent speech does not generally lead to comprehension errors for human listeners. Recent research has shown that important cues to disfluency can be found in the syntactic and semantic structures conveyed by the word sequence, and in the phonological and phonetic structures signaled by acoustic features local to the disfluency interval. These cues identify the components of the disfluent regionthe reparandum (REP), edit phrase (EDIT), and alteration (ALT)--- and their junctures. Work on automatic disfluency detection has shown that the most successful approach combines both lexical and acoustic features, with explicit models of the lexical-syntactic and prosodic features that pattern systematically with disfluent intervals [1,6].
Acoustic correlates of non‐modal phonation in telephone speech

Non‐modal phonation conveys both linguistic and paralinguistic information, and is distinguished by acoustic source and filter features. Detecting non‐modal phonation in speech requires reliable F0 analysis, a problem for telephone‐band speech, where F0 analysis frequently fails. An approach is demonstrated to the detection of creaky phonation in telephone speech based on robust F0 and spectral analysis. The F0 analysis relies on an autocorrelation algorithm applied to the inverse‐filtered speech signal and succeeds in regions of non‐modal phonation where the non‐filtered F0 analysis typically fails. In addition to the extracted F0 values, spectral amplitude is measured at the first two harmonics (H1, H2) and the first three formants (A1, A2, A3). F0 and spectral tilt are measured from 300 samples of modal and creaky voice vowels, selected from Switchboard telephone speech using auditory and visual criteria. Results show successful F0 detection in creaky voice regions, with distinctive low F0, and statist...
How Prosody Improves Word Recognition

Prosody has been traditionally regarded as useless for word recognition. In this paper, we provide a schematic view describing how prosody can help word recognition. We provide our view in terms of a Bayesian network that models the stochastic dependence among acoustic observation, word, prosody, syntax and meaning, and an information-theoretic analysis proving that the mutual information between acoustic observation and correct word hypotheses improves if prosody is jointly modeled with word in a prosody dependent speech recognition framework. We also report our experiment on Radio News Corpus in which prosody has improved word recognition accuracy by 2.5%.
Intertranscriber reliability of prosodic labeling on telephone conversation using toBI

Two transcribers have labeled prosodic events independently on a subset of Switchboard corpus using adapted ToBI (TOnes and Break Indices) system. Transcriptions of two types of pitch accents (H* and L*), phrasal accents (Hand L-) and boundary tones (H% and L%) encoded independently by two transcribers are compared for intertranscriber reliability. Two commonly used methods of reliability measurement, ‘transcriber-pair-word’ comparison and kappa statistic, are used for comparison with previous reports on the intertranscriber consistency. The results obtained from transcriber-pair-word comparison are: The overall agreement on the presence or absence and choice of pitch accent is 86.57%. The agreement on the presence or absence and the choice of phrasal accent is 85.63%. The presence and choice of boundary tone is 89.33%. When both transcribers agreed that there is at least a phrasal tone, the agreement on the choice of the type of either phrasal accent or boundary tone is 73.86%. The kappa coefficient of agreement (K) of 0.7 to 1 indicates the degree of reliability to be from good to perfect. A kappa coefficient of 0.75 is obtained for agreement on the presence or absence of pitch accents, 0.67 for the presence of phrasal accents, and 0.61 for the strength of disjuncture between phrasal accent and boundary tone. Comparison of the present results with those of previous reliability studies [1][2][3][4] suggests that some higher agreement rates for this study may result from our adoption of fewer labeling distinctions in the transcription of pitch accent events. The results for phrase boundary labeling suggest that spontaneous speech of the type found in the Switchboard corpus is harder to code for the degree of disjuncture between prosodic domains than is read speech.
Stop consonant classification by dynamic formant trajectory

A cable reel assembly particularly adapted for, but not limited to, a system for providing electrical connection of power and data signals between an orbiter vehicle (12), such as a space shuttle, and a recovered satellite (14) and being comprised of two mutually opposing ring type structures (18, 20) having +/-180 DEG relative rotation with one of the structures being held in fixed position while the other structure is rotatable. Motor controlled berthing latches (161, 162, 163) and umbilical cable connectors (171, 172) for the satellite are located on the rim (26) of the rotatable ring structure (18). The electrical cable assembly (56) is fed in two sections (58, 60) from the orbiter vehicle into the outer rim portion (40) of the fixed ring structure (20) where they are directed inwardly and attached to two concentrically coiled metal bands (67, 68) whose respective ends are secured to inner and outer post members of circular sets of guide pins (70, 72) located on opposing circular plate members (28, 42), one rotatable and one fixed, forming part of and located along a central axis (44) of the two structures. The cable sections (58, 60) are fed out as three output cable sections (62, 64, 66) through openings (63,65) in the central portion of the circular plate (28) of the rotatable ring structure (18) where they are directed to the latches (161, 162, 163) and connectors (171, 172) located on its rim (26).
A factorial HMM approach to simultaneous recognition of isolated digits spoken by multiple talkers on one audio channel

This paper addresses the novel problem of recognizing digits spoken simultaneously by two different talkers. A factorial hidden Markov model architecture is proposed to accurately model the simultaneous utterance of two digits. Nadas' (1999) MIXMAX approximation is extended to a mixture of Gaussians observation PDF which enables the implementation of the proposed system. The multiple digit recognizer is found to successfully recognize pairs of simultaneous utterances of digits at 0db SNR with up to 89% accuracy.
A Maximum Likelihood Prosody Recognizer

Automatic prosody recognition (APR) is of fundamental importance for automatic speech understanding. In this paper, we propose a maximum likelihood prosody recognizer consisting of a GMM-based acoustic model that models the distribution of the phone-level acoustic-prosodic observations (pitch, duration and energy) and an ANN-based language model that models the word-level stochastic dependence between prosody and syntax. Our experiments on the Radio News Corpus show that our recognizer is able to achieve 84% pitch accent recognition accuracy and 93% intonational phrase boundary (IPB) recognition accuracy in a leave-one-speaker-out task which has exceeded previous reported results on the same corpus. The same recognizer is tested on a subset of Switchboard corpus. The accuracies are degraded but still significantly better than the chance levels.
Children's emotion recognition in an intelligent tutoring scenario

This paper presents an approach to automatically recognize emotion which children exhibit in an intelligent tutoring system. Emotion recognition can assist the computer agent to adapt its tutorial strategies to improve the efficiency of knowledge transmission. In this study, we detect three emotional classes: confidence, puzzle, and hesitation. Emotion is detected by means of lexical, prosodic, spectral, and syntactic analyses of users’ speech. An automatic speech recognition system serves as the fundamental constituent of the system. A robust classification and regression tree (CART) integrates the various information sources together for final decision. The effectiveness of the proposed approach has been tested on data collected by Wizard-of-Oz (WoZ) experiments. Our emotion recognition was speaker-independent, and yielded 91.3% accuracy. The test results showed that the spectral and duration-related prosodic features played very important roles in emotion recognition.
Automatic recognition of pitch movements using multilayer perceptron and time-Delay Recursive neural network

This letter demonstrates hidden Markov model (HMM), multilayer perceptron (MLP), and time-delay recursive neural network (TDRNN) architectures for the purpose of recognizing pitch accents given observation of the F0 and energy trajectories. At an insertion error rate of 25%, the deletion error rates of the MLP, TDRNN, and HMM are 13.2%, 7.9%, and 32.7%, respectively, despite the fact that both MLP and TDRNN have 70% fewer trainable parameters than the HMM. Error analysis suggests that low-pitch accents may require long-term context to correctly recognize, while high-pitch accents may be recognizable based on local pitch contour.
Modeling pronunciation variation using artificial neural networks for English spontaneous speech

Pronunciation variation in conversational speech has caused significant amount of word errors in large vocabulary automatic speech recognition. Rule-based approaches and decision-tree based approaches have been previously proposed to model pronunciation variation. In this paper, we report our work on modeling pronunciation variation using artificial neural networks (ANN). The results we achieved are significantly better than previously published ones on two different corpora, indicating that ANN may be better suited for modeling pronunciation variation than other statistical models that have been previously investigated. Our experiments indicate that binary distinctive features can be used to effectively represent the phonological context. We also find that including pitch accent feature in input improves the prediction of pronunciation variation on a ToBI-labeled subset of the Switchboard corpus.
Semantic analysis for a speech user interface in an intelligent tutoring system

In this paper, we describe the strategy of semantic analysis for a speech user interface that is designed for a multimodal intelligent tutoring system. The semantic analysis involves three phases: semantic parsing, salient words/phrases spotting, and accented word detection. Semantic parsing attempts to represent the recognized sentence with a well-formed semantic frame. The recognized sentence consists of the a posterior most probably hypothesized words given the acoustic evidence, and is compliant with the grammatical knowledge that is represented by a semantic language model. The salient words/phrases are useful when semantic parsing fails. The accented words are useful when the user response is out of our expectations, and assist to make the computer agent smarter and smarter.
Prosody dependent speech recognition on american radio news speech

Prosody (the melody and rhythm of natural speech), although important for human speech recognition, has not been fully utilized in large vocabulary continuous speech recognition. In this dissertation, we propose a novel “prosody-dependent speech recognition” framework, in which word and prosody are recognized simultaneously for the purpose of improving word recognition accuracy. We review the linguistic literature on how prosody is used in human speech communication, what the structure and function of prosody is, and how prosody affects acoustic realization of the segmental and suprasegmental units of speech. We conduct information-theoretic analysis, proving that when prosody is modeled, better word recognition can be achieved through the interaction between the acoustic model and the language model. We conduct detailed experiments to determine the set of allophonic HMMs or probability distributions that are sensitive to prosody, under the guidance of linguistic prior knowledge and empirical selection rules. We measure the effects of prosody on the language modeling and on the pronunciation modeling, and propose a factored approach, which leverages the strong dependence of prosody over syntax, to improve the robustness of the N-gram language modeling. We also develop an automatic prosody labeling system as a way to reduce the human labeling cost based on a simplified version of the Tones and Break Indices system. In the word recognition experiment on the Boston University Radio News Corpus, we find that our system is able to reduce word error rate by as much as an absolute 11%, as compared with a conventional prosody-independent HMM-based automatic speech recognizer that has comparable parameter count. Our research confirms that explicit modeling of prosody in HMM based automatic speech recognizers can improve word recognition on American Radio News speech.
Acoustic Differentiation of ip and IP Boundary Levels: Comparison of L- and L-L% in the Switchboard Corpus

Prosodic phrase boundaries, regardless of level of disjuncture, can be signaled by variation in pitch, loudness, and finalsyllable length. In an attempt to find acoustically distinctive characteristics correlated with ip (intermediate phrase) versus IP (intonation phrase) labels in a ToBI-labeled subset of the Switchboard corpus, we compared F0 drop, intensity drop, and nucleus duration in the phrase-final rime for L- and L-L% boundary labels. The results indicate no significant difference in F0 or intensity drop, but final-syllable lengthening as measured by nucleus duration differed significantly between the two boundary levels. Additionally, F0 aperiodicity associated with creaky voice was found to occur more frequently at L-L% than at L- boundaries. These results provide empirical corroboration of the statement that F0 does not reliably differentiate L- from L-L% , and support previous findings that degree of finalsyllable lengthening  and presence of creaky phonation [10] are correlated with differences in perceived level of phrasal disjuncture.
An automatic prosody labeling system using ANN-based syntactic-prosodic model and GMM-based acoustic-prosodic model

Automatic prosody labeling is important for both speech synthesis and automatic speech understanding. Humans use both syntactic cues and acoustic cues to develop their prediction of prosody for a given utterance. This process can be effectively modeled by an ANN-based syntactic-prosodic model that predicts prosody from syntax and a GMM-based acoustic-prosodic model that predicts prosody from acoustic-prosodic observations. Our experiments on the Radio News Corpus show that ANN is effective in learning the stochastic mapping from the syntactic representation of word strings to prosody labels, with an accuracy of 82.7% for pitch accent labeling and 90.5% for intonational phrase boundary (IPB) labeling. When acoustic observations and reasonably accurate phoneme transcriptions are given, a GMM-based acoustic-prosodic model, coupled with the syntactical-prosodic model, can achieve 84% pitch accent recognition accuracy and 93% IPB recognition accuracy. These results are obtained using different speakers for training and testing and have considerably exceeded all previously reported results on the same corpus, especially for the task of IPB detection.
Model enforcement: a unified feature transformation framework for classification and recognition

Bayesian classifiers rely on models of the a priori and class-conditional feature distributions; the classifier is trained by optimizing these models to best represent features observed in a training corpus according to certain criterion. In many problems of interest, the true class-conditional feature probability density function (PDF) is not a member of the set of PDFs the classifier can represent. Previous research has shown that the effect of this problem may be reduced either by improving the models or by transforming the features used in the classifier. This paper addresses this model mismatch problem in statistical identification, classification, and recognition systems. We formulate the problem as the problem of minimizing the relative entropy, which is also known as the Kullback-Leibler distance, between the true conditional PDF and the hypothesized probabilistic model. Based on this formulation, we provide a computationally efficient solution to the problem based on volume-preserving maps; existing linear transform designs are shown to be special cases of the proposed solution. Using this result, we propose the symplectic maximum likelihood transform (SMLT), which is a nonlinear volume-preserving extension of the maximum likelihood linear transform (MLLT). This approach has many applications in statistical modeling, classification, and recognition. We apply it to the maximum likelihood estimation (MLE) of the joint PDF of order statistics and show a significant increase in the likelihood for the same number of parameters. We provide also phoneme recognition experiments that show recognition accuracy improvement compared with using the baseline Mel-Frequency Cepstrum Coefficient (MFCC) features or using MLLT. We present an iterative algorithm to jointly estimate the parameters of the symplectic map and the probabilistic model for both applications.
Source separation using particle filters

Our goal is to study the statistical methods for source separation based on temporal and frequency specific features by using particle filtering. Particle filtering is an advanced state-space Bayesian estimation technique that supports non-Gaussian and nonlinear models along with time-varying noise, allowing for a more accurate model of the underlying system dynamics. We present a system that combines standard speech processing techniques in a novel method to separate two noisy speech sources. The system models the pitch and amplitude over time separately, and adopts particle filtering to reduce complexity by generating a discrete distribution that approximates well the desired continuous distribution. Preliminary results that demonstrate the separation of two noisy sources using this system are presented.
RECOGNITION OF PROSODIC FACTORS AND DETECTION OF LANDMARKS FOR IMPROVEMENTS TO CONTINUOUS SPEECH RECOGNITION SYSTEMS

This thesis examines the usefulness of including prosodic and phonetic context information in the phoneme model of a speech recognizer. This is done creating a series of prosodic and phonetic models and then comparing the log likelihoods of each model. The comparison of log likelihoods shows that both prosodic and phonetic context information improve the phoneme model for most phonemes. The prosodic and phonetic context information is then modeled in two separate recognizers to show that while phonetic context modeling does provide some useful information for the recognizer, prosody provides much more. This thesis also experiments with the detection of landmarks, the transitions between two phonemes, using binary classification. The experiments presented show that landmarks can be detected with high accuracy and are useful for phone classification.  Humans can interpret and understand spoken language with relative ease. Computers, however , struggle with the task of recognizing continuous speech. Part of the problem is that speech recognizers implemented on computers are not robust to environmental and acoustic variations. For example, a speech recognizer trained under quiet conditions fails miserably when any amount of background noise is added. A speech recognizer trained on a specific person's voice will not perform as well for a different talker. Even the choice of microphone affects the performance of a recognizer. The accuracy of a recognizer trained on speech recorded with one microphone will decrease if the original microphone is replaced with a different one. Another problem is that speech itself is variable. No word or sound is produced in exactly the same way twice. Another factor that plays into the poor performance of speech recognizers is that natural human conversational speech contains thousands of words; many of which have similar pronunciations. When a speech recognizer is trained and tested on a database containing only a few isolated, discrete words, recognition rates are high. Liu et al. , Samouelian [2], and Li-Peng et al.  each use different methods to implement isolated word recognition systems and each report that their different recognizers can identify over 90% of the words correctly. However, the word error rate (WER) increases significantly when attempts are made to recognize continuous conversational speech. Liu et al.  and Eide et al.  each report WER's of 50% or more when performing recognition experiments on corpora of 1 conversational telephone speech. The research presented in this thesis attempts to address many of …