nils-herrmann/clean_scirepeval huggingface dataset:
The Necessity of Parsing for Predicate Argument Recognition

Broad-coverage corpora annotated with semantic role, or argument structure, information are becoming available for the first time. Statistical systems have been trained to automatically label semantic roles from the output of statistical parsers on unannotated text. In this paper, we quantify the effect of parser accuracy on these systems' performance, and examine the question of whether a flatter "chunked" representation of the input can be as effective for the purposes of semantic role identification.
Automatic Labeling of Semantic Roles

We present a system for identifying the semantic relationships, or semantic roles, filled by constituents of a sentence within a semantic frame. Given an input sentence and a target word and frame, the system labels constituents with either abstract semantic roles, such as Agent or Patient, or more domain-specific semantic roles, such as Speaker, Message, and Topic. The system is based on statistical classifiers trained on roughly 50,000 sentences that were hand-annotated with semantic roles by the FrameNet semantic labeling project. We then parsed each training sentence into a syntactic tree and extracted various lexical and syntactic features, including the phrase type of each constituent, its grammatical function, and its position in the sentence. These features were combined with knowledge of the predicate verb, noun, or adjective, as well as information such as the prior probabilities of various combinations of semantic roles. We used various lexical clustering algorithms to generalize across possible fillers of roles. Test sentences were parsed, were annotated with these features, and were then passed through the classifiers. Our system achieves 82 accuracy in identifying the semantic role of presegmented constituents. At the more difficult task of simultaneously segmenting constituents and identifying their semantic role, the system achieved 65 precision and 61 recall. Our study also allowed us to compare the usefulness of different features and feature combination methods in the semantic role labeling task. We also explore the integration of role labeling with statistical syntactic parsing and attempt to generalize to predicates unseen in the training data.
Form variation of English function words in conversation

Function words (the, that, and, of, . . . ) vary widely in pronunciation. Understanding this variation is essential both for cognitive modeling of lexical production and for computer speech recognition and synthesis. This study investigates which factors affect the forms of function words, especially whether they have a fuller pronunciation (e.g., , , , ) or a more reduced or lenited pronunciation (e.g., , , , ). It is based on over 8000 occurrences of ten frequent English function words in a four-hour sample from conversations from the Switchboard corpus. Ordinary linear and logistic regression models were used to examine variation in the length of the words, in the form of their vowel (basic, full, or reduced), and whether final obstruents were present or not. For all these measures, after controlling for segmental context, rate of speech, and other important factors, there are strong independent effects that made function words more likely to be longer or have a fuller form (1) when neighboring disfluencies (such as filled pauses uh and um) indicate that the speaker was encountering problems in planning the utterance; (2) when the word is unexpected, i.e less predictable in context; (3) when the word is either utterance-initial or utterance-final. Looking at the phenomenon in a different way, function words are more likely to be shorter and to have less full forms in fluent speech, in predictable positions or multi-word collocations, and utterance-internally. Also considered are other factors such as sex (women are more likely to use fuller forms, even after controlling for rate of speech, for example), and some of the differences among the ten function words in their response to the factors.
Corpus Variation and Parser Performance

Most work in statistical parsing has focused on a single corpus: the Wall Street Journal portion of the Penn Treebank. While this has allowed for quantitative comparison of parsing techniques, it has left open the question of how other types of text might a ect parser performance, and how portable parsing models are across corpora. We examine these questions by comparing results for the Brown and WSJ corpora, and also consider which parts of the parser's probability model are particularly tuned to the corpus on which it was trained. This leads us to a technique for pruning parameters to reduce the size of the parsing model.
Forms of English Function Words — Effects of Disfluencies , Turn Position , Age and Sex , and Predictability

This study examines the role of several non–phonetic factors in the reduction of ten frequent English function words ( I, and, the, that, a, you, to, of, it , and in) in the phoneticallytranscribed portion of the Switchboard corpus of spontaneous telephone conversations. Using ordinary linear and logistic regression models, we examined the length of the words and whether their vowels were full or reduced. We show that function words are more likely to be longer or unreduced when they are turn–initial or utterance–final, when the speaker is female (mostly but not completely due to slower rate of speech) and when the word is surprising given the previous or following words. Finally, focusing on finer details of the effect of planning problems on reduction, we show that filled pauses ( uh and um) are the strongest factor in predicting lengthening of a previous function word. The results bear on issues in speech recognition and models of speech production.
A Dynamic Model of Aspectual

This paper describes results of a dynamic model of aspectual composition that demonstrates how features necessary for planning and controlling actions can also motivate and ground simple analyses of a number of as-pectual phenomena. A novel feature of the model is an active computational representation for verb semantics called x-schemas, an extension of the Petri net formalism that can encode goals, resources and other features aaecting aspect. Vexing problems of aspectual composition lend themselves to simple analyses in terms of the context-sensitive interaction between verb-speciic x-schemas and a controller x-schema that captures important regularities in the evolution of events. The resulting x-schemas can be elaborated and constrained by such factors as tense, temporal modiiers, nominals and pragmatic context, providing a rich representation that supports simulative inference in language understanding .
A Dynamic Model of Aspectual Composition

This paper describes results of a dynamic model of aspectual composition that demonstrates how features necessary for planning and controlling actions can also motivate and ground simple analyses of a number of aspectual phenomena. A novel feature of the model is an active computational representation for verb semantics called x-schemas, an extension of the Petri net formalism that can encode goals, resources and other features a ecting aspect. Vexing problems of aspectual composition lend themselves to simple analyses in terms of the context-sensitive interaction between verb-speci c x-schemas and a controller x-schema that captures important regularities in the evolution of events. The resulting x-schemas can be elaborated and constrained by such factors as tense, temporal modi ers, nominals and pragmatic context, providing a rich representation that supports simulative inference in language understanding.
Learning Bias and Phonological-Rule Induction

A fundamental debate in the machine learning of language has been the role of prior knowledge in the learning process. Purely nativist approaches, such as the Principles and Parameters model, build parameterized linguistic generalizations directly into the learning system. Purely empirical approaches use a general, domain-independent learning rule (Error Back-Propagation, Instance-based Generalization, Minimum Description Length) to learn linguistic generalizations directly from the data.In this paper we suggest that an alternative to the purely nativist or purely empiricist learning paradigms is to represent the prior knowledge of language as a set of abstract learning biases, which guide an empirical inductive learning algorithm. We test our idea by examining the machine learning of simple Sound Pattern of English (SPE)-style phonological rules. We represent phonological rules as finite-state transducers that accept underlying forms as input and generate surface forms as output. We show that OSTIA, a general-purpose transducer induction algorithm, was incapable of learning simple phonological rules like flapping. We then augmented OSTIA with three kinds of learning biases that are specific to natural language phonology, and that are assumed explicitly or implicitly by every theory of phonology: faithfulness (underlying segments tend to be realized similarly on the surface), community (similar segments behave similarly), and context (phonological rules need access to variable in their context). These biases are so fundamental to generative phonology that they are left implicit in many theories. But explicitly modifying the OSTIA algorithm with these biases allowed it to learn more compact, accurate, and general transducers, and our implementation successfully learns a number of rules from English and German. Furthermore, we show that some of the remaining errors in our augmented model are due to implicit biases in the traditional SPE-style rewrite system that are not similarly represented in the transducer formalism, suggesting that while transducers may be formally equivalent to SPE-style rules, they may not have identical evaluation procedures.Because our biases were applied to the learning of very simple SPE-style rules, and to a non-psychologically-motivated and nonprobabilistic theory of purely deterministic transducers, we do not expect that our model as implemented has any practical use as a phonological learning device, nor is it intended as a cognitive model of human learning. Indeed, because of the noise and nondeterminism inherent to linguistic data, we feel strongly that stochastic algorithms for language induction are much more likely to be a fruitful research direction. Our model is rather intended to suggest the kind of biases that may be added to other empiricist induction models, and the way in which they may be added, in order to build a cognitively and computationally plausible learning model for phonological rules.
Applications of Technology in Providing Transit Information

A number of efforts to electronically disseminate transit information are described and compared, leading up to a detailed analysis of one volunteer project to place transit information for the San Francisco Bay Area on the Internet. This service provides route, schedule, and fare information for a dozen transit operators. The system is used by more than 600 people daily, a number that is growing steadily. The results of a user survey are discussed, revealing that a large number of the system's users are regular transit users. Many are browsing instead of searching for specific information, and a smaller number use the system to check schedules for transit routes they use every day. The emergence of the Internet as a global standard gives transit operators a new opportunity to provide detailed information directly to their patrons at low cost and around the clock.
APPLICATIONS OF TECHNOLOGY IN PROVIDING TRANSIT INFORMATION

A number of efforts to electronically disseminate transit information are described and compared, leading up to a detailed analysis of one volunteer project to place transit information for the San Francisco Bay Area on the Internet. This service provides route, schedule, and fare information for a dozen transit operators. The system is used by more than 600 people daily, a number that is growing steadily. The results of a user survey are discussed, revealing that a large number of the system's users are regular transit users. Many are browsing instead of searching for specific information, and a smaller number use the system to check schedules for transit routes they use every day. The emergence of the Internet as a global standard gives transit operators a new opportunity to provide detailed information directly to their patrons at low cost and around the clock.
Automatic Induction of Finite State Transducers for Simple Phonological Rules

This paper presents a method for learning phonological rules from sample pairs of underlying and surface forms, without negative evidence. The learned rules are represented as finite state transducers that accept underlying forms as input and generate surface forms as output. The algorithm for learning them is an extension of the OSTIA algorithm for learning general subsequential finite state transducers. Although OSTIA is capable of learning arbitrary s.f.s.t's in the limit, large dictionaries of actual English pronunciations did not give enough samples to correctly induce phonological rules. We then augmented OSTIA with two kinds of knowledge specific to natural language phonology, biases from "universal grammar". One bias is that underlying phones are often realized as phonetically similar or identical surface phones. The other biases phonological rules to apply across natural phonological classes. The additions helped in learning more compact, accurate, and general transducers than the unmodified OSTIA algorithm. An implementation of the algorithm successfully learns a number of English postlexical rules.
SPEECH OGLE: Indexing Uncertainty for Spoken Document Search

The paper presents the Position Specific Posterior Lattice (PSPL), a novel lossy representation of automatic speech recognition lattices that naturally lends itself to efficient indexing and subsequent relevance ranking of spoken documents.In experiments performed on a collection of lecture recordings --- MIT iCampus data --- the spoken document ranking accuracy was improved by 20% relative over the commonly used baseline of indexing the 1-best output from an automatic speech recognizer.The inverted index built from PSPL lattices is compact --- about 20% of the size of 3-gram ASR lattices and 3% of the size of the uncompressed speech --- and it allows for extremely fast retrieval. Furthermore, little degradation in performance is observed when pruning PSPL lattices, resulting in even smaller indexes --- 5% of the size of 3-gram ASR lattices.
Position Specific Posterior Lattices for Indexing Speech

The paper presents the Position Specific Posterior Lattice, a novel representation of automatic speech recognition lattices that naturally lends itself to efficient indexing of position information and subsequent relevance ranking of spoken documents using proximity.In experiments performed on a collection of lecture recordings --- MIT iCampus data --- the spoken document ranking accuracy was improved by 20% relative over the commonly used baseline of indexing the 1-best output from an automatic speech recognizer. The Mean Average Precision (MAP) increased from 0.53 when using 1-best output to 0.62 when using the new lattice representation. The reference used for evaluation is the output of a standard retrieval engine working on the manual transcription of the speech collection.Albeit lossy, the PSPL lattice is also much more compact than the ASR 3-gram lattice from which it is computed --- which translates in reduced inverted index size as well --- at virtually no degradation in word-error-rate performance. Since new paths are introduced in the lattice, the ORACLE accuracy increases over the original ASR lattice.
Indexing uncertainty for spoken document search

The paper presents the Position Specific Posterior Lattice, a novel lossy representation of automatic speech recognition lattices that naturally lends itself to efficient indexing and subsequent relevance ranking of spoken documents. Albeit lossy, the PSPL lattice is much more compact than the ASR 3-gram lattice from which it is computed, at virtually no degradation in word-error-rate performance. Since new paths are introduced in the lattice, the “oracle” accuracy increases over the original ASR lattice. In experiments performed on a collection of lecture recordings — MIT iCampus database — the spoken document ranking accuracy was improved by 20% relative over the commonly used baseline of indexing the 1-best output from an automatic speech recognizer. The Mean Average Precision (MAP) increased from 0.53 when using 1-best output to 0.62 when using the new lattice representation. The reference used for evaluation is the output of a standard retrieval engine working on the manual transcription of the speech collection.
Parsing Conversational Speech Using Enhanced Segmentation

The lack of sentence boundaries and presence of disfluencies pose difficulties for parsing conversational speech. This work investigates the effects of automatically detecting these phenomena on a probabilistic parser's performance. We demonstrate that a state-of-the-art segmenter, relative to a pause-based segmenter, gives more than 45% of the possible error reduction in parser performance, and that presentation of interruption points to the parser improves performance over using sentence boundaries alone.
Conditional Maximum Likelihood Estimation Using Rational Function Growth Transform

We present a study on conditional maximum likelihood (CML) estimation of probability models by means of a well known technique that generalizes the Baum-Eagon inequality  from polynomials to rational functions. The main advantage of the rational function growth transform (RFGT) method  is that it keeps the model parameter values — probabilities — properly normalized at each iteration. As a case study we apply the technique to discriminatively train a Na ̈ ıve B yes (NB) classifier; the same procedure is at the basis of discriminative training of HMMs in speech recognition . The NB model trained under the maximum likelihood (ML) and CML criteria, respectively, is used on a text classification problem. Smoothing is found to be a key component in increasing the classification accuracy. A simple modification of the algorithm increases the convergence speed significantly — as measured by likelihood and classification accuracy increase per iteration — over a straightforward implementation of RFGT. The model trained under the CML criterion achieves a relative improvement of 40% in classification accuracy over its ML counterpart . The NB model can also be re-parameterized as a standard conditional exponential model encoutered in maximum entropy (MaxEnt) estimation . Although the two parameterizations are in principle equivalent and should lead to the same model when trained under CML, the conditional exponential model estimated using improved iterative scaling (IIS) and smoothing with a Gaussian prior  outperforms the smoothed NB model estimated using RFGT when evaluated in terms of classification accuracy.
Conditional ML Estimation Using Rational Function Growth Transform

We present a study on conditional maximum likelihood (CML) estimation of probability models by means of a well known technique that generalizes the Baum-Eagon inequality  from polynomials to rational functions. The main advantage of the rational function growth transform (RFGT) method  is that it keeps the model parameter values — probabilities — properly normalized at each iteration. As a case study we apply the technique to discriminatively train a Na¨ive Bayes (NB) classiﬁer; the same procedure is at the basis of discriminative training of HMMs in speech recognition . The NB model trained under the maximum likelihood (ML) and CML criteria, respectively, is used on a text classiﬁcation problem. Smoothing is found to be a key component in increasing the classiﬁcation accuracy. A simple modiﬁcation of the algorithm increases the convergence speed signiﬁcantly — as measured by likelihood and classiﬁcation accuracy increase per iteration — over a straightforward implementation of RFGT. The model trained under the CML criterion achieves a relative improvement of 40% in classiﬁcation accuracy over its ML counterpart . The NB model can also be re-parameterized as a standard conditional exponential model encoutered in maximum entropy (MaxEnt) estimation . Although the two parameterizations are in principle equivalent and should lead to the same model when trained under CML, the conditional exponential model estimated using improved iterative scaling (IIS) and smoothing with a Gaussian prior  outperforms the smoothed NB model estimated using RFGT when evaluated in terms of classiﬁcation accuracy.
Conditional Maximum Likelihood Estimation of Naive Bayes Probability Models Using Rational Function Growth Transform

We present a method for conditional maximum likelihood estimation of Naive Bayes models that employs a well known technique relying on a generalization of the Baum-Eagon inequality from polynomials to rational functions. The main advantage of the procedure is that it keeps the model parameter values (probabilities) properly normalized at each iteration. We apply the model trained under the maximum likelihood and conditional maximum likelihood criteria, respectively, to a text classification problem. A simple modification of the algorithm increases the convergence speed significantly over a straightforward implementation. The model trained under the conditional maximum likelihood criterion achieves a relative improvement of 40% in classification accuracy over its maximum likelihood counterpart on a text classification task.
Is word error rate a good indicator for spoken language understanding accuracy

It is a conventional wisdom in the speech community that better speech recognition accuracy is a good indicator for better spoken language understanding accuracy, given a fixed understanding component. The findings in this work reveal that this is not always the case. More important than word error rate reduction, the language model for recognition should be trained to match the optimization objective for understanding. In this work, we applied a spoken language understanding model as the language model in speech recognition. The model was obtained with an example-based learning algorithm that optimized the understanding accuracy. Although the speech recognition word error rate is 46% higher than the trigram model, the overall slot understanding error can be reduced by as much as 17%.
Speech utterance classification

The paper presents a series of experiments on speech utterance classification performed on the ATIS corpus. We compare the performance of n-gram classifiers with that of Naive Bayes and maximum entropy classifiers. The n-gram classifiers have the advantage that one can use a single pass system (concurrent speech recognition and classification) whereas for Naive Bayes or maximum entropy classification we use a two-stage system: speech recognition followed by classification. Substantial relative improvements (up to 55%) in classification accuracy can be obtained using discriminative. training methods that belong to the class of conditional maximum likelihood techniques.
Discriminative training of n-gram classifiers for speech and text routing

We present a method for conditional maximum likelihood estimation of N-gram models used for text or speech utterance classification. The method employs a well known technique relying on a generalization of the Baum-Eagon inequality from polynomials to rational functions. The best performance is achieved for the 1-gram classifier where conditional maximum likelihood training reduces the class error rate over a maximum likelihood classifier by 45% relative.