nils-herrmann/clean_scirepeval huggingface dataset:
Language Model Adaptation with MAP Estimation and the Perceptron Algorithm

In this paper, we contrast two language model adaptation approaches: MAP estimation and the perceptron algorithm. Used in isolation, we show that MAP estimation outperforms the latter approach, for reasons which argue for combining the two approaches. When combined, the resulting system provides a 0.7 percent absolute reduction in word error rate over MAP estimation alone. In addition, we demonstrate that, in a multi-pass recognition scenario, it is better to use the perceptron algorithm on early pass word lattices, since the improved error rate improves acoustic model adaptation.
A generalized construction of integrated speech recognition transducers

We showed in previous work that weighted finite-state transducers provide a common representation for many components of a speech recognition system and described general algorithms for combining these representations to build a single optimized and compact transducer integrating all these components, directly mapping from HMM states to words. This approach works well for certain well-controlled input transducers, but presents some problems related to the efficiency of composition and the applicability of determinization and weight-pushing with more general transducers. We generalize our prior construction of the integrated speech recognition transducer to work with an arbitrary number of component transducers and, to a large extent, release the constraints imposed on the type of input transducers by providing more general solutions to these problems. This generalization allowed us to deal with cases where our prior optimization did not apply. Our experiments in the AT&T HMIHY 0300 task and an AT&T VoiceTone task show the efficiency of our generalized optimization technique. We report a 1.6 recognition speed-up in the HMIHY 0300 task, 1.8 speed-up in a VoiceTone task using a word-based language model, and 1.7 using a class-based model.
Meta-data conditional language modeling

Automatic speech recognition (ASR) often occurs in circumstances in which knowledge external to the speech signal, or meta-data, is given. For example, a company receiving a call from a customer might have access to a database record of that customer. Conditioning the ASR models directly on this information to improve the transcription accuracy is hampered because, generally, the meta-data takes on many values and a training corpus has little data for each meta-data condition. The paper presents an algorithm to construct language models conditioned on such metadata. It uses tree-based clustering of the the training data to derive automatically meta-data projections, useful as language model conditioning contexts. The algorithm was tested on a multiple domain voice mail transcription task. We compare the performance of an adapted system aware of the domain shift to a system that only has meta-data to infer that fact. The meta-data used were the caller ID strings associated with the voice mail messages. The meta-data adapted system matched the performance of the system adapted using the domain knowledge explicitly.
Good-Turing estimation from word lattices for unsupervised language model adaptation

We present a comparison of using the weighted word lattice output of a recognizer versus its one-best transcription for unsupervised language model adaptation. We begin with a general analysis of how to smooth word probabilities when the sample is hidden, as is the case with recognizer lattices. For each smoothing technique for the known sample case, we show there is a natural generalization to the hidden case. In particular, we use this generalization with the well-known Good-Turing estimate on word lattices, and show results using Monte Carlo methods for building Katz backoff models. In our realistic adaptation task, with mismatched acoustic and language models, we find that Katz backoff models trained on word lattice samples provide a small, consistent benefit over those trained on one-best output, most notably when there is a limited amount of adaptation data (less than 100 hours). Thus, while the recognizer one-best transcription can provide an effective approximation for the purpose of language model adaptation under certain circumstances, the word lattice provides information that can be exploited for more robust language modeling.
Unsupervised language model adaptation

This paper investigates unsupervised language model adaptation, from ASR transcripts. N-gram counts from these transcripts can be used either to adapt an existing n-gram model or to build an n-gram model from scratch. Various experimental results are reported on a particular domain adaptation task, namely building a customer care application starting from a general voicemail transcription system. The experiments investigate the effectiveness of various adaptation strategies, including iterative adaptation and self-adaptation on the test data. They show an error rate reduction of 3.9% over the unadapted baseline performance, from 28% to 24.1%, using 17 hours of unsupervised adaptation material. This is 51% of the 7.7% adaptation gain obtained by supervised adaptation. Self-adaptation on the test data resulted in a 1.3% improvement over the baseline.
Supervised and unsupervised PCFG adaptation to novel domains

This paper investigates adapting a lexicalized probabilistic context-free grammar (PCFG) to a novel domain, using maximum a posteriori (MAP) estimation. The MAP framework is general enough to include some previous model adaptation approaches, such as corpus mixing in Gildea (2001), for example. Other approaches falling within this framework are more effective. In contrast to the results in Gildea (2001), we show F-measure parsing accuracy gains of as much as 2.5% for high accuracy lexicalized parsing through the use of out-of-domain treebanks, with the largest gains when the amount of indomain data is small. MAP adaptation can also be based on either supervised or unsupervised adaptation data. Even when no in-domain treebank is available, unsupervised techniques provide a substantial accuracy gain over unadapted grammars, as much as nearly 5% F-measure improvement.
Generalized Algorithms for Constructing Statistical Language Models

Recent text and speech processing applications such as speech mining raise new and more general problems related to the construction of language models. We present and describe in detail several new and efficient algorithms to address these more general problems and report experimental results demonstrating their usefulness. We give an algorithm for computing efficiently the expected counts of any sequence in a word lattice output by a speech recognizer or any arbitrary weighted automaton; describe a new technique for creating exact representations of n-gram language models by weighted automata whose size is practical for offline use even for a vocabulary size of about 500,000 words and an n-gram order n = 6; and present a simple and more general technique for constructing class-based language models that allows each class to represent an arbitrary weighted automaton. An efficient implementation of our algorithms and techniques has been incorporated in a general software library for language modeling, the GRM Library, that includes many other text and grammar processing functionalities.
Markov Parsing: Lattice Rescoring with a Statistical Parser

We present a generalization of an incremental statistical parsing algorithm that allows for the re-scoring of lattices of word hypotheses, for use by a speech recognizer. This approach contrasts with other lattice parsing algorithms, which either do not provide scores for strings in the lattice (i.e. they just produce parse trees) or use search techniques (e.g. A-star) to find the best paths through the lattice, without re-scoring every arc. We show that a very large efficiency gain can be had in processing 1000-best lists without reducing word accuracy when the lists are encoded in lattices instead of trees. Further, this allows for processing arbitrary lattices without n-best extraction. This can lead to more interesting methods of combination with other models, both acoustic and language, through, for example, adaptation or confusion matrices.
Robust Probabilistic Predictive Syntactic Processing

This thesis presents a broad-coverage probabilistic top-down parser, and its application to the problem of language modeling for speech recognition. The parser builds fully connected derivations incrementally, in a single pass from left-to-right across the string. We argue that the parsing approach that we have adopted is well-motivated from a psycholinguistic perspective, as a model that captures probabilistic dependencies between lexical items, as part of the process of building connected syntactic structures. The basic parser and conditional probability models are presented, and empirical results are provided for its parsing accuracy on both newspaper text and spontaneous telephone conversations. Modifications to the probability model are presented that lead to improved performance. A new language model which uses the output of the parser is then defined. Perplexity and word error rate reduction are demonstrated over trigram models, even when the trigram is trained on significantly more data. Interpolation on a word-by-word basis with a trigram model yields additional improvements.
Probabilistic Top-Down Parsing and Language Modeling

This paper describes the functioning of a broad-coverage probabilistic top-down parser, and its application to the problem of language modeling for speech recognition. The paper first introduces key notions in language modeling and probabilistic parsing, and briefly reviews some previous approaches to using syntactic structure for language modeling. A lexicalized probabilistic top-down parser is then presented, which performs very well, in terms of both the accuracy of returned parses and the efficiency with which they are found, relative to the best broad-coverage statistical parsers. A new language model that utilizes probabilistic top-down parsing is then outlined, and empirical results show that it improves upon previous work in test corpus perplexity. Interpolation with a trigram model yields an exceptional improvement relative to the improvement observed by other models, demonstrating the degree to which the information captured by our parsing model is orthogonal to that captured by a trigram model. A small recognition experiment also demonstrates the utility of the model.
Noun-phrase co-occurrence statistics for semi-automatic semantic lexicon construction

Generating semantic lexicons semi-automatically could be a great time saver, relative to creating them by hand. In this paper, we present an algorithm for extracting potential entries for a category from an on-line corpus, based upon a small set of exemplars. Our algorithm finds more correct terms and fewer incorrect ones than previous work in this area. Additionally, the entries that are generated potentially provide broader coverage of the category than would occur to an individual coding them by hand. Our algorithm finds many terms not included within Wordnet (many more than previous algorithms), and could be viewed as an ``enhancer'' of existing broad-coverage resources.
Measuring Efficiency in High-accuracy, Broad-coverage Statistical Parsing

Very little attention has been paid to the comparison of efficiency between high accuracy statistical parsers. This paper proposes one machine-independent metric that is general enough to allow comparisons across very different parsing architectures. This metric, which we call "events considered", measures the number of "events", however they are defined for a particular parser, for which a probability must be calculated, in order to find the parse. It is applicable to single-pass or multi-stage parsers. We discuss the advantages of the metric, and demonstrate its usefulness by using it to compare two parsers which differ in several fundamental ways.
Prosodic constraints and the learner’s environment: a corpus study

Jakobson (1968) was one of the first to propose that children’s early u tterances would exhibit ‘unmarked’ linguistic structures. For example, h e predicted that children would first use segments such as /p/, /n/, and /a/ that are wi dely attested in the world’s languages. Given the variability found in childr en’s early utterances, even within a specific language, this hypothesis has been somewh at difficult to test, and has therefore remained somewhat controversial. Recent developments in phonological theory, where ‘unmarked’ structures such as binary feet (or ‘minimal words’) play an important role, have renewed interest in the status of ‘unmarked’ structures in natural language (e. . McCarthy and Prince, 1994). Researchers of language acquisition have been quick to recognize the importance of such developments, especially with respect to higher level prosodic units such as syllables and prosodic wor ds (e.g. Fikkert, 1994;  1995; Gnanadesikan, 1995; Pater, 1997; Ota, 19 99). For example, it has been noted that many children learning English and Dutch init ially avoid the use of coda consonants, producing early words with unmarked ‘cor ’ CV syllables, whereCVC word targets are often realized as CVCV, with an epenthetic final vowel: (1) /bVk/ ! [bVk ] Matthei (1989)
Noun-Phrase Co-Occurence Statistics for Semi-Automatic Semantic Lexicon Construction

Generating semantic lexicons semi-automatically could be a great time saver, relative to creating them by hand. In this paper, we present an algorithm for extracting potential entries for a category from an on-line corpus, based upon a small set of exemplars. Our algorithm finds more correct terms and fewer incorrect ones than previous work in this area. Additionally, the entries that are generated potentially provide broader coverage of the category than would occur to an individual coding them by hand. Our algorithm finds many terms not included within Wordnet (many more than previous algorithms), and could be viewed as an "enhancer" of existing broad-coverage resources.
Support Vector Machines and Other Kernel Methods

Support Vector Machines (SVMs) and other Kernel Methods have emerged as a predominant family of methods in machine learning in the last ten years. The goal of this tutorial is to provide a basic understanding of the ideas behind kernel methods sufficient to allow practical application of the methods and to provide background for those interested in researching further into the subject. First we will use an intuitive geometric examination of SVM classification to introduce SVMs and understand the principals behind them. We will provide intuitions behind the extensive statistical learning theory underlying the approach without going into mathematical detail. Then, we will use case studies in regression and principal component analysis to show how these basic principals can be applied to make practical robust nonlinear versions of other linear inference methods. The power and flexibility of kernel methods comes from their ability to easily use different kernel functions. We will investigate how by changing kernel functions, the same kernel method can be applied to both traditional vector data as well as non-vector data such as strings and graphs. We will then examine some of the practical issues in support vector machines such as parameter selection, algorithms, and software. We will conclude with a discussion of the strengths and limitations of kernel methods.  I. What are Support Vector Machine and Kernel Methods? II. Intuitive guide to SVM Classification a. Linear classification b. Capacity control c. Nonlinear classification III. Case studies a. Regression b. PCA
Kernelized set-membership approach to nonlinear adaptive filtering

In linear filtering, the set-membership normalized least mean squares (SM-NLMS) algorithm has been shown to exhibit desirable features of selective update and optimized variable step size. In this paper, a kernel approach to the SM-NLMS algorithm is presented that makes it feasible to address nonlinear problems. An online greedy approximation technique to achieve sparsity is discussed. Simulation results are presented for two practical problems: equalization of nonlinear inter-symbol interference (ISI) channels and predistortion of nonlinear high power amplifiers (HPA).
Proceedings of the eleventh ACM SIGKDD international conference on Knowledge discovery in data mining

It is our great pleasure to welcome you to the 11th ACM SIGKDD International Conference on Knowledge Discovery and Data Mining -- KDD'05. The KDD conferences provide a forum for novel research results and interesting applications in the areas of data mining and knowledge discovery. The conference series gives researchers and practitioners a unique opportunity to share their perspectives with others and to present new research ideas, applications, solutions, tools, systems, and research directions broadly related to knowledge discovery and data mining.The call for papers attracted 465 research submissions and 73 industrial submissions from around the world. The program committee accepted 40 research papers, 36 research posters, 14 industrial papers and 11 industrial posters. In addition, this year's conference includes three plenary talks and two panels.Putting together KDD'05 was a team effort. First, we would like to thank the authors for providing the content of the program and the program committee and external reviewers, who worked very hard in reviewing papers and providing suggestions for their improvements. Second, we would like to thank the Organizing Committee, who also worked very hard, and, as volunteers, didn't get much for it. I strongly encourage you to thank them, and even, perhaps, to let them get out of the elevator ahead of you. Finally, we would like to thank our sponsor, ACM SIGKDD, for their continued support.Daniel Hudson Burnham (1846-1912) was a partner in the Chicago based architecture firm Burnham and Root. Burnham and Root created the foundation for the modern skyscraper by using a floating foundation of cement that provided a stable foundation even when, as in many Chicago locations, it was not possible to reach bedrock. Burnham and Root was also the lead architect for, and in charge of, construction for the World Columbian Exposition (1893), which celebrated the 400th anniversary of the arrival of Columbus to North America. The World Columbian Exposition was the largest World's Fair to that date and drew 27.5 million attendees at a time when the US population was about 65 million. To achieve this, many logistic obstacles had to be overcome. Burnham's style is nicely captured by a quote associated with him: "Make no little plans. They have no magic to strike man's blood and probably will themselves not be realized.This is the theme we have chosen for this year's KDD Conference. The field of data mining and knowledge discovery is over ten years old, and is now mature enough to begin to make some big plans and to tackle some very difficult problems and challenges. We will begin discussions about these challenges at this year's conference and continue them throughout the year. Over the coming year, please look to the SIGKDD Explorations for further information.We hope that you will find this program interesting and thought provoking and that the conference will provide you with a valuable opportunity to share ideas with other researchers and practitioners from institutions around the world.
Inductive Transfer using Kernel Multitask Latent Analysis

We develop the Kernel Multitask Latent Analysis (KMLA) method for modeling many-to-many relationships between inputs and responses, and show how it can be applied to inductive transfer problems in bioseparations. KMLA performs dimensionality reduction targeted towards a multitask loss function much like Kernel Partial Least Squares (KPLS). KPLS is limited to least squares multiple regression while KMLA is a more general approach that can utilize widely-used convex loss functions for inference tasks. KMLA achieves inductive transfer between tasks by forcing the tasks to share the same latent features. In the bioseparation problem, the goal is to predict the retention times for a novel anion-chromatography system; only a few retention times are known for the target systems, while many protein retention times are known for the related systems. KMLA is used with semi-supervised loss functions that do not require that all proteins have responses for all the systems. Results are presented for both regression and ranking losses. KMLA significantly outperforms both single task KMLA and KPLS, and the existing missing response algorithm for multitask PLS extended to kernels.
Column-generation boosting methods for mixture of kernels

We devise a boosting approach to classification and regression based on column generation using a mixture of kernels. Traditional kernel methods construct models based on a single positive semi-definite kernel with the type of kernel predefined and kernel parameters chosen according to cross-validation performance. Our approach creates models that are mixtures of a library of kernel models, and our algorithm automatically determines kernels to be used in the final model. The 1-norm and 2-norm regularization methods are employed to restrict the ensemble of kernel models. The proposed method produces sparser solutions, and thus significantly reduces the testing time. By extending the column generation (CG) optimization which existed for linear programs with 1-norm regularization to quadratic programs with 2-norm regularization, we are able to solve many learning formulations by leveraging various algorithms for constructing single kernel models. By giving different priorities to columns to be generated, we are able to scale CG boosting to large datasets. Experimental results on benchmark data are included to demonstrate its effectiveness.
Decision-tree learning in dwell point policies in autonomous vehicle storage and retrieval systems (AVSRS)

Autonomous vehicle storage and retrieval system (A VSRS) is a new material handling technology for unit load storage and retrieval. The dwell point issue is one of important aspects to increase the throughput. We examine two dwell point policies of an IO point policy and a last transaction floor policy. We combine two policies from findings in a decision tree. The result indicates significant improvements. The findings from a decision tree are used as inputs in the control system so that the control system selects an appropriate policy based on a current system state.
Sparse Regression Ensembles in Infinite and Finite Hypothesis Spaces

We examine methods for constructing regression ensembles based on a linear program (LP). The ensemble regression function consists of linear combinations of base hypotheses generated by some boosting-type base learning algorithm. Unlike the classification case, for regression the set of possible hypotheses producible by the base learning algorithm may be infinite. We explicitly tackle the issue of how to define and solve ensemble regression when the hypothesis space is infinite. Our approach is based on a semi-infinite linear program that has an infinite number of constraints and a finite number of variables. We show that the regression problem is well posed for infinite hypothesis spaces in both the primal and dual spaces. Most importantly, we prove there exists an optimal solution to the infinite hypothesis space problem consisting of a finite number of hypothesis. We propose two algorithms for solving the infinite and finite hypothesis problems. One uses a column generation simplex-type algorithm and the other adopts an exponential barrier approach. Furthermore, we give sufficient conditions for the base learning algorithm and the hypothesis set to be used for infinite regression ensembles. Computational results show that these methods are extremely promising.