nils-herrmann/clean_scirepeval huggingface dataset:
Nonlinear State Space Estimation With Neural Networks And The Em Algorithm

In this paper, we derive an EM algorithm for nonlinear state space models. We use it to estimate jointly the neural network weights, the model uncertainty and the noise in the data. In the E-step we apply a forward-backward Rauch-Tung-Striebel smoother to compute the network weights. For the M-step, we derive expressions to compute the model uncertainty and the measurement noise. We nd that the method is intrinsically very powerful, simple, elegant and stable.
On Sequential Bayesian Logistic Regression

The Extended Kalman Filter (EKF) algorithm for identification of a state space model is shown to be a sensible tool in estimating a Logistic Regression Model sequentially. A Gaussian probability density over the parameters of the Logistic model is propagated on a sample by sample basis. Two other approaches, the Laplace Approximation and the Variational Approximation are compared with the state space formulation. Features of the latter approach, such as the possibility of inferring noise levels by maximising the ‘innovation probability’ are discussed. Experimental illustrations of these ideas on a synthetic and a real world problems are shown.
Parametric subspace modelling of speech transitions

We report on attempting to capture segmental transition information for speech recognition tasks. The slowly varying dynamics of spectral trajectories carries much discriminant information that is very crudely modelled by traditional approaches such as HMMs. In attempts such as recurrent neural networks there is the hope, but not convincing demonstration, that such transitional information could be captured. We start from the very different position of explicitly capturing the trajectory of short time spectral parameter vectors on a subspace in which the temporal sequence information is preserved (time constrained principal component analysis). On this subspace, we attempt a parametric modelling of the trajectory, and compute a distance metric to perform classification of diphones. Much of the discriminant information is still retained in this subspace. This is illustrated on the isolated transitions /bee/,/dee/ and /gee/.
Maximum Realisable Performance: a Principled Method for Enhancing Performance by Using Multiple Clas

A novel method is described for obtaining superior classiication performance over a variable range of classiication costs. By analysis of a set of existing classiiers using a receiver operating characteristic (ROC) curve, a set of new realisable classiiers may be obtained by a principled random combination of two of the existing classiiers. These classiiers lie on the convex hull that contains the original ROC points for the existing classiiers. This hull is the maximum realisable ROC (MRROC). A theorem for this method is derived and proved from an observation about ROC data, and experimental results verify that a superior classii-cation system may be constructed using only the existing classiiers and the information of the original ROC data. This new system is shown to produce the MRROC, and as such provides a powerful technique for improving classiication systems in problem domains within which classii-cation costs may not be known a priori Lovell et al., 1997b, Lovell et al., 1997a].
Comparing solution methodologies for speech enhancement within a Bayesian framework

Speech enhancement is the area of speech processing concerned with the reconstruction of speech corrupted by some noise process. This report states the problem of enhancing speech corrupted by additive white Gaussian noise within a model-based Bayesian framework, and proceeds to develop and compare three strategies to estimate the model parameters and the clean speech. Each of these methods estimates the additive noise variance from the observed noise corrupted data, alongside the speech model parameters, eliminating the need for off-line estimation of this value. The first two methods are the Evidence Maximisation and Expectation Maximisation (EM) algorithms, which essentially perform maximum a posteriori (MAP) estimation of the model parameters, and, based on this estimate, reconstruct the clean speech. The difference between these two methods lies in the fact that the former performs the maximisation directly in the parameter space, whereas the latter iteratively max-imises for the parameters by conditioning on the current estimate of the clean speech. The third method entertained is the Gibbs sampler, which is a special case of the general class of Markov chain Monte Carlo (MCMC) algorithms. In contrast with the first two methods, which are deterministic, the Gibbs sampler is essentially stochastic in nature, and sets up a Markov chain that generates samples from the joint posterior distribution for the model parameters and clean speech. These samples are subsequently used to calculate Monte Carlo estimates for the desired quantities. The methods are evaluated and compared on real speech corrupted by both stationary and non-stationary noise processes.
An improved speech production model for voiced speech utilising seasonal AR-AR model and Markov chain Monte Carlo simulation

Summaryofreconstruction andestimationresultsunderavarietyofcorrupt-ing noise conditions when running the sampling algorithm on the data of Figure 11.All the SNR values are given in dB.It should be pointed out that even though, from an algorithmic point of view, thesignal reconstruction step is very efﬁciently implemented, it is still computationallyintensive, especially for large values of the effective order of the state-space system p+q  , which commonly occur when processing speech signals. There are a numberof ways in which the computational efﬁciency can be increased, and a couple of obvi-ous suggestions are presented below. Due to the nature of the model, the state vector,and resulting matrices, are sparse, allowing for efﬁcient sparse matrix operations tobe employed. Furthermore, approximately 85% of the computational time is spent inupdating the state covariance matrices in the Kalman ﬁlter and smoother. If the sys-tem is known to be time-invariant and stationary (as would be the case over a singleframe of speech), these matrices converge within the ﬁrst few iterations, eliminatingthe need for the recursive updates. In some special cases the steady-state values forthese matrices can be obtained by an analytic solution of the Riccati equation, elimi-nating the updating step completely.
Sequential Monte Carlo methods for optimisation of neural network models

We discuss a novel strategy for training neural networks using sequential Monte Carlo algorithms and propose a new hybrid gradient descent/sampling importance resampling algorithm (HySIR). In terms of both computational time and accuracy, the hybrid SIR is a clear improvement over conventional sequential Monte Carlo techniques. The new algorithm may be viewed as a global optimisation strategy, which allows us to learn the probability distributions of the network weights and outputs in a sequential framework. It is well suited to applications involving on-line, nonlinear and non-Gaussian signal processing. We show how the new algorithm outperforms extended Kalman lter training on several problems. In particular, we address the problem of pricing option contracts, traded in nancial markets. In this context, we are able to estimate the one-step-ahead probability density functions of the options prices.
Realisable Classifiers: Improving Operating Performance on Variable Cost Problems

A novel method is described for obtaining superior classification performance over a variable range of classification costs. By analysis of a set of existing classifiers using a receiver operating characteristic ( )c urve, a set of new realisable classifiers may be obtained by a random combination of two of the existing classifiers. These classifiers lie on the convex hull that contains the original points for the existing classifiers. This hull is the maximum realisable ( ). A theorem for this method is derived and proved from an observation about data, and experimental results verify that a superior classification system may be constructed using only the existing classifiers and the information of the original data. This new system is shown to produce the , and as such provides a powerful technique for improving classification systems in problem domains within which classification costs may not be known ap riori. Empirical results are presented for artificial data, and for two real world data sets: an image segmentation task and the diagnosis of abnormal thyroid condition.
Parcel: Feature Subset Selection in Variable Cost Domains

The vast majority of classification systems are designed with a single set of features, and optimised to a single specified cost. However, in examples such as medical and financial risk modelling, costs are known to vary subsequent to system design. In this paper, we present a design method for feature selection in the presence of varying costs. Starting from the Wilcoxon nonparametric statistic for the performance of a classification system, we introduce a concept called the maximum realisable receiver operating characteristic ( ), and prove a related theorem. A novel criterion for feature selection, based on the area under the curve, is then introduced. This leads to a framework which we call Parcel. This has the flexibility to use different combinations of features at different operating points on the resulting curve. Empirical support for each stage in our approach is provided by experiments on real world problems, with Parcel achieving superior results.
Speech enhancement in a Bayesian framework

We present an approach for the enhancement of speech signals corrupted by additive white noise of Gaussian statistics. The speech enhancement problem is treated as a signal estimation problem within a Bayesian framework. The conventional all-pole speech production model is assumed to govern the behaviour of the clean speech signal. The additive noise level and all-pole model gain are automatically inferred during the speech enhancement process. The strength of the Bayesian approach developed in this paper lies in its ability to perform speech enhancement without the usual requirement of estimating the level of the corrupting noise from "silence" segments of the corrupted signal. The performance of the Bayesian approach is compared to that of the Lim & Oppenheim (1978) framework, to which it follows a similar iterative nature. A significant quality improvement is obtained over the Lim & Oppenheim framework.
Markov chain Monte Carlo methods for speech enhancement

This paper investigates a Bayesian approach to the enhancement of speech signals corrupted by additive white Gaussian noise. Parametric models for the speech and noise processes are constructed, leading to a posterior distribution for the model parameters and uncorrupted speech samples given the observed noisy speech samples. Being analytically intractable, inferences concerning these variables are performed using Markov chain Monte Carlo (MCMC) methods. The efficiency of the sampling scheme within this framework is further improved by employing state-space techniques based on the Kalman filter.
MARKOV CHAINMONTE CARLO ESTIMATION FOR THE SEASONAL AUTOREGRESSIVE PROCESS WITH APPLICATION TO PITCHMODELLING

Many time series exhibit long term dependencies that cannot accurately be accounted for by standard autoregressive (AR) type models, unless the model order is chosen to match the frequency of these correlations. One such example is the glottal excita-tion waveform, which comprises of highly correlated pulses, separated by the pitch period. Increasing the AR order leads to models that are less parsimonious than required by the data, and subsequent poor generalisation. Often improved modelling accuracy can be achieved by altering the structure of the model to better account for the longer term effects. The seasonal AR process is one such model, and results from a simple extension to the standard AR process. This report considers the application of the seasonal AR process to modelling the glottal waveform. The parameter estimation problem is stated within a Bayesian framework, leading to analytically intractable in-tegrals. These integrals are elegantly approximated by Markov chain Monte Carlo (MCMC) sampling strategies. Results are presented for both synthetic and real data sets.
Nonlinear state space learning with EM and neural networks

In this paper, we derive the EM algorithm for nonlinear state space models. We show how this algorithm, in conjunction with the well known techniques of Kalman smoothing, can be used for nonlinear system identification. A multilayer perceptron, whose derivatives are computed by backpropagation, is used to generate the measurements mapping. We found that the methodic is intrinsically very powerful, simple, elegant and stable. However, it exhibits very slow convergence.
The EM algorithm and neural networks for nonlinear state space estimation

In this paper, we derive the EM algorithm for nonlinear state space models. We show how this algorithm, in conjunction with the well known techniques of Kalman smoothing, can be used for nonlinear system identi cation. A multi-layer perceptron, whose derivatives are computed by back-propagation, is used to generate the measurements mapping. We nd that the method is intrinsically very powerful, simple, elegant and stable. However, it exhibits very slow convergence.
Regularisation in Sequential Learning Algorithms

In this paper, we discuss regularisation in online/sequential learning algorithms. In environments where data arrives sequentially, techniques such as cross-validation to achieve regularisation or model selection are not possible. Further, bootstrapping to determine a confidence level is not practical. To surmount these problems, a minimum variance estimation approach that makes use of the extended Kalman algorithm for training multi-layer perceptrons is employed. The novel contribution of this paper is to show the theoretical links between extended Kalman filtering, Sutton's variable learning rate algorithms and Mackay's Bayesian estimation framework. In doing so, we propose algorithms to overcome the need for heuristic choices of the initial conditions and noise covariance matrices in the Kalman approach.
Average-Case Learning Curves for Radial Basis Function Networks

The application of statistical physics to the study of the learning curves of feedforward connectionist networks has to date been concerned mostly with perceptron-like networks. Recent work has extended the theory to networks such as committee machines and parity machines, and an important direction for current and future research is the extension of this body of theory to further connectionist networks. In this article, we use this formalism to investigate the learning curves of gaussian radial basis function networks (RBFNs) having fixed basis functions. (These networks have also been called generalized linear regression models.) We address the problem of learning linear and nonlinear, realizable and unrealizable, target rules from noise-free training examples using a stochastic training algorithm. Expressions for the generalization error, defined as the expected error for a network with a given set of parameters, are derived for general gaussian RBFNs, for which all parameters, including centers and spread parameters, are adaptable. Specializing to the case of RBFNs with fixed basis functions (basis functions having parameters chosen without reference to the training examples), we then study the learning curves for these networks in the limit of high temperature.
Hierarchical Bayesian-Kalman models for regularisation and ARD in sequential learning

In this paper, we show that a hierarchical Bayesian modelling approach to sequential learning leads to many interesting attributes such as regularisation and automatic relevance determination. We identify three inference levels within this hierarchy, namely model selection, parameter estimation and noise estimation. In environments where data arrives sequentially, techniques such as cross-validation to achieve regularisation or model selection are not possible. The Bayesian approach, with extended Kalman ltering at the parameter estimation level, allows for regularisation within a minimum variance framework. A multi-layer perceptron is used to generate the extended Kalman lter nonlinear measurements mapping. We describe several algorithms at the noise estimation level, which allow us to implement adaptive regularisation and automatic relevance determination of model inputs and basis functions. An important contribution of this paper is to show the theoretical links between adaptive noise estimation in extended Kalman ltering, multiple adaptive learning rates and multiple smoothing regularisation coe cients.
Generalisation and regularisation by Gaussian filter convolution of radial basis function networks

This paper proposes a new technique called Regularisation by Convolution to improve the generalisation of GaRBF networks. The technique is based on the convolution after training of GaRBF networks with gaussian filters and, consequently, is independent of the learning stage and algorithm. Moreover, it alleviates the need for retraining. The performance of Regularisation by Convolution in improving regression is illustrated for Wahba’s problem.
The past, present, and future of neural networks for signal processing

The article provides a review of the fundamental of neural networks and reports recent progress. Topics covered include dynamic modeling, model-based neural networks, statistical learning, eigenstructure-based processing, active learning, and generalization capability. Current and potential applications of neural networks are also described in detail. Those applications include optical character recognition, speech recognition and synthesis, automobile and aircraft control, image analysis and neural vision, and several medical applications. Essentially, neural networks have become a very effective tool in signal processing, particularly in various recognition tasks.
Neurocomputing applications in post-operative liver transplant monitoring

This paper explores the potential for the application of neurocomputing technology to the domain of post-operative liver transplant monitoring. The investigation compares a neural network model with two classical statistical techniques using biochemical information obtained from a set of liver transplant patients. Each approach combines the results of a number of liver function tests to predict the presence of allograft rejection. Each system is assessed, relative to the clinical gold standard, in terms of its overall accuracy and degree of advance warning offered. Applying nonlinear methods does offer an advantage over the traditional linear techniques. The underlying structure of the data set has also been determined using k-means cluster analysis. This analysis suggests important directions for future investigation including the use of temporal information. Preliminary results of incorporating this temporal information are also presented.
On the use of Expected Attainable Discrimination for feature selection in large scale medical risk p

This report investigates the use of expected attainable discrimination (EAD) as a measure to select discrete valued features in two-class prediction problems. In essence, EAD tells us the performance we could expect to achieve with a simple histogram probability density model of a given dataset. For discrete valued features, this kind of density model is bias-free but can have large variance. Given insuucient training data, such a model's test set performance will be lower than that of a suitably biased model. In light of this, we explore the usefulness of EAD for feature selection.
Application of neural networks to postoperative liver transplant monitoring

This paper explores the feasibility and efficacy of applying artificial neural network technology to assist with the clinical management of human liver transplant recipients. We describe a novel application of neural network technology to this domain and present results from three experiments which assess the performance gains obtained. These experiments directly compare the statistical techniques, logistic regression and discriminant analysis with multi-layer perceptrons (MLPs) for performing rejection risk assessment. This paper documents an analysis of progressively more sophisticated modelling techniques, together with a discussion of the advantages and disadvantages of each approach. These experiments lead us to conclude that MLPs offer significant advantages over traditional statistical methods in this domain. Finally, the paper introduces a discussion, together with interim results, of the future directions being explored in this research program. In particular, this includes the use of temporal information to further enhance the performance of the most promising of the connectionist systems described here.
Non-linear speech transition visualization

Modelling context effects and segmental transitions in speech recognition systems is very important. Explicitly modelling segmental transitions in a RNN framework would circumvent these problems. We present an interesting application of Principal Curves, an algorithm to extract a non-linear summary of p-dimensional data firstly published by Hastie and Stuetzle (1989). The algorithm can be used to visualize non-linear transient characteristics in speech. We show that between-phone characteristics found within diphones can be used as discriminant information to distinguish ambiguous phones. The technique used is explained and illustrated on the examples /bah/, /dah/ and /gah/.
Using upper bounds on attainable discrimination to select discrete valued features

Selection of features that will permit accurate pattern classification is, in general, a difficult task. However, if a particular data set is represented by discrete valued features, it becomes possible to determine empirically the contribution that each feature makes to the discrimination between classes. We describe how to calculate the maximum discrimination possible in a two alternative forced choice decision problem, when discrete valued features are used to represent a given data set. (In this paper, we measure discrimination in terms of the area under the receiver operating characteristic (ROC) curve.) Since this bound corresponds to the upper limit of classification achievable by any classifier (with that given data representation), we can use it to assess whether recognition errors are due to a lack of separability in the data or shortcomings in the classification technique. In comparison to the training and testing of artificial neural networks, the empirical bound on discrimination can be efficiently calculated, allowing an experimenter to decide whether subsequent development of neural network models is warranted. We extend the discrimination bound method so that we can estimate both the maximum and average discrimination we can expect on unseen test data. These estimation techniques are the basis of a backwards elimination algorithm that can be used to rank features in order of their discriminative power. We use two problems to demonstrate this feature selection process: classification of the Mushroom Database, and a real-world, pregnancy related medical risk prediction task-assessment of risk of perinatal death.