nils-herrmann/clean_scirepeval huggingface dataset:
Fisher Scoring and a Mixture of Modes Approach for Approximate Inference and Learning in Nonlinear State Space Models

We present Monte-Carlo generalized EM equations for learning in nonlinear state space models. The difficulties lie in the Monte-Carlo E-step which consists of sampling from the posterior distribution of the hidden variables given the observations. The new idea presented in this paper is to generate samples from a Gaussian approximation to the true posterior from which it is easy to obtain independent samples. The parameters of the Gaussian approximation are either derived from the extended Kalman filter or the Fisher scoring algorithm. In case the posterior density is multimodal we propose to approximate the posterior by a sum of Gaussians (mixture of modes approach). We show that sampling from the approximate posterior densities obtained by the above algorithms leads to better models than using point estimates for the hidden states. In our experiment, the Fisher scoring algorithm obtained a better approximation of the posterior mode than the EKF. For a multimodal distribution, the mixture of modes approach gave superior results.
Call-Based Fraud Detection in Mobile Communication Networks Using a Hierarchical Regime-Switching Model

Fraud causes substantial losses to telecommunication carriers. Detection systems which automatically detect illegal use of the network can be used to alleviate the problem. Previous approaches worked on features derived from the call patterns of individual users. In this paper we present a call-based detection system based on a hierarchical regime-switching model. The detection problem is formulated as an inference problem on the regime probabilities. Inference is implemented by applying the junction tree algorithm to the underlying graphical model. The dynamics are learned from data using the EM algorithm and subsequent discriminative training. The methods are assessed using fraud data from a real mobile communication network.
Fraud detection in communication networks using neural and probabilistic methods

Fraud detection refers to the attempt to detect illegitimate usage of a communication network. Three methods to detect fraud are presented. Firstly, a feed-forward neural network based on supervised learning is used to learn a discriminative function to classify subscribers using summary statistics. Secondly, a Gaussian mixture model is used to model the probability density of subscribers' past behavior so that the probability of current behavior can be calculated to detect any abnormalities from the past behavior. Lastly, Bayesian networks are used to describe the statistics of a particular user and the statistics of different fraud scenarios. The Bayesian networks can be used to infer the probability of fraud given the subscribers' behavior. The data features are derived from toll tickets. The experiments show that the methods detect over 85% of the fraudsters in our testing set without causing false alarms.
Neural nets at Siemens

The high potential for applications of neural nets was realized in the corporate research divisions of Siemens in Munich and Princeton 10 years ago. The broad range of applications on which the central research division cooperates with the business units is illustrated. Furthermore. a sampling of representative applications is given. The application spectrum can be categorized into several main classes which are modeling, optimization, forecast, decision support, control and diagnosis. The specific applications range from single large scale solutions (e.g. steel production and processing) to solutions with an appreciable multiplication factor (e.g. pulp production, water purification) and solutions for mass products (e.g. decoding of TV signals).
Averaging Regularized Estimators

We compare the performance of averaged regularized estimators. We show that the improvement in performance that can be achieved by averaging depends critically on the degree of regularization which is used in training the individual estimators. We compare four different averaging approaches: simple averaging, bagging, variance-based weighting, and variance-based bagging. In any of the averaging methods, the greatest degree of improvementif compared to the individual estimatorsis achieved if no or only a small degree of regularization is used. Here, variance-based weighting and variance-based bagging are superior to simple averaging or bagging. Our experiments indicate that better performance for both individual estimators and for averaging is achieved in combination with regularization. With increasing degrees of regularization, the two bagging-based approaches (bagging and variance-based bagging) outperform the individual estimators, simple averaging, and variance-based weighting. Bagging and variance-based bagging seem to be the overall best combining methods over a wide range of degrees of regularization.
Nonlinear Markov Networks for Continuous Variables

We address the problem of learning structure in nonlinear Markov networks with continuous variables. This can be viewed as non-Gaussian multidimensional density estimation exploiting certain conditional independencies in the variables. Markov networks are a graphical way of describing conditional independencies well suited to model relationships which do not exhibit a natural causal ordering. We use neural network structures to model the quantitative relationships between variables. The main focus in this paper will be on learning the structure for the purpose of gaining insight into the underlying process. Using two data sets we show that interesting structures can be found using our approach. Inference will be briefly addressed.
A Solution for Missing Data in Recurrent Neural Networks with an Application to Blood Glucose Prediction

We consider neural network models for stochastic nonlinear dynamical systems where measurements of the variable of interest are only available at irregular intervals i.e. most realizations are missing. Difficulties arise since the solutions for prediction and maximum likelihood learning with missing data lead to complex integrals, which even for simple cases cannot be solved analytically. In this paper we propose a specific combination of a nonlinear recurrent neural predictive model and a linear error model which leads to tractable prediction and maximum likelihood adaptation rules. In particular, the recurrent neural network can be trained using the real-time recurrent learning rule and the linear error model can be trained by an EM adaptation rule, implemented using forward-backward Kalman filter equations. The model is applied to predict the glucose/insulin metabolism of a diabetic patient where blood glucose measurements are only available a few times a day at irregular intervals. The new model shows considerable improvement with respect to both recurrent neural networks trained with teacher forcing or in a free running mode and various linear models.
Missing and noisy data in nonlinear time-series prediction

We discuss the issue of missing and noisy data in nonlinear time-series prediction. We derive fundamental equations both for prediction and for training. Our discussion shows that if measurements are noisy or missing, treating the time series as a static input/output mapping problem (the usual time-delay neural network approach) is suboptimal. We describe approximations of the solutions which are based on stochastic simulations. A special case is K-step prediction in which a one-step predictor is iterated K times. Our solutions provide error bars for prediction with missing or noisy data and for K-step prediction. Using the K-step iterated logistic map as an example, we show that the proposed solutions are a considerable improvement over simple heuristic solutions. Using our formalism we derive algorithms for training recurrent networks, for control of stochastic systems and for reinforcement learning problems.
Improved Gaussian Mixture Density Estimates Using Bayesian Penalty Terms and Network Averaging

We compare two regularization methods which can be used to improve the generalization capabilities of Gaussian mixture density estimates. The first method uses a Bayesian prior on the parameter space. We derive EM (Expectation Maximization) update rules which maximize the a posterior parameter probability. In the second approach we apply ensemble averaging to density estimation. This includes Breiman's "bagging", which recently has been found to produce impressive results for classification networks.
Retrieving lightly annotated images using image similarities

Users' search needs are often represented by words and images are retrieved according to such textual queries. Annotation words assigned to the stored images are most useful to connect queries to the images. However, due to annotation cost, quite limited amount of annotation words are available in many cases. When annotations are not given at all, there needs to be some techniques that assign annotations automatically. When only a few annotation words are given to each image (lightly annotated), there need to be some enhancement techniques that best use the available annotations. We address the later problem by estimating word associations to fill in the lexical gap between queries and annotations. The model of word associations can be learned from the data. However, since images are only lightly annotated, their sparseness in computing word associations becomes crucial. To compensate the sparseness, we propose a novel data exploration technique in which image similarities contribute to the estimation of word associations on the assumption that similar images have similar semantic concepts. We experimentally show the potential benefit of our approach.
A Hybrid Generative/Discriminative Approach to Semi-Supervised Classifier Design

Semi-supervised classifier design that simultaneously utilizes both labeled and unlabeled samples is a major research issue in machine learning. Existing semisupervised learning methods belong to either generative or discriminative approaches. This paper focuses on probabilistic semi-supervised classifier design and presents a hybrid approach to take advantage of the generative and discriminative approaches. Our formulation considers a generative model trained on labeled samples and a newly introduced bias correction model. Both models belong to the same model family. The proposed hybrid model is constructed by combining both generative and bias correction models based on the maximum entropy principle. The parameters of the bias correction model are estimated by using training data, and combination weights are estimated so that labeled samples are correctly classified. We use naive Bayes models as the generative models to apply the hybrid approach to text classification problems. In our experimental results on three text data sets, we confirmed that the proposed method significantly outperformed pure generative and discriminative methods when the classification performances of the both methods were comparable.
A Classifier Design Based on Combining Multiple Components by Maximum Entropy Principle

Designing high performance classifiers for structured data consisting of multiple components is an important and challenging research issue in the field of machine learning. Although the main component of structured data plays an important role when designing classifiers, additional components may contain beneficial information for classification. This paper focuses on a probabilistic classifier design for multiclass classification based on the combination of main and additional components. Our formulation separately considers component generative models and constructs the classifier by combining these trained models based on the maximum entropy principle. We use naive Bayes models as the component generative models for text and link components so that we can apply our classifier design to document and web page classification problems. Our experimental results for three test collections confirmed that the proposed method effectively combined the main and additional components to improve classification performance.
Selection of Shared-State Hidden Markov Model Structure Using Bayesian Criterion

A Shared-State Hidden Markov Model (SS-HMM) has been widely used as an acoustic model in speech recognition. In this paper, we propose a method for constructing SS-HMMs within a practical Bayesian framework. Our method derives the Bayesian model selection criterion for the SS-HMM based on the variational Bayesian approach. The appropriate phonetic decision tree structure of the SS-HMM is found by using the Bayesian criterion. Unlike the conventional asymptotic criteria, this criterion is applicable even in the case of an insufficient amount of training data. The experimental results on isolated word recognition demonstrate that the proposed method does not require the tuning parameter that must be tuned according to the amount of training data, and is useful for selecting the appropriate SS-HMM structure for practical use. key words: speech recognition, shared-state HMM, model structure selection, variational Bayes, Bayesian criterion
Variational bayesian estimation and clustering for speech recognition

In this paper, we propose variational Bayesian estimation and clustering for speech recognition (VBEC), which is based on the variational Bayesian (VB) approach. VBEC is a total Bayesian framework: all speech recognition procedures (acoustic modeling and speech classification) are based on VB posterior distribution, unlike the maximum likelihood (ML) approach based on ML parameters. The total Bayesian framework generates two major Bayesian advantages over the ML approach for the mitigation of over-training effects, as it can select an appropriate model structure without any data set size condition, and can classify categories robustly using a predictive posterior distribution. By using these advantages, VBEC: 1) allows the automatic construction of acoustic models along two separate dimensions, namely, clustering triphone hidden Markov model states and determining the number of Gaussians and 2) enables robust speech classification, based on Bayesian predictive classification using VB posterior distributions. The capabilities of the VBEC functions were confirmed in large vocabulary continuous speech recognition experiments for read and spontaneous speech tasks. The experiments confirmed that VBEC automatically constructed accurate acoustic models and robustly classified speech, i.e., totally mitigated the over-training effects with high word accuracies due to the VBEC functions.
Modeling network growth with directional attachment and communities

Aiming to construct a more precise probabilistic model for the short-term growth process of real-world growing networks such as the Web, we propose a new network growth model and its learning algorithm. Unlike the conventional network growth models, we incorporate directional attachment and community structure for this purpose. We first show that the proposed model also exhibits a degree distribution with a power-law tail. In experiments using real Web data, we demonstrate that prediction of the dynamics of a given growing network can be improved by incorporating directional attachment and community structure. In experiments using synthetic data, we also demonstrate that prediction performance can definitely be improved by incorporating community structure.
Extended Tied-Mixture HMMs for Both Labeled and Unlabeled Time Series Data

Classifier design often faces a lack of sufficient labeled data because the class labels are identified by experienced analysts and therefore collecting labeled data often costs much. To mitigate this problem, several learning methods have been proposed to effectively use unlabeled data that can be inexpensively collected. In these methods, however, only static data have been considered; time series unlabeled data cannot be dealt with by these methods. Focusing on Hidden Markov Models (HMMs), in this paper we first present an extension of HMMs, named Extended Tied-Mixture HMMs (ETM-HMMs), in which both labeled and unlabeled time series data can be utilized simultaneously. We also formally derive a learning algorithm for the ETM-HMMs based on the maximum likelihood framework. Experimental results using synthetic and real time series data show that we can obtain a certainly better classification accuracy when unlabeled time series data are added to labeled training data than the case only labeled data are used.
Document Clustering at NTCIR-4 Workshop: Limiting Search Space of the K-Means Method Using Word Occurrence

In this paper, we propose a new document clustering method based on the K-means method (kmeans). In our method, we allow only finite candidate vectors to be representative vectors of kmeans. We also propose a method for constructing these candidate vectors using documents that have the same word. We participated in NTCIR-4 WEB Task D (Topic Classification Task) and experimentally compared our method with kmeans on this task.
BAYESIAN ACOUSTIC MODELING FOR SPONTANEOUS SPEECH RECOGNITION

Accurate acoustic model construction for spontaneous speech recognition requires that various speech fluctuation factors such as speaking variations and speaker variances are dealt with. The Bayesian approach has advantages for the speech fluctuation modeling because it enables an appropriate model selection for given speech data, unlike the maximum likelihood approach. However, the Bayesian approach includes complicated integrals that have prevented it from being realized in a large-scale task such as spontaneous speech recognition. In this paper, we apply a practical Bayesian framework: Variational Bayesian Estimation and Clustering for speech recognition (VBEC), to spontaneous speech recognition. In particular, we focus on the selection of an appropriate acoustic model structure. The effectiveness of the VBEC is shown through recognition experiments using real spontaneous speech data.
Extended Parametric Mixture Model for Robust Multi-labeled Text Categorization

In this paper, we propose an extended variant of the parametric mixture model (PMM), which we recently proposed for multi-class and multi-labeled text categorization. In the extended model (EPMM), latent categories are incorporated in the PMM so that it can adaptively control the model’s flexibility according to the data while maintaining the validity of parametric mixture assumption of the original PMM. In the multi-label setting, we experimentally compare a Naive Bayes classifier (NB), Support Vector Machines (SVM), PMM and EPMM for their robustness against classification noise as well as classification performance. The results show that EPMM provides higher classification performance than PMM while keeping the advantage of greater robustness against noise than that by NB and SVM.
Application of variational Bayesian estimation and clustering to acoustic model adaptation

We apply Variational Bayesian estimation and clustering for speech recognition (VBEC) to an acoustic model adaptation. VBEC can estimate parameter posteriors even when a model includes hidden variables, by using variational Bayesian approach. In addition, VBEC can select an appropriate model structure in clustering triphone states, according to the amount of available adaptation data. Unlike a conventional Bayesian method such as maximum a posteriori (MAP), VBEC is useful even in the case of small amounts of data, because the amount of data per one Gaussian increases due to the model structure selection, and over-training is suppressed. We conduct an off-line supervised adaptation experiment on isolated word recognition, and show the advantage of the proposed method over the conventional method, especially when dealing with small amounts of adaptation data.
Use of unlabeled time series data in hidden Markov models

In this paper, the authors propose a method of assigning probabilistic class labels to unlabeled time-series data that can be collected inexpensively to use together with labeled data as training data in order to improve the performance of hidden Markov models (HMM) as classifiers. Conventional methods of using unlabeled data according to probabilistic class labels were only for static data. They could not deal with time-series data, which are used with HMMs. Therefore, the authors extended the conventional methods as follows. First, they formed extended tied-mixture hidden Markov models (ETM-HMMs) by mixing multiple HMMs that exist independently for each class. Then, they introduced the extended Baum-Welch (EBW) algorithm as the learning algorithm from time-series data in which labeled and unlabeled data coexist in the ETM-HMMs. The authors then assumed a situation in which there existed insufficient labeled training data and used the proposed method to perform classification experiments for Japanese sign language data and voice data. When unlabeled data were added to the labeled training data according to the proposed method, the classification error rate for Japanese sign language signs improved from 38.7% to 30.4%. Also, by making the model more detailed in addition to adding unlabeled data, the classification error rate for phonemes improved from 69.1% to 52.9%.
Adaptive clustering using modular learning architecture

We present a clustering method for data that are continuously distributed within each class and where there is no overlap between the classes. With the method the data are clustered by on-line processing, starting at a point where the number of classes is unknown. The proposed method uses ideas derived from statistical mechanics and applies them to a modular learning architecture: learning progresses by regarding the data as particles and regarding each module as a space in which the particles are distributed. The learning method estimates the energy state that matches the particle distribution. As a result of learning, continuously distributed data are clustered in the same module. At the same time, certain modules appear that contain no data. Consequently, the number of classes is estimated by defining each module containing data as a class. This paper describes the proposed method in detail and shows the results of clustering experiments undertaken using two-dimensional artificial data and facial images.
Exploitation of Unlabeled Sequences in Hidden Markov Models

This paper presents a method for effectively using unlabeled sequential data in the learning of hidden Markov models (HMMs). With the conventional approach, class labels for unlabeled data are assigned deterministically by HMMs learned from labeled data. Such labeling often becomes unreliable when the number of labeled data is small. We propose an extended Baum-Welch (EBW) algorithm in which the labeling is undertaken probabilistically and iteratively so that the labeled and unlabeled data likelihoods are improved. Unlike the conventional approach, the EBW algorithm guarantees convergence to a local maximum of the likelihood. Experimental results on gesture data and speech data show that when labeled training data are scarce, by using unlabeled data, the EBW algorithm improves the classification performance of HMMs more robustly than the conventional naive labeling (NL) approach.
Cross-Entropy Directed Embedding of Network Data

We present a novel approach to embedding data represented by a network into a low-dimensional Euclidean space. Unlike existing methods, the proposed method attempts to minimize an energy function based on the cross-entropy between desirable and embedded node configurations without directly utilizing pairwise distances between nodes. We also propose a natural criterion to effectively evaluate an embedded network layout in terms of how well node connectivities are preserved. Experimental results show that the proposed method provides better layouts than those produced by some of the well-known embedding methods in terms of the proposed criterion. We believe that our method produces a natural embedding of a large-scale network suitable for analyzing by manual browsing in a two- or three-dimensional Euclidean space.