Above are papers related to the following paper: [Paper_ID] Paper #32698. 	[Title] Ensemble Learning for Hidden Markov Models [Abstract] The standard method for training Hidden Markov Models optimizes a point estimate of the model parameters. This estimate, which can be viewed as the maximum of a posterior probability density over the model parameters, may be susceptible to over-fitting, and contains no indication of parameter uncertainty. Also, this maximum may be unrepresentative of the posterior probability distribution. In this paper we study a method in which we optimize an ensemble which approximates the entire posterior probability distribution. The ensemble learning algorithm requires the same resources as the traditional Baum-Welch algorithm. The traditional training algorithm for hidden Markov models is an expectation-maximization (EM) algorithm (Dempster et al. 1977) known as the Baum-Welch algorithm. It is a maximum likelihood method, or, with a simple modification, a penalized maximum likelihood method, which can be viewed as maximizing a posterior probability density over the model parameters. Recently, Hinton and van Camp (1993) developed a technique known as ensemble learning (see also MacKay (1995) for a review). Whereas maximum a posteriori methods optimize a point estimate of the parameters, in ensemble learning an ensemble is optimized, so that it approximates the entire posterior probability distribution over the parameters. The objective function that is optimized is a variational free energy (Feynman 1972) which measures the relative entropy between the approximating ensemble and the true distribution. In this paper we derive and test an ensemble learning algorithm for hidden Markov models, building on Neal Please summarise all the given information.

Above are papers related to the following paper: [Paper_ID] Paper #46468. 	[Title] A Statistical Approach to Decision Tree Modeling [Abstract] A statistical approach to decision tree modeling is described. In this approach, each decision in the tree is modeled parametrically as is the process by which an output is generated from an input and a sequence of decisions. The resulting model yields a likelihood measure of goodness of fit, allowing ML and MAP estimation techniques to be utilized. An efficient algorithm is presented to estimate the parameters in the tree. The model selection problem is presented and several alternative proposals are considered. A hidden Markov version of the tree is described for data sequences that have temporal dependencies. Please summarise all the given information.

Above are papers related to the following paper: [Paper_ID] Paper #24966. 	[Title] A Practical Bayesian Framework for Backprop Networks [Abstract] A quantitative and practical Bayesian framework is described for learning of mappings in feedforward networks. The framework makes possible: (1) objective comparisons between solutions using alternative network architectures; (2) objective stopping rules for network pruning or growing procedures; (3) objective choice of magnitude and type of weight decay terms or additive regularisers (for penalising large weights, etc.); (4) a measure of the effective number of well-determined parameters in a model; (5) quantified estimates of the error bars on network parameters and on network output; (6) objective comparisons with alternative learning and interpolation models such as splines and radial basis functions. The Bayesian `evidence' automatically embodies `Occam's razor,' penalising over-flexible and over-complex models. The Bayesian approach helps detect poor underlying assumptions in learning models. For learning models well matched to a problem, a good correlation between generalisation ability and the Bayesian evidence is obtained. This paper makes use of the Bayesian framework for regularisation and model comparison described in the companion paper `Bayesian interpolation' (MacKay, 1991a). This framework is due to Gull and Skilling (Gull, 1989a). Please summarise all the given information.

Above are papers related to the following paper: [Paper_ID] Paper #755082. 	[Title] DRAFT Cluster-Weighted Modeling for Time Series Prediction and Characterization [Abstract]  Please summarise all the given information.

Above are papers related to the following paper: [Paper_ID] Paper #1136397. 	[Title] State Reconstruction for Determining Predictability in Driven Nonlinear Acoustical Systems [Abstract]  Please summarise all the given information.

Above are papers related to the following paper: [Paper_ID] Paper #1104300. 	[Title] Efficient Stochastic Source Coding and an Application to a Bayesian Network Source Model [Abstract] Brendan J. Frey and Geoffrey E. Hinton 1997. Efficient stochastic source coding and an application to a Bayesian network source model. The Computer Journal 40, 157-165. In this paper, we introduce a new algorithm called "bits-back coding" that makes stochastic source codes efficient. For a given one-to-many source code, we show that this algorithm can actually be more efficient than the algorithm that always picks the shortest codeword. Optimal efficiency is achieved when codewords are chosen according to the Boltzmann distribution based on the codeword lengths. It turns out that a commonly used technique for determining parameters | maximum likelihood estimation | actually minimizes the bits-back coding cost when codewords are chosen according to the Boltzmann distribution. A tractable approximation to maximum likelihood estimation | the generalized expectation maximization algorithm | minimizes the bits-back coding cost. After presenting a binary Bayesian network model that assigns exponentially many codewords to each symbol, we show how a tractable approximation to the Boltzmann distribution can be used for bits-back coding. We illustrate the performance of bits-back coding using using nonsynthetic data with a binary Bayesian network source model that produces 2 60 possible codewords for each input symbol. The rate for bits-back coding is nearly one half of that obtained by picking the shortest codeword for each symbol. Please summarise all the given information.

Above are papers related to the following paper: [Paper_ID] Paper #1717. 	[Title] On Convergence Properties of the EM Algorithm for Gaussian Mixtures [Abstract] We build up the mathematical connection between the "Expectation-Maximization" (EM) algorithm and gradient-based approaches for maximum likelihood learning of finite Gaussian mixtures. We show that the EM step in parameter space is obtained from the gradient via a projection matrix P , and we provide an explicit expression for the matrix. We then analyze the convergence of EM in terms of special properties of P and provide new results analyzing the effect that P has on the likelihood surface. Based on these mathematical results, we present a comparative discussion of the advantages and disadvantages of EM and other algorithms for the learning of Gaussian mixture models. This report describes research done at the Center for Biological and Computational Learning and the Artificial Intelligence Laboratory of the Massachusetts Institute of Technology. Support for the Center is provided in part by a grant from the National Science Foundation under contract ASC-9217041. Support for the laboratory's artificial intelligence research is provided in part by the Advanced Research Projects Agency of the Department of Defense under Office of Naval Research contract N00000-00-A-0000. The authors were also supported by the HK RGC Earmarked Grant CUHK250/94E, by a grant from the McDonnell-Pew Foundation, by a grant from ATR Human Information Processing Research Laboratories, by a grant from Siemens Corporation, and by grant N00014-90-1-0777 from the Office of Naval Research. Michael I. Jordan is an NSF Presidential Young Investigator. Please summarise all the given information.

Above are papers related to the following paper: [Paper_ID] Paper #1136310. 	[Title] The Expectation-Maximization Algorithm for MAP Estimation [Abstract] The Expectation-Maximization algorithm given by Dempster et al (1977) has enjoyed considerable popularity for solving MAP estimation problems. This note gives a simple derivation of the algorithm, due to Luttrell (1994), that better illustrates the convergence properties of the algorithm and its variants. The algorithm is illustrated with two examples: pooling data from multiple noisy sources and fitting a mixture density. Please summarise all the given information.

Above are papers related to the following paper: [Paper_ID] Paper #116790. 	[Title] Sequential Update of Bayesian Network Structure [Abstract] There is an obvious need for improving the performance and accuracy of a Bayesian network as new data is observed. Because of errors in model construction and changes in the dynamics of the domains, we cannot afford to ignore the information in new data. While sequential update of parameters for a fixed structure can be accomplished using standard techniques, sequential update of network structure is still an open problem. In this paper, we investigate sequential update of Bayesian networks were both parameters and structure are expected to change. We introduce a new approach that allows for the flexible manipulation of the tradeoff between the quality of the learned networks and the amount of information that is maintained about past observations. We formally describe our approach including the necessary modifications to the scoring functions for learning Bayesian networks, evaluate its effectiveness through and empirical study, and extend it to the case of missing data. Please summarise all the given information.

Above are papers related to the following paper: [Paper_ID] Paper #239800. 	[Title] Ensemble Learning and Evidence Maximization [Abstract] Ensemble learning by variational free energy minimization is a tool introduced to neural networks by Hinton and van Camp in which learning is described in terms of the optimization of an ensemble of parameter vectors. The optimized ensemble is an approximation to the posterior probability distribution of the parameters. This tool has now been applied to a variety of statistical inference problems. In this paper I study a linear regression model with both parameters and hyper-parameters. I demonstrate that the evidence approximation for the optimization of regularization constants can be derived in detail from a free energy minimization viewpoint. Please summarise all the given information.

Above are papers related to the following paper: [Paper_ID] Paper #40124. 	[Title] Probabilistic Networks: New Models and New Methods [Abstract] In this paper I describe the implementation of a probabilistic regression model in BUGS. BUGS is a program that carries out Bayesian inference on statistical problems using a simulation technique known as Gibbs sampling. It is possible to implement surprisingly complex regression models in this environment. I demonstrate the simultaneous inference of an interpolant and an input-dependent noise level. Please summarise all the given information.

Above are papers related to the following paper: [Paper_ID] Paper #62274. 	[Title] A Comparison of New and Old Algorithms for A Mixture Estimation Problem [Abstract] We investigate the problem of estimating the proportion vector which maximizes the likelihood of a given sample for a mixture of given densities. We adapt a framework developed for supervised learning and give simple derivations for many of the standard iterative algorithms like gradient projection and EM. In this framework, the distance between the new and old proportion vectors is used as a penalty term. The square distance leads to the gradient projection update, and the relative entropy to a new update which we call the exponentiated gradient update (EG ). Curiously, when a second order Taylor expansion of the relative entropy is used, we arrive at an update EM which, for = 1, gives the usual EM update. Experimentally, both the EM -update and the EG -update for &gt; 1 outperform the EM algorithm and its variants. We also prove a polynomial bound on the rate of convergence of the EG algorithm. Please summarise all the given information.

Above are papers related to the following paper: [Paper_ID] Paper #1140547. 	[Title] Localist Attractor Networks [Abstract] Attractor networks, which map a continuous input space to a discrete output space, are useful for pattern completion, cleaning up noisy or missing features in an input. However, designing a net to have a given set of attractors is notoriously tricky; training procedures are CPU intensive and often produce spurious attractors and ill-conditioned attractor basins. These difficulties occur because each connection in the network participates in the encoding of multiple attractors. We describe an alternative formulation of attractor networks in which the encoding of knowledge is local, not distributed. Although localist attractor nets have similar dynamics to their distributed counterparts, they are much easier to work with and interpret. We propose a statistical formulation of localist attractor net dynamics, which yields a convergence proof and a mathematical interpretation of model parameters. We present simulation experiments that explore the behavior of lo-calist attractor nets, showing that they produce a gang effectthe presence of an attractor enhances the attractor basins of neighboring attractorsand that spurious attractors occur only at points of symmetry in state space. Please summarise all the given information.

Above are papers related to the following paper: [Paper_ID] Paper #62417. 	[Title] Free energy coding  In [Abstract] In this paper, we introduce a new approach to the problem of optimal compression when a source code produces multiple codewords for a given symbol. It may seem that the most sensible codeword to use in this case is the shortest one. However, in the proposed free energy approach, random codeword selection yields an effective codeword length that can be less than the shortest codeword length. If the random choices are Boltzmann distributed, the effective length is optimal for the given source code. The expectation-maximization parameter estimation algorithms minimize this effective codeword length. We illustrate the performance of free energy coding on a simple problem where a compression factor of two is gained by using the new method. Please summarise all the given information.

Above are papers related to the following paper: [Paper_ID] Paper #35922. 	[Title] Mean Field Theory for Sigmoid Belief Networks [Abstract] We develop a mean field theory for sigmoid belief networks based on ideas from statistical mechanics. Our mean field theory provides a tractable approximation to the true probability distribution in these networks; it also yields a lower bound on the likelihood of evidence. We demonstrate the utility of this framework on a benchmark problem in statistical pattern recognition|the classification of handwritten digits. Please summarise all the given information.

Above are papers related to the following paper: [Paper_ID] Paper #71904. 	[Title] Free Energy Minimization Algorithm for Decoding and Cryptanalysis three binary vectors: s of length N [Abstract] where A is a binary matrix. Our task is to infer s given z and A, and given assumptions about the statistical properties of s and n. This problem arises in the decoding of a noisy signal transmitted using a linear code A, and in the inference of the sequence of a linear feedback shift register (LFSR) from noisy observations [1, 2]. Please summarise all the given information.

Above are papers related to the following paper: [Paper_ID] Paper #95718. 	[Title] Developments in Probabilistic Modelling with Neural Networks|Ensemble Learning [Abstract] In this paper I give a review of ensemble learning using a simple example. Please summarise all the given information.

Above are papers related to the following paper: [Paper_ID] Paper #95719. 	[Title] Choice of Basis for Laplace Approximation [Abstract] Maximum a posteriori optimization of parameters and the Laplace approximation for the marginal likelihood are both basis-dependent methods. This note compares two choices of basis for models parameterized by probabilities, showing that it is possible to improve on the traditional choice, the probability simplex, by transforming to the `softmax' basis. Please summarise all the given information.

Above are papers related to the following paper: [Paper_ID] Paper #1121254. 	[Title] On Structured Variational Approximations [Abstract] The problem of approximating a probability distribution occurs frequently in many areas of applied mathematics, including statistics, communication theory, machine learning, and the theoretical analysis of complex systems such as neural networks. Saul and Jordan (1996) have recently proposed a powerful method for efficiently approximating probability distributions known as structured variational approximations. In structured variational approximations, exact algorithms for probability computation on tractable substructures are combined with variational methods to handle the interactions between the substructures which make the system as a whole intractable. In this note, I present a mathematical result which can simplify the derivation of struc tured variational approximations in the exponential family of distributions. Please summarise all the given information.

Above are papers related to the following paper: [Paper_ID] Paper #390889. 	[Title] A HIERARCHICAL COMMUNITY OF EXPERTS [Abstract] We describe a directed acyclic graphical model that contains a hierarchy of linear units and a mechanism for dynamically selecting an appropriate subset of these units to model each observation. The non-linear selection mechanism is a hierarchy of binary units each of which gates the output of one of the linear units. There are no connections from linear units to binary units, so the generative model can be viewed as a logistic belief net (Neal 1992) which selects a skeleton linear model from among the available linear units. We show that Gibbs sampling can be used to learn the parameters of the linear and binary units even when the sampling is so brief that the Markov chain is far from equilibrium. Please summarise all the given information.

Above are papers related to the following paper: [Paper_ID] Paper #4335. 	[Title] A VIEW OF THE EM ALGORITHM THAT JUSTIFIES INCREMENTAL, SPARSE, AND OTHER VARIANTS [Abstract] The EM algorithm performs maximum likelihood estimation for data in which some variables are unobserved. We present a function that resembles negative free energy and show that the M step maximizes this function with respect to the model parameters and the E step maximizes it with respect to the distribution over the unobserved variables. From this perspective, it is easy to justify an incremental variant of the EM algorithm in which the distribution for only one of the unobserved variables is recalculated in each E step. This variant is shown empirically to give faster convergence in a mixture estimation problem. A variant of the algorithm that exploits sparse conditional distributions is also described, and a wide range of other variant algorithms are also seen to be possible. Please summarise all the given information.

Above are papers related to the following paper: [Paper_ID] Paper #19697. 	[Title] Keeping Neural Networks Simple by Minimizing the Description Length of the Weights [Abstract] Supervised neural networks generalize well if there is much less information in the weights than there is in the output vectors of the training cases. So during learning, it is important to keep the weights simple by penalizing the amount of information they contain. The amount of information in a weight can be controlled by adding Gaussian noise and the noise level can be adapted during learning to optimize the trade-off between the expected squared error of the network and the amount of information in the weights. We describe a method of computing the derivatives of the expected squared error and of the amount of information in the noisy weights in a network that contains a layer of non-linear hidden units. Provided the output units are linear, the exact derivatives can be computed efficiently without time-consuming Monte Carlo simulations. The idea of minimizing the amount of information that is required to communicate the weights of a neural network leads to a number of interesting schemes for encoding the weights. Please summarise all the given information.

Above are papers related to the following paper: [Paper_ID] Paper #131318. 	[Title] FLAT MINIMA Neural Computation 9(1):1-42 (1997) [Abstract] We present a new algorithm for finding low complexity neural networks with high generalization capability. The algorithm searches for a "flat" minimum of the error function. A flat minimum is a large connected region in weight-space where the error remains approximately constant. An MDL-based, Bayesian argument suggests that flat minima correspond to "simple" networks and low expected overfitting. The argument is based on a Gibbs algorithm variant and a novel way of splitting generalization error into underfitting and overfitting error. Unlike many previous approaches, ours does not require Gaussian assumptions and does not depend on a "good" weight prior instead we have a prior over input/output functions, thus taking into account net architecture and training set. Although our algorithm requires the computation of second order derivatives, it has backprop's order of complexity. Automatically, it effectively prunes units, weights, and input lines. Various experiments with feedforward and recurrent nets are described. In an application to stock market prediction, flat minimum search outperforms (1) conventional backprop, (2) weight decay, (3) "optimal brain surgeon" / "optimal brain damage". We also provide pseudo code of the algorithm (omitted from the NC-version). Please summarise all the given information.

Paper #32698 has the title `Ensemble Learning for Hidden Markov Models  
` and the abstract `The standard method for training Hidden Markov Models optimizes a point estimate of the model parameters. This estimate, which can be viewed as the maximum of a posterior probability density over the model parameters, may be susceptible to over-fitting, and contains no indication of parameter uncertainty. Also, this maximum may be unrepresentative of the posterior probability distribution. In this paper we study a method in which we optimize an ensemble which approximates the entire posterior probability distribution. The ensemble learning algorithm requires the same resources as the traditional Baum-Welch algorithm. The traditional training algorithm for hidden Markov models is an expectation-maximization (EM) algorithm (Dempster et al. 1977) known as the Baum-Welch algorithm. It is a maximum likelihood method, or, with a simple modification, a penalized maximum likelihood method, which can be viewed as maximizing a posterior probability density over the model parameters. Recently, Hinton and van Camp (1993) developed a technique known as ensemble learning (see also MacKay (1995) for a review). Whereas maximum a posteriori methods optimize a point estimate of the parameters, in ensemble learning an ensemble is optimized, so that it approximates the entire posterior probability distribution over the parameters. The objective function that is optimized is a variational free energy (Feynman 1972) which measures the relative entropy between the approximating ensemble and the true distribution. In this paper we derive and test an ensemble learning algorithm for hidden Markov models, building on Neal 
`.
Question: Which category should Paper #32698 be classified as ?
You can select one from ['Neural_Networks', 'Case_Based', 'Theory', 'Reinforcement_Learning', 'Probabilistic_Methods', 'Rule_Learning', 'Genetic_Algorithms'].


Paper #32698 has the title `Ensemble Learning for Hidden Markov Models  
` and the abstract `The standard method for training Hidden Markov Models optimizes a point estimate of the model parameters. This estimate, which can be viewed as the maximum of a posterior probability density over the model parameters, may be susceptible to over-fitting, and contains no indication of parameter uncertainty. Also, this maximum may be unrepresentative of the posterior probability distribution. In this paper we study a method in which we optimize an ensemble which approximates the entire posterior probability distribution. The ensemble learning algorithm requires the same resources as the traditional Baum-Welch algorithm. The traditional training algorithm for hidden Markov models is an expectation-maximization (EM) algorithm (Dempster et al. 1977) known as the Baum-Welch algorithm. It is a maximum likelihood method, or, with a simple modification, a penalized maximum likelihood method, which can be viewed as maximizing a posterior probability density over the model parameters. Recently, Hinton and van Camp (1993) developed a technique known as ensemble learning (see also MacKay (1995) for a review). Whereas maximum a posteriori methods optimize a point estimate of the parameters, in ensemble learning an ensemble is optimized, so that it approximates the entire posterior probability distribution over the parameters. The objective function that is optimized is a variational free energy (Feynman 1972) which measures the relative entropy between the approximating ensemble and the true distribution. In this paper we derive and test an ensemble learning algorithm for hidden Markov models, building on Neal 
`.
Question: Which category should Paper #32698 be classified as ?
You can select one from ['Neural_Networks', 'Case_Based', 'Theory', 'Reinforcement_Learning', 'Probabilistic_Methods', 'Rule_Learning', 'Genetic_Algorithms'].

This paper should be classified into: Probabilistic_Methods