Above are papers related to the following paper: [Paper_ID] Paper #1123576. 	[Title] Density Networks and their Application to Protein Modelling [Abstract] I define a latent variable model in the form of a neural network for which only target outputs are specified; the inputs are unspecified. Although the inputs are missing, it is still possible to train this model by placing a simple probability distribution on the unknown inputs and maximizing the probability of the data given the parameters. The model can then discover for itself a description of the data in terms of an underlying latent variable space of lower dimensionality. I present preliminary results of the application of these models to protein data. Please summarise all the given information.

Above are papers related to the following paper: [Paper_ID] Paper #1135746. 	[Title] Learning Markov chains with variable memory length from noisy output [Abstract] The problem of modeling complicated data sequences, such as DNA or speech, often arises in practice. Most of the algorithms select a hypothesis from within a model class assuming that the observed sequence is the direct output of the underlying generation process. In this paper we consider the case when the output passes through a memoryless noisy channel before observation. In particular, we show that in the class of Markov chains with variable memory length, learning is affected by factors, which, despite being super-polynomial, are still small in some practical cases. Markov models with variable memory length, or probabilistic finite suffix automata, were introduced in learning theory by Ron, Singer and Tishby who also described a polynomial time learning algorithm [11, 12]. We present a modification of the algorithm which uses a noise-corrupted sample and has knowledge of the noise structure. The same algorithm is still viable if the noise is not known exactly but a good estimation is available. Finally, some experimental results are presented for removing noise from corrupted English text, and to measure how the performance of the learning algorithm is affected by the size of the noisy sample and the noise rate. Please summarise all the given information.

Above are papers related to the following paper: [Paper_ID] Paper #1131267. 	[Title] Parallel Markov chain Monte Carlo sampling. [Abstract] Markov chain Monte Carlo (MCMC) samplers have proved remarkably popular as tools for Bayesian computation. However, problems can arise in their application when the density of interest is high dimensional and strongly correlated. In these circumstances the sampler may be slow to traverse the state space and mixing is poor. In this article we offer a partial solution to this problem. The state space of the Markov chain is augmented to accommodate multiple chains in parallel. Updates to individual chains are based around a genetic style crossover operator acting on `parent' states drawn from the population of chains. This process makes efficient use of gradient information implicitly encoded within the distribution of states across the population. Empirical studies support the claim that the crossover operator acting on a parallel population of chains improves mixing. This is illustrated with an example of sampling a high dimensional posterior probability density from a complex predictive model. By adopting a latent variable approach the methodology is extended to deal with variable selection and model averaging in high dimensions. This is illustrated with an example of knot selection for a spline interpolant. Please summarise all the given information.

Above are papers related to the following paper: [Paper_ID] Paper #24966. 	[Title] A Practical Bayesian Framework for Backprop Networks [Abstract] A quantitative and practical Bayesian framework is described for learning of mappings in feedforward networks. The framework makes possible: (1) objective comparisons between solutions using alternative network architectures; (2) objective stopping rules for network pruning or growing procedures; (3) objective choice of magnitude and type of weight decay terms or additive regularisers (for penalising large weights, etc.); (4) a measure of the effective number of well-determined parameters in a model; (5) quantified estimates of the error bars on network parameters and on network output; (6) objective comparisons with alternative learning and interpolation models such as splines and radial basis functions. The Bayesian `evidence' automatically embodies `Occam's razor,' penalising over-flexible and over-complex models. The Bayesian approach helps detect poor underlying assumptions in learning models. For learning models well matched to a problem, a good correlation between generalisation ability and the Bayesian evidence is obtained. This paper makes use of the Bayesian framework for regularisation and model comparison described in the companion paper `Bayesian interpolation' (MacKay, 1991a). This framework is due to Gull and Skilling (Gull, 1989a). Please summarise all the given information.

Above are papers related to the following paper: [Paper_ID] Paper #1129608. 	[Title] A New Look at Tree Models for Multiple Sequence Alignment [Abstract] Evolutionary trees are frequently used as the underlying model in the design of algorithms, optimization criteria and software packages for multiple sequence alignment (MSA). In this paper, we reexamine the suitability of trees as a universal model for MSA in light of the broad range of biological questions that MSA's are used to address. A tree model consists of a tree topology and a model of accepted mutations along the branches. After surveying the major applications of MSA, examples from the molecular biology literature are used to illustrate situations in which this tree model fails. This occurs when the relationship between residues in a column cannot be described by a tree; for example, in some structural and functional applications of MSA. It also occurs in situations, such as lateral gene transfer, where an entire gene cannot be modeled by a unique tree. In cases of nonparsimonous data or convergent evolution, it may be difficult to find a consistent mutational model. We hope that this survey will promote dialogue between biologists and computer scientists, leading to more biologically realistic research on MSA. Please summarise all the given information.

Above are papers related to the following paper: [Paper_ID] Paper #1152904. 	[Title] GIBBS-MARKOV MODELS [Abstract] In this paper we present a framework for building probabilistic automata parameterized by context-dependent probabilities. Gibbs distributions are used to model state transitions and output generation, and parameter estimation is carried out using an EM algorithm where the M-step uses a generalized iterative scaling procedure. We discuss relations with certain classes of stochastic feedforward neural networks, a geometric interpretation for parameter estimation, and a simple example of a statistical language model constructed using this methodology. Please summarise all the given information.

Above are papers related to the following paper: [Paper_ID] Paper #1131149. 	[Title] Least Absolute Shrinkage is Equivalent to Quadratic Penalization [Abstract] Adaptive ridge is a special form of ridge regression, balancing the quadratic penalization on each parameter of the model. This paper shows the equivalence between adaptive ridge and lasso (least absolute shrinkage and selection operator). This equivalence states that both procedures produce the same estimate. Least absolute shrinkage can thus be viewed as a particular quadratic penalization. From this observation, we derive an EM algorithm to compute the lasso solution. We finally present a series of applications of this type of algorithm in regres sion problems: kernel regression, additive modeling and neural net training. Please summarise all the given information.

Above are papers related to the following paper: [Paper_ID] Paper #202639. 	[Title] PREDICTING SUNSPOTS AND EXCHANGE RATES WITH CONNECTIONIST NETWORKS [Abstract] We investigate the effectiveness of connectionist networks for predicting the future continuation of temporal sequences. The problem of overfitting, particularly serious for short records of noisy data, is addressed by the method of weight-elimination: a term penalizing network complexity is added to the usual cost function in back-propagation. The ultimate goal is prediction accuracy. We analyze two time series. On the benchmark sunspot series, the networks outperform traditional statistical approaches. We show that the network performance does not deteriorate when there are more input units than needed. Weight-elimination also manages to extract some part of the dynamics of the notoriously noisy currency exchange rates and makes the network solution interpretable. Please summarise all the given information.

Above are papers related to the following paper: [Paper_ID] Paper #1152272. 	[Title] GIBBS-MARKOV MODELS [Abstract] In this paper we present a framework for building probabilistic automata parameterized by context-dependent probabilities. Gibbs distributions are used to model state transitions and output generation, and parameter estimation is carried out using an EM algorithm where the M-step uses a generalized iterative scaling procedure. We discuss relations with certain classes of stochastic feedforward neural networks, a geometric interpretation for parameter estimation, and a simple example of a statistical language model constructed using this methodology. Please summarise all the given information.

Above are papers related to the following paper: [Paper_ID] Paper #145176. 	[Title] Bayesian Non-linear Modelling for the Prediction Competition [Abstract] The 1993 energy prediction competition involved the prediction of a series of building energy loads from a series of environmental input variables. Non-linear regression using `neural networks' is a popular technique for such modeling tasks. Since it is not obvious how large a time-window of inputs is appropriate, or what preprocessing of inputs is best, this can be viewed as a regression problem in which there are many possible input variables, some of which may actually be irrelevant to the prediction of the output variable. Because a finite data set will show random correlations between the irrelevant inputs and the output, any conventional neural network (even with reg-ularisation or `weight decay') will not set the coefficients for these junk inputs to zero. Thus the irrelevant variables will hurt the model's performance. The Automatic Relevance Determination (ARD) model puts a prior over the regression parameters which embodies the concept of relevance. This is done in a simple and `soft' way by introducing multiple regularisation constants, one associated with each input. Using Bayesian methods, the regularisation constants for junk inputs are automatically inferred to be large, preventing those inputs from causing significant overfitting. Please summarise all the given information.

Above are papers related to the following paper: [Paper_ID] Paper #1105433. 	[Title] Priors for Infinite Networks [Abstract] Technical Report CRG-TR-94-1 Department of Computer Science University of Toronto 10 King's College Road Toronto, Canada M5S 1A4 Abstract Bayesian inference begins with a prior distribution for model parameters that is meant to capture prior beliefs about the relationship being modeled. For multilayer perceptron networks, where the parameters are the connection weights, the prior lacks any direct meaning | what matters is the prior over functions computed by the network that is implied by this prior over weights. In this paper, I show that priors over weights can be defined in such a way that the corresponding priors over functions reach reasonable limits as the number of hidden units in the network goes to infinity. When using such priors, there is thus no need to limit the size of the network in order to avoid "overfitting". The infinite network limit also provides insight into the properties of different priors. A Gaussian prior for hidden-to-output weights results in a Gaussian process prior for functions, which can be smooth, Brownian, or fractional Brownian, depending on the hidden unit activation function and the prior for input-to-hidden weights. Quite different effects can be obtained using priors based on non-Gaussian stable distributions. In networks with more than one hidden layer, a combination of Gaussian and non-Gaussian priors appears most interesting. Please summarise all the given information.

Above are papers related to the following paper: [Paper_ID] Paper #124828. 	[Title] Ensemble Training: Some Recent Experiments with Postal Zip Data [Abstract] Recent findings suggest that a classification scheme based on an ensemble of networks is an effective way to address overfitting. We study optimal methods for training an ensemble of networks. Some recent experiments on Postal Zip-code character data suggest that weight decay may not be an optimal method for controlling the variance of a classifier. Please summarise all the given information.

Above are papers related to the following paper: [Paper_ID] Paper #92065. 	[Title] ADAPTIVE REGULARIZATION [Abstract] Regularization, e.g., in the form of weight decay, is important for training and optimization of neural network architectures. In this work we provide a tool based on asymptotic sampling theory, for iterative estimation of weight decay parameters. The basic idea is to do a gradient descent in the estimated generalization error with respect to the regularization parameters. The scheme is implemented in our Designer Net framework for network training and pruning, i.e., is based on the diagonal Hessian approximation. The scheme does not require essential computational overhead in addition to what is needed for training and pruning. The viability of the approach is demonstrated in an experiment concerning prediction of the chaotic Mackey-Glass series. We find that the optimized weight decays are relatively large for densely connected networks in the initial pruning phase, while they decrease as pruning proceeds. Please summarise all the given information.

Above are papers related to the following paper: [Paper_ID] Paper #1152162. 	[Title] GIBBS-MARKOV MODELS [Abstract] In this paper we present a framework for building probabilistic automata parameterized by context-dependent probabilities. Gibbs distributions are used to model state transitions and output generation, and parameter estimation is carried out using an EM algorithm where the M-step uses a generalized iterative scaling procedure. We discuss relations with certain classes of stochastic feedforward neural networks, a geometric interpretation for parameter estimation, and a simple example of a statistical language model constructed using this methodology. Please summarise all the given information.

Above are papers related to the following paper: [Paper_ID] Paper #10531. 	[Title] Using Dirichlet Mixture Priors to Derive Hidden Markov Models for Protein Families [Abstract] A Bayesian method for estimating the amino acid distributions in the states of a hidden Markov model (HMM) for a protein family or the columns of a multiple alignment of that family is introduced. This method uses Dirichlet mixture densities as priors over amino acid distributions. These mixture densities are determined from examination of previously constructed HMMs or multiple alignments. It is shown that this Bayesian method can improve the quality of HMMs produced from small training sets. Specific experiments on the EF-hand motif are reported, for which these priors are shown to produce HMMs with higher likelihood on unseen data, and fewer false positives and false negatives in a database search task. Please summarise all the given information.

Above are papers related to the following paper: [Paper_ID] Paper #43698. 	[Title] Dirichlet Mixtures: A Method for Improving Detection of Weak but Significant Protein Sequence Homology [Abstract] UCSC Technical Report UCSC-CRL-96-09 Abstract This paper presents the mathematical foundations of Dirichlet mixtures, which have been used to improve database search results for homologous sequences, when a variable number of sequences from a protein family or domain are known. We present a method for condensing the information in a protein database into a mixture of Dirichlet densities. These mixtures are designed to be combined with observed amino acid frequencies, to form estimates of expected amino acid probabilities at each position in a profile, hidden Markov model, or other statistical model. These estimates give a statistical model greater generalization capacity, such that remotely related family members can be more reliably recognized by the model. Dirichlet mixtures have been shown to outperform substitution matrices and other methods for computing these expected amino acid distributions in database search, resulting in fewer false positives and false negatives for the families tested. This paper corrects a previously published formula for estimating these expected probabilities, and contains complete derivations of the Dirichlet mixture formulas, methods for optimizing the mixtures to match particular databases, and suggestions for efficient implementation. Please summarise all the given information.

Above are papers related to the following paper: [Paper_ID] Paper #31927. 	[Title] A Generalized Hidden Markov Model for the Recognition of Human Genes in DNA [Abstract] We present a statistical model of genes in DNA. A Generalized Hidden Markov Model (GHMM) provides the framework for describing the grammar of a legal parse of a DNA sequence (Stormo & Haussler 1994). Probabilities are assigned to transitions between states in the GHMM and to the generation of each nucleotide base given a particular state. Machine learning techniques are applied to optimize these probabilities using a standardized training set. Given a new candidate sequence, the best parse is deduced from the model using a dynamic programming algorithm to identify the path through the model with maximum probability. The GHMM is flexible and modular, so new sensors and additional states can be inserted easily. In addition, it provides simple solutions for integrating cardinality constraints, reading frame constraints, "indels", and homology searching. The description and results of an implementation of such a gene-finding model, called Genie, is presented. The exon sensor is a codon frequency model conditioned on windowed nucleotide frequency and the preceding codon. Two neural networks are used, as in (Brunak, Engelbrecht, & Knudsen 1991), for splice site prediction. We show that this simple model performs quite well. For a cross-validated standard test set of 304 genes [ftp://www-hgc.lbl.gov/pub/genesets] in human DNA, our gene-finding system identified up to 85% of protein-coding bases correctly with a specificity of 80%. 58% of exons were exactly identified with a specificity of 51%. Genie is shown to perform favorably compared with several other gene-finding systems. Please summarise all the given information.

Above are papers related to the following paper: [Paper_ID] Paper #641976. 	[Title] A comparison of some error estimates for neural network models  Summary [Abstract] We discuss a number of methods for estimating the standard error of predicted values from a multi-layer perceptron. These methods include the delta method based on the Hessian, bootstrap estimators, and the "sandwich" estimator. The methods are described and compared in a number of examples. We find that the bootstrap methods perform best, partly because they capture variability due to the choice of starting weights. Please summarise all the given information.

Above are papers related to the following paper: [Paper_ID] Paper #239800. 	[Title] Ensemble Learning and Evidence Maximization [Abstract] Ensemble learning by variational free energy minimization is a tool introduced to neural networks by Hinton and van Camp in which learning is described in terms of the optimization of an ensemble of parameter vectors. The optimized ensemble is an approximation to the posterior probability distribution of the parameters. This tool has now been applied to a variety of statistical inference problems. In this paper I study a linear regression model with both parameters and hyper-parameters. I demonstrate that the evidence approximation for the optimization of regularization constants can be derived in detail from a free energy minimization viewpoint. Please summarise all the given information.

Above are papers related to the following paper: [Paper_ID] Paper #194617. 	[Title] Finding Genes in DNA with a Hidden Markov Model [Abstract] This study describes a new Hidden Markov Model (HMM) system for segmenting uncharacterized genomic DNA sequences into exons, introns, and intergenic regions. Separate HMM modules were designed and trained for specific regions of DNA: exons, introns, intergenic regions, and splice sites. The models were then tied together to form a biologically feasible topology. The integrated HMM was trained further on a set of eukaryotic DNA sequences, and tested by using it to segment a separate set of sequences. The resulting HMM system, which is called VEIL (Viterbi Exon-Intron Locator), obtains an overall accuracy on test data of 92% of total bases correctly labelled, with a correlation coefficient of 0.73. Using the more stringent test of exact exon prediction, VEIL correctly located both ends of 53% of the coding exons, and 49% of the exons it predicts are exactly correct. These results compare favorably to the best previous results for gene structure prediction, and demonstrate the benefits of using HMMs for this problem. Please summarise all the given information.

Above are papers related to the following paper: [Paper_ID] Paper #40124. 	[Title] Probabilistic Networks: New Models and New Methods [Abstract] In this paper I describe the implementation of a probabilistic regression model in BUGS. BUGS is a program that carries out Bayesian inference on statistical problems using a simulation technique known as Gibbs sampling. It is possible to implement surprisingly complex regression models in this environment. I demonstrate the simultaneous inference of an interpolant and an input-dependent noise level. Please summarise all the given information.

Above are papers related to the following paper: [Paper_ID] Paper #671293. 	[Title] Selecting Input Variables Using Mutual Information and Nonparametric Density Estimation [Abstract] In learning problems where a connectionist network is trained with a finite sized training set, better generalization performance is often obtained when unneeded weights in the network are eliminated. One source of unneeded weights comes from the inclusion of input variables that provide little information about the output variables. We propose a method for identifying and eliminating these input variables. The method first determines the relationship between input and output variables using nonparametric density estimation and then measures the relevance of input variables using the information theoretic concept of mutual information. We present results from our method on a simple toy problem and a nonlinear time series. Please summarise all the given information.

Above are papers related to the following paper: [Paper_ID] Paper #65212. 	[Title] Bayesian Training of Backpropagation Networks by the Hybrid Monte Carlo Method [Abstract] It is shown that Bayesian training of backpropagation neural networks can feasibly be performed by the "Hybrid Monte Carlo" method. This approach allows the true predictive distribution for a test case given a set of training cases to be approximated arbitrarily closely, in contrast to previous approaches which approximate the posterior weight distribution by a Gaussian. In this work, the Hybrid Monte Carlo method is implemented in conjunction with simulated annealing, in order to speed relaxation to a good region of parameter space. The method has been applied to a test problem, demonstrating that it can produce good predictions, as well as an indication of the uncertainty of these predictions. Appropriate weight scaling factors are found automatically. By applying known techniques for calculation of "free energy" differences, it should also be possible to compare the merits of different network architectures. The work described here should also be applicable to a wide variety of statistical models other than neural networks. Please summarise all the given information.

Above are papers related to the following paper: [Paper_ID] Paper #63549. 	[Title] A Practical Monte Carlo Implementation of Bayesian Learning [Abstract]  Please summarise all the given information.

Above are papers related to the following paper: [Paper_ID] Paper #1105344. 	[Title] In Advances in Neural Information Processing Systems 8  Gaussian Processes for Regression [Abstract] The Bayesian analysis of neural networks is difficult because a simple prior over weights implies a complex prior distribution over functions. In this paper we investigate the use of Gaussian process priors over functions, which permit the predictive Bayesian analysis for fixed values of hyperparameters to be carried out exactly using matrix operations. Two methods, using optimization and averaging (via Hybrid Monte Carlo) over hyperparameters have been tested on a number of challenging problems and have produced excellent results. Please summarise all the given information.

Above are papers related to the following paper: [Paper_ID] Paper #79809. 	[Title] MDL and MML Similarities and Differences (Introduction to Minimum Encoding Inference Part III) [Abstract] Tech Report 207 Department of Computer Science, Monash University, Clayton, Vic. 3168, Australia Abstract: This paper continues the introduction to minimum encoding inductive inference given by Oliver and Hand. This series of papers was written with the objective of providing an introduction to this area for statisticians. We describe the message length estimates used in Wallace's Minimum Message Length (MML) inference and Rissanen's Minimum Description Length (MDL) inference. The differences in the message length estimates of the two approaches are explained. The implications of these differences for applications are discussed. Please summarise all the given information.

Above are papers related to the following paper: [Paper_ID] Paper #1152194. 	[Title]  [Abstract] Draft A Brief Introduction to Neural Networks Richard D. De Veaux Lyle H. Ungar Williams College University of Pennsylvania Abstract Artificial neural networks are being used with increasing frequency for high dimensional problems of regression or classification. This article provides a tutorial overview of neural networks, focusing on back propagation networks as a method for approximating nonlinear multivariable functions. We explain, from a statistician's vantage point, why neural networks might be attractive and how they compare to other modern regression techniques. KEYWORDS: nonparametric regression; function approximation; backpropagation. 1 Introduction Networks that mimic the way the brain works; computer programs that actually LEARN patterns; forecasting without having to know statistics. These are just some of the many claims and attractions of artificial neural networks. Neural networks (we will henceforth drop the term artificial, unless we need to distinguish them from biological neural networks) seem to be everywhere these days, and at least in their advertising, are able to do all that statistics can do without all the fuss and bother of having to do anything except buy a piece of software. Neural networks have been successfully used for many different applications including robotics, chemical process control, speech recognition, optical character recognition, credit card fraud detection, interpretation of chemical spectra and vision for autonomous navigation of vehicles. (Pointers to the literature are given at the end of this article.) In this article we will attempt to explain how one particular type of neural network, feedforward networks with sigmoidal activation functions ("backpropagation networks") actually works, how it is "trained", and how it compares with some more well known statistical techniques. As an example of why someone would want to use a neural network, consider the problem of recognizing hand written ZIP codes on letters. This is a classification problem, where the 1 Please summarise all the given information.

Above are papers related to the following paper: [Paper_ID] Paper #1104449. 	[Title] Using Neural Networks for Descriptive Statistical Analysis of Educational Data [Abstract] In this paper we discuss the methodological issues of using a class of neural networks called Mixture Density Networks (MDN) for discriminant analysis. MDN models have the advantage of having a rigorous probabilistic interpretation, and they have proven to be a viable alternative as a classification procedure in discrete domains. We will address both the classification and interpretive aspects of discriminant analysis, and compare the approach to the traditional method of linear discrimin- ants as implemented in standard statistical packages. We show that the MDN approach adopted performs well in both aspects. Many of the observations made are not restricted to the particular case at hand, and are applicable to most applications of discriminant analysis in educational research. fl URL: http://www.cs.Helsinki.FI/research/cosco/ Please summarise all the given information.

Above are papers related to the following paper: [Paper_ID] Paper #686532. 	[Title] Homology Detection via Family Pairwise Search a straightforward generalization of pairwise sequence comparison algorithms to [Abstract] The function of an unknown biological sequence can often be accurately inferred by identifying sequences homologous to the original sequence. Given a query set of known homologs, there exist at least three general classes of techniques for finding additional homologs: pairwise sequence comparisons, motif analysis, and hidden Markov modeling. Pairwise sequence comparisons are typically employed when only a single query sequence is known. Hidden Markov models (HMMs), on the other hand, are usually trained with sets of more than 100 sequences. Motif-based methods fall in between these two extremes. Please summarise all the given information.

Above are papers related to the following paper: [Paper_ID] Paper #1106370. 	[Title] The Effective Size of a Neural Network: A Principal Component Approach [Abstract] Often when learning from data, one attaches a penalty term to a standard error term in an attempt to prefer simple models and prevent overfitting. Current penalty terms for neural networks, however, often do not take into account weight interaction. This is a critical drawback since the effective number of parameters in a network usually differs dramatically from the total number of possible parameters. In this paper, we present a penalty term that uses Principal Component Analysis to help detect functional redundancy in a neural network. Results show that our new algorithm gives a much more accurate estimate of network complexity than do standard approaches. As a result, our new term should be able to improve techniques that make use of a penalty term, such as weight decay, weight pruning, feature selection, Bayesian, and prediction-risk tech niques. Please summarise all the given information.

Above are papers related to the following paper: [Paper_ID] Paper #197452. 	[Title] Best-First Model Merging for Dynamic Learning and Recognition [Abstract] Best-first model merging is a general technique for dynamically choosing the structure of a neural or related architecture while avoiding overfitting. It is applicable to both learning and recognition tasks and often generalizes significantly better than fixed structures. We demonstrate the approach applied to the tasks of choosing radial basis functions for function learning, choosing local affine models for curve and constraint surface modelling, and choosing the structure of a balltree or bumptree to maximize efficiency of access. Please summarise all the given information.

Above are papers related to the following paper: [Paper_ID] Paper #23116. 	[Title] Bayesian Methods for Adaptive Models [Abstract]  Please summarise all the given information.

Above are papers related to the following paper: [Paper_ID] Paper #1063773. 	[Title] A Gentle Guide to Multiple Alignment Version Please send comments, critique, flames and praise Instructions [Abstract] Prerequisites. An understanding of the dynamic programming (edit distance) approach to pairwise sequence alignment is useful for parts 1.3, 1.4, and 2. Also, familiarity with the use of Internet resources would be helpful for part 3. For the former, see Chapters 1.1 - 1.3, and for the latter, see Chapter 2 of the Hypertext Book of the GNA-VSNS Biocomputing Course at http://www.techfak.uni-bielefeld.de/bcd/Curric/welcome.html. General Rationale. You will understand why Multiple Alignment is considered a challenging problem, you will study approaches that try to reduce the number of steps needed to calculate the optimal solution, and you will study fast heuristics. In a case study involving immunoglobulin sequences, you will study multiple alignments obtained from WWW servers, recapitulating results from an original paper. Revision History. Version 1.01 on 17 Sep 1995. Expanded Ex.9. Updated Ex.46. Revised Solution Sheet -re- Ex.3+12. Marked more Exercises by "A" (to be submitted to the Instructor). Various minor clarifications in content Please summarise all the given information.

Above are papers related to the following paper: [Paper_ID] Paper #686559. 	[Title] Parameterization studies for the SAM and HMMER methods of hidden Markov model generation [Abstract] Multiple sequence alignment of distantly related viral proteins remains a challenge to all currently available alignment methods. The hidden Markov model approach offers a new, flexible method for the generation of multiple sequence alignments. The results of studies attempting to infer appropriate parameter constraints for the generation of de novo HMMs for globin, kinase, aspartic acid protease, and ribonuclease H sequences by both the SAM and HMMER methods are described. Please summarise all the given information.

Above are papers related to the following paper: [Paper_ID] Paper #131042. 	[Title] Efficient Implementation of Gaussian Processes [Abstract] Neural networks and Bayesian inference provide a useful framework within which to solve regression problems. However their parameterization means that the Bayesian analysis of neural networks can be difficult. In this paper, we investigate a method for regression using Gaussian process priors which allows exact Bayesian analysis using matrix manipulations. We discuss the workings of the method in detail. We will also detail a range of mathematical and numerical techniques that are useful in applying Gaussian processes to general problems including efficient approximate matrix inversion methods developed by Skilling. Please summarise all the given information.

Above are papers related to the following paper: [Paper_ID] Paper #286562. 	[Title] Hidden Markov Model Analysis of Motifs in Steroid Dehydrogenases and their Homologs [Abstract]  Please summarise all the given information.

Above are papers related to the following paper: [Paper_ID] Paper #1129442. 	[Title] Minimum-Risk Profiles of Protein Families Based on Statistical Decision Theory [Abstract] Statistical decision theory provides a principled way to estimate amino acid frequencies in conserved positions of a protein family. The goal is to minimize the risk function, or the expected squared-error distance between the estimates and the true population frequencies. The minimum-risk estimates are obtained by adding an optimal number of pseudocounts to the observed data. Two formulas are presented, one for pseudocounts based on marginal amino acid frequencies and one for pseudocounts based on the observed data. Experimental results show that profiles constructed using minimal-risk estimates are more discriminating than those constructed using existing methods. Please summarise all the given information.

Above are papers related to the following paper: [Paper_ID] Paper #95719. 	[Title] Choice of Basis for Laplace Approximation [Abstract] Maximum a posteriori optimization of parameters and the Laplace approximation for the marginal likelihood are both basis-dependent methods. This note compares two choices of basis for models parameterized by probabilities, showing that it is possible to improve on the traditional choice, the probability simplex, by transforming to the `softmax' basis. Please summarise all the given information.

Above are papers related to the following paper: [Paper_ID] Paper #31336. 	[Title] The megaprior heuristic for discovering protein sequence patterns [Abstract] Several computer algorithms for discovering patterns in groups of protein sequences are in use that are based on fitting the parameters of a statistical model to a group of related sequences. These include hidden Markov model (HMM) algorithms for multiple sequence alignment, and the MEME and Gibbs sampler algorithms for discovering motifs. These algorithms are sometimes prone to producing models that are incorrect because two or more patterns have been combined. The statistical model produced in this situation is a convex combination (weighted average) of two or more different models. This paper presents a solution to the problem of convex combinations in the form of a heuristic based on using extremely low variance Dirichlet mixture priors as part of the statistical model. This heuristic, which we call the megaprior heuristic, increases the strength (i.e., decreases the variance) of the prior in proportion to the size of the sequence dataset. This causes each column in the final model to strongly resemble the mean of a single component of the prior, regardless of the size of the dataset. We describe the cause of the convex combination problem, analyze it mathematically, motivate and describe the implementation of the megaprior heuristic, and show how it can effectively eliminate the problem of convex combinations in protein sequence pattern discovery. Please summarise all the given information.

Above are papers related to the following paper: [Paper_ID] Paper #27627. 	[Title] Information-based objective functions for active data selection [Abstract] Learning can be made more efficient if we can actively select particularly salient data points. Within a Bayesian learning framework, objective functions are discussed which measure the expected informativeness of candidate measurements. Three alternative specifications of what we want to gain information about lead to three different criteria for data selection. All these criteria depend on the assumption that the hypothesis space is correct, which may prove to be their main weakness. Please summarise all the given information.

Above are papers related to the following paper: [Paper_ID] Paper #1124844. 	[Title] Stochastic Decomposition of DNA Sequences Using Hidden Markov Models [Abstract] This work presents an application of a machine learning for characterizing an important property of natural DNA sequences compositional inhomogeneity. Compositional segments often correspond to meaningful biological units. Taking into account such inhomogeneity is a prerequisite of successful recognition of functional features in DNA sequences, especially, protein-coding genes. Here we present a technique for DNA segmentation using hidden Markov models. A DNA sequence is represented by a chain of homogeneous segments, each described by one of a few statistically discriminated hidden states, whose contents form a first-order Markov chain. The technique is used to describe and compare chromosomes I and IV of the completely sequenced Saccharomyces cerevisiae (yeast) genome. Our results indicate the existence of a few well separated states, which gives support to the isochore theory. We also explore the model's likelihood landscape and analyze the dynamics of the optimization process, thus addressing the problem of reliability of the obtained optima and efficiency of the algorithms. Please summarise all the given information.

Above are papers related to the following paper: [Paper_ID] Paper #145134. 	[Title] Brief Papers Computing Second Derivatives in Feed-Forward Networks: A Review [Abstract] The calculation of second derivatives is required by recent training and analysis techniques of connectionist networks, such as the elimination of superfluous weights, and the estimation of confidence intervals both for weights and network outputs. We here review and develop exact and approximate algorithms for calculating second derivatives. For networks with jwj weights, simply writing the full matrix of second derivatives requires O(jwj 2 ) operations. For networks of radial basis units or sigmoid units, exact calculation of the necessary intermediate terms requires of the order of 2h + 2 backward/forward-propagation passes where h is the number of hidden units in the network. We also review and compare three approximations (ignoring some components of the second derivative, numerical differentiation, and scoring). Our algorithms apply to arbitrary activation functions, networks, and error functions (for instance, with connections that skip layers, or radial basis functions, or cross-entropy error and Softmax units, etc.). Please summarise all the given information.

Above are papers related to the following paper: [Paper_ID] Paper #1106671. 	[Title] Bayesian Learning in Feed Forward Neural Networks [Abstract] Bayesian methods are applicable to complex modeling tasks. In this review, the principles of Bayesian inference are presented and applied to neural network models. Several approximate implementations are discussed, and their advantages over conventional fre-quentist model training and selection are outlined. It is argued that Bayesian methods are preferable to traditional approaches, although empirical evidence for this is still sparse. Please summarise all the given information.

Above are papers related to the following paper: [Paper_ID] Paper #1107567. 	[Title] Learning Algorithms with Applications to Robot Navigation and Protein Folding [Abstract]  Please summarise all the given information.

Above are papers related to the following paper: [Paper_ID] Paper #19697. 	[Title] Keeping Neural Networks Simple by Minimizing the Description Length of the Weights [Abstract] Supervised neural networks generalize well if there is much less information in the weights than there is in the output vectors of the training cases. So during learning, it is important to keep the weights simple by penalizing the amount of information they contain. The amount of information in a weight can be controlled by adding Gaussian noise and the noise level can be adapted during learning to optimize the trade-off between the expected squared error of the network and the amount of information in the weights. We describe a method of computing the derivatives of the expected squared error and of the amount of information in the noisy weights in a network that contains a layer of non-linear hidden units. Provided the output units are linear, the exact derivatives can be computed efficiently without time-consuming Monte Carlo simulations. The idea of minimizing the amount of information that is required to communicate the weights of a neural network leads to a number of interesting schemes for encoding the weights. Please summarise all the given information.

Above are papers related to the following paper: [Paper_ID] Paper #31349. 	[Title] Meta-MEME: Motif-based Hidden Markov Models of Protein Families [Abstract]  Please summarise all the given information.

Above are papers related to the following paper: [Paper_ID] Paper #131318. 	[Title] FLAT MINIMA Neural Computation 9(1):1-42 (1997) [Abstract] We present a new algorithm for finding low complexity neural networks with high generalization capability. The algorithm searches for a "flat" minimum of the error function. A flat minimum is a large connected region in weight-space where the error remains approximately constant. An MDL-based, Bayesian argument suggests that flat minima correspond to "simple" networks and low expected overfitting. The argument is based on a Gibbs algorithm variant and a novel way of splitting generalization error into underfitting and overfitting error. Unlike many previous approaches, ours does not require Gaussian assumptions and does not depend on a "good" weight prior instead we have a prior over input/output functions, thus taking into account net architecture and training set. Although our algorithm requires the computation of second order derivatives, it has backprop's order of complexity. Automatically, it effectively prunes units, weights, and input lines. Various experiments with feedforward and recurrent nets are described. In an application to stock market prediction, flat minimum search outperforms (1) conventional backprop, (2) weight decay, (3) "optimal brain surgeon" / "optimal brain damage". We also provide pseudo code of the algorithm (omitted from the NC-version). Please summarise all the given information.

Above are papers related to the following paper: [Paper_ID] Paper #31353. 	[Title] Hidden Markov Models in Computational Biology: Applications to Protein Modeling UCSC-CRL-93-32 Keywords: Hidden Markov Models, [Abstract] Hidden Markov Models (HMMs) are applied to the problems of statistical modeling, database searching and multiple sequence alignment of protein families and protein domains. These methods are demonstrated on the globin family, the protein kinase catalytic domain, and the EF-hand calcium binding motif. In each case the parameters of an HMM are estimated from a training set of unaligned sequences. After the HMM is built, it is used to obtain a multiple alignment of all the training sequences. It is also used to search the SWISS-PROT 22 database for other sequences that are members of the given protein family, or contain the given domain. The HMM produces multiple alignments of good quality that agree closely with the alignments produced by programs that incorporate three-dimensional structural information. When employed in discrimination tests (by examining how closely the sequences in a database fit the globin, kinase and EF-hand HMMs), the HMM is able to distinguish members of these families from non-members with a high degree of accuracy. Both the HMM and PRO-FILESEARCH (a technique used to search for relationships between a protein sequence and multiply aligned sequences) perform better in these tests than PROSITE (a dictionary of sites and patterns in proteins). The HMM appears to have a slight advantage Please summarise all the given information.

Above are papers related to the following paper: [Paper_ID] Paper #1154042. 	[Title] 5 Bayesian estimation 5.1 Introduction [Abstract] This chapter takes a different standpoint to address the problem of learning. We will here reason only in terms of probability, and make extensive use of the chain rule known as "Bayes' rule". A fast definition of the basics in probability is provided in appendix A for quick reference. Most of this chapter is a review of the methods of Bayesian learning applied to our modelling purposes. Some original analyses and comments are also provided in section 5.8, 5.11 and 5.12. There is a latent rivalry between "Bayesian" and "Orthodox" statistics. It is by no means our intention to enter this kind of controversy. We are perfectly willing to accept orthodox as well as unorthodox methods, as long as they are scientifically sound and provide good results when applied to learning tasks. The same disclaimer applies to the two frameworks presented here. They have been the object of heated controversy in the past 3 years in the neural networks community. We will not take side, but only present both frameworks, with their strong points and their weaknesses. In the context of this work, the "Bayesian frameworks" are especially interesting as the provide some continuous update rules that can be used during regularised cost minimisation to yield an automatic selection of the regularisation level. Unlike the methods presented in chapter 3, it is not necessary to try several regularisation levels and perform as many optimisations. The Bayesian framework is the only one in which training is achieved through a one-pass optimisation procedure. Please summarise all the given information.

Paper #1123576 has the title `Density Networks and their Application to Protein Modelling  
` and the abstract `I define a latent variable model in the form of a neural network for which only target outputs are specified; the inputs are unspecified. Although the inputs are missing, it is still possible to train this model by placing a simple probability distribution on the unknown inputs and maximizing the probability of the data given the parameters. The model can then discover for itself a description of the data in terms of an underlying latent variable space of lower dimensionality. I present preliminary results of the application of these models to protein data. 
`.
Question: Which category should Paper #1123576 be classified as ?
You can select one from ['Neural_Networks', 'Case_Based', 'Theory', 'Reinforcement_Learning', 'Probabilistic_Methods', 'Rule_Learning', 'Genetic_Algorithms'].


Paper #1123576 has the title `Density Networks and their Application to Protein Modelling  
` and the abstract `I define a latent variable model in the form of a neural network for which only target outputs are specified; the inputs are unspecified. Although the inputs are missing, it is still possible to train this model by placing a simple probability distribution on the unknown inputs and maximizing the probability of the data given the parameters. The model can then discover for itself a description of the data in terms of an underlying latent variable space of lower dimensionality. I present preliminary results of the application of these models to protein data. 
`.
Question: Which category should Paper #1123576 be classified as ?
You can select one from ['Neural_Networks', 'Case_Based', 'Theory', 'Reinforcement_Learning', 'Probabilistic_Methods', 'Rule_Learning', 'Genetic_Algorithms'].

This paper should be classified into: Neural_Networks