nils-herrmann/clean_scirepeval huggingface dataset:
Tractable Bayesian learning of tree belief networks

In this paper we present decomposable priors, a family of priors over structure and parameters of tree belief nets for which Bayesian learning with complete observations is tractable, in the sense that the posterior is also decomposable and can be completely determined analytically in polynomial time. Our result is the first where computing the normalization constant and averaging over a super-exponential number of graph structures can be performed in polynomial time. This follows from two main results: First, we show that factored distributions over spanning trees in a graph can be integrated in closed form. Second, we examine priors over tree parameters and show that a set of assumptions similar to Heckerman, Geiger and Chickering (1995) constrain the tree parameter priors to be a compactly parametrized product of Dirichlet distributions. Besides allowing for exact Bayesian learning, these results permit us to formulate a new class of tractable latent variable models in which the likelihood of a data point is computed through an ensemble average over tree structures.
Tutorial on variational approximation methods

Tutorial topics • A bit of history • Examples of variational methods • A brief intro to graphical models • Variational mean field theory – Accuracy of variational mean field – Structured mean field theory • Variational methods in Bayesian estimation • Convex duality and variational factorization methods – Example: variational inference and the QMR-DT Variational methods •  " finding the extremum of an integral involving a function and its derivatives " Example: finding the trajectory of a particle under external field • The key idea here is that the problem of interest is formulated as an optimization problem Variational methods cont'd • Variational methods have a long history in physics, statistics, control theory as well as economics. – calculus of variations (physics) – linear/non-linear moments problems (statistics) – dynamic programming (control theory) • Variational formulations appear naturally also in machine learning contexts: – regularization theory – maximum entropy estimation • Recently variational methods been used and further developed in the context of approximate inference and estimation Examples of variational methods • In classical examples the formulation itself is given but for us this is one of the key problems • We provide here a few examples that highlight 1. how to cast problems as optimization problems 2. how to find an approximate solution when the exact solution is not feasible • The examples we use involve a) finite element methods for solving differential equations b) large deviation methods (Chernoff bound)
Sequentially Fitting "Inclusive" Trees for Inference in Noisy-OR Networks

An important class of problems can be cast as inference in noisy-OR Bayesian networks, where the binary state of each variable is a logical OR of noisy versions of the states of the variable's parents. For example, in medical diagnosis, the presence of a symptom can be expressed as a noisy-OR of the diseases that may cause the symptom - on some occasions, a disease may fail to activate the symptom. Inference in richly-connected noisy-OR networks is intractable, but approximate methods (e.g., variational techniques) are showing increasing promise as practical solutions. One problem with most approximations is that they tend to concentrate on a relatively small number of modes in the true posterior, ignoring other plausible configurations of the hidden variables. We introduce a new sequential variational method for bipartite noisy-OR networks, that favors including all modes of the true posterior and models the posterior distribution as a tree. We compare this method with other approximations using an ensemble of networks with network statistics that are comparable to the QMR-DT medical diagnostic network.
Feature Selection and Dualities in Maximum Entropy Discrimination

Incorporating feature selection into a classification or regression method often carries a number of advantages. In this paper we formalize feature selection specifically from a discriminative perspective of improving classification/ regression accuracy. The feature selection method is developed as an extension to the recently proposed maximum entropy discrimination (MED) framework. We describe MED as a flexible (Bayesian) regularization approach that subsumes, e.g., support vector classification, regression and exponential family models. For brevity, we restrict ourselves primarily to feature selection in the context of linear classification/regression methods and demonstrate that the proposed approach indeed carries substantial improvements in practice. Moreover, we discuss and develop various extensions of feature selection, including the problem of dealing with example specific but unobserved degrees of freedom - alignments or invariants.
Bayesian parameter estimation via variational methods

We consider a logistic regression model with a Gaussian prior distribution over the parameters. We show that an accurate variational transformation can be used to obtain a closed form approximation to the posterior distribution of the parameters thereby yielding an approximate posterior predictive model. This approach is readily extended to binary graphical model with complete observations. For graphical models with incomplete observations we utilize an additional variational transformation and again obtain a closed form approximation to the posterior. Finally, we show that the dual of the regression problem gives a latent variable density model, the variational formulation of which leads to exactly solvable EM updates.
Using the Fisher Kernel Method to Detect Remote Protein Homologies

A new method, called the Fisher kernel method, for detecting remote protein homologies is introduced and shown to perform well in classifying protein domains by SCOP superfamily. The method is a variant of support vector machines using a new kernel function. The kernel function is derived from a hidden Markov model. The general approach of combining generative models like HMMs with discriminative methods such as support vector machines may have applications in other areas of biosequence analysis as well.
Maximum Entropy Discrimination

We present a general framework for discriminative estimation based on the maximum entropy principle and its extensions. All calculations involve distributions over structures and/or parameters rather than specific settings and reduce to relative entropy projections. This holds even when the data is not separable within the chosen parametric class, in the context of anomaly detection rather than classification, or when the labels in the training set are uncertain or incomplete. Support vector machines are naturally subsumed under this class and we provide several extensions. We are also able to estimate exactly and efficiently discriminative distributions over tree structures of class-conditional models within this framework. Preliminary experimental results are indicative of the potential in these techniques.
Variational methods and the QMR-DT database

We describe a variational approximation method for efficient probabilistic inference in dense graphical models. Variational methods are deterministic approximation procedures that provide bounds on probabilities of interest. They provide alternatives to approximate inference methods based on stochastic sampling or search. We exemplify variational methods by an application to the problem of diagnostic inference in the QMR-DT database. The QMR-DT database is a large-scale graphical model based on statistical and expert knowledge in internal medicine. The size and complexity of the database render exact probabilistic diagnosis infeasible for all but a small set of cases. We describe a variational inference algorithm for the QMR-DT database, evaluate the accuracy of our algorithm on a set of standard diagnostic cases and compare to stochastic sampling methods.
Variational probabilistic inference and the QMR-DT database

We describe a variational approximation method for eecient inference in large-scale probabilistic models. Variational methods are deterministic procedures that provide approximations to marginal and conditional probabilities of interest. They provide alternatives to approximate inference methods based on stochastic sampling or search. We describe a variational approach to the problem of diagnostic inference in the \Quick Medical Reference" (QMR) database. The QMR database is a large-scale probabilistic graphical model built on statistical and expert knowledge. Exact probabilistic inference is infeasible in this model for all but a small set of cases. We evaluate our variational inference algorithm on a large set of diagnostic test cases, comparing the algorithm to a state-of-the-art stochastic sampling method.
IMPROVING THE MEAN FIELD APPROXIMATIONVIA THE USE OF MIXTURE

Mean eld methods provide computationally eecient approximations to posterior probability distributions for graphical models. Simple mean eld methods make a completely factorized approximation to the posterior, which is unlikely to be accurate when the posterior is multi-modal. Indeed, if the posterior is multi-modal, only one of the modes can be captured. To improve the mean eld approximation in such cases, we employ mixture models as posterior approximations, where each mixture component is a factorized distribution. We describe eecient methods for optimizing the parameters in these models.
Improving the Mean Field Approximation Via the Use of Mixture Distributions

Mean field methods provide computationally efficient approximations to posterior probability distributions for graphical models. Simple mean field methods make a completely factorized approximation to the posterior, which is unlikely to be accurate when the posterior is multimodal. Indeed, if the posterior is multi-modal, only one of the modes can be captured. To improve the mean field approximation in such cases, we employ mixture models as posterior approximations, where each mixture component is a factorized distribution. We describe efficient methods for optimizing the Parameters in these models.
Exploiting Generative Models in Discriminative Classifiers

Generative probability models such as hidden Markov models provide a principled way of treating missing information and dealing with variable length sequences. On the other hand, discriminative methods such as support vector machines enable us to construct flexible decision boundaries and often result in classification performance superior to that of the model based approaches. An ideal classifier should combine these two complementary approaches. In this paper, we develop a natural way of achieving this combination by deriving kernel functions for use in discriminative methods such as support vector machines from generative probability models. We provide a theoretical justification for this combination as well as demonstrate a substantial improvement in the classification performance in the context of DNA and protein sequence analysis.
Variational methods for inference and estimation in graphical models

Graphical models enhance the representational power of probability models through qualitative characterization of their properties. This also leads to greater efficiency in terms of the computational algorithms that empower such representations. The increasing complexity of these models, however, quickly renders exact probabilistic calculations infeasible. We propose a principled framework for approximating graphical models based on variational methods. 
We develop variational techniques from the perspective that unifies and expands their applicability to graphical models. These methods allow the (recursive) computation of upper and lower bounds on the quantities of interest. Such bounds yield considerably more information than mere approximations and provide an inherent error metric for tailoring the approximations individually to the cases considered. These desirable properties, concomitant to the variational methods, are unlikely to arise as a result of other deterministic or stochastic approximations. 
The thesis consists of the development of this variational methodology for probabilistic inference, Bayesian estimation, and towards efficient diagnostic reasoning in the domain of internal medicine.
Approximating Posterior Distributions in Belief Networks Using Mixtures

Exact inference in densely connected Bayesian networks is computationally intractable, and so there is considerable interest in developing effective approximation schemes. One approach which has been adopted is to bound the log likelihood using a mean-field approximating distribution. While this leads to a tractable algorithm, the mean field distribution is assumed to be factorial and hence unimodal. In this paper we demonstrate the feasibility of using a richer class of approximating distributions based on mixtures of mean field distributions. We derive an efficient algorithm for updating the mixture parameters and apply it to the problem of learning in sigmoid belief networks. Our results demonstrate a systematic improvement over simple mean field theory as the number of mixture components is increased.
Vc Dimension Further Information 3.3 Optimization Algorithms 3 Learning from Data

A measure of the complexity of a model. Knowledge of the VC dimension permits an estimate to be made of the diierence between performance on the training set and performance on a test set. 29 Deening terms Classiication A learning problem in which the goal is to assign input vectors to one of a number of (usually mutually exclusive) classes. Boltzmann machine An undirected network of discrete valued random variables, where an energy function is associated with each of the links, and for which a probability distribution is deened by the Boltzmann distribution. Cost function A function of the adaptive parameters of a model whose minimum is used to deene suitable values for those parameters. It may consist of a likelihood function and additional terms. Decision tree A network that performs a sequence of classiicatory decisions on an input vector and produces an output vector that is conditional on the outcome of the decision sequence. Density estimation The problem of modeling a probability distribution from a nite set of examples drawn from that distribution. Discriminant function A function of the input vector which can be used to assign inputs to classes in a classiication problem. Hidden Markov model A graphical probabilistic model characterized by a state vector, an output vector, a state transition matrix, an emission matrix and an initial state distribution. Likelihood function The probability of observing a particular data set under the assumption of a given parametrized model, expressed as a function of the adaptive parameters of the model. Mixture model A probability model which consists of a linear combination of simpler component probability models. Multilayer perceptron The most common form of neural network model, consisting of successive linear transformations followed by processing with nonlinear activation functions. Overrtting The problem in which a model which is too complex captures too much of the noise in the data, leading to poor generalization. Radial basis function network A common network model consisting of a linear combination of basis functions each of which is a function of the diierence between the input vector and a center vector.  A learning problem in which the goal is to map each input vector to a real-valued output vector.  A technique for controlling model complexity and improving generalization by the addition of a penalty term to the cost function. In this chapter we have emphasized the links between neural networks and statistical pattern …
MASSACHUSETTS INSTITUTE OF TECHNOLOGY ARTIFICIAL INTELLIGENCE LABORATORY and CENTER FOR BIOLOGICAL AND COMPUTATIONAL LEARNING DEPARTMENT OF BRAIN AND COGNITIVE SCIENCES

Sigmoid type belief networks, a class of probabilistic neural networks, provide a natural framework for compactly representing probabilistic information in a variety of unsupervised and supervised learning problems. Often the parameters used in these networks need to be learned from examples. Unfortunately, estimating the parameters via exact probabilistic calculations (i.e, the EM-algorithm) is intractable even for networks with fairly small numbers of hidden units. We propose to avoid the infeasibility of the E step by bounding likelihoods instead of computing them exactly. We introduce extended and complementary representations for these networks and show that the estimation of the network parameters can be made fast (reduced to quadratic optimization) by performing the estimation in either of the alternative domains. The complementary networks can be used for continuous density estimation as well.  In Advances in Neural Information Processing Systems 8, MIT Press. This report describes research done at the Dept. of Brain and Cognitive Sciences, the Center for Biological and Computational Learning, and the Arti cial Intelligence Laboratory of the Massachusetts Institute of Technology. Support for CBCL is provided in part by a grant from the NSF (ASC{9217041). Support for the laboratory's arti cial intelligence research is provided in part by the Advanced Research Projects Agency of the Dept. of Defense. The authors were supported by a grant from the McDonnell-Pew Foundation, by a grant from Siemens Corporation, by a grant >from Daimler-Benz Systems Technology Research, and by a grant from the O ce of Naval Research. Michael I. Jordan is a NSF Presidential Young Investigator.
Recursive Algorithms for Approximating Probabilities in Graphical Models

We develop a recursive node-elimination formalism for efficiently approximating large probabilistic networks. No constraints are set on the network topologies. Yet the formalism can be straightforwardly integrated with exact methods whenever they are/become applicable. The approximations we use are controlled: they maintain consistently upper and lower bounds on the desired quantities at all times. We show that Boltzmann machines, sigmoid belief networks, or any combination (i.e., chain graphs) can be handled within the same framework. The accuracy of the methods is verified experimentally.
A variational approach to Bayesian logistic regression problems and their extensions

We consider a logistic regression model with a Gaussian prior distribution over the parameters. We show that accurate variational techniques can be used to obtain a closed form posterior distribution over the parameters given the data thereby yielding a posterior predictive model. The results are readily extended to (binary) belief networks. For belief networks we also derive closed form posteriors in the presence of missing values. Finally, we show that the dual of the regression problem gives a latent variable density model, the variational formulation of which leads to exactly solvable EM updates.
Fast Learning by Bounding Likelihoods in Sigmoid Type Belief Networks

Sigmoid type belief networks, a class of probabilistic neural networks, provide a natural framework for compactly representing probabilistic information in a variety of unsupervised and supervised learning problems. Often the parameters used in these networks need to be learned from examples. Unfortunately, estimating the parameters via exact probabilistic calculations (i.e, the EM-algorithm) is intractable even for networks with fairly small numbers of hidden units. We propose to avoid the infeasibility of the E step by bounding likelihoods instead of computing them exactly. We introduce extended and complementary representations for these networks and show that the estimation of the network parameters can be made fast (reduced to quadratic optimization) by performing the estimation in either of the alternative domains. The complementary networks can be used for continuous density estimation as well.
Approximations of Bayesian networks through KL minimisation

Exact inference in large, complex Bayesian networks is computationally intractable. Approximate schemes are therefore of great importance for real world computation. In this paper we consider an approximation scheme in which the original Bayesian network is approximated by another Bayesian network. The approximating network is optimised by an iterative procedure, which minimises the Kullback-Leibler divergence between the two networks. The procedure is guaranteed to converge to a local minimum of the Kullback-Leibler divergence. An important question in this scheme is how to choose the structure of the approximating network. In this paper we show how redundant structures of the approximating model can be pruned in advance. Simulation results of model optimisation are provided to illustrate the methods.
Modeling Bayesian Networks by Learning from Experts

Bayesian network modeling by domain experts is still mainly a process of trial and error. The structure of the graph and the specification of the conditional probability tables (ePTs) are in practice often fiddled until a desired model behavior is obtained. We describe a development tool in which graph specification and ePT modeling are fully separated. Furthermore, the tuning of ePTs is handled automatically. The development tool consists of a database in which the graph description and the desired probabilistic behavior of the network are separately stored. From this database, the graph is constructed and the ePTs are numerically optimized in order to minimize the error between desired and actual behavior. The tool may be helpful in both development and maintenance of probabilistic expert systems. A demo is provided. A numerical example illustrates the methodology.
Approximations with Reweighted Generalized Belief Propagation

In (Wainwright et al., 2002) a new general class of upper bounds on the log partition function of arbitrary undirected graphical models has been developed. This bound is constructed by taking convex combinations of tractable distributions. The experimental results published so far concentrates on combinations of tree-structured distributions leading to a convexified Bethe free energy, which is minimized by the tree-reweighted belief propagation algorithm. One of the favorable properties of this class of approximations is that increasing the complexity of the approximation is guaranteed to increase the precision. The lack of this guarantee is notorious in standard generalized belief propagation. We increase the complexity of the approximating distributions by taking combinations of junction trees, leading to a convexified Kikuchi free energy, which is minimized by reweighted generalized belief propagation. Experimental results for Ising grids as well as for fully connected Ising models are presented illustrating advantages and disadvantages of the reweighting method in approximate inference.
Deterministic Generative Models for Fast Feature Discovery

We propose a vector quantisation method which does not only provide a compact description of data vectors in terms codebook vectors, but also gives an explanation of codebook vectors as binary combinations of elementary features. This corresponds to the intuitive notion that, in the real world, patterns can be usefully thought of as being constructed by compositions from simpler features. The model can be understood as a generative model, in which the codebook vector is generated by a hidden binary state vector. The model is non-probabilistic in the sense that it assigns each data vector to a single codebook vector. We describe exact and approximate learning algorithms for learning deterministic feature representations. In contrast to probabilistic models, the deterministic approach allows the use of message propagation algorithms within the learning scheme. These are compared with standard mean-field/Gibbs sampling learning. We show that Generative Vector Quantisation gives a good performance in large scale real world tasks like image compression and handwritten digit analysis with up to 400 data dimensions.