nils-herrmann/clean_scirepeval huggingface dataset:
Efficient Approximations for the Marginal Likelihood of Bayesian Networks with Hidden Variables

We discuss Bayesian methods for model averaging and model selection among Bayesian-network models with hidden variables. In particular, we examine large-sample approximations for the marginal likelihood of naive-Bayes models in which the root node is hidden. Such models are useful for clustering or unsupervised learning. We consider a Laplace approximation and the less accurate but more computationally efficient approximation known as the Bayesian Information Criterion (BIC), which is equivalent to Rissanen's (1987) Minimum Description Length (MDL). Also, we consider approximations that ignore some off-diagonal elements of the observed information matrix and an approximation proposed by Cheeseman and Stutz (1995). We evaluate the accuracy of these approximations using a Monte-Carlo gold standard. In experiments with artificial and real examples, we find that (1) none of the approximations are accurate when used for model averaging, (2) all of the approximations, with the exception of BIC/MDL, are accurate for model selection, (3) among the accurate approximations, the Cheeseman–Stutz and Diagonal approximations are the most computationally efficient, (4) all of the approximations, with the exception of BIC/MDL, can be sensitive to the prior distribution over model parameters, and (5) the Cheeseman–Stutz approximation can be more accurate than the other approximations, including the Laplace approximation, in situations where the parameters in the maximum a posteriori configuration are near a boundary.
Model-Based Clustering and Visualization of Navigation Patterns on a Web Site

We present a new methodology for exploring and analyzing navigation patterns on a web site. The patterns that can be analyzed consist of sequences of URL categories traversed by users. In our approach, we first partition site users into clusters such that users with similar navigation paths through the site are placed into the same cluster. Then, for each cluster, we display these paths for users within that cluster. The clustering approach we employ is model-based (as opposed to distance-based) and partitions users according to the order in which they request web pages. In particular, we cluster users by learning a mixture of first-order Markov models using the Expectation-Maximization algorithm. The runtime of our algorithm scales linearly with the number of clusters and with the size of the data; and our implementation easily handles hundreds of thousands of user sessions in memory. In the paper, we describe the details of our method and a visualization tool based on it called WebCANVAS. We illustrate the use of our approach on user-traffic data from msnbc.com.
An Experimental Comparison of Model-Based Clustering Methods

We compare the three basic algorithms for model-based clustering on high-dimensional discrete-variable datasets. All three algorithms use the same underlying model: a naive-Bayes model with a hidden root node, also known as a multinomial-mixture model. In the first part of the paper, we perform an experimental comparison between three batch algorithms that learn the parameters of this model: the Expectation–Maximization (EM) algorithm, a “winner take all” version of the EM algorithm reminiscent of the K-means algorithm, and model-based agglomerative clustering. We find that the EM algorithm significantly outperforms the other methods, and proceed to investigate the effect of various initialization methods on the final solution produced by the EM algorithm. The initializations that we consider are (1) parameters sampled from an uninformative prior, (2) random perturbations of the marginal distribution of the data, and (3) the output of agglomerative clustering. Although the methods are substantially different, they lead to learned models that are similar in quality.
ARMA Time-Series Modeling with Graphical Models

We express the classic ARMA time-series model as a directed graphical model. In doing so, we find that the deterministic relationships in the model make it effectively impossible to use the EM algorithm for learning model parameters. To remedy this problem, we replace the deterministic relationships with Gaussian distributions having a small variance, yielding the stochastic ARMA (σARMA) model. This modification allows us to use the EM algorithm to learn parameters and to forecast, even in situations where some data is missing. This modification, in conjunction with the graphical-model approach, also allows us to include cross predictors in situations where there are multiple time series and/or additional non-temporal covariates. More surprising, experiments suggest that the move to stochastic ARMA yields improved accuracy through better smoothing. We demonstrate improvements afforded by cross prediction and better smoothing on real data.
Joint Discovery of Haplotype Blocks and Complex Trait Associations from SNP Sequences

Haplotypes, the global patterns of DNA sequence variation, have important implications for identifying complex traits. Recently, blocks of limited haplotype diversity have been discovered in human chromosomes, intensifying the research on modelling the block structure as well as the transitions or co-occurrence of the alleles in these blocks as a way to compress the variability and infer the associations more robustly. The haplotype block structure analysis is typically complicated by the fact that the phase information for each SNP is missing, i.e., the observed allele pairs are not given in a consistent order across the sequence. The techniques for circumventing this require additional information, such as family data, or a more complex sequencing procedure. In this paper we present a hierarchical statistical model and the associated learning and inference algorithms that simultaneously deal with the allele ambiguity per locus, missing data, block estimation, and the complex trait association. While the block structure may differ from the structures inferred by other methods, which use the pedigree information or previously known alleles, the parameters we estimate, including the learned block structure and the estimated block transitions per locus, define a good model of variability in the set. The method is completely data-driven and can detect Chron's disease from the SNP data taken from the human chromosome 5q31 with the detection rate of 80% and a small error variance.
Accelerating EM for Large Databases

The EM algorithm is a popular method for parameter estimation in a variety of problems involving missing data. However, the EM algorithm often requires significant computational resources and has been dismissed as impractical for large databases. We present two approaches that significantly reduce the computational cost of applying the EM algorithm to databases with a large number of cases, including databases with large dimensionality. Both approaches are based on partial E-steps for which we can use the results of Neal and Hinton (In Jordan, M. (Ed.), Learning in Graphical Models, pp. 355–371. The Netherlands: Kluwer Academic Publishers) to obtain the standard convergence guarantees of EM. The first approach is a version of the incremental EM algorithm, described in Neal and Hinton (1998), which cycles through data cases in blocks. The number of cases in each block dramatically effects the efficiency of the algorithm. We provide amethod for selecting a near optimal block size. The second approach, which we call lazy EM, will, at scheduled iterations, evaluate the significance of each data case and then proceed for several iterations actively using only the significant cases. We demonstrate that both methods can significantly reduce computational costs through their application to high-dimensional real-world and synthetic mixture modeling problems for large databases.
Bayesian Networks for Data Mining

A Bayesian network is a graphical model that encodesprobabilistic relationships among variables of interest. When used inconjunction with statistical techniques, the graphical model hasseveral advantages for data modeling. One, because the model encodesdependencies among all variables, it readily handles situations wheresome data entries are missing. Two, a Bayesian network can be used tolearn causal relationships, and hence can be used to gain understanding about a problem domain and to predict the consequencesof intervention. Three, because the model has both a causal andprobabilistic semantics, it is an ideal representation for combiningprior knowledge (which often comes in causal form) and data. Four,Bayesian statistical methods in conjunction with Bayesian networksoffer an efficient and principled approach for avoiding theoverfitting of data. In this paper, we discuss methods for constructing Bayesian networks from prior knowledge and summarizeBayesian statistical methods for using data to improve these models.With regard to the latter task, we describe methods for learning boththe parameters and structure of a Bayesian network, includingtechniques for learning with incomplete data. In addition, we relateBayesian-network methods for learning to techniques for supervised andunsupervised learning. We illustrate the graphical-modeling approachusing a real-world case study.
Large-Sample Learning of Bayesian Networks is NP-Hard

In this paper, we provide new complexity results for algorithms that learn discretevariable Bayesian networks from data. Our results apply whenever the learning algorithm uses a scoring criterion that favors the simplest model able to represent the generative distribution exactly. Our results therefore hold whenever the learning algorithm uses a consistent scoring criterion and is applied to a sufficiently large dataset. We show that identifying high-scoring structures is NP-hard, even when we are given an independence oracle, an inference oracle, and/or an information oracle. Our negative results also apply when learning discrete-variable Bayesian networks in which each node has at most k parents, for all k ≥ 3.
The Variational Bayesian EM Algorithm for Incomplete Data : with Application to Scoring Graphical Model Structures

We present an efficient procedure for estimating the marginal likelihood of probabilistic models with latent variables or incomplete data. This method constructs and optimises a lower bound on the marginal likelihood using variational calculus, resulting in an iterative algorithm which generalises the EM algorithm by maintaining posterior distributions over both latent variables and parameters. We define the family of conjugate-exponential models—which includes finite mixtures of exponential family models, factor analysis, hidden Markov models, linear state-space models, and other models of interest—for which this bound on the marginal likelihood can be computed very simply through a modification of the standard EM algorithm. In particular, we focus on applying these bounds to the problem of scoring discrete directed graphical model structures (Bayesian networks). Extensive simulations comparing the variational bounds to the usual approach based on the Bayesian Information Criterion (BIC) and to a sampling-based gold standard method known as Annealed Importance Sampling (AIS) show that variational bounds substantially outperform BIC in finding the correct model structure at relatively little computational cost, while approaching the performance of the much more costly AIS procedure. Using AIS allows us to provide the first serious case study of the tightness of variational bounds. We also analyse the perfomance of AIS through a variety of criteria, and outline directions in which this work can be extended.
Targeted Advertising on the Web with Inventory Management

Companies that maintain Web sites can make considerable revenue by running advertisements, and they therefore compete to attract advertisers. The ability to deliver high click-through rates on a site can attract advertisers and, under an appropriate pricing model, can also increase revenue directly. Consequently, companies can benefit from delivery systems that display advertisements selectively to those visitors most likely to click though. To satisfy contractual obligations, however, these systems must simultaneously manage inventory. We developed a delivery system that maximizes click-through rate given inventory-management constraints in the form of advertisement quotas. The system uses predictive segments in conjunction with a linear program to perform the constrained optimization. Using a real Web site (msn.com), we demonstrated the efficacy of the system. We can generalize our system to find revenue-optimal advertisement schedules under a wide variety of pricing models.
Gaussian Processes to Speed up Hybrid Monte Carlo for Expensive Bayesian Integrals

Hybrid Monte Carlo (HMC) is often the method of choice for computing Bayesian integrals that are not analytically tractable. However the success of this method may require a very large number of evaluations of the (un-normalized) posterior and its partial derivatives. In situations where the posterior is computationally costly to evaluate, this may lead to an unacceptable computational load for HMC. I propose to use a Gaussian Process model of the (log of the) posterior for most of the computations required by HMC. Within this scheme only occasional evaluation of the actual posterior is required to guarantee that the samples generated have exactly the desired distribution, even if the GP model is somewhat inaccurate. The method is demonstrated on a 10 dimensional problem, where 200 evaluations suffice for the generation of 100 roughly independent points from the posterior. Thus, the proposed scheme allows Bayesian treatment of models with posteriors that are computationally demanding, such as models involving computer simulation.
Markov chain Monte Carlo-based approaches for inference in computationally intensive inverse problems

A typical setup for many inverse problems is that one wishes to update beliefs about a spatially dependent set of inputs x given rather indirect observations y. Here, the inputs and observed outputs are related by the complex physical relationship y = ζ(x) + . Applications include medical and geological tomography, hydrology, and the modeling of physical and biological systems. We consider applications where the physical relationship ζ(x) can be well approximated by detailed simulation code η(x). When the forward simulation code η(x) is sufficiently fast, Bayesian inference can, in principle, be carried out via Markov chain Monte Carlo (MCMC). Difficulties arise for two main reasons: • Even though the code may accurately represent the physical process, there are a large number of unknown, but required, inputs that must be calibrated to match the observed data y. • The computational burden of the fastest available forward simulators is often large enough that approaches for speeding up the MCMC calculations are required. This paper develops approaches for specifying effective low-dimensional representations of the inputs x along with MCMC approaches for sampling the posterior distribution. In particular we consider augmenting the basic formulation with fast, possibly coarsened, formulations to improve MCMC performance. This approach can be very easily implemented in a parallel computing environment. We give examples in single photon emission computed tomography and in hydrology.
A Statistical Approach to Modeling Genomic Aberrations in Cancer Cells

Whereas most cells in the body carry the normal complement of 23 chromosome pairs, the cells within a cancerous tumor very often present highly abnormal genomic structure. Deletions, amplifications, rearrangements and mutations are common at various scales and are highly variable amongst tumors, as indicated by molecular technologies which enable ever better measurement. It is an important statistical problem to separate sporadic abnormalities from those that may not be sporadic and that may have some biological significance. We discuss a modeling strategy for genomic aberration data which allows us to to infer combinations of aberrations that together increase the chance that a precancerous cell will have a descendant tumor lineage. The likelihood component involves a network of pathway structures. Markov chain Monte Carlo is used to sample from the space of these oncogenic networks. We illustrate the methodology with comparative genomic hybridizations from a recent breast cancer study, and we derive a likelihood formula for the larger class of tree-like networks.
Recommendation as a Stochastic Sequential Decision Problem

Recommender systems — systems that suggest to users in e-commerce sites items that might interest them — adopt a static view of the recommendation process and treat it as a prediction problem. In an earlier paper, we argued that it is more appropriate to view the problem of generating recommendations as a sequential decision problem and, consequently, that Markov decision processes (MDPs) provide a more appropriate model for recommender systems. MDPs introduce two benefits: they take into account the long-term effects of each recommendation, and they take into account the expected value of each recommendation. The use of MDPs in a commercial site raises three fundamental problems: providing an adequate initial model, updating this model online as new items (e.g., books) arrive, and coping with the enormous state-space of this model. In past work, we dealt with the first problem. In this paper we consider the second, and especially, the third problem, which is of greater concern to researchers in decision-theoretic planning. We show that although the model we consider has roughly 1011 states, we can quickly provide an approximate solution by utilizing its special structure. Our memory requirements – a serious concern for commercial online applications – are modest; and the overall resource requirements of our system are comparable to those of a well-known commercial recommender system that uses a simpler and less accurate model. Our system is one of a handful of deployed commercial recommender systems as well as one of a handful of MDP-based deployed systems. It has been running at www.mitos.co.il, a commercial online bookstore, since August, 2002.
Non-Centered Parameterisations for Hierarchical Models and Data Augmentation

In this paper, we will compare centered and non-centered parameterisations for classes of hierarchical models. Our examples will include variance component models, random effect models, hidden Markov process models, and partially observed diffusion models. We will investigate the construction of non-centered methods by the use of state space expansion techniques, and will introduce methods for devising partially non-centered parameterisations, many of which are data-dependent.
A relationship between randomized manipulation and parameter independence

We argue that a Bayesian will usually need to specify a joint prior density of the conditional probabilities of Causal Bayesian network (CBN). We show that in order to do this, under very mild conditions it will be necessary to demand that this joint prior density exhibits the properties of local and global independence. To make the connection between prior independence and causality, it is first necessary to strengthen slightly the assumptions of factorization invariance under manipulation which induces randomized intervention. We introduce the hypercausal BN (HCBN) that asserts a set of factorizations of densities which are invariant to a class of "do" operations larger than those considered by Pearl. We show that if a BN is assumed to be hypercausal, then the prior distributions on the probabilities of the idle system must exhibit local and global independence.
Density Modeling and Clustering Using Dirichlet Diffusion Trees

I introduce a family of prior distributions over multivariate distributions, based on the use of a “Dirichlet diffusion tree” to generate exchangeable data sets. These priors can be viewed as generalizations of Dirichlet processes and of Dirichlet process mixtures, but unlike simple mixtures, they can capture the hierarchical structure present in many distributions, by means of the latent diffusion tree underlying the data. This latent tree also provides a hierarchical clustering of the data, which, unlike ad hoc clustering methods, comes with probabilistic indications of uncertainty. The relevance of each variable to the clustering can also be determined. Although Dirichlet diffusion trees are defined in terms of a continuoustime process, posterior inference involves only finite-dimensional quantities, allowing computation to be performed by reasonably efficient Markov chain Monte Carlo methods. The methods are demonstrated on problems of modeling a two-dimensional density and of clustering gene expression data.
Expected Utility Estimation via Cross-Validation

We discuss practical methods for the assessment, comparison and selection of complex hierarchical Bayesian models. A natural way to assess the goodness of the model is to estimate its future predictive capability by estimating expected utilities. Instead of just making a point estimate, it is important to obtain the distribution of the expected utility estimate in order to describe the associated uncertainty. We synthesize and extend the previous work in several ways. We give a unified presentation from the Bayesian viewpoint emphasizing the assumptions made and propose practical methods to obtain the distributions of the expected utility estimates. We discuss the properties of two practical methods, the importance sampling leave-one-out and the k-fold cross-validation. We propose a quick and generic approach based on the Bayesian bootstrap for obtaining samples from the distributions of the expected utility estimates. These distributions can also be used for model comparison, for example, by computing the probability of one model having a better expected utility than some other model. We discuss how the crossvalidation approach differs from other predictive density approaches, and the relationship of cross-validation to information criteria approaches, which can also be used to estimate the expected utilities. We illustrate the discussion with one toy and two real world examples.
Learning Bayesian Networks From Dependency Networks: A Preliminary Study

In this paper we describe how to learn Bayesian networks from a summary of complete data in the form of a dependency network rather than from data directly. This method allows us to gain the advantages of both representations: scalable algorithms for learning dependency networks and convenient inference with Bayesian networks. Our approach is to use a dependency network as an “oracle” for the statistics needed to learn a Bayesian network. We show that the general problem is NP-hard and develop a greedy search algorithm. We conduct a preliminary experimental evaluation and find that the prediction accuracy of the Bayesian networks constructed from our algorithm almost equals that of Bayesian networks learned directly from the data.
Staged Mixture Modelling and Boosting

In this paper, we introduce and evaluate a data-driven staged mixture modeling tcchnique for building density, regression, and classification models. Our basic approach is to sequentially add components to a finite mixture model using the structural expectation maximization (SEM) algorithm. We show that our technique is qualitatively similar to boosting. This correspondence is a natural byproduct of the fact that we use the SEM algorithm to sequentially fit the mixture model. Finally, in our experimental evaluation, we demonstrate the effectiveness of our approach on a variety of prediction and density estimation tasks using real-world data.