nils-herrmann/clean_scirepeval huggingface dataset:
Approximate Expectation Maximization

We discuss the integration of the expectation-maximization (EM) algorithm for maximum likelihood learning of Bayesian networks with belief propagation algorithms for approximate inference. Specifically we propose to combine the outer-loop step of convergent belief propagation algorithms with the M-step of the EM algorithm. This then yields an approximate EM algorithm that is essentially still double loop, with the important advantage of an inner loop that is guaranteed to converge. Simulations illustrate the merits of such an approach.
Promedas : A clinical diagnosticdecisionsupport system

Promedasis a medicalpatient-specific clinical diagnostic decision support systemsbasedon graphicalprobabilistic models.Promedasaimsto improve the quality andefficiency of the diagnosticprocess,while reducingits costs at thesametime. Modern-daymedicaldiagnosisis averycomplex process,requiring accuratepatient data, a profoundunderstandingof themedicalliteratureand many yearsof clinical experience.Diagnosisis a process, by whichadoctorsearchesfor thecause (disease)that best explains the symptomsof a patient. The searchprocessis sequential,in the sensethatpatientsymptomssuggestsomeinitial teststo be performed.Basedon the outcomeof thesetests,a tentati ve hypothesisis formulated about the possiblecause(s). Basedon this hypothesis,subsequent testsareorderedto confirm or reject this hypothesis.The processmay proceedin several iterationsuntil the patient is finally diagnosedwith sufficient certaintyandthe causeof thesymptomsis established. A significant part of the diagnosticprocessis standardizedin theform of protocols.Theseare setsof rules that prescribewhich teststo perform and in which order, basedon the patient symptomsandprevious testresults.Theserules form a decisiontree, whosenodesare intermediatestagesin thediagnosticprocessandwhose branchespoint to additional testing, depending on thecurrenttestresults.Theprotocolsaredefinedin eachcountryby a committeeof medical experts. In themayorityof thediagnosesthatareencountered,the guidelinesaresufficiently accurateto make the correctdiagnosis. For these’routine’ cases,a decisionsupportsystemis not needed. In 10-20% of the cases,however, the diagostic processis moredifficult. As a resultof the uncertaintyabout the correctdiagnosisand about the next actionsto perform, the decisionsmade by differentphysiciansat differentstagesof the diagnosticprocessdo not alwaysagreeandlack “rationalization”. In thesecases,normallya collegueis consultedthat is particularlyspecialized in this diseaseor the literatureis consulted.For thesedifficult casescomputerbaseddecisionsupport is of addedvalue. Since1996, we are developing a clinical diagnosticdecisionsupportsystem(CDDSS),called Promedas.The CDDSSis a computerprogram thatcontainsall relevantknowledgeabouta certain medical(sub)domainand generatespatient specificdiagnosticadvicein theform of a list of likely diagnosesand suggestionsfor additional laboratoryteststhat are expectedto be particularly informative to establishor rule out any of thediagnosesconsidered. Promedasis basedonmedicalexpertknowledge, acquiredfrom the medical literature and textbooksby the medicalspecialistsin our project team. In order to obtain a clinically relevant model it is crucial that the modelingis doneby the medicalexpertsthemselves. The domainof internalmedicineconsistsof many sub-domains (diabetes,lipids, endocrinology, etc.) and only specialistswith a specific specializationin the sub-domainareable to do suchmodelingaccurately. Theexpertknowledgeis storedin a database,in suchawaythatextensionandmaintenanceof the expertknowledgeis easilyfacilitated.
Combining Exploration and Reliability in Coevolution

Coevolution can in principle circumvent the difficult problem of designing a fitness function; this may remove harmful biases, and thereby improve search performance. Recently, based on Evolutionary Multi-Objective Optimization, the feasibility of accurate evaluation in coevolution has been demonstrated theoretically. A current challenge in coevolution is how this theoretical promise may be translated into practical algorithms. We investigate a setup consisting of multiple subpopulations, each with a varying degree of exploration. It is shown that by allowing these subpopulations to interact, the desirable aims of exploration and reliability can be combined.
Approximate Expectation

We discuss the integration of the expectation-maximization (EM) algorithm for maximum likelihood learning of Bayesian networks with belief propagation algorithms for approximate inference. Specifically we propose to combine the outer-loop step of convergent belief propagation algorithms with the M-step of the EM algorithm. This then yields an approximate EM algorithm that is essentially still double loop, with the important advantage of an inner loop that is guaranteed to converge. Simulations illustrate the merits of such an approach.
Optimising newspaper sales using neural-Bayesian technology

We describe a software system, called just enough delivery (JED), for optimising single-copy newspaper sales, based on a combination of neural and Bayesian technology. The prediction model is a huge feedforward neural network, in which each output corresponds to the sales prediction for a single outlet. Input-to-hidden weights are shared between outlets. The hidden-to-output weights are specific to each outlet, but linked through the introduction of priors. All weights and hyperparameters can be inferred using (empirical) Bayesian inference. The system has been tested on data for several different newspapers and magazines. Consistent performance improvements of 1 to 3% more sales with the same total amount of deliveries have been obtained.
IPF for Discrete Chain Factor Graphs

Iterative Proportional Fitting (IPF), combined with EM, is commonly used as an algorithm for likelihood maximization in undirected graphical models. In this paper, we present two iterative algorithms that generalize upon IPF. The first one is for likelihood maximization in discrete chain factor graphs, which we define as a wide class of discrete variable models including undirected graphical models and Bayesian networks, but also chain graphs and sigmoid belief networks. The second one is for conditional likelihood maximization in standard undirected models and Bayesian networks. In both algorithms, the iteration steps are expressed in closed form. Numerical simulations show that the algorithms are competitive with state of the art methods.
”PROMEDAS” : a probabilistic decision support system for medical diagnosis

The use of patient-specific Decision Support Systems (DSS) may improve the quality and efficiency of health care, while reducing its costs at the same time. The adoption of such a system is largely compatible with the principles of " Evidence Based Medicine " and patient oriented care. PROMEDAS (PRObabilistic MEdical Diagnostic Advisory System) is a prototype DSS, based on a probabilistic model and advanced computational techniques. The system offers patient specific diagnostic advice. It presents a differential diagnosis and it supports the diagnostic process by indicating the most useful next step in the diagnostic process. The system is intended to support diagnosis making in the setting of the outpatient clinic and for educational purposes. Its target-users are general internists, super specialists (i.e. endocrinologists, rheumatologists), interns and residents, medical students and others working in the hospital environment. Currently, PROMEDAS is a stand alone application. In the future PROMEDAS may be integrated with a Hospital Information System and an Electronic Patient Record. This will facilitate its use in practice, and may augment its acceptance. PROMEDAS is based on medical expert knowledge, acquired from the literature by the medical specialists in our project team. The acquired knowledge is stored in a database, in such a way that extension and maintenance of the expert knowledge is facilitated. Currently, the database covers large parts of endocrinology and lymphoma diagnostics. In near future, parts of vasculair medicine will be entered as well. From (parts of) this database, Bayesian network and an interface for PROMEDAS is automatically compiled. The network is the underlying model of PROMEDAS. Bayesian inference is used to query the system. The PROMEDAS project is funded by STW (project nr: NNN5322). The goal of the project is to demonstrate that an accurate diagnostic DSS covering a large diagnostic repertoire in internal medicine is possible. The key technical innovation is the use of advanced approximate inference methods which allow Bayesian inference to be applied to large problem instances (patent submitted). 3 4 " Promedas " , a prototype DSS
Belief networks / Bayesian networks

In modeling real world tasks, one inevitably has to deal with uncertainty. This uncertainty is due to the fact that many facts are unknown and or simply ignored and summarized. Suppose that one morning you find out that your grass is wet. Is it due to rain, or is it due to the sprinkler? If there is no other information, you can only talk in terms of probabilities. In a probabilistic model approach, you could try to enumerate the states of all variables (grass: wet or dry; rained: true or false; sprinkler: on or off ), and assign probabilities to each combination of states. Ideally, these probabilities will be proportional to the relative frequencies of the occurrence of the combinations of states. The elegance of the probabilistic approach resides in the fact that the probabilistic model on these three variables is correct, consistent and automatically includes context dependency. For instance, you can use the model to compute the probability that it has rained in the context that the grass is wet. You will find an increase in the probability that it has rained. However, in the context that the grass is wet and that the sprinkler has been left on, the probability that it has rained is generally (for sensible choices of the conditional probabilities) lower. In systems that are rule-based rather than based on probability theory context dependency is not fully modeled. In such systems invalid conclusions can be drawn easily. For instance, in a system with context free rules, concatenation of the rule: ‘sprinkler on’ implies ‘wet grass’ with the rule: ‘wet grass’ implies ‘it has rained’, will lead to the incorrect conclusion that ‘sprinkler on’ implies ‘it has rained’. A drawback of probabilistic models is their computational complexity. In problems with many variables the approach in which all combinations of states are enumerated in the model will lead to huge computational problems. The reason is that the number of combinations of states grows exponentially with the number of variables. Even if one manages to parameterize the probabilities in an efficient way, the problem is still not easily solved: inference (i.e. computing probabilities of variables of interest) requires the summation over all (exponentially many) states of the remaining variables. Graphical models provide a remedy. They include Bayesian networks (also known as belief networks), Hidden Markov models, Markov fields, naive Bayes and many others. In graphical models, the probability distribution is defined in terms of local quantities, involving only a few variables. In particular, the local structure can be represented by a graph; hence the name graphical model. The local quantities are glued together according to the laws of probability theory, such that they define a unique and consistent global probability distribution.
Fractional Belief Propagation

We consider loopy belief propagation for approximate inference in probabilistic graphical models. A limitation of the standard algorithm is that clique marginals are computed as if there were no loops in the graph. To overcome this limitation, we introduce fractional belief propagation. Fractional belief propagation is formulated in terms of a family of approximate free energies, which includes the Bethe free energy and the naive mean-field free as special cases. Using the linear response correction of the clique marginals, the scale parameters can be tuned. Simulation results illustrate the potential merits of the approach.
Mean field theory for graphical models

In this chapter, mean eld theory is introduced from an information theoretic view point. The mean eld approximation is deened as the factorized distribution that is closest to the target distribution. When using the KL divergence to deene closeness, this factorized distribution must have equal marginals as the target distribution. Such marginals can be approximately computed by using a Taylor series expansion in the couplings around the factorized distribution. To lowest order in the couplings, the usual naive mean eld equations are obtained and to second order, one obtains the TAP equations. An important advantage of this procedure is that it does not require the concept of a free energy. Therefore, it can be applied to arbitrary probability distributions, such as arising in asymmetric stochastic neural networks and graphical models. 1.1  During the last few years, the use of probabilistic methods in artii-cial intelligence and machine learning has gained enormous popularity. In particular, probabilistic graphical models have become the preferred method for knowledge representation and reasoning 1]. The advantage of the probabilistic approach is that all assumptions are made explicit in the modeling process and that consequences, such as predictions on novel data, are assumption free and follow from a mechanistic computation. The drawback of the probabilistic approach is that the method is intractable. This means that the typical computation scales exponentially with the problem size. Recently, a number of authors have proposed methods for approximate inference in large graphical models. The simplest approach gives a lower bound on the probability of a subset of variables using Jenssen's inequality 2]. The method involves the minimization of the KL divergence between the target probability distribution p and some 'simple' variational distribution q. The method can be applied to a any probability model, whether directed or undirected. The Boltzmann-Gibbs distributions is widely used in physics, and mean eld theory has been known for these distributions for a long time. For instance, for the Ising model on a square lattice, it is known as the Bragg-Williams approximation 3] and it is generalized to other models in the Landau theory 4]. One can show that the above lower bound corresponds to the rst term in a Taylor series expansion of the free energy around a factorized model. This Taylor series can be continued and the second order term is known as the Thouless Anderson Palmer (TAP) correction 5, 6, 7, 8]. The second …
Novel iteration schemes for the Cluster Variation Method

The Cluster Variation method is a class of approximation methods containing the Bethe and Kikuchi approximations as special cases. We derive two novel iteration schemes for the Cluster Variation  One is a fixed point iteration scheme which gives a significant improvement over loopy BP. mean field and TAP methods on directed graphical models. The other is a gradient based method, that is guaranteed to converge and is shown to give useful results on random graphs with mild frustration. We conclude that the methods are of significant practical value for large inference problems.
Second Order Approximations for Probability Models

In this paper, we derive a second order mean field theory for directed graphical probability models. By using an information theoretic argument it is shown how this can be done in the absense of a partition function. This method is a direct generalisation of the well-known TAP approximation for Boltzmann Machines. In a numerical example, it is shown that the method greatly improves the first order mean field approximation. For a restricted class of graphical models, so-called single overlap graphs, the second order method has comparable complexity to the first order method. For sigmoid belief networks, the method is shown to be particularly fast and effective.
Variational methods for approximate reasoning in graphical models

Exact inference in large and complex graphical models e g Bayesian networks is computationally intractable Approximate schemes are therefore of great importance for real world computation In this paper we consider a general scheme in which the original intractable graphical model is approximated by a model with a tractable structure The approximatingmodel is optimised by an iterative procedure which minimises the Kullback Leibler divergence between the two models The procedure is guaranteed to converge to a local minimum of the Kullback Leibler divergence The scheme provides a bridge between naive mean eld theory and exact computation Simulation results are provided to illustrate the method
Variational Approximations between Mean Field Theory and the Junction Tree Algorithm

Recently, variational approximations such as the mean field approximation have received much interest. We extend the standard mean field method by using an approximating distribution that factorises into cluster potentials. This includes undirected graphs, directed acyclic graphs and junction trees. We derive generalized mean field equations to optimize the cluster potentials. We show that the method bridges the gap between the standard mean field approximation and the exact junction tree algorithm. In addition, we address the problem of how to choose the graphical structure of the approximating distribution. From the generalised mean field equations we derive rules to simplify the structure of the approximating distribution in advance without affecting the quality of the approximation. We also show how the method fits into some other variational approximations that are currently popular.
Generative vector quantisation

Based on the assumption that a pattern is constructed out of features which are either fully present or absent, we propose a vector quantisation method which constructs patterns using binary combinations of features. For this model there exists an efficient EM-like learning algorithm which learns a set of representative codebook vectors. In terms of a generative model, the collection of allowed binary states 'generates' the set of codebook vectors. Thus, the method provides not only a compact description of the data in terms of clusters, but also an explanation of the individual clusters in terms of common elementary features. Preliminary results on image compression and handwritten digit analysis indicate that our approach is a computationally inexpensive alternative to more complex probabilistic generative graphical models.
A Development Protocol for a Diagnostic DSS

A diagnostic decision support system (DSS) in medicine is an expert system that aids the physician in the determination of the diagnosis based on findings and test results. The DSS can be divided into 2 different types of components: the knowledge component and the information system component. Methods from software engineering, knowledge engineering and management are combined into a dynamic development cycle that allows stepwise update and refinement. The development of the knowledge component is based on knowledge engineering. In the starting phases, rapid prototyping is convenient to determine and evaluate the specification. The content of the knowledge base is frequently updated during its life time. For this purpose, a knowledge modelling protocol is supplied.
Variational Belief Networks for Approximate InferenceWim Wiegerinck

Exact inference in large, densely connected probabilistic networks is computa-tionally intractable, and approximate schemes are therefore of great importance. One approach is to use mean eld theory, in which the exact log-likelihood is bounded from below using a simpler approximating distribution. In the standard mean eld theory, the approximating distribution is factorial. We propose instead to use a (tractable) belief network as an approximating distribution. The resulting compact framework is analogous to standard mean eld theory and no additional bounds are required, in contrast to other recently proposed extensions. We derive mean eld equations which provide an eecient iterative algorithm to optimize the parameters of the approximating belief network. Simulation results indicate a considerable improvement on previously reported methods.
Mean Field Theory based on Belief Networks for Approximate Inference

Exact inference in large, densely connected belief networks is computationally intractable, and approximate schemes are therefore of great importance. In the context of approximate inference in sigmoid belief networks, mean field theory has received much interest. In this method the exact log-likelihood is bounded from below using a mean field approximating distribution. In the standard mean field theory, the approximating distribution is assumed to be factorial. In this paper we propose to use a (tractable) belief network as an approximating distribution. We show that belief networks fit very well into mean field theory, and no additional bounds are required. We derive mean field equations which provide an efficient iterative algorithm to optimize the parameters of the approximating distribution. Simulation results on an inference problem indicates a considerable improvement over existing mean field methods.
Tractable Undirected Approximations for Graphical Models

Graphical models provide a broad framework for probabilistic inference, with application to such diverse areas as speech recognition (Hidden Markov Models), medical diagnosis (Belief networks) and artificial intelligence (Boltzmann Machines). However, the computing time is typically exponential in the number of nodes in the graph. We present a general framework for a class of approximating models, based on the Kullback-Leibler divergence between an approximating graph and the original graph. We concentrate here on undirected approximations of both intractable directed and undirected graphical models. Simulation results on a small benchmark problem suggest that this method compares favourably against others previously reported in the literature.