nils-herrmann/clean_scirepeval huggingface dataset:
An Idiosyncratic Journey Beyond Mean Field Theory

I try to clarify the relationships between different ways of deriving or correcting mean field theory, and present ”translations” between the language of physicists and that of computer scientists. The connecting thread between the different methods described here is the Gibbs free energy. After introducing the inference problem we are interested in analyzing, I will define the Gibbs free energy, and describe how to derive a mean field approximation to it using a variational approach. I will then explain how one might re-derive and correct the mean field and TAP free energies using high temperature expansions with constrained one-node beliefs. I will explore the relationships between the high-temperature expansion approach, the Bethe approximation, and the belief propagation algorithm, and point out in particular the equivalence of the Bethe approximation and belief propagation. Finally, I will describe Kikuchi approximations to the Gibbs Free energy and advertise new belief propagation algorithms that efficiently compute beliefs equivalent to those obtained from the Kikuchi free energy. To appear as a chapter in “Advanced Mean Field  Theory and Practice”, eds. D. Saad and M. Opper, MIT Press, 2000. This work may not be copied or reproduced in whole or in part for any commercial purpose. Permission to copy in whole or in part without payment of fee is granted for nonprofit educational and research purposes provided that all such whole or partial copies include the following: a notice that such copying is by permission of Mitsubishi Electric Information Technology Center America; an acknowledgment of the authors and individual contributions to the work; and all applicable portions of the copyright notice. Copying, reproduction, or republishing for any other purpose shall require a license with payment of fee to Mitsubishi Electric Information Technology Center America. All rights reserved.
Generalized Belief Propagation

Belief propagation (BP) was only supposed to work for treelike networks but works surprisingly well in many applications involving networks with loops, including turbo codes. However, there has been little understanding of the algorithm or the nature of the solutions it finds for general graphs. 
 
We show that BP can only converge to a stationary point of an approximate free energy, known as the Bethe free energy in statistical physics. This result characterizes BP fixed-points and makes connections with variational approaches to approximate inference. 
 
More importantly, our analysis lets us build on the progress made in statistical physics since Bethe's approximation was introduced in 1935. Kikuchi and others have shown how to construct more accurate free energy approximations, of which Bethe's approximation is the simplest. Exploiting the insights from our analysis, we derive generalized belief propagation (GBP) versions of these Kikuchi approximations. These new message passing algorithms can be significantly more accurate than ordinary BP, at an adjustable increase in complexity. We illustrate such a new GBP algorithm on a grid Markov network and show that it gives much more accurate marginal probabilities than those found using ordinary BP.
Tangible interaction + graphical interpretation: a new approach to 3D modeling

Construction toys are a superb medium for geometric models. We argue that such toys, suitably instrumented or sensed, could be the inspiration for a new generation of easy-to-use, tangible modeling systems—especially if the tangible modeling is combined with graphical-interpretation techniques for enhancing nascent models automatically. The three key technologies needed to realize this idea are embedded computation, vision-based acquisition, and graphical interpretation. We sample these technologies in the context of two novel modeling systems: physical building blocks that self-describe, interpret, and decorate the structures into which they are assembled; and a system for scanning, interpreting, and animating clay figures.
Inference for biochemical reaction systems based on a weak noise approximation

The marginal likelihood function �(x, t) is defined as the probability that a process with drift vector f(x) starting at state x and time t generated all observations yk with tk ≥ t. In theory, �(x, t) can be calculated exactly by solving the backward Fokker-Planck equation of the diffusion model with the end condition �(x, tn) = p(yn|x(tn)) and the jump conditions �(x, t − ) = �(x, t + ) p(yk|x(tk)). Here �(x, t − ) and �(x, t + ) denote the values of �(x, t) before and after the k-th observation. Instead we obtain an approximate solution in the weak noise limit by setting D → ǫD and assuming a scaling form �(x, t) ≈ z(t)exp[− 1 2ǫ (x −b(t)) ⊤ S −1 (t)(x −b(t))] for ǫ → 0. To leading order, we find ODEs for b(t), S(t), and z(t). By solving them backwards in time, we obtain approximations to the drift g(x, t) of the posterior process and the free energy F(θ) ≡ −ln p(y1, . . . , yn|θ), i.e. the negative log-likelihood of the data. The posterior statistics can now be computed either by applying the weak noise limit again (leading to an approximate Gaussian posterior process), or more simply by sampling. Additionally, type II maximum likelihood estimates of the parameters θ can be obtained by minimising F(θ) ≈ (2π) d/2 det(S(t1)) 1/2 z(t1). We have applied our method to the stochastic Lotka-Volterra model and an autoregulatory gene network . In both cases the original process, which generated the observations, lies inside the inferred confidence interval most of the time. Thus our algorithm gives reliable information about the state of the reaction system, even if the internal noise is rather large. Parameter estimation also works well, as the inferred values are reasonable close to the real values of the reaction constants. We are currently working on full Bayesian inference by using a Hybrid Monte Carlo method, which samples from the posterior p(θ|y1, . . . , yn) ≈ 1 Z exp(−F(θ))p(θ).
Sequential, Bayesian Geostatistics: A Principled Method for Large Data Sets

The principled statistical application of Gaussian random field models used in geostatistics has historically been limited to data sets of a small size. This limitation is imposed by the requirement to store and invert the covariance matrix of all the samples to obtain a predictive distribution at unsampled locations, or to use likelihood-based covariance estimation. Various ad hoc approaches to solve this problem have been adopted, such as selecting a neighborhood region and/or a small number of observations to use in the kriging process, but these have no sound theoretical basis and it is unclear what information is being lost. In this article, we present a Bayesian method for estimating the posterior mean and covariance structures of a Gaussian random field using a sequential estimation algorithm. By imposing sparsity in a well-defined framework, the algorithm retains a subset of “basis vectors” that best represent the “true” posterior Gaussian random field model in the relative entropy sense. This allows a principled treatment of Gaussian random field models on very large data sets. The method is particularly appropriate when the Gaussian random field model is regarded as a latent variable model, which may be nonlinearly related to the observations. We show the application of the sequential, sparse Bayesian estimation in Gaussian random field models and discuss its merits and drawbacks.
A statistical physics approach for the analysis of machine learning algorithms on real data

We combine the replica approach of statistical physics with a variational technique to make it applicable for the analysis of machine learning algorithms on real data. The method is applied to Gaussian process models and their relative, the support vector machine. We discuss the quality of our theoretical results in comparison to experiments. As a key result, we apply our theory on real world benchmark data and show its potential for practical applications by deriving approximate expressions for data averaged performance measures which hold for general data distributions and allow us to optimize the performance of the learning algorithm.
Bayesian analysis of the scatterometer wind retrieval inverse problem: some new approaches

The retrieval of wind vectors from satellite scatterometer observations is a non‐linear inverse problem. A common approach to solving inverse problems is to adopt a Bayesian framework and to infer the posterior distribution of the parameters of interest given the observations by using a likelihood model relating the observations to the parameters, and a prior distribution over the parameters. We show how Gaussian process priors can be used efficiently with a variety of likelihood models, using local forward (observation) models and direct inverse models for the scatterometer. We present an enhanced Markov chain Monte Carlo method to sample from the resulting multimodal posterior distribution. We go on to show how the computational complexity of the inference can be controlled by using a sparse, sequential Bayes algorithm for estimation with Gaussian processes. This helps to overcome the most serious barrier to the use of probabilistic, Gaussian process methods in remote sensing inverse problems, which is the prohibitively large size of the data sets. We contrast the sampling results with the approximations that are found by using the sparse, sequential Bayes algorithm.
Dicussion on the meeting on ‘Statistical approaches to inverse problems’

Johnstone, Kerkyacharian, Picard and Raimondo Johnstone, Kerkyacharian, Picard and Raimondo are interested in the inverse problem of estimating f where f has been convolved with g and then contaminated with white noise. This popular problem has been tackled by a wide variety of procedures and wavelet methods have recently generated considerable interest. Donoho’s (1995) seminal wavelet–vaguelette paper introduced the notion that wavelets would be a good choice for the representation of f since real life objects, such as images, are more likely to be efficiently represented using wavelets when compared with, for example, Fourier representations. Johnstone and his colleagues have moved the field on significantly. In particular, their procedure is more direct than wavelet–vaguelette or Abramovich and Silverman’s (1998) vaguelette–wavelet method; it can handle boxcar blur theoretically and practically, they have rates of convergence forp =2 (p defines the type of loss) and the paper innovates through use of the new maxiset approach. For me, the most appealing of these innovations is that of enabling the treatment of boxcar blur which is one of the most common types of inverse problem. However, is it really, really, the case that for rational a nothing can be done? Formula (4) compels us to say no, nothing can, but naively it still feels wrong.  More specifically, the zl are zero-mean Gaussian variables which are complex valued and satisfy E.zlzk/= δlk. One question is why threshold the βk and not the yl directly? The covariance of the βk is given by cov.βk, βl/=n ∑ m ΨkmΨ l m
The Complexity of Learning with Supportvector Machines - A Statistical Physics Study

Supportvector Machines (SVMs) have been introduced in recent years by V. Vapnik [ 1, 2] and his collaborators as a novel approach in machine learning (for detailed reviews, see e.g. [3-5]. SVMs seem to have a variety of advantages over otherlearning machines which make them an important alternative to neural networks. E.g., the training of SVMs is performed by minimizing a cost function which has a single minimum only. Despite the mathematical simplicity and elegance of SVM training, SVMs are able to learn tasks of high complexity. In fact, unlike neural networks, which have a fixed number of adjustable parameters, the flexibility of an SVM is not fixed in advance but can grow with the data. This seems like a very pleasant feature because it allows SVMs to memorize arbitrary sets of examples perfectly. However, such an unbounded flexibility could present also a potential danger. For practical applications, when the number of training examples grows large, we would like the learning process to converge to a solution which makes the machine able to generalize well on data that it has not observed before An SVM on the other hand, observing an increasing number of examples might generate hypotheses about the learning task which become more and more complex by exploring more and more of its unbounded learning capabilities. Can we expect that such a process will actually converge to a satisfactory solution when the actual task to be learnt is of .finite (but unknown) complexity? To address this question, we present results of an analysis which is based on a theoretical approach from Statistical Physics. As discussed in the contributions of Michael Biehl and Wolfgang Kinzel in this volume, the method has been already applied successfully to a variety of neural network learning problems (for a detailed review of methods and results, see e.g [6, 7]). It provided exact results for the performance of such machines in controlled analytical experiments on statistical ensembles of typical learning tasks. The approach will allow us to study the generalization performance of SVMs in an important limit where the dimensionality of the space of input features grows large.
An Approximate Analytical Approach to Resampling Averages

Using a novel reformulation, we develop a framework to compute approximate resampling data averages analytically. The method avoids multiple retraining of statistical models on the samples. Our approach uses a combination of the replica "trick" of statistical physics and the TAP approach for approximate Bayesian inference. We demonstrate our approach on regression with Gaussian processes. A comparison with averages obtained by Monte-Carlo sampling shows that our method achieves good accuracy.
Sparse, Sequential Bayesian Geostatistics

Biography Dr. Dan Cornford is a lecture in Computer Science and works in the Neural Computing Research Group at Aston University. Research interests are in the field of spatial statistics, space-time modelling and data assimilation. Lehel Csato is a post-doc in the same group working on an EPSRC grant (GR/R61857/01) looking at applying sparse sequential Gaussian processes to data assimilation. Manfred Opper is a Reader in the same group, with research interests in statistical physics, sequential learning and Bayesian learning.
Region growing with pulse-coupled neural networks: an alternative to seeded region growing

The seeded region growing (SRG) algorithm is a fast robust parameter-free method for segmenting intensity images given initial seed locations for each region. The requirement of predetermined seeds means that the model cannot operate fully autonomously. In this paper, we demonstrate a novel region growing variant of the pulse-coupled neural network (PCNN), which offers comparable performance to the SRG and is able to generate seed locations internally, opening the way to fully autonomous operation.
Sparse On-Line Gaussian Processes

We develop an approach for sparse representations of gaussian process (GP) models (which are Bayesian types of kernel machines) in order to overcome their limitations for large data sets. The method is based on a combination of a Bayesian on-line algorithm, together with a sequential construction of a relevant subsample of the data that fully specifies the prediction of the GP model. By using an appealing parameterization and projection techniques in a reproducing kernel Hilbert space, recursions for the effective parameters and a sparse gaussian approximation of the posterior process are obtained. This allows for both a propagation of predictions and Bayesian error measures. The significance and robustness of our approach are demonstrated on a variety of experiments.
Data Assimilation with Sequential Gaussian Processes

We study a data assimilation problem using Gaussian processes (GPs) where the GPs act as latent variables for the observations. Inference is done using a convenient parameterisation and sequential learning for a faster algorithm. We are addressing the disadvantage of the GPs, namely the quadratic scaling of the parameters with data and eliminate the scaling by using a fixed number of parameters. The result is a sparse representation that allows us to treat problems with a large number of observations. We apply our method to the prediction of wind fields over the ocean surface from scatterometer data.
Asymptotic Universality for Learning Curves of Support Vector Machines

Using methods of Statistical Physics, we investigate the role of model complexity in learning with support vector machines (SVMs). We show the advantages of using SVMs with kernels of infinite complexity on noisy target rules, which, in contrast to common theoretical beliefs, are found to achieve optimal generalization error although the training error does not converge to the generalization error. Moreover, we find a universal asymptotics of the learning curves which only depend on the target rule but not on the SVM kernel.
Universal learning curves of support vector machines.

Using methods of statistical physics, we investigate the role of model complexity in learning with support vector machines (SVMs), which are an important alternative to neural networks. We show the advantages of using SVMs with kernels of infinite complexity on noisy target rules, which, in contrast to common theoretical beliefs, are found to achieve optimal generalization error although the training error does not converge to the generalization error. Moreover, we find a universal asymptotics of the learning curves which depend only on the target rule but not on the SVM kernel.
Advanced mean field methods: theory and practice

A major problem in modern probabilistic modeling is the huge computational complexity involved in typical calculations with multivariate probability distributions when the number of random variables is large. Because exact computations are infeasible in such cases and Monte Carlo sampling techniques may reach their limits, there is a need for methods that allow for efficient approximate computations. One of the simplest approximations is based on the mean field method, which has a long history in statistical physics. The method is widely used, particularly in the growing field of graphical models. Researchers from disciplines such as statistical physics, computer science, and mathematical statistics are studying ways to improve this and related methods and are exploring novel application areas. Leading approaches include the variational approach, which goes beyond factorizable distributions to achieve systematic improvements; the TAP (Thouless-Anderson-Palmer) approach, which incorporates correlations by including effective reaction terms in the mean field theory; and the more general methods of graphical models. Bringing together ideas and techniques from these diverse disciplines, this book covers the theoretical foundations of advanced mean field methods, explores the relation between the different approaches, examines the quality of the approximation obtained, and demonstrates their application to various areas of probabilistic modeling.
Eecient Approaches to Gaussian Process Classiication

We present three simple approximations for the calculation of the posterior mean in Gaussian Process classiication. The rst two methods are related to mean eld ideas known in Statistical Physics. The third approach is based on Bayesian online approach which was motivated by recent results in the Statistical Mechanics of Neural Networks. We present simulation results showing: 1. that the mean eld Bayesian evidence may be used for hyperparameter tuning and 2. that the online approach may achieve a low training error fast.
On-line learning from a finite training set: A solvable model

We discuss the problem of on-line learning from a finite training set with feedforward neural networks. Defining a modified learning rule, which randomly chooses inputs and weights to be updated, the dynamics of learning can be treated within a diffusion approximation in the thermodynamic limit. No assumption on the generation of data is made. Explicit results for the stationary distribution and relaxation times can be found for a network with linear transfer function. Assuming self-averaging of the diffusion term, a general relation between on-line learning and batch learning with an effective temperature is established.
Worst Case Prediction over Sequences under Log Loss

We consider the game of sequentially assigning probabilities to future data based on past observations under logarithmic loss. We are not making probabilistic assumptions about the generation of the data, but consider a situation where a player tries to minimize his loss relative to the loss of the (with hindsight) best distribution from a target class for the worst sequence of data. We give bounds on the minimax regret in terms of the metric entropies of the target class with respect to suitable distances between distributions.
A Bayesian approach to on-line learning

Online learning is discussed from the viewpoint of Bayesian statistical inference. By replacing the true posterior distribution with a simpler parametric distribution, one can define an online algorithm by a repetition of two steps: An update of the approximate posterior, when a new example arrives, and an optimal projection into the parametric family. Choosing this family to be Gaussian, we show that the algorithm achieves asymptotic efficiency. An application to learning in single layer neural networks is given.
Statistical mechanics of Support Vector networks.

Using methods of Statistical Physics, we investigate the generalization performance of support vector machines (SVMs), which have been recently introduced as a general alternative to neural networks. For nonlinear classification rules, the generalization error saturates on a plateau, when the number of examples is too small to properly estimate the coefficients of the nonlinear part. When trained on simple rules, we find that SVMs overfit only weakly. The performance of SVMs is strongly enhanced, when the distribution of the inputs has a gap in feature space.
Bayes-optimal performance in a discrete space

Statistical mechanics techniques have been used with success to study and understand key properties of inferential learning [1, 2]. This approach provides explicit and detailed results that are in many ways complementary to the more general, but also more blend results obtained by statistics. The case of non-smooth problems, in which the parameters that have to be estimated take discrete values, is of particular interest. On the one hand, many of the results from statistics can no longer be applied, while on the other hand, the estimation of these parameters is often a computationally hard problem. In this paper, we present a detailed analysis of a simple model of unsupervised learning [3, 4, 5, 6, 7, 8], involving a single symmetry breaking vector with binary components ±1 and highlight the differences with the case of smooth components. In particular we compare the results from Gibbs learning and Bayes learning with the ones for the best binary vector and a vector which minimizes a variationally optimal potential. The problem is as follows: p N -dimensional patterns {ξ, μ = 1, ..., p}, are sampled independently from a distribution P (ξ|B) ∼ δ(ξ ·ξ−N) exp [