nils-herrmann/clean_scirepeval huggingface dataset:
Dynamics of batch training in a perceptron

Early stopping and weight decay are studied in a linear perceptron using a new simplified approach to the dynamics in the thermodynamical limit. The approach is directly deduced from the gradient descent weight update. It allows an exact description of the dynamics of the batch training process. The results are compared with a recent study of early stopping and weight decay based on the equilibrium statistical mechanics approach. It is shown that the equilibrium results for early stopping are good approximations and are exact for weight decay. Furthermore, in the dynamical approach it is possible to determine the necessary number of training steps to fulfil certain termination conditions. It can be shown that asymptotically, i.e. if the number of examples is large, only two batch steps are required to reach the optimal convergence, if the learning rate is optimally chosen.
Finite-Dimensional Approximation of Gaussian Processes

Gaussian process (GP) prediction suffers from O(n3) scaling with the data set size n. By using a finite-dimensional basis to approximate the GP predictor, the computational complexity can be reduced. We derive optimal finite-dimensional predictors under a number of assumptions, and show the superiority of these predictors over the Projected Bayes Regression method (which is asymptotically optimal). We also show how to calculate the minimal model size for a given n. The calculations are backed up by numerical experiments.
Convexity, internal representations and the statistical mechanics of neural networks

We present an approach to the statistical mechanics of feedforward neural networks which is based on counting realizable internal representations by utilizing convexity properties of the weight space. For a toy model, our method yields storage capacities based on an annealed approximation, which are in close agreement with one-step replica symmetry-breaking results obtained from a standard approach. For a single-layer perceptron, a combinatorial result for the number of realizable output combinations is recovered and generalized to fixed stabilities.
MUTUAL INFORMATION, METRIC ENTROPY AND CUMULATIVE RELATIVE ENTROPY RISK

is a set of probability distributions with a common dominating measure on a complete separable metric space Y. A state θ * ∈Θ is chosen by Nature. A statistician obtains n independent observations Y 1 ,..., Y n from Y distributed according to P θ . For each time t between 1 and n, based on the observations Y 1 ,..., Y t-1 , the statistician produces an estimated distribution P t for P θ* and suffers a loss L(P θ* , P t ). The cumulative risk for the statistician is the average total loss up to time n. Of special interest in information theory, data compression, mathematical finance, computational learning theory and statistical mechanics is the special case when the loss L(P θ* , P t ) is the relative entropy between the true distribution P θ* and the estimated distribution P t . Here the cumulative Bayes risk from time 1 to n is the mutual information between the random parameter Θ* and the observations Y 1 ,... Y n . New bounds on this mutual information are given in terms of the Laplace transform of the Hellinger distance between pairs of distributions indexed by parameters in Θ. From these, bounds on the cumulative minimax risk are given in terms of the metric entropy of Θ with respect to the Hellinger distance. The assumptions required for these bounds are very general and do not depend on the choice of the dominating measure. They apply to both finite- and infinite-dimensional Θ. They apply in some cases where Y is infinite dimensional, in some cases where Y is not compact, in some cases where the distributions are not smooth and in some parametric cases where asymptotic normality of the posterior distribution fails.
Metric Entropy and Minimax Risk in Classification

We apply recent results on the minimax risk in density estimation to the related problem of pattern classification. The notion of loss we seek to minimize is an information theoretic measure of how well we can predict the classification of future examples, given the classification of previously seen examples. We give an asymptotic characterization of the minimax risk in terms of the metric entropy properties of the class of distributions that might be generating the examples. We then use these results to characterize the minimax risk in the special case of noisy two-valued classification problems in terms of the Assouad density and the Vapnik-Chervonenkis dimension.
MUTUAL INFORMATION, METRIC ENTROPY, AND RISK IN ESTIMATION OF PROBABILITY DISTRIBUTIONS

Assume $\{P_\theta: \theta \in \Theta\}$ is a set of probability distributions with a common dominating measure on a complete separable metric space $Y$. A state $\theta^* \in \Theta$ is chosen by Nature. A statistician gets $n$ independent observations $Y_1, \ldots, Y_n$ distributed according to $P_{\theta^*}$. For each time $t$ between 1 and $n$, based on the observations $Y_1, \ldots, Y_{t-1}$, the statistician produces an estimated distribution $Q_t$ for $P_{\theta^*}$, and suffers a loss equal to the average of L(P_{\theta^*},Q_t)$ over the observations. The cumulative risk for the statistician is the average total loss up to time $n$. Of special interest in information theory, data compression, mathematical finance, computational learning theory and statistical physics is the special case when the loss $L(P_{\theta^*},Q_t)$ is the relative entropy between the true distribution $P_{\theta^*}$ and the estimated distribution $Q_t$. Here the cumulative Bayes risk is the mutual information between the random parameter $\Theta^*$ and the observations $Y_1, \ldots, Y_n$. New bounds on this mutual information are given in terms of the Laplace transform of the Hellinger distance between distributions in $\Theta$. From these, bounds on the cumulative minimax risk are given in terms of the metric entropy of $\Theta$ with respect to the Hellinger distance. The assumptions required for these bounds are very general and do not depend on the choice of the dominating measure. They apply to both finite and infinite dimensional $\Theta$. They apply in some cases where $Y$ is infinite dimensional, in some cases where $Y$ is not compact, in some cases where the distributions are not smooth, and in some parametric cases where asymptotic normality of the posterior distribution fails. Using these bounds for cumulative relative entropy risk, we also examine the Bayes and minimax risk of this game at specific times $t$ for various loss functions $L$, including the relative entropy, the squared Hellinger distance, and the $L_1$ distance.
Mutual Information and Bayes Methods for Learning a Distribution

Each parameter w in an abstract parameter space W is associated with a diier-ent probability distribution on a set Y. A parameter w is chosen at random from W according to some a priori distribution on W , a n d n conditionally independent random variables Y n = Y 1 : : : Y n are observed with common distribution determined by w. Viewing W as a random variable, we obtain bounds on the mutual information between the random variable W , giving the choice of parameter , and the random variable Y n , giving the sequence of observations. This quantity is the cumulative risk in predicting Y 1 : : : Y n under the log loss, minus the risk if the true parameter w is known. The upper bounds are stated in terms of the Laplace transform of the rate of growth of the volume of relative e n tropy neighborhoods in the parameter space W , and the lower bounds are given in terms of the corresponding quantity using Hellinger neighborhoods. We show how these bounds can be interpreted in terms of an average local dimension of the parameter space W under suitable conditions.
Maximal stability in unsupervised learning

A perceptron can classify a set of random patterns into two groups. The maximal gap between the two groups is calculated using replica theory. The replica-symmetric solution is shown to be unstable and gives results which are qualitatively different from the one-step replica-symmetry breaking solution (RSB1). The results of a calculation without the replica method are given, yielding an exact upper bound for the gap which almost coincides with the RSB1 solution. Finding the classification of a set of patterns with a maximal gap is a difficult combinatorial optimization problem. An algorithm is developed which gives large gaps.
On the relaxation of infinite-range spin glasses

The relaxation of the Sherrington-Kirkpatrick model for spin glasses is studied at low temperatures. A recently developed numerical method is used by which an infinitely large system is simulated, hence finite size effects are avoided. For parallel dynamics the decay of the energy as well as of the magnetization is investigated. For low temperatures we find evidence for a relaxation into a state which is characterized by non-equilibrium values of the energy and a non-vanishing remanent magnetization.
GENERAL BOUNDS FOR PREDICTIVE ERRORS IN SUPERVISED LEARNING

Within a Bayesian framework, we calculate general upper and lower bounds for a cumulative entropic error, which measures the success in the supervised learning of an unknown rule from examples. This performance measure is equivalent to the mutual information between the data and the parameter that specifies the rule to be learnt. Both bounds match asymptotically, when the number m of observed data grows large. Under mild conditions, we find that the information gain from observing a new example decreases universally like d/m. Here d is a dimension that is defined from the scaling of small volumes with respect to a suitable distance in the space of rules.
General bounds on the mutual information between a parameter and n conditionally independent observations

Each parameter in an abstract parameter space is associated with a di erent probability distribution on a set Y A parameter is chosen at random from ac cording to some a priori distribution on and n condi tionally independent random variables Y n Y Yn are observed with common distribution determined by We obtain bounds on the mutual information be tween the random variable giving the choice of pa rameter and the random variable Y n giving the se quence of observations We also bound the supremum of the mutual information over choices of the prior dis tribution on These quantities have applications in density estimation computational learning theory uni versal coding hypothesis testing and portfolio selection theory The bounds are given in terms of the metric and information dimensions of the parameter space with respect to the Hellinger distance
Gaussian Processes for Machine Learning

Gaussian processes (GPs) provide a principled, practical, probabilistic approach to learning in kernel machines. GPs have received growing attention in the machine learning community over the past decade. The book provides a long-needed, systematic and unified treatment of theoretical and practical aspects of GPs in machine learning. The treatment is comprehensive and self-contained, targeted at researchers and students in machine learning and applied statistics. The book deals with the supervised learning problem for both regression and classification, and includes detailed algorithms. A wide variety of covariance (kernel) functions are presented and their properties discussed. Model selection is discussed both from a Bayesian and classical perspective. Many connections to other well-known techniques from machine learning and statistics are discussed, including support vector machines, neural networks, splines, regularization networks, relevance vector machines and others. Theoretical issues including learning curves and the PAC-Bayesian framework are treated, and several approximation methods for learning with large datasets are discussed. The book contains illustrative examples and exercises. Code and datasets can be obtained on the web. Appendices provide mathematical background and a discussion of Gaussian Markov processes.
Unsupervised Learning of Multiple Aspects of Moving Objects from Video

A popular framework for the interpretation of image sequences is based on the layered model; see e.g. Wang and Adelson , Irani et al. . Jojic and Frey  provide a generative probabilistic model framework for this task. However, this layered models do not explicitly account for variation due to changes in the pose and self occlusion. In this paper we show that if the motion of the object is large so that different aspects (or views) of the object are visible at different times in the sequence, we can learn appearance models of the different aspects using a mixture modelling approach.
Gaussian Processes for Machine Learning (Adaptive Computation and Machine Learning)

Gaussian processes (GPs) provide a principled, practical, probabilistic approach to learning in kernel machines. GPs have received increased attention in the machine-learning community over the past decade, and this book provides a long-needed systematic and unified treatment of theoretical and practical aspects of GPs in machine learning. The treatment is comprehensive and self-contained, targeted at researchers and students in machine learning and applied statistics. The book deals with the supervised-learning problem for both regression and classification, and includes detailed algorithms. A wide variety of covariance (kernel) functions are presented and their properties discussed. Model selection is discussed both from a Bayesian and a classical perspective. Many connections to other well-known techniques from machine learning and statistics are discussed, including support-vector machines, neural networks, splines, regularization networks, relevance vector machines and others. Theoretical issues including learning curves and the PAC-Bayesian framework are treated, and several approximation methods for learning with large datasets are discussed. The book contains illustrative examples and exercises, and code and datasets are available on the Web. Appendixes provide mathematical background and a discussion of Gaussian Markov processes.
Datasets and Code

The datasets used for experiments in this book and implementations of the algorithms presented are available for download at the website of the book: The programs are short stand-alone implementations and not part of a larger package. They are meant to be simple to understand and modify for a desired purpose. Some of the programs allow specification of covariance functions from a selection provided, or to link in user defined covariance code. For some of the plots, code is provided which produces a similar plot, as this may be a convenient way of conveying the details.
Factorial Switching Kalman Filters for Condition Monitoring in Neonatal Intensive Care

The observed physiological dynamics of an infant receiving intensive care are affected by many possible factors, including interventions to the baby, the operation of the monitoring equipment and the state of health. The Factorial Switching Kalman Filter can be used to infer the presence of such factors from a sequence of observations, and to estimate the true values where these observations have been corrupted. We apply this model to clinical time series data and show it to be effective in identifying a number of artifactual and physiological patterns.
How to Pretend That Correlated Variables Are Independent by Using Difference Observations

In many areas of data modeling, observations at different locations (e.g., time frames or pixel locations) are augmented by differences of nearby observations (e.g., features in speech recognition, Gabor jets in image analysis). These augmented observations are then often modeled as being independent. How can this make sense? We provide two interpretations, showing (1) that the likelihood of data generated from an autoregressive process can be computed in terms of independent augmented observations and (2) that the augmented observations can be given a coherent treatment in terms of the products of experts model (Hinton, 1999).
Fast Learning of Sprites using Invariant Features

A popular framework for the interpretation of image sequences is the layers or sprite model of e.g. Wang and Adelson (1994), Irani et al. (1994). Jojic and Frey (2001) provide a generative probabilistic model framework for this task, but their algorithm is slow as it needs to search over discretized transformations (e.g. translations, or affines) for each layer. In this paper we show that by using invariant features (e.g. Lowe’s SIFT features) and clustering their motions we can reduce or eliminate the search and thus learn the sprites much faster. We demonstrate our algorithm on two image sequences.
Pascal Visual Object Classes Challenge Results

The goal of this challenge is to recognize objects from a number of visual object classes in images of realistic scenes. It is fundamentally a supervised learning learning problem in that a training set of labelled images is provided. The object classes are: motorbikes, bicycles, people and cars. Twelve participants entered the challenge. A full description of the challenge including software and image sets is available on the web page http://www.pascal-network.org/challenges/VOC/voc/index.html.
Gaussian Markov Processes

Gaussian processes (GPs) provide a principled, practical, probabilistic approach to learning in kernel machines. GPs have received increased attention in the machine-learning community over the past decade, and this book provides a long-needed systematic and unified treatment of theoretical and practical aspects of GPs in machine learning. The treatment is comprehensive and self-contained, targeted at researchers and students in machine learning and applied statistics.The book deals with the supervised-learning problem for both regression and classification, and includes detailed algorithms. A wide variety of covariance (kernel) functions are presented and their properties discussed. Model selection is discussed both from a Bayesian and a classical perspective. Many connections to other well-known techniques from machine learning and statistics are discussed, including support-vector machines, neural networks, splines, regularization networks, relevance vector machines and others. Theoretical issues including learning curves and the PAC-Bayesian framework are treated, and several approximation methods for learning with large datasets are discussed. The book contains illustrative examples and exercises, and code and datasets are available on the Web. Appendixes provide mathematical background and a discussion of Gaussian Markov processes.
Fast Learning of Sprites using Invariant

A popular framework for the interpretation of image sequences is the layers or sprite model of e.g. Wang and Adelson (1994), Irani et al. (1994). Jojic and Frey (2001) provide a generative probabilistic model framework for this task, but their algorithm is slow as it needs to search over discretized transformations (e.g. translations, or affines) for each layer. In this paper we show that by using invariant features (e.g. Lowe’s SIFT features) and clustering their motions we can reduce or eliminate the search and thus learn the sprites much faster. We demonstrate our algorithm on two image sequences.
Probabilistic in Silico Prediction of Protein-Peptide Interactions

Peptide recognition modules (PRMs) are specialised compact protein domains that mediate many important protein-protein interactions. They are responsible for the assembly of critical macromolecular complexes and biochemical pathways [Pawson and Scott, 1997], and they have been implicated in carcinogenesis and various other human diseases [Sudol and Hunter, 2000]. PRMs recognise and bind to peptide ligands that contain a specific structural motif. This paper introduces a novel discriminative model which models these PRMs and allows prediction of their behaviour, which we compare with a recently proposed generative model. We find that on a yeast two-hybrid dataset, the generative model performs better when background sequences are included, while our discriminative model performs better when the evaluation is focused on discriminating between the SH3 domains. Our model is also evaluated on a phage display dataset, where it consistently out-performed the generative model.
Understanding Gaussian Process Regression Using the Equivalent Kernel

The equivalent kernel  is a way of understanding how Gaussian process regression works for large sample sizes based on a continuum limit. In this paper we show how to approximate the equivalent kernel of the widely-used squared exponential (or Gaussian) kernel and related kernels. This is easiest for uniform input densities, but we also discuss the generalization to the non-uniform case. We show further that the equivalent kernel can be used to understand the learning curves for Gaussian processes, and investigate how kernel smoothing using the equivalent kernel compares to full Gaussian process regression.
Fast Unsupervised Greedy Learning of Multiple Objects and Parts from Video

Williams and Titsias (2004) have shown how to carry out unsupervised greedy learning of multiple objects from images (GLOMO), building on the work of Jojic and Frey (2001). In this paper we show that the earlier work on GLOMO can be greatly speeded up for video sequence data by carrying out approximate tracking of the multiple objects in the scene. Our method is applied to raw image sequence data and extracts the objects one at a time. First, the moving background is learned, and moving objects are found at later stages. The algorithm recursively updates an appearance model of the tracked object so that possible occlusion of the object is taken into account which makes tracking stable. We apply this method to learn multiple objects in image sequences as well as articulated parts of the human body.
Greedy Learning of Multiple Objects in Images Using Robust Statistics and Factorial Learning

We consider data that are images containing views of multiple objects. Our task is to learn about each of the objects present in the images. This task can be approached as a factorial learning problem, where each image must be explained by instantiating a model for each of the objects present with the correct instantiation parameters. A major problem with learning a factorial model is that as the number of objects increases, there is a combinatorial explosion of the number of configurations that need to be considered. We develop a method to extract object models sequentially from the data by making use of a robust statistical method, thus avoiding the combinatorial explosion, and present results showing successful extraction of objects from real images.
Upper and Lower Bounds on the Learning Curve for Gaussian Processes

In this paper we introduce and illustrate non-trivial upper and lower bounds on the learning curves for one-dimensional Guassian Processes. The analysis is carried out emphasising the effects induced on the bounds by the smoothness of the random process described by the Modified Bessel and the Squared Exponential covariance functions. We present an explanation of the early, linearly-decreasing behavior of the learning curves and the bounds as well as a study of the asymptotic behavior of the curves. The effects of the noise level and the lengthscale on the tightness of the bounds are also discussed.
On a Connection between Kernel PCA and Metric Multidimensional Scaling

In this note we show that the kernel PCA algorithm of Schölkopf, Smola, and Müller (Neural Computation, 10, 1299–1319.) can be interpreted as a form of metric multidimensional scaling (MDS) when the kernel function k(x, y) is isotropic, i.e. it depends only on ‖x − y‖. This leads to a metric MDS algorithm where the desired configuration of points is found via the solution of an eigenproblem rather than through the iterative optimization of the stress objective function. The question of kernel choice is also discussed.