nils-herrmann/clean_scirepeval huggingface dataset:
Analysis of Mean Field Annealing in Substactive Interference Cancellation

In this contribution we derive the cost function corresponding to the linear complexity Subtractive Interference Cancellation with tangent hyperbolic tentative decisions. We use the cost function to analyse the fix-points of solving the Subtractive Interference Cancellation equations. The analysis show that we can control the slope of the tangent hyperbolic functions so that the corresponding cost function is convex. We also show that increasing the slope can make the cost non-convex. Going from the convex regime into the non-convex regime, we prove that the bifurcation of the fix-points, for non-singular signal correlation matrices, consist of the fix-point of interest together with a saddle node bifurcation. This proves that tracking the solution from low slopes, with a convex cost, gradually increasing to higher slopes, with non-convex cost, can bring us to the best solution being very close to the optimal determined by enumeration. This tracking is the idea behind annealing. We show Monte Carlo studies with a substantial signal to noise ratio gain compared to not using annealing. We also show how annealing can be used to increase capacity at a given target bit error rate. In fact this capacity gain is the same obtained by Improved Parallel Interference Cancellation making us believe that the latter includes a mechanism to avoid local minima similar to annealing.
FIELD IMPLEMENTATION OF BAYESIAN ICA

In this contribution we review the mean field approach to Bayesian independent component analysis (ICA) recently developed by the authors [1, 2]. For the chosen setting of additive Gaussian noise on the measured signal and Maximum Likelihood II estimation of the mixing matrix and the noise, the expected sufficient statistics are obtained from the two first posterior moments of the sources. These can be effectively estimated using variational mean field theory and its linear response correction. We give an application to feature extraction in neuro-imaging using a binary (stimuli/no stimuli) source paradigm. Finally, we discuss the possibilities of extending the framework to convolutive mixtures, temporal and ‘spatial’ source prior correlations, identification of common sources in mixtures of different media and ICA for density estimation.
Adaptive and self-averaging Thouless-Anderson-Palmer mean-field theory for probabilistic modeling.

We develop a generalization of the Thouless-Anderson-Palmer (TAP) mean-field approach of disorder physics, which makes the method applicable to the computation of approximate averages in probabilistic models for real data. In contrast to the conventional TAP approach, where the knowledge of the distribution of couplings between the random variables is required, our method adapts to the concrete set of couplings. We show the significance of the approach in two ways: Our approach reproduces replica symmetric results for a wide class of toy models (assuming a nonglassy phase) with given disorder distributions in the thermodynamic limit. On the other hand, simulations on a real data model demonstrate that the method achieves more accurate predictions as compared to conventional TAP approaches.
TAP Gibbs Free Energy, Belief Propagation and Sparsity

The adaptive TAP Gibbs free energy for a general densely connected probabilistic model with quadratic interactions and arbritary single site constraints is derived. We show how a specific sequential minimization of the free energy leads to a generalization of Minka's expectation propagation. Lastly, we derive a sparse representation version of the sequential algorithm. The usefulness of the approach is demonstrated on classification and density estimation with Gaussian processes and on an independent component analysis problem.
MEAN FIELD IMPLEMENTATION OF BAYESIAN ICA

In this contribution we review the mean field approach to Bayesian independent component analysis (ICA) recently developed by the authors [1, 2]. For the chosen setting of additive Gaussian noise on the measured signal and Maximum Likelihood II estimation of the mixing matrix and the noise, the expected sufficient statistics are obtained from the two first posterior moments of the sources. These can be effectively estimated using variational mean field theory and its linear response correction. We give an application to feature extraction in neuro-imaging using a binary (stimuli/no stimuli) source paradigm. Finally, we discuss the possibilities of extending the framework to convolutive mixtures, temporal and ‘spatial’ source prior correlations, identification of common sources in mixtures of different media and ICA for density estimation.
Computing with Finite and Infinite Networks

Using statistical mechanics results, I calculate learning curves (average generalization error) for Gaussian processes (GPs) and Bayesian neural networks (NNs) used for regression. Applying the results to learning a teacher defined by a two-layer network, I can directly compare GP and Bayesian NN learning. I find that a GP in general requires O(ds)-training examples to learn input features of order s(d is the input dimension), whereas a NN can learn the task with order the number of adjustable weights training examples. Since a GP can be considered as an infinite NN, the results show that even in the Bayesian approach, it is important to limit the complexity of the learning machine. The theoretical findings are confirmed in simulations with analytical GP learning and a NN mean field algorithm.
Ensemble Learning and Linear Response Theory for ICA

We propose a general Bayesian framework for performing independent component analysis (ICA) which relies on ensemble learning and linear response theory known from statistical physics. We apply it to both discrete and continuous sources. For the continuous source the underdetermined (overcomplete) case is studied. The naive mean-field approach fails in this case whereas linear response theory-which gives an improved estimate of covariances-is very efficient. The examples given are for sources without temporal correlations. However, this derivation can easily be extended to treat temporal correlations. Finally, the framework offers a simple way of generating new ICA algorithms without needing to define the prior distribution of the sources explicitly.
Gaussian Processes for Classification: Mean-Field Algorithms

We derive a mean-field algorithm for binary classification with gaussian processes that is based on the TAP approach originally proposed in statistical physics of disordered systems. The theory also yields an approximate leave-one-out estimator for the generalization error, which is computed with no extra computational cost. We show that from the TAP approach, it is possible to derive both a simpler naive mean-field theory and support vector machines (SVMs) as limiting cases. For both mean-field algorithms and support vector machines, simulation results for three small benchmark data sets are presented. They show that one may get state-of-the-art performance by using the leave-one-out estimator for model selection and the built-in leave-one-out estimators are extremely precise when compared to the exact leave-one-out estimate. The second result is taken as strong support for the internal consistency of the mean-field approach.
Gaussian processes and SVM: Mean field results and leave-one-out

In this chapter, we elaborate on the well-known relationship between Gaussian processes (GP) and Support Vector Machines (SVM). Secondly, we present approximate solutions for two computational problems arising in GP and SVM. The first one is the calculation of the posterior mean for GP classifiers using a `naive' mean field approach. The second one is a leave-one-out estimator for the generalization error of SVM based on a linear response method. Simulation results on a benchmark dataset show similar performances for the GP mean field algorithm and the SVM algorithm. The approximate leave-one-out estimator is found to be in very good agreement with the exact leave-one-out error.
Embedded Systems Design with 8051 Microcontrollers : Hardware and Software

From the Publisher: 
The explosive proliferation of built-in computers and the variety of design methods developed in both industry and academia necessitates the sort of pragmatic guidance offered in Embedded Systems Design with 8051 Microcontrollers. This reference/text explains the developments in microcontroller technology and provides lucid instructions on its many and varied applications - focusing on the popular 8-bit microcontroller, the 8051, and the 83C552.
Optimal online learning : a Bayesian approachSara

A recently proposed Bayesian approach to online learning is applied to learning a rule deened as a noisy single layer perceptron. In the Bayesian online approach, the exact posterior distribution is approximated by a simple parametric posterior that is updated online as new examples are incorporated to the dataset. In the case of binary weights, the approximate posterior is chosen to be a biased binary distribution. The resulting online algorithm is shown to outperform several other online approaches to this problem. Neural networks are adaptive systems characterized by a set of parameters w, the weights and biases that specify the connectivity among the neuronal computational elements. Consider the case of multilayer architectures, which implement deterministic maps from an input space fxg onto an output space fyg. For a xed architecture, the properties of the speciic input-output map y = f w (x) implemented by the network depend only on the values selected for the network parameters w. Of particular interest is the ability of these systems to learn from examples. In the supervised learning scenario investigated here, each example is in the form of an input-output pair (x; y). A training set of size t is denoted by D t = f(x ; y); 1 tg; every example is an input-output pair labeled by the integer counting index. Traditional formulations of the learning problem are based on a dynamical prescription for the adaptation of the parameters w. The learning process 1 We acknowledge the support of CONNECT, the Computational Neural Network Center at the Niels Bohr Institute, Copenhagen, Denmark; CONNECT provided the hospitable environment where most of the discussions that led to this work took place.
Efficient Approaches to Gaussian Process Classification

We present three simple approximations for the calculation of the posterior mean in Gaussian Process classification. The first two methods are related to mean field ideas known in Statistical Physics. The third approach is based on Bayesian online approach which was motivated by recent results in the Statistical Mechanics of Neural Networks. We present simulation results showing: 1. that the mean field Bayesian evidence may be used for hyperparameter tuning and 2. that the online approach may achieve a low training error fast.
Optimal Perceptron Learning : anOnline Bayesian

The recently proposed Bayesian approach to online learning is applied to learning a rule deened as a noisy single layer perceptron with either continuous or binary weights. In the Bayesian online approach the exact posterior distribution is approximated by a simpler paramet-ric posterior that is updated online as new examples are incorporated to the dataset. In the case of continuous weights, the approximate posterior is chosen to be Gaussian. The computational complexity of the resulting online algorithm is found to be at least as high as that of the Bayesian ooine approach, making the online approach less attractive. A Hebbian approximation based on casting the full covariance matrix into an isotropic diagonal form signiicantly reduces the computational complexity and yields a previously identiied optimal Hebbian algorithm. In the case of binary weights, the approximate posterior is chosen to be a biased binary distribution. The resulting online algorithm is derived and shown to outperform several other online approaches to this problem.
Optimal learning in multilayer neural networks

The generalization performance of two learning algorithms, Bayes algorithm and the ``optimal learning'' algorithm, on two classification tasks is studied theoretically. In the first example the task is defined by a restricted two-layer network, a committee machine, and in the second the task is defined by the so-called prototype problem. The architecture of the learning machine, in both cases, is defined to be a committee machine. For both tasks the optimal learning algorithm, which is optimal when the solution is restricted to a specific architecture, performs worse than the overall optimal Bayes algorithm. However, both algorithms perform better than the conventional stochastic Gibbs algorithm, especially for the prototype problem in which the task and the learning machine are very different.
A Mean Eld Algorithm for Bayes Learning in Large Feed-forward Neural Networks

We present a mean eld algorithm to realise Bayes optimal predictions in large feed-forward networks. The derivation of the algorithm is based on methods developed within statistical mechanics of disordered systems. The algorithm also provides a leave-one-out cross-validation test of the predictions. Simulations show excellent agreement with theoretical results of statistical mechanics. Furthermore we derive other simpler mean eld algorithms and make a comparison study.
Bayesian online learning in the perceptron

In a Bayesian approach to online learning a simple approximate parametric form for posterior is updated in each online learning step. Usually in online learning only an estimate of the solution is updated. The Bayesian online approach is applied to two simple learning scenarios, learning a perceptron rule with respectively a spherical and a binary weight prior. In the rst case we rederive the results for the optimal Hebb-type online algorithm for spherical input distribution.
The Effect of Correlated Input Data on the Dynamics of Learning

The convergence properties of the gradient descent algorithm in the case of the linear perceptron may be obtained from the response function. We derive a general expression for the response function and apply it to the case of data with simple input correlations. It is found that correlations severely may slow down learning. This explains the success of PCA as a method for reducing training time. Motivated by this finding we furthermore propose to transform the input data by removing the mean across input variables as well as examples to decrease correlations. Numerical findings for a medical classification problem are in fine agreement with the theoretical results.
A Mean Field Algorithm for Bayes Learning in Large Feed-forward Neural Networks

We present an algorithm which is expected to realise Bayes optimal predictions in large feed-forward networks. It is based on mean field methods developed within statistical mechanics of disordered systems. We give a derivation for the single layer perceptron and show that the algorithm also provides a leave-one-out cross-validation test of the predictions. Simulations show excellent agreement with theoretical results of statistical mechanics.
ARACNE: An Algorithm for the Reconstruction of Gene Regulatory Networks in a Mammalian Cellular Context

BackgroundElucidating gene regulatory networks is crucial for understanding normal cell physiology and complex pathologic phenotypes. Existing computational methods for the genome-wide "reverse engineering" of such networks have been successful only for lower eukaryotes with simple genomes. Here we present ARACNE, a novel algorithm, using microarray expression profiles, specifically designed to scale up to the complexity of regulatory networks in mammalian cells, yet general enough to address a wider range of network deconvolution problems. This method uses an information theoretic approach to eliminate the majority of indirect interactions inferred by co-expression methods.ResultsWe prove that ARACNE reconstructs the network exactly (asymptotically) if the effect of loops in the network topology is negligible, and we show that the algorithm works well in practice, even in the presence of numerous loops and complex topologies. We assess ARACNE's ability to reconstruct transcriptional regulatory networks using both a realistic synthetic dataset and a microarray dataset from human B cells. On synthetic datasets ARACNE achieves very low error rates and outperforms established methods, such as Relevance Networks and Bayesian Networks. Application to the deconvolution of genetic networks in human B cells demonstrates ARACNE's ability to infer validated transcriptional targets of the cMYC proto-oncogene. We also study the effects of misestimation of mutual information on network reconstruction, and show that algorithms based on mutual information ranking are more resilient to estimation errors.ConclusionARACNE shows promise in identifying direct transcriptional interactions in mammalian cellular networks, a problem that has challenged existing reverse engineering algorithms. This approach should enhance our ability to use microarray data to elucidate functional mechanisms that underlie cellular processes and to identify molecular targets of pharmacological compounds in mammalian cellular networks.
Coincidences And Entropies of Random Variables with Very Large Alphabets

We examine the recently introduced NSB estimator of entropies of severely undersampled discrete variables and devise a procedure for calculating the involved integrals. We discover that the output of the estimator has a well defined limit for large cardinalities of the variables being studied. Thus one can estimate entropies with no a priori assumptions about these cardinalities, and a closed form solution for such estimates is given.
Fluctuation-Dissipation Theorem and Models of Learning

Advances in statistical learning theory have resulted in a multitude of different designs of learning machines. But which ones are implemented by brains and other biological information processors? We analyze how various abstract Bayesian learners perform on different data and argue that it is difficult to determine which learningtheoretic computation is performed by a particular organism using just its performance in learning a stationary target (learning curve). Based on the fluctuationdissipation relation in statistical physics, we then discuss a different experimental setup that might be able to solve the problem.
Entropy and information in neural spike trains: progress on the sampling problem.

The major problem in information theoretic analysis of neural responses and other biological data is the reliable estimation of entropy-like quantities from small samples. We apply a recently introduced Bayesian entropy estimator to synthetic data inspired by experiments, and to real experimental spike trains. The estimator performs admirably even very deep in the undersampled regime, where other techniques fail. This opens new possibilities for the information theoretic analysis of experiments, and may be of general interest as an example of learning from limited data.
Conditional Network Analysis Identifies Candidate Regulator Genes in Human B Cells

Cellular phenotypes are determined by the dynamical activity of networks of co-regulated genes. Elucidating such networks is crucial for the understanding of normal cell physiology as well as for the dissection of complex pathologic phenotypes. Existing methods for such "reverse engineering" of genetic networks from microarray expression data have been successful only in prokaryotes (E. coli) and lower eukaryotes (S. cerevisiae) with relatively simple genomes. Additionally, they have mostly attempted to reconstruct average properties about the network connectivity without capturing the highly conditional nature of the interactions. In this paper we extend the ARACNE algorithm, which we recently introduced and successfully applied to the reconstruction of whole-genome transcriptional networks from mammalian cells, precisely to link the existence of specific network structures to the expression or lack thereof of specific regulator genes. This is accomplished by analyzing thousands of alternative network topologies generated by constraining the data set on the presence or absence of putative regulator genes. By considering interactions that are consistently supported across several such constraints, we identify many transcriptional interactions that would not have been detectable by the original method. By selecting genes that produce statistically significant changes in network topology, we identify novel candidate regulator genes. Further analysis shows that transcription factors, kinases, phosphatases, and other gene families known to effect biochemical interactions, are significantly overrepresented among the set of candidate regulator genes identified in silico, indirectly supporting the validity of the approach.
Reverse Engineering of the Yeast Transcriptional Network Using the ARACNE algorithm

Cellular phenotypes are determined by dynamical activity of networks of co-regulated genes. Elucidating such networks is crucial for the understanding of normal cell physiology as well as for the dissection of complex pathologic phenotypes. Recently we have shown that ARACNE, a novel information-theoretic algorithm for reverse engineering of transcriptional networks using microarray data, holds significant promise for the genome-wide analysis of mammalian networks, which had never been performed in silico. In this paper we present the application of ARACNE to reverse engineering of transcriptional networks in the yeast Saccharomyces cerevisiae. This provides another platform for further comparisons of the new method to a variety of established ones, which have been extensively used to analyze this important model organism. Moreover, it provides an additional genome-wide interactome for the yeast using a method that is shown to produce very few false positive interactions, both in vivo and in silico. Additionally, we investigate the global topological properties of the reconstructed networks and determine that the scale-free structure suggested by existing models should be taken cautiously. Finally, analysis of the ARN1 sub-network, which has also been investigated using Bayesian Networks, shows that ARACNE is able to distinguish the key regulatory elements of this pathway from within a large cluster of co-regulated genes.
Potential and field singularity at a surface point charge

The behavior of the magnetic potential near a point charge (fluxon) located at a curved regular boundary surface is shown to be essentially different from that of a volume point charge. In addition to the usual inverse distance singularity, two singular terms are generally present. The first of them, a logarithmic one, is axially symmetric with respect to the boundary normal at the charge location, and proportional to the sum of the two principal curvatures of the boundary surface at this point, that is, to the local mean curvature. The second term is asymmetric and proportional to the difference of the two principal curvatures in question; it is also bounded at the charge location. Both terms vanish, apparently, if the charge is at a planar point of the boundary, and only in this case. The field in the charge vicinity behaves accordingly, featuring generally two singular terms proportional to the inverse distance, in addition to the main inverse distance squared singularity. This result is significant, in particular, for studying the interaction of magnetic vortices in type II superconductors.