nils-herrmann/clean_scirepeval huggingface dataset:
Occam factors and model-independent Bayesian learning of continuous distributions

Learning of a smooth but nonparametric probability density can be regularized using methods of quantum field theory. We implement a field theoretic prior numerically, test its efficacy, and show that the data and the phase space factors arising from the integration over the model space determine the free parameter of the theory ("smoothness scale") self-consistently. This persists even for distributions that are atypical in the prior and is a step towards a model independent theory for learning continuous distributions. Finally, we point out that a wrong parametrization of a model family may sometimes be advantageous for small data sets.
Inference of entropies of discrete random variables with unknown cardinalities

We examine the recently introduced NSB estimator of entropi es of severely undersampled discrete variables and devise a proc edure for calculating the involved integrals. We discover that the outpu t of the estimator has a well defined limit for large cardinalities of the var iables being studied. Thus one can estimate entropies with no a priori ass umptions about these cardinalities, and a closed form solution for su ch estimates is given.
On the interaction of point charges in an arbitrary domain

We develop a systematic approach to calculating the electrostatic force between point charges in an arbitrary domain with arbitrary boundary conditions. When the boundary is present, the simple expression for the force acting on a charge as “the charge times the field it is placed in” becomes ill-defined. However, this rule can be salvaged if the field in question is redefined to only include all terms that do not diverge at the charge position, in particular, those due to the charge itself. The proof requires handling the self-action energy divergence for point charges, which is accomplished by means of a geometrical regularization.
Learning Continuous Distributions: Simulations With Field Theoretic Priors

Learning of a smooth but nonparametric probability density can be regularized using methods of Quantum Field Theory. We implement a field theoretic prior numerically, test its efficacy, and show that the free parameter of the theory ('smoothness scale') can be determined self consistently by the data; this forms an infinite dimensional generalization of the MDL principle. Finally, we study the implications of one's choice of the prior and the parameterization and conclude that the smoothness scale determination makes density estimation very weakly sensitive to the choice of the prior, and that even wrong choices can be advantageous for small data sets.
Information theory and learning: a physical approach

We try to establish a unified information theoretic approach to learning and to explore some of its applications. First, we define {\em predictive information} as the mutual information between the past and the future of a time series, discuss its behavior as a function of the length of the series, and explain how other quantities of interest studied previously in learning theory - as well as in dynamical systems and statistical mechanics - emerge from this universally definable concept. We then prove that predictive information provides the {\em unique measure for the complexity} of dynamics underlying the time series and show that there are classes of models characterized by {\em power-law growth of the predictive information} that are qualitatively more complex than any of the systems that have been investigated before. Further, we investigate numerically the learning of a nonparametric probability density, which is an example of a problem with power-law complexity, and show that the proper Bayesian formulation of this problem provides for the `Occam' factors that punish overly complex models and thus allow one {\em to learn not only a solution within a specific model class, but also the class itself} using the data only and with very few a priori assumptions. We study a possible {\em information theoretic method} that regularizes the learning of an undersampled discrete variable, and show that learning in such a setup goes through stages of very different complexities. Finally, we discuss how all of these ideas may be useful in various problems in physics, statistics, and, most importantly, biology.
Explicit Green's function of a boundary value problem for a sphere and trapped flux analysis in Gravity Probe B experiment

Magnetic flux trapped on the surface of superconducting rotors of the Gravity Probe B (GP-B) experiment produces some signal in the superconducting quantum interference device readout. For the needs of GP-B error analysis and simulation of data reduction, this signal is calculated and analyzed in this article. We first solve a magnetostatic problem for a point source on the surface of a sphere, finding the closed form elementary expression for the corresponding Green’s function. Second, we calculate the flux through the pick-up loop as a function of the source position. Next, the time dependence of a source position, caused by rotor motion according to a symmetric top model, and thus the time signature of its flux are determined, and the spectrum of the trapped flux signal is analyzed. Finally, a multipurpose program of trapped flux signal generation based on the above results is described, various examples of the signal obtained by means of this program are given, and their features are discussed. Signals of up to 100 fluxons, i.e., 100 pairs of positive and negative point sources, are examined.
On the influence of the gravitating vacuum on the dynamics of homogeneous isotropic models in gauge theories of gravity

The analysis of solutions of gravitational equations for homogeneous isotropic models in the presence of a gravitating vacuum (cosmological constant ) is given in the frame of various gauge theories of gravity on the basis of qualitative theory of dynamic systems. The equation of state of matter is given in the form of a linear dependence of the pressure p on the energy density with restriction . It is shown that regularization of metric derivatives takes place in the case of sufficiently high values of , and some are obtained which are regular in the metrics solutions for superdense gravitating systems.
Information geometry for turbo decoding

Turbo codes are known as a class of error-correcting codes which have high error-correcting performance with efficient decoding algorithm. Characteristics of the iterative decoding algorithm have been studied in detail through a variety of numerical experiments, but theoretical results are still insufficient. In this paper, this issue is addressed from the information geometrical viewpoint. As a result, a mathematical framework for analyzing turbo codes is obtained, and some of the fundamental properties of turbo decoding are elucidated based on this framework. Recently, it has been pointed out that the turbo decoding algorithm is related to the decoding algorithm of low-density parity check codes, the computation method of Bethe approximation in statistical physics, and the belief propagation algorithm of Bayesian networks. The mathematical framework given in the present paper can also be used to analyze these wide classes of iterative computation methods, and hence represent a new analysis tool.  2005; .
Information geometry of turbo and LDPC codes

Turbo and LDPC (low-density parity check) codes are simple and new types of error correction codes which give a powerful and practical performance of error correction. Although experimental results show their efficacy, further theoretical analysis is necessary, which is not straightforward. We have built a unified framework of turbo and LDPC codes based on information geometry. The framework helps our intuitive understanding of the codes and opens a new prospect of further analysis. We have revealed some properties of these codes in the proposed framework. This paper summarizes the results.
Information geometry for turbo decoding

Turbo codes are known as a class of error-correcting codes which have high error-correcting performance with efficient decoding algorithm. Characteristics of the iterative decoding algorithm have been studied in detail through a variety of numerical experiments, but theoretical results are still insufficient. In this paper, this issue is addressed from the information geometrical viewpoint. As a result, a mathematical framework for analyzing turbo codes is obtained, and some of the fundamental properties of turbo decoding are elucidated based on this framework. Recently, it has been pointed out that the turbo decoding algorithm is related to the decoding algorithm of low-density parity check codes, the computation method of Bethe approximation in statistical physics, and the belief propagation algorithm of Bayesian networks. The mathematical framework given in the present paper can also be used to analyze these wide classes of iterative computation methods, and hence represent a new analysis tool.  2005; . DOI 10.1002/scj.10359
Stochastic Reasoning, Free Energy, and Information Geometry

Belief propagation (BP) is a universal method of stochastic reasoning. It gives exact inference for stochastic models with tree interactions and works surprisingly well even if the models have loopy interactions. Its performance has been analyzed separately in many fields, such as AI, statistical physics, information theory, and information geometry. This article gives a unified framework for understanding BP and related methods and summarizes the results obtained in many fields. In particular, BP and its variants, including tree reparameterization and concave-convex procedure, are reformulated with information-geometrical terms, and their relations to the free energy function are elucidated from an information-geometrical viewpoint. We then propose a family of new algorithms. The stabilities of the algorithms are analyzed, and methods to accelerate them are investigated.
Information geometry of turbo and low-density parity-check codes

Since the proposal of turbo codes in 1993, many studies have appeared on this simple and new type of codes which give a powerful and practical performance of error correction. Although experimental results strongly support the efficacy of turbo codes, further theoretical analysis is necessary, which is not straightforward. It is pointed out that the iterative decoding algorithm of turbo codes shares essentially similar ideas with low-density parity-check (LDPC) codes, with Pearl's belief propagation algorithm applied to a cyclic belief diagram, and with the Bethe approximation in statistical physics. Therefore, the analysis of the turbo decoding algorithm will reveal the mystery of those similar iterative methods. In this paper, we recapture and extend the geometrical framework initiated by Richardson to the information geometrical framework of dual affine connections, focusing on both of the turbo and LDPC decoding algorithms. The framework helps our intuitive understanding of the algorithms and opens a new prospect of further analysis. We reveal some properties of these codes in the proposed framework, including the stability and error analysis. Based on the error analysis, we finally propose a correction term for improving the approximation.
Information Geometry of α-Projection in Mean Field Approximation

Information geometry is applied to mean field approximation for elucidating its properties in the spin glass model or the Boltzmann machine. The α-divergence is used for approximation, where α-geodesic projection plays an important role. The naive mean field approximation and TAP approximation are studied from the point of view of information geometry, which treats the intrinsic geometric structures of a family of probability distributions. The bifurcation of the α-projection is studied, at which the uniqueness of the α-approximation is broken.
Combined approach of array processing and independent component analysis for blind separation of acoustic signals

Two array signal processing techniques are combined with independent component analysis (ICA) to enhance the performance of blind separation of acoustic signals in a reflective environment. The first technique is the subspace method which reduces the effect of room reflection when the system is used in a room. Room reflection is one of the biggest problems in blind source separation (BSS) in acoustic environments. The second technique is a method of solving permutation. For employing the subspace method, ICA must be used in the frequency domain, and precise permutation is necessary for all frequencies. In this method, a physical property of the mixing matrix, i.e., the coherency in adjacent frequencies, is utilized to solve the permutation. The experiments in a meeting room showed that the subspace method improved the rate of automatic speech recognition from 50% to 68% and that the method of solving permutation achieves performance that closely approaches that of the correct permutation, differing by only 4% in recognition rate.
A METHOD OF ICA IN TIME-FREQUENCY DOMAIN

AMETHODOFICAINTIME-FREQUENCYDOMAINShiroIkedaPRESTO,JSTHirosawa2-1,Wako,351-0198,JapanShiro.Ikeda@brain.riken.go.jpNoboruMurataRIKENBSINob oru.Murata@brain.rikABSTRACTWeprop oseametho dofICAforseparatingconvolu-tivemixturesofacousticsignals.Thesignalsrecordedinarealenvironmentarenotinstantaneousbutconvolutivemixtures,b ecauseofthedelayandreections.Inordertoseparatethesesignals,itisef-fectivetotransformthesignalsintime-frequencydo-main.Thedicultp ointintheseapproachesistheambiguityofthep ermutationandamplitudewhichisunavoidableinoriginalICAproblem.Sincetheba-sicICAapproachescannotsolvetheseambiguity,wneedanotherapproachtosolvethem.Wemployedtheenvelop esofsignalstosolveit,andhade-velop edsomealgorithms.Inthisarticle,weshotheoutlineofourorignalmetho d,andsomeextensionsit.Theyare,theon-lineversionandauditorysceneanalysisproblem.1.INTRODUCTIONOneofthego o dapplicationsICAisseparatingacousticsignalsrecordedinarealenvironment.Thisproblemiswellknownasthename,\co cktailpartyproblem".Whatisdicultfortheseproblemsthatthesignalsincludedelaysandreections(Fig.1).
A METHOD OF BLIND SEPARATION ON TEMPORAL STRUCTURE OF SIGNALS

In this article, we propose an Blind Source Separation algorithm for convolutive mixture of signals. We propose a method of separating signals in the time-frequency domain. We apply the decorrelation method proposed by Molgedey and Schuster on spectrogram and reconstruct separated signals focusing on the temporal structure of the signals. We show some results of experiments with both artificially controlled data and speech data recorded in the real environment.
Information Geometrical Framework for Analyzing Belief Propagation Decoder

The mystery of belief propagation (BP) decoder, especially of the turbo decoding, is studied from information geometrical viewpoint. The loopy belief network (BN) of turbo codes makes it difficult to obtain the true "belief" by BP, and the characteristics of the algorithm and its equilibrium are not clearly understood. Our study gives an intuitive understanding of the mechanism, and a new framework for the analysis. Based on the framework, we reveal basic properties of the turbo decoding.
Belief propagation and turbo code: Information geometrical view

In this article, we describe the information geometrical understanding of the belief propagation decoder, especially of the turbo decoding. The turbo decoding was proposed by Berrou et al. early in 90’s, and many studies have been appeared on this practical and powerful error correcting code. Even though many experimental results support the potential of the turbo decoding, there is not sufficient theoretical analysis for the decoding method. We investigate the problem from information geometrical viewpoint. From the new viewpoint, we establish a new framework for analyzing the turbo code, and reveal basic properties.
FACTOR ANALYSIS PREPROCESSING FOR ICA

One of the reasons ICA (Independent Component Analysis) became so popular is that ICA is a promising tools for a lot of applications. One of the attractive applications is the biological data analysis. There are a lot of works on neurobiological data analysis such as EEG (Electroencephalography), fMRI (functional Magnetic Resonance Imaging), and MEG (Magnetoencephalography), and they show interesting results. However, there still remain some problems to be solved. Many neurobiological data includes a large amount of noises, and also the number of independent components is unknown. These problems make it difficult to obtain good results by ICA algorithms. We discuss an approach to separate the data which contain additive noise without knowing the number of independent components. Our approach uses factor analysis as the preprocessing of the ICA algorithm, instead of PCA (Principal Component Analysis), which is the major preprocessing in many ICA algorithms. In the new preprocessing, the number of the sources and the amount of sensor noise are estimated. After the preprocessing, an ICA algorithm is used to estimate the separation matrix and mixing system. Through the experiments with MEG data and fMRI data, we show this approach is effective.
Information-Geometrical Significance of Sparsity in Gallager Codes

We report a result of perturbation analysis on decoding error of the belief propagation decoder for Gallager codes. The analysis is based on information geometry, and it shows that the principal term of decoding error at equilibrium comes from the m-embedding curvature of the log-linear submanifold spanned by the estimated pseudoposteriors, one for the full marginal, and K for partial posteriors, each of which takes a single check into account, where K is the number of checks in the Gallager code. It is then shown that the principal error term vanishes when the parity-check matrix of the code is so sparse that there are no two columns with overlap greater than 1.
ICA for noisy neurobiological data

ICA (independent component analysis) is a technique for analyzing multi-variant data. Lots of results are reported in the field of neurobiological data analysis such as EEG (electroencephalography), MRI (magnetic resonance imaging), and MEG (magnetoencephalography) using ICA. But there still remain problems. In most of the neurobiological data, there is a large amount of noise, and the number of independent components is unknown which gives difficulties for many ICA algorithms. We discuss an approach to separate noise-contaminated data without knowing the number of independent components. The idea is to replace PCA (principal component analysis), which is used as the preprocessing of many ICA algorithms, with factor analysis. In the new preprocessing, the number of the sources and the amount of the noise are estimated. After the preprocessing, an ICA algorithm is used to estimate the separation matrix and mixing system. Through experiments with MEG data, we show this approach is effective.
EVALUATION AND REAL-TIME IMPLEMENTATION OF BLIND SOURCE SEPARATION SYSTEM USING TIME-DELAYED DECORRELATION

Blind source separation based on the time-delayed decor-relation, which had been extended to a convolved mixture problem, is evaluated and implemented in a DSP system. The evaluation is conducted using the data recorded in an anechoic chamber and those in a listening room with moderate reverberation. The crosstalks and the scores of automatic speech recognition are measured. The hardware consists of dual DSPs and a host PC. The results of the benchmark is shown.
Acceleration of the EM algorithm

The EM algorithm is used for many applications including Boltzmann machine, stochastic Perceptron and HMM. This algorithm gives an iterating procedure for calculating the MLE of stochastic models which have hidden random variables. It is simple, but the convergence is slow. We also have “Fisher’s scoring method”. Its convergence is faster, but the calculation is heavy. We show that by using the EM algorithm recursively, we can connect these two methods and accelerate the EM algorithm. Also Louis, Meng and Rubin showed they can accelerate the EM algorithm, but our algorithm is simpler. We show some numerical simulations with our algorithm.
Acceleration of the EM algorithm

The EM algorithm is used for many applications, including the Boltzmann machine, stochastic Perceptron, and HMM. This algorithm gives an iterating procedure for calculating the MLE of stochastic models which have hidden random variables. It is simple, but the convergence is slow. We also have the Fisher scoring method. Its convergence is faster, but the calculation load is heavy. We show that by using the EM algorithm recursively, we can connect these two methods and accelerate the EM algorithm. Also, Louis, Meng, and Rubin showed they can accelerate the EM algorithm, but our algorithm is simpler. We present some numerical simulations using our algorithm.
Combining Independent Component Analysis and Sound Stream Segregation

This paper reports the issues and results of AI Challenge: \Understanding Three Simultaneous Speeches". First, the issues of the Challenge are revisited. We emphasis the importance of information fusion of various attributes of speeches (sounds) in separating speeches from a mixture of sounds. This emphasis is supported by comparing two methods of speech separation; computational auditory scene analysis approach that employs the attributes of sound sources and sound transmitting channel, and blind source separation approach that dispenses with these attributes. Although these two approaches are usually considered as opposite with regards to whether sound attributes is used or not, we conclude that they di er in the ways of using sound attributes. Next, a new algorithm for information fusion is proposed. Sound attributes extracted by tracking harmonic structures and sound source directions as well as by independent component analysis are fused according to sound ontology. Finally, the error reduction rate of the 1best/10-best word recognition of each speaker performed on 200 mixtures of two women's and one man's utterances of an isolated word is reported.
An Approach to Blind Source Separation of Speech Signals

In this paper we introduce a new technique for blind source separation of speech signals. We focused on the temporal structure of signals which is not always the case in other major approaches. The idea is to apply the decorrelation method proposed by Molgedey and Schuster in time-frequency domain. We show some results of experiments with artificial data and speech data recorded in the real environment. Our algorithm needs considerably straightforward calculation and includes only a few parameters to be tuned.
Convergence of the Wake-Sleep Algorithm

The W-S (Wake-Sleep) algorithm is a simple learning rule for the models with hidden variables. It is shown that this algorithm can be applied to a factor analysis model which is a linear version of the Helmholtz machine. But even for a factor analysis model, the general convergence is not proved theoretically. In this article, we describe the geometrical understanding of the W-S algorithm in contrast with the EM (Expectation-Maximization) algorithm and the em algorithm. As the result, we prove the convergence of the W-S algorithm for the factor analysis model. We also show the condition for the convergence in general models.
A Method of Blind Separation Based on Temporal Structure of Signals

In this article, we propose an Blind Source Separation algorithm for convolutive mixture of signals. We propose a method of separating signals in the time-frequency domain. We apply the decorrelation method proposed by Molgedey and Schuster on spectrogram and reconstruct separated signals focusing on the temporal structure of the signals. We show some results of experiments with both artificially controlled data and speech data recorded in the real environment.