natural gradient learning for over and under complete bases in ica independent component analysis or blind source separation is a new technique of extracting independent signals from mixtures it is applicable even when the number of independent sources is unknown and is larger or smaller than the number of observed mixture signals this article extends the natural gradient learning algorithm to be applicable to these overcomplete and undercomplete cases here the observed signals are assumed to be whitened by preprocessing so that we use the natural riemannian gradient in stiefel manifolds

matrix momentum for practical natural gradient learning an on line learning rule based on the introduction of a matrix momentum term is presented aimed at alleviating the computational costs of standard natural gradient learning the new rule natural gradient matrix momentum is analysed in the case of two layer feed forward neural network learning via methods of statistical physics it appears to provide a practical algorithm that performs as well as standard natural gradient descent in both the transient and asymptotic regimes but with a hugely reduced complexity

sufficient dimension reduction via squared loss mutual information estimation the goal of sufficient dimension reduction in supervised learning is to find the low dimensional subspace of input features that contains all of the information about the output values that the input features possess in this letter we propose a novel sufficient dimension reduction method using a squared loss variant of mutual information as a dependency measure we apply a density ratio estimator for approximating squared loss mutual information that is formulated as a minimum contrast estimator on parametric or nonparametric models since cross validation is available for choosing an appropriate model our method does not require any prespecified structure on the underlying distributions we elucidate the asymptotic bias of our estimator on parametric models and the asymptotic convergence rate on nonparametric models the convergence analysis utilizes the uniform tail bound of a u process and the convergence rate is characterized by the bracketing entropy of the model we then develop a natural gradient algorithm on the grassmann manifold for sufficient subspace search the analytic formula of our estimator allows us to compute the gradient efficiently numerical experiments show that the proposed method compares favorably with existing dimension reduction approaches on artificial and benchmark data sets

information geometry on hierarchy of probability distributions an exponential family or mixture family of probability distributions has a natural hierarchical structure this paper gives an orthogonal decomposition of such a system based on information geometry a typical example is the decomposition of stochastic dependency among a number of random variables in general they have a complex structure of dependencies pairwise dependency is easily represented by correlation but it is more difficult to measure effects of pure triplewise or higher order interactions dependencies among these variables stochastic dependency is decomposed quantitatively into an orthogonal sum of pairwise triplewise and further higher order dependencies this gives a new invariant decomposition of joint entropy this problem is important for extracting intrinsic interactions in firing patterns of an ensemble of neurons and for estimating its functional connections the orthogonal decomposition is given in a wide class of hierarchical structures including both exponential and mixture families as an example we decompose the dependency in a higher order markov chain into a sum of those in various lower order markov chains

the geometry of turbo decoding dynamics the spectacular performance offered by turbo codes sparked intense interest in them a considerable amount of research has simplified formalized and extended the ideas inherent in the original turbo code construction nevertheless the nature of the relatively simple ad hoc turbo decoding algorithm has remained something of a mystery we present a geometric interpretation of the turbo decoding algorithm the geometric perspective clearly indicates the relationship between turbo decoding and maximum likelihood decoding analysis of the geometry leads to new results concerning existence of fixed points conditions for uniqueness conditions for stability and proximity to maximum likelihood decoding

complexity issues in natural gradient descent method for training multilayer perceptrons the natural gradient descent method is applied to train an n m 1 multilayer perceptron based on an efficient scheme to represent the fisher information matrix for an n m 1 stochastic multilayer perceptron a new algorithm is proposed to calculate the natural gradient without inverting the fisher information matrix explicitly when the input dimension n is much larger than the number of hidden neurons m the time complexity of computing the natural gradient is o n

equivariant adaptive selective transmission in this paper we consider the problem of selective transmission the dual of the blind source separation task in which a set of independent source signals are adaptively premixed prior to a nondispersive physical mixing process so that each source can be independently monitored in the far field following similar procedures for information theoretic blind source separation we derive a stochastic gradient algorithm for iteratively estimating the premixing matrix in the selective transmission problem and through a simple modification we obtain a second algorithm whose performance is equivariant with respect to the channel s mixing characteristics the local stability conditions for the algorithms about any selective transmission solution are shown to be the same as those for similar source separation algorithms practical implementation issues are discussed including the estimation of the combined system matrix and the reordering and scaling of the received signals within the algorithm mean square error based selective transmission algorithms are also derived for performance comparison purposes simulations indicate the useful behavior of the premixing algorithms for selective transmission

information geometry of mean field approximation i present a general theory of mean field approximation based on information geometry and applicable not only to boltzmann machines but also to wider classes of statistical models using perturbation expansion of the kullback divergence or plefka expansion in statistical physics a formulation of mean field approximation of general orders is derived it includes in a natural way the naive mean field approximation and is consistent with the thouless anderson palmer tap approach and the linear response theorem in statistical physics

fractional variations for dynamical systems hamilton and lagrange approaches fractional generalization of an exterior derivative for calculus of variations is defined the hamilton and lagrange approaches are considered fractional hamilton and euler lagrange equations are derived fractional equations are obtained by fractional variation of lagrangian and hamiltonian that have only integer derivatives

blind separation of a mixture of uniformly distributed source signals a novel approach a new efficient algorithm for blind separation of uniformly distributed sources is proposed the mixing matrix is assumed to be orthogonal by prewhitening the observed signals the learning rule adaptively estimates the mixing matrix by conceptually rotating a unit hypercube so that all output signal components are contained within or on the hypercube under some ideal constraints it has been theoretically shown that the algorithm is very similar to an ideal convergent algorithm which is much faster than the existing convergent algorithms the algorithm has been generalized to take care of the noisy signals by adaptively dilating the hypercube in conjunction with its rotation

cccp algorithms to minimize the bethe and kikuchi free energies convergent alternatives to belief propagation this article introduces a class of discrete iterative algorithms that are provably convergent alternatives to belief propagation bp and generalized belief propagation gbp our work builds on recent results by yedidia freeman and weiss 2000 who showed that the fixed points of bp and gbp algorithms correspond to extrema of the bethe and kikuchi free energies respectively we obtain two algorithms by applying cccp to the bethe and kikuchi free energies respectively cccp is a procedure introduced here for obtaining discrete iterative algorithms by decomposing a cost function into a concave and a convex part we implement our cccp algorithms on two and three dimensional spin glasses and compare their results to bp and gbp our simulations show that the cccp algorithms are stable and converge very quickly the speed of cccp is similar to that of bp and gbp unlike cccp bp will often not converge for these problems gbp usually but not always converges the results found by cccp applied to the bethe or kikuchi free energies are equivalent or slightly better than those found by bp or gbp respectively when bp and gbp converge note that for these and other problems bp and gbp give very accurate results see yedidia et al 2000 and failure to converge is their major error mode finally we point out that our algorithms have a large range of inference and learning applications

normal forms and linearization of resonant vector fields with multiple eigenvalues abstract we discuss the linearization and normal forms of resonant vector fields x x a x a x where a has one double or triple eigenvalue or a pair of double eigenvalues we present a simple way of identifying the resonant monomials that have to appear in its normal form and also effective conditions on the nonlinearity a x for the resonant vector field to be linearizable

belief propagation dykstra s algorithm and iterated information projections belief propagation is shown to be an instance of a hybrid between two projection algorithms in the convex programming literature dykstra s algorithm with cyclic bregman projections and an alternating bregman projections algorithm via this connection new results concerning the convergence and performance of belief propagation can be proven by exploiting the corresponding literature about the two projections algorithms it hybridizes in this regard it is identified that the lack of guaranteed convergence for belief propagation results from the asymmetry of its bregman divergence by proving that when the associated hybrid projection algorithm generalization is used with a symmetric bregman divergence it always converges additionally by characterizing factorizations that are close to acyclic in a manner independent of their girth a new collection of distributions for which belief propagation is guaranteed to perform well is identified using the new projection algorithm framework

a mean field optimal control formulation for global optimization this paper is concerned with variational optimal control constructions whose solution yields a sampling algorithm the particular form of the sampling algorithm considered here is a particle filter designed to numerically approximate the solution to the global optimization problem the theoretical significance of this study comes from its variational aspects specifically the control input represents the solution of a mean field type optimal control problem its parametric counterpart obtained when a parametric form of density is known a priori is shown to be equivalent to the natural gradient algorithm explicit formulae for the filter are derived when the objective function is quadratic and the density is gaussian the optimal control construction of the particle filter is a significant departure from the classical importance sampling resampling based approaches

robust independent component analysis by iterative maximization of the kurtosis contrast with algebraic optimal step size independent component analysis ica aims at decomposing an observed random vector into statistically independent variables deflation based implementations such as the popular one unit fastica algorithm and its variants extract the independent components one after another a novel method for deflationary ica referred to as robustica is put forward in this paper this simple technique consists of performing exact line search optimization of the kurtosis contrast function the step size leading to the global maximum of the contrast along the search direction is found among the roots of a fourth degree polynomial this polynomial rooting can be performed algebraically and thus at low cost at each iteration among other practical benefits robustica can avoid prewhitening and deals with real and complex valued mixtures of possibly noncircular sources alike the absence of prewhitening improves asymptotic performance the algorithm is robust to local extrema and shows a very high convergence speed in terms of the computational cost required to reach a given source extraction quality particularly for short data records these features are demonstrated by a comparative numerical analysis on synthetic data robustica s capabilities in processing real world data involving noncircular complex strongly super gaussian sources are illustrated by the biomedical problem of atrial activity aa extraction in atrial fibrillation af electrocardiograms ecgs where it outperforms an alternative ica based technique

an alternative perspective on adaptive independent component analysis algorithms this article develops an extended independent component analysis algorithm for mixtures of arbitrary subgaussian and supergaussian sources the gaussian mixture model of pearson is employed in deriving a closedform generic score function for strictly subgaussian sources this is combined with the score function for a unimodal supergaussian density to provide a computationally simple yet powerful algorithm for performing independent component analysis on arbitrary mixtures of nongaussian sources

recurrent sampling models for the helmholtz machine many recent analysis by synthesis density estimation models of cortical learning and processing have made the crucial simplifying assumption that units within a single layer are mutually independent given the states of units in the layer below or the layer above in this article we suggest using either a markov random field or an alternative stochastic sampling architecture to capture explicitly particular forms of dependence within each layer we develop the architectures in the context of real and binary helmholtz machines recurrent sampling can be used to capture correlations within layers in the generative or the recognition models and we also show how these can be combined

energy efficient distributed filtering with wireless sensor networks we consider a wireless sensor network wsn that monitors a physical field and communicates pertinent data to a distant fusion center fc we study the case of a binary valued hidden natural field observed in a significant amount of gaussian clutter which is relevant to applications like detection of plumes or oil slicks the considerable spatio temporal dependencies found in natural fields can be exploited to improve the reliability of the detection estimation of hidden phenomena while this problem has been previously treated using kernel regression techniques we formulate it as a task of delay free filtering on a process observed by the sensors we propose a distributed scalable implementation of the filter within the network this is achieved by i exploiting the localized spatio temporal dependencies to define a hidden markov model hmm in terms of an exponential family with o n parameters where n is the size of the wsn ii using a reduced state approximation of the propagated probability mass function and iii making a tractable approximation of model marginals by using iterated decoding algorithms like the gibbs sampler gs mean field decoding mfd iterated conditional modes icm and broadcast belief propagation bbp we compare the marginalization algorithms in terms of their information geometry performance complexity and communication load finally we analyze the energy efficiency of the proposed distributed filter relative to brute force data fusion it is demonstrated that when the fc is sufficiently far away from the sensor array distributed filtering is significantly more energy efficient and can increase the lifetime of the wsn by one to two orders of magnitude

statistical trajectory of an approximate em algorithm for probabilistic image processing we calculate analytically a statistical average of trajectories of an approximate expectation maximization em algorithm with generalized belief propagation gbp and a gaussian graphical model for the estimation of hyperparameters from observable data in probabilistic image processing a statistical average with respect to observed data corresponds to a configuration average for the random field ising model in spin glass theory in the present paper hyperparameters which correspond to interactions and external fields of spin systems are estimated by an approximate em algorithm a practical algorithm is described for gray level image restoration based on a gaussian graphical model and gbp the gbp approach corresponds to the cluster variation method in statistical mechanics our main result in the present paper is to obtain the statistical average of the trajectory in the approximate em algorithm by using loopy belief propagation and gbp with respect to degraded images generated from a probability density function with true values of hyperparameters the statistical average of the trajectory can be expressed in terms of recursion formulas derived from some analytical calculations

integration of stochastic models by minimizing α divergence when there are a number of stochastic models in the form of probability distributions one needs to integrate them mixtures of distributions are frequently used but exponential mixtures also provide a good means of integration this letter proposes a one parameter family of integration called integration which includes all of these well known integrations these are generalizations of various averages of numbers such as arithmetic geometric and harmonic averages there are psychophysical experiments that suggest that integrations are used in the brain the divergence between two distributions is defined which is a natural generalization of kullback leibler divergence and hellinger distance and it is proved that integration is optimal in the sense of minimizing divergence the theory is applied to generalize the mixture of experts and the product of experts to the mixture of experts the predictive distribution is also stated in the bayesian framework

nonextensive statistics entropic gravity and gravitational force in a non integer dimensional space abstract based on the connection between tsallis nonextensive statistics and fractional dimensional space in this work we have introduced with the aid of verlinde s formalism the newton constant in a fractal space as a function of the nonextensive constant with this result we have constructed a curve that shows the direct relation between tsallis nonextensive parameter and the dimension of this fractal space we have demonstrated precisely that there are ambiguities between the results due to verlinde s approach and the ones due to fractional calculus formalism we have shown precisely that these ambiguities appear only for spaces with dimensions different from three a possible solution for this ambiguity was proposed here

blind source separation semiparametric statistical approach the semiparametric statistical model is used to formulate the problem of blind source separation the method of estimating functions is applied to this problem it is shown that an estimator of the mixing matrix or its learning version can be described in terms of an estimating function the statistical efficiencies of these algorithms are studied the main results are as follows 1 the space consisting of all the estimating functions is derived 2 the space is decomposed into the orthogonal sum of the admissible part and a redundant ancillary part for any estimating function one can find a better or equally good estimator in the admissible part 3 the fisher efficient that is asymptotically best estimating functions are derived 4 the stability of learning algorithms is studied

on line versus off line learning from random examples general results i propose a general model of on line learning from random examples which when applied to a smooth realizable stochastic rule yields the same asymptotic generalization error rate as optimal batch algorithms the approach is based on an iterative gaussian approximation to the posterior gibbs distribution of rule parameters

statistical modelling of higher order correlations in pools of neural activity abstract simultaneous recordings from multiple neural units allow us to investigate the activity of very large neural ensembles to understand how large ensembles of neurons process sensory information it is necessary to develop suitable statistical models to describe the response variability of the recorded spike trains using the information geometry framework it is possible to estimate higher order correlations by assigning one interaction parameter to each degree of correlation leading to a 2 n 1 dimensional model for a population with n neurons however this model suffers greatly from a combinatorial explosion and the number of parameters to be estimated from the available sample size constitutes the main intractability reason of this approach to quantify the extent of higher than pairwise spike correlations in pools of multiunit activity we use an information geometric approach within the framework of the extended central limit theorem considering all possible contributions from higher order spike correlations the identification of a deformation parameter allows us to provide a statistical characterisation of the amount of higher order correlations in the case of a very large neural ensemble significantly reducing the number of parameters avoiding the sampling problem and inferring the underlying dynamical properties of the network within pools of multiunit neural activity

serial updating rule for blind separation derived from the method of scoring in the context of blind source separation the method of scoring based on the inverse of the fisher information matrix fim becomes the serial updating learning rule with an equivariant property this learning rule can be simplified to a low complexity algorithm by using the asymptotic form of the ftm around the equilibrium the simplified learning rule is still general enough to include some existing equivariant blind separation algorithms as its special cases

preconditioned stochastic gradient descent stochastic gradient descent sgd still is the workhorse for many practical problems however it converges slow and can be difficult to tune it is possible to precondition sgd to accelerate its convergence remarkably but many attempts in this direction either aim at solving specialized problems or result in significantly more complicated methods than sgd this paper proposes a new method to adaptively estimate a preconditioner such that the amplitudes of perturbations of preconditioned stochastic gradient match that of the perturbations of parameters to be optimized in a way comparable to newton method for deterministic optimization unlike the preconditioners based on secant equation fitting as done in deterministic quasi newton methods which assume positive definite hessian and approximate its inverse the new preconditioner works equally well for both convex and nonconvex optimizations with exact or noisy gradients when stochastic gradient is used it can naturally damp the gradient noise to stabilize sgd efficient preconditioner estimation methods are developed and with reasonable simplifications they are applicable to large scale problems experimental results demonstrate that equipped with the new preconditioner without any tuning effort preconditioned sgd can efficiently solve many challenging problems like the training of a deep neural network or a recurrent neural network requiring extremely long term memories

adaptive method of realizing natural gradient learning for multilayer perceptrons the natural gradient learning method is known to have ideal performances for on line training of multilayer perceptrons it avoids plateaus which give rise to slow convergence of the backpropagation method it is fisher efficient whereas the conventional method is not however for implementing the method it is necessary to calculate the fisher information matrix and its inverse which is practically very difficult this article proposes an adaptive method of directly obtaining the inverse of the fisher information matrix it generalizes the adaptive gauss newton algorithms and provides a solid theoretical justification of them simulations show that the proposed adaptive method works very well for realizing natural gradient learning

approximate riemannian conjugate gradient learning for fixed form variational bayes variational bayesian vb methods are typically only applied to models in the conjugate exponential family using the variational bayesian expectation maximisation vb em algorithm or one of its variants in this paper we present an efficient algorithm for applying vb to more general models the method is based on specifying the functional form of the approximation such as multivariate gaussian the parameters of the approximation are optimised using a conjugate gradient algorithm that utilises the riemannian geometry of the space of the approximations this leads to a very efficient algorithm for suitably structured approximations it is shown empirically that the proposed method is comparable or superior in efficiency to the vb em in a case where both are applicable we also apply the algorithm to learning a nonlinear state space model and a nonlinear factor analysis model for which the vb em is not applicable for these models the proposed algorithm outperforms alternative gradient based methods by a significant margin

a globally convergent mc algorithm with an adaptive learning rate this brief deals with the problem of minor component analysis mca artificial neural networks can be exploited to achieve the task of mca recent research works show that convergence of neural networks based mca algorithms can be guaranteed if the learning rates are less than certain thresholds however the computation of these thresholds needs information about the eigenvalues of the autocorrelation matrix of data set which is unavailable in online extraction of minor component from input data stream in this correspondence we introduce an adaptive learning rate into the ojan mca algorithm such that its convergence condition does not depend on any unobtainable information and can be easily satisfied in practical applications

natural evolution strategies this paper presents natural evolution strategies nes a recent family of black box optimization algorithms that use the natural gradient to update a parameterized search distribution in the direction of higher expected fitness we introduce a collection of techniques that address issues of convergence robustness sample complexity computational complexity and sensitivity to hyperparameters this paper explores a number of implementations of the nes family such as general purpose multi variate normal distributions and separable distributions tailored towards search in high dimensional spaces experimental results show best published performance on various standard benchmarks as well as competitive performance on others

newton stein method an optimization method for glms via stein s lemma we consider the problem of efficiently computing the maximum likelihood estimator in generalized linear models glms when the number of observations is much larger than the number of coefficients n p 1 in this regime optimization algorithms can immensely benefit from approximate second order information we propose an alternative way of constructing the curvature information by formulating it as an estimation problem and applying a stein type lemma which allows further improvements through subsampling and eigenvalue thresholding our algorithm enjoys fast convergence rates resembling that of second order methods with modest per iteration cost we provide its convergence analysis for the general case where the rows of the design matrix are samples from a sub gaussian distribution we show that the convergence has two phases a quadratic phase followed by a linear phase finally we empirically demonstrate that our algorithm achieves the highest performance compared to various optimization algorithms on several data sets

information geometry of boltzmann machines a boltzmann machine is a network of stochastic neurons the set of all the boltzmann machines with a fixed topology forms a geometric manifold of high dimension where modifiable synaptic weights of connections play the role of a coordinate system to specify networks a learning trajectory for example is a curve in this manifold it is important to study the geometry of the neural manifold rather than the behavior of a single network in order to know the capabilities and limitations of neural networks of a fixed topology using the new theory of information geometry a natural invariant riemannian metric and a dual pair of affine connections on the boltzmann neural network manifold are established the meaning of geometrical structures is elucidated from the stochastic and the statistical point of view this leads to a natural modification of the boltzmann machine learning rule

blind separation of uniformly distributed signals a general approach a general algorithm for blind separation of uniformly distributed signals is presented first maximum likelihood equations are obtained for dealing with this task it is difficult to obtain a closed form maximum likelihood solution for arbitrary mixing matrix the learning rules are obtained based on the geometric interpretation of the maximum likelihood estimator the algorithm under special constraint of orthogonal mixing matrix is the same as the o 1 t sup 2 convergent algorithm special noise correction mechanisms are incorporated in the algorithm and it has been found that the algorithm exhibits stable performance even in the presence of large amount of noise

learning dynamics on different timescales the special character of certain degrees of freedom in two layered neural networks is investigated for on line learning of realizable rules our analysis shows that the dynamics of these degrees of freedom can be put on a faster timescale than those remaining with the profit of speeding up the overall adaptation process this is shown for two groups of degrees of freedom second layer weights and bias weights for the former case our analysis provides a theoretical explanation of phenomenological findings the resulting learning algorithm is compared with natural gradient descent in order to check whether the proposed scaling can be naturally derived from that type of learning rule

a generalized contrast function and stability analysis for overdetermined blind separation of instantaneous mixtures in this letter the problem of blind separation of n independent sources from their m linear instantaneous mixtures is considered first a generalized contrast function is defined as a valuable extension of the existing classical and nonsymmetrical contrast functions it is applicable to the overdetermined blind separation m n with an unknown number of sources because not only independent components but also redundant ones are allowed in the outputs of a separation system second a natural gradient learning algorithm developed primarily for the complete case m n is shown to work as well with an n m or m m separating matrix for each optimizes a certain mutual information contrast function finally we present stability analysis for a newly proposed generalized orthogonal natural gradient algorithm which can perform the overdetermined blind separation when n is unknown obtaining an expectable result that its local stability conditions are slightly stricter than those of the conventional natural gradient algorithm using an invertible mixing matrix m n

a unifying information theoretic framework for independent component analysis abstract we show that different theories recently proposed for independent component analysis ica lead to the same iterative learning algorithm for blind separation of mixed independent sources we review those theories and suggest that information theory can be used to unify several lines of research pearlmutter and parra 1 and cardoso 2 showed that the infomax approach of bell and sejnowski 3 and the maximum likelihood estimation approach are equivalent we show that negentropy maximization also has equivalent properties and therefore all three approaches yield the same learning rule for a fixed nonlinearity girolami and fyfe 4 have shown that the nonlinear principal component analysis pca algorithm of karhunen and joutsensalo 5 and oja 6 can also be viewed from information theoretic principles since it minimizes the sum of squares of the fourth order marginal cumulants and therefore approximately minimizes the mutual information 7 lambert 8 has proposed different bussgang cost functions for multichannel blind deconvolution we show how the bussgang property relates to the infomax principle finally we discuss convergence and stability as well as future research issues in blind source separation

statistical dynamics of on line independent component analysis the learning dynamics of on line independent component analysis is analysed in the limit of large data dimension we study a simple hebbian learning algorithm that can be used to separate out a small number of non gaussian components from a high dimensional data set the de mixing matrix parameters are confined to a stiefel manifold of tall orthogonal matrices and we introduce a natural gradient variant of the algorithm which is appropriate to learning on this manifold for large input dimension the parameter trajectory of both algorithms passes through a sequence of unstable fixed points each described by a diffusion process in a polynomial potential choosing the learning rate too large increases the escape time from each of these fixed points effectively trapping the learning in a sub optimal state in order to avoid these trapping states a very low learning rate must be chosen during the learning transient resulting in learning time scales of o n2 or o n3 iterations where n is the data dimension escape from each sub optimal state results in a sequence of symmetry breaking events as the algorithm learns each source in turn this is in marked contrast to the learning dynamics displayed by related on line learning algorithms for multilayer neural networks and principal component analysis although the natural gradient variant of the algorithm has nice asymptotic convergence properties it has an equivalent transient dynamics to the standard hebbian algorithm

adaptive online learning algorithms for blind separation maximum entropy and minimum mutual information there are two major approaches for blind separation maximum entropy me and minimum mutual information mmi both can be implemented by the stochastic gradient descent method for obtaining the demixing matrix the mi is the contrast function for blind separation the entropy is not to justify the me the relation between me and mmi is first elucidated by calculating the first derivative of the entropy and proving that the mean subtraction is necessary in applying the me and at the solution points determined by the mi the me will not update the demixing matrix in the directions of increasing the cross talking second the natural gradient instead of the ordinary gradient is introduced to obtain efficient algorithms because the parameter space is a riemannian space consisting of matrices the mutual information is calculated by applying the gram charlier expansion to approximate probability density functions of the outputs finally we propose an efficient learning algorithm that incorporates with an adaptive method of estimating the unknown cumulants it is shown by computer simulation that the convergence of the stochastic descent algorithms is improved by using the natural gradient and the adaptively estimated cumulants

loop calculus for nonbinary alphabets using concepts from information geometry the bethe approximation is a well known approximation of the partition function used in statistical physics recently an equality relating the partition function and its bethe approximation was obtained for graphical models with binary variables by chertkov and chernyak in this equality the multiplicative error in the bethe approximation is represented as a weighted sum over all generalized loops in the graphical model in this paper the equality is generalized to graphical models with nonbinary alphabet using concepts from information geometry

the concave convex procedure the concave convex procedure cccp is a way to construct discrete time iterative dynamical systems that are guaranteed to decrease global optimization and energy functions monotonically this procedure can be applied to almost any optimization problem and many existing algorithms can be interpreted in terms of it in particular we prove that all expectation maximization algorithms and classes of legendre minimization and variational bounding algorithms can be reexpressed in terms of cccp we show that many existing neural network and mean field theory algorithms are also examples of cccp the generalized iterative scaling algorithm and sinkhorn s algorithm can also be expressed as cccp by changing variables cccp can be used both as a new way to understand and prove the convergence of existing optimization algorithms and as a procedure for generating new algorithms

possible generalization of boltzmann gibbs statistics with the use of a quantity normally scaled in multifractals a generalized form is postulated for entropy namelys q k 1 i 1 w p i q q 1 whereq characterizes the generalization andp i are the probabilities associated withw microscopic configurations w the main properties associated with this entropy are established particularly those corresponding to the microcanonical and canonical ensembles the boltzmann gibbs statistics is recovered as theq 1 limit

dynamics of learning in multilayer perceptrons near singularities the dynamical behavior of learning is known to be very slow for the multilayer perceptron being often trapped in the plateau it has been recently understood that this is due to the singularity in the parameter space of perceptrons in which trajectories of learning are drawn the space is riemannian from the point of view of information geometry and contains singular regions where the riemannian metric or the fisher information matrix degenerates this paper analyzes the dynamics of learning in a neighborhood of the singular regions when the true teacher machine lies at the singularity we give explicit asymptotic analytical solutions trajectories both for the standard gradient sgd and natural gradient ngd methods it is clearly shown in the case of the sgd method that the plateau phenomenon appears in a neighborhood of the critical regions where the dynamical behavior is extremely slow the analysis of the ngd method is much more difficult because the inverse of the fisher information matrix diverges we conquer the difficulty by introducing the blow down technique used in algebraic geometry the ngd method works efficiently and the state converges directly to the true parameters very quickly while it staggers in the case of the sgd method the analytical results are compared with computer simulations showing good agreement the effects of singularities on learning are thus qualitatively clarified for both standard and ngd methods

estimating functions of independent component analysis for temporally correlated signals this article studies a general theory of estimating functions of independent component analysis when the independent source signals are temporarily correlated estimating functions are used for deriving both batch and on line learning algorithms and they are applicable to blind cases where spatial and temporal probability structures of the sources are unknown most algorithms proposed so far can be analyzed in the framework of estimating functions an admissible class of estimating functions is derived and related efficient on line learning algorithms are introduced we analyze dynamical stability and statistical efficiency of these algorithms different from the independently and identically distributed case the algorithms work even when only the second order moments are used the method of simultaneous diagonalization of cross covariance matrices is also studied from the point of view of estimating functions

cauchy noise loss for stochastic optimization of random matrix models via free deterministic equivalents abstract for random matrix models the parameter estimation based on the traditional likelihood functions is not straightforward in particular when we have only one sample matrix we introduce a new parameter optimization method for random matrix models which works even in such a case the method is based on the spectral distribution instead of the traditional likelihood in the method the cauchy noise has an essential role because the free deterministic equivalent which is a tool in free probability theory allows us to approximate the spectral distribution perturbed by cauchy noises by a smooth and accessible density function moreover we study an asymptotic property of determination gap which has a similar role as generalization gap besides we propose a new dimensionality recovery method for the signal plus noise model and experimentally demonstrate that it recovers the rank of the signal part even if the true rank is not small it is a simultaneous rank selection and parameter estimation procedure

analysis of natural gradient descent for multilayer neural networks natural gradient descent ngd is an on line algorithm for redefining the steepest descent direction an analysis of ngd for training a multilayer neural network is presented using statistical mechanics the performance can be significantly improved using ngd algorithm and can be used for both the transient and asymptotic stages of learning

superefficiency in blind source separation blind source separation is the problem of extracting independent signals from their mixtures without knowing the mixing coefficients nor the probability distributions of source signals and may be applied to eeg and meg imaging of the brain it is already known that certain algorithms work well for the extraction of independent components the present paper is concerned with superefficiency of these based on the statistical and dynamical analysis in a statistical estimation using t examples the covariance of any two extracted independent signals converges to 0 of the order of 1 t on line dynamics shows that the covariance is of the order of spl eta when the learning rate spl eta is fixed to a small constant in contrast with the above general properties a surprising superefficiency holds in blind source separation under certain conditions where superefficiency implies that covariance decreases in the order of 1 t sup 2 or of spl eta sup 2 the paper uses the natural gradient learning algorithm and method of estimating functions to obtain superefficient procedures for both batch estimation and on line learning a standardized estimating function is introduced to this end superefficiency does not imply that the error variances of the extracted signals decrease in the order of 1 t sup 2 or spl eta sup 2 but implies that their covariances and independencies do

natural gradient works efficiently in learning when a parameter space has a certain underlying structure the ordinary gradient of a function does not represent its steepest direction but the natural gradient does information geometry is used for calculating the natural gradients in the parameter space of perceptrons the space of matrices for blind source separation and the space of linear dynamical systems for blind source deconvolution the dynamical behavior of natural gradient online learning is analyzed and is proved to be fisher efficient implying that it has asymptotically the same performance as the optimal batch estimation of parameters this suggests that the plateau phenomenon which appears in the backpropagation learning algorithm of multilayer perceptrons might disappear or might not be so serious when the natural gradient is used an adaptive method of updating the learning rate is proposed and analyzed

higher order boltzmann machines and entropy bounds we examine some aspects of the interface area between mathematical statistics and statistical physics relevant to the study of boltzmann machines the boltzmann machine learning algorithm is based on a variational principle gibbs lemma for relative entropy this fact suggests the possibility of a scheme of successive approximations here we consider successive approximations parametrized by the order of many body interactions among individual units we prove bounds on the gain in relative entropy in the crucial step of adding and estimating by hebb s rule a new parameter we address the problem of providing on the basis of local observations upper and lower bounds on the entropy while upper bounds are easily obtained by subadditivity lower bounds involve localization of hirschman bounds on a dual quantum system

smooth linearization near a fixed point in this paper we extend a theorem of sternberg and bi leckii we study a vector field or a diffeomorphism in the vicinity of a hyperbolic fixed point we assume that the eigenvalues of the linear part a at the fixed point satisfy qth order algebraic inequalities where q 2 2 then there is ck linearization in the vicinity of the fixed point where k is the q smoothness of a we give an explicit and simple algorithm for com puting k an illustrative example from celestial mechanics is included which shows that our main theorem is an improvement over previously known theories we also show that if the vector field or diffeomorphism depends smoothly on a parameter then the linearizing conjugation de pends smoothly on the parameter

pruning using parameter and neuronal metrics in this article we introduce a measure of optimality for architecture selection algorithms for neural networks the distance from the original network to the new network in a metric defined by the probability distributions of all possible networks we derive two pruning algorithms one based on a metric in parameter space and the other based on a metric in neuron space which are closely related to well known architecture selection algorithms such as gobs our framework extends the theoretically range of validity of gobs and therefore can explain results observed in previous experiments in addition we give some computational improvements for these algorithms

a stochastic quasi newton method for large scale optimization the question of how to incorporate curvature information into stochastic approximation methods is challenging the direct application of classical quasi newton updating techniques for deterministic optimization leads to noisy curvature estimates that have harmful effects on the robustness of the iteration in this paper we propose a stochastic quasi newton method that is efficient robust and scalable it employs the classical bfgs update formula in its limited memory form and is based on the observation that it is beneficial to collect curvature information pointwise and at spaced intervals one way to do this is through subsampled hessian vector products this technique differs from the classical approach that would compute differences of gradients at every iteration and where controlling the quality of the curvature estimates can be difficult we present numerical results on problems arising in machine learning that suggest that the proposed method shows much promise

turbo decoding as iterative constrained maximum likelihood sequence detection the turbo decoder was not originally introduced as a solution to an optimization problem which has impeded attempts to explain its excellent performance here it is shown that the turbo decoder is an iterative method seeking a solution to an intuitively pleasing constrained optimization problem in particular the turbo decoder seeks the maximum likelihood sequence mls under the false assumption that the input to the encoders are chosen independently of each other in the parallel case or that the output of the outer encoder is chosen independently of the input to the inner encoder in the serial case to control the error introduced by the false assumption the optimizations are performed subject to a constraint on the probability that the independent messages happen to coincide when the constraining probability equals one the global maximum of the constrained optimization problem is the maximum likelihood sequence detection mlsd allowing for a theoretical connection between turbo decoding and mlsd it is then shown that the turbo decoder is a nonlinear block gauss seidel iteration that aims to solve the optimization problem by zeroing the gradient of the lagrangian with a lagrange multiplier of 1 some conditions for the convergence for the turbo decoder are then given by adapting the existing literature for gauss seidel iterations

information geometry of turbo and low density parity check codes since the proposal of turbo codes in 1993 many studies have appeared on this simple and new type of codes which give a powerful and practical performance of error correction although experimental results strongly support the efficacy of turbo codes further theoretical analysis is necessary which is not straightforward it is pointed out that the iterative decoding algorithm of turbo codes shares essentially similar ideas with low density parity check ldpc codes with pearl s belief propagation algorithm applied to a cyclic belief diagram and with the bethe approximation in statistical physics therefore the analysis of the turbo decoding algorithm will reveal the mystery of those similar iterative methods in this paper we recapture and extend the geometrical framework initiated by richardson to the information geometrical framework of dual affine connections focusing on both of the turbo and ldpc decoding algorithms the framework helps our intuitive understanding of the algorithms and opens a new prospect of further analysis we reveal some properties of these codes in the proposed framework including the stability and error analysis based on the error analysis we finally propose a correction term for improving the approximation

information geometric algorithm for estimating switching probabilities in space varying hmm this paper proposes an iterative natural gradient algorithm to perform the optimization of switching probabilities in a space varying hidden markov model in the context of human activity recognition in long range surveillance the proposed method is a version of the gradient method developed under an information geometric viewpoint where the usual euclidean metric is replaced by a riemannian metric on the space of transition probabilities it is shown that the change in metric provides advantages over more traditional approaches namely 1 it turns the original constrained optimization into an unconstrained optimization problem 2 the optimization behaves asymptotically as a newton method and yields faster convergence than other methods for the same computational complexity and 3 the natural gradient vector is an actual contravariant vector on the space of probability distributions for which an interpretation as the steepest descent direction is formally correct experiments on synthetic and real world problems focused on human activity recognition in long range surveillance settings show that the proposed methodology compares favorably with the state of the art algorithms developed for the same purpose

correctness of local probability propagation in graphical models with loops graphical models such as bayesian networks and markov networks represent joint distributions over a set of variables by means of a graph when the graph is singly connected local propagation rules of the sort proposed by pearl 1988 are guaranteed to converge to the correct posterior probabilities recently a number of researchers have empirically demonstrated good performance of these same local propagation schemes on graphs with loops but a theoretical understanding of this performance has yet to be achieved for graphical models with a single loop we derive an analytical relationship between the probabilities computed using local propagation and the correct marginals using this relationship we show a category of graphical models with loops for which local propagation gives rise to provably optimal maximum a posteriori assignments although the computed marginals will be incorrect we also show how nodes can use local information in the messages they receive in order to correct their computed marginals we discuss how these results can be extended to graphical models with multiple loops and show simulation results suggesting that some properties of propagation on single loop graphs may hold for a larger class of graphs specifically we discuss the implication of our results for understanding a class of recently proposed error correcting codes known as turbo codes

fractional dirac bracket and quantization for constrained systems so far it is not well known how to deal with dissipative systems there are many paths of investigation in the literature and none of them present a systematic and general procedure to tackle the problem on the other hand it is well known that the fractional formalism is a powerful alternative when treating dissipative problems in this paper we propose a detailed way of attacking the issue using fractional calculus to construct an extension of the dirac brackets in order to carry out the quantization of nonconservative theories through the standard canonical way we believe that by using the extended dirac bracket definition it will be possible to analyze more deeply gauge theories starting with second class systems

statistical inference for the population landscape via moment adjusted stochastic gradients modern statistical inference tasks often require iterative optimization methods to compute the solution convergence analysis from an optimization viewpoint informs us only how well the solution is approximated numerically but overlooks the sampling nature of the data in contrast recognizing the randomness in the data statisticians are keen to provide uncertainty quantification or confidence for the solution obtained by using iterative optimization methods the paper makes progress along this direction by introducing moment adjusted stochastic gradient descent a new stochastic optimization method for statistical inference we establish non asymptotic theory that characterizes the statistical distribution for certain iterative methods with optimization guarantees on the statistical front the theory allows for model misspecification with very mild conditions on the data for optimization the theory is flexible for both convex and non convex cases remarkably the moment adjusting idea motivated from error standardization in statistics achieves a similar effect to acceleration in first order optimization methods that are used to fit generalized linear models we also demonstrate this acceleration effect in the non convex setting through numerical experiments

minimum volume constrained nonnegative matrix factorization enhanced ability of learning parts nonnegative matrix factorization nmf with minimum volume constraint mvc is exploited in this paper our results show that mvc can actually improve the sparseness of the results of nmf this sparseness is l0 norm oriented and can give desirable results even in very weak sparseness situations thereby leading to the significantly enhanced ability of learning parts of nmf the close relation between nmf sparse nmf and the mvc nmf is discussed first then two algorithms are proposed to solve the mvc nmf model one is called quadratic programming mvc nmf qp mvc nmf which is based on quadratic programming and the other is called negative glow mvc nmf ng mvc nmf because it uses multiplicative updates incorporating natural gradient ingeniously the qp mvc nmf algorithm is quite efficient for small scale problems and the ng mvc nmf algorithm is more suitable for large scale problems simulations show the efficiency and validity of the proposed methods in applications of blind source separation and human face images analysis

on the relationship between belief propagation decoding and joint maximum likelihood detection belief propagation via a novel reinterpretation of the bethe free energy s pseudo dual is shown to be related to a novel relaxation of maximum likelihood detection via a constrained optimization the conventional maximum likelihood detection falls out for a zero constraint and belief propagation s fixed points are obtained for other constraint values

equivariant adaptive source separation source separation consists of recovering a set of independent signals when only mixtures with unknown coefficients are observed this paper introduces a class of adaptive algorithms for source separation that implements an adaptive version of equivariant estimation and is henceforth called equivariant adaptive separation via independence easi the easi algorithms are based on the idea of serial updating this specific form of matrix updates systematically yields algorithms with a simple structure for both real and complex mixtures most importantly the performance of an easi algorithm does not depend on the mixing matrix in particular convergence rates stability conditions and interference rejection levels depend only on the normalized distributions of the source signals closed form expressions of these quantities are given via an asymptotic performance analysis the theme of equivariance is stressed throughout the paper the source separation problem has an underlying multiplicative structure the parameter space forms a matrix multiplicative group we explore the favorable consequences of this fact on implementation performance and optimization of easi algorithms

blind source separation by sparse decomposition in a signal dictionary the blind source separation problem is to extract the underlying source signals from a set of linear mixtures where the mixing matrix is unknown this situation is common in acoustics radio medical signal and image processing hyperspectral imaging and other areas we suggest a two stage separation process a priori selection of a possibly overcomplete signal dictionary for instance a wavelet frame or a learned dictionary in which the sources are assumed to be sparsely representable followed by unmixing the sources by exploiting the their sparse representability we consider the general case of more sources than mixtures but also derive a more efficient algorithm in the case of a nonovercomplete dictionary and an equal numbers of sources and mixtures experiments with artificial signals and musical sounds demonstrate significantly better separation than other known techniques

nonlinear kalman filtering with divergence minimization we consider the nonlinear kalman filtering problem using kullback leibler kl and alpha divergence measures as optimization criteria unlike linear kalman filters nonlinear kalman filters do not have closed form gaussian posteriors because of a lack of conjugacy due to the nonlinearity in the likelihood in this paper we propose novel algorithms to approximate this posterior by optimizing the forward and reverse forms of the kl divergence as well as the alpha divergence that contains these two as limiting cases unlike previous approaches our algorithms do not make approximations to the divergences being optimized but use monte carlo techniques to derive unbiased algorithms for direct optimization we assess performance on radar and sensor tracking and options pricing showing general improvement over the extended unscented and ensemble kalman filters as well as competitive performance with particle filtering

the relation between information theory and the differential geometry approach to statistics abstract cencov has shown that the riemannian metric on the probability simplex xi 1 defined by ds 2 dx i 2 x i has an invariance property under certain probabilistically natural mappings no other riemannian metric has the same property the geometry associated with this metric is shown to lead almost automatically to measures of divergence between probability distributions which are associated with kullback bhattacharyya and matusita certain vector fields are associated in a natural way with random variables the integral curves of these vector fields yield the maximum entropy or minimum divergence estimates of probabilities some other consequences of this geometric view are also explored

stochastic reasoning free energy and information geometry belief propagation bp is a universal method of stochastic reasoning it gives exact inference for stochastic models with tree interactions and works surprisingly well even if the models have loopy interactions its performance has been analyzed separately in many fields such as ai statistical physics information theory and information geometry this article gives a unified framework for understanding bp and related methods and summarizes the results obtained in many fields in particular bp and its variants including tree reparameterization and concave convex procedure are reformulated with information geometrical terms and their relations to the free energy function are elucidated from an information geometrical viewpoint we then propose a family of new algorithms the stabilities of the algorithms are analyzed and methods to accelerate them are investigated

learning dynamics of a single polar variable complex valued neuron this letter investigates the characteristics of the complex valued neuron model with parameters represented by polar coordinates called polar variable complex valued neuron the parameters of the polar variable complex valued neuron are unidentifiable the plateau phenomenon can occur during learning of the polar variable complex valued neuron furthermore computer simulations suggest that a single polar variable complex valued neuron has the following characteristics in the case of using the steepest gradient descent method with square error 1 unidentifiable parameters singular points degrade the learning speed and 2 a plateau can occur during learning when the weight is attracted to the singular point the learning tends to become stuck however computer simulations also show that the steepest gradient descent method with amplitude phase error and the complex valued natural gradient method could reduce the effects of the singular points the learning dynamics near singular points depends on the error functions and the training algorithms used

linearization of resonant vector fields a method allowing the formal linearization of a large class of vector fields with resonant eigenvalues is presented the admissible nonlinearities being characterized by conditions that are easy to check this method also gives information on the terms that are actually present in a nonlinear normal form of a given resonant vector field

an information maximization approach to blind separation and blind deconvolution we derive a new self organizing learning algorithm that maximizes the information transferred in a network of nonlinear units the algorithm does not assume any knowledge of the input distributions and is defined here for the zero noise limit under these conditions information maximization has extra properties not found in the linear case linsker 1989 the nonlinearities in the transfer function are able to pick up higher order moments of the input distributions and perform something akin to true redundancy reduction between units in the output representation this enables the network to separate statistically independent components in the inputs a higher order generalization of principal components analysis we apply the network to the source separation or cocktail party problem successfully separating unknown mixtures of up to 10 speakers we also show that a variant on the network architecture is able to perform blind deconvolution cancellation of unknown echoes and reverberation in a speech signal finally we derive dependencies of information transfer on time delays we suggest that information maximization provides a unifying framework for problems in blind signal processing

online model selection based on the variational bayes the bayesian framework provides a principled way of model selection this framework estimates a probability distribution over an ensemble of models and the prediction is done by averaging over the ensemble of models accordingly the uncertainty of the models is taken into account and complex models with more degrees of freedom are penalized however integration over model parameters is often intractable and some approximation scheme is needed recently a powerful approximation scheme called the variational bayes vb method has been proposed this approach defines the free energy for a trial probability distribution which approximates a joint posterior probability distribution over model parameters and hidden variables the exact maximization of the free energy gives the true posterior distribution the vb method uses factorized trial distributions the integration over model parameters can be done analytically and an iterative expectation maximization like algorithm whose convergence is guaranteed is derived in this article we derive an online version of the vb algorithm and prove its convergence by showing that it is a stochastic approximation for finding the maximum of the free energy by combining sequential model selection procedures the online vb method provides a fully online learning method with a model selection mechanism in preliminary experiments using synthetic data the online vb method was able to adapt the model structure to dynamic environments

statistical theory of learning curves under entropic loss criterion the present paper elucidates a universal property of learning curves which shows how the generalization error training error and the complexity of the underlying stochastic machine are related and how the behavior of a stochastic machine is improved as the number of training examples increases the error is measured by the entropic loss it is proved that the generalization error converges to h0 the entropy of the conditional distribution of the true machine as h0 m 2t while the training error converges as h0 m 2t where t is the number of examples and m shows the complexity of the network when the model is faithful implying that the true machine is in the model m is reduced to m the number of modifiable parameters this is a universal law because it holds for any regular machine irrespective of its structure under the maximum likelihood estimator similar relations are obtained for the bayes and gibbs learning algorithms these learning curves show the relation among the accuracy of learning the complexity of a model and the number of training examples

source separation in post nonlinear mixtures we address the problem of separation of mutually independent sources in nonlinear mixtures first we propose theoretical results and prove that in the general case it is not possible to separate the sources without nonlinear distortion therefore we focus our work on specific nonlinear mixtures known as post nonlinear mixtures these mixtures constituted by a linear instantaneous mixture linear memoryless channel followed by an unknown and invertible memoryless nonlinear distortion are realistic models in many situations and emphasize interesting properties i e in such nonlinear mixtures sources can be estimated with the same indeterminacies as in instantaneous linear mixtures the separation structure of nonlinear mixtures is a two stage system namely a nonlinear stage followed by a linear stage the parameters of which are updated to minimize an output independence criterion expressed as a mutual information criterion the minimization of this criterion requires knowledge or estimation of source densities or of their log derivatives a first algorithm based on a gram charlier expansion of densities is proposed unfortunately it fails for hard nonlinear mixtures a second algorithm based on an adaptive estimation of the log derivative of densities leads to very good performance even with hard nonlinearities experiments are proposed to illustrate these results