natural gradient learning for over and under complete bases in ica independent component analysis or blind source separation is a new technique of extracting independent signals from mixtures it is applicable even when the number of independent sources is unknown and is larger or smaller than the number of observed mixture signals this article extends the natural gradient learning algorithm to be applicable to these overcomplete and undercomplete cases here the observed signals are assumed to be whitened by preprocessing so that we use the natural riemannian gradient in stiefel manifolds

matrix momentum for practical natural gradient learning an on line learning rule based on the introduction of a matrix momentum term is presented aimed at alleviating the computational costs of standard natural gradient learning the new rule natural gradient matrix momentum is analysed in the case of two layer feed forward neural network learning via methods of statistical physics it appears to provide a practical algorithm that performs as well as standard natural gradient descent in both the transient and asymptotic regimes but with a hugely reduced complexity

learning maximum entropy models from finite size data sets a fast data driven algorithm allows sampling from the posterior distribution maximum entropy models provide the least constrained probability distributions that reproduce statistical properties of experimental datasets in this work we characterize the learning dynamics that maximizes the log likelihood in the case of large but finite datasets we first show how the steepest descent dynamics is not optimal as it is slowed down by the inhomogeneous curvature of the model parameters space we then provide a way for rectifying this space which relies only on dataset properties and does not require large computational efforts we conclude by solving the long time limit of the parameters dynamics including the randomness generated by the systematic use of gibbs sampling in this stochastic framework rather than converging to a fixed point the dynamics reaches a stationary distribution which for the rectified dynamics reproduces the posterior distribution of the parameters we sum up all these insights in a rectified data driven algorithm that is fast and by sampling from the parameters posterior avoids both under and overfitting along all the directions of the parameters space through the learning of pairwise ising models from the recording of a large population of retina neurons we show how our algorithm outperforms the steepest descent method

sufficient dimension reduction via squared loss mutual information estimation the goal of sufficient dimension reduction in supervised learning is to find the low dimensional subspace of input features that contains all of the information about the output values that the input features possess in this letter we propose a novel sufficient dimension reduction method using a squared loss variant of mutual information as a dependency measure we apply a density ratio estimator for approximating squared loss mutual information that is formulated as a minimum contrast estimator on parametric or nonparametric models since cross validation is available for choosing an appropriate model our method does not require any prespecified structure on the underlying distributions we elucidate the asymptotic bias of our estimator on parametric models and the asymptotic convergence rate on nonparametric models the convergence analysis utilizes the uniform tail bound of a u process and the convergence rate is characterized by the bracketing entropy of the model we then develop a natural gradient algorithm on the grassmann manifold for sufficient subspace search the analytic formula of our estimator allows us to compute the gradient efficiently numerical experiments show that the proposed method compares favorably with existing dimension reduction approaches on artificial and benchmark data sets

statistical theory of learning curves under entropic loss criterion the present paper elucidates a universal property of learning curves which shows how the generalization error training error and the complexity of the underlying stochastic machine are related and how the behavior of a stochastic machine is improved as the number of training examples increases the error is measured by the entropic loss it is proved that the generalization error converges to h0 the entropy of the conditional distribution of the true machine as h0 m 2t while the training error converges as h0 m 2t where t is the number of examples and m shows the complexity of the network when the model is faithful implying that the true machine is in the model m is reduced to m the number of modifiable parameters this is a universal law because it holds for any regular machine irrespective of its structure under the maximum likelihood estimator similar relations are obtained for the bayes and gibbs learning algorithms these learning curves show the relation among the accuracy of learning the complexity of a model and the number of training examples

complexity issues in natural gradient descent method for training multilayer perceptrons the natural gradient descent method is applied to train an n m 1 multilayer perceptron based on an efficient scheme to represent the fisher information matrix for an n m 1 stochastic multilayer perceptron a new algorithm is proposed to calculate the natural gradient without inverting the fisher information matrix explicitly when the input dimension n is much larger than the number of hidden neurons m the time complexity of computing the natural gradient is o n

equivariant adaptive selective transmission in this paper we consider the problem of selective transmission the dual of the blind source separation task in which a set of independent source signals are adaptively premixed prior to a nondispersive physical mixing process so that each source can be independently monitored in the far field following similar procedures for information theoretic blind source separation we derive a stochastic gradient algorithm for iteratively estimating the premixing matrix in the selective transmission problem and through a simple modification we obtain a second algorithm whose performance is equivariant with respect to the channel s mixing characteristics the local stability conditions for the algorithms about any selective transmission solution are shown to be the same as those for similar source separation algorithms practical implementation issues are discussed including the estimation of the combined system matrix and the reordering and scaling of the received signals within the algorithm mean square error based selective transmission algorithms are also derived for performance comparison purposes simulations indicate the useful behavior of the premixing algorithms for selective transmission

an iterative inversion approach to blind source separation we present an iterative inversion ii approach to blind source separation bss it consists of a quasi newton method for the resolution of an estimating equation obtained from the implicit inversion of a robust estimate of the mixing system the resulting learning rule includes several existing algorithms for bss as particular cases giving them a novel and unified interpretation it also provides a justification of the cardoso and laheld 1996 step size normalization the ii method is first presented for instantaneous mixtures and then extended to the problem of blind separation of convolutive mixtures finally we derive the necessary and sufficient asymptotic stability conditions for both the instantaneous and convolutive methods to converge

blind separation of a mixture of uniformly distributed source signals a novel approach a new efficient algorithm for blind separation of uniformly distributed sources is proposed the mixing matrix is assumed to be orthogonal by prewhitening the observed signals the learning rule adaptively estimates the mixing matrix by conceptually rotating a unit hypercube so that all output signal components are contained within or on the hypercube under some ideal constraints it has been theoretically shown that the algorithm is very similar to an ideal convergent algorithm which is much faster than the existing convergent algorithms the algorithm has been generalized to take care of the noisy signals by adaptively dilating the hypercube in conjunction with its rotation

least squares independent component analysis accurately evaluating statistical independence among random variables is a key element of independent component analysis ica in this letter we employ a squared loss variant of mutual information as an independence measure and give its estimation method our basic idea is to estimate the ratio of probability densities directly without going through density estimation thereby avoiding the difficult task of density estimation in this density ratio approach a natural cross validation procedure is available for hyperparameter selection thus all tuning parameters such as the kernel width or the regularization parameter can be objectively optimized this is an advantage over recently developed kernel based independence measures and is a highly useful property in unsupervised learning problems such as ica based on this novel independence measure we develop an ica algorithm named least squares independent component analysis

a numerical investigation of blow up in reaction diffusion problems with traveling heat sources this paper studies the numerical solution of a reaction diffusion differential equation with traveling heat sources according to the fact that the locations of heat sources are known we add auxiliary mesh points exactly at heat sources and present a novel moving mesh algorithm for solving the problem several examples are provided to demonstrate the efficiency of the new moving mesh method especially in the case of two or three traveling heat sources moreover numerical results illustrate that the speed of the movement of the heat source is critical for blow up when there is only one traveling heat source for the case of two traveling heat sources blow up depends not only on the speed but also on the distance between the two traveling heat sources

a mean field optimal control formulation for global optimization this paper is concerned with variational optimal control constructions whose solution yields a sampling algorithm the particular form of the sampling algorithm considered here is a particle filter designed to numerically approximate the solution to the global optimization problem the theoretical significance of this study comes from its variational aspects specifically the control input represents the solution of a mean field type optimal control problem its parametric counterpart obtained when a parametric form of density is known a priori is shown to be equivalent to the natural gradient algorithm explicit formulae for the filter are derived when the objective function is quadratic and the density is gaussian the optimal control construction of the particle filter is a significant departure from the classical importance sampling resampling based approaches

robust independent component analysis by iterative maximization of the kurtosis contrast with algebraic optimal step size independent component analysis ica aims at decomposing an observed random vector into statistically independent variables deflation based implementations such as the popular one unit fastica algorithm and its variants extract the independent components one after another a novel method for deflationary ica referred to as robustica is put forward in this paper this simple technique consists of performing exact line search optimization of the kurtosis contrast function the step size leading to the global maximum of the contrast along the search direction is found among the roots of a fourth degree polynomial this polynomial rooting can be performed algebraically and thus at low cost at each iteration among other practical benefits robustica can avoid prewhitening and deals with real and complex valued mixtures of possibly noncircular sources alike the absence of prewhitening improves asymptotic performance the algorithm is robust to local extrema and shows a very high convergence speed in terms of the computational cost required to reach a given source extraction quality particularly for short data records these features are demonstrated by a comparative numerical analysis on synthetic data robustica s capabilities in processing real world data involving noncircular complex strongly super gaussian sources are illustrated by the biomedical problem of atrial activity aa extraction in atrial fibrillation af electrocardiograms ecgs where it outperforms an alternative ica based technique

an alternative perspective on adaptive independent component analysis algorithms this article develops an extended independent component analysis algorithm for mixtures of arbitrary subgaussian and supergaussian sources the gaussian mixture model of pearson is employed in deriving a closedform generic score function for strictly subgaussian sources this is combined with the score function for a unimodal supergaussian density to provide a computationally simple yet powerful algorithm for performing independent component analysis on arbitrary mixtures of nongaussian sources

recurrent sampling models for the helmholtz machine many recent analysis by synthesis density estimation models of cortical learning and processing have made the crucial simplifying assumption that units within a single layer are mutually independent given the states of units in the layer below or the layer above in this article we suggest using either a markov random field or an alternative stochastic sampling architecture to capture explicitly particular forms of dependence within each layer we develop the architectures in the context of real and binary helmholtz machines recurrent sampling can be used to capture correlations within layers in the generative or the recognition models and we also show how these can be combined

energy efficient distributed filtering with wireless sensor networks we consider a wireless sensor network wsn that monitors a physical field and communicates pertinent data to a distant fusion center fc we study the case of a binary valued hidden natural field observed in a significant amount of gaussian clutter which is relevant to applications like detection of plumes or oil slicks the considerable spatio temporal dependencies found in natural fields can be exploited to improve the reliability of the detection estimation of hidden phenomena while this problem has been previously treated using kernel regression techniques we formulate it as a task of delay free filtering on a process observed by the sensors we propose a distributed scalable implementation of the filter within the network this is achieved by i exploiting the localized spatio temporal dependencies to define a hidden markov model hmm in terms of an exponential family with o n parameters where n is the size of the wsn ii using a reduced state approximation of the propagated probability mass function and iii making a tractable approximation of model marginals by using iterated decoding algorithms like the gibbs sampler gs mean field decoding mfd iterated conditional modes icm and broadcast belief propagation bbp we compare the marginalization algorithms in terms of their information geometry performance complexity and communication load finally we analyze the energy efficiency of the proposed distributed filter relative to brute force data fusion it is demonstrated that when the fc is sufficiently far away from the sensor array distributed filtering is significantly more energy efficient and can increase the lifetime of the wsn by one to two orders of magnitude

bayesian texture classification from indirect observations using fast sampling a bayesian method for texture model choice from blurred and noisy i e indirect observations is presented the textures are modeled by stationary random fields with various distribution laws either gaussian or scale mixtures of gaussians the power spectral densities of the fields are modeled by parametric functions and the aim is to select the most appropriate model among a set of candidates this is achieved by computing the a posteriori model probabilities through parameter marginalization the marginalization is done by sampling and harmonic mean approach considering separately each model in a within model sampling strategy the highly nonlinear dependency with respect to the parameters imposes the use of the metropolis hastings sampler moreover to achieve efficient sampling the paper proposes a new fast algorithm based on the fisher information matrix the fisher metropolis hastings

a linear discriminant analysis method based on mutual information maximization we present a new linear discriminant analysis method based on information theory where the mutual information between linearly transformed input data and the class labels is maximized first we introduce a kernel based estimate of mutual information with a variable kernel size furthermore we devise a learning algorithm that maximizes the mutual information w r t the linear transformation two experiments are conducted the first one uses a toy problem to visualize and compare the transformation vectors in the original input space the second one evaluates the performance of the method for classification by employing cross validation tests on four datasets from the uci repository various classifiers are investigated our results show that this method can significantly boost class separability over conventional methods especially for nonlinear classification

on line versus off line learning from random examples general results i propose a general model of on line learning from random examples which when applied to a smooth realizable stochastic rule yields the same asymptotic generalization error rate as optimal batch algorithms the approach is based on an iterative gaussian approximation to the posterior gibbs distribution of rule parameters

blind source separation semiparametric statistical approach the semiparametric statistical model is used to formulate the problem of blind source separation the method of estimating functions is applied to this problem it is shown that an estimator of the mixing matrix or its learning version can be described in terms of an estimating function the statistical efficiencies of these algorithms are studied the main results are as follows 1 the space consisting of all the estimating functions is derived 2 the space is decomposed into the orthogonal sum of the admissible part and a redundant ancillary part for any estimating function one can find a better or equally good estimator in the admissible part 3 the fisher efficient that is asymptotically best estimating functions are derived 4 the stability of learning algorithms is studied

how to center deep boltzmann machines this work analyzes centered restricted boltzmann machines rbms and centered deep boltzmann machines dbms where centering is done by subtracting offset values from visible and hidden variables we show analytically that i centered and normal boltzmann machines bms and thus rbms and dbms are different parameterizations of the same model class such that any normal bm rbm dbm can be transformed to an equivalent centered bm rbm dbm and vice versa and that this equivalence generalizes to artificial neural networks in general ii the expected performance of centered binary bms rbms dbms is invariant under simultaneous flip of data and offsets for any off set value in the range of zero to one iii centering can be reformulated as a different update rule for normal bms rbms dbms and iv using the enhanced gradient is equivalent to setting the offset values to the average over model and data mean furthermore we present numerical simulations suggesting that i optimal generative performance is achieved by subtracting mean values from visible as well as hidden variables ii centered binary rbms dbms reach significantly higher log likelihood values than normal binary rbms dbms iii centering variants whose offsets depend on the model mean like the enhanced gradient suffer from severe divergence problems iv learning is stabilized if an exponentially moving average over the batch means is used for the offset values instead of the current batch mean which also prevents the enhanced gradient from severe divergence v on a similar level of log likelihood values centered binary rbms dbms have smaller weights and bigger bias parameters than normal binary rbms dbms vi centering leads to an update direction that is closer to the natural gradient which is extremely efficient for training as we show for small binary rbms vii centering eliminates the need for greedy layer wise pre training of dbms which often even deteriorates the results independently of whether centering is used or not and ix centering is also beneficial for auto encoders

serial updating rule for blind separation derived from the method of scoring in the context of blind source separation the method of scoring based on the inverse of the fisher information matrix fim becomes the serial updating learning rule with an equivariant property this learning rule can be simplified to a low complexity algorithm by using the asymptotic form of the ftm around the equilibrium the simplified learning rule is still general enough to include some existing equivariant blind separation algorithms as its special cases

preconditioned stochastic gradient descent stochastic gradient descent sgd still is the workhorse for many practical problems however it converges slow and can be difficult to tune it is possible to precondition sgd to accelerate its convergence remarkably but many attempts in this direction either aim at solving specialized problems or result in significantly more complicated methods than sgd this paper proposes a new method to adaptively estimate a preconditioner such that the amplitudes of perturbations of preconditioned stochastic gradient match that of the perturbations of parameters to be optimized in a way comparable to newton method for deterministic optimization unlike the preconditioners based on secant equation fitting as done in deterministic quasi newton methods which assume positive definite hessian and approximate its inverse the new preconditioner works equally well for both convex and nonconvex optimizations with exact or noisy gradients when stochastic gradient is used it can naturally damp the gradient noise to stabilize sgd efficient preconditioner estimation methods are developed and with reasonable simplifications they are applicable to large scale problems experimental results demonstrate that equipped with the new preconditioner without any tuning effort preconditioned sgd can efficiently solve many challenging problems like the training of a deep neural network or a recurrent neural network requiring extremely long term memories

adaptive method of realizing natural gradient learning for multilayer perceptrons the natural gradient learning method is known to have ideal performances for on line training of multilayer perceptrons it avoids plateaus which give rise to slow convergence of the backpropagation method it is fisher efficient whereas the conventional method is not however for implementing the method it is necessary to calculate the fisher information matrix and its inverse which is practically very difficult this article proposes an adaptive method of directly obtaining the inverse of the fisher information matrix it generalizes the adaptive gauss newton algorithms and provides a solid theoretical justification of them simulations show that the proposed adaptive method works very well for realizing natural gradient learning

approximate riemannian conjugate gradient learning for fixed form variational bayes variational bayesian vb methods are typically only applied to models in the conjugate exponential family using the variational bayesian expectation maximisation vb em algorithm or one of its variants in this paper we present an efficient algorithm for applying vb to more general models the method is based on specifying the functional form of the approximation such as multivariate gaussian the parameters of the approximation are optimised using a conjugate gradient algorithm that utilises the riemannian geometry of the space of the approximations this leads to a very efficient algorithm for suitably structured approximations it is shown empirically that the proposed method is comparable or superior in efficiency to the vb em in a case where both are applicable we also apply the algorithm to learning a nonlinear state space model and a nonlinear factor analysis model for which the vb em is not applicable for these models the proposed algorithm outperforms alternative gradient based methods by a significant margin

variational inference a review for statisticians abstractone of the core problems of modern statistics is to approximate difficult to compute probability densities this problem is especially important in bayesian statistics which frames all inference about unknown quantities as a calculation involving the posterior density in this article we review variational inference vi a method from machine learning that approximates probability densities through optimization vi has been used in many applications and tends to be faster than classical methods such as markov chain monte carlo sampling the idea behind vi is to first posit a family of densities and then to find a member of that family which is close to the target density closeness is measured by kullback leibler divergence we review the ideas behind mean field variational inference discuss the special case of vi applied to exponential family models present a full example with a bayesian mixture of gaussians and derive a variant that uses stochastic optimization to scale up to massive data

natural evolution strategies this paper presents natural evolution strategies nes a recent family of black box optimization algorithms that use the natural gradient to update a parameterized search distribution in the direction of higher expected fitness we introduce a collection of techniques that address issues of convergence robustness sample complexity computational complexity and sensitivity to hyperparameters this paper explores a number of implementations of the nes family such as general purpose multi variate normal distributions and separable distributions tailored towards search in high dimensional spaces experimental results show best published performance on various standard benchmarks as well as competitive performance on others

pegasos primal estimated sub gradient solver for svm we describe and analyze a simple and effective stochastic sub gradient descent algorithm for solving the optimization problem cast by support vector machines svm we prove that the number of iterations required to obtain a solution of accuracy epsilon is tilde o 1 epsilon where each iteration operates on a single training example in contrast previous analyses of stochastic gradient descent methods for svms require omega 1 epsilon 2 iterations as in previously devised svm solvers the number of iterations also scales linearly with 1 where is the regularization parameter of svm for a linear kernel the total run time of our method is tilde o d lambda epsilon where d is a bound on the number of non zero features in each example since the run time does not depend directly on the size of the training set the resulting algorithm is especially suited for learning from large datasets our approach also extends to non linear kernels while working solely on the primal objective function though in this case the runtime does depend linearly on the training set size our algorithm is particularly well suited for large text classification problems where we demonstrate an order of magnitude speedup over previous svm learning methods

fast algorithms for mutual information based independent component analysis this paper provides fast algorithms to perform independent component analysis based on the mutual information criterion the main ingredient is the binning technique and the use of cardinal splines which allows the fast computation of the density estimator over a regular grid using a discretized form of the entropy the criterion can be evaluated quickly together with its gradient which can be expressed in terms of the score functions both offline and online separation algorithms have been developed our density entropy and score estimators also have their own interest

a globally convergent mc algorithm with an adaptive learning rate this brief deals with the problem of minor component analysis mca artificial neural networks can be exploited to achieve the task of mca recent research works show that convergence of neural networks based mca algorithms can be guaranteed if the learning rates are less than certain thresholds however the computation of these thresholds needs information about the eigenvalues of the autocorrelation matrix of data set which is unavailable in online extraction of minor component from input data stream in this correspondence we introduce an adaptive learning rate into the ojan mca algorithm such that its convergence condition does not depend on any unobtainable information and can be easily satisfied in practical applications

information geometry of boltzmann machines a boltzmann machine is a network of stochastic neurons the set of all the boltzmann machines with a fixed topology forms a geometric manifold of high dimension where modifiable synaptic weights of connections play the role of a coordinate system to specify networks a learning trajectory for example is a curve in this manifold it is important to study the geometry of the neural manifold rather than the behavior of a single network in order to know the capabilities and limitations of neural networks of a fixed topology using the new theory of information geometry a natural invariant riemannian metric and a dual pair of affine connections on the boltzmann neural network manifold are established the meaning of geometrical structures is elucidated from the stochastic and the statistical point of view this leads to a natural modification of the boltzmann machine learning rule

blind separation of uniformly distributed signals a general approach a general algorithm for blind separation of uniformly distributed signals is presented first maximum likelihood equations are obtained for dealing with this task it is difficult to obtain a closed form maximum likelihood solution for arbitrary mixing matrix the learning rules are obtained based on the geometric interpretation of the maximum likelihood estimator the algorithm under special constraint of orthogonal mixing matrix is the same as the o 1 t sup 2 convergent algorithm special noise correction mechanisms are incorporated in the algorithm and it has been found that the algorithm exhibits stable performance even in the presence of large amount of noise

learning dynamics on different timescales the special character of certain degrees of freedom in two layered neural networks is investigated for on line learning of realizable rules our analysis shows that the dynamics of these degrees of freedom can be put on a faster timescale than those remaining with the profit of speeding up the overall adaptation process this is shown for two groups of degrees of freedom second layer weights and bias weights for the former case our analysis provides a theoretical explanation of phenomenological findings the resulting learning algorithm is compared with natural gradient descent in order to check whether the proposed scaling can be naturally derived from that type of learning rule

a generalized contrast function and stability analysis for overdetermined blind separation of instantaneous mixtures in this letter the problem of blind separation of n independent sources from their m linear instantaneous mixtures is considered first a generalized contrast function is defined as a valuable extension of the existing classical and nonsymmetrical contrast functions it is applicable to the overdetermined blind separation m n with an unknown number of sources because not only independent components but also redundant ones are allowed in the outputs of a separation system second a natural gradient learning algorithm developed primarily for the complete case m n is shown to work as well with an n m or m m separating matrix for each optimizes a certain mutual information contrast function finally we present stability analysis for a newly proposed generalized orthogonal natural gradient algorithm which can perform the overdetermined blind separation when n is unknown obtaining an expectable result that its local stability conditions are slightly stricter than those of the conventional natural gradient algorithm using an invertible mixing matrix m n

a theory for learning by weight flow on stiefel grassman manifold recently we introduced the concept of neural network learning on stiefel grassman manifold for multilayer perceptron like networks contributions of other authors have also appeared in the scientific literature about this topic this article presents a general theory for it and illustrates how existing theories may be explained within the general framework proposed here

a double variational bayesian framework in random fourier features for indefinite kernels random fourier features rffs have been successfully employed to kernel approximation in large scale situations the rationale behind rff relies on bochner s theorem but the condition is too strict and excludes many widely used kernels e g dot product kernels violates the shift invariant condition and indefinite kernels violates the positive definite pd condition in this article we present a unified rff framework for indefinite kernel approximation in the reproducing kernel kre n spaces rkkss besides our model is also suited to approximate a dot product kernel on the unit sphere as it can be transformed into a shift invariant but indefinite kernel by the kolmogorov decomposition scheme an indefinite kernel in rkks can be decomposed into the difference of two unknown pd kernels the spectral distribution of each underlying pd kernel can be formulated as a nonparametric bayesian gaussian mixtures model based on this we propose a double infinite gaussian mixture model in rff by placing the dirichlet process prior it takes full advantage of high flexibility on the number of components and has the capability of approximating indefinite kernels on a wide scale in model inference we develop a non conjugate variational algorithm with a sub sampling scheme for the posterior inference it allows for the non conjugate case in our model and is quite efficient due to the sub sampling strategy experimental results on several large classification data sets demonstrate the effectiveness of our nonparametric bayesian model for indefinite kernel approximation when compared to other representative random feature based methods

explicit runge kutta pairs with one more derivative evaluation than the minimum existing classes of 4 5 5 6 and 6 7 pairs use the minimum number of derivative evaluations per step this ensures that the cost of each step is minimized however the cost per unit step is not necessarily minimized because it may be possible to take larger steps if an extra evaluation is used the change in efficiency when one extra evaluation per step is used is investigated for classes of 4 5 5 6 and 6 7 pairs with the minimum number of evaluations the aim is to show that using an extra evaluation does not necessarily increase the cost per unit step and that in some cases the cost may decrease the paper begins with the derivation of the new classes then the new and existing classes are compared using standard functions of the free parameters following this numerical comparisons of representative pairs from the new and existing classes are presented the new 4 5 and 5 6 pairs are found to be more efficient and just as reliable as the existing 4 5 and 5 6 pairs for the 6 7

a unifying information theoretic framework for independent component analysis abstract we show that different theories recently proposed for independent component analysis ica lead to the same iterative learning algorithm for blind separation of mixed independent sources we review those theories and suggest that information theory can be used to unify several lines of research pearlmutter and parra 1 and cardoso 2 showed that the infomax approach of bell and sejnowski 3 and the maximum likelihood estimation approach are equivalent we show that negentropy maximization also has equivalent properties and therefore all three approaches yield the same learning rule for a fixed nonlinearity girolami and fyfe 4 have shown that the nonlinear principal component analysis pca algorithm of karhunen and joutsensalo 5 and oja 6 can also be viewed from information theoretic principles since it minimizes the sum of squares of the fourth order marginal cumulants and therefore approximately minimizes the mutual information 7 lambert 8 has proposed different bussgang cost functions for multichannel blind deconvolution we show how the bussgang property relates to the infomax principle finally we discuss convergence and stability as well as future research issues in blind source separation

statistical dynamics of on line independent component analysis the learning dynamics of on line independent component analysis is analysed in the limit of large data dimension we study a simple hebbian learning algorithm that can be used to separate out a small number of non gaussian components from a high dimensional data set the de mixing matrix parameters are confined to a stiefel manifold of tall orthogonal matrices and we introduce a natural gradient variant of the algorithm which is appropriate to learning on this manifold for large input dimension the parameter trajectory of both algorithms passes through a sequence of unstable fixed points each described by a diffusion process in a polynomial potential choosing the learning rate too large increases the escape time from each of these fixed points effectively trapping the learning in a sub optimal state in order to avoid these trapping states a very low learning rate must be chosen during the learning transient resulting in learning time scales of o n2 or o n3 iterations where n is the data dimension escape from each sub optimal state results in a sequence of symmetry breaking events as the algorithm learns each source in turn this is in marked contrast to the learning dynamics displayed by related on line learning algorithms for multilayer neural networks and principal component analysis although the natural gradient variant of the algorithm has nice asymptotic convergence properties it has an equivalent transient dynamics to the standard hebbian algorithm

adaptive online learning algorithms for blind separation maximum entropy and minimum mutual information there are two major approaches for blind separation maximum entropy me and minimum mutual information mmi both can be implemented by the stochastic gradient descent method for obtaining the demixing matrix the mi is the contrast function for blind separation the entropy is not to justify the me the relation between me and mmi is first elucidated by calculating the first derivative of the entropy and proving that the mean subtraction is necessary in applying the me and at the solution points determined by the mi the me will not update the demixing matrix in the directions of increasing the cross talking second the natural gradient instead of the ordinary gradient is introduced to obtain efficient algorithms because the parameter space is a riemannian space consisting of matrices the mutual information is calculated by applying the gram charlier expansion to approximate probability density functions of the outputs finally we propose an efficient learning algorithm that incorporates with an adaptive method of estimating the unknown cumulants it is shown by computer simulation that the convergence of the stochastic descent algorithms is improved by using the natural gradient and the adaptively estimated cumulants

algorithms for nonnegative independent component analysis we consider the task of solving the independent component analysis ica problem x as given observations x with a constraint of nonnegativity of the source random vector s we refer to this as nonnegative independent component analysis and we consider methods for solving this task for independent sources with nonzero probability density function pdf p s down to s 0 it is sufficient to find the orthonormal rotation y wz of prewhitened sources z vx which minimizes the mean squared error of the reconstruction of z from the rectified version y sup of y we suggest some algorithms which perform this both based on a nonlinear principal component analysis pca approach and on a geodesic search method driven by differential geometry considerations we demonstrate the operation of these algorithms on an image separation problem which shows in particular the fast convergence of the rotation and geodesic methods and apply the approach to a musical audio analysis task

dynamics of learning in multilayer perceptrons near singularities the dynamical behavior of learning is known to be very slow for the multilayer perceptron being often trapped in the plateau it has been recently understood that this is due to the singularity in the parameter space of perceptrons in which trajectories of learning are drawn the space is riemannian from the point of view of information geometry and contains singular regions where the riemannian metric or the fisher information matrix degenerates this paper analyzes the dynamics of learning in a neighborhood of the singular regions when the true teacher machine lies at the singularity we give explicit asymptotic analytical solutions trajectories both for the standard gradient sgd and natural gradient ngd methods it is clearly shown in the case of the sgd method that the plateau phenomenon appears in a neighborhood of the critical regions where the dynamical behavior is extremely slow the analysis of the ngd method is much more difficult because the inverse of the fisher information matrix diverges we conquer the difficulty by introducing the blow down technique used in algebraic geometry the ngd method works efficiently and the state converges directly to the true parameters very quickly while it staggers in the case of the sgd method the analytical results are compared with computer simulations showing good agreement the effects of singularities on learning are thus qualitatively clarified for both standard and ngd methods

bankruptcy analysis with self organizing maps in learning metrics we introduce a method for deriving a metric locally based on the fisher information matrix into the data space a self organizing map som is computed in the new metric to explore financial statements of enterprises the metric measures local distances in terms of changes in the distribution of an auxiliary random variable that reflects what is important in the data in this paper the variable indicates bankruptcy within the next few years the conditional density of the auxiliary variable is first estimated and the change in the estimate resulting from local displacements in the primary data space is measured using the fisher information matrix when a self organizing map is computed in the new metric it still visualizes the data space in a topology preserving fashion but represents the local directions in which the probability of bankruptcy changes the most

cauchy noise loss for stochastic optimization of random matrix models via free deterministic equivalents abstract for random matrix models the parameter estimation based on the traditional likelihood functions is not straightforward in particular when we have only one sample matrix we introduce a new parameter optimization method for random matrix models which works even in such a case the method is based on the spectral distribution instead of the traditional likelihood in the method the cauchy noise has an essential role because the free deterministic equivalent which is a tool in free probability theory allows us to approximate the spectral distribution perturbed by cauchy noises by a smooth and accessible density function moreover we study an asymptotic property of determination gap which has a similar role as generalization gap besides we propose a new dimensionality recovery method for the signal plus noise model and experimentally demonstrate that it recovers the rank of the signal part even if the true rank is not small it is a simultaneous rank selection and parameter estimation procedure

estimating functions of independent component analysis for temporally correlated signals this article studies a general theory of estimating functions of independent component analysis when the independent source signals are temporarily correlated estimating functions are used for deriving both batch and on line learning algorithms and they are applicable to blind cases where spatial and temporal probability structures of the sources are unknown most algorithms proposed so far can be analyzed in the framework of estimating functions an admissible class of estimating functions is derived and related efficient on line learning algorithms are introduced we analyze dynamical stability and statistical efficiency of these algorithms different from the independently and identically distributed case the algorithms work even when only the second order moments are used the method of simultaneous diagonalization of cross covariance matrices is also studied from the point of view of estimating functions

analysis of natural gradient descent for multilayer neural networks natural gradient descent ngd is an on line algorithm for redefining the steepest descent direction an analysis of ngd for training a multilayer neural network is presented using statistical mechanics the performance can be significantly improved using ngd algorithm and can be used for both the transient and asymptotic stages of learning

natural gradient works efficiently in learning when a parameter space has a certain underlying structure the ordinary gradient of a function does not represent its steepest direction but the natural gradient does information geometry is used for calculating the natural gradients in the parameter space of perceptrons the space of matrices for blind source separation and the space of linear dynamical systems for blind source deconvolution the dynamical behavior of natural gradient online learning is analyzed and is proved to be fisher efficient implying that it has asymptotically the same performance as the optimal batch estimation of parameters this suggests that the plateau phenomenon which appears in the backpropagation learning algorithm of multilayer perceptrons might disappear or might not be so serious when the natural gradient is used an adaptive method of updating the learning rate is proposed and analyzed

higher order boltzmann machines and entropy bounds we examine some aspects of the interface area between mathematical statistics and statistical physics relevant to the study of boltzmann machines the boltzmann machine learning algorithm is based on a variational principle gibbs lemma for relative entropy this fact suggests the possibility of a scheme of successive approximations here we consider successive approximations parametrized by the order of many body interactions among individual units we prove bounds on the gain in relative entropy in the crucial step of adding and estimating by hebb s rule a new parameter we address the problem of providing on the basis of local observations upper and lower bounds on the entropy while upper bounds are easily obtained by subadditivity lower bounds involve localization of hirschman bounds on a dual quantum system

superefficiency in blind source separation blind source separation is the problem of extracting independent signals from their mixtures without knowing the mixing coefficients nor the probability distributions of source signals and may be applied to eeg and meg imaging of the brain it is already known that certain algorithms work well for the extraction of independent components the present paper is concerned with superefficiency of these based on the statistical and dynamical analysis in a statistical estimation using t examples the covariance of any two extracted independent signals converges to 0 of the order of 1 t on line dynamics shows that the covariance is of the order of spl eta when the learning rate spl eta is fixed to a small constant in contrast with the above general properties a surprising superefficiency holds in blind source separation under certain conditions where superefficiency implies that covariance decreases in the order of 1 t sup 2 or of spl eta sup 2 the paper uses the natural gradient learning algorithm and method of estimating functions to obtain superefficient procedures for both batch estimation and on line learning a standardized estimating function is introduced to this end superefficiency does not imply that the error variances of the extracted signals decrease in the order of 1 t sup 2 or spl eta sup 2 but implies that their covariances and independencies do

pruning using parameter and neuronal metrics in this article we introduce a measure of optimality for architecture selection algorithms for neural networks the distance from the original network to the new network in a metric defined by the probability distributions of all possible networks we derive two pruning algorithms one based on a metric in parameter space and the other based on a metric in neuron space which are closely related to well known architecture selection algorithms such as gobs our framework extends the theoretically range of validity of gobs and therefore can explain results observed in previous experiments in addition we give some computational improvements for these algorithms

a stochastic quasi newton method for large scale optimization the question of how to incorporate curvature information into stochastic approximation methods is challenging the direct application of classical quasi newton updating techniques for deterministic optimization leads to noisy curvature estimates that have harmful effects on the robustness of the iteration in this paper we propose a stochastic quasi newton method that is efficient robust and scalable it employs the classical bfgs update formula in its limited memory form and is based on the observation that it is beneficial to collect curvature information pointwise and at spaced intervals one way to do this is through subsampled hessian vector products this technique differs from the classical approach that would compute differences of gradients at every iteration and where controlling the quality of the curvature estimates can be difficult we present numerical results on problems arising in machine learning that suggest that the proposed method shows much promise

information geometric algorithm for estimating switching probabilities in space varying hmm this paper proposes an iterative natural gradient algorithm to perform the optimization of switching probabilities in a space varying hidden markov model in the context of human activity recognition in long range surveillance the proposed method is a version of the gradient method developed under an information geometric viewpoint where the usual euclidean metric is replaced by a riemannian metric on the space of transition probabilities it is shown that the change in metric provides advantages over more traditional approaches namely 1 it turns the original constrained optimization into an unconstrained optimization problem 2 the optimization behaves asymptotically as a newton method and yields faster convergence than other methods for the same computational complexity and 3 the natural gradient vector is an actual contravariant vector on the space of probability distributions for which an interpretation as the steepest descent direction is formally correct experiments on synthetic and real world problems focused on human activity recognition in long range surveillance settings show that the proposed methodology compares favorably with the state of the art algorithms developed for the same purpose

statistical inference for the population landscape via moment adjusted stochastic gradients modern statistical inference tasks often require iterative optimization methods to compute the solution convergence analysis from an optimization viewpoint informs us only how well the solution is approximated numerically but overlooks the sampling nature of the data in contrast recognizing the randomness in the data statisticians are keen to provide uncertainty quantification or confidence for the solution obtained by using iterative optimization methods the paper makes progress along this direction by introducing moment adjusted stochastic gradient descent a new stochastic optimization method for statistical inference we establish non asymptotic theory that characterizes the statistical distribution for certain iterative methods with optimization guarantees on the statistical front the theory allows for model misspecification with very mild conditions on the data for optimization the theory is flexible for both convex and non convex cases remarkably the moment adjusting idea motivated from error standardization in statistics achieves a similar effect to acceleration in first order optimization methods that are used to fit generalized linear models we also demonstrate this acceleration effect in the non convex setting through numerical experiments

minimum volume constrained nonnegative matrix factorization enhanced ability of learning parts nonnegative matrix factorization nmf with minimum volume constraint mvc is exploited in this paper our results show that mvc can actually improve the sparseness of the results of nmf this sparseness is l0 norm oriented and can give desirable results even in very weak sparseness situations thereby leading to the significantly enhanced ability of learning parts of nmf the close relation between nmf sparse nmf and the mvc nmf is discussed first then two algorithms are proposed to solve the mvc nmf model one is called quadratic programming mvc nmf qp mvc nmf which is based on quadratic programming and the other is called negative glow mvc nmf ng mvc nmf because it uses multiplicative updates incorporating natural gradient ingeniously the qp mvc nmf algorithm is quite efficient for small scale problems and the ng mvc nmf algorithm is more suitable for large scale problems simulations show the efficiency and validity of the proposed methods in applications of blind source separation and human face images analysis

equivariant adaptive source separation source separation consists of recovering a set of independent signals when only mixtures with unknown coefficients are observed this paper introduces a class of adaptive algorithms for source separation that implements an adaptive version of equivariant estimation and is henceforth called equivariant adaptive separation via independence easi the easi algorithms are based on the idea of serial updating this specific form of matrix updates systematically yields algorithms with a simple structure for both real and complex mixtures most importantly the performance of an easi algorithm does not depend on the mixing matrix in particular convergence rates stability conditions and interference rejection levels depend only on the normalized distributions of the source signals closed form expressions of these quantities are given via an asymptotic performance analysis the theme of equivariance is stressed throughout the paper the source separation problem has an underlying multiplicative structure the parameter space forms a matrix multiplicative group we explore the favorable consequences of this fact on implementation performance and optimization of easi algorithms

newton stein method an optimization method for glms via stein s lemma we consider the problem of efficiently computing the maximum likelihood estimator in generalized linear models glms when the number of observations is much larger than the number of coefficients n p 1 in this regime optimization algorithms can immensely benefit from approximate second order information we propose an alternative way of constructing the curvature information by formulating it as an estimation problem and applying a stein type lemma which allows further improvements through subsampling and eigenvalue thresholding our algorithm enjoys fast convergence rates resembling that of second order methods with modest per iteration cost we provide its convergence analysis for the general case where the rows of the design matrix are samples from a sub gaussian distribution we show that the convergence has two phases a quadratic phase followed by a linear phase finally we empirically demonstrate that our algorithm achieves the highest performance compared to various optimization algorithms on several data sets

blind source separation by sparse decomposition in a signal dictionary the blind source separation problem is to extract the underlying source signals from a set of linear mixtures where the mixing matrix is unknown this situation is common in acoustics radio medical signal and image processing hyperspectral imaging and other areas we suggest a two stage separation process a priori selection of a possibly overcomplete signal dictionary for instance a wavelet frame or a learned dictionary in which the sources are assumed to be sparsely representable followed by unmixing the sources by exploiting the their sparse representability we consider the general case of more sources than mixtures but also derive a more efficient algorithm in the case of a nonovercomplete dictionary and an equal numbers of sources and mixtures experiments with artificial signals and musical sounds demonstrate significantly better separation than other known techniques

nonlinear kalman filtering with divergence minimization we consider the nonlinear kalman filtering problem using kullback leibler kl and alpha divergence measures as optimization criteria unlike linear kalman filters nonlinear kalman filters do not have closed form gaussian posteriors because of a lack of conjugacy due to the nonlinearity in the likelihood in this paper we propose novel algorithms to approximate this posterior by optimizing the forward and reverse forms of the kl divergence as well as the alpha divergence that contains these two as limiting cases unlike previous approaches our algorithms do not make approximations to the divergences being optimized but use monte carlo techniques to derive unbiased algorithms for direct optimization we assess performance on radar and sensor tracking and options pricing showing general improvement over the extended unscented and ensemble kalman filters as well as competitive performance with particle filtering

the relation between information theory and the differential geometry approach to statistics abstract cencov has shown that the riemannian metric on the probability simplex xi 1 defined by ds 2 dx i 2 x i has an invariance property under certain probabilistically natural mappings no other riemannian metric has the same property the geometry associated with this metric is shown to lead almost automatically to measures of divergence between probability distributions which are associated with kullback bhattacharyya and matusita certain vector fields are associated in a natural way with random variables the integral curves of these vector fields yield the maximum entropy or minimum divergence estimates of probabilities some other consequences of this geometric view are also explored

learning dynamics of a single polar variable complex valued neuron this letter investigates the characteristics of the complex valued neuron model with parameters represented by polar coordinates called polar variable complex valued neuron the parameters of the polar variable complex valued neuron are unidentifiable the plateau phenomenon can occur during learning of the polar variable complex valued neuron furthermore computer simulations suggest that a single polar variable complex valued neuron has the following characteristics in the case of using the steepest gradient descent method with square error 1 unidentifiable parameters singular points degrade the learning speed and 2 a plateau can occur during learning when the weight is attracted to the singular point the learning tends to become stuck however computer simulations also show that the steepest gradient descent method with amplitude phase error and the complex valued natural gradient method could reduce the effects of the singular points the learning dynamics near singular points depends on the error functions and the training algorithms used

an information maximization approach to blind separation and blind deconvolution we derive a new self organizing learning algorithm that maximizes the information transferred in a network of nonlinear units the algorithm does not assume any knowledge of the input distributions and is defined here for the zero noise limit under these conditions information maximization has extra properties not found in the linear case linsker 1989 the nonlinearities in the transfer function are able to pick up higher order moments of the input distributions and perform something akin to true redundancy reduction between units in the output representation this enables the network to separate statistically independent components in the inputs a higher order generalization of principal components analysis we apply the network to the source separation or cocktail party problem successfully separating unknown mixtures of up to 10 speakers we also show that a variant on the network architecture is able to perform blind deconvolution cancellation of unknown echoes and reverberation in a speech signal finally we derive dependencies of information transfer on time delays we suggest that information maximization provides a unifying framework for problems in blind signal processing

training faster by separating modes of variation in batch normalized models batch normalization bn is essential to effectively train state of the art deep convolutional neural networks cnn it normalizes the layer outputs during training using the statistics of each mini batch bn accelerates training procedure by allowing to safely utilize large learning rates and alleviates the need for careful initialization of the parameters in this work we study bn from the viewpoint of fisher kernels that arise from generative probability models we show that assuming samples within a mini batch are from the same probability density function then bn is identical to the fisher vector of a gaussian distribution that means batch normalizing transform can be explained in terms of kernels that naturally emerge from the probability density function that models the generative process of the underlying data distribution consequently it promises higher discrimination power for the batch normalized mini batch however given the rectifying non linearities employed in cnn architectures distribution of the layer outputs show an asymmetric characteristic therefore in order for bn to fully benefit from the aforementioned properties we propose approximating underlying data distribution not with one but a mixture of gaussian densities deriving fisher vector for a gaussian mixture model gmm reveals that batch normalization can be improved by independently normalizing with respect to the statistics of disentangled sub populations we refer to our proposed soft piecewise version of batch normalization as mixture normalization mn through extensive set of experiments on cifar 10 and cifar 100 using both a 5 layers deep cnn and modern inception v3 architecture we show that mixture normalization reduces required number of gradient updates to reach the maximum test accuracy of the batch normalized model by sim 31 47 31 47 across a variety of training scenarios replacing even a few bn modules with mn in the 48 layers deep inception v3 architecture is sufficient to not only obtain considerable training acceleration but also better final test accuracy we show that similar observations are valid for 40 and 100 layers deep densenet architectures as well we complement our study by evaluating the application of mixture normalization to the generative adversarial networks gans where mode collapse hinders the training process we solely replace a few batch normalization layers in the generator with our proposed mixture normalization our experiments using deep convolutional gan dcgan on cifar 10 show that mixture normalized dcgan not only provides an acceleration of sim 58 58 but also reaches lower better frechet inception distance fid of 33 35 compared to 37 56 of its batch normalized counterpart

online model selection based on the variational bayes the bayesian framework provides a principled way of model selection this framework estimates a probability distribution over an ensemble of models and the prediction is done by averaging over the ensemble of models accordingly the uncertainty of the models is taken into account and complex models with more degrees of freedom are penalized however integration over model parameters is often intractable and some approximation scheme is needed recently a powerful approximation scheme called the variational bayes vb method has been proposed this approach defines the free energy for a trial probability distribution which approximates a joint posterior probability distribution over model parameters and hidden variables the exact maximization of the free energy gives the true posterior distribution the vb method uses factorized trial distributions the integration over model parameters can be done analytically and an iterative expectation maximization like algorithm whose convergence is guaranteed is derived in this article we derive an online version of the vb algorithm and prove its convergence by showing that it is a stochastic approximation for finding the maximum of the free energy by combining sequential model selection procedures the online vb method provides a fully online learning method with a model selection mechanism in preliminary experiments using synthetic data the online vb method was able to adapt the model structure to dynamic environments

riemannian optimal identification method for linear systems with symmetric positive definite matrix this article develops identification methods for linear continuous time symmetric systems such as electrical network systems multiagent network systems and temperature dynamics in buildings to this end we formulate three system identification problems for the corresponding discrete time systems the first is a least squares problem in which we wish to minimize the sum of squared errors between the true and model outputs on the product manifold of the manifold of symmetric positive definite matrices and two euclidean spaces in the second problem to reduce the search dimensions the product manifold is replaced with the quotient set under a specified group action by the orthogonal group in the third problem the manifold of symmetric positive definite matrices in the first problem is replaced by the manifold of matrices with only positive diagonal elements in particular we examine the quotient geometry in the second problem we propose riemannian conjugate gradient methods for the three problems and select initial points using a popular subspace method the effectiveness of our proposed methods is demonstrated through numerical simulations and comparisons with the gauss newton method which is one of the most popular approach for solving least squares problems

source separation in post nonlinear mixtures we address the problem of separation of mutually independent sources in nonlinear mixtures first we propose theoretical results and prove that in the general case it is not possible to separate the sources without nonlinear distortion therefore we focus our work on specific nonlinear mixtures known as post nonlinear mixtures these mixtures constituted by a linear instantaneous mixture linear memoryless channel followed by an unknown and invertible memoryless nonlinear distortion are realistic models in many situations and emphasize interesting properties i e in such nonlinear mixtures sources can be estimated with the same indeterminacies as in instantaneous linear mixtures the separation structure of nonlinear mixtures is a two stage system namely a nonlinear stage followed by a linear stage the parameters of which are updated to minimize an output independence criterion expressed as a mutual information criterion the minimization of this criterion requires knowledge or estimation of source densities or of their log derivatives a first algorithm based on a gram charlier expansion of densities is proposed unfortunately it fails for hard nonlinear mixtures a second algorithm based on an adaptive estimation of the log derivative of densities leads to very good performance even with hard nonlinearities experiments are proposed to illustrate these results