natural gradient learning for over and under complete bases in ica independent component analysis or blind source separation is a new technique of extracting independent signals from mixtures it is applicable even when the number of independent sources is unknown and is larger or smaller than the number of observed mixture signals this article extends the natural gradient learning algorithm to be applicable to these overcomplete and undercomplete cases here the observed signals are assumed to be whitened by preprocessing so that we use the natural riemannian gradient in stiefel manifolds

sensitivity analysis for antenna near field imaging although imaging and inverse scattering problems have been thoroughly studied during the last century there is only a partial understanding of these complex problems most of the efforts have been placed on the development of efficient inversion algorithms and mathematical uniqueness results in comparison there are very few results and a limited knowledge about the information content in the inversion data in this paper we provide a mathematical framework for sensitivity analysis of antenna near field imaging problems based on the multipole expansion of the electromagnetic field and the fisher information to quantify the quality of data by exploiting this framework a fundamental relation for accuracy and resolution is formulated based on the crameacuter rao bound crb the sensitivity analysis is illustrated using a relevant example with cylindrical measurement data

matrix momentum for practical natural gradient learning an on line learning rule based on the introduction of a matrix momentum term is presented aimed at alleviating the computational costs of standard natural gradient learning the new rule natural gradient matrix momentum is analysed in the case of two layer feed forward neural network learning via methods of statistical physics it appears to provide a practical algorithm that performs as well as standard natural gradient descent in both the transient and asymptotic regimes but with a hugely reduced complexity

factorization in lorentz spaces with an application to centralizers abstract we give an explicit formula to factorize functions in the lorentz spaces l p q some applications to centralizers twisted sums and interpolation theory are included

on actor critic algorithms in this article we propose and analyze a class of actor critic algorithms these are two time scale algorithms in which the critic uses temporal difference learning with a linearly parameterized approximation architecture and the actor is updated in an approximate gradient direction based on information provided by the critic we show that the features for the critic should ideally span a subspace prescribed by the choice of parameterization of the actor we study actor critic algorithms for markov decision processes with polish state and action spaces we state and prove two results regarding their convergence

complexity issues in natural gradient descent method for training multilayer perceptrons the natural gradient descent method is applied to train an n m 1 multilayer perceptron based on an efficient scheme to represent the fisher information matrix for an n m 1 stochastic multilayer perceptron a new algorithm is proposed to calculate the natural gradient without inverting the fisher information matrix explicitly when the input dimension n is much larger than the number of hidden neurons m the time complexity of computing the natural gradient is o n

topological size of some subsets in certain calderon lozanowskiĭ spaces abstract for i 1 2 3 let i be young functions a topological measure space e an ideal of measurable complex valued functions defined on and e i be the corresponding calderon lozanowski space our aim in this paper is to give under mild conditions several results on topological size in the sense of baire of the sets f g e 1 e 2 f g e 3 and f g e 1 e 2 x v f g x is well defined where denotes the convolution or pointwise product of functions and v a compact neighborhood our results sharpen and unify the related results obtained in diverse areas during recent thirty years

blind separation of a mixture of uniformly distributed source signals a novel approach a new efficient algorithm for blind separation of uniformly distributed sources is proposed the mixing matrix is assumed to be orthogonal by prewhitening the observed signals the learning rule adaptively estimates the mixing matrix by conceptually rotating a unit hypercube so that all output signal components are contained within or on the hypercube under some ideal constraints it has been theoretically shown that the algorithm is very similar to an ideal convergent algorithm which is much faster than the existing convergent algorithms the algorithm has been generalized to take care of the noisy signals by adaptively dilating the hypercube in conjunction with its rotation

multiple emitter location and signal parameter estimation processing the signals received on an array of sensors for the location of the emitter is of great enough interest to have been treated under many special case assumptions the general problem considers sensors with arbitrary locations and arbitrary directional characteristics gain phase polarization in a noise interference environment of arbitrary covariance matrix this report is concerned first with the multiple emitter aspect of this problem and second with the generality of solution a description is given of the multiple signal classification music algorithm which provides asymptotically unbiased estimates of 1 number of incident wavefronts present 2 directions of arrival doa or emitter locations 3 strengths and cross correlations among the incident waveforms 4 noise interference strength examples and comparisons with methods based on maximum likelihood ml and maximum entropy me as well as conventional beamforming are included an example of its use as a multiple frequency estimator operating on time series is included

an alternative perspective on adaptive independent component analysis algorithms this article develops an extended independent component analysis algorithm for mixtures of arbitrary subgaussian and supergaussian sources the gaussian mixture model of pearson is employed in deriving a closedform generic score function for strictly subgaussian sources this is combined with the score function for a unimodal supergaussian density to provide a computationally simple yet powerful algorithm for performing independent component analysis on arbitrary mixtures of nongaussian sources

recurrent sampling models for the helmholtz machine many recent analysis by synthesis density estimation models of cortical learning and processing have made the crucial simplifying assumption that units within a single layer are mutually independent given the states of units in the layer below or the layer above in this article we suggest using either a markov random field or an alternative stochastic sampling architecture to capture explicitly particular forms of dependence within each layer we develop the architectures in the context of real and binary helmholtz machines recurrent sampling can be used to capture correlations within layers in the generative or the recognition models and we also show how these can be combined

exact maximum likelihood parameter estimation of superimposed exponential signals in noise a unified framework for the exact maximum likelihood estimation of the parameters of superimposed exponential signals in noise encompassing both the time series and the array problems is presented an exact expression for the ml criterion is derived in terms of the linear prediction polynomial of the signal and an iterative algorithm for the maximization of this criterion is presented the algorithm is equally applicable in the case of signal coherence in the array problem simulation shows the estimator to be capable of providing more accurate frequency estimates than currently existing techniques the algorithm is similar to those independently derived by kumaresan et al in addition to its practical value the present formulation is used to interpret previous methods such as prony s pisarenko s and modifications thereof

on line versus off line learning from random examples general results i propose a general model of on line learning from random examples which when applied to a smooth realizable stochastic rule yields the same asymptotic generalization error rate as optimal batch algorithms the approach is based on an iterative gaussian approximation to the posterior gibbs distribution of rule parameters

simulation based optimization of markov reward processes this paper proposes a simulation based algorithm for optimizing the average reward in a finite state markov reward process that depends on a set of parameters as a special case the method applies to markov decision processes where optimization takes place within a parametrized set of policies the algorithm relies on the regenerative structure of finite state markov processes involves the simulation of a single sample path and can be implemented online a convergence result with probability 1 is provided

analysis of the asymptotic relative efficiency of the music algorithm an analytical performance evaluation of the errors of the direction of arrival estimates obtained by the music algorithm for uncorrelated sources is provided explicit asymptotic formulas are derived for the means and the covariance of the estimates the covariances are then compared to the cramer rao lower bound it is shown that for a single course the music algorithm is asymptotically efficient for multiple sources the algorithm is not efficient in general however it approaches asymptotic efficiency when the snrs signal to noise ratios of all sources tend to infinity it is illustrated by several test cases that the relative efficiency of the music algorithm is nearly one under a wide range of parameter variations the analytic performance evaluation thus confirms empirical evidence to the excellent performance of the music algorithm for narrowband signals

direction finding for diversely polarized signals using polynomial rooting a direction finding algorithm for diversely polarized arrays that is based on polynomial rooting is presented using polynomial rooting instead of search reduces significantly the computational requirements of the algorithm and enhances resolution the basic method is limited to linear uniformly spaced arrays however using an interpolation technique the algorithm is extended to a larger class of arrays the performance of the proposed algorithm is analyzed computer simulations are used to demonstrate the performance of the new technique and to verify the performance analysis

a unifying information theoretic framework for independent component analysis abstract we show that different theories recently proposed for independent component analysis ica lead to the same iterative learning algorithm for blind separation of mixed independent sources we review those theories and suggest that information theory can be used to unify several lines of research pearlmutter and parra 1 and cardoso 2 showed that the infomax approach of bell and sejnowski 3 and the maximum likelihood estimation approach are equivalent we show that negentropy maximization also has equivalent properties and therefore all three approaches yield the same learning rule for a fixed nonlinearity girolami and fyfe 4 have shown that the nonlinear principal component analysis pca algorithm of karhunen and joutsensalo 5 and oja 6 can also be viewed from information theoretic principles since it minimizes the sum of squares of the fourth order marginal cumulants and therefore approximately minimizes the mutual information 7 lambert 8 has proposed different bussgang cost functions for multichannel blind deconvolution we show how the bussgang property relates to the infomax principle finally we discuss convergence and stability as well as future research issues in blind source separation

information geometry of boltzmann machines a boltzmann machine is a network of stochastic neurons the set of all the boltzmann machines with a fixed topology forms a geometric manifold of high dimension where modifiable synaptic weights of connections play the role of a coordinate system to specify networks a learning trajectory for example is a curve in this manifold it is important to study the geometry of the neural manifold rather than the behavior of a single network in order to know the capabilities and limitations of neural networks of a fixed topology using the new theory of information geometry a natural invariant riemannian metric and a dual pair of affine connections on the boltzmann neural network manifold are established the meaning of geometrical structures is elucidated from the stochastic and the statistical point of view this leads to a natural modification of the boltzmann machine learning rule

learning dynamics on different timescales the special character of certain degrees of freedom in two layered neural networks is investigated for on line learning of realizable rules our analysis shows that the dynamics of these degrees of freedom can be put on a faster timescale than those remaining with the profit of speeding up the overall adaptation process this is shown for two groups of degrees of freedom second layer weights and bias weights for the former case our analysis provides a theoretical explanation of phenomenological findings the resulting learning algorithm is compared with natural gradient descent in order to check whether the proposed scaling can be naturally derived from that type of learning rule

convergence results for some temporal difference methods based on least squares we consider finite state markov decision processes and prove convergence and rate of convergence results for certain least squares policy evaluation algorithms of the type known as lspe lambda these are temporal difference methods for constructing a linear function approximation of the cost function of a stationary policy within the context of infinite horizon discounted and average cost dynamic programming we introduce an average cost method patterned after the known discounted cost method and we prove its convergence for a range of constant stepsize choices we also show that the convergence rate of both the discounted and the average cost methods is optimal within the class of temporal difference methods analysis and experiment indicate that our methods are substantially and often dramatically faster than td lambda as well as more reliable

adaptive online learning algorithms for blind separation maximum entropy and minimum mutual information there are two major approaches for blind separation maximum entropy me and minimum mutual information mmi both can be implemented by the stochastic gradient descent method for obtaining the demixing matrix the mi is the contrast function for blind separation the entropy is not to justify the me the relation between me and mmi is first elucidated by calculating the first derivative of the entropy and proving that the mean subtraction is necessary in applying the me and at the solution points determined by the mi the me will not update the demixing matrix in the directions of increasing the cross talking second the natural gradient instead of the ordinary gradient is introduced to obtain efficient algorithms because the parameter space is a riemannian space consisting of matrices the mutual information is calculated by applying the gram charlier expansion to approximate probability density functions of the outputs finally we propose an efficient learning algorithm that incorporates with an adaptive method of estimating the unknown cumulants it is shown by computer simulation that the convergence of the stochastic descent algorithms is improved by using the natural gradient and the adaptively estimated cumulants

fundamental limitations for doa and polarization estimation with applications in array signal processing using a generic electromagnetic model based on the spherical vector harmonics we provide the cramer rao lower bound crb for direction on arrival doa and polarization estimation with arbitrary multiport antennas by reciprocity the receiving multiport antenna is characterized in terms of the transmission matrix from incident voltage waves to multipole coefficients of the outgoing spherical vector waves explicit results of the crb for the gaussian case is given in terms of the transmission matrix the spherical vector harmonics and its spatial derivatives numerical examples are included to illustrate the results

local approach to kadec klee properties in symmetric function spaces abstract we prove several results concerning local approach to kadec klee properties with respect to global local convergence in measure in symmetric banach function spaces which may be of independent interest moreover we prove characterizations of these properties in the lorentz spaces finally we show applications of h g and h l points to the local best dominated approximation problems in banach lattices

analysis of natural gradient descent for multilayer neural networks natural gradient descent ngd is an on line algorithm for redefining the steepest descent direction an analysis of ngd for training a multilayer neural network is presented using statistical mechanics the performance can be significantly improved using ngd algorithm and can be used for both the transient and asymptotic stages of learning

natural gradient works efficiently in learning when a parameter space has a certain underlying structure the ordinary gradient of a function does not represent its steepest direction but the natural gradient does information geometry is used for calculating the natural gradients in the parameter space of perceptrons the space of matrices for blind source separation and the space of linear dynamical systems for blind source deconvolution the dynamical behavior of natural gradient online learning is analyzed and is proved to be fisher efficient implying that it has asymptotically the same performance as the optimal batch estimation of parameters this suggests that the plateau phenomenon which appears in the backpropagation learning algorithm of multilayer perceptrons might disappear or might not be so serious when the natural gradient is used an adaptive method of updating the learning rate is proposed and analyzed

natural actor critic algorithms we present four new reinforcement learning algorithms based on actor critic natural gradient and function approximation ideas and we provide their convergence proofs actor critic reinforcement learning methods are online approximations to policy iteration in which the value function parameters are estimated using temporal difference learning and the policy parameters are updated by stochastic gradient descent methods based on policy gradients in this way are of special interest because of their compatibility with function approximation methods which are needed to handle large or infinite state spaces the use of temporal difference learning in this way is of special interest because in many applications it dramatically reduces the variance of the gradient estimates the use of the natural gradient is of interest because it can produce better conditioned parameterizations and has been shown to further reduce variance in some cases our results extend prior two timescale convergence results for actor critic methods by konda and tsitsiklis by using temporal difference learning in the actor and by incorporating natural gradients our results extend prior empirical studies of natural actor critic methods by peters vijayakumar and schaal by providing the first convergence proofs and the first fully incremental algorithms

factorization of tent spaces and hankel operators it is shown that the factorization tpq tp t q for tent spaces proved by r r coifman et al 1985 j funct anal 62 304 335 for p q q 2 holds true for all 0 p q from this certain strong factorization theorems are derived for spaces hps of fractional derivatives of hp functions and more general triebel spaces in particular it is proved that hps hp bmoas applications considered include characterizations of symbols of bounded hankel operators h hp hqs complex interpolation of tent spaces and carleson measure theorems for derivatives of hp functions

duality for symmetric hardy spaces of noncommutative martingales we show the dual spaces of conditional hardy space and symmetric hardy space of noncommutative martingales we derive relationship between the symmetric hardy space of noncommutative martingales and its conditioned version

higher order boltzmann machines and entropy bounds we examine some aspects of the interface area between mathematical statistics and statistical physics relevant to the study of boltzmann machines the boltzmann machine learning algorithm is based on a variational principle gibbs lemma for relative entropy this fact suggests the possibility of a scheme of successive approximations here we consider successive approximations parametrized by the order of many body interactions among individual units we prove bounds on the gain in relative entropy in the crucial step of adding and estimating by hebb s rule a new parameter we address the problem of providing on the basis of local observations upper and lower bounds on the entropy while upper bounds are easily obtained by subadditivity lower bounds involve localization of hirschman bounds on a dual quantum system

on geometric structure of symmetric spaces abstract in this article we discuss local approach to strict k monotonicity and local uniform rotundity in symmetric spaces we prove several general results on local structure of symmetric spaces e showing relation between strict monotonicity and strict k monotonicity and the kadec klee property for global convergence in measure we also present the full criteria for points of upper k monotonicity in lorentz spaces p w for degenerated weight function w next we characterize local uniform rotundity in symmetric spaces e proving several correspondences between x e a point of local uniform rotundity and its decreasing rearrangement x and absolute value x finally we apply these results to find complete criteria for local uniform rotundity of lorentz spaces p w

pointwise products of some banach function spaces and factorization abstract the well known factorization theorem of lozanovski may be written in the form l 1 e e where means the pointwise product of banach ideal spaces a natural generalization of this problem would be the question when one can factorize f through e i e when f e m e f where m e f is the space of pointwise multipliers from e to f properties of m e f were investigated in our earlier paper 41 and here we collect and prove some properties of the construction e f the formulas for pointwise product of calderon lozanovski e spaces lorentz spaces and marcinkiewicz spaces are proved these results are then used to prove factorization theorems for such spaces finally it is proved in theorem 11 that under some natural assumptions a rearrangement invariant banach function space may be factorized through a marcinkiewicz space

maximum likelihood signal estimation for polarization sensitive arrays the authors consider the problem of separating and estimating the waveforms of superimposed signals received by a polarization sensitive array signal estimation is accomplished by a two step maximum likelihood procedure i the directions of arrival and polarization parameters of all the signals are estimated ii the estimated signal is obtained as a linear combination of the array outputs with weights which are computed from the estimated direction polarization parameters the objective of this study is to analyze the quality of the estimated signal in terms of the output signal to interference ratio sir and output signal to noise ratio snr closed form expressions are derived for the output sir and snr of a general diversely polarized array by evaluating these expressions for selected test cases it is shown that polarization sensitive arrays can provide significantly higher output sir and snr than uniformly polarized arrays the performance improvement is especially significant for closed spaced sources with sufficiently different polarization characteristics

equivariant adaptive source separation source separation consists of recovering a set of independent signals when only mixtures with unknown coefficients are observed this paper introduces a class of adaptive algorithms for source separation that implements an adaptive version of equivariant estimation and is henceforth called equivariant adaptive separation via independence easi the easi algorithms are based on the idea of serial updating this specific form of matrix updates systematically yields algorithms with a simple structure for both real and complex mixtures most importantly the performance of an easi algorithm does not depend on the mixing matrix in particular convergence rates stability conditions and interference rejection levels depend only on the normalized distributions of the source signals closed form expressions of these quantities are given via an asymptotic performance analysis the theme of equivariance is stressed throughout the paper the source separation problem has an underlying multiplicative structure the parameter space forms a matrix multiplicative group we explore the favorable consequences of this fact on implementation performance and optimization of easi algorithms

symmetrization factorization and arithmetic of quasi banach function spaces abstract we investigate relations between symmetrizations of quasi banach function spaces and constructions such as calderon lozanovski spaces pointwise product spaces and pointwise multipliers we show that under reasonable assumptions the symmetrization commutates with these operations we determine also the spaces of pointwise multipliers between lorentz spaces and cesaro spaces the methods that we develop may be regarded as an arithmetic of quasi banach function spaces and proofs of theorem 3 theorem 4 theorem 6 give a kind of tutorial for these methods finally the above results will be used in proofs of some factorization results

local structure of symmetrizations e with applications abstract we study the local structure of the separated point x in the space e with respect to properties of its nonincreasing rearrangement x in the space e in particular we get the characterizations of local structure in the lorentz and marcinkiewicz spaces it is worth to mention that the local structure of a separated point can be applicable to the local best dominated approximation problems in banach lattices

an actor critic algorithm with second order actor and critic actor critic algorithms solve dynamic decision making problems by optimizing a performance metric of interest over a user specified parametric class of policies they employ a combination of an actor making policy improvement steps and a critic computing policy improvement directions many existing algorithms use a steepest ascent method to improve the policy which is known to suffer from slow convergence for ill conditioned problems in this paper we first develop an estimate of the hessian matrix containing the second derivatives of the performance metric with respect to policy parameters using this estimate we introduce a new second order policy improvement method and couple it with a critic using a second order learning method we establish almost sure convergence of the new method to a neighborhood of a policy parameter stationary point we compare the new algorithm with some existing algorithms in two applications and demonstrate that it leads to significantly faster convergence

the relation between information theory and the differential geometry approach to statistics abstract cencov has shown that the riemannian metric on the probability simplex xi 1 defined by ds 2 dx i 2 x i has an invariance property under certain probabilistically natural mappings no other riemannian metric has the same property the geometry associated with this metric is shown to lead almost automatically to measures of divergence between probability distributions which are associated with kullback bhattacharyya and matusita certain vector fields are associated in a natural way with random variables the integral curves of these vector fields yield the maximum entropy or minimum divergence estimates of probabilities some other consequences of this geometric view are also explored

integral representation of product factorable bilinear operators and summability of bilinear maps on c k spaces abstract we present a constructive technique to represent classes of bilinear operators that allow a factorization through a bilinear product providing a general version of the well known characterization of integral bilinear forms as elements of the dual of an injective tensor product we show that this general method fits with several known situations coming from different contexts harmonic analysis c algebras c k spaces operator theory polynomials providing a unified approach to the integral representation of a broad class of bilinear operators some examples and applications are also shown regarding for example operator spaces and summability properties of bilinear maps

an information maximization approach to blind separation and blind deconvolution we derive a new self organizing learning algorithm that maximizes the information transferred in a network of nonlinear units the algorithm does not assume any knowledge of the input distributions and is defined here for the zero noise limit under these conditions information maximization has extra properties not found in the linear case linsker 1989 the nonlinearities in the transfer function are able to pick up higher order moments of the input distributions and perform something akin to true redundancy reduction between units in the output representation this enables the network to separate statistically independent components in the inputs a higher order generalization of principal components analysis we apply the network to the source separation or cocktail party problem successfully separating unknown mixtures of up to 10 speakers we also show that a variant on the network architecture is able to perform blind deconvolution cancellation of unknown echoes and reverberation in a speech signal finally we derive dependencies of information transfer on time delays we suggest that information maximization provides a unifying framework for problems in blind signal processing

statistical theory of learning curves under entropic loss criterion the present paper elucidates a universal property of learning curves which shows how the generalization error training error and the complexity of the underlying stochastic machine are related and how the behavior of a stochastic machine is improved as the number of training examples increases the error is measured by the entropic loss it is proved that the generalization error converges to h0 the entropy of the conditional distribution of the true machine as h0 m 2t while the training error converges as h0 m 2t where t is the number of examples and m shows the complexity of the network when the model is faithful implying that the true machine is in the model m is reduced to m the number of modifiable parameters this is a universal law because it holds for any regular machine irrespective of its structure under the maximum likelihood estimator similar relations are obtained for the bayes and gibbs learning algorithms these learning curves show the relation among the accuracy of learning the complexity of a model and the number of training examples

superefficiency in blind source separation blind source separation is the problem of extracting independent signals from their mixtures without knowing the mixing coefficients nor the probability distributions of source signals and may be applied to eeg and meg imaging of the brain it is already known that certain algorithms work well for the extraction of independent components the present paper is concerned with superefficiency of these based on the statistical and dynamical analysis in a statistical estimation using t examples the covariance of any two extracted independent signals converges to 0 of the order of 1 t on line dynamics shows that the covariance is of the order of spl eta when the learning rate spl eta is fixed to a small constant in contrast with the above general properties a surprising superefficiency holds in blind source separation under certain conditions where superefficiency implies that covariance decreases in the order of 1 t sup 2 or of spl eta sup 2 the paper uses the natural gradient learning algorithm and method of estimating functions to obtain superefficient procedures for both batch estimation and on line learning a standardized estimating function is introduced to this end superefficiency does not imply that the error variances of the extracted signals decrease in the order of 1 t sup 2 or spl eta sup 2 but implies that their covariances and independencies do