estimating the errors on measured entropy and mutual information abstract information entropy and the related quantity mutual information are used extensively as measures of complexity and to identify nonlinearity in dynamical systems expressions for the probability distribution of entropies and mutual informations calculated from finite amounts of data exist in the literature but the expressions have seldom been used in the field of nonlinear dynamics in this paper formulae for estimating the errors on observed information entropies and mutual informations are derived using the standard error analysis familiar to physicists their validity is demonstrated by numerical experiment for illustration the formulae are then used to evaluate the errors on the time lagged mutual information of the logistic map

nested newton s method for ica and post factor analysis two distinct topics are dealt with first a new method for independent component analysis ica has been constructed that exploits the invariance of criteria under component wise scaling which is intrinsic to ica this practical and simple ica method is called the nested newton s method when the number of the channel of observation is less than a certain level factor analysis fa is ineffective bound for fa the target of this paper is these cases three of many concrete advantages of the nested newton s method are addressed i it is robust against gaussian noise and outperforms existing methods such as jade and fast ica especially under gaussian noise conditions ii it is highly stable globally iii each step resolves itself into two dimensional 2 d matrix problems there is thus no need to deal with gigantic matrices which means that fewer computational resources are required second a method called post factor analysis post fa is described that is aimed to be useful as post processing for ica although it is functionally similar to conventional fa post fa is a completely new method and is more powerful than conventional fa in compensation for its stronger assumption that there are mutually independent sources behind observations by fully making use of this assumption post fa is capable of estimating the noise variance beyond the known limit for fa furthermore it improves the accuracy of ica to a considerable extent any ica algorithm without prewhitening pre wh or pre factor analysis pre fa can be used for preprocessing although the nested method is a good candidate

blind source separation by nonstationarity of variance a cumulant based approach blind separation of source signals usually relies either on the nongaussianity of the signals or on their linear autocorrelations a third approach was introduced by matsuoka et al 1995 who showed that source separation can be performed by using the nonstationarity of the signals in particular the nonstationarity of their variances in this paper we show how to interpret the nonstationarity due to a smoothly changing variance in terms of higher order cross cumulants this is based on the time correlation of the squares energies of the signals and leads to a simple optimization criterion using this criterion we construct a fixed point algorithm that is computationally very efficient

variational iteration method for autonomous ordinary differential systems in this paper a new iteration technique is proposed to solve autonomous ordinary differential systems in this method general lagrange multipliers are introduced to construct correction functionals for the systems the multipliers in the functionals can be identified by the variational theory the initial approximations can be freely chosen with possible unknown constants which can be determined by imposing the boundary initial conditions some examples are given the results reveal that the method is very effective and convenient

monitoring the formation of kernel based topographic maps in a hybrid som kmer model a new lattice disentangling monitoring algorithm for a hybrid self organizing map kernel based maximum entropy learning rule som kmer model is proposed it aims to overcome topological defects owing to a rapid decrease of the neighborhood range over the finite running time in topographic map formation the empirical results demonstrate that the proposed approach is able to accelerate the formation of a topographic map and at the same time to simplify the monitoring procedure

approximation of scattered data using smooth grid functions we present a three stage scheme for constructing smooth grid functions approximating data defined over scattered point sets in rs the scheme is useful for approximating large scattered data sets and is particularly successful when the data points are unevenly distributed the paper includes several examples of grid surface construction over the plane

extended gaussianization method for blind separation of post nonlinear mixtures the linear mixture model has been investigated in most articles tackling the problem of blind source separation recently several articles have addressed a more complex model blind source separation bss of postnonlinear pnl mixtures these mixtures are assumed to be generated by applying an unknown invertible nonlinear distortion to linear instantaneous mixtures of some independent sources the gaussianization technique for bss of pnl mixtures emerged based on the assumption that the distribution of the linear mixture of independent sources is gaussian in this letter we review the gaussianization method and then extend it to apply to pnl mixture in which the linear mixture is close to gaussian our proposed method approximates the linear mixture using the cornishfisher expansion we choose the mutual information as the independence measurement to develop a learning algorithm to separate pnl mixtures this method provides better applicability and accuracy we then discuss the sufficient condition for the method to be valid the characteristics of the nonlinearity do not affect the performance of this method with only a few parameters to tune our algorithm has a comparatively low computation finally we present experiments to illustrate the efficiency of our method

kernel based equiprobabilistic topographic map formation we introduce a new unsupervised competitive learning rule the kernel based maximum entropy learning rule kmer which performs equiprobabilistic topographic map formation in regular fixed topology lattices for use with nonparametric density estimation as well as nonparametric regression analysis the receptive fields of the formal neurons are overlapping radially symmetric kernels compatible with radial basis functions rbfs but unlike other learning schemes the radii of these kernels do not have to be chosen in an ad hoc manner the radii are adapted to the local input density together with the weight vectors that define the kernel centers so as to produce maps of which the neurons have an equal probability to be active equiprobabilistic maps both an online and a batch version of the learning rule are introduced which are applied to nonparametric density estimation and regression respectively the application envisaged is blind source separation bss from nonlinear noisy mixtures

principal component analysis and blind separation of sources for optical imaging of intrinsic signals the analysis of data sets from optical imaging of intrinsic signals requires the separation of signals which accurately reflect stimulated neuronal activity mapping signal from signals related to background activity here we show that blind separation of sources by extended spatial decorrelation esd is a powerful method for the extraction of the mapping signal from the total recorded signal esd is based on the assumptions i that each signal component varies smoothly across space and ii that every component has zero cross correlation functions with the other components in contrast to the standard analysis of optical imaging data the proposed method i is applicable to nonorthogonal stimulus conditions ii can remove the global signal blood vessel patterns and movement artifacts iii works without ad hoc assumptions about the data structure in the frequency domain and iv provides a confidence measure for the signals z score we first demonstrate on orientation maps from cat and ferret visual cortex that principal component analysis which acts as a preprocessing step to esd can already remove global signals from image stacks as long as data stacks for at least two not necessarily orthogonal stimulus conditions are available we then show that the full esd analysis can further reduce global signal components and finally concentrate the mapping signal within a single component both for differential image stacks and for image stacks recorded during presentation of a single stimulus

robust independent component analysis by iterative maximization of the kurtosis contrast with algebraic optimal step size independent component analysis ica aims at decomposing an observed random vector into statistically independent variables deflation based implementations such as the popular one unit fastica algorithm and its variants extract the independent components one after another a novel method for deflationary ica referred to as robustica is put forward in this paper this simple technique consists of performing exact line search optimization of the kurtosis contrast function the step size leading to the global maximum of the contrast along the search direction is found among the roots of a fourth degree polynomial this polynomial rooting can be performed algebraically and thus at low cost at each iteration among other practical benefits robustica can avoid prewhitening and deals with real and complex valued mixtures of possibly noncircular sources alike the absence of prewhitening improves asymptotic performance the algorithm is robust to local extrema and shows a very high convergence speed in terms of the computational cost required to reach a given source extraction quality particularly for short data records these features are demonstrated by a comparative numerical analysis on synthetic data robustica s capabilities in processing real world data involving noncircular complex strongly super gaussian sources are illustrated by the biomedical problem of atrial activity aa extraction in atrial fibrillation af electrocardiograms ecgs where it outperforms an alternative ica based technique

least squares for second order elliptic problems abstract in this paper we introduce and analyze two least squares methods for second order elliptic differential equations with mixed boundary conditions these methods extend to problems which involve oblique derivative boundary conditions as well as nonsymmetric and indefinite problems as long as the original problem has a unique solution with the methods to be developed neumann and oblique boundary conditions are imposed weakly and thus avoid compatibility conditions on the finite element subspaces the resulting least squares approximations are unconditionally stable no conditions on the step size h and will be shown to converge at an optimal rate the first least squares method involves a discrete computable h 1 norm of the residual and stabilization terms consisting of the jumps at the interelement boundaries and a weighted elementwise l 2 norm of the residual over the finite elements this method is developed without the introduction of additional problem variables the second method involves the use of the flux as an additional unknown although this method is similar to the least squares method for first order systems introduced in 7 it differs in that discontinuous finite elements are allowed it is also more general in that it extends to the oblique boundary problem

blind separation of parametric nonlinear mixtures of possibly autocorrelated and non stationary sources in this paper we present a new method formulated in a maximum likelihood framework for blindly separating nonlinear mixtures of statistically independent signals our method exploits on the one hand the knowledge of the parametric model of the mixing transformation with unknown parameter values and on the other hand the possible structure of source signals i e their autocorrelation and or nonstationarity one of the main advantages of the proposed method is that it can be implemented even if the analytical expression of the inverse model is unknown the method is first addressed in a general configuration then detailed for two special cases i e a simple bijective toy model and a linear quadratic model the study of the toy model is interesting because of its simplicity and its global bijectivity which allows us to focus our efforts on parameter estimation the linear quadratic model is chosen due to its capacity to describe real world mixing phenomena simulation results using the toy model and using a subclass of the linear quadratic model i e the bilinear model show that taking into account the nonlinearity of the mixing transformations and the structure of signals considerably improves separation performance

separation theorem for independent subspace analysis and its consequences independent component analysis ica the theory of mixed independent non gaussian sources has a central role in signal processing computer vision and pattern recognition one of the most fundamental conjectures of this research field is that independent subspace analysis isa the extension of the ica problem where groups of sources are independent can be solved by traditional ica followed by grouping the ica components the conjecture called isa separation principle i has been rigorously proven for some distribution types recently ii forms the basis of the state of the art isa solvers iii enables one to estimate the unknown number and the dimensions of the sources efficiently and iv can be extended to generalizations of the isa task such as different linear controlled post nonlinear complex valued partially observed problems as well as to problems dealing with nonparametric source dynamics here we shall review the advances on this field

variational iteration method new development and applications variational iteration method has been favourably applied to various kinds of nonlinear problems the main property of the method is in its flexibility and ability to solve nonlinear equations accurately and conveniently in this paper recent trends and developments in the use of the method are reviewed major applications to nonlinear wave equation nonlinear fractional differential equations nonlinear oscillations and nonlinear problems arising in various engineering applications are surveyed the confluence of modern mathematics and symbol computation has posed a challenge to developing technologies capable of handling strongly nonlinear equations which cannot be successfully dealt with by classical methods variational iteration method is uniquely qualified to address this challenge the flexibility and adaptation provided by the method have made the method a strong candidate for approximate analytical solutions this paper outlines the basic conceptual framework of variational iteration technique with application to nonlinear problems both achievements and limitations are discussed with direct reference to approximate solutions for nonlinear equations a new iteration formulation is suggested to overcome the shortcoming a very useful formulation for determining approximately the period of a nonlinear oscillator is suggested examples are given to illustrate the solution procedure

optimal alternating direction implicit parameters for nonsymmetric systems of linear equations the determination of optimal parameters for the well known alternating direction implicit adi method leads to a minimization problem for rational functions of given degree l on compact sets e and f that contain the eigenvalues of the matrices involved if e and f are real intervals the solution of this problem was given by zolotarev more than 100 years ago and by wachspress in the 1960s however in the complex case e g if the linear system arises from the discretization of non self adjoint elliptic differential equations little is known about this problem in this paper two approaches to the zolotarev problem in the complex plane are considered first generalized leja points are considered as an example for rational functions which are asymptotically minimal in addition for rectangular eigenvalue domains e and f the exact minimal solutions for l 2 are constructed and it is shown by numerical examples that the results obtained using the optimal parameters of degree 2 are comparable to tho

faithful representation of separable distributions a geometric approach to data representation incorporating information theoretic ideas is presented the task of finding a faithful representation where the input distribution is evenly partitioned into regions of equal mass is addressed for input consisting of mixtures of statistically independent sources we treat independent component analysis ica as a computational geometry problem first we consider the separation of sources with sharply peaked distribution functions where the ica problem becomes that of finding high density directions in the input distribution second we consider the more general problem for arbitrary input distributions where ica is transformed into the task of finding an aligned equipartition by modifying the kohonen self organized feature maps we arrive at neural networks with local interactions that optimize coding while simultaneously performing source separation the local nature of our approach results in networks with nonlinear ica capabilities

bayes estimators of generalized entropies the order q tsallis and renyi entropy receive broad applications in the statistical analysis of complex phenomena a generic problem arises however when these entropies need to be estimated from observed data the finite size of data sets can lead to serious systematic and statistical errors in numerical estimates in this paper we focus upon the problem of estimating generalized entropies from finite samples and derive the bayes estimator of the order q tsallis entropy including the order 1 i e the shannon entropy under the assumption of a uniform prior probability density the bayes estimator yields in general the smallest mean quadratic deviation from the true parameter as compared with any other estimator exploiting the functional relationship between and we use the bayes estimator of to estimate the renyi entropy we compare these novel estimators with the frequency count estimators for and we find by numerical simulations that the bayes estimator reduces statistical errors of order q entropy estimates for bernoulli as well as for higher order markov processes derived from the complete genome of the prokaryote haemophilus influenzae

controlling the magnification factor of self organizing feature maps the magnification exponents occurring in adaptive map formation algorithms like kohonen s self organizing feature map deviate for the information theoretically optimal value 1 as well as from the values that optimize e g the mean square distortion error 1 3 for one dimensional maps at the same time models for categorical perception such as the perceptual magnet effect which are based on topographic maps require negative magnification exponents 0 we present an extension of the self organizing feature map algorithm which utilizes adaptive local learning step sizes to actually control the magnification properties of the map by change of a single parameter maps with optimal information transfer with various minimal reconstruction errors or with an inverted magnification can be generated analytic results on this new algorithm are complemented by numerical simulations

an optimization approach to signal extraction from noisy multivariate data we consider a problem of blind signal extraction from noisy multivariate data in which each datum represents a system s response observed under a particular experimental condition our prototype example is multipixel functional images of brain activity in response to a set of prescribed experimental stimuli we present a novel multivariate analysis technique which identifies the different activity patterns signals that are attributable to specific experimental conditions without a priori knowledge about the signal or the noise characteristics the extracted signals which we term the generalized indicator functions are optimal in the sense that they maximize a weighted difference between the signal variance and the noise variance with an appropriate choice of the weighting parameter the method returns a set of images whose signal to noise ratios satisfy some user defined level of significance we demonstrate the performance of our method in optical intrinsic signal imaging of cat cortical area 17 we find that the method performs effectively and robustly in all tested data which include both real experimental data and numerically simulated data the method of generalized indicator functions is related to canonical variate analysis a multivariate analysis technique that directly solves for the maxima of the signal to noise ratio but important theoretical and practical differences exist which can make our method more appropriate in certain situations

ica in signals with multiplicative noise independent component analysis ica has been shown in the last few years to be a very useful tool in blind separation of sources and feature extraction however at least in its simpler form its utility is reduced to the case that the outputs are linear mixtures of independent sources this excludes signals with multiplicative noise in this paper ica is extended to this situation in order to do this the special structure that appears in this new model is first studied and then the multiplicative ica method is designed such that it uses this structure to find the mixture of the sources in the noisy environment the local and global convergence properties of the method are also studied and its performance compared with standard ica methods

misep method for postnonlinear blind source separation in this letter a standard postnonlinear blind source separation algorithm is proposed based on the misep method which is widely used in linear and nonlinear independent component analysis to best suit a wide class of postnonlinear mixtures we adapt the misep method to incorporate a priori information of the mixtures in particular a group of three layered perceptrons and a linear network are used as the unmixing system to separate sources in the postnonlinear mixtures and another group of three layered perceptron is used as the auxiliary network the learning algorithm for the unmixing system is then obtained by maximizing the output entropy of the auxiliary network the proposed method is applied to postnonlinear blind source separation of both simulation signals and real speech signals and the experimental results demonstrate its effectiveness and efficiency in comparison with existing methods

numerical solution of a biological population model using he s variational iteration method this paper presents numerical solution of a degenerate parabolic equation arising in the spatial diffusion of biological populations the variational iteration method and adomian decomposition method are used for solving this equation and then numerical results are compared with each other showing that the variational iteration method leads to more accurate results furthermore variational iteration method overcomes the difficulty arising in calculating the adomian s polynomials which is an important advantage over the adomian decomposition method

kernel based nonlinear blind source separation we propose ktdsep a kernel based algorithm for nonlinear blind source separation bss it combines complementary research fields kernel feature spaces and bss using temporal information this yields an efficient algorithm for nonlinear bss with invertible nonlinearity key assumptions are that the kernel feature space is chosen rich enough to approximate the nonlinearity and that signals of interest contain temporal information both assumptions are fulfilled for a wide set of real world applications the algorithm works as follows first the data are implicitly mapped to a high possibly infinite dimensional kernel feature space in practice however the data form a smaller submanifold in feature space even smaller than the number of training data points a fact that has already been used by for example reduced set techniques for support vector machines we propose to adapt to this effective dimension as a preprocessing step and to construct an orthonormal basis of this submanifold the latter dimension reduction step is essential for making the subsequent application of bss methods computationally and numerically tractable in the reduced space we use a bss algorithm that is based on second order temporal decorrelation finally we propose a selection procedure to obtain the original sources from the extracted nonlinear components automatically experiments demonstrate the excellent performance and efficiency of our ktdsep algorithm for several problems of nonlinear bss and for more than two sources

nonlinear component analysis as a kernel eigenvalue problem a new method for performing a nonlinear form of principal component analysis is proposed by the use of integral operator kernel functions one can efficiently compute principal components in high dimensional feature spaces related to input space by some nonlinear map for instance the space of all possible five pixel products in 16 16 images we give the derivation of the method and present experimental results on polynomial feature extraction for pattern recognition

assessing causality from multivariate time series in this work we propose a general nonparametric test of causality for weakly dependent time series more precisely we study the problem of attribution i e the proper comparison of the relative influence that two or more external dynamics trigger on a given system of interest we illustrate the possible applications of the proposed methodology in very different fields like physiology and climate science

a unifying information theoretic framework for independent component analysis abstract we show that different theories recently proposed for independent component analysis ica lead to the same iterative learning algorithm for blind separation of mixed independent sources we review those theories and suggest that information theory can be used to unify several lines of research pearlmutter and parra 1 and cardoso 2 showed that the infomax approach of bell and sejnowski 3 and the maximum likelihood estimation approach are equivalent we show that negentropy maximization also has equivalent properties and therefore all three approaches yield the same learning rule for a fixed nonlinearity girolami and fyfe 4 have shown that the nonlinear principal component analysis pca algorithm of karhunen and joutsensalo 5 and oja 6 can also be viewed from information theoretic principles since it minimizes the sum of squares of the fourth order marginal cumulants and therefore approximately minimizes the mutual information 7 lambert 8 has proposed different bussgang cost functions for multichannel blind deconvolution we show how the bussgang property relates to the infomax principle finally we discuss convergence and stability as well as future research issues in blind source separation

a generalized contrast function and stability analysis for overdetermined blind separation of instantaneous mixtures in this letter the problem of blind separation of n independent sources from their m linear instantaneous mixtures is considered first a generalized contrast function is defined as a valuable extension of the existing classical and nonsymmetrical contrast functions it is applicable to the overdetermined blind separation m n with an unknown number of sources because not only independent components but also redundant ones are allowed in the outputs of a separation system second a natural gradient learning algorithm developed primarily for the complete case m n is shown to work as well with an n m or m m separating matrix for each optimizes a certain mutual information contrast function finally we present stability analysis for a newly proposed generalized orthogonal natural gradient algorithm which can perform the overdetermined blind separation when n is unknown obtaining an expectable result that its local stability conditions are slightly stricter than those of the conventional natural gradient algorithm using an invertible mixing matrix m n

a fast fixed point algorithm for independent component analysis we introduce a novel fast algorithm for independent component analysis which can be used for blind source separation and feature extraction we show how a neural network learning rule can be transformed into a fixedpoint iteration which provides an algorithm that is very simple does not depend on any user defined parameters and is fast to converge to the most accurate solution allowed by the data the algorithm finds one at a time all nongaussian independent components regardless of their probability distributions the computations can be performed in either batch mode or a semiadaptive manner the convergence of the algorithm is rigorously proved and the convergence speed is shown to be cubic some comparisons to gradient based algorithms are made showing that the new algorithm is usually 10 to 100 times faster sometimes giving the solution in just a few iterations

statistical dynamics of on line independent component analysis the learning dynamics of on line independent component analysis is analysed in the limit of large data dimension we study a simple hebbian learning algorithm that can be used to separate out a small number of non gaussian components from a high dimensional data set the de mixing matrix parameters are confined to a stiefel manifold of tall orthogonal matrices and we introduce a natural gradient variant of the algorithm which is appropriate to learning on this manifold for large input dimension the parameter trajectory of both algorithms passes through a sequence of unstable fixed points each described by a diffusion process in a polynomial potential choosing the learning rate too large increases the escape time from each of these fixed points effectively trapping the learning in a sub optimal state in order to avoid these trapping states a very low learning rate must be chosen during the learning transient resulting in learning time scales of o n2 or o n3 iterations where n is the data dimension escape from each sub optimal state results in a sequence of symmetry breaking events as the algorithm learns each source in turn this is in marked contrast to the learning dynamics displayed by related on line learning algorithms for multilayer neural networks and principal component analysis although the natural gradient variant of the algorithm has nice asymptotic convergence properties it has an equivalent transient dynamics to the standard hebbian algorithm

generalized redundancies for time series analysis abstract extensions to various information theoretic quantities such as entropy redundancy and mutual information are discussed in the context of their role in nonlinear time series analysis we also discuss linearized versions of these quantities and their use as benchmarks in tests for nonlinearity many of these quantities can be expressed in terms of the generalized correlation integral and this expression permits us to more clearly exhibit the relationships of these quantities to each other and to other commonly used nonlinear statistics such as the bds and green savit statistics further numerical estimation of these quantities is found to be more accurate and more efficient when the the correlation integral is employed in the computation finally we consider several local versions of these quantities including a local kolmogorov sinai entropy which gives an estimate of variability of the short term predictability

box spline interpolation a computational study abstract we present a computational study of box spline interpolation in two space dimensions we are in particular concerned with the problem of computing the box spline coefficients in the presence of large data sets the issues of convergence and regularity of the interpolants are also considered from a computational point of view

a generic framework for blind source separation in structured nonlinear models this paper is concerned with blind source separation in nonlinear models special attention is paid to separability issues results show that separation is impossible in the general case however for specific nonlinear models the problem becomes tractable a generic set of parametric nonlinear mixtures is considered this set has the lie group structure a group structure with continuous binary operation in the parameter set a definition of a relative gradient is given and is used to design both batch and stochastic algorithms for the latter it is shown how a proper use of the relative gradient leads to equivariant adaptive algorithms

super exponential methods for blind deconvolution a class of iterative methods for solving the blind deconvolution problem i e for recovering the input of an unknown possibly nonminimum phase linear system by observation of its output is presented these methods are universal do not require prior knowledge of the input distribution are computationally efficient and statistically stable and converge to the desired solution regardless of initialization at a very fast rate the effects of finite length of the data finite length of the equalizer and additive noise in the system on the attainable performance intersymbol interference are analyzed it is shown that in many cases of practical interest the performance of the proposed methods is far superior to linear prediction methods even for minimum phase systems recursive and sequential algorithms are also developed which allow real time implementation and adaptive equalization of time varying systems

blind separation of post nonlinear mixtures using linearizing transformations and temporal decorrelation we propose two methods that reduce the post nonlinear blind source separation problem pnl bss to a linear bss problem the first method is based on the concept of maximal correlation we apply the alternating conditional expectation ace algorithm a powerful technique from non parametric statistics to approximately invert the componentwise non linear functions the second method is a gaussianizing transformation which is motivated by the fact that linearly mixed signals before nonlinear transformation are approximately gaussian distributed this heuristic but simple and efficient procedure works as good as the ace method using the framework provided by ace convergence can be proven the optimal transformations obtained by ace coincide with the sought after inverse functions of the nonlinearities after equalizing the nonlinearities temporal decorrelation separation tdsep allows us to recover the source signals numerical simulations testing ace td and gauss td on realistic examples are performed with excellent results

measuring statistical dependences in a time series we propose two methods to measure all linear and nonlinear statistical dependences in a stationary time series presuming ergodicity the measures can be obtained from efficient numerical algorithms

natural discriminant analysis using interactive potts models natural discriminant analysis based on interactive potts models is developed in this work a generative model composed of piece wise multivariate gaussian distributions is used to characterize the input space exploring the embedded clustering and mixing structures and developing proper internal representations of input parameters the maximization of a log likelihood function measuring the fitness of all input parameters to the generative model and the minimization of a design cost summing up square errors between posterior outputs and desired outputs constitutes a mathematical framework for discriminant analysis we apply a hybrid of the mean field annealing and the gradient descent methods to the optimization of this framework and obtain multiple sets of interactive dynamics which realize coupled potts models for discriminant analysis the new learning process is a whole process of component analysis clustering analysis and labeling analysis its major improvement compared to the radial basis function and the support vector machine is described by using some artificial examples and a real world application to breast cancer diagnosis

how to generate ordered maps by maximizing the mutual information between input and output signals a learning rule that performs gradient ascent in the average mutual information between input and an output signal is derived for a system having feedforward and lateral interactions several processes emerge as components of this learning rule hebb like modification and cooperation and competition among processing nodes topographic map formation is demonstrated using the learning rule an analytic expression relating the average mutual information to the response properties of nodes and their geometric arrangement is derived in certain cases this yields a relation between the local map magnification factor and the probability distribution in the input space the results provide new links between unsupervised learning and information theoretic optimization in a system whose properties are biologically motivated

towards a theory of early visual processing we propose a theory of the early processing in the mammalian visual pathway the theory is formulated in the language of information theory and hypothesizes that the goal of this processing is to recode in order to reduce a generalized redundancy subject to a constraint that specifies the amount of average information preserved in the limit of no noise this theory becomes equivalent to barlow s redundancy reduction hypothesis but it leads to very different computational strategies when noise is present a tractable approach for finding the optimal encoding is to solve the problem in successive stages where at each stage the optimization is performed within a restricted class of transfer functions we explicitly find the solution for the class of encodings to which the parvocellular retinal processing belongs namely linear and nondivergent transformations the solution shows agreement with the experimentally observed transfer functions at all levels of signal to noise

computational geometry an introduction from the reviews this book offers a coherent treatment at the graduate textbook level of the field that has come to be known in the last decade or so as computational geometry the book is well organized and lucidly written a timely contribution by two founders of the field it clearly demonstrates that computational geometry in the plane is now a fairly well understood branch of computer science and mathematics it also points the way to the solution of the more challenging problems in dimensions higher than two

application of he s variational iteration method for solving the cauchy reaction diffusion problem in this paper the solution of cauchy reaction diffusion problem is presented by means of variational iteration method reaction diffusion equations have special importance in engineering and sciences and constitute a good model for many systems in various fields application of variational iteration technique to this problem shows the rapid convergence of the sequence constructed by this method to the exact solution moreover this technique does not require any discretization linearization or small perturbations and therefore it reduces significantly the numerical computations

two methods for solving integral equations the objective of this work is to introduce two efficient strategies that produce the accurate solutions to linear and nonlinear volterra and fredholm integral equations of the second kind for illustration several linear nonlinear and integro differential examples have been tested and the results showed improvements over existing techniques

blind separation of mixture of independent sources through a quasi maximum likelihood approach we propose two methods for separating mixture of independent sources without any precise knowledge of their probability distribution they are obtained by considering a maximum likelihood ml solution corresponding to some given distributions of the sources and relaxing this assumption afterward the first method is specially adapted to temporally independent non gaussian sources and is based on the use of nonlinear separating functions the second method is specially adapted to correlated sources with distinct spectra and is based on the use of linear separating filters a theoretical analysis of the performance of the methods has been made a simple procedure for optimally choosing the separating functions is proposed further in the second method a simple implementation based on the simultaneous diagonalization of two symmetric matrices is provided finally some numerical and simulation results are given illustrating the performance of the method and the good agreement between the experiments and the theory

a blind source separation technique using second order statistics separation of sources consists of recovering a set of signals of which only instantaneous linear mixtures are observed in many situations no a priori information on the mixing matrix is available the linear mixture should be blindly processed this typically occurs in narrowband array processing applications when the array manifold is unknown or distorted this paper introduces a new source separation technique exploiting the time coherence of the source signals in contrast with other previously reported techniques the proposed approach relies only on stationary second order statistics that are based on a joint diagonalization of a set of covariance matrices asymptotic performance analysis of this method is carried out some numerical simulations are provided to illustrate the effectiveness of the proposed method

a reliable treatment for mixed volterra fredholm integral equations the main goal of this paper is to demonstrate the use of the modified decomposition method for mixed nonlinear volterra fredholm integral equations the modified method combined with the noise terms phenomena may provide the exact solution by using two iterations only two numerical illustrations are given to show the pertinent features of the technique

variational iteration method some recent results and new interpretations this paper is an elementary introduction to the concepts of variational iteration method first the main concepts in variational iteration method such as general lagrange multiplier restricted variation correction functional are explained heuristically subsequently the solution procedure is systematically addressed in particular for nonlinear oscillators particular attention is paid throughout the paper to give an intuitive grasp for the method the main motivation is to put things together in a convenient form for later reference and systematic use

the convergence of he s variational iteration method for solving integral equations in this paper several integral equations are solved by he s variational iteration method in general case then we consider the convergence of he s variational iteration method for solving integral equations

detection and characterization of changes of the correlation structure in multivariate time series we propose a method based on the equal time correlation matrix as a sensitive detector for phase shape correlations in multivariate data sets the key point of the method is that changes of the degree of synchronization between time series provoke level repulsions between eigenstates at both edges of the spectrum of the correlation matrix consequently detailed information about the correlation structure of the multivariate data set is imprinted into the dynamics of the eigenvalues and into the structure of the corresponding eigenvectors the performance of the technique is demonstrated by application to n f tori autoregressive models and coupled chaotic systems the high sensitivity the comparatively small computational effort and the excellent time resolution of the method recommend it for application to the analysis of complex spatially extended nonstationary systems

infinite hidden markov models for unusual event detection in video we address the problem of unusual event detection in a video sequence invariant subspace analysis isa is used to extract features from the video and the time evolving properties of these features are modeled via an infinite hidden markov model ihmm which is trained using ldquonormalrdquo ldquotypicalrdquo video the ihmm retains a full posterior density function on all model parameters including the number of underlying hmm states anomalies unusual events are detected subsequently if a low likelihood is observed when associated sequential features are submitted to the trained ihmm a hierarchical dirichlet process framework is employed in the formulation of the ihmm the evaluation of posterior distributions for the ihmm is achieved in two ways via markov chain monte carlo and using a variational bayes formulation comparisons are made to modeling based on conventional maximum likelihood based hmms as well as to dirichlet process based gaussian mixture models

an introduction to kernel based learning algorithms this paper provides an introduction to support vector machines kernel fisher discriminant analysis and kernel principal component analysis as examples for successful kernel based learning methods we first give a short background about vapnik chervonenkis theory and kernel feature spaces and then proceed to kernel based learning in supervised and unsupervised scenarios including practical and algorithmic considerations we illustrate the usefulness of kernel algorithms by discussing applications such as optical character recognition and dna analysis

semiparametrically efficient inference based on signed ranks in symmetric independent component models we consider semiparametric location scatter models for which the p variate observation is obtained as x lambda z mu where mu is a p vector lambda is a full rank p p matrix and the unobserved random p vector z has marginals that are centered and mutually independent but are otherwise unspecified as in blind source separation and independent component analysis ica the parameter of interest throughout the paper is lambda on the basis of n i i d copies of x we develop under a symmetry assumption on z signed rank one sample testing and estimation procedures for lambda we exploit the uniform local and asymptotic normality ulan of the model to define signed rank procedures that are semiparametrically efficient under correctly specified densities yet as usual in rank based inference the proposed procedures remain valid correct asymptotic size under the null for hypothesis testing and root n consistency for point estimation under a very broad range of densities we derive the asymptotic properties of the proposed procedures and investigate their finite sample behavior through simulations

blind source separation by sparse decomposition in a signal dictionary the blind source separation problem is to extract the underlying source signals from a set of linear mixtures where the mixing matrix is unknown this situation is common in acoustics radio medical signal and image processing hyperspectral imaging and other areas we suggest a two stage separation process a priori selection of a possibly overcomplete signal dictionary for instance a wavelet frame or a learned dictionary in which the sources are assumed to be sparsely representable followed by unmixing the sources by exploiting the their sparse representability we consider the general case of more sources than mixtures but also derive a more efficient algorithm in the case of a nonovercomplete dictionary and an equal numbers of sources and mixtures experiments with artificial signals and musical sounds demonstrate significantly better separation than other known techniques

independent factor analysis we introduce the independent factor analysis ifa method for recovering independent hidden sources from their observed mixtures ifa generalizes and unifies ordinary factor analysis fa principal component analysis pca and independent component analysis ica and can handle not only square noiseless mixing but also the general case where the number of mixtures differs from the number of sources and the data are noisy ifa is a two step procedure in the first step the source densities mixing matrix and noise covariance are estimated from the observed data by maximum likelihood for this purpose we present an expectation maximization em algorithm which performs unsupervised learning of an associated probabilistic model of the mixing situation each source in our model is described by a mixture of gaussians thus all the probabilistic calculations can be performed analytically in the second step the sources are reconstructed from the observed data by an optimal nonlinear estimator a variational approximation of this algorithm is derived for cases with a large number of sources where the exact algorithm becomes intractable our ifa algorithm reduces to the one for ordinary fa when the sources become gaussian and to an em algorithm for pca in the zero noise limit we derive an additional em algorithm specifically for noiseless ifa this algorithm is shown to be superior to ica since it can learn arbitrary source densities from the data beyond blind separation ifa can be used for modeling multidimensional data by a highly constrained mixture of gaussians and as a tool for nonlinear signal encoding

information transfer in continuous processes we discuss a recently proposed quantity called transfer entropy which uses time series data to measure the amount of information transferred from one process to another in order to understand its foundation merits and limitations we review some aspects of information theoretic functionals while for symbol sequences these measures have an intuitive interpretation their application to continuous state processes and in particular their estimation from finite data sets is problematic for mutual information finite length scale estimates converge from below and can thus be used to reject the assumption that the observed processes are independent however mutual information does not provide any directional information conversely transfer entropy does resolve the directionality of information exchange but no similar monotonic convergence seems to hold thus only in the case of zero transfer entropy in one direction we can reliably infer an asymmetry of the information exchange

an efficient adi solver for scattered data problems with global smoothing for the approximate representation of large data sets over a parameter domain inr2 many geological and other applications require the construction of surfaces which have minimal energy i e minimal curvature one way to achieve this is by solving a fourth order elliptic partial differential equation its discretization by a difference scheme makes it particularly easy to incorporate appropriate approximations of known data points in this paper we investigate the performance of different solution methods for the resulting symmetric linear system of equations since this is the most cpu demanding step in the scattered data approximation procedure specifically we test first the performance of a preconditioned conjugate gradient method with an ssor and an rilu preconditioner however since the partial differential operator does not contain mixed derivatives using an alternating direction implicit scheme adi method which has been employed successfully in the past for second order problems together with a cholesky factorization of the corresponding one dimensional operators has also been tried for the problem at hand the computational studies that we have performed here show that for our problem an instationary adi method is superior to the above mentioned preconditioned conjugate gradient solvers both with respect to work load and accuracy of the solution for the fourth order model problem considered in this paper the instationary adi method with wachspress parameters results in a number of iterations that is essentially independent of the number of variables

surrogate time series before we apply nonlinear techniques e g those inspired by chaos theory to dynamical phenomena occurring in nature it is necessary to first ask if the use of such advanced techniques is justified by the data while many processes in nature seem very unlikely a priori to be linear the possible nonlinear nature might not be evident in specific aspects of their dynamics the method of surrogate data has become a very popular tool to address such a question however while it was meant to provide a statistically rigorous foolproof framework some limitations and caveats have shown up in its practical use in this paper recent efforts to understand the caveats avoid the pitfalls and to overcome some of the limitations are reviewed and augmented by new material in particular we will discuss specific as well as more general approaches to constrained randomisation providing a full range of examples new algorithms will be introduced for unevenly sampled and multivariate data and for surrogate spike trains the main limitation which lies in the interpretability of the test results will be illustrated through instructive case studies we will also discuss some implementational aspects of the realisation of these methods in the tisean software package

multilevel preconditioning this paper is concerned with multilevel techniques for preconditioning linear systems arising from galerkin methods for elliptic boundary value problems a general estimate is derived which is based on the characterization of besov spaces in terms of weighted sequence norms related to corresponding multilevel expansions the result brings out clearly how the various ingredients of a typical multilevel setting affect the growth rate of the condition numbers in particular our analysis indicates how to realize even uniformly bounded condition numbers for example the general results are used to show that the bramble pasciak xu preconditioner for piecewise linear finite elements gives rise to uniformly bounded condition numbers even when the refinements of the underlying triangulations are highly nonuniform furthermore they are applied to a general multivariate setting of refinable shift invariant spaces in particular covering those induced by various types of wavelets

what does the retina know about natural scenes by examining the experimental data on the statistical properties of natural scenes together with retinal contrast sensitivity data we arrive at a first principle theoretical hypothesis for the purpose of retinal processing and its relationship to an animal s environment we argue that the retinal goal is to transform the visual input as much as possible into a statistically independent basis as the first step in creating a redundancy reduced representation in the cortex as suggested by barlow the extent of this whitening of the input is limited however by the need to suppress input noise our explicit theoretical solutions for the retinal filters also show a simple dependence on mean stimulus luminance they predict an approximate weber law at low spatial frequencies and a de vries rose law at high frequencies assuming that the dominant source of noise is quantum we generate a family of contrast sensitivity curves as a function of mean luminance this family is compared to psychophysical data

variational iteration method for solving cubic nonlinear schrodinger equation the variational iteration method is applied to solve the cubic nonlinear schrodinger cnls equation in one and two space variables in both cases we will reduce the cnls equation to a coupled system of nonlinear equations numerical experiments are made to verify the efficiency of the method comparison with the theoretical solution shows that the variational iteration method is of high accuracy

stochastic trapping in a solvable model of on line independent component analysis previous analytical studies of on line independent component analysis ica learning rules have focused on asymptotic stability and efficiency in practice the transient stages of learning are often more significant in determining the success of an algorithm this is demonstrated here with an analysis of a hebbian ica algorithm which can find a small number of nongaussian components given data composed of a linear mixture of independent source signals an idealized data model is considered in which the sources comprise a number of nongaussian and gaussian sources and a solution to the dynamics is obtained in the limit where the number of gaussian sources is infinite previous stability results are confirmed by expanding around optimal fixed points where a closed form solution to the learning dynamics is obtained however stochastic effects are shown to stabilize otherwise unstable suboptimal fixed points conditions required to destabilize one such fixed point are obtained for the case of a single nongaussian component indicating that the initial learning rate required to escape successfully is very low o n 2 where n is the data dimension resulting in very slow learning typically requiring o n3 iterations simulations confirm that this picture holds for a finite system

characterization of strange attractors a new measure of strange attractors is introduced which offers a practical algorithm to determine their character from the time series of a single observable the relation of this new measure to fractal dimension and information theoretic entropy is discussed

an information maximization approach to blind separation and blind deconvolution we derive a new self organizing learning algorithm that maximizes the information transferred in a network of nonlinear units the algorithm does not assume any knowledge of the input distributions and is defined here for the zero noise limit under these conditions information maximization has extra properties not found in the linear case linsker 1989 the nonlinearities in the transfer function are able to pick up higher order moments of the input distributions and perform something akin to true redundancy reduction between units in the output representation this enables the network to separate statistically independent components in the inputs a higher order generalization of principal components analysis we apply the network to the source separation or cocktail party problem successfully separating unknown mixtures of up to 10 speakers we also show that a variant on the network architecture is able to perform blind deconvolution cancellation of unknown echoes and reverberation in a speech signal finally we derive dependencies of information transfer on time delays we suggest that information maximization provides a unifying framework for problems in blind signal processing

regularized spectral matching for blind source separation application to fmri imaging the main contribution of this paper is to present a bayesian approach for solving the noisy instantaneous blind source separation problem based on second order statistics of the time varying spectrum the success of the blind estimation relies on the nonstationarity of the second order statistics and their intersource diversity choosing the time frequency domain as the signal representation space and transforming the data by a short time fourier transform stft our method presents a simple em algorithm that can efficiently deal with the time varying spectrum diversity of the sources the estimation variance of the stft is reduced by averaging across time frequency subdomains the algorithm is demonstrated on a standard functional resonance imaging fmri experiment involving visual stimuli in a block design explicitly taking into account the noise in the model the proposed algorithm has the advantage of extracting only relevant task related components and considers the remaining components artifacts to be noise

independent component analysis using potts models we explore the extending application of potts encoding to the task of independent component analysis which primarily deals with the problem of minimizing the kullback leibler divergence between the joint distribution and the product of all marginal distributions of output components the competitive mechanism of potts neurons is used to encode the overlapping projections from observations to output components based on these projections the marginal distributions and the entropy of output components are made tractable for computation and the adaptation of the de mixing matrix toward independent output components is obtained the potts model for ica is well formulated by an objective function subject to a set of constraints which leads to a novel energy function a hybrid of the mean field annealing and the gradient descent method is applied to the energy function our approach to independent component analysis presents a new criterion for ica the performance of the potts model for ica given by our numerical simulations is encouraging

source separation in post nonlinear mixtures we address the problem of separation of mutually independent sources in nonlinear mixtures first we propose theoretical results and prove that in the general case it is not possible to separate the sources without nonlinear distortion therefore we focus our work on specific nonlinear mixtures known as post nonlinear mixtures these mixtures constituted by a linear instantaneous mixture linear memoryless channel followed by an unknown and invertible memoryless nonlinear distortion are realistic models in many situations and emphasize interesting properties i e in such nonlinear mixtures sources can be estimated with the same indeterminacies as in instantaneous linear mixtures the separation structure of nonlinear mixtures is a two stage system namely a nonlinear stage followed by a linear stage the parameters of which are updated to minimize an output independence criterion expressed as a mutual information criterion the minimization of this criterion requires knowledge or estimation of source densities or of their log derivatives a first algorithm based on a gram charlier expansion of densities is proposed unfortunately it fails for hard nonlinear mixtures a second algorithm based on an adaptive estimation of the log derivative of densities leads to very good performance even with hard nonlinearities experiments are proposed to illustrate these results