statistical consistency of kernel canonical correlation analysis while kernel canonical correlation analysis cca has been applied in many contexts the convergence of finite sample estimates of the associated functions to their population counterparts has not yet been established this paper gives a mathematical proof of the statistical convergence of kernel cca providing a theoretical justification for the method the proof uses covariance operators defined on reproducing kernel hilbert spaces and analyzes the convergence of their empirical estimates of finite rank to their population counterparts which can have infinite rank the result also gives a sufficient condition for convergence on the regularization coefficient involved in kernel cca this should decrease as n 1 3 where n is the number of data

uniform asymptotics for polynomials orthogonal with respect to varying exponential weights and applications to universality questions in random matrix theory we consider asymptotics for orthogonal polynomials with respect to varying exponential weights wn x dx e nv x dx on the line as n the potentials v are assumed to be real analytic with sufficient growth at infinity the principle results concern plancherel rotach type asymptotics for the orthogonal polynomials down to the axis using these asymptotics we then prove universality for a variety of statistical quantities arising in the theory of random matrix models some of which have been considered recently in 31 and also in 4 an additional application concerns the asymptotics of the recurrence coefficients and leading coefficients for the orthonormal polynomials see also 4 the orthogonal polynomial problem is formulated as a riemann hilbert problem following 19 20 the riemann hilbert problem is analyzed in turn using the steepest descent method introduced in 12 and further developed in 11 13 a critical role in our method is played by the equilibrium measure d v for v as analyzed in 8 1999 john wiley sons inc

correlation functions cluster functions and spacing distributions for random matrices the usual formulas for the correlation functions in orthogonal and symplectic matrix models express them as quaternion determinants from this representation one can deduce formulas for spacing probabilities in terms of fredholm determinants of matrix valued kernels the derivations of the various formulas are somewhat involved in this article we present a direct approach which leads immediately to scalar kernels for the unitary ensembles and matrix kernels for the orthogonal and symplectic ensembles and the representations of the correlation functions cluster functions and spacing distributions in terms of them

asymptotics and integrable structures for biorthogonal polynomials associated to a random two matrix model abstract we give a rigorous construction of complete families of biorthonormal polynomials associated to a planar measure of the form e n v x w y 2 xy d x d y for polynomial v and w we are further able to show that the zeroes of these polynomials are all real and distinct a complex analytical construction of the biorthonormal polynomials is given in terms of a non local riemann hilbert problem which given our prior result provides an avenue for developing uniform asymptotics for the statistical distributions of these zeroes as n becomes large the biorthonormal polynomials considered here play a fundamental role in the analysis of certain random multi matrix models we show that the evolutions of the recursion matrices for the polynomials induced by linear deformations of v and w coincide with a semi infinite generalization of the completely integrable full kostant toda lattice this connection could be relevant for understanding aspects of scaling limits for the multi matrix model

emergence of phase and shift invariant features by decomposition of natural images into independent feature subspaces olshausen and field 1996 applied the principle of independence maximization by sparse coding to extract features from natural images this leads to the emergence of oriented linear filters that have simultaneous localization in space and in frequency thus resembling gabor functions and simple cell receptive fields in this article we show that the same principle of independence maximization can explain the emergence of phase and shift invariant features similar to those found in complex cells this new kind of emergence is obtained by maximizing the independence between norms of projections on linear subspaces instead of the independence of simple linear filter outputs the norms of the projections on such independent feature subspaces then indicate the values of invariant features

multivariate mimo fir inverses the problem of computing exact finite impulse response fir inverses for multivariate multiple input multiple output mimo fir systems is considered necessary and sufficient conditions for invertibility are given along with computation techniques random systems and structured systems are defined necessary and sufficient conditions for the almost sure invertibility of structured random systems are derived bounds on the orders of the inverse filters are computed

separation theorem for independent subspace analysis and its consequences independent component analysis ica the theory of mixed independent non gaussian sources has a central role in signal processing computer vision and pattern recognition one of the most fundamental conjectures of this research field is that independent subspace analysis isa the extension of the ica problem where groups of sources are independent can be solved by traditional ica followed by grouping the ica components the conjecture called isa separation principle i has been rigorously proven for some distribution types recently ii forms the basis of the state of the art isa solvers iii enables one to estimate the unknown number and the dimensions of the sources efficiently and iv can be extended to generalizations of the isa task such as different linear controlled post nonlinear complex valued partially observed problems as well as to problems dealing with nonparametric source dynamics here we shall review the advances on this field

on the complete integrability and linearization of nonlinear ordinary differential equations ii third order equations we introduce a method for finding general solutions of third order nonlinear differential equations by extending the modified prelle singer method we describe a procedure to deduce all the integrals of motion associated with the given equation so that the general solution follows straightforwardly from these integrals the method is illustrated with several examples further we propose a powerful method of identifying linearizing transformations the proposed method not only unifies all the known linearizing transformations systematically but also introduces a new and generalized linearizing transformation in addition to the above we provide an algorithm to invert the non local linearizing transformation through this procedure the general solution for the original nonlinear equation can be obtained from the solution of the linear ordinary differential equation

undercomplete blind subspace deconvolution we introduce the blind subspace deconvolution bssd problem which is the extension of both the blind source deconvolution bsd and the independent subspace analysis isa tasks we examine the case of the undercomplete bssd ubssd applying temporal concatenation we reduce this problem to isa the associated high dimensional isa problem can be handled by a recent technique called joint f decorrelation jfd similar decorrelation methods have been used previously for kernel independent component analysis kernel ica more precisely the kernel canonical correlation kcca technique is a member of this family and as is shown in this paper the kernel generalized variance kgv method can also be seen as a decorrelation method in the feature space these kernel based algorithms will be adapted to the isa task in the numerical examples we i examine how efficiently the emerging higher dimensional isa tasks can be tackled and ii explore the working and advantages of the derived kernel isa methods

model selection for convolutive ica with an application to spatiotemporal analysis of eeg we present a new algorithm for maximum likelihood convolutive independent component analysis ica in which components are unmixed using stable autoregressive filters determined implicitly by estimating a convolutive model of the mixing process by introducing a convolutive mixing model for the components we show how the order of the filters in the model can be correctly detected using bayesian model selection we demonstrate a framework for deconvolving a subspace of independent components in electroencephalography eeg initial results suggest that in some cases convolutive mixing may be a more realistic model for eeg signals than the instantaneous ica model

on the distribution of the length of the longest increasing subsequence of random permutations let sn be the group of permutations of 1 2 n if 7r e sn we say that 7 i1 7f ik is an increasing subsequence in 7r if il i2 ik and 7r ii 7r i2 7r ik let 1n r be the length of the longest increasing subsequence for example if n 5 and 7r is the permutation 5 1 3 2 4 in one line notation thus 7r 1 5 7r 2 1 then the longest increasing subsequences are 1 2 4 and 1 3 4 and n 3 equip sn with uniform distribution

kernel methods for measuring independence we introduce two new functionals the constrained covariance and the kernel mutual information to measure the degree of independence of random variables these quantities are both based on the covariance between functions of the random variables in reproducing kernel hilbert spaces rkhss we prove that when the rkhss are universal both functionals are zero if and only if the random variables are pairwise independent we also show that the kernel mutual information is an upper bound near independence on the parzen window estimate of the mutual information analogous results apply for two correlation based dependence functionals introduced earlier we show the kernel canonical correlation and the kernel generalised variance to be independence measures for universal kernels and prove the latter to be an upper bound on the mutual information near independence the performance of the kernel dependence functionals in measuring independence is verified in the context of independent component analysis

blind separation of multi dimensional components via subspace decomposition performance analysis a prevalent approach to blind separation of multi dimensional data is a two step procedure in the first step the data are assigned a one dimensional model a separating algorithm is applied according to this model this step corresponds to classical blind source separation bss in the second step the output is assigned into groups representing the multi dimensional components in this paper we consider an even more general case in which the subpartition of the components in the first step may be into elements of any dimension not necessarily one we consider a piecewise stationary model and assume that the number and dimensions of the underlying multi dimensional components are known we obtain a closed form analytical expression for the mean square error mse in the estimation of the multi dimensional components using this two step procedure as expected this approach is suboptimal in the presence of finite data size errors therefore we can predict the expected gain from using the correct model of the components over any finer decomposition thereof in addition we demonstrate the dependence of this gain on the model parameters

second order multidimensional ica performance analysis independent component analysis ica and blind source separation bss deal with extracting a number of mutually independent elements from a set of observed linear mixtures motivated by various applications this paper considers a more general and more flexible model the sources can be partitioned into groups exhibiting dependence within a given group but independence between two different groups we argue that this is tantamount to considering multidimensional components as opposed to the standard ica case which is restricted to one dimensional components the core of the paper is devoted to the statistical analysis of the blind separation of multidimensional components based on second order statistics in a piecewise stationary model we develop the likelihood and the associated estimating equations for the gaussian case we obtain closed form expressions for the fisher information matrix and the cramer rao bound of the de mixing parameters as well as the mean square error mse of the component estimates the derived mse is valid also for non gaussian data our analysis is verified through numerical experiments and its performance is compared to classical ica in various dependence scenarios quantifying the gain in the accuracy of component recovery in presence of multidimensional components

bootstrap rank ordered conditional mutual information brocmi a nonlinear input variable selection method for water resources modeling the input variable selection problem has recently garnered much interest in the time series modeling community especially within water resources applications demonstrating that information theoretic nonlinear based input variable selection algorithms such as partial mutual information pmi selection pmis provide an improved representation of the modeled process when compared to linear alternatives such as partial correlation input selection pcis pmis is a popular algorithm for water resources modeling problems considering nonlinear input variable selection however this method requires the specification of two nonlinear regression models each with parametric settings that greatly influence the selected input variables other attempts to develop input variable selection methods using conditional mutual information cmi an analog to pmi have been formulated under different parametric pretenses such as k nearest neighbor knn statistics or kernel density estimates kde in this paper we introduce a new input variable selection method based on cmi that uses a nonparametric multivariate continuous probability estimator based on edgeworth approximations ea we improve the ea method by considering the uncertainty in the input variable selection procedure by introducing a bootstrap resampling procedure that uses rank statistics to order the selected input sets we name our proposed method bootstrap rank ordered cmi brocmi we demonstrate the superior performance of brocmi when compared to cmi based alternatives ea kde and knn pmis and pcis input variable selection algorithms on a set of seven synthetic test problems and a real world urban water demand uwd forecasting experiment in ottawa canada

a fast fixed point algorithm for independent component analysis we introduce a novel fast algorithm for independent component analysis which can be used for blind source separation and feature extraction we show how a neural network learning rule can be transformed into a fixedpoint iteration which provides an algorithm that is very simple does not depend on any user defined parameters and is fast to converge to the most accurate solution allowed by the data the algorithm finds one at a time all nongaussian independent components regardless of their probability distributions the computations can be performed in either batch mode or a semiadaptive manner the convergence of the algorithm is rigorously proved and the convergence speed is shown to be cubic some comparisons to gradient based algorithms are made showing that the new algorithm is usually 10 to 100 times faster sometimes giving the solution in just a few iterations

kernel independent component analysis we present a class of algorithms for independent component analysis ica which use contrast functions based on canonical correlations in a reproducing kernel hilbert space on the one hand we show that our contrast functions are related to mutual information and have desirable mathematical properties as measures of statistical dependence on the other hand building on recent developments in kernel methods we show that these criteria and their derivatives can be computed efficiently minimizing these criteria leads to flexible and robust algorithms for ica we illustrate with simulations involving a wide variety of source distributions showing that our algorithms outperform many of the presently known algorithms

on the statistical mechanics approach in the random matrix theory integrated density of states we consider the ensemble of random symmetricn n matrices specified by an orthogonal invariant probability distribution we treat this distribution as a gibbs measure of a mean field type model this allows us to show that the normalized eigenvalue counting function of this ensemble converges in probability to a nonrandom limit asn and that this limiting distribution is the solution of a certain self consistent equation

beyond independent components trees and clusters we present a generalization of independent component analysis ica where instead of looking for a linear transform that makes the data components independent we look for a transform that makes the data components well fit by a tree structured graphical model this tree dependent component analysis tca provides a tractable and flexible approach to weakening the assumption of independence in ica in particular tca allows the underlying graph to have multiple connected components and thus the method is able to find clusters of components such that components are dependent within a cluster and independent between clusters finally we make use of a notion of graphical models for time series due to brillinger 1996 to extend these ideas to the temporal setting in particular we are able to fit models that incorporate tree structured dependencies among multiple time series

toda flows with infinitely many variables abstract the toda flow and related flows extend naturally to operators in hilbert space and the purpose of this paper is to describe these flows and to analyse some of their special properties

a method of integration over matrix variables the integral over twon n hermitan matricesz g c dadbexp tr a 2 b 2 2cab g n a 4 b 4 is evaluated in the limit of largen for this purpose use is made of the theory of diffusion equation and that of orthogonal polynomials with a non local weight the above integral arises in the study of the planar approximation to quantum field theory

semiclassical asymptotics of orthogonal polynomials riemann hilbert problem and universality in the matrix model we derive semiclassical asymptotics for the orthogonal polynomials pn z on the line with respect to the exponential weight exp inv z where v z is a double well quartic polynomial in the limit when n n 1 we assume that n n cri for some 0 where cr is the critical value which separates orthogonal polynomials with two cuts from the ones with one cut simultaneously we derive semiclassical asymptotics for the recursive coe cients of the orthogonal polynomials and we show that these coe cients form a cycle of period two which drifts slowly with the change of the ratio n n the proof of the semiclassical asymptotics is based on the methods of the theory of integrable systems and on the analysis of the appropriate matrix riemannhilbert problem as an application of the semiclassical asymptotics of the orthogonal polynomials we prove the universality of the local distribution of eigenvalues in the matrix model with the double well quartic interaction in the presence of two cuts contents

separation of synchronous sources through phase locked matrix factorization in this paper we study the separation of synchronous sources sss problem which deals with the separation of sources whose phases are synchronous this problem cannot be addressed through independent component analysis methods because synchronous sources are statistically dependent we present a two step algorithm called phase locked matrix factorization plmf to perform sss we also show that sss is identifiable under some assumptions and that any global minimum of plmfs cost function is a desirable solution for sss we extensively study the algorithm on simulated data and conclude that it can perform sss with various numbers of sources and sensors and with various phase lags between the sources both in the ideal i e perfectly synchronous and nonnoisy case and with various levels of additive noise in the observed signals and of phase jitter in the sources