finiteness of conical algorithms with ω subdivisions in this paper the problem of finding the global optimum of a concave function over a polytope is considered a well known class of algorithms for this problem is the class of conical algorithms in particular the conical algorithm based on the so called subdivision strategy is considered it is proved that for any given accuracy e 0 this algorithm stops in a finite time by returning an e optimal solution for the problem while it is convergent for e 0

finding minimum entropy codes to determine whether a particular sensory event is a reliable predictor of reward or punishment it is necessary to know the prior probability of that event if the variables of a sensory representation normally occur independently of each other then it is possible to derive the prior probability of any logical function of the variables from the prior probabilities of the individual variables without any additional knowledge hence such a representation enormously enlarges the scope of definable events that can be searched for reliable predictors finding a minimum entropy code is a possible method of forming such a representation and methods for doing this are explored in this paper the main results are 1 to show how to find such a code when the probabilities of the input states form a geometric progression as is shown to be nearly true for keyboard characters in normal text 2 to show how a minimum entropy code can be approximated by repeatedly recoding pairs triples etc of an original 7 bit code for keyboard characters 3 to prove that in some cases enlarging the capacity of the output channel can lower the entropy

a hybrid technique for blind separation of non gaussian and time correlated sources using a multicomponent approach blind inversion of a linear and instantaneous mixture of source signals is a problem often encountered in many signal processing applications efficient fastica efica offers an asymptotically optimal solution to this problem when all of the sources obey a generalized gaussian distribution at most one of them is gaussian and each is independent and identically distributed i i d in time likewise weights adjusted second order blind identification wasobi is asymptotically optimal when all the sources are gaussian and can be modeled as autoregressive ar processes with distinct spectra nevertheless real life mixtures are likely to contain both gaussian ar and non gaussian i i d sources rendering wasobi and efica severely suboptimal in this paper we propose a novel scheme for combining the strengths of efica and wasobi in order to deal with such hybrid mixtures simulations show that our approach outperforms competing algorithms designed for separating similar mixtures

exact mean integrated squared error an exact and easily computable expression for the mean integrated squared error mise for the kernel estimator of a general normal mixture density is given for gaussian kernels of arbitrary order this provides a powerful new way of understanding density estimation which complements the usual tools of simulation and asymptotic analysis the family of normal mixture densities is very flexible and the formulae derived allow simple exact analysis for a wide variety of density shapes a number of applications of this method giving important new insights into kernel density estimation are presented among these is the discovery that the usual asymptotic approximations to the mise can be quite inaccurate especially when the underlying density contains substantial fine structure and also strong evidence that the practical importance of higher order kernels is surprisingly small for moderate sample sizes 1 introduction substantial research has been devoted to kernel density estimation this is because it provides a simple yet appealing context in which to study problems and issues that arise in all types of nonparametric curve estimation this includes regression spectral density and hazard estimation and also a variety of other estimators including histograms splines and orthogonal series three important and useful tools for understanding the behavior of nonparametric curve estimators are asymptotic analysis simulation and numerical calculation of error criteria each of these methods provides many useful insights into the complicated structure present in the study of curve estimation however each has its limitations as well rhe strength of asymptotic analysis is that it frequently allows simultaneous study of many different specific examples through general results applying to entire classes of settings the weakness of asymptotics is that they only describe behavior in the limit this is still very useful in many situations because the asymptotics describe the actual situation quite well however it is less useful when the asymptotics have not yet kicked in that is in studying situations where the asymptotically dominant effect has not taken over yet perhaps the biggest drawback to asymptotics is that it is very difficult to determine in a given situation which of these possibilities is occurring

information maximization and independent component analysis is there a difference this article provides a detailed and rigorous analysis of the two commonly used methods for redundancy reduction linear independent component analysis ica posed as a direct minimization of a suitably chosen redundancy measure and information maximization infomax of a continuous stochastic signal transmitted through an appropriate nonlinear network the article shows analytically that ica based on the kullbackleibler information as a redundancy measure and infomax lead to the same solution if the parameterization of the output nonlinear functions in the latter method is sufficiently rich furthermore this work discusses the alternative redundancy measures not based on the kullback leibler information distance the practical issues of applying ica and infomax are also discussed and illustrated on the problem of extracting statistically independent factors from a linear pixel by pixel mixture of images

a fast fixed point algorithm for independent component analysis we introduce a novel fast algorithm for independent component analysis which can be used for blind source separation and feature extraction we show how a neural network learning rule can be transformed into a fixedpoint iteration which provides an algorithm that is very simple does not depend on any user defined parameters and is fast to converge to the most accurate solution allowed by the data the algorithm finds one at a time all nongaussian independent components regardless of their probability distributions the computations can be performed in either batch mode or a semiadaptive manner the convergence of the algorithm is rigorously proved and the convergence speed is shown to be cubic some comparisons to gradient based algorithms are made showing that the new algorithm is usually 10 to 100 times faster sometimes giving the solution in just a few iterations

new families of skewed higher order kernel estimators to solve the bss ica problem for multimodal sources mixtures this letter suggests two new types of asymmetrical higher order kernels hok that are generated using the orthogonal polynomials laguerre positive or right skew and bessel negative or left skew these skewed hok are implemented in the blind source separation independent component analysis bss ica algorithm the tests for these proposed hok are accomplished using three scenarios to simulate a real environment using actual sound sources an environment of mixtures of multimodal fast changing probability density function pdf sources that represent a challenge to the symmetrical hok and an environment of an adverse case near gaussian the separation is performed by minimizing the mutual information mi among the mixed sources the performance of the skewed kernels is compared to the performance of the standard kernels such as epanechnikov bisquare trisquare and gaussian and the performance of the symmetrical hok generated using the polynomials chebyshev1 chebyshev2 gegenbauer jacobi and legendre to the tenth order the gaussian hok are generated using the hermite polynomial and the wand and schucany procedure the comparison among the 96 kernels is based on the average intersymbol interference ratio aisir and the time needed to complete the separation in terms of aisir the skewed kernels performance is better than that of the standard kernels and rivals most of the symmetrical kernels performance the importance of these new skewed hok is manifested in the environment of the multimodal pdf mixtures in such an environment the skewed hok come in first place compared with the symmetrical hok these new families can substitute for symmetrical hoks in such applications

a theory for multiresolution signal decomposition the wavelet representation multiresolution representations are effective for analyzing the information content of images the properties of the operator which approximates a signal at a given resolution were studied it is shown that the difference of information between the approximation of a signal at the resolutions 2 sup j 1 and 2 sup j where j is an integer can be extracted by decomposing this signal on a wavelet orthonormal basis of l sup 2 r sup n the vector space of measurable square integrable n dimensional functions in l sup 2 r a wavelet orthonormal basis is a family of functions which is built by dilating and translating a unique function psi x this decomposition defines an orthogonal multiresolution representation called a wavelet representation it is computed with a pyramidal algorithm based on convolutions with quadrature mirror filters wavelet representation lies between the spatial and fourier domains for images the wavelet representation differentiates several spatial orientations the application of this representation to data compression in image coding texture discrimination and fractal analysis is discussed

adaptive learning in cartesian product of reproducing kernel hilbert spaces we propose a novel adaptive learning algorithm based on iterative orthogonal projections in the cartesian product of multiple reproducing kernel hilbert spaces rkhss the objective is to estimate or track nonlinear functions that are supposed to contain multiple components such as i linear and nonlinear components and ii high and low frequency components in this case the use of multiple rkhss permits a compact representation of multicomponent functions the proposed algorithm is where two different methods of the author meet multikernel adaptive filtering and the algorithm of hyperplane projection along affine subspace hypass in a particular case the sum space of the rkhss is isomorphic under a straightforward correspondence to the product space and hence the proposed algorithm can also be regarded as an iterative projection method in the sum space the efficacy of the proposed algorithm is shown by numerical examples

complexity pursuit separating interesting components from time series a generalization of projection pursuit for time series that is signals with time structure is introduced the goal is to find projections of time series that have interesting structure defined using criteria related to kolmogoroff complexity or coding length interesting signals are those that can be coded with a short code length we derive a simple approximation of coding length that takes into account both the nongaussianity and the autocorrelations of the time series also we derive a simple algorithm for its approximative optimization the resulting method is closely related to blind separation of nongaussian time dependent source signals

a constructive approach to linear lyapunov functions for positive switched systems using collatz wielandt sets we establish the link between linear lyapunov functions for positive switched systems and corresponding collatz wielandt sets this leads to an algorithm to compute a linear lyapunov function whenever a lyapunov function exists

on polynomial based projection indices for exploratory projection pursuit we develop asymptotic theory for two polynomial based methods of estimating orientation in projection pursuit density approximation one of the techniques uses legendre polynomials and has been proposed and implemented by friedman 1 the other employs hermite functions issues of smoothing parameter choice and robustness are addressed it is shown that each method can be used to construct sqrt n consistent estimates of the projection which maximizes distance from normality although the former can only be employed in that manner when the underlying distribution has extremely light tails the former can be used very generally to measure low frequency departure from normality

on stochastic approximation of the eigenvectors and eigenvalues of the expectation of a random matrix abstract in applications of signal processing and pattern recognition eigenvectors and eigenvalues of the statistical mean of a random matrix sequence are needed iterative methods are suggested and analyzed in which no sample moments are used convergence is shown by stochastic approximation theory

nonlinear principal components i absolutely continuous random variables with positive bounded densities nonlinear principal components for an absolutely continuous random vector x with positive bounded density are defined as the solution of a variational problem in a suitable function space in this way transformations depending on all the components of x are obtained some properties of nonlinear principal components are proved in particular it is shown that the set of nonlinear principal transformations of x is an orthonormal basis for the function space associated with the optimal problem the spectral decomposition of x and its covariance matrix with respect to this basis are given a notion of marginal nonlinear principal components is sketched and the relations with nonlinear principal components are shown finally treating the case of random vectors distributed on unbounded domains the existence problem is shown to be related to the global existence of the moment generating function of x since it is not restrictive definitions and results are stated in terms of a uniformly distributed random vector

independent factor analysis we introduce the independent factor analysis ifa method for recovering independent hidden sources from their observed mixtures ifa generalizes and unifies ordinary factor analysis fa principal component analysis pca and independent component analysis ica and can handle not only square noiseless mixing but also the general case where the number of mixtures differs from the number of sources and the data are noisy ifa is a two step procedure in the first step the source densities mixing matrix and noise covariance are estimated from the observed data by maximum likelihood for this purpose we present an expectation maximization em algorithm which performs unsupervised learning of an associated probabilistic model of the mixing situation each source in our model is described by a mixture of gaussians thus all the probabilistic calculations can be performed analytically in the second step the sources are reconstructed from the observed data by an optimal nonlinear estimator a variational approximation of this algorithm is derived for cases with a large number of sources where the exact algorithm becomes intractable our ifa algorithm reduces to the one for ordinary fa when the sources become gaussian and to an em algorithm for pca in the zero noise limit we derive an additional em algorithm specifically for noiseless ifa this algorithm is shown to be superior to ica since it can learn arbitrary source densities from the data beyond blind separation ifa can be used for modeling multidimensional data by a highly constrained mixture of gaussians and as a tool for nonlinear signal encoding

robust recursive least squares learning algorithm for principal component analysis a learning algorithm for the principal component analysis pca is developed based on the least square minimization the dual learning rate parameters are adjusted adaptively to make the proposed algorithm capable of fast convergence and high accuracy for extracting all principal components the proposed algorithm is robust to the error accumulation existing in the sequential pca algorithm we show that all information needed for pca can he completely represented by the unnormalized weight vector which is updated based only on the corresponding neuron input output product the updating of the normalized weight vector can be referred to as a leaky hebb s rule the convergence of the proposed algorithm is briefly analyzed we also establish the relation between oja s rule and the least squares learning rule finally the simulation results are given to illustrate the effectiveness of this algorithm for pca and tracking time varying directions of arrival

estimation of entropy and mutual information we present some new results on the nonparametric estimation of entropy and mutual information first we use an exact local expansion of the entropy function to prove almost sure consistency and central limit theorems for three of the most commonly used discretized information estimators the setup is related to grenander s method of sieves and places no assumptions on the underlying probability measure generating the data second we prove a converse to these consistency theorems demonstrating that a misapplication of the most common estimation techniques leads to an arbitrarily poor estimate of the true information even given unlimited data this inconsistency theorem leads to an analytical approximation of the bias valid in surprisingly small sample regimes and more accurate than the usual 1 n formula of miller and madow over a large region of parameter space the two most practical implications of these results are negative 1 information estimates in a certain data regime are likely contaminated by bias even if bias corrected estimators are used and 2 confidence intervals calculated by standard techniques drastically underestimate the error of the most common estimation methods finally we note a very useful connection between the bias of entropy estimators and a certain polynomial approximation problem by casting bias calculation problems in this approximation theory framework we obtain the best possible generalization of known asymptotic bias results more interesting this framework leads to an estimator with some nice properties the estimator comes equipped with rigorous bounds on the maximum error over all possible underlying probability distributions and this maximum error turns out to be surprisingly small we demonstrate the application of this new estimator on both real and simulated data

an information maximization approach to blind separation and blind deconvolution we derive a new self organizing learning algorithm that maximizes the information transferred in a network of nonlinear units the algorithm does not assume any knowledge of the input distributions and is defined here for the zero noise limit under these conditions information maximization has extra properties not found in the linear case linsker 1989 the nonlinearities in the transfer function are able to pick up higher order moments of the input distributions and perform something akin to true redundancy reduction between units in the output representation this enables the network to separate statistically independent components in the inputs a higher order generalization of principal components analysis we apply the network to the source separation or cocktail party problem successfully separating unknown mixtures of up to 10 speakers we also show that a variant on the network architecture is able to perform blind deconvolution cancellation of unknown echoes and reverberation in a speech signal finally we derive dependencies of information transfer on time delays we suggest that information maximization provides a unifying framework for problems in blind signal processing

superefficiency in blind source separation blind source separation is the problem of extracting independent signals from their mixtures without knowing the mixing coefficients nor the probability distributions of source signals and may be applied to eeg and meg imaging of the brain it is already known that certain algorithms work well for the extraction of independent components the present paper is concerned with superefficiency of these based on the statistical and dynamical analysis in a statistical estimation using t examples the covariance of any two extracted independent signals converges to 0 of the order of 1 t on line dynamics shows that the covariance is of the order of spl eta when the learning rate spl eta is fixed to a small constant in contrast with the above general properties a surprising superefficiency holds in blind source separation under certain conditions where superefficiency implies that covariance decreases in the order of 1 t sup 2 or of spl eta sup 2 the paper uses the natural gradient learning algorithm and method of estimating functions to obtain superefficient procedures for both batch estimation and on line learning a standardized estimating function is introduced to this end superefficiency does not imply that the error variances of the extracted signals decrease in the order of 1 t sup 2 or spl eta sup 2 but implies that their covariances and independencies do