selection of variables in cluster analysis an empirical comparison of eight procedures eight different variable selection techniques for model based and non model based clustering are evaluated across a wide range of cluster structures it is shown that several methods have difficulties when non informative variables i e random noise are included in the model furthermore the distribution of the random noise greatly impacts the performance of nearly all of the variable selection procedures overall a variable selection technique based on a variance to range weighting procedure coupled with the largest decreases in within cluster sums of squares error performed the best on the other hand variable selection methods used in conjunction with finite mixture models performed the worst

nonparametric multivariate density estimation a comparative study the paper algorithmically and empirically studies two major types of nonparametric multivariate density estimation techniques where no assumption is made about the data being drawn from any of known parametric families of distribution the first type is the popular kernel method and several of its variants which uses locally tuned radial basis e g gaussian functions to interpolate the multidimensional density the second type is based on an exploratory projection pursuit technique which interprets the multidimensional density through the construction of several 1d densities along highly interesting projections of multidimensional data performance evaluations using training data from mixture gaussian and mixture cauchy densities are presented the results show that the curse of dimensionality and the sensitivity of control parameters have a much more adverse impact on the kernel density estimators than on the projection pursuit density estimators

flexible discriminant analysis by optimal scoring abstract fisher s linear discriminant analysis is a valuable tool for multigroup classification with a large number of predictors one can find a reduced number of discriminant coordinate functions that are optimal for separating the groups with two such functions one can produce a classification map that partitions the reduced space into regions that are identified with group membership and the decision boundaries are linear this article is about richer nonlinear classification schemes linear discriminant analysis is equivalent to multiresponse linear regression using optimal scorings to represent the groups in this paper we obtain nonparametric versions of discriminant analysis by replacing linear regression by any nonparametric regression method in this way any multiresponse regression technique such as mars or neural networks can be postprocessed to improve its classification performance

svm decision boundary based discriminative subspace induction we study the problem of linear dimension reduction for classification with a focus on sufficient dimension reduction i e finding subspaces without loss of discrimination power first we formulate the concept of sufficient subspace for classification in parallel terms as for regression then we present a new method to estimate the smallest sufficient subspace based on an improvement of decision boundary analysis dba the main idea is to combine dba with support vector machines svm to overcome the inherent difficulty of dba in small sample size situations while keeping dba s estimation simplicity the compact representation of svm boundary results in a significant gain in both speed and accuracy over previous dba implementations alternatively this technique can be viewed as a way to reduce the run time complexity of svm itself comparative experiments on one simulated and four real world benchmark datasets highlight the superior performance of the proposed approach

edgeworth approximation of multivariate differential entropy we develop the general multivariate case of the edgeworth approximation of differential entropy and show that it can be more accurate than the nearest neighbor method in the multivariate case and that it scales better with sample size furthermore we introduce mutual information estimation as an application

nonlinear modeling of scattered multivariate data and its application to shape change we are given a set of points in a space of high dimension for instance this set may represent many visual appearances of an object a face or a hand we address the problem of approximating this set by a manifold in order to have a compact representation of the object appearance when the scattering of this set is approximately an ellipsoid then the problem has a well known solution given by principal components analysis pca however in some situations like object displacement learning or face learning this linear technique may be ill adapted and nonlinear approximation has to be introduced the method we propose can be seen as a nonlinear pca nlpca the main difficulty being that the data are not ordered we propose an index which favors the choice of axes preserving the closest point neighborhoods these axes determine an order for visiting all the points when smoothing finally a new criterion called generalization error is introduced to determine the smoothing rate that is the knot number for the spline fitting experimental results conclude this paper the method is tested on artificial data and on two databases used in visual learning

kernel methods in machine learning we review machine learning methods employing positive definite kernels these methods formulate learning and estimation problems in a reproducing kernel hilbert space rkhs of functions defined on the data domain expanded in terms of a kernel working in linear spaces of function has the benefit of facilitating the construction and analysis of learning algorithms while at the same time allowing large classes of functions the latter include nonlinear functions as well as functions defined on nonvectorial data we cover a wide range of methods ranging from binary classifiers to sophisticated methods for estimation with structured data

the effect of noise on a class of energy based learning rules we study the selectivity properties of neurons based on bcm and kurtosis energy functions in a general case of noisy high dimensional input space the proposed approach which is used for characterization of the stable states can be generalized to a whole class of energy functions we characterize the critical noise levels beyond which the selectivity is destroyed we also perform a quantitative analysis of such transitions which shows interesting dependency on data set size we observe that the robustness to noise of the bcm neuron bienenstock cooper munro 1982 intrator cooper 1992 increases as a function of dimensionality we explicitly compute the separability limit of bcm and kurtosis learning rules in the case of a bimodal input distribution numerical simulations show a stronger robustness of the bcm rule for practical data set size when compared with kurtosis

mixture densities maximum likelihood and the em algorithm the problem of estimating the parameters which determine a mixture density has been the subject of a large diverse body of literature spanning nearly ninety years during the last two decades the

significance levels in exploratory projection pursuit summary in exploratory data analysis projection pursuit methods explore the nonlinear structure of high dimensional data it is useful to have a significance test to help us decide whether apparent structure is real or just the effect of noise monte carlo methods can be helpful to achieve this but unfortunately in this case they are computationally expensive in this paper under a suitable null hypothesis we derive a theoretical approximation for the p value associated with friedman s projection pursuit index friedman 1987 the result of monte carlo simulations is compared with our analytical result some practical aspects are discussed

what is the goal of sensory coding a number of recent attempts have been made to describe early sensory coding in terms of a general information processing strategy in this paper two strategies are contrasted both strategies take advantage of the redundancy in the environment to produce more effective representations the first is described as a compact coding scheme a compact code performs a transform that allows the input to be represented with a reduced number of vectors cells with minimal rms error this approach has recently become popular in the neural network literature and is related to a process called principal components analysis pca a number of recent papers have suggested that the optimal compact code for representing natural scenes will have units with receptive field profiles much like those found in the retina and primary visual cortex however in this paper it is proposed that compact coding schemes are insufficient to account for the receptive field properties of cells in the mammalian visual pathway in contrast it is proposed that the visual system is near to optimal in representing natural scenes only if optimality is defined in terms of sparse distributed coding in a sparse distributed code all cells in the code have an equal response probability across the class of images but have a low response probability for any single image in such a code the dimensionality is not reduced rather the redundancy of the input is transformed into the redundancy of the firing pattern of cells it is proposed that the signature for a sparse code is found in the fourth moment of the response distribution i e the kurtosis in measurements with 55 calibrated natural scenes the kurtosis was found to peak when the bandwidths of the visual code matched those of cells in the mammalian visual cortex codes resembling wavelet transforms are proposed to be effective because the response histograms of such codes are sparse i e show high kurtosis when presented with natural scenes it is proposed that the structure of the image that allows sparse coding is found in the phase spectrum of the image it is suggested that natural scenes to a first approximation can be considered as a sum of self similar local functions the inverse of a wavelet possible reasons for why sensory systems would evolve toward sparse coding are presented

feature selection for unsupervised learning in this paper we identify two issues involved in developing an automated feature subset selection algorithm for unlabeled data the need for finding the number of clusters in conjunction with feature selection and the need for normalizing the bias of feature selection criteria with respect to dimension we explore the feature selection problem and these issues through fssem feature subset selection using expectation maximization em clustering and through two different performance criteria for evaluating candidate feature subsets scatter separability and maximum likelihood we present proofs on the dimensionality biases of these feature criteria and present a cross projection normalization scheme that can be applied to any criterion to ameliorate these biases our experiments show the need for feature selection the need for addressing these two issues and the effectiveness of our proposed solutions

data clustering using a model granular magnet we present a new approach to clustering based on the physical properties of an inhomogeneous ferromagnet no assumption is made regarding the underlying distribution of the data we assign a potts spin to each data point and introduce an interaction between neighboring points whose strength is a decreasing function of the distance between the neighbors this magnetic system exhibits three phases at very low temperatures it is completely ordered all spins are aligned at very high temperatures the system does not exhibit any ordering and in an intermediate regime clusters of relatively strongly coupled spins become ordered whereas different clusters remain uncorrelated this intermediate phase is identified by a jump in the order parameters the spin spin correlation function is used to partition the spins and the corresponding data points into clusters we demonstrate on three synthetic and three real data sets how the method works detailed comparison to the performance of other techniques clearly indicates the relative success of our method

statistical principles in image modeling images of natural scenes contain a rich variety of visual patterns to learn and recognize these patterns from natural images it is necessary to construct statistical models for these patterns in this review article we describe three statistical principles for modeling image patterns the sparse coding principle the minimax entropy principle and the meaningful alignment principle we explain these three principles and their relationships in the context of modeling images as compositions of gabor wavelets these three principles correspond to three regimes of composition patterns of gabor wavelets and these three regimes are connected by changes in scale or resolution

recent developments in nonparametric density estimation advances in computation and the fast and cheap computational facilities now available to statisticians have had a significant impact upon statistical research and especially the development of nonparametric data analysis procedures in particular theoretical and applied research on nonparametric density estimation has had a noticeable influence on related topics such as nonparametric regression nonparametric discrimination and nonparametric pattern recognition this article reviews recent developments in nonparametric density estimation and includes topics that have been omitted from review articles and books on the subject the early density estimation methods such as the histogram kernel estimators and orthogonal series estimators are still very popular and recent research on them is described different types of restricted maximum likelihood density estimators including order restricted estimators maximum penalized likelihood estimators and sieve estimators are discussed where restrictions are imposed upon the class of densities or on the form of the likelihood function nonparametric density estimators that are data adaptive and lead to locally smoothed estimators are also discussed these include variable partition histograms estimators based on statistically equivalent blocks nearest neighbor estimators variable kernel estimators and adaptive kernel estimators for the multivariate case extensions of methods of univariate density estimation are usually straightforward but can be computationally expensive a method of multivariate density estimation that did not spring from a univariate generalization is described namely projection pursuit density estimation in which both dimensionality reduction and density estimation can be pursued at the same time finally some areas of related research are mentioned such as nonparametric estimation of functionals of a density robust parametric estimation semiparametric models and density estimation for censored and incomplete data directional and spherical data and density estimation for dependent sequences of observations

dimension selection for feature selection and dimension reduction with principal and independent component analysis this letter is concerned with the problem of selecting the best or most informative dimension for dimension reduction and feature extraction in high dimensional data the dimension of the data is reduced by principal component analysis subsequent application of independent component analysis to the principal component scores determines the most nongaussian directions in the lower dimensional space a criterion for choosing the optimal dimension based on bias adjusted skewness and kurtosis is proposed this new dimension selector is applied to real data sets and compared to existing methods simulation studies for a range of densities show that the proposed method performs well and is more appropriate for nongaussian data than existing methods

orthogonal principal planes factor analysis and principal component analysis result in computing a new coordinate system which is usually rotated to obtain a better interpretation of the results in the present paper the idea of rotation to simple structure is extended to two dimensions while the classical definition of simple structure is aimed at rotating one dimensional factors the extension to a simple structure for two dimensions is based on the rotation of planes the resulting planes principal planes reveal a better view of the data than planes spanned by factors from classical rotation and hence allow a more reliable interpretation the usefulness of the method as well as the effectiveness of a proposed algorithm are demonstrated by simulation experiments and an example

sliced inverse regression for dimension reduction abstract modern advances in computing power have greatly widened scientists scope in gathering and investigating information from many variables information which might have been ignored in the past yet to effectively scan a large pool of variables is not an easy task although our ability to interact with data has been much enhanced by recent innovations in dynamic graphics in this article we propose a novel data analytic tool sliced inverse regression sir for reducing the dimension of the input variable x without going through any parametric or nonparametric model fitting process this method explores the simplicity of the inverse view of regression that is instead of regressing the univariate output variable y against the multivariate x we regress x against y forward regression and inverse regression are connected by a theorem that motivates this method the theoretical properties of sir are investigated under a model of the form y f 1 x k x e where the k s are the unknown

continual reassessment method for ordered groups summary we investigate the two group continual reassessment method for a dose finding study in which we anticipate some ordering between the groups this is a situation in which for either group we have little or almost no knowledge about which of the available dose levels will correspond to the maximum tolerated dose mtd but we may have quite strong knowledge concerning which of the two groups will have the higher level of mtd if indeed they do not have the same mtd the motivation for studying this problem came from an investigation into a new therapy for acute leukemia in children the background to this study is discussed there were two groups of patients one group already received heavy prior therapy while the second group had received relatively much lighter prior therapy it was therefore anticipated that the second group would have an mtd higher or at least as high as the first generally likelihood methods or equivalently the use of noninformative bayes priors can be used to model the main aspects of the study i e the mtd for one of the groups reserving more informative bayes modeling to be applied to the secondary features of the study these secondary features may simply be the direction of the difference between the mtd levels for the two groups or possibly information on the potential gap between the two mtds

projection pursuit mixture density estimation in this paper we seek a gaussian mixture model gmm of an n variate probability density function usually the parameters of gmms are determined in the original n dimensional space by optimizing a maximum likelihood ml criterion a practical deficiency of this method of fitting gmms is its poor performance when dealing with high dimensional data since a large sample size is needed to match the accuracy that is possible in low dimensions we propose a method for fitting the gmm based on the projection pursuit strategy this gmm is highly constrained and hence its ability to model structure in subspaces is enhanced compared to a direct ml fitting of a gmm in high dimensions our method is closely related to recently developed independent factor analysis ifa mixture models the comparisons with ml fitting of gmm in n dimensions and ifa mixtures show that the proposed method is an attractive choice for fitting gmms using small sizes of training sets

blind source separation using temporal predictability a measure of temporal predictability is defined and used to separate linear mixtures of signals given any set of statistically independent source signals it is conjectured here that a linear mixture of those signals has the following property the temporal predictability of any signal mixture is less than or equal to that of any of its component source signals it is shown that this property can be used to recover source signals from a set of linear mixtures of those signals by finding an un mixing matrix that maximizes a measure of temporal predictability for each recovered signal this matrix is obtained as the solution to a generalized eigenvalue problem such problems have scaling characteristics of o n3 where n is the number of signal mixtures in contrast to independent component analysis the temporal predictability method requires minimal assumptions regarding the probability density functions of source signals it is demonstrated that the method can separate signal mixtures in which each mixture is a linear combination of source signals with supergaussian subgaussian and gaussian probability density functions and on mixtures of voices and music

projection pursuit density estimation abstract the projection pursuit methodology is applied to the multivariate density estimation problem the resulting nonparametric procedure is often less biased than the kernel and near neighbor methods in addition graphical information is produced that can be used to help gain geometric insight into the multivariate data distribution

use of bias term in projection pursuit learning improves approximation and convergence properties in a regression problem one is given a multidimensional random vector x the components of which are called predictor variables and a random variable y called response a regression surface describes a general relationship between x and y a nonparametric regression technique that has been successfully applied to high dimensional data is projection pursuit regression ppr the regression surface is approximated by a sum of empirically determined univariate functions of linear combinations of the predictors projection pursuit learning ppl formulates ppr using a 2 layer feedforward neural network the smoothers in ppr are nonparametric whereas those in ppl are based on hermite functions of some predefined highest order r we demonstrate that ppl networks in the original form do not have the universal approximation property for any finite r and thus cannot converge to the desired function even with an arbitrarily large number of hidden units but by including a bias term in each linear projection of the predictor variables ppl networks can regain these capabilities independent of the exact choice of r experimentally it is shown in this paper that this modification increases the rate of convergence with respect to the number of hidden units improves the generalization performance and makes it less sensitive to the setting of r finally we apply ppl to chaotic time series prediction and obtain superior results compared with the cascade correlation architecture

a rotation test to verify latent structure in multivariate regression models we have the opportunity to look for hidden structure unrelated to the observed predictors however when one fits a model involving such latent variables it is important to be able to tell if the structure is real or just an artifact of correlation in the regression errors we develop a new statistical test based on random rotations for verifying the existence of latent variables the rotations are carefully constructed to rotate orthogonally to the column space of the regression model we find that only non gaussian latent variables are detectable a finding that parallels a well known phenomenon in independent components analysis we base our test on a measure of non gaussianity in the histogram of the principal eigenvector components instead of on the eigenvalue the method finds and verifies some latent dichotomies in the microarray data from the agemap consortium

a family of nonparametric density estimation algorithms a new methodology for density estimation is proposed the methodology which builds on the one developed by tabak and vanden eijnden normalizes the data points through the composition of simple maps the parameters of each map are determined through the maximization of a local quadratic approximation to the log likelihood various candidates for the elementary maps of each step are proposed criteria for choosing one includes robustness computational simplicity and good behavior in high dimensional settings a good choice is that of localized radial expansions which depend on a single parameter all the complexity of arbitrary possibly convoluted probability densities can be built through the composition of such simple maps 2012 wiley periodicals inc

approximating conditional distribution functions using dimension reduction motivated by applications to prediction and forecasting we sug gest methods for approximating the conditional distribution function of a random variable y given a dependent random d vector x the idea is to estimate not the distribution of y x but that of y t x where the unit vectoris selected so that the approximation is opti mal under a least squares criterion we show thatmay be estimated root n consistently furthermore estimation of the conditional distri bution function of y givent x has the same first order asymptotic properties that it would enjoy ifwere known the proposed method is illustrated using both simulated and real data examples showing its effectiveness for both independent datasets and data from time series numerical work corroborates the theoretical result thatcan be estimated particularly accurately

self organization of symmetry networks transformation invariance from the spontaneous symmetry breaking mechanism symmetry networks use permutation symmetries among synaptic weights to achieve transformation invariant response this article proposes a generic mechanism by which such symmetries can develop during unsupervised adaptation it is shown analytically that spontaneous symmetry breaking can result in the discovery of unknown invariances of the data s probability distribution it is proposed that a role of sparse coding is to facilitate the discovery of statistical invariances by this mechanism it is demonstrated that the statistical dependences that exist between simplecell like threshold feature detectors when exposed to temporally uncorrelated natural image data can drive the development of complex cell like invariances via single cell hebbian adaptation a single learning rule can generate both simple cell like and complex cell like receptive fields

a unifying information theoretic framework for independent component analysis abstract we show that different theories recently proposed for independent component analysis ica lead to the same iterative learning algorithm for blind separation of mixed independent sources we review those theories and suggest that information theory can be used to unify several lines of research pearlmutter and parra 1 and cardoso 2 showed that the infomax approach of bell and sejnowski 3 and the maximum likelihood estimation approach are equivalent we show that negentropy maximization also has equivalent properties and therefore all three approaches yield the same learning rule for a fixed nonlinearity girolami and fyfe 4 have shown that the nonlinear principal component analysis pca algorithm of karhunen and joutsensalo 5 and oja 6 can also be viewed from information theoretic principles since it minimizes the sum of squares of the fourth order marginal cumulants and therefore approximately minimizes the mutual information 7 lambert 8 has proposed different bussgang cost functions for multichannel blind deconvolution we show how the bussgang property relates to the infomax principle finally we discuss convergence and stability as well as future research issues in blind source separation

a design for phase i trials in completely or partially ordered groups we propose a design for dose finding for cytotoxic agents in completely or partially ordered groups of patients by completely ordered groups we mean that prior to the study there is clinical information that would indicate that for a given dose the groups can be ordered with respect to the probability of toxicity at that dose with partially ordered groups at a given dose only some of the groups can be ordered with respect to the probability of toxicity at that dose the method we propose includes elements of the parametric model used in the continual reassessment method combined with the hwang peddada order restricted estimation procedure we evaluate the operating characteristics of these designs in a family of dose toxicity curves representing complete and partial orders copyright 2017 john wiley sons ltd

tail probabilities of the maxima of multilinear forms and their applications let z be a k way array consisting of independent standard normal variables for column vectors h1 hk define a multilinear form of degree k by h 1 otimes cdots otimes h k vec z we derive formulas for upper tail probabilities of the maximum of a multilinear form with respect to the hi s under the condition that the hi s are unit vectors and of its standardized statistic obtained by dividing by the norm of z we also give formulas for the maximum of a symmetric multilinear form h 1 otimes cdots otimes h k sym z where sym z denotes the symmetrization of z with respect to indices these classes of statistics are used for testing hypotheses in the analysis of variance of multiway layout data and for testing multivariate normality in order to derive the tail probabilities we employ a geometric approach developed by hotelling weyl and sun upper and lower bounds for the tail probabilities are given by reexamining sun s results some numerical examples are given to illustrate the practical usefulness of the obtained formulas including the upper and lower bounds

cluster identification using projections this article describes a procedure to identify clusters in multivariate data using information obtained from the univariate projections of the sample data onto certain directions the directions are chosen as those that minimize and maximize the kurtosis coefficient of the projected data it is shown that under certain conditions these directions provide the largest separation for the different clusters the projected univariate data are used to group the observations according to the values of the gaps or spacings between consecutive ordered observations these groupings are then combined over all projection directions the behavior of the method is tested on several examples and compared to k means mclust and the procedure proposed by jones and sibson in 1987 the proposed algorithm is iterative affine equivariant flexible robust to outliers fast to implement and seems to work well in practice

linear discriminant analysis for two classes via removal of classification structure a new method for two class linear discriminant analysis called removal of classification structure is proposed its novelty lies in the transformation of the data along an identified discriminant direction into data without discriminant information and iteration to obtain the next discriminant direction it is free to search for discriminant directions oblique to each other and ensures that the informative directions already found will not be chosen again at a later stage the efficacy of the method is examined for two discriminant criteria studies with a wide spectrum of synthetic data sets and a real data set indicate that the discrimination quality of these criteria can be improved by the proposed method

on the combination of supervised and unsupervised learning abstract the bias variance dilemma is addressed in the context of neural networks a bias constraint based on prior knowledge about the underlying distribution of the data is discussed as a means for reducing the overall error measure of a classifier

review papers recent developments in nonparametric density estimation abstract advances in computation and the fast and cheap computational facilities now available to statisticians have had a significant impact upon statistical research and especially the development of nonparametric data analysis procedures in particular theoretical and applied research on nonparametric density estimation has had a noticeable influence on related topics such as nonparametric regression nonparametric discrimination and nonparametric pattern recognition this article reviews recent developments in nonparametric density estimation and includes topics that have been omitted from review articles and books on the subject the early density estimation methods such as the histogram kernel estimators and orthogonal series estimators are still very popular and recent research on them is described different types of restricted maximum likelihood density estimators including order restricted estimators maximum penalized likelihood estimators and sieve estimators are discussed where re

exploratory projection pursuit abstract a new projection pursuit algorithm for exploring multivariate data is presented that has both statistical and computational advantages over previous methods a number of practical issues concerning its application are addressed a connection to multivariate density estimation is established and its properties are investigated through simulation studies and application to real data the goal of exploratory projection pursuit is to use the data to find low one two or three dimensional projections that provide the most revealing views of the full dimensional data with these views the human gift for pattern recognition can be applied to help discover effects that may not have been anticipated in advance since linear effects are directly captured by the covariance structure of the variable pairs which are straightforward to estimate the emphasis here is on the discovery of nonlinear effects such as clustering or other general nonlinear associations among the variables although arbitrary

multimodality exploration by an unsupervised projection pursuit neural network graphical inspection of multimodality is demonstrated using unsupervised lateral inhibition neural networks three projection pursuit indexes are compared on low dimensional simulated and real world data principal components legendre polynomial and projection pursuit network

the nonlinear statistics of high contrast patches in natural images recently there has been a great deal of interest in modeling the non gaussian structures of natural images however despite the many advances in the direction of sparse coding and multi resolution analysis the full probability distribution of pixels values in a neighborhood has not yet been described in this study we explore the space of data points representing the values of 3 3 high contrast patches from optical and 3d range images we find that the distribution of data is extremely sparse with the majority of the data points concentrated in clusters and non linear low dimensional manifolds furthermore a detailed study of probability densities allows us to systematically distinguish between images of different modalities optical versus range which otherwise display similar marginal distributions our work indicates the importance of studying the full probability distribution of natural images not just marginals and the need to understand the intrinsic dimensionality and nature of the data we believe that object like structures in the world and the sensor properties of the probing device generate observations that are concentrated along predictable shapes in state space our study of natural image statistics accounts for local geometries such as edges in natural scenes but does not impose such strong assumptions on the data as independent components or sparse coding by linear change of bases

on the link between l1 pca and ica principal component analysis pca based on l1 norm maximization is an emerging technique that has drawn growing interest in the signal processing and machine learning research communities especially due to its robustness to outliers the present work proves that l1 norm pca can perform independent component analysis ica under the whitening assumption however when the source probability distributions fulfil certain conditions the l1 norm criterion needs to be minimized rather than maximized which can be accomplished by simple modifications on existing optimal algorithms for l1 pca if the sources have symmetric distributions we show in addition that l1 pca is linked to kurtosis optimization a number of numerical experiments illustrate the theoretical results and analyze the comparative performance of different algorithms for ica via l1 pca although our analysis is asymptotic in the sample size this equivalence opens interesting new perspectives for performing ica using optimal algorithms for l1 pca with guaranteed global convergence while inheriting the increased robustness to outliers of the l1 norm criterion

moments of random variables a systems theoretic interpretation moments of continuous random variables admitting a probability density function are studied we show that under certain assumptions the moments of a random variable can be characterized in terms of a sylvester equation and of the steady state output response of a specific interconnected system this allows to interpret well known notions and results of probability theory and statistics in the language of systems theory including the sum of independent random variables the notion of mixture distribution and results from renewal theory the theory developed is based on tools from center manifold theory the theory of the steady state response of nonlinear systems and the theory of output regulation our formalism is illustrated by means of several examples and can be easily adapted to the case of discrete and multivariate random variables

differential log likelihood for evaluating and learning gaussian mixtures we introduce a new unbiased metric for assessing the quality of density estimation based on gaussian mixtures called differential log likelihood as an application we determine the optimal smoothness and the optimal number of kernels in gaussian mixtures furthermore we suggest a learning strategy for gaussian mixture density estimation and compare its performance with log likelihood maximization for a wide range of real world data sets

when is the range of a multiplier on a banach algebra closed in this paper we prove the theorem let a be a banach algebra with a bounded approximate identity bai such that every proper closed ideal of a is contained in a proper closed ideal with a bai then a multiplier t a a has a closed range iff t factors as a product of an idempotent multiplier and an invertible multiplier

a characterization of the closed unital ideals of the fourier stieltjes algebra b g of a locally compact amenable group g abstract let g be a locally compact amenable group b g its fourier stieltjes algebra and i be a closed ideal of it in this paper we prove the following result the ideal i has a unit element iff it is principal this is the noncommutative version of the glicksberg host parreau theorem the paper also contains an abstract version of this theorem

topological centers of certain dual algebras let a be a banach algebra with a bounded approximate identity let z1 and z2 be respectively the topological centers of the algebras a and a a in this paper for weakly sequentially complete banach algebras in particular for the group and fourier algebras l1 g and a g we study the sets z1 z2 the relations between them and with several other subspaces of a or a

fast and robust fixed point algorithms for independent component analysis independent component analysis ica is a statistical method for transforming an observed multidimensional random vector into components that are statistically as independent from each other as possible we use a combination of two different approaches for linear ica comon s information theoretic approach and the projection pursuit approach using maximum entropy approximations of differential entropy we introduce a family of new contrast functions for ica these contrast functions enable both the estimation of the whole decomposition by minimizing mutual information and estimation of individual independent components as projection pursuit directions the statistical properties of the estimators based on such contrast functions are analyzed under the assumption of the linear mixture model and it is shown how to choose contrast functions that are robust and or of minimum variance finally we introduce simple fixed point algorithms for practical optimization of the contrast functions

learning active basis model for object detection and recognition this article proposes an active basis model a shared sketch algorithm and a computational architecture of sum max maps for representing learning and recognizing deformable templates in our generative model a deformable template is in the form of an active basis which consists of a small number of gabor wavelet elements at selected locations and orientations these elements are allowed to slightly perturb their locations and orientations before they are linearly combined to generate the observed image the active basis model in particular the locations and the orientations of the basis elements can be learned from training images by the shared sketch algorithm the algorithm selects the elements of the active basis sequentially from a dictionary of gabor wavelets when an element is selected at each step the element is shared by all the training images and the element is perturbed to encode or sketch a nearby edge segment in each training image the recognition of the deformable template from an image can be accomplished by a computational architecture that alternates the sum maps and the max maps the computation of the max maps deforms the active basis to match the image data and the computation of the sum maps scores the template matching by the log likelihood of the deformed active basis

predictions of the spontaneous symmetry breaking theory for visual code completeness and spatial scaling in single cell learning rules this article shows analytically that single cell learning rules that give rise to oriented and localized receptive fields when their synaptic weights are randomly and independently initialized according to a plausible assumption of zero prior information will generate visual codes that are invariant under two dimensional translations rotations and scale magnifications provided that the statistics of their training images are sufficiently invariant under these transformations such codes span different image locations orientations and size scales with equal economy thus single cell rules could account for the spatial scaling property of the cortical simple cell code this prediction is tested computationally by training with natural scenes it is demonstrated that a single cell learning rule can give rise to simple cell receptive fields spanning the full range of orientations image locations and spatial frequencies except at the extreme high and low frequencies at which the scale invariance of the statistics of digitally sampled images must ultimately break down because of the image boundary and the finite pixel resolution thus no constraint on completeness or any other coupling between cells is necessary to induce the visual code to span wide ranges of locations orientations and size scales this prediction is made using the theory of spontaneous symmetry breaking which we have previously shown can also explain the data driven self organization of a wide variety of transformation invariances in neurons responses such as the translation invariance of complex cell response

complexity pursuit separating interesting components from time series a generalization of projection pursuit for time series that is signals with time structure is introduced the goal is to find projections of time series that have interesting structure defined using criteria related to kolmogoroff complexity or coding length interesting signals are those that can be coded with a short code length we derive a simple approximation of coding length that takes into account both the nongaussianity and the autocorrelations of the time series also we derive a simple algorithm for its approximative optimization the resulting method is closely related to blind separation of nongaussian time dependent source signals

robust singular value decompositions a new approach to projection pursuit abstract robust location and covariance estimators are developed via general m estimation for covariance matrix eigenvectors and eigenvalues the solution to this gm estimation problem is obtained by transforming it into a series of robust regression problems based on a new algorithm for the singular value decomposition it is shown here that the singular value decomposition can be represented as an iteration of two steps a least squares regression fit of the data matrix followed by a rotation to the regression hyperplanes an algorithm to obtain the solution to this gm estimation problem is presented along with results of a monte carlo study and examples of its application in addition it is shown how the output of this algorithm can be used to numerically search for multivariate outliers which is especially useful in exploratory data analysis with high dimensional data and large sample sizes where standard graphical techniques are difficult to implement because the algorithm computes robust estima

edgeworth expanded gaussian mixture density modeling instead of increasing the order of the edgeworth expansion of a single gaussian kernel we suggest using mixtures of edgeworth expanded gaussian kernels of moderate order we introduce a simple closed form solution for estimating the kernel parameters based on weighted moment matching furthermore we formulate the extension to the multivariate case which is not always feasible with algebraic density approximation procedures

robust projection indices loosely speaking a robust projection index is one that prefers projections involving true clusters over projections consisting of a cluster and an outlier we introduce a mathematical definition of one dimensional index robustness and describe a numerical experiment to measure it we design five new indices based on measuring divergence from student s t distribution which are intended to be especially robust the experiment shows that they are more robust than several established indices the experiment also reveals more generally that the robustness of moment indices depends on the number of approximation terms providing additional practical guidance for existing projection pursuit implementations we investigate the theoretical properties of one new student t index and hall s index and show that the new index automatically adapts its robustness to the degree of outlier contamination we conclude by outlining the possibilities for extending our experiments to both higher dimensions and other new indices

on the forward and backward algorithms of projection pursuit this article provides a historic review of the forward and backward projection pursuit algorithms previously thought to be equivalent and points out an important difference between the two in doing so a small error in the original exploratory projection pursuit article by friedman j amer statist assoc 82 1987 249 266 is corrected the implication of the difference is briefly discussed in the context of an application in which projection pursuit density estimation is used as a building block for nonparametric discriminant analysis

principal information theoretic approaches 2000 principal information theoretic approaches journal of the american statistical association vol 95 no 452 pp 1349 1353

a unified model for probabilistic principal surfaces principal curves and surfaces are nonlinear generalizations of principal components and subspaces respectively they can provide insightful summary of high dimensional data not typically attainable by classical linear methods solutions to several problems such as proof of existence and convergence faced by the original principal curve formulation have been proposed in the past few years nevertheless these solutions are not generally extensible to principal surfaces the mere computation of which presents a formidable obstacle consequently relatively few studies of principal surfaces are available we previously 2000 proposed the probabilistic principal surface pps to address a number of issues associated with current principal surface algorithms pps uses a manifold oriented covariance noise model based on the generative topographical mapping gtm which can be viewed as a parametric formulation of kohonen s self organizing map building on the pps we introduce a unified covariance model that implements pps 0 1 by varying the clamping parameter spl alpha then we comprehensively evaluate the empirical performance of pps gtm and the manifold aligned gtm on three popular benchmark data sets it is shown in two different comparisons that the pps outperforms the gtm under identical parameter settings convergence of the pps is found to be identical to that of the gtm and the computational overhead incurred by the pps decreases to 40 percent or less for more complex manifolds these results show that the generalized pps provides a flexible and effective way of obtaining principal surfaces

consistency of continual reassessment method under model misspecification summary this paper studies large sample behaviour of the continual reassessment method used to determine the target dose levels in phase i clinical trials a sequential maximum likelihood procedure is investigated for local estimation of a given targeted toxicity level consistency and asymptotic normality for the maximum likelihood estimates are established under model misspecification and it is shown that in general the recommended dose level will converge to the target level there are however situations in which the recommended dose may converge to a level close to the target level but not necessarily the closest this contrasts with the conclusions of o quigley pepe fisher 1990 and we describe this situation as well as providing the conditions under which convergence is assured

maximum likelihood topographic map formation we introduce a new unsupervised learning algorithm for kernel based topographic map formation of heteroscedastic gaussian mixtures that allows for a unified account of distortion error vector quantization log likelihood and kullback leibler divergence

estimating the direction in which a data set is most interesting estimation of orientation is a key operation at each step in projection pursuit since projection pursuit is a nonparametric algorithm and since even low dimensional approximations to the target function must converge to their limits at rates considerably slower than n 1 2 where n is sample size then it might be thought that the same is true of orientation estimates it is shown in the present paper that this is not the case and that estimation of orientation is a parametric operation in the sense that under mild nonparametric assumptions correctly chosen kernel type orientation estimates converge to their limits at rate n 1 2 this property is not enjoyed by standard projection pursuit orientation estimates which converge at a slower rate than n 1 2 most attention in the present paper is focussed on the case of projection pursuit density approximation but it is pointed out that our arguments hold generally an important practical conclusion is that data should be smoothed less when estimating orientation than when constructing the final projection pursuit approximation

projection pursuit in high dimensions projection pursuit is a classical exploratory data analysis method to detect interesting low dimensional structures in multivariate data originally projection pursuit was applied mostly to data of moderately low dimension motivated by contemporary applications we here study its properties in high dimensional settings specifically we analyze the asymptotic properties of projection pursuit on structureless multivariate gaussian data with an identity covariance as both dimension p and sample size n tend to infinity with p n 0 our main results are that i if then there exist projections whose corresponding empirical cumulative distribution function can approximate any arbitrary distribution and ii if 0 not all limiting distributions are possible however depending on the value of various non gaussian distributions may still be approximated in contrast if we restrict to sparse projections involving only a few of the p variables then asymptotically all empirical cumulative distribution functions are gaussian and iii if 0 then asymptotically all projections are gaussian some of these results extend to mean centered sub gaussian data and to projections into k dimensions hence in the small n large p setting unless sparsity is enforced and regardless of the chosen projection index projection pursuit may detect an apparent structure that has no statistical significance furthermore our work reveals fundamental limitations on the ability to detect non gaussian signals in high dimensional data in particular through independent component analysis and related non gaussian component analysis

blind source separation by sparse decomposition in a signal dictionary the blind source separation problem is to extract the underlying source signals from a set of linear mixtures where the mixing matrix is unknown this situation is common in acoustics radio medical signal and image processing hyperspectral imaging and other areas we suggest a two stage separation process a priori selection of a possibly overcomplete signal dictionary for instance a wavelet frame or a learned dictionary in which the sources are assumed to be sparsely representable followed by unmixing the sources by exploiting the their sparse representability we consider the general case of more sources than mixtures but also derive a more efficient algorithm in the case of a nonovercomplete dictionary and an equal numbers of sources and mixtures experiments with artificial signals and musical sounds demonstrate significantly better separation than other known techniques

limitations of nonlinear pca as performed with generic neural networks kramer s 1991 nonlinear principal components analysis nlpca neural networks are feedforward autoassociative networks with five layers the third layer has fewer nodes than the input or output layers this paper proposes a geometric interpretation for kramer s method by showing that nlpca fits a lower dimensional curve or surface through the training data the first three layers project observations onto the curve or surface giving scores the last three layers define the curve or surface the first three layers are a continuous function which we show has several implications nlpca projections are suboptimal producing larger approximation error nlpca is unable to model curves and surfaces that intersect themselves and nlpca cannot parameterize curves with parameterizations having discontinuous jumps we establish results on the identification of score values and discuss their implications on interpreting score values we discuss the relationship between nlpca and principal curves and surfaces another nonlinear feature extraction method

linear dimensionality reduction survey insights and generalizations linear dimensionality reduction methods are a cornerstone of analyzing high dimensional data due to their simple geometric interpretations and typically attractive computational properties these methods capture many data features of interest such as covariance dynamical structure correlation between data sets input output relationships and margin between data classes methods have been developed with a variety of names and motivations in many fields and perhaps as a result the connections between all these methods have not been highlighted here we survey methods from this disparate literature as optimization programs over matrix manifolds we discuss principal component analysis factor analysis linear multidimensional scaling fisher s linear discriminant analysis canonical correlations analysis maximum autocorrelation factors slow feature analysis sufficient dimensionality reduction undercomplete independent component analysis linear regression distance metric learning and more this optimization framework gives insight to some rarely discussed shortcomings of well known methods such as the suboptimality of certain eigenvector solutions modern techniques for optimization over matrix manifolds enable a generic linear dimensionality reduction solver which accepts as input data and an objective to be optimized and returns as output an optimal low dimensional projection of the data this simple optimization framework further allows straightforward generalizations and novel variants of classical methods which we demonstrate here by creating an orthogonal projection canonical correlations analysis more broadly this survey and generic solver suggest that linear dimensionality reduction can move toward becoming a blackbox objective agnostic numerical technology

robust estimation of dispersion matrices and principal components abstract this paper uses monte carlo methods to compare the performances of several robust procedures for estimating a correlation matrix and its principal components the estimators are formed either from separate bivariate analyses or by simultaneous manipulation of all variables by using techniques such as multivariate trimming and m estimation the m estimators stand up exceptionally well they and the multivariate trimming procedure are especially effective at estimating the principal components including a near singularity however the m estimators can break down relatively easily when the dimensionality is large and the outliers are asymmetric with missing data the element wise approach becomes more attractive

a new method for one dimensional linear feature transformations abstract we propose a method for finding a linear transformation of an initial pattern space into a one dimensional new space which optimizes the l2 distance between density functions we use an orthogonal expansion with hermite functions to compute the criterion we discuss both the truncation and the statistical variation error for this expansion to avoid false local optima caused by sample variation we choose the starting point with a coarse step optimization method experimental results with two class and multi class both unimodal and multimodal are presented

on principal hessian directions for data visualization and dimension reduction another application of stein s lemma abstract modern graphical tools have enhanced our ability to learn many things from data directly with much user friendly graphical software available we are encouraged to plot a lot more often than before the benefits from direct interaction with graphics have been enormous but trailing behind these high tech advances is the issue of appropriate guidance on what to plot there are too many directions to project a high dimensional data set and unguided plotting can be time consuming and fruitless in a recent article li set up a statistical framework for study on this issue based on a notion of effective dimension reduction edr directions they are the directions to project a high dimensional input variable for the purpose of effectively viewing and studying its relationship with an output variable a methodology sliced inverse regression was introduced and shown to be useful in finding edr directions this article introduces another method for finding edr directions it begins with the observat

bivariate isotonic design for dose finding with ordered groups we consider the problem of dose finding where subjects can be stratified into two populations with possibly different susceptibility to toxicity the goal is to find the maximally tolerated dose for each population we propose a non parametric design for this problem toxicity is estimated using the bivariate isotonic regression estimator the new design is compared with the two sample continual reassessment method

centroidal voronoi tessellations applications and algorithms a centroidal voronoi tessellation is a voronoi tessellation whose generating points are the centroids centers of mass of the corresponding voronoi regions we give some applications of such tessellations to problems in image compression quadrature finite difference methods distribution of resources cellular biology statistics and the territorial behavior of animals we discuss methods for computing these tessellations provide some analyses concerning both the tessellations and the methods for their determination and finally present the results of some numerical experiments