selection of variables in cluster analysis an empirical comparison of eight procedures eight different variable selection techniques for model based and non model based clustering are evaluated across a wide range of cluster structures it is shown that several methods have difficulties when non informative variables i e random noise are included in the model furthermore the distribution of the random noise greatly impacts the performance of nearly all of the variable selection procedures overall a variable selection technique based on a variance to range weighting procedure coupled with the largest decreases in within cluster sums of squares error performed the best on the other hand variable selection methods used in conjunction with finite mixture models performed the worst

machine learning and nonparametric bandit theory in its most basic form bandit theory is concerned with the design problem of sequentially choosing members from a given collection of random variables so that the regret i e r sub n spl sigma sub j spl mu spl mu sub j et sub n j grows as slowly as possible with increasing n here spl mu sub j is the expected value of the bandit arm i e random variable indexed by j t sub n j is the number of times arm j has been selected in the first n decision stages and spl mu sup sup sub j spl mu sub j the present paper contributes to the theory by considering the situation in which observations are dependent to begin with the dependency is presumed to depend only on past observations of the same arm but later we allow that it may be with respect to the entire past and that the set of arms is infinite this brings queues and more generally controlled markov processes into our purview thus our black box methodology is suitable for the case when the only observables are cost values and in particular the probability structure and loss function are unknown to the designer the conclusion of the analysis is that under lenient conditions using algorithms prescribed herein risk growth is commensurate with that in the simplest i i d cases our methods represent an alternative to stochastic approximation perturbation analysis ideas for tuning queues

svm decision boundary based discriminative subspace induction we study the problem of linear dimension reduction for classification with a focus on sufficient dimension reduction i e finding subspaces without loss of discrimination power first we formulate the concept of sufficient subspace for classification in parallel terms as for regression then we present a new method to estimate the smallest sufficient subspace based on an improvement of decision boundary analysis dba the main idea is to combine dba with support vector machines svm to overcome the inherent difficulty of dba in small sample size situations while keeping dba s estimation simplicity the compact representation of svm boundary results in a significant gain in both speed and accuracy over previous dba implementations alternatively this technique can be viewed as a way to reduce the run time complexity of svm itself comparative experiments on one simulated and four real world benchmark datasets highlight the superior performance of the proposed approach

nonlinear modeling of scattered multivariate data and its application to shape change we are given a set of points in a space of high dimension for instance this set may represent many visual appearances of an object a face or a hand we address the problem of approximating this set by a manifold in order to have a compact representation of the object appearance when the scattering of this set is approximately an ellipsoid then the problem has a well known solution given by principal components analysis pca however in some situations like object displacement learning or face learning this linear technique may be ill adapted and nonlinear approximation has to be introduced the method we propose can be seen as a nonlinear pca nlpca the main difficulty being that the data are not ordered we propose an index which favors the choice of axes preserving the closest point neighborhoods these axes determine an order for visiting all the points when smoothing finally a new criterion called generalization error is introduced to determine the smoothing rate that is the knot number for the spline fitting experimental results conclude this paper the method is tested on artificial data and on two databases used in visual learning

kernel methods in machine learning we review machine learning methods employing positive definite kernels these methods formulate learning and estimation problems in a reproducing kernel hilbert space rkhs of functions defined on the data domain expanded in terms of a kernel working in linear spaces of function has the benefit of facilitating the construction and analysis of learning algorithms while at the same time allowing large classes of functions the latter include nonlinear functions as well as functions defined on nonvectorial data we cover a wide range of methods ranging from binary classifiers to sophisticated methods for estimation with structured data

the effect of noise on a class of energy based learning rules we study the selectivity properties of neurons based on bcm and kurtosis energy functions in a general case of noisy high dimensional input space the proposed approach which is used for characterization of the stable states can be generalized to a whole class of energy functions we characterize the critical noise levels beyond which the selectivity is destroyed we also perform a quantitative analysis of such transitions which shows interesting dependency on data set size we observe that the robustness to noise of the bcm neuron bienenstock cooper munro 1982 intrator cooper 1992 increases as a function of dimensionality we explicitly compute the separability limit of bcm and kurtosis learning rules in the case of a bimodal input distribution numerical simulations show a stronger robustness of the bcm rule for practical data set size when compared with kurtosis

global stochastic optimization with low dispersion point sets this study concerns a generic model free stochastic optimization problem requiring the minimization of a risk function defined on a given bounded domain in a euclidean space smoothness assumptions regarding the risk function are hypothesized and members of the underlying space of probabilities are presumed subject to a large deviation principle however the risk function may well be nonconvex and multimodal a general approach to finding the risk minimizer on the basis of decision observation pairs is proposed it consists of repeatedly observing pairs over a collection of design points principles are derived for choosing the number of these design points on the basis of an observation budget and for allocating the observations between these points in both prescheduled and adaptive settings on the basis of these principles large deviation type bounds of the minimizer in terms of sample size are established

significance levels in exploratory projection pursuit summary in exploratory data analysis projection pursuit methods explore the nonlinear structure of high dimensional data it is useful to have a significance test to help us decide whether apparent structure is real or just the effect of noise monte carlo methods can be helpful to achieve this but unfortunately in this case they are computationally expensive in this paper under a suitable null hypothesis we derive a theoretical approximation for the p value associated with friedman s projection pursuit index friedman 1987 the result of monte carlo simulations is compared with our analytical result some practical aspects are discussed

what is the goal of sensory coding a number of recent attempts have been made to describe early sensory coding in terms of a general information processing strategy in this paper two strategies are contrasted both strategies take advantage of the redundancy in the environment to produce more effective representations the first is described as a compact coding scheme a compact code performs a transform that allows the input to be represented with a reduced number of vectors cells with minimal rms error this approach has recently become popular in the neural network literature and is related to a process called principal components analysis pca a number of recent papers have suggested that the optimal compact code for representing natural scenes will have units with receptive field profiles much like those found in the retina and primary visual cortex however in this paper it is proposed that compact coding schemes are insufficient to account for the receptive field properties of cells in the mammalian visual pathway in contrast it is proposed that the visual system is near to optimal in representing natural scenes only if optimality is defined in terms of sparse distributed coding in a sparse distributed code all cells in the code have an equal response probability across the class of images but have a low response probability for any single image in such a code the dimensionality is not reduced rather the redundancy of the input is transformed into the redundancy of the firing pattern of cells it is proposed that the signature for a sparse code is found in the fourth moment of the response distribution i e the kurtosis in measurements with 55 calibrated natural scenes the kurtosis was found to peak when the bandwidths of the visual code matched those of cells in the mammalian visual cortex codes resembling wavelet transforms are proposed to be effective because the response histograms of such codes are sparse i e show high kurtosis when presented with natural scenes it is proposed that the structure of the image that allows sparse coding is found in the phase spectrum of the image it is suggested that natural scenes to a first approximation can be considered as a sum of self similar local functions the inverse of a wavelet possible reasons for why sensory systems would evolve toward sparse coding are presented

data clustering using a model granular magnet we present a new approach to clustering based on the physical properties of an inhomogeneous ferromagnet no assumption is made regarding the underlying distribution of the data we assign a potts spin to each data point and introduce an interaction between neighboring points whose strength is a decreasing function of the distance between the neighbors this magnetic system exhibits three phases at very low temperatures it is completely ordered all spins are aligned at very high temperatures the system does not exhibit any ordering and in an intermediate regime clusters of relatively strongly coupled spins become ordered whereas different clusters remain uncorrelated this intermediate phase is identified by a jump in the order parameters the spin spin correlation function is used to partition the spins and the corresponding data points into clusters we demonstrate on three synthetic and three real data sets how the method works detailed comparison to the performance of other techniques clearly indicates the relative success of our method

an adaptive huber method for non linear systems of weakly singular second kind volterra integral equations abstract numerical methods for systems of weakly singular volterra integral equations are rarely considered in the literature especially if the equations involve non linear dependencies between unknowns and their integrals in the present work an adaptive huber method for such systems is proposed by extending the method previously formulated for single weakly singular second kind volterra equations the method is tested on example systems of integral equations involving integrals with kernels k t t 1 2 k t exp t t 1 2 where 0 and k t 1 the magnitude of the errors and practical accuracy orders observed for ie systems are comparable to those for single ies in cases when the solution vector is not differentiable at t 0 the estimation of errors at t 0 is found somewhat less reliable for ie systems than it was for single ies the stability of the ie systems solved appears to be sufficient in practice for the numerical stability of the method

statistical principles in image modeling images of natural scenes contain a rich variety of visual patterns to learn and recognize these patterns from natural images it is necessary to construct statistical models for these patterns in this review article we describe three statistical principles for modeling image patterns the sparse coding principle the minimax entropy principle and the meaningful alignment principle we explain these three principles and their relationships in the context of modeling images as compositions of gabor wavelets these three principles correspond to three regimes of composition patterns of gabor wavelets and these three regimes are connected by changes in scale or resolution

sliced inverse regression for dimension reduction abstract modern advances in computing power have greatly widened scientists scope in gathering and investigating information from many variables information which might have been ignored in the past yet to effectively scan a large pool of variables is not an easy task although our ability to interact with data has been much enhanced by recent innovations in dynamic graphics in this article we propose a novel data analytic tool sliced inverse regression sir for reducing the dimension of the input variable x without going through any parametric or nonparametric model fitting process this method explores the simplicity of the inverse view of regression that is instead of regressing the univariate output variable y against the multivariate x we regress x against y forward regression and inverse regression are connected by a theorem that motivates this method the theoretical properties of sir are investigated under a model of the form y f 1 x k x e where the k s are the unknown

newton like extremum seeking for the control of thermoacoustic instability in practice the convergence rate and stability of perturbation based extremum seeking schemes can be very sensitive to the curvature of the plant map an example of this can be seen in the use of extremum seeking to reduce the amplitude of thermoacoustic oscillations in premixed gas turbine combustors this sensitivity to the plant map curvature arises from the use of a gradient descent adaptation algorithm such extremum seeking schemes may need to be conservatively tuned in order to maintain stability over a wide range of operating conditions resulting in slower optimization than could be achieved for a fixed operating condition this can severely reduce the effectiveness of perturbation based extremum seeking schemes in some applications in this paper a sinusoidally perturbed extremum seeking scheme using a newton like step is developed non local stability results for the scheme are formulated using a lyapunov analysis a local analysis of the scheme is given to investigate the influence of plant dynamics and to show that the local rate of convergence is independent of the plant map curvature the benefit of this plant map curvature independence is then experimentally demonstrated in minimizing the thermoacoustic oscillations in a model premixed combustor

blind source separation using temporal predictability a measure of temporal predictability is defined and used to separate linear mixtures of signals given any set of statistically independent source signals it is conjectured here that a linear mixture of those signals has the following property the temporal predictability of any signal mixture is less than or equal to that of any of its component source signals it is shown that this property can be used to recover source signals from a set of linear mixtures of those signals by finding an un mixing matrix that maximizes a measure of temporal predictability for each recovered signal this matrix is obtained as the solution to a generalized eigenvalue problem such problems have scaling characteristics of o n3 where n is the number of signal mixtures in contrast to independent component analysis the temporal predictability method requires minimal assumptions regarding the probability density functions of source signals it is demonstrated that the method can separate signal mixtures in which each mixture is a linear combination of source signals with supergaussian subgaussian and gaussian probability density functions and on mixtures of voices and music

projection pursuit density estimation abstract the projection pursuit methodology is applied to the multivariate density estimation problem the resulting nonparametric procedure is often less biased than the kernel and near neighbor methods in addition graphical information is produced that can be used to help gain geometric insight into the multivariate data distribution

efficient algorithms to solve singular integral equations of abel type in the present paper we obtain the approximate solution of abel s integral equation by using the following powerful efficient but simple methods i he s homotopy perturbation method hpm ii modified homotopy perturbation method mhpm iii adomian decomposition method adm and iv modified adomian decomposition method madm the validity and applicability of these techniques are illustrated through various particular cases which demonstrate their efficiency and simplicity in solving these types of integral equations compared with the other existing methods

use of bias term in projection pursuit learning improves approximation and convergence properties in a regression problem one is given a multidimensional random vector x the components of which are called predictor variables and a random variable y called response a regression surface describes a general relationship between x and y a nonparametric regression technique that has been successfully applied to high dimensional data is projection pursuit regression ppr the regression surface is approximated by a sum of empirically determined univariate functions of linear combinations of the predictors projection pursuit learning ppl formulates ppr using a 2 layer feedforward neural network the smoothers in ppr are nonparametric whereas those in ppl are based on hermite functions of some predefined highest order r we demonstrate that ppl networks in the original form do not have the universal approximation property for any finite r and thus cannot converge to the desired function even with an arbitrarily large number of hidden units but by including a bias term in each linear projection of the predictor variables ppl networks can regain these capabilities independent of the exact choice of r experimentally it is shown in this paper that this modification increases the rate of convergence with respect to the number of hidden units improves the generalization performance and makes it less sensitive to the setting of r finally we apply ppl to chaotic time series prediction and obtain superior results compared with the cascade correlation architecture

global random optimization by simultaneous perturbation stochastic approximation we examine the theoretical and numerical global convergence properties of a certain ldquogradient freerdquo stochastic approximation algorithm called the ldquosimultaneous perturbation stochastic approximation spsa rdquo that has performed well in complex optimization problems we establish two theorems on the global convergence of spsa the first involving the well known method of injected noise the second theorem establishes conditions under which ldquobasicrdquo spsa without injected noise can achieve convergence in probability to a global optimum a result with important practical benefits

a family of nonparametric density estimation algorithms a new methodology for density estimation is proposed the methodology which builds on the one developed by tabak and vanden eijnden normalizes the data points through the composition of simple maps the parameters of each map are determined through the maximization of a local quadratic approximation to the log likelihood various candidates for the elementary maps of each step are proposed criteria for choosing one includes robustness computational simplicity and good behavior in high dimensional settings a good choice is that of localized radial expansions which depend on a single parameter all the complexity of arbitrary possibly convoluted probability densities can be built through the composition of such simple maps 2012 wiley periodicals inc

strong bounds for multidimensional spacings let u1 u2 u n be independent random vectors uniformly distributed on 0 1 d we define the k th maximal spacing n k associated to u1 u1 as the k th maximal possible value of the length of a side of a d dimensional square block in 0 1 d which does not intersect the sample and which cannot be enlarged without doing so our main result is that n k n 1 log n 0 log2n 1 d almost surely as n other bounds are proposed for the limiting almost sure behavior of n k as n

approximating conditional distribution functions using dimension reduction motivated by applications to prediction and forecasting we sug gest methods for approximating the conditional distribution function of a random variable y given a dependent random d vector x the idea is to estimate not the distribution of y x but that of y t x where the unit vectoris selected so that the approximation is opti mal under a least squares criterion we show thatmay be estimated root n consistently furthermore estimation of the conditional distri bution function of y givent x has the same first order asymptotic properties that it would enjoy ifwere known the proposed method is illustrated using both simulated and real data examples showing its effectiveness for both independent datasets and data from time series numerical work corroborates the theoretical result thatcan be estimated particularly accurately

a unifying information theoretic framework for independent component analysis abstract we show that different theories recently proposed for independent component analysis ica lead to the same iterative learning algorithm for blind separation of mixed independent sources we review those theories and suggest that information theory can be used to unify several lines of research pearlmutter and parra 1 and cardoso 2 showed that the infomax approach of bell and sejnowski 3 and the maximum likelihood estimation approach are equivalent we show that negentropy maximization also has equivalent properties and therefore all three approaches yield the same learning rule for a fixed nonlinearity girolami and fyfe 4 have shown that the nonlinear principal component analysis pca algorithm of karhunen and joutsensalo 5 and oja 6 can also be viewed from information theoretic principles since it minimizes the sum of squares of the fourth order marginal cumulants and therefore approximately minimizes the mutual information 7 lambert 8 has proposed different bussgang cost functions for multichannel blind deconvolution we show how the bussgang property relates to the infomax principle finally we discuss convergence and stability as well as future research issues in blind source separation

adaptive search algorithms for discrete stochastic optimization a smooth best response approach this paper considers simulation based optimization of the performance of a regime switching stochastic system over a finite set of feasible configurations inspired by the stochastic fictitious play learning rules in game theory we propose an adaptive random search algorithm that uses a smooth best response sampling strategy and tracks the set of global optima yet distributes the search so that most of the effort is spent on simulating the system performance at the global optima the algorithm responds properly to the random unpredictable jumps of the global optimum even when the observations data are temporally correlated as long as a weak law of large numbers holds numerical examples show that the proposed scheme yields faster convergence and superior efficiency for finite sample lengths compared with several existing random search and pure exploration methods in the literature

on the combination of supervised and unsupervised learning abstract the bias variance dilemma is addressed in the context of neural networks a bias constraint based on prior knowledge about the underlying distribution of the data is discussed as a means for reducing the overall error measure of a classifier

exploratory projection pursuit abstract a new projection pursuit algorithm for exploring multivariate data is presented that has both statistical and computational advantages over previous methods a number of practical issues concerning its application are addressed a connection to multivariate density estimation is established and its properties are investigated through simulation studies and application to real data the goal of exploratory projection pursuit is to use the data to find low one two or three dimensional projections that provide the most revealing views of the full dimensional data with these views the human gift for pattern recognition can be applied to help discover effects that may not have been anticipated in advance since linear effects are directly captured by the covariance structure of the variable pairs which are straightforward to estimate the emphasis here is on the discovery of nonlinear effects such as clustering or other general nonlinear associations among the variables although arbitrary

the nonlinear statistics of high contrast patches in natural images recently there has been a great deal of interest in modeling the non gaussian structures of natural images however despite the many advances in the direction of sparse coding and multi resolution analysis the full probability distribution of pixels values in a neighborhood has not yet been described in this study we explore the space of data points representing the values of 3 3 high contrast patches from optical and 3d range images we find that the distribution of data is extremely sparse with the majority of the data points concentrated in clusters and non linear low dimensional manifolds furthermore a detailed study of probability densities allows us to systematically distinguish between images of different modalities optical versus range which otherwise display similar marginal distributions our work indicates the importance of studying the full probability distribution of natural images not just marginals and the need to understand the intrinsic dimensionality and nature of the data we believe that object like structures in the world and the sensor properties of the probing device generate observations that are concentrated along predictable shapes in state space our study of natural image statistics accounts for local geometries such as edges in natural scenes but does not impose such strong assumptions on the data as independent components or sparse coding by linear change of bases

moments of random variables a systems theoretic interpretation moments of continuous random variables admitting a probability density function are studied we show that under certain assumptions the moments of a random variable can be characterized in terms of a sylvester equation and of the steady state output response of a specific interconnected system this allows to interpret well known notions and results of probability theory and statistics in the language of systems theory including the sum of independent random variables the notion of mixture distribution and results from renewal theory the theory developed is based on tools from center manifold theory the theory of the steady state response of nonlinear systems and the theory of output regulation our formalism is illustrated by means of several examples and can be easily adapted to the case of discrete and multivariate random variables

fast and robust fixed point algorithms for independent component analysis independent component analysis ica is a statistical method for transforming an observed multidimensional random vector into components that are statistically as independent from each other as possible we use a combination of two different approaches for linear ica comon s information theoretic approach and the projection pursuit approach using maximum entropy approximations of differential entropy we introduce a family of new contrast functions for ica these contrast functions enable both the estimation of the whole decomposition by minimizing mutual information and estimation of individual independent components as projection pursuit directions the statistical properties of the estimators based on such contrast functions are analyzed under the assumption of the linear mixture model and it is shown how to choose contrast functions that are robust and or of minimum variance finally we introduce simple fixed point algorithms for practical optimization of the contrast functions

learning active basis model for object detection and recognition this article proposes an active basis model a shared sketch algorithm and a computational architecture of sum max maps for representing learning and recognizing deformable templates in our generative model a deformable template is in the form of an active basis which consists of a small number of gabor wavelet elements at selected locations and orientations these elements are allowed to slightly perturb their locations and orientations before they are linearly combined to generate the observed image the active basis model in particular the locations and the orientations of the basis elements can be learned from training images by the shared sketch algorithm the algorithm selects the elements of the active basis sequentially from a dictionary of gabor wavelets when an element is selected at each step the element is shared by all the training images and the element is perturbed to encode or sketch a nearby edge segment in each training image the recognition of the deformable template from an image can be accomplished by a computational architecture that alternates the sum maps and the max maps the computation of the max maps deforms the active basis to match the image data and the computation of the sum maps scores the template matching by the log likelihood of the deformed active basis

robust singular value decompositions a new approach to projection pursuit abstract robust location and covariance estimators are developed via general m estimation for covariance matrix eigenvectors and eigenvalues the solution to this gm estimation problem is obtained by transforming it into a series of robust regression problems based on a new algorithm for the singular value decomposition it is shown here that the singular value decomposition can be represented as an iteration of two steps a least squares regression fit of the data matrix followed by a rotation to the regression hyperplanes an algorithm to obtain the solution to this gm estimation problem is presented along with results of a monte carlo study and examples of its application in addition it is shown how the output of this algorithm can be used to numerically search for multivariate outliers which is especially useful in exploratory data analysis with high dimensional data and large sample sizes where standard graphical techniques are difficult to implement because the algorithm computes robust estima

edgeworth expanded gaussian mixture density modeling instead of increasing the order of the edgeworth expansion of a single gaussian kernel we suggest using mixtures of edgeworth expanded gaussian kernels of moderate order we introduce a simple closed form solution for estimating the kernel parameters based on weighted moment matching furthermore we formulate the extension to the multivariate case which is not always feasible with algebraic density approximation procedures

staffing multiskill call centers via linear programming and simulation we study an iterative cutting plane algorithm on an integer program for minimizing the staffing costs of a multiskill call center subject to service level requirements that are estimated by simulation we solve a sample average version of the problem where the service levels are expressed as functions of the staffing for a fixed sequence of random numbers driving the simulation an optimal solution of this sample problem is also an optimal solution to the original problem when the sample size is large enough several difficulties are encountered when solving the sample problem especially for large problem instances and we propose practical heuristics to deal with these difficulties we report numerical experiments with examples of different sizes the largest example corresponds to a real life call center with 65 types of calls and 89 types of agents skill groups

adaptive stochastic approximation by the simultaneous perturbation method stochastic approximation sa has long been applied for problems of minimizing loss functions or root finding with noisy input information as with all stochastic search algorithms there are adjustable algorithm coefficients that must be specified and that can have a profound effect on algorithm performance it is known that choosing these coefficients according to an sa analog of the deterministic newton raphson algorithm provides an optimal or near optimal form of the algorithm however directly determining the required hessian matrix or jacobian matrix for root finding to achieve this algorithm form has often been difficult or impossible in practice the paper presents a general adaptive sa algorithm that is based on a simple method for estimating the hessian matrix while concurrently estimating the primary parameters of interest the approach applies in both the gradient free optimization kiefer wolfowitz and root finding stochastic gradient based robbins monro settings and is based on the simultaneous perturbation sp idea introduced previously the algorithm requires only a small number of loss function or gradient measurements per iteration independent of the problem dimension to adaptively estimate the hessian and parameters of primary interest aside from introducing the adaptive sp approach the paper presents practical implementation guidance asymptotic theory and a nontrivial numerical evaluation also included is a discussion and numerical analysis comparing the adaptive sp approach with the iterate averaging approach to accelerated sa

stochastic approximation methods for systems over an infinitehorizon the paper develops efficient and general stochastic approximation sa methods for improving the operation of parametrized systems of either the continuous or discrete event dynamical systems types and which are of interest over a long time period for example one might wish to optimize or improve the stationary or average cost per unit time performance by adjusting the systems parameters the number of applications and the associated literature are increasing at a rapid rate this is partly due to the increasing activity in computing pathwise derivatives and adapting them to the average cost problem although the original motivation and the examples come from an interest in the infinite horizon problem the techniques and results are of general applicability in sa we present an updating and review of powerful ordinary differential equation type methods in a fairly general context and based on weak convergence ideas the results and proof techniques are applicable to a wide variety of applications exploiting the full potential of these ideas can greatly simplify and extend much current work their breadth as well as the relative ease of using the basic ideas are illustrated in detail via typical examples drawn from discrete event dynamical systems piecewise deterministic dynamical systems and a stochastic differential equations model in these particular illustrations we use either infinitesimal perturbation analysis type estimators mean square derivative type estimators or finite difference type estimators markov and non markov models are discussed the algorithms for distributed asynchronous updating as well as the fully synchronous schemes are developed

on the forward and backward algorithms of projection pursuit this article provides a historic review of the forward and backward projection pursuit algorithms previously thought to be equivalent and points out an important difference between the two in doing so a small error in the original exploratory projection pursuit article by friedman j amer statist assoc 82 1987 249 266 is corrected the implication of the difference is briefly discussed in the context of an application in which projection pursuit density estimation is used as a building block for nonparametric discriminant analysis

a unified model for probabilistic principal surfaces principal curves and surfaces are nonlinear generalizations of principal components and subspaces respectively they can provide insightful summary of high dimensional data not typically attainable by classical linear methods solutions to several problems such as proof of existence and convergence faced by the original principal curve formulation have been proposed in the past few years nevertheless these solutions are not generally extensible to principal surfaces the mere computation of which presents a formidable obstacle consequently relatively few studies of principal surfaces are available we previously 2000 proposed the probabilistic principal surface pps to address a number of issues associated with current principal surface algorithms pps uses a manifold oriented covariance noise model based on the generative topographical mapping gtm which can be viewed as a parametric formulation of kohonen s self organizing map building on the pps we introduce a unified covariance model that implements pps 0 1 by varying the clamping parameter spl alpha then we comprehensively evaluate the empirical performance of pps gtm and the manifold aligned gtm on three popular benchmark data sets it is shown in two different comparisons that the pps outperforms the gtm under identical parameter settings convergence of the pps is found to be identical to that of the gtm and the computational overhead incurred by the pps decreases to 40 percent or less for more complex manifolds these results show that the generalized pps provides a flexible and effective way of obtaining principal surfaces

an adaptive huber method for nonlinear systems of volterra integral equations with weakly singular kernels and solutions abstract numerical methods for solving nonlinear systems of weakly singular volterra integral equations vies possessing weakly singular solutions appear almost nonexistent in the literature except for a few treatments of single first kind abel equations to reduce this gap an extension is presented of the adaptive huber method designed for vies with singular kernels such as k t t 1 2 and k t exp t t 1 2 where 0 and a variety of nonsingular kernels the method was thus far restricted to bounded solutions having at least two derivatives under a number of assumptions specified the extension applies to solutions u t that can be written as sums of singular components c t 1 2 with unknown coefficients c and nonsingular components u t in the solution process factor t 1 2 is handled analytically whereas c and u t are determined numerically computational experiments reveal that the extended method determines singular solutions equally well as the unextended method determined nonsingular solutions the method is intended primarily for a class of vies encountered in electroanalytical chemistry but it can also be of interest to other application areas

simplifying neural networks by soft weight sharing one way of simplifying neural networks so they generalize better is to add an extra term to the error function that will penalize complexity simple versions of this approach include penalizing the sum of the squares of the weights or penalizing the number of nonzero weights we propose a more complicated penalty term in which the distribution of weight values is modeled as a mixture of multiple gaussians a set of weights is simple if the weights have high probability density under the mixture model this can be achieved by clustering the weights into subsets with the weights in each cluster having very similar values since we do not know the appropriate means or variances of the clusters in advance we allow the parameters of the mixture model to adapt at the same time as the network learns simulations on two different problems demonstrate that this complexity term is more effective than previous complexity terms

blind source separation by sparse decomposition in a signal dictionary the blind source separation problem is to extract the underlying source signals from a set of linear mixtures where the mixing matrix is unknown this situation is common in acoustics radio medical signal and image processing hyperspectral imaging and other areas we suggest a two stage separation process a priori selection of a possibly overcomplete signal dictionary for instance a wavelet frame or a learned dictionary in which the sources are assumed to be sparsely representable followed by unmixing the sources by exploiting the their sparse representability we consider the general case of more sources than mixtures but also derive a more efficient algorithm in the case of a nonovercomplete dictionary and an equal numbers of sources and mixtures experiments with artificial signals and musical sounds demonstrate significantly better separation than other known techniques

linear dimensionality reduction survey insights and generalizations linear dimensionality reduction methods are a cornerstone of analyzing high dimensional data due to their simple geometric interpretations and typically attractive computational properties these methods capture many data features of interest such as covariance dynamical structure correlation between data sets input output relationships and margin between data classes methods have been developed with a variety of names and motivations in many fields and perhaps as a result the connections between all these methods have not been highlighted here we survey methods from this disparate literature as optimization programs over matrix manifolds we discuss principal component analysis factor analysis linear multidimensional scaling fisher s linear discriminant analysis canonical correlations analysis maximum autocorrelation factors slow feature analysis sufficient dimensionality reduction undercomplete independent component analysis linear regression distance metric learning and more this optimization framework gives insight to some rarely discussed shortcomings of well known methods such as the suboptimality of certain eigenvector solutions modern techniques for optimization over matrix manifolds enable a generic linear dimensionality reduction solver which accepts as input data and an objective to be optimized and returns as output an optimal low dimensional projection of the data this simple optimization framework further allows straightforward generalizations and novel variants of classical methods which we demonstrate here by creating an orthogonal projection canonical correlations analysis more broadly this survey and generic solver suggest that linear dimensionality reduction can move toward becoming a blackbox objective agnostic numerical technology

robust estimation of dispersion matrices and principal components abstract this paper uses monte carlo methods to compare the performances of several robust procedures for estimating a correlation matrix and its principal components the estimators are formed either from separate bivariate analyses or by simultaneous manipulation of all variables by using techniques such as multivariate trimming and m estimation the m estimators stand up exceptionally well they and the multivariate trimming procedure are especially effective at estimating the principal components including a near singularity however the m estimators can break down relatively easily when the dimensionality is large and the outliers are asymmetric with missing data the element wise approach becomes more attractive

on the convergence rates of ipa and fdc derivative estimators we show that under the sufficient conditions usually given for infinitesimal perturbation analysis ipa to apply for derivative estimation a finite difference scheme with common random numbers fdc has the same order of convergence namely o n 1 2 provided that the size of the finite difference interval converges to zero fast enough this holds for both one and two sided fdc this also holds for different variants of ipa such as some versions of smoothed perturbation analysis spa which is based on conditional expectation finally this also holds for the estimation of steady state performance measures by truncated horizon estimators under some ergodicity assumptions our developments do not involve monotonicity but are based on continuity and smoothness we give examples and numerical illustrations which show that the actual difference in mean square error mse between ipa and fdc is typically negligible we also obtain the order of convergence of that difference which is faster than the c

multivariate stochastic approximation using a simultaneous perturbation gradient approximation the problem of finding a root of the multivariate gradient equation that arises in function minimization is considered when only noisy measurements of the function are available a stochastic approximation sa algorithm for the general kiefer wolfowitz type is appropriate for estimating the root the paper presents an sa algorithm that is based on a simultaneous perturbation gradient approximation instead of the standard finite difference approximation of keifer wolfowitz type procedures theory and numerical experience indicate that the algorithm can be significantly more efficient than the standard algorithms in large dimensional problems

feedback and weighting mechanisms for improving jacobian estimates in the adaptive simultaneous perturbation algorithm it is known that a stochastic approximation sa analogue of the deterministic newton raphson algorithm provides an asymptotically optimal or near optimal form of stochastic search however directly determining the required jacobian matrix or hessian matrix for optimization has often been difficult or impossible in practice this paper presents a general adaptive sa algorithm that is based on a simple method for estimating the jacobian matrix while concurrently estimating the primary parameters of interest relative to prior methods for adaptively estimating the jacobian matrix the paper introduces two enhancements that generally improve the quality of the estimates for underlying jacobian hessian matrices thereby improving the quality of the estimates for the primary parameters of interest the first enhancement rests on a feedback process that uses previous jacobian estimates to reduce the error in the current estimate the second enhancement is based on an optimal weighting of per iteration jacobian estimates from the use of simultaneous perturbations the algorithm requires only a small number of loss function or gradient measurements per iteration independent of the problem dimension to adaptively estimate the jacobian matrix and parameters of primary interest

budget dependent convergence rate of stochastic approximation convergence rate results are derived for a stochastic optimization problem where a performance measure is minimized with respect to a vector parameter t assuming that a gradient estimator is available and that both the bias and the variance of the estimator are known functions of the budget devoted to its computation the gradient estimator is employed in conjunction with a stochastic approximation sa algorithm our interest is to figure out how to allocate the total available computational budget to the successive sa iterations the effort is devoted to solving the asymptotic version of this problem by finding the convergence rate of sa toward the optimizer first as a function of the number of iterations and then as a function of the total computational effort as a result the optimal rate of increase of the computational budget per iteration can be found explicit expressions for the case where the computational budget devoted to an iteration is a polynomial in the iteration number and where the bias and variance of the gradient estimator are polynomials of the computational budget are derived applications include the optimization of steady state simulation models with likelihood ratio perturbation analysis or finite difference gradient estimators optimization of infinite horizon models with discounting optimization of functions of several expectations and so on several examples are discussed our results readily generalize to general root finding problems

a globally convergent stochastic approximation by combining a constrained kiefer wolfowitz search with an automatic learning algorithm it is shown that asymptotically normal convergence of an estimator to a global optimum under reasonably lenient assumptions can be attained it is enough that the objective function be smooth and locally strictly convex at its minima the central conclusion is that if theta n is the estimate produced by the method shown at the nth decision epoch then for some global minimizer theta n 1 3 theta n theta is asymptotically normally distributed this coincides with the conventional kiefer wolfowitz convergence rate to a local optimum whereas this study was motivated by needs of machine learning the basic plan would seem applicable to root finding tasks and to other types of stochastic approximation algorithms

on principal hessian directions for data visualization and dimension reduction another application of stein s lemma abstract modern graphical tools have enhanced our ability to learn many things from data directly with much user friendly graphical software available we are encouraged to plot a lot more often than before the benefits from direct interaction with graphics have been enormous but trailing behind these high tech advances is the issue of appropriate guidance on what to plot there are too many directions to project a high dimensional data set and unguided plotting can be time consuming and fruitless in a recent article li set up a statistical framework for study on this issue based on a notion of effective dimension reduction edr directions they are the directions to project a high dimensional input variable for the purpose of effectively viewing and studying its relationship with an output variable a methodology sliced inverse regression was introduced and shown to be useful in finding edr directions this article introduces another method for finding edr directions it begins with the observat