application of the karhunen loeve procedure for the characterization of human faces the use of natural symmetries mirror images in a well defined family of patterns human faces is discussed within the framework of the karhunen loeve expansion this results in an extension of the data and imposes even and odd symmetry on the eigenfunctions of the covariance matrix without increasing the complexity of the calculation the resulting approximation of faces projected from outside of the data set onto this optimal basis is improved on average

generalized principal component analysis projection of saturated model parameters principal component analysis pca is very useful for a wide variety of data analysis tasks but its implicit connection to the gaussian distribution can be undesirable for discrete data such as bi

piecewise linear skeletonization using principal curves proposes an algorithm to find piecewise linear skeletons of handwritten characters by using principal curves the development of the method was inspired by the apparent similarity between the definition of principal curves smooth curves which pass through the middle of a cloud of points and medial axes smooth curves that run equidistantly from the contours of a character image the central fitting and smoothing step of the algorithm is an extension of the polygonal line algorithm which approximates principal curves of data sets by piecewise linear curves the polygonal line algorithm is extended to find principal graphs and complemented with two steps specific to the task of skeletonization an initialization method to capture the approximate topology of the character and a collection of restructuring operations to improve the structural quality of the skeleton produced by the initialization method an advantage of our approach over existing methods is that we optimize the skeleton graph by minimizing an intuitive and explicit objective function that captures the two competing criteria of smoothing the skeleton and fitting it closely to the pixels of the character image we tested the algorithm on isolated handwritten digits and images of continuous handwriting the results indicated that the proposed algorithm can find a smooth medial axis in the great majority of a wide variety of character templates and that it substantially improves the pixel wise skeleton obtained by traditional thinning methods

a bayesian information criterion for singular models summary we consider approximate bayesian model choice for model selection problems that involve models whose fisher information matrices may fail to be invertible along other competing submodels such singular models do not obey the regularity conditions underlying the derivation of schwarz s bayesian information criterion bic and the penalty structure in bic generally does not reflect the frequentist large sample behaviour of the marginal likelihood although large sample theory for the marginal likelihood of singular models has been developed recently the resulting approximations depend on the true parameter value and lead to a paradox of circular reasoning guided by examples such as determining the number of components in mixture models the number of factors in latent factor models or the rank in reduced rank regression we propose a resolution to this paradox and give a practical extension of bic for singular model selection problems

performance of ds cdma systems with optimal hard decision parallel interference cancellation we study a multiuser detection system for code division multiple access cdma we show that applying multistage hard decision parallel interference cancellation hd pic significantly improves performance compared to the matched filter system in multistage hd pic estimates of the interfering signals are used iteratively to improve knowledge of the desired signal we use large deviation theory to show that the bit error probability bep is exponentially small when the number of users is fixed and the processing gain increases we investigate the exponential rate of the bep after several stages of hd pic we propose to use the exponential rate of the bep as a measure of performance rather than the signal to noise ratio snr which is often not reliable in multiuser detection models when the system is lightly loaded we show that the exponential rate of the bep remains fixed after a finite number of stages resulting in an optimal hard decision system when the number of users becomes large the exponential rate of the bep converges to log 2 2 1 4 we provide guidelines for the number of stages necessary to obtain this asymptotic exponential rate we also give chernoff bounds on the beps these estimates show that the beps are quite small as long as k o n log n when the number of stages of hd pic is fixed and even exponentially small when k o n for the optimal hd pic system and where k is the number of users in the system and n is the processing gain finally we extend the results to the case where the number of stages depends on k in a certain manner the above results are proved for a noiseless channel and we argue that we expect similar results in a noisy channel as long as the two sided spectrum of the noise decreases proportionally to n

products of gaussians and probabilistic minor component analysis recently hinton introduced the products of experts architecture for density estimation where individual expert probabilities are multiplied and renormalized we consider products of gaussian pancakes equally elongated in all directions except one and prove that the maximum likelihood solution for the model gives rise to a minor component analysis solution we also discuss the covariance structure of sums and products of gaussian pancakes or one factor probabilistic principal component analysis models

bayes optimality motivated linear and multilayered perceptron based dimensionality reduction dimensionality reduction is the process of mapping high dimension patterns to a lower dimension subspace when done prior to classification estimates obtained in the lower dimension subspace are more reliable for some classifiers there is also an improvement in performance due to the removal of the diluting effect of redundant information a majority of the present approaches to dimensionality reduction are based on scatter matrices or other statistics of the data which do not directly correlate to classification accuracy the optimality criteria of choice for the purposes of classification is the bayes error usually however bayes error is difficult to express analytically we propose an optimality criteria based on an approximation of the bayes error and use it to formulate a linear and a nonlinear method of dimensionality reduction the nonlinear method we propose relies on using a multilayered perceptron which produces as output the lower dimensional representation it thus differs from autoassociative like multilayered perceptrons which have been proposed and used for dimensionality reduction our results show that the nonlinear method is as anticipated superior to the linear method in that it can perform unfolding of a nonlinear manifold in addition the nonlinear method we propose provides substantially better lower dimension representation for classification purposes than fisher s linear discriminant fld and two other nonlinear methods of dimensionality reduction that are often used

error probabilities for binary direct sequence spread spectrum communications with random signature sequences binary direct sequence spread spectrum multiple access communications an additive white gaussian noise channel and a coherent correlation receiver are considered an expression for the output of the receiver is obtained for the case of random signature sequences and the corresponding characteristic function is determined the expression is used to study the density function of the multiple access interference and to determine arbitrarily tight upper and lower bounds on the average probability of error the bounds which are obtained without making a gaussian approximation are compared to results obtained using a gaussian approximation the effects of transmitter power the length of the signature sequences and the number of interfering transmitters are illustrated each transmitter is assumed to have the same power although the general approach can accommodate the case of transmitters with unequal powers

anomaly preserving ell _ scriptscriptstyle 2 infty optimal dimensionality reduction over a grassmann manifold in this paper we address the problem of redundancy reduction of high dimensional noisy signals that may contain anomaly rare vectors which we wish to preserve since anomaly data vectors contribute weakly to the l2 norm of the signal as compared to the noise l2 based criteria are unsatisfactory for obtaining a good representation of these vectors as a remedy a new approach named min max svd mx svd was recently proposed for signal subspace estimation by attempting to minimize the maximum of data residual l2 norms denoted as l2 l and designed to represent well both abundant and anomaly measurements however the mx svd algorithm is greedy and only approximately minimizes the proposed l2 l norm of the residuals in this paper we develop an optimal algorithm for the minization of the l2 l norm of data misrepresentation residuals which we call maximum orthogonal complements optimal subspace estimation moose the optimization is performed via a natural conjugate gradient learning approach carried out on the set of n dimensional subspaces in irm m n which is a grassmann manifold the results of applying moose mx svd and l2 based approaches are demonstrated both on simulated and real hyperspectral data

cauchy noise loss for stochastic optimization of random matrix models via free deterministic equivalents abstract for random matrix models the parameter estimation based on the traditional likelihood functions is not straightforward in particular when we have only one sample matrix we introduce a new parameter optimization method for random matrix models which works even in such a case the method is based on the spectral distribution instead of the traditional likelihood in the method the cauchy noise has an essential role because the free deterministic equivalent which is a tool in free probability theory allows us to approximate the spectral distribution perturbed by cauchy noises by a smooth and accessible density function moreover we study an asymptotic property of determination gap which has a similar role as generalization gap besides we propose a new dimensionality recovery method for the signal plus noise model and experimentally demonstrate that it recovers the rank of the signal part even if the true rank is not small it is a simultaneous rank selection and parameter estimation procedure

feature extraction using two dimensional maximum embedding difference in this paper we propose a novel method combining graph embedding and difference criterion techniques for image feature extraction namely two dimensional maximum embedding difference 2dmed this method directly extracts the optimal projective vectors from 2d image matrices by simultaneously considering characteristic that is the intraclass compactness graph the margin graph and inter class separability graph respectively in this method it is not necessary to convert the image matrix into high dimensional image vector so that much computational time would be saved in addition the proposed method preserves the manifold reconstruction relationships in the low dimensional subspace experimental results on the orl yale face and usps database show the effectiveness of the proposed method

conditional importance sampling estimators we give a unified presentation of the conditional importance sampling estimators we show that they are always better than their nonconditional counterparts we then present the large deviation theory associated with these estimators in particular we give conditional simulation distributions that are optimal in the sense that they are efficient interestingly enough these distributions will not in general be the usual exponential shifts we give examples showing how to use the theory developed

a unifying probabilistic perspective for spectral dimensionality reduction insights and new models we introduce a new perspective on spectral dimensionality reduction which views these methods as gaussian markov random fields grfs our unifying perspective is based on the maximum entropy principle which is in turn inspired by maximum variance unfolding the resulting model which we call maximum entropy unfolding meu is a nonlinear generalization of principal component analysis we relate the model to laplacian eigenmaps and isomap we show that parameter fitting in the locally linear embedding lle is approximate maximum likelihood meu we introduce a variant of lle that performs maximum likelihood exactly acyclic lle alle we show that meu and alle are competitive with the leading spectral approaches on a robot navigation visualization and a human motion capture data set finally the maximum likelihood perspective allows us to introduce a new approach to dimensionality reduction based on l1 regularization of the gaussian random field via the graphical lasso

probabilistic tensor canonical polyadic decomposition with orthogonal factors tensor canonical polyadic decomposition cpd which recovers the latent factor matrices from multidimensional data is an important tool in signal processing in many applications some of the factor matrices are known to have orthogonality structure and this information can be exploited to improve the accuracy of latent factors recovery however existing methods for cpd with orthogonal factors all require the knowledge of tensor rank which is difficult to acquire and have no mechanism to handle outliers in measurements to overcome these disadvantages in this paper a novel tensor cpd algorithm based on the probabilistic inference framework is devised in particular the problem of tensor cpd with orthogonal factors is interpreted using a probabilistic model based on which an inference algorithm is proposed that alternatively estimates the factor matrices recovers the tensor rank and mitigates the outliers simulation results using synthetic data and real world applications are presented to illustrate the excellent performance of the proposed algorithm in terms of accuracy and robustness

independent factor analysis we introduce the independent factor analysis ifa method for recovering independent hidden sources from their observed mixtures ifa generalizes and unifies ordinary factor analysis fa principal component analysis pca and independent component analysis ica and can handle not only square noiseless mixing but also the general case where the number of mixtures differs from the number of sources and the data are noisy ifa is a two step procedure in the first step the source densities mixing matrix and noise covariance are estimated from the observed data by maximum likelihood for this purpose we present an expectation maximization em algorithm which performs unsupervised learning of an associated probabilistic model of the mixing situation each source in our model is described by a mixture of gaussians thus all the probabilistic calculations can be performed analytically in the second step the sources are reconstructed from the observed data by an optimal nonlinear estimator a variational approximation of this algorithm is derived for cases with a large number of sources where the exact algorithm becomes intractable our ifa algorithm reduces to the one for ordinary fa when the sources become gaussian and to an em algorithm for pca in the zero noise limit we derive an additional em algorithm specifically for noiseless ifa this algorithm is shown to be superior to ica since it can learn arbitrary source densities from the data beyond blind separation ifa can be used for modeling multidimensional data by a highly constrained mixture of gaussians and as a tool for nonlinear signal encoding

image outlier detection and feature extraction via l1 norm based 2d probabilistic pca this paper introduces an l1 norm based probabilistic principal component analysis model on 2d data l1 2dppca based on the assumption of the laplacian noise model the laplacian or l1 density function can be expressed as a superposition of an infinite number of gaussian distributions under this expression a bayesian inference can be established based on the variational expectation maximization approach all the key parameters in the probabilistic model can be learned by the proposed variational algorithm it has experimentally been demonstrated that the newly introduced hidden variables in the superposition can serve as an effective indicator for data outliers experiments on some publicly available databases show that the performance of l1 2dppca has largely been improved after identifying and removing sample outliers resulting in more accurate image reconstruction than the existing pca based methods the performance of feature extraction of the proposed method generally outperforms other existing algorithms in terms of reconstruction errors and classification accuracy

a direct formulation for sparse pca using semidefinite programming given a covariance matrix we consider the problem of maximizing the variance explained by a particular linear combination of the input variables while constraining the number of nonzero coefficients in this combination this problem arises in the decomposition of a covariance matrix into sparse factors or sparse principal component analysis pca and has wide applications ranging from biology to finance we use a modification of the classical variational representation of the largest eigenvalue of a symmetric matrix where cardinality is constrained and derive a semidefinite programming based relaxation for our problem we also discuss nesterov s smooth minimization technique applied to the semidefinite program arising in the semidefinite relaxation of the sparse pca problem the method has complexity o n 4 sqrt log n epsilon where n is the size of the underlying covariance matrix and epsilon is the desired absolute accuracy on the optimal value of the problem

maximal likelihood correspondence estimation for face recognition across pose due to the misalignment of image features the performance of many conventional face recognition methods degrades considerably in across pose scenario to address this problem many image matching based methods are proposed to estimate semantic correspondence between faces in different poses in this paper we aim to solve two critical problems in previous image matching based correspondence learning methods 1 fail to fully exploit face specific structure information in correspondence estimation and 2 fail to learn personalized correspondence for each probe image to this end we first build a model termed as morphable displacement field mdf to encode face specific structure information of semantic correspondence from a set of real samples of correspondences calculated from 3d face models then we propose a maximal likelihood correspondence estimation mlce method to learn personalized correspondence based on maximal likelihood frontal face assumption after obtaining the semantic correspondence encoded in the learned displacement we can synthesize virtual frontal images of the profile faces for subsequent recognition using linear discriminant analysis method with pixel intensity features state of the art performance is achieved on three multipose benchmarks i e cmu pie feret and multipie databases owe to the rational mdf regularization and the usage of novel maximal likelihood objective the proposed mlce method can reliably learn correspondence between faces in different poses even in complex wild environment i e labeled face in the wild database

construction of a unique self adjoint generator for a symmetric local semigroup abstract a definition is given of a symmetric local semigroup of unbounded operators p t 0 t t for some t 0 on a hilbert space h such that p t is eventually densely defined as t 0 it is shown that there exists a unique unbounded below self adjoint operator h on h such that p t is a restriction of e th as an application it is proven that h0 v is essentially self adjoint where e th0 is an lp contractive semigroup and v is multiplication by a real measurable function such that v l2 e and e v l1 for some e 0

non stationary random vibration analysis of structures under multiple correlated normal random excitations abstract an algorithm that integrates karhunen loeve expansion kle and the finite element method fem is proposed to perform non stationary random vibration analysis of structures under excitations represented by multiple random processes that are correlated in both time and spatial domains in kle the auto covariance functions of random excitations are discretized using orthogonal basis functions the kle for multiple correlated random excitations relies on expansions in terms of correlated sets of random variables reflecting the cross covariance of the random processes during the response calculations the eigenfunctions of kle used to represent excitations are applied as forcing functions to the structure the proposed algorithm is applied to a 2dof system a 2d cantilever beam and a 3d aircraft wing under both stationary and non stationary correlated random excitations two methods are adopted to obtain the structural responses a the modal method and b the direct method both the methods provide the statistics of the dynamic response with sufficient accuracy the structural responses under the same type of correlated random excitations are bounded by the response obtained by perfectly correlated and uncorrelated random excitations the structural response increases with a decrease in the correlation length and with an increase in the correlation magnitude the proposed methodology can be applied for the analysis of any complex structure under any type of random excitation

maximum a posteriori estimation of linear shape variation with application to vertebra and cartilage modeling the estimation of covariance matrices is a crucial step in several statistical tasks especially when using few samples of a high dimensional representation of shapes the standard maximum likelihood estimation ml of the covariance matrix can be far from the truth is often rank deficient and may lead to unreliable results in this paper we discuss regularization by prior knowledge using maximum a posteriori map estimates we compare ml to map using a number of priors and to tikhonov regularization we evaluate the covariance estimates on both synthetic and real data and we analyze the estimates influence on a missing data reconstruction task where high resolution vertebra and cartilage models are reconstructed from incomplete and lower dimensional representations our results demonstrate that our methods outperform the traditional ml method and tikhonov regularization

analytic continuation of local representations of symmetric spaces we prove the following analytic continuation theorem which applies to any virtual representation of any symmetric space g k the problem of passing from the euclidean group to the poincare group appears first to have been addressed and solved this way by klein and landau let g be a lie group k a closed subgroup and an involutive automorphism with k as fixed point subgroup if g k m is the corresponding symmetric lie algebra we form g k im and let g denote the simply connected lie group with g as lie algebra we consider virtual representations of g on a fixed complex hilbert space h adopting the definitions due to j frohlich k osterwalder and e seiler in particular g 1 g possibly unbounded operators for g in a neighborhood of e in g we prove that every such continues analytically to a strongly continuous unitary representation of g on h our theorem extends results due to klein landau frohlich et al and others earlier for special cases previous results were known only for special g k and then only for certain

links between ppca and subspace methods for complete gaussian density estimation high dimensional density estimation is a fundamental problem in pattern recognition and machine learning areas in this letter we show that for complete high dimensional gaussian density estimation two widely used methods probabilistic principal component analysis and a typical subspace method using eigenspace decomposition actually give the same results additionally we present a unified view from the aspect of robust estimation of the covariance matrix

large deviations for code division multiple access systems we derive approximations for the probability of a bit error for a code division multiple access cdma system with one stage soft decision parallel interference cancellation more precisely we derive the exponential rates jk with cancellation and ik without cancellation of a cdma system with k users and processing gain equal to n as n to infty whereas the rates ik follow explicitly from cramer s theorem the rates jk are given in terms of an optimization problem that can be evaluated numerically we prove that jk ik for k ge3 which shows that interference cancellation is effective for the case without interference cancellation we investigate the second order bahadur rao asymptotics for the case with interference cancellation we can obtain second order asymptotics only for k 3 together the limits provide excellent approximations for the probability of a bit error in a wide range of interest

danco an intrinsic dimensionality estimator exploiting angle and norm concentration abstract in the past decade the development of automatic intrinsic dimensionality estimators has gained considerable attention due to its relevance in several application fields however most of the proposed solutions prove to be not robust on noisy datasets and provide unreliable results when the intrinsic dimensionality of the input dataset is high and the manifold where the points are assumed to lie is nonlinearly embedded in a higher dimensional space in this paper we propose a novel intrinsic dimensionality estimator danco and its faster variant fastdanco which exploit the information conveyed both by the normalized nearest neighbor distances and by the angles computed on couples of neighboring points the effectiveness and robustness of the proposed algorithms are assessed by experiments on synthetic and real datasets by the comparative evaluation with state of the art methodologies and by significance tests

linear multiuser detectors for synchronous code division multiple access channels under the assumptions of symbol synchronous transmissions and white gaussian noise the authors analyze the detection mechanism at the receiver comparing different detectors by their bit error rates in the low background noise region and by their worst case behavior in a near far environment where the received energies of the users are not necessarily similar optimum multiuser detection achieves important performance gains over conventional single user detection at the expense of computational complexity that grows exponentially with the number of users it is shown that in the synchronous case the performance achieved by linear multiuser detectors is similar to that of optimum multiuser detection attention is focused on detectors whose linear memoryless transformation is a generalized inverse of the matrix of signature waveform crosscorrelations and on the optimum linear detector it is shown that the generalized inverse detectors exhibit the same degree of near far resistance as the optimum multiuser detectors the optimum linear detector is obtained

infinite max margin factor analysis via data augmentation this paper addresses the bayesian estimation of the discriminative probabilistic latent models especially the mixture models we develop the max margin factor analysis mmfa model which utilizes the latent variable support vector machine lvsvm as the classification criterion in the latent space to learn a discriminative subspace with max margin constraint furthermore to deal with multimodally distributed data we further extend mmfa to infinite gaussian mixture model and develop the infinite max margin factor analysis immfa model via the consideration of dirichlet process mixtures dpm it jointly learns clustering max margin classifiers and the discriminative latent space in a united framework to improve the prediction performance moreover both of mmfa and immfa are natural to handle outlier rejection problem since the observations are described by a single or a mixture of gaussian distributions additionally thanks to the conjugate property the parameters in the two models can be inferred efficiently via the simple gibbs sampler finally we implement our models on synthesized and real world data including multimodally distributed datasets and measured radar echo data to validate the classification and rejection performance of the proposed models jointly learning fa and svm mmfa is proposed to get a discriminative subspace clustering the dataset in the subspace by dpm mmfa is extended to immfa thanks to the jointly learning framework they gain good prediction performance having the data description ability the proposed models can reject outlier samples in bayesian framework parameters can be inferred efficiently by the gibbs sampler

complete monotonicity for inverse powers of some combinatorially defined polynomials we prove the complete monotonicity on 0 infty n for suitable inverse powers of the spanning tree polynomials of graphs and more generally of the basis generating polynomials of certain classes of matroids this generalizes a result of szeg and answers among other things a long standing question of lewy and askey concerning the positivity of taylor coefficients for certain rational functions our proofs are based on two ab initio methods for proving that p beta is completely monotone on a convex cone c the determinantal method and the quadratic form method these methods are closely connected with harmonic analysis on euclidean jordan algebras or equivalently on symmetric cones we furthermore have a variety of constructions that given such polynomials can create other ones with the same property among these are algebraic analogues of the matroid operations of deletion contraction direct sum parallel connection series connection and 2 sum the complete monotonicity of p beta for some beta 0 can be viewed as a strong quantitative version of the half plane property hurwitz stability for p and is also related to the rayleigh property for matroids

subspace identification for doa estimation in massive full dimension mimo systems bad data mitigation and automatic source enumeration in this paper the direction of arrival doa estimation problem for massive multiple input multiple output mimo systems with a two dimensional 2d array also known as full dimension mimo is investigated assuming no knowledge of path number noise power path gain correlations and bad data statistics based on the variational bayesian framework a novel iterative algorithm for subspace identification operating on tensor represented data is proposed with integrated features of effective bad data mitigation and automatic source enumeration the subspace recovered from the proposed algorithm not only enables existing 2d doa estimators to be readily applied if the number of signal paths is less than the number of horizontal antennas and vertical antennas the subspaces in elevation and azimuth domains can be separately estimated from which one dimensional 1d doa estimators can be utilized thus further lowering the complexity simulation results are presented to illustrate the excellent performance of the proposed subspace recovery method and subsequent doa estimation in terms of accuracy and robustness

weighted nuclear norm minimization and its applications to low level vision as a convex relaxation of the rank minimization model the nuclear norm minimization nnm problem has been attracting significant research interest in recent years the standard nnm regularizes each singular value equally composing an easily calculated convex norm however this restricts its capability and flexibility in dealing with many practical problems where the singular values have clear physical meanings and should be treated differently in this paper we study the weighted nuclear norm minimization wnnm problem which adaptively assigns weights on different singular values as the key step of solving general wnnm models the theoretical properties of the weighted nuclear norm proximal wnnp operator are investigated albeit nonconvex we prove that wnnp is equivalent to a standard quadratic programming problem with linear constrains which facilitates solving the original problem with off the shelf convex optimization solvers in particular when the weights are sorted in a non descending order its optimal solution can be easily obtained in closed form with wnnp the solving strategies for multiple extensions of wnnm including robust pca and matrix completion can be readily constructed under the alternating direction method of multipliers paradigm furthermore inspired by the reweighted sparse coding scheme we present an automatic weight setting method which greatly facilitates the practical implementation of wnnm the proposed wnnm methods achieve state of the art performance in typical low level vision tasks including image denoising background subtraction and image inpainting

a global solution to the schrodinger equation from henstock to feynman one of the key elements of feynman s formulation of non relativistic quantum mechanics is a so called feynman path integral it plays an important role in the theory but it appears as a postulate based on intuition rather than a well defined object all previous attempts to supply feynman s theory with rigorous mathematics underpinning based on the physical requirements have not been satisfactory the difficulty comes from the need to define a measure on the infinite dimensional space of paths and to create an integral that would possess all of the properties requested by feynman in the present paper we consider a new approach to defining the feynman path integral based on the theory developed by muldowney a modern theory of random variable with applications in stochastic calcolus financial mathematics and feynman integration john wiley sons inc new jersey 2012 muldowney uses the henstock integration technique and deals with non absolute integrability of the fresnel integrals in orde

linear dimensionality reduction survey insights and generalizations linear dimensionality reduction methods are a cornerstone of analyzing high dimensional data due to their simple geometric interpretations and typically attractive computational properties these methods capture many data features of interest such as covariance dynamical structure correlation between data sets input output relationships and margin between data classes methods have been developed with a variety of names and motivations in many fields and perhaps as a result the connections between all these methods have not been highlighted here we survey methods from this disparate literature as optimization programs over matrix manifolds we discuss principal component analysis factor analysis linear multidimensional scaling fisher s linear discriminant analysis canonical correlations analysis maximum autocorrelation factors slow feature analysis sufficient dimensionality reduction undercomplete independent component analysis linear regression distance metric learning and more this optimization framework gives insight to some rarely discussed shortcomings of well known methods such as the suboptimality of certain eigenvector solutions modern techniques for optimization over matrix manifolds enable a generic linear dimensionality reduction solver which accepts as input data and an objective to be optimized and returns as output an optimal low dimensional projection of the data this simple optimization framework further allows straightforward generalizations and novel variants of classical methods which we demonstrate here by creating an orthogonal projection canonical correlations analysis more broadly this survey and generic solver suggest that linear dimensionality reduction can move toward becoming a blackbox objective agnostic numerical technology

a maximum likelihood interpretation for slow feature analysis the brain extracts useful features from a maelstrom of sensory information and a fundamental goal of theoretical neuroscience is to work out how it does so one proposed feature extraction strategy is motivated by the observation that the meaning of sensory data such as the identity of a moving visual object is often more persistent than the activation of any single sensory receptor this notion is embodied in the slow feature analysis sfa algorithm which uses slowness as a heuristic by which to extract semantic information from multidimensional time series here we develop a probabilistic interpretation of this algorithm showing that inference and learning in the limiting case of a suitable probabilistic model yield exactly the results of sfa similar equivalences have proved useful in interpreting and extending comparable algorithms such as independent component analysis for sfa we use the equivalent probabilistic model as a conceptual springboard with which to motivate several novel extensions to the algorithm

optimally adaptive transform coding the optimal linear block transform for coding images is well known to be the karhunen loeve transformation klt however the assumption of stationarity in the optimality condition is far from valid for images images are composed of regions whose local statistics may vary widely across an image while the use of adaptation can result in improved performance there has been little investigation into the optimality of the criterion upon which the adaptation is based in this paper we propose a new transform coding method in which the adaptation is optimal the system is modular consisting of a number of modules corresponding to different classes of the input data each module consists of a linear transformation whose bases are calculated during an initial training period the appropriate class for a given input vector is determined by the subspace classifier the performance of the resulting adaptive system is shown to be superior to that of the optimal nonadaptive linear transformation this method can also be used as a segmentor the segmentation it performs is independent of variations in illumination in addition the resulting class representations are analogous to the arrangement of the directionally sensitive columns in the visual cortex

practical approaches to principal component analysis in the presence of missing values principal component analysis pca is a classical data analysis technique that finds linear transformations of data that retain the maximal amount of variance we study a case where some of the data values are missing and show that this problem has many features which are usually associated with nonlinear models such as overfitting and bad locally optimal solutions a probabilistic formulation of pca provides a good foundation for handling missing values and we provide formulas for doing that in case of high dimensional and very sparse data overfitting becomes a severe problem and traditional algorithms for pca are very slow we introduce a novel fast algorithm and extend it to variational bayesian learning different versions of pca are compared in artificial experiments demonstrating the effects of regularization and modeling of posterior variance the scalability of the proposed algorithm is demonstrated by applying it to the netflix problem

intrinsic grassmann averages for online linear robust and nonlinear subspace learning principal component analysis pca and kernel principal component analysis kpca are fundamental methods in machine learning for dimensionality reduction the former is a technique for finding this approximation in finite dimensions and the latter is often in an infinite dimensional reproducing kernel hilbert space rkhs in this paper we present a geometric framework for computing the principal linear subspaces in both situations as well as for the robust pca case that amounts to computing the intrinsic average on the space of all subspaces the grassmann manifold points on this manifold are defined as the subspaces spanned by k tuples of observations the intrinsic grassmann average of these subspaces are shown to coincide with the principal components of the observations when they are drawn from a gaussian distribution we show similar results in the rkhs case and provide an efficient algorithm for computing the projection onto the this average subspace the result is a method akin to kpca which is substantially faster further we present a novel online version of the kpca using our geometric framework competitive performance of all our algorithms are demonstrated on a variety of real and synthetic data sets

error probability for direct sequence spread spectrum multiple access communications part i upper and lower bounds upper and lower bounds on the average probability of error are obtained for direct sequence spread spectrum multipleaccess communications systems with additive white gaussian noise channels the bounds which are developed from convexity properties of the error probability function are valid for systems in which the maximum multiple access interference does not exceed the desired signal and the signature sequence period is equal to the duration of the data pulse the tightness of the bounds is examined for systems with a small number of simultaneously active transmitters this is accomplished by comparisons of the upper and lower bounds for several values of the system parameters the bounds are also compared with an approximation based on the signal to noise ratio and with the chernoff upper bound

automated model selection in covariance estimation and spatial whitening of meg and eeg signals abstract magnetoencephalography and electroencephalography m eeg measure non invasively the weak electromagnetic fields induced by post synaptic neural currents the estimation of the spatial covariance of the signals recorded on m eeg sensors is a building block of modern data analysis pipelines such covariance estimates are used in brain computer interfaces bci systems in nearly all source localization methods for spatial whitening as well as for data covariance estimation in beamformers the rationale for such models is that the signals can be modeled by a zero mean gaussian distribution while maximizing the gaussian likelihood seems natural it leads to a covariance estimate known as empirical covariance ec it turns out that the ec is a poor estimate of the true covariance when the number of samples is small to address this issue the estimation needs to be regularized the most common approach downweights off diagonal coefficients while more advanced regularization methods are based on shrinkage techniques or generative models with low rank assumptions probabilistic pca ppca and factor analysis fa using cross validation all of these models can be tuned and compared based on gaussian likelihood computed on unseen data we investigated these models on simulations one electroencephalography eeg dataset as well as magnetoencephalography meg datasets from the most common meg systems first our results demonstrate that different models can be the best depending on the number of samples heterogeneity of sensor types and noise properties second we show that the models tuned by cross validation are superior to models with hand selected regularization hence we propose an automated solution to the often overlooked problem of covariance estimation of m eeg signals the relevance of the procedure is demonstrated here for spatial whitening and source localization of meg signals

a constrained em algorithm for independent component analysis we introduce a novel way of performing independent component analysis using a constrained version of the expectation maximization em algorithm the source distributions are modeled as d one dimensional mixtures of gaussians the observed data are modeled as linear mixtures of the sources with additive isotropic noise this generative model is fit to the data using constrained em the simpler soft switching approach is introduced which uses only one parameter to decide on the sub or supergaussian nature of the sources we explain how our approach relates to independent factor analysis

robust covariance matrix estimation in heterogeneous low rank context this paper addresses the problem of robust covariance matrix cm estimation in the context of a disturbance composed of a low rank lr heterogeneous clutter plus an additive white gaussian noise the lr clutter is modeled by a spherically invariant random vector with assumed high clutter to noise ratio in such a context adaptive process should require less training samples than classical methods to reach equivalent performance as in a full rank clutter configuration the main issue is that classical robust estimators of the cm cannot be computed in the undersampled configuration to overcome this issue the current approach is based on regularization methods nevertheless most of these solutions are enforcing the estimate to be well conditioned while in our context it should be lr structured this paper therefore addresses this issue and derives an algorithm to compute the maximum likelihood estimator of the cm for the considered disturbance model several relaxations and robust generalizations of the result are discussed performance is finally illustrated on numerical simulations and on a space time adaptive processing for airborne radar application

bit to bit error dependence in slotted ds ssma packet systems with random signature sequences a technique is developed to find an accurate approximation to the probability of data bit error and the probability of packet success in a direct sequence spread spectrum multiple access ds ssma packet radio system with random signature sequences an improved gaussian approximation to the probability of data bit error is performed packet performance is analyzed by using the theory of moment spaces to gain insight into the effect of bit to bit error dependence caused by interfering signal relative delays and phases which are assumed constant over the duration of a desired packet numerical results show that if no error control exists in the desired packet or if block error control is used when multiple access interference is high the error dependence increases the average probability of packet success beyond that predicted by models which use independent bit errors however when block error control is used and the multiple access interference is low the bit error dependencies cause a reduction in packet error performance

from sample similarity to ensemble similarity probabilistic distance measures in reproducing kernel hilbert space this paper addresses the problem of characterizing ensemble similarity from sample similarity in a principled manner using a reproducing kernel as a characterization of sample similarity we suggest a probabilistic distance measure in the reproducing kernel hilbert space rkhs as the ensemble similarity assuming normality in the rkhs we derive analytic expressions for probabilistic distance measures that are commonly used in many applications such as chernoff distance or the bhattacharyya distance as its special case kullback leibler divergence etc since the reproducing kernel implicitly embeds a nonlinear mapping our approach presents a new way to study these distances whose feasibility and efficiency is demonstrated using experiments with synthetic and real examples further we extend the ensemble similarity to the reproducing kernel for ensemble and study the ensemble similarity for more general data representations

sparse exponential family principal component analysis we propose a sparse exponential family principal component analysis sepca method suitable for any type of data following exponential family distributions to achieve simultaneous dimension reduction and variable selection for better interpretation of the results because of the generality of exponential family distributions the method can be applied to a wide range of applications in particular when analyzing high dimensional next generation sequencing data and genetic mutation data in genomics the use of sparsity inducing penalty helps produce sparse principal component loading vectors such that the principal components can focus on informative variables by using an equivalent dual form of the formulated optimization problem for sepca we derive optimal solutions with efficient iterative closed form updating rules the results from both simulation experiments and real world applications have demonstrated the superiority of our sepca in reconstruction accuracy and computational efficiency over traditional exponential family pca epca the existing sparse pca spca and sparse logistic pca slpca algorithms highlightsa sparse exponential family principal component analysis is proposed it has wide applications to any data following exponential family distributions efficient solutions can be achieved with closed form update rules the performance is enhanced with appropriate assumption of the data distribution this model is flexible and highly extensible

accuracy of pseudo inverse covariance learning a random matrix theory analysis for many learning problems estimates of the inverse population covariance are required and often obtained by inverting the sample covariance matrix increasingly for modern scientific data sets the number of sample points is less than the number of features and so the sample covariance is not invertible in such circumstances the moore penrose pseudo inverse sample covariance matrix constructed from the eigenvectors corresponding to nonzero sample covariance eigenvalues is often used as an approximation to the inverse population covariance matrix the reconstruction error of the pseudo inverse sample covariance matrix in estimating the true inverse covariance can be quantified via the frobenius norm of the difference between the two the reconstruction error is dominated by the smallest nonzero sample covariance eigenvalues and diverges as the sample size becomes comparable to the number of features for high dimensional data we use random matrix theory techniques and results to study the reconstruction error for a wide class of population covariance matrices we also show how bagging and random subspace methods can result in a reduction in the reconstruction error and can be combined to improve the accuracy of classifiers that utilize the pseudo inverse sample covariance matrix we test our analysis on both simulated and benchmark data sets

error probability for direct sequence spread spectrum multiple access communications part ii approximations approximations are obtained for the average probability of error in an asynchronous direct sequence spread spectrum multiple access communications system both binary and quaternary systems are considered and the chip waveforms are allowed to be arbitrary time limited waveforms with time duration equal to the inverse of the chip rate the approximation is based on the integration of the characteristic function of the multiple access interference the amount of computation required to evaluate this approximation grows only linearly with the product of the number of simultaneous transmitters and the number of chips per bit the accuracy of the approximation is extremely good in most cases but it can be improved if necessary by an application of a series expansion numerical results are presented for specific chip waveforms and signature sequences

performance of ds ssma communications in impulsive channels part i linear correlation receivers the performance of digital linear correlation receivers is studied in a multiuser environment there are assumed to be two types of sources interfering with data transmission multiple access interference and additive channel noise which is attributed to impulsive noise sources in the environment the contribution of multiple access interference is examined by considering k asynchronous users transmitting simultaneously over a linear channel using the binary psk direct sequence spreadspectrum multiple access ds ssma technique alternatively the effects of the non gaussian impulsive channel in such a system are studied by modeling the samples of noise after front end filtering errorprobability performance under these conditions is compared to that for additive white gaussian noise awgn channels due to computational complexity exact analysis is limited here to systems utilizing short spreading sequences computationally simple methods are proposed for approximating the average error probability when the length of the signature sequences is large furthermore some asymptotic results are obtained for the case of infinitely long sequences in all cases performance variation is examined as the shape of the noise density varies with snr held constant the results of this analysis indicate that the presence of impulsive noise can cause significant performance degradation over that predicted from an awgn model even when the total noise power does not increase

a covariance free iterative algorithm for distributed principal component analysis on vertically partitioned data in this paper a covariance free iterative algorithm is developed to achieve distributed principal component analysis on high dimensional data sets that are vertically partitioned we have proved that our iterative algorithm converges monotonously with an exponential rate different from existing techniques that aim at approximating the global pca our covariance free iterative distributed pca cidpca algorithm can estimate the principal components directly without computing the sample covariance matrix therefore a significant reduction on transmission costs can be achieved furthermore in comparison to existing distributed pca techniques cidpca can provide more accurate estimations of the principal components and classification results we have demonstrated the superior performance of cidpca through the studies of multiple real world data sets

performance analysis of single user and multiuser detectors for optical code division multiple access communication systems code division multiple access cdma is a powerful multiplexing scheme particularly suited to the optical domain with its broad spectrum unfortunately analysis of optical cdma ocdma systems has been almost exclusively restricted to systems employing a correlation detector which suffer high degradation when received user powers are dissimilar this study examines the performance of an optimum single user detector and two multiuser detectors for ocdma in the presence of additive light intensity noise and poisson detection the optimized single user detector is the maximum likelihood detector given the interference from other users is a random process with known distribution one multiuser detector is based on a local search algorithm for maximizing the likelihood function the second multiuser detector is a generalization of the optical multistage detector to accept soft decision from the previous stage approximations to the error probabilities of the ocdma system based on either random signature sequences or deterministic codes are derived for each detector using a large deviations theory approach the asymptotic multiuser efficiency is found to be related to the large deviations rate function the analysis shows that the correlation detector performance decreases rapidly as the number of users increases while the optimized single user detector and the two multiuser detectors proposed perform considerably better and show less sensitivity to unequal received user powers

expandable factor analysis bayesian sparse factor models have proven useful for characterizing dependence in multivariate data but scaling computation to large numbers of samples and dimensions is problematic we propose expandable factor analysis for scalable inference in factor models when the number of factors is unknown the method relies on a continuous shrinkage prior for efficient maximum a posteriori estimation of a low rank and sparse loadings matrix the structure of the prior leads to an estimation algorithm that accommodates uncertainty in the number of factors we propose an information criterion to select the hyperparameters of the prior expandable factor analysis has better false discovery rates and true positive rates than its competitors across diverse simulation settings we apply the proposed approach to a gene expression study of ageing in mice demonstrating superior results relative to four competing methods

relevance units latent variable model and nonlinear dimensionality reduction a new dimensionality reduction method called relevance units latent variable model rulvm is proposed in this paper rulvm has a close link with the framework of gaussian process latent variable model gplvm and it originates from a recently developed sparse kernel model called relevance units machine rum rum follows the idea of relevance vector machine rvm under the bayesian framework but releases the constraint that relevance vectors rvs have to be selected from the input vectors rum treats relevance units rus as part of the parameters to be learned from the data as a result a rum maintains all the advantages of rvm and offers superior sparsity rulvm inherits the advantages of sparseness offered by the rum and the experimental result shows that rulvm algorithm possesses considerable computational advantages over gplvm algorithm

generalized widder theorem via fractional moments abstract we provide a necessary and sufficient condition for the representability of a function as the classical multidimensional laplace transform when the support of the representing measure is contained in some generalized semi algebraic set this is done by employing a method of putinar and vasilescu m putinar f h vasilescu solving moment problems by dimensional extension ann of math 2 149 3 1999 1087 1107 for the corresponding multidimensional moment problem

rank estimation and redundancy reduction of high dimensional noisy signals with preservation of rare vectors in this paper we address the problem of redundancy reduction of high dimensional noisy signals that may contain anomaly rare vectors which we wish to preserve for example when applying redundancy reduction techniques to hyperspectral images it is essential to preserve anomaly pixels for target detection purposes since rare vectors contribute weakly to the norm of the signal as compared to the noise based criteria are unsatisfactory for obtaining a good representation of these vectors the proposed approach combines and norms for both signal subspace and rank determination and considers two aspects one aspect deals with signal subspace estimation aiming to minimize the maximum of data residual norms denoted as for a given rank conjecture the other determines whether the rank conjecture is valid for the obtained signal subspace by applying extreme value theory results to model the distribution of the noise norm these two operations are performed alternately using a suboptimal greedy algorithm which makes the proposed approach practically plausible the algorithm was applied on both synthetically simulated data and on a real hyperspectral image producing better results than common based methods

direct sequence spread spectrum multiple access communications with random signature sequences a large deviations analysis a direct sequence spread spectrum multiple access bit error probability analysis is developed using large deviations theory let m denote the number of interfering spread spectrum signals and let n denote the signature sequence length then the large deviations limit is as n to infinity with m fixed a tight asymptotic expression for the bit error probability is proven and in addition recent large deviations results with the importance sampling monte carlo estimation technique are applied to obtain accurate and computationally efficient estimates of the bit error probability for finite values of m and n the large deviations point of view is compared also to the conventional asymptotics of central limit theory and the associated gaussian approximation the gaussian approximation is accurate and the ratio m n is moderately large and all signals have roughly equal power in the near far situation however the gaussian approximation is quite poor in contrast large deviations techniques are more accurate in the near far situation and it is here that these methods provide some important practical insight

the indian buffet process an introduction and review the indian buffet process is a stochastic process defining a probability distribution over equivalence classes of sparse binary matrices with a finite number of rows and an unbounded number of columns this distribution is suitable for use as a prior in probabilistic models that represent objects using a potentially infinite array of features or that involve bipartite graphs in which the size of at least one class of nodes is unknown we give a detailed derivation of this distribution and illustrate its use as a prior in an infinite latent feature model we then review recent applications of the indian buffet process in machine learning discuss its extensions and summarize its connections to other stochastic processes

probabilistic principal component analysis principal component analysis pca is a ubiquitous technique for data analysis and processing but one which is not based upon a probability model in this paper we demonstrate how the principal axes of a set of observed data vectors may be determined through maximum likelihood estimation of parameters in a latent variable model closely related to factor analysis we consider the properties of the associated likelihood function giving an em algorithm for estimating the principal subspace iteratively and discuss the advantages conveyed by the definition of a probability density function for pca

blind separation of sound sources from the principle of least spatial entropy abstract the aim of the paper is to offer a method for separating incoherent and compact sound sources which may overlap in both the space and frequency domains this is found of interest in acoustical applications involving the identification and ranking of sound sources stemming from different physical origins the principle proceeds in two steps the first one being reminiscent to source reconstruction e g as in near field acoustical holography and the second one to blind source separation specifically the source mixture is first expanded into a linear combination of spatial basis functions whose coefficients are set by backpropagating the pressures measured by an array of microphones to the source domain this leads to a formulation similar but no identical to blind source separation in the second step these coefficients are blindly separated into uncorrelated latent variables assigned to incoherent virtual sources these are shown to be defined up to an arbitrary rotation a unique set of sound sources is finally recovered by searching for that rotation by conjugate gradient descent in the stiefel manifold of unitary matrices which maximizes their spatial compactness as measured either by their spatial variance or their spatial entropy this results in the proposal of two separation criteria coined least spatial variance and least spatial entropy respectively the same concept of spatial entropy which is central to the paper is also exploited in defining a new criterion the entropic l curve dedicated to determining the number of active sound sources the idea consists in considering the number of sources that achieves the best compromise between a low spatial entropy as expected from compact sources and a low statistical entropy as expected from a low residual error the proposed methodology is validated on both laboratory experiments and numerical data and illustrated on an industrial example concerned with the ranking of sound sources on a diesel engine at the same time its robustness to the estimated number of active sources is demonstrated

minimax estimation in sparse canonical correlation analysis canonical correlation analysis is a widely used multivariate statistical technique for exploring the relation between two sets of variables this paper considers the problem of estimating the leading canonical correlation directions in high dimensional settings recently under the assumption that the leading canonical correlation directions are sparse various procedures have been proposed for many high dimensional applications involving massive data sets however there has been few theoretical justification available in the literature in this paper we establish rate optimal nonasymptotic minimax estimation with respect to an appropriate loss function for a wide range of model spaces two interesting phenomena are observed first the minimax rates are not affected by the presence of nuisance parameters namely the covariance matrices of the two sets of random variables though they need to be estimated in the canonical correlation analysis problem second we allow the presence of the residual canonical correlation directions however they do not influence the minimax rates under a mild condition on eigengap a generalized sin theta theorem and an empirical process bound for gaussian quadratic forms under rank constraint are used to establish the minimax upper bounds which may be of independent interest

exponential family factors for bayesian factor analysis expressing data as linear functions of a small number of unknown variables is a useful approach employed by several classical data analysis methods e g factor analysis principal component analysis or latent semantic indexing these models represent the data using the product of two factors in practice one important concern is how to link the learned factors to relevant quantities in the context of the application to this end various specialized forms of the factors have been proposed to improve interpretability toward developing a unified view and clarifying the statistical significance of the specialized factors we propose a bayesian model family we employ exponential family distributions to specify various types of factors which provide a unified probabilistic formulation a gibbs sampling procedure is constructed as a general computation routine we verify the model by experiments in which the proposed model is shown to be effective in both emulating existing models and motivating new model designs for particular problem settings

minimum probability of error for asynchronous gaussian multiple access channels consider a gaussian multiple access channel shared by k users who transmit asynchronously independent data streams by modulating a set of assigned signal waveforms the uncoded probability of error achievable by optimum multiuser detectors is investigated it is shown that the k user maximum likelihood sequence detector consists of a bank of single user matched filters followed by a viterbi algorithm whose complexity per binary decision is o 2 k the upper bound analysis of this detector follows an approach based on the decomposition of error sequences the issues of convergence and tightness of the bounds are examined and it is shown that the minimum multiuser error probability is equivalent in the iow noise region to that of a single user system with reduced power these results show that the proposed multiuser detectors afford important performance gains over conventional single user systems in which the signal constellation carries the entire burden of complexity required to achieve a given performance level

the em algorithm for kernel matrix completion with auxiliary data in biological data it is often the case that observed data are available only for a subset of samples when a kernel matrix is derived from such data we have to leave the entries for unavailable samples as missing in this paper the missing entries are completed by exploiting an auxiliary kernel matrix derived from another information source the parametric model of kernel matrices is created as a set of spectral variants of the auxiliary kernel matrix and the missing entries are estimated by fitting this model to the existing entries for model fitting we adopt the em algorithm distinguished from the em algorithm of dempster et al 1977 based on the information geometry of positive definite matrices we will report promising results on bacteria clustering experiments using two marker sequences 16s and gyrb

regularized coplanar discriminant analysis for dimensionality reduction abstract the dimensionality reduction methods based on linear embedding such as neighborhood preserving embedding npe sparsity preserving projections spp and collaborative representation based projections crp try to preserve a certain kind of linear representation for each sample after projection however in the transformed low dimensional space the linear relationship between the samples may be changed which cannot make the linear representation based classifiers such as sparse representation based classifier src to achieve higher recognition accuracy in this paper we propose a new linear dimensionality reduction algorithm called regularized coplanar discriminant analysis rcda to address this problem it simultaneously seeks a linear projection matrix and some linear representation coefficients that make the samples from the same class coplanar and the samples from different classes not coplanar the proposed regularization term balances the bias from the optimal linear representation and that from the class mean to avoid overfitting the training data and overcomes the matrix singularity in solving the linear representation coefficients an alternative optimization approach is proposed to solve the rcda model experiments are done on several benchmark face databases and hyperspectral image databases and results show that rcda can obtain better performance than other dimensionality reduction methods

extensions and generalizations of a theorem of widder and of the theory of symmetric local semigroups the theory of symmetric local semigroups due to a klein and l landau j funct anal 44 1981 121 136 is generalized to semigroups indexed by subsets of rn for n 1 the result implies a similar result of a e nussbaum j funct anal 48 1982 213 223 it is further generalized to semigroups that are symmetric local in some directions and unitary in others the results are used to give a simple proof of a devinatz s duke math j 22 1955 185 192 and n i akhiezer s the classical moment problem and some related questions hafner new york 1965 generalization of a theorem of widder concerning the representation of functions as laplace integrals this result is extended to the representation as a laplace integral of a function taking values in b r the set of bounded linear operators on a hilbert space r also a theorem is proved encompassing both the result of devinatz and akhiezer and bochner s theorem on the representation of positive definite functions as fourier integrals

multi parameter local semi groups of hermetian operators abstract it is proven that if a weakly continuous multi parameter family of hermetian operators in a hilbert space whose domains are not necessarily dense but whose union is dense and which has the semi group property for the parameters belonging to an open convex set in euclidean n space with 0 a limit point then the operators have self adjoint extensions which have a joint spectral representation

on large deviations theory and asymptotically efficient monte carlo estimation it is shown how large deviations theory can be applied to construct an asymptotically efficient simulation distribution for monte carlo simulation using the importance sampling technique a sufficient and necessary condition is given for the asymptotic efficiency of the candidate simulation distributions this is done in the multidimensional setting that is required by many practical simulation problems the result obtained is applied primarily in two areas first the generalization of previous work dealing with functionals of markov chains is discussed a second area of application is the simulation of nonlinear systems with gaussian inputs as an example of a system with gaussian inputs the simulation of a digital communication channel with nonlinear isi intersymbol interference characteristic of satellite data links is considered in the case of linear isi the asymptotically efficient exponentially twisted distribution turns out to agree with a method previously proposed by d lu and k yao 1988 the large deviations point of view provides some useful insight on how to extent the lu and yao method to nonlinear channels

reconstructing bifurcation diagrams from noisy time series using nonlinear autoregressive models we introduce a formalism for the reconstruction of bifurcation diagrams from noisy time series the method consists in finding a parametrized predictor function whose bifurcation structure is similar to that of the given system the reconstruction algorithm is composed of two stages model selection and bifurcation parameter identification in the first stage an appropriate model that best represents all the given time series is selected a nonlinear autoregressive model with polynomial terms is employed in this study the identification of the bifurcation parameters from among the many model parameters is done in the second stage the algorithm works well even for a limited number of time series

generalization performance of regularization networks and support vector machines via entropy numbers of compact operators we derive new bounds for the generalization error of kernel machines such as support vector machines and related regularization networks by obtaining new bounds on their covering numbers the proofs make use of a viewpoint that is apparently novel in the field of statistical learning theory the hypothesis class is described in terms of a linear operator mapping from a possibly infinite dimensional unit ball in feature space into a finite dimensional space the covering numbers of the class are then determined via the entropy numbers of the operator these numbers which characterize the degree of compactness of the operator can be bounded in terms of the eigenvalues of an integral operator induced by the kernel function used by the machine as a consequence we are able to theoretically explain the effect of the choice of kernel function on the generalization performance of support vector machines

nonlinear component analysis as a kernel eigenvalue problem a new method for performing a nonlinear form of principal component analysis is proposed by the use of integral operator kernel functions one can efficiently compute principal components in high dimensional feature spaces related to input space by some nonlinear map for instance the space of all possible five pixel products in 16 16 images we give the derivation of the method and present experimental results on polynomial feature extraction for pattern recognition

performance analysis of ssma communication systems with spreading sequences of markov chains large deviations principle versus the central limit theorem in this paper we consider asynchronous spread spectrum multiple access ssma communication systems with spreading sequences of markov chains we show that given phase shifts and time delays the empirical mean of the multiple access interference mai for such systems satisfies the conditional large deviations principle ldp with the convex good rate function by refinement of the large deviations analysis we also obtain exact asymptotics of bit error probabilities in such systems comparing theoretical expressions of bit error probabilities with experimental results we confirm that for too small numbers of users compared to the lengths of spreading sequences the central limit asymptotics become invalid but the large deviations asymptotics turn out to be relevant

a least squares framework for component analysis over the last century component analysis ca methods such as principal component analysis pca linear discriminant analysis lda canonical correlation analysis cca locality preserving projections lpp and spectral clustering sc have been extensively used as a feature extraction step for modeling classification visualization and clustering ca techniques are appealing because many can be formulated as eigen problems offering great potential for learning linear and nonlinear representations of data in closed form however the eigen formulation often conceals important analytic and computational drawbacks of ca techniques such as solving generalized eigen problems with rank deficient matrices e g small sample size problem lacking intuitive interpretation of normalization factors and understanding commonalities and differences between ca methods this paper proposes a unified least squares framework to formulate many ca methods we show how pca lda cca lpp sc and its kernel and regularized extensions correspond to a particular instance of least squares weighted kernel reduced rank regression ls wkrrr the ls wkrrr formulation of ca methods has several benefits 1 provides a clean connection between many ca techniques and an intuitive framework to understand normalization factors 2 yields efficient numerical schemes to solve ca techniques 3 overcomes the small sample size problem 4 provides a framework to easily extend ca methods we derive weighted generalizations of pca lda sc and cca and several new ca techniques

minimax bounds for sparse pca with noisy high dimensional data we study the problem of estimating the leading eigenvectors of a high dimensional population covariance matrix based on independent gaussian observations we establish a lower bound on the minimax risk of estimators under the l2 loss in the joint limit as dimension and sample size increase to infinity under various models of sparsity for the population eigenvectors the lower bound on the risk points to the existence of different regimes of sparsity of the eigenvectors we also propose a new method for estimating the eigenvectors by a two stage coordinate selection scheme

em algorithms for ml factor analysis the details of em algorithms for maximum likelihood factor analysis are presented for both the exploratory and confirmatory models the algorithm is essentially the same for both cases and involves only simple least squares regression operations the largest matrix inversion required is for aq q symmetric matrix whereq is the matrix of factors the example that is used demonstrates that the likelihood for the factor analysis model may have multiple modes that are not simply rotations of each other such behavior should concern users of maximum likelihood factor analysis and certainly should cast doubt on the general utility of second derivatives of the log likelihood as measures of precision of estimation

nonlinear kernel based statistical pattern analysis the eigenstructure of the second order statistics of a multivariate random population can be inferred from the matrix of pairwise combinations of inner products of the samples therefore it can be also efficiently obtained in the implicit high dimensional feature spaces defined by kernel functions we elaborate on this property to obtain general expressions for immediate derivation of nonlinear counterparts of a number of standard pattern analysis algorithms including principal component analysis data compression and denoising and fisher s discriminant the connection between kernel methods and nonparametric density estimation is also illustrated using these results we introduce the kernel version of mahalanobis distance which originates nonparametric models with unexpected and interesting properties and also propose a kernel version of the minimum squared error mse linear discriminant function this learning machine is particularly simple and includes a number of generalized linear models such as the potential functions method or the radial basis function rbf network our results shed some light on the relative merit of feature spaces and inductive bias in the remarkable generalization properties of the support vector machine svm although in most situations the svm obtains the lowest error rates exhaustive experiments with synthetic and natural data show that simple kernel machines based on pseudoinversion are competitive in problems with appreciable class overlapping

reflection positivity and conformal symmetry abstract a reflection positive hilbert space is a triple e e where e is a hilbert space a unitary involution and e a closed subspace on which the hermitian form v w v w is positive semidefinite for a triple g s where g is a lie group an involutive automorphism of g and s a subsemigroup invariant under the involution s s s 1 a unitary representation of g on e e is called reflection positive if g g and s e e this is the first in a series of papers in which we develop a new and systematic approach to reflection positive representations based on reflection positive distributions and reflection positive distribution vectors this approach is most natural to obtain classification results in particular in the abelian case among the tools we develop is a generalization of the bochner schwartz theorem to positive definite distributions on open convex cones we further illustrate our techniques with a non abelian example by constructing reflection positive distribution vectors for complementary series representations of the conformal group o 1 n 1 r of the sphere s n

an introduction to kernel based learning algorithms this paper provides an introduction to support vector machines kernel fisher discriminant analysis and kernel principal component analysis as examples for successful kernel based learning methods we first give a short background about vapnik chervonenkis theory and kernel feature spaces and then proceed to kernel based learning in supervised and unsupervised scenarios including practical and algorithmic considerations we illustrate the usefulness of kernel algorithms by discussing applications such as optical character recognition and dna analysis

directed principal component analysis we consider a problem involving estimation of a high dimensional covariance matrix that is the sum of a diagonal matrix and a low rank matrix and making a decision based on the resulting estimate such problems arise for example in portfolio management where a common approach employs principal component analysis pca to estimate factors used in constructing the low rank term of the covariance matrix the decision problem is typically treated separately with the estimated covariance matrix taken to be an input to an optimization problem we propose directed pca an efficient algorithm that takes the decision objective into account when estimating the covariance matrix directed pca effectively adjusts factors that would be produced by pca so that they better guide the specific decision at hand we demonstrate through computational studies that directed pca yields significant benefit and we prove theoretical results establishing that the degree of improvement over conventional pca can be arbitrarily large

input space versus feature space in kernel based methods this paper collects some ideas targeted at advancing our understanding of the feature spaces associated with support vector sv kernel functions we first discuss the geometry of feature space in particular we review what is known about the shape of the image of input space under the feature space map and how this influences the capacity of sv methods following this we describe how the metric governing the intrinsic geometry of the mapped surface can be computed in terms of the kernel using the example of the class of inhomogeneous polynomial kernels which are often used in sv pattern recognition we then discuss the connection between feature space and input space by dealing with the question of how one can given some vector in feature space find a preimage exact or approximate in input space we describe algorithms to tackle this issue and show their utility in two applications of kernel methods first we use it to reduce the computational complexity of sv decision functions second we combine it with the kernel pca algorithm thereby constructing a nonlinear statistical denoising technique which is shown to perform well on real world data

learning harmonium models with infinite latent features undirected latent variable models represent an important class of graphical models that have been successfully developed to deal with various tasks one common challenge in learning such models is to determine the number of hidden units that are unknown a priori although bayesian nonparametrics have provided promising results in bypassing the model selection problem in learning directed bayesian networks very little effort has been made toward applying bayesian nonparametrics to learn undirected latent variable models in this paper we present the infinite exponential family harmonium iefh a bipartite undirected latent variable model that automatically determines the number of latent units from an unbounded pool we also present two important extensions of iefh to 1 multiview iefh for dealing with heterogeneous data and 2 infinite maximum margin harmonium immh for incorporating supervising side information to learn predictive latent features we develop variational inference algorithms to learn model parameters our methods are computationally competitive because of the avoidance of selecting the number of latent units our extensive experiments on real image datasets and text datasets appear to demonstrate the benefits of iefh and immh inherited from bayesian nonparametrics and max margin learning such results were not available until now and contribute to expanding the scope of bayesian nonparametrics to learn the structures of undirected latent variable models