learning eigenfunctions links spectral embedding and kernel pca in this letter we show a direct relation between spectral embedding methods and kernel principal components analysis and how both are special cases of a more general learning problem learning the principal eigenfunctions of an operator defined from a kernel and the unknown data generating density whereas spectral embedding methods provided only coordinates for the training points the analysis justifies a simple extension to out of sample examples the nystrom formula for multidimensional scaling mds spectral clustering laplacian eigenmaps locally linear embedding lle and isomap the analysis provides for all such spectral embedding methods the definition of a loss function whose empirical average is minimized by the traditional algorithms the asymptotic expected value of that loss defines a generalization performance and clarifies what these algorithms are trying to learn experiments with lle isomap spectral clustering and mds show that this out of sample embedding formula generalizes well with a level of error comparable to the effect of small perturbations of the training set on the embedding

regression on manifolds estimation of the exterior derivative collinearity and near collinearity of predictors cause difficulties when doing regression in these cases variable selection becomes untenable because of mathematical issues concerning the existence and numerical stability of the regression coefficients and interpretation of the coefficients is ambiguous because gradients are not defined using a differential geometric interpretation in which the regression coefficients are interpreted as estimates of the exterior derivative of a function we develop a new method to do regression in the presence of collinearities our regularization scheme can improve estimation error and it can be easily modified to include lasso type regularization these estimators also have simple extensions to the large p small n context

the relative value of labeled and unlabeled samples in pattern recognition with an unknown mixing parameter we observe a training set q composed of l labeled samples x sub 1 spl theta sub 1 x sub l spl theta sub l and u unlabeled samples x sub 1 x sub u the labels spl theta sub i are independent random variables satisfying pr spl theta sub i 1 spl eta pr spl theta sub i 2 1 spl eta the labeled observations x sub i are independently distributed with conditional density f sub spl theta i spl middot given spl theta sub i let x sub 0 spl theta sub 0 be a new sample independently distributed as the samples in the training set we observe x sub 0 and we wish to infer the classification spl theta sub 0 in this paper we first assume that the distributions f sub 1 spl middot and f sub 2 spl middot are given and that the mixing parameter is unknown we show that the relative value of labeled and unlabeled samples in reducing the risk of optimal classifiers is the ratio of the fisher informations they carry about the parameter spl eta we then assume that two densities g sub 1 spl middot and g sub 2 spl middot are given but we do not know whether g sub 1 spl middot f sub 1 spl middot and g sub 2 spl middot f sub 2 spl middot or if the opposite holds nor do we know spl eta thus the learning problem consists of both estimating the optimum partition of the observation space and assigning the classifications to the decision regions here we show that labeled samples are necessary to construct a classification rule and that they are exponentially more valuable than unlabeled samples

pairwise sparsity preserving embedding for unsupervised subspace learning and classification two novel unsupervised dimensionality reduction techniques termed sparse distance preserving embedding sdpe and sparse proximity preserving embedding sppe are proposed for feature extraction and classification sdpe and sppe perform in the clean data space recovered by sparse representation and enhanced euclidean distances over noise removed data are employed to measure pairwise similarities of points in extracting informative features sdpe and sppe aim at preserving pairwise similarities between data points in addition to preserving the sparse characteristics this paper calculates the sparsest representation of all vectors jointly by a convex optimization the sparsest codes enable certain local information of data to be preserved and can endow sdpe and sppe a natural discriminating power adaptive neighborhood and robust characteristic against noise and errors in delivering low dimensional embeddings we also mathematically show sdpe and sppe can be effectively extended for discriminant learning in a supervised manner the validity of sdpe and sppe is examined by extensive simulations comparison with other related state of the art unsupervised algorithms show that promising results are delivered by our techniques

discrete hessian eigenmaps method for dimensionality reduction for a given set of data points lying on a low dimensional manifold embedded in a high dimensional space the dimensionality reduction is to recover a low dimensional parametrization from the data set the recently developed hessian eigenmaps method is a mathematically rigorous method that also sets a theoretical framework for the nonlinear dimensionality reduction problem in this paper we develop a discrete version of the hessian eigenmaps method and present an analysis giving conditions under which the method works as intended as an application a procedure to modify the standard constructions of k nearest neighborhoods is presented to ensure that hessian lle can recover the original coordinates up to an affine transformation

time finite element discretization of hamilton s law of varying action hamilton s law of varying action is used as a variational source for the derivation of finite element discretization procedure in the time domain three different versions of the proposed algorithms are presented and verified for accuracy and stability the first one is the high precision finite time element analogous to the standard finite elements with cutoff frequency the second version is the step by step one time element from which the unconditionally stable with slightly altered accuracy third algorithm is derived the new operator connected with the proposed algorithms bears attractive properties of much greater accuracy than other existing stable methods and easy computer implementation thus the work herein shows that the reservations expressed against the use of finite elements in time domain seem unjustified

manifold regularization a geometric framework for learning from labeled and unlabeled examples we propose a family of learning algorithms based on a new form of regularization that allows us to exploit the geometry of the marginal distribution we focus on a semi supervised framework that incorporates labeled and unlabeled data in a general purpose learner some transductive graph learning algorithms and standard methods including support vector machines and regularized least squares can be obtained as special cases we use properties of reproducing kernel hilbert spaces to prove new representer theorems that provide theoretical basis for the algorithms as a result in contrast to purely graph based approaches we obtain a natural out of sample extension to novel examples and so are able to handle both transductive and truly semi supervised settings we present experimental evidence suggesting that our semi supervised algorithms are able to use unlabeled data effectively finally we have a brief discussion of unsupervised and fully supervised learning within our general framework

dictionary pair learning on grassmann manifolds for image denoising image denoising is a fundamental problem in computer vision and image processing that holds considerable practical importance for real world applications the traditional patch based and sparse coding driven image denoising methods convert 2d image patches into 1d vectors for further processing thus these methods inevitably break down the inherent 2d geometric structure of natural images to overcome this limitation pertaining to the previous image denoising methods we propose a 2d image denoising model namely the dictionary pair learning dpl model and we design a corresponding algorithm called the dpl on the grassmann manifold dplg algorithm the dplg algorithm first learns an initial dictionary pair i e the left and right dictionaries by employing a subspace partition technique on the grassmann manifold wherein the refined dictionary pair is obtained through a sub dictionary pair merging the dplg obtains a sparse representation by encoding each image patch only with the selected sub dictionary pair the non zero elements of the sparse representation are further smoothed by the graph laplacian operator to remove the noise consequently the dplg algorithm not only preserves the inherent 2d geometric structure of natural images but also performs manifold smoothing in the 2d sparse coding space we demonstrate that the dplg algorithm also improves the structural similarity values of the perceptual visual quality for denoised images using the experimental evaluations on the benchmark images and berkeley segmentation data sets moreover the dplg also produces the competitive peak signal to noise ratio values from popular image denoising algorithms

fast approximate k nn graph construction for high dimensional data via recursive lanczos bisection nearest neighbor graphs are widely used in data mining and machine learning a brute force method to compute the exact knn graph takes dn2 time for n data points in the d dimensional euclidean space we propose two divide and conquer methods for computing an approximate knn graph in dnt time for high dimensional data large d the exponent t 1 2 is an increasing function of an internal parameter which governs the size of the common region in the divide step experiments show that a high quality graph can usually be obtained with small overlaps that is for small values of t a few of the practical details of the algorithms are as follows first the divide step uses an inexpensive lanczos procedure to perform recursive spectral bisection after each conquer step an additional refinement step is performed to improve the accuracy of the graph finally a hash table is used to avoid repeating distance calculations during the divide and conquer process the combination of these techniques is shown to yield quite effective algorithms for building knn graphs

spectral embedded clustering a framework for in sample and out of sample spectral clustering spectral clustering sc methods have been successfully applied to many real world applications the success of these sc methods is largely based on the manifold assumption namely that two nearby data points in the high density region of a low dimensional data manifold have the same cluster label however such an assumption might not always hold on high dimensional data when the data do not exhibit a clear low dimensional manifold structure e g high dimensional and sparse data the clustering performance of sc will be degraded and become even worse than k means clustering in this paper motivated by the observation that the true cluster assignment matrix for high dimensional data can be always embedded in a linear space spanned by the data we propose the spectral embedded clustering sec framework in which a linearity regularization is explicitly added into the objective function of sc methods more importantly the proposed sec framework can naturally deal with out of sample data we also present a new laplacian matrix constructed from a local regression of each pattern and incorporate it into our sec framework to capture both local and global discriminative information for clustering comprehensive experiments on eight real world high dimensional datasets demonstrate the effectiveness and advantages of our sec framework over existing sc methods and k means based clustering methods our sec framework significantly outperforms sc using the nystrom algorithm on unseen data

learning the dynamical system behind sensory data this letter looks at the physics behind sensory data by identifying the parameters that govern the physical system and estimating them from sensory observations we extend takens s delay embedding theorem to the dynamical systems controlled by parameters an embedding of the product space of the phase and the parameter spaces of the dynamical system can be obtained by the delay embedding map provided that the parameter of the dynamical system changes slowly the reconstruction error is bounded for slowly varying parameters a manifold learning technique is applied to the embedding obtained to extract a low dimensional global coordinate system representing the product space the phase space of the deterministic dynamics can be contracted by using the adjacency relationship in time which enables recovery of only the parameter space as examples the manifolds of synthetic and real world vowels with time varying fundamental frequency f0 are analyzed and the f0 contours are extracted by an unsupervised algorithm experimental results show that the proposed method leads to robust performance under various noise conditions and rapid changes of f0 compared with the current state of the art f0 estimation algorithms

normalized cuts and image segmentation we propose a novel approach for solving the perceptual grouping problem in vision rather than focusing on local features and their consistencies in the image data our approach aims at extracting the global impression of an image we treat image segmentation as a graph partitioning problem and propose a novel global criterion the normalized cut for segmenting the graph the normalized cut criterion measures both the total dissimilarity between the different groups as well as the total similarity within the groups we show that an efficient computational technique based on a generalized eigenvalue problem can be used to optimize this criterion we applied this approach to segmenting static images as well as motion sequences and found the results to be very encouraging

proximity based frameworks for generating embeddings from multi output data this paper is about supervised and semi supervised dimensionality reduction dr by generating spectral embeddings from multi output data based on the pairwise proximity information two flexible and generic frameworks are proposed to achieve supervised dr sdr for multilabel classification one is able to extend any existing single label sdr to multilabel via sample duplication referred to as mesd the other is a multilabel design framework that tackles the sdr problem by computing weight proximity matrices based on simultaneous feature and label information referred to as mope as a generalization of many current techniques a diverse set of different schemes for label based proximity calculation as well as a mechanism for combining label based and feature based weight information by considering information importance and prioritization are proposed for mope additionally we summarize many current spectral methods for unsupervised dr udr single multilabel sdr and semi supervised dr ssdr and express them under a common template representation as a general guide to researchers in the field we also propose a general framework for achieving ssdr by combining existing sdr and udr models and also a procedure of reducing the computational cost via learning with a target set of relation features the effectiveness of our proposed methodologies is demonstrated with experiments with document collections for multilabel text categorization from the natural language processing domain

multibody system analysis based on hamilton s weak principle the double pendulum a simple system that exhibits complex dynamics is used to demonstrate a method based on hamilton s weak principle hwp for assembling and solving the maximum coordinate set equations for a multibody system results calculated using hwp are compared to a reference solution obtained by integrating the ordinary differential equations ode in minimum coordinate set form the hwp solution is shown to be comparable to the ode solution in accuracy and computational efe ciency these results suggest an alternative architecture for multibody system analysis which implements the maximum coordinate set form of the system equations and a simple efe cient and robust solution algorithm

self regularized fixed rank representation for subspace segmentation abstract consider a set of data points generated from various linear subspaces subspace segmentation tasks which are important in fields such as computer vision and image processing aim to partition the set of data points so as to recover these subspaces the subspace segmentation method fixed rank representation frr was introduced to remedy the problem of insufficient sampling in classical low rank representation lrr in many subspace segmentation applications frr has achieved much better results than those of lrr in this paper a new frr related algorithm called self regularized fixed rank representation srfrr is proposed in srfrr a laplacian regularizer is constructed using the coefficient matrix obtained by srfrr itself further by proving that the laplacian regularizer can be transformed into a structure constraint on the coefficient matrix we show that another existing method sparse frr sfrr is a special case of srfrr to implement the srfrr method we present two optimization algorithms experiments on both synthetic and real databases show that srfrr outperforms some existing frr and lrr related algorithms

adaptive graph construction using data self representativeness for pattern classification graph construction from data constitutes a pre stage in many machine learning and computer vision tasks like semi supervised learning manifold learning and spectral clustering the influence of graph construction procedures on learning tasks and their related applications has only received limited study despite its critical impact on accuracy state of the art graphs are built via sparse coding adopting 1 regularization those graphs exhibit good performance in many computer vision applications however the locality and similarity among instances are not explicitly used in the coding scheme furthermore due to the use of 1 regularization these construction approaches can be computationally expensive in this paper we investigate graph construction using the data self representativeness property by incorporating a variant of locality constrained linear coding llc we introduce and derive four variants for graph construction these variants adopt a two phase llc tpllc compared with the recent 1 graphs our proposed objective function associated with three variants has an analytical solution and thus is more efficient a key element of the proposed methods is the second phase of coding that allows data closeness or locality to be naturally incorporated it performs a coding over some selected relevant samples and reinforces the individual regularization terms by exploiting the coefficients estimated in the first phase comprehensive experimental results using several benchmark datasets show that it can achieve or outperform existing state of the art results furthermore it is shown to be more efficient than the robust 1 graph construction schemes

progressive match expansion via coherent subspace constraint in this paper a quasi dense matching algorithm called progressive match expansion via coherent subspace pecs is proposed our algorithm starts from a set of sparse seeds and iteratively expands potential matches in the region around the seeds until no more matches could be found the main difference from previous methods is that our method computes a coherent non rigid transformation to relate the matching pixels instead of using local planar affine model to approximate the real scene surface within the expanding region the points in the expanding region are treated as a whole and all the matches satisfying the transformation are found at once first dense sift descriptors are extracted and a subspace encoding the feature similarity between two point sets is computed this embedding pulls two points closer if they have similar features and makes the non rigid transformation learning more robust second a coherent non rigid transformation in the subspace is computed to move one point set towards the other and the spatially aligned two points denote a match the proposed model is robust to scenes with non smooth surface and experiment results reveal the good performance in both 2d image matching and 3d reconstruction

improved data visualisation through nonlinear dissimilarity modelling inherent to state of the art dimension reduction algorithms is the assumption that global distances between observations are euclidean despite the potential for altogether non euclidean data manifolds we demonstrate that a non euclidean manifold chart can be approximated by implementing a universal approximator over a dictionary of dissimilarity measures building on recent developments in the field this approach is transferable across domains such that observations can be vectors distributions graphs and time series for instance our novel dissimilarity learning method is illustrated with four standard visualisation datasets showing the benefits over the linear dissimilarity learning approach

local linear regression on manifolds and its geometric interpretation high dimensional data analysis has been an active area and the main focus areas have been variable selection and dimension reduction in practice it occurs often that the variables are located on an unknown lower dimensional nonlinear manifold under this manifold assumption one purpose of this article is regression and gradient estimation on the manifold and another is developing a new tool for manifold learning as regards the first aim we suggest directly reducing the dimensionality to the intrinsic dimension d of the manifold and performing the popular local linear regression llr on a tangent plane estimate an immediate consequence is a dramatic reduction in the computational time when the ambient space dimension p d we provide rigorous theoretical justification of the convergence of the proposed regression and gradient estimators by carefully analyzing the curvature boundary and nonuniform sampling effects we propose a bandwidth selector that can handle heteroscedastic errors with re

nonlinear component analysis as a kernel eigenvalue problem a new method for performing a nonlinear form of principal component analysis is proposed by the use of integral operator kernel functions one can efficiently compute principal components in high dimensional feature spaces related to input space by some nonlinear map for instance the space of all possible five pixel products in 16 16 images we give the derivation of the method and present experimental results on polynomial feature extraction for pattern recognition

riemannian manifold learning recently manifold learning has been widely exploited in pattern recognition data analysis and machine learning this paper presents a novel framework called riemannian manifold learning rml based on the assumption that the input high dimensional data lie on an intrinsically low dimensional riemannian manifold the main idea is to formulate the dimensionality reduction problem as a classical problem in riemannian geometry that is how to construct coordinate charts for a given riemannian manifold we implement the riemannian normal coordinate chart which has been the most widely used in riemannian geometry for a set of unorganized data points first two input parameters the neighborhood size k and the intrinsic dimension d are estimated based on an efficient simplicial reconstruction of the underlying manifold then the normal coordinates are computed to map the input high dimensional data into a low dimensional space experiments on synthetic data as well as real world images demonstrate that our algorithm can learn intrinsic geometric structures of the data preserve radial geodesic distances and yield regular embeddings

semisupervised multicategory classification with imperfect model semisupervised learning has been of growing interest over the past years and many methods have been proposed while existing semisupervised methods have shown some promising empirical performances their development has been based largely on heuristics in this paper we investigate semisupervised multicategory classification with an imperfect mixture density model in the proposed model the training data come from a probability distribution which can be modeled imperfectly by an identifiable mixture distribution furthermore we propose a semisupervised multicategory classification method and establish its generalization error bounds the theoretical analysis illustrates that the proposed method can utilize unlabeled data effectively and can achieve fast convergence rate

clustering documents with labeled and unlabeled documents using fuzzy semi kmeans while focusing on document clustering this work presents a fuzzy semi supervised clustering algorithm called fuzzy semi kmeans the fuzzy semi kmeans is an extension of k means clustering model and it is inspired by an em algorithm and a gaussian mixture model additionally the fuzzy semi kmeans provides the flexibility to employ different fuzzy membership functions to measure the distance between data this work employs gaussian weighting function to conduct experiments but cosine similarity function can be used as well this work conducts experiments on three data sets and compares fuzzy semi kmeans with several methods the experimental results indicate that fuzzy semi kmeans can generally outperform the other methods

toward optimal manifold hashing via discrete locally linear embedding binary code learning also known as hashing has received increasing attention in large scale visual search by transforming high dimensional features to binary codes the original euclidean distance is approximated via hamming distance more recently it is advocated that it is the manifold distance rather than the euclidean distance that should be preserved in the hamming space however it retains as an open problem to directly preserve the manifold structure by hashing in particular it first needs to build the local linear embedding in the original feature space and then quantize such embedding to binary codes such a two step coding is problematic and less optimized besides the off line learning is extremely time and memory consuming which needs to calculate the similarity matrix of the original data in this paper we propose a novel hashing algorithm termed discrete locality linear embedding hashing dllh which well addresses the above challenges the dllh directly reconstructs the manifold structure in the hamming space which learns optimal hash codes to maintain the local linear relationship of data points to learn discrete locally linear embedding codes we further propose a discrete optimization algorithm with an iterative parameters updating scheme moreover an anchor based acceleration scheme termed anchor dllh is further introduced which approximates the large similarity matrix by the product of two low rank matrices experimental results on three widely used benchmark data sets i e cifar10 nus wide and youtube face have shown superior performance of the proposed dllh over the state of the art approaches

nonlinear embedding preserving multiple local linearities locally linear embedding lle is one of the effective and efficient algorithms for nonlinear dimensionality reduction this paper discusses the stability of lle focusing on the optimal weights for extracting local linearity behind the considered manifold it is proven that there are multiple sets of weights that are approximately optimal and can be used to improve the stability of lle a new algorithm using multiple weights is then proposed together with techniques for constructing multiple weights this algorithm is called as nonlinear embedding preserving multiple local linearities neml neml improves the preservation of local linearity and is more stable than lle a short analysis for neml is also given for isometric manifolds neml is compared with the local tangent space alignment ltsa in methodology since both of them adopt multiple local constraints numerical examples are given to show the improvement and efficiency of neml

least squares methods for mixed type equations a least squares method for the numerical solution of a class of second order mixed type partial differential equations is analyzed the method offers a unified computational treatment of both elliptic and hyperbolic parts of a given domain moreover it has the advantage over other methods that its implementation is independent of specific properties of a given problem convergence of the approximation is proved in a natural norm assuming smooth solutions a particular least squares method is implemented using bicubic spline polynomials and numerical results for test problems are presented

intrinsic dimension estimation of manifolds by incising balls dimensionality reduction is a very important tool in data mining intrinsic dimension of data sets is a key parameter for dimensionality reduction however finding the correct intrinsic dimension is a challenging task in this paper a new intrinsic dimension estimation method is presented the estimator is derived by finding the exponential relationship between the radius of an incising ball and the number of samples included in the ball the method is compared with the previous dimension estimation methods experiments have been conducted on synthetic and high dimensional image data sets and on data sets of the santa fe time series competition and the results show that the new method is accurate and robust

the convergence rate of semi supervised regression with quadratic loss abstract it is known that the semi supervised learning deals with learning algorithms with less labeled samples and more unlabeled samples one of the problems in this field is to show at what extent the performance depends upon the unlabeled number a kind of modified semi supervised regularized regression with quadratic loss is provided the convergence rate for the error estimate is given in expectation mean it is shown that the learning rate is controlled by the number of the unlabeled samples and the algorithm converges with the increasing of the unlabeled sample number

a finite element method for a boundary value problem of mixed type we consider the numerical solution of the boundary value problem 1 lu k y u xx u yy c x y u f u gamma 0 cup gamma 1 0 where k y gtreqless 0 for y gtreqless 0 0 and 1 are parts of the boundary of a bounded simply connected regiong inr 2 g is bounded fory 0 by a piecewise smooth curve 0 which intersects the liney 0 at the pointsa 1 0 andb 0 0 fory 0 g is bounded by a piecewise smooth curve 1 througha which meets the characteristic of 1 issued fromb at pointc and by the curve 2 which consists of the portioncb of the characteristic throughb using a weak formulation based on different spaces of test and trial functions we construct a galerkin procedure for the above boundary value problem existence uniqueness and uniform stability of an approximate solution is proven and a priori error bounds are given

numerical investigation of constrained direct solutions using hamilton s law this paper presents numerical results obtained for a dynamic system that is modeled using hamilton s law of varying action and for which geometric constraints are enforced explicitly these results are compared with numerical results for the same system that were obtained from ordinary differential equations differential algebraic equations and a formulation based on hamilton s law in which the constraints were enforced implicitly this comparison had two objectives first to show that numerical results from the formulation based on hamilton s law and explicit constraints were equal or superior to results from other formulations second to verify that explicit constraints must be enforced at the end of the time step both objectives were successfully accomplished

characterizing the functional connectivity diversity of the insula cortex subregions diversity curves and behavior the connectivity of the insula cortex is diverse we present new models to characterize the resting state connectional diversity of the human insula cortex and perform model selection using high quality fmri data from the human connectome project we first attempt to parcellate the insula into distinct subregions using traditional clustering methods but find that the resulting subregions are not homogeneous and that the optimal number of subregions is substantially influenced by data smoothness we then introduce the concept of a diversity curve which we use to continuously parameterize the insula s laplacian eigenmap with respect to streamlines propagated through the eigenmap s gradient field to perform model selection we compare the insula s diversity curve to benchmark diversity curves for i two distinct regions ii a continuum of gradual change and iii an absence of any connectional diversity i e homogenous region of the three benchmarks tested we find that the insula s connectional diversity is most parsimoniously modeled as continuum of gradual change from dorsal posterior to ventral anterior we find that individuals who score high on measures of positive affect self efficacy emotion recognition motor dexterity and gustation show greater diversity within the anterior insula our findings are replicated using data from a second fmri session we conclude that the functional connectivity diversity of the insula can be characterized parsimoniously as a continuum avoiding the vexed task of determining an optimal number of insula subregions and that inter individual variation in this continuum can explain significant variation in behavior

a least squares framework for component analysis over the last century component analysis ca methods such as principal component analysis pca linear discriminant analysis lda canonical correlation analysis cca locality preserving projections lpp and spectral clustering sc have been extensively used as a feature extraction step for modeling classification visualization and clustering ca techniques are appealing because many can be formulated as eigen problems offering great potential for learning linear and nonlinear representations of data in closed form however the eigen formulation often conceals important analytic and computational drawbacks of ca techniques such as solving generalized eigen problems with rank deficient matrices e g small sample size problem lacking intuitive interpretation of normalization factors and understanding commonalities and differences between ca methods this paper proposes a unified least squares framework to formulate many ca methods we show how pca lda cca lpp sc and its kernel and regularized extensions correspond to a particular instance of least squares weighted kernel reduced rank regression ls wkrrr the ls wkrrr formulation of ca methods has several benefits 1 provides a clean connection between many ca techniques and an intuitive framework to understand normalization factors 2 yields efficient numerical schemes to solve ca techniques 3 overcomes the small sample size problem 4 provides a framework to easily extend ca methods we derive weighted generalizations of pca lda sc and cca and several new ca techniques

image completion by diffusion maps and spectral relaxation we present a framework for image inpainting that utilizes the diffusion framework approach to spectral dimensionality reduction we show that on formulating the inpainting problem in the embedding domain the domain to be inpainted is smoother in general particularly for the textured images thus the textured images can be inpainted through simple exemplar based and variational methods we discuss the properties of the induced smoothness and relate it to the underlying assumptions used in contemporary inpainting schemes as the diffusion embedding is nonlinear and noninvertible we propose a novel computational approach to approximate the inverse mapping from the inpainted embedding space to the image domain we formulate the mapping as a discrete optimization problem solved through spectral relaxation the effectiveness of the presented method is exemplified by inpainting real images where it is shown to compare favorably with contemporary state of the art schemes

a galerkin method for elliptic hyperbolic type equations a certain galerkin type method is proposed for the numerical solution of a class of second order linear mixed type partial differential equations convergence of such an approximation is proved in the whole region of definition error estimates are derived in a natural norm assuming smooth solutions numerical experiments are conducted using bicubic splines the obtained results show that solutions are approximated equally well in both elliptic and hyperbolic regions

face recognition using elasticfaces though principle component analysis pca and locality preserving projections lpps are two of the most popular linear methods for face recognition pca can only see the euclidean structure of the training set and lpp preserves the nonlinear submanifold structure hidden in the training set in this paper we propose the elastic preserving projections epps which by incorporating the merits of the local geometry and the global information of the training set epp outputs a sample subspace which simultaneously preserves the local geometrical structure and exploits the global information of the training set different from some other linear dimensionality reduction methods epp can be deemed as learning both the coordinates and the affinities between sample points furthermore the effectiveness of our proposed algorithm is analyzed theoretically and confirmed by some experiments on several well known face databases the obtained results indicate that epp significantly outperforms its other rival algorithms

low dimensional embedding of fmri datasets abstract we propose a novel method to embed a functional magnetic resonance imaging fmri dataset in a low dimensional space the embedding optimally preserves the local functional coupling between fmri time series and provides a low dimensional coordinate system for detecting activated voxels to compute the embedding we build a graph of functionally connected voxels we use the commute time instead of the geodesic distance to measure functional distances on the graph because the commute time can be computed directly from the eigenvectors of a symmetric version the graph probability transition matrix we use these eigenvectors to embed the dataset in low dimensions after clustering the datasets in low dimensions coherent structures emerge that can be easily interpreted we performed an extensive evaluation of our method comparing it to linear and nonlinear techniques using synthetic datasets and in vivo datasets we analyzed datasets from the ebc competition obtained with subjects interacting in an urban virtual reality environment our exploratory approach is able to detect independently visual areas v1 v2 v5 mt auditory areas and language areas our method can be used to analyze fmri collected during natural stimuli

data visualization and dimensionality reduction using kernel maps with a reference point in this paper a new kernel based method for data visualization and dimensionality reduction is proposed a reference point is considered corresponding to additional constraints taken in the problem formulation in contrast with the class of kernel eigenmap methods the solution coordinates in the low dimensional space is characterized by a linear system instead of an eigenvalue problem the kernel maps with a reference point are generated from a least squares support vector machine ls svm core part that is extended with an additional regularization term for preserving local mutual distances together with reference point constraints the kernel maps possess primal and dual model representations and provide out of sample extensions e g for validation based tuning the method is illustrated on toy problems and real life data sets

density sensitive semisupervised inference semisupervised methods are techniques for using labeled data x 1 y 1 ldots x n y n together with unlabeled data x n 1 ldots x n to make predictions these methods invoke some assumptions that link the marginal distribution p x of x to the regression function f x for example it is common to assume that f is very smooth over high density regions of p x many of the methods are ad hoc and have been shown to work in specific examples but are lacking a theoretical foundation we provide a minimax framework for analyzing semisupervised methods in particular we study methods based on metrics that are sensitive to the distribution p x our model includes a parameter alpha that controls the strength of the semisupervised assumption we then use the data to adapt to alpha

approximate orthogonal sparse embedding for dimensionality reduction locally linear embedding lle is one of the most well known manifold learning methods as the representative linear extension of lle orthogonal neighborhood preserving projection onpp has attracted widespread attention in the field of dimensionality reduction in this paper a unified sparse learning framework is proposed by introducing the sparsity or l 1 norm learning which further extends the lle based methods to sparse cases theoretical connections between the onpp and the proposed sparse linear embedding are discovered the optimal sparse embeddings derived from the proposed framework can be computed by iterating the modified elastic net and singular value decomposition we also show that the proposed model can be viewed as a general model for sparse linear and nonlinear kernel subspace learning based on this general model sparse kernel embedding is also proposed for nonlinear sparse feature extraction extensive experiments on five databases demonstrate that the proposed sparse learning framework performs better than the existing subspace learning algorithm particularly in the cases of small sample sizes

robust locally linear embedding in the past few years some nonlinear dimensionality reduction nldr or nonlinear manifold learning methods have aroused a great deal of interest in the machine learning community these methods are promising in that they can automatically discover the low dimensional nonlinear manifold in a high dimensional data space and then embed the data points into a low dimensional embedding space using tractable linear algebraic techniques that are easy to implement and are not prone to local minima despite their appealing properties these nldr methods are not robust against outliers in the data yet so far very little has been done to address the robustness problem in this paper we address this problem in the context of an nldr method called locally linear embedding lle based on robust estimation techniques we propose an approach to make lle more robust we refer to this approach as robust locally linear embedding rlle we also present several specific methods for realizing this general rlle approach experimental results on both synthetic and real world data show that rlle is very robust against outliers

pattern vectors from algebraic graph theory graph structures have proven computationally cumbersome for pattern analysis the reason for this is that before graphs can be converted to pattern vectors correspondences must be established between the nodes of structures which are potentially of different size to overcome this problem in this paper we turn to the spectral decomposition of the laplacian matrix we show how the elements of the spectral matrix for the laplacian can be used to construct symmetric polynomials that are permutation invariants the coefficients of these polynomials can be used as graph features which can be encoded in a vectorial manner we extend this representation to graphs in which there are unary attributes on the nodes and binary attributes on the edges by using the spectral decomposition of a hermitian property matrix that can be viewed as a complex analogue of the laplacian to embed the graphs in a pattern space we explore whether the vectors of invariants can be embedded in a low dimensional space using a number of alternative strategies including principal components analysis pca multidimensional scaling mds and locality preserving projection lpp experimentally we demonstrate that the embeddings result in well defined graph clusters our experiments with the spectral representation involve both synthetic and real world data the experiments with synthetic data demonstrate that the distances between spectral feature vectors can be used to discriminate between graphs on the basis of their structure the real world experiments show that the method can be used to locate clusters of graphs

laplacian eigenmaps for dimensionality reduction and data representation one of the central problems in machine learning and pattern recognition is to develop appropriate representations for complex data we consider the problem of constructing a representation for data lying on a low dimensional manifold embedded in a high dimensional space drawing on the correspondence between the graph laplacian the laplace beltrami operator on the manifold and the connections to the heat equation we propose a geometrically motivated algorithm for representing the high dimensional data the algorithm provides a computationally efficient approach to nonlinear dimensionality reduction that has locality preserving properties and a natural connection to clustering some potential applications and illustrative examples are discussed

linear dimensionality reduction survey insights and generalizations linear dimensionality reduction methods are a cornerstone of analyzing high dimensional data due to their simple geometric interpretations and typically attractive computational properties these methods capture many data features of interest such as covariance dynamical structure correlation between data sets input output relationships and margin between data classes methods have been developed with a variety of names and motivations in many fields and perhaps as a result the connections between all these methods have not been highlighted here we survey methods from this disparate literature as optimization programs over matrix manifolds we discuss principal component analysis factor analysis linear multidimensional scaling fisher s linear discriminant analysis canonical correlations analysis maximum autocorrelation factors slow feature analysis sufficient dimensionality reduction undercomplete independent component analysis linear regression distance metric learning and more this optimization framework gives insight to some rarely discussed shortcomings of well known methods such as the suboptimality of certain eigenvector solutions modern techniques for optimization over matrix manifolds enable a generic linear dimensionality reduction solver which accepts as input data and an objective to be optimized and returns as output an optimal low dimensional projection of the data this simple optimization framework further allows straightforward generalizations and novel variants of classical methods which we demonstrate here by creating an orthogonal projection canonical correlations analysis more broadly this survey and generic solver suggest that linear dimensionality reduction can move toward becoming a blackbox objective agnostic numerical technology

semi supervised approaches to efficient evaluation of model prediction performance in many modern machine learning applications the outcome is expensive or time consuming to collect whereas the predictor information is easy to obtain semi supervised ss learning aims at utilizing large amounts of unlabelled data along with small amounts of labelled data to improve the efficiency of a classical supervised approach though numerous ss learning classification and prediction procedures have been proposed in recent years no methods currently exist to evaluate the prediction performance of a working regression model in the context of developing phenotyping algorithms derived from electronic medical records we present an efficient two step estimation procedure for evaluating a binary classifier based on various prediction performance measures in the ss setting in step i the labelled data are used to obtain a non parametrically calibrated estimate of the conditional risk function in step ii ss estimates of the prediction accuracy parameters are constructed based on the estimated conditional risk function and the unlabelled data we demonstrate that under mild regularity conditions the estimators proposed are consistent and asymptotically normal importantly the asymptotic variance of the ss estimators is always smaller than that of the supervised counterparts under correct model specification we also correct for potential overfitting bias in the ss estimators in finite samples with cross validation and we develop a perturbation resampling procedure to approximate their distributions our proposals are evaluated through extensive simulation studies and illustrated with two real electronic medical record studies aiming to develop phenotyping algorithms for rheumatoid arthritis and multiple sclerosis

manifold regularized reinforcement learning this paper introduces a novel manifold regularized reinforcement learning scheme for continuous markov decision processes smooth feature representations for value function approximation can be automatically learned using the unsupervised manifold regularization method the learned features are data driven and can be adapted to the geometry of the state space furthermore the scheme provides a direct basis representation extension for novel samples during policy learning and control the performance of the proposed scheme is evaluated on two benchmark control tasks i e the inverted pendulum and the energy storage problem simulation results illustrate the concepts of the proposed scheme and show that it can obtain excellent performance

principal manifolds and nonlinear dimensionality reduction via tangent space alignment we present a new algorithm for manifold learning and nonlinear dimensionality reduction based on a set of unorganized data points sampled with noise from a parameterized manifold the local geometry of the manifold is learned by constructing an approximation for the tangent space at each data point and those tangent spaces are then aligned to give the global coordinates of the data points with respect to the underlying manifold we also present an error analysis of our algorithm showing that reconstruction errors can be quite small in some cases we illustrate our algorithm using curves and surfaces both in two dimensional three dimensional 2d 3d euclidean spaces and in higher dimensional euclidean spaces we also address several theoretical and algorithmic issues for further research and improvements

manifold regularization and semi supervised learning some theoretical analyses manifold regularization belkin et al 2006 is a geometrically motivated framework for machine learning within which several semi supervised algorithms have been constructed here we try to provide some theoretical understanding of this approach our main result is to expose the natural structure of a class of problems on which manifold regularization methods are helpful we show that for such problems no supervised learner can learn effectively on the other hand a manifold based learner that knows the manifold or learns it from unlabeled examples can learn with relatively few labeled examples our analysis follows a minimax style with an emphasis on finite sample results in terms of n the number of labeled examples these results allow us to properly interpret manifold regularization and related spectral and geometric algorithms in terms of their potential use in semi supervised learning

graph spectral image smoothing using the heat kernel a new method for smoothing both gray scale and color images is presented that relies on the heat diffusion equation on a graph we represent the image pixel lattice using a weighted undirected graph the edge weights of the graph are determined by the gaussian weighted distances between local neighboring windows we then compute the associated laplacian matrix the degree matrix minus the adjacency matrix anisotropic diffusion across this weighted graph structure with time is captured by the heat equation and the solution i e the heat kernel is found by exponentiating the laplacian eigensystem with time image smoothing is accomplished by convolving the heat kernel with the image and its numerical implementation is realized by using the krylov subspace technique the method has the effect of smoothing within regions but does not blur region boundaries we also demonstrate the relationship between our method standard diffusion based pdes fourier domain signal processing and spectral clustering experiments and comparisons on standard images illustrate the effectiveness of the method