generalized multiple kernel learning with data dependent priors multiple kernel learning mkl and classifier ensemble are two mainstream methods for solving learning problems in which some sets of features views are more informative than others or the features views within a given set are inconsistent in this paper we first present a novel probabilistic interpretation of mkl such that maximum entropy discrimination with a noninformative prior over multiple views is equivalent to the formulation of mkl instead of using the noninformative prior we introduce a novel data dependent prior based on an ensemble of kernel predictors which enhances the prediction performance of mkl by leveraging the merits of the classifier ensemble with the proposed probabilistic framework of mkl we propose a hierarchical bayesian model to learn the proposed data dependent prior and classification model simultaneously the resultant problem is convex and other information e g instances with either missing views or missing labels can be seamlessly incorporated into the data dependent priors furthermore a variety of existing mkl models can be recovered under the proposed mkl framework and can be readily extended to incorporate these priors extensive experiments demonstrate the benefits of our proposed framework in supervised and semisupervised settings as well as in tasks with partial correspondence among multiple views

regression on manifolds estimation of the exterior derivative collinearity and near collinearity of predictors cause difficulties when doing regression in these cases variable selection becomes untenable because of mathematical issues concerning the existence and numerical stability of the regression coefficients and interpretation of the coefficients is ambiguous because gradients are not defined using a differential geometric interpretation in which the regression coefficients are interpreted as estimates of the exterior derivative of a function we develop a new method to do regression in the presence of collinearities our regularization scheme can improve estimation error and it can be easily modified to include lasso type regularization these estimators also have simple extensions to the large p small n context

multiview hessian regularization for image annotation the rapid development of computer hardware and internet technology makes large scale data dependent models computationally tractable and opens a bright avenue for annotating images through innovative machine learning algorithms semisupervised learning ssl therefore received intensive attention in recent years and was successfully deployed in image annotation one representative work in ssl is laplacian regularization lr which smoothes the conditional distribution for classification along the manifold encoded in the graph laplacian however it is observed that lr biases the classification function toward a constant function that possibly results in poor generalization in addition lr is developed to handle uniformly distributed data or single view data although instances or objects such as images and videos are usually represented by multiview features such as color shape and texture in this paper we present multiview hessian regularization mhr to address the above two problems in lr based image annotation in particular mhr optimally combines multiple hr each of which is obtained from a particular view of instances and steers the classification function that varies linearly along the data manifold we apply mhr to kernel least squares and support vector machines as two examples for image annotation extensive experiments on the pascal voc 07 dataset validate the effectiveness of mhr by comparing it with baseline algorithms including lr and hr

learning graphs with monotone topology properties and multiple connected components recent papers have formulated the problem of learning graphs from data as an inverse covariance estimation problem with graph laplacian constraints while such problems are convex existing methods cannot guarantee that solutions will have specific graph topology properties e g being a tree which are desirable for some applications the problem of learning a graph with topology properties is in general non convex in this paper we propose an approach to solve these problems by decomposing them into two sub problems for which efficient solutions are known specifically a graph topology inference gti step is employed to select a feasible graph topology then a graph weight estimation gwe step is performed by solving a generalized graph laplacian estimation problem where edges are constrained by the topology found in the gti step our main result is a bound on the error of the gwe step as a function of the error in the gti step this error bound indicates that the gti step should be solved using an algorithm that approximates the data similarity matrix by another matrix whose entries have been thresholded to zero to have the desired type of graph topology the gti stage can leverage existing methods which are typically based on minimizing the total weight of removed edges since the gwe stage is an inverse covariance estimation problem with linear constraints it can be solved using existing convex optimization methods we demonstrate that our approach can achieve good results for both synthetic and texture image data

adaptive filtering in subbands with critical sampling analysis experiments and application to acoustic echo cancellation an exact analysis of the critically subsampled two band modelization scheme is given and it is demonstrated that adaptive cross filters between the subbands are necessary for modelization with small output errors it is shown that perfect reconstruction filter banks can yield exact modelization these results are extended to the critically subsampled multiband schemes and important computational savings are seen to be achieved by using good quality filter banks the problem of adaptive identification in critically subsampled subbands is considered and an appropriate adaptation algorithm is derived the authors give a detailed analysis of the computational complexity of all the discussed schemes and experimentally verify the theoretical results that are obtained the adaptive behavior of the subband schemes that were tested is discussed

consistency of spectral clustering consistency is a key property of all statistical procedures analyzing randomly sampled data surprisingly despite decades of work little is known about consistency of most clustering algorithms in this paper we investigate consistency of the popular family of spectral clustering algorithms which clusters the data with the help of eigenvectors of graph laplacian matrices we develop new methods to establish that for increasing sample size those eigenvectors converge to the eigenvectors of certain limit operators as a result we can prove that one of the two major classes of spectral clustering normalized clustering converges under very general conditions while the other unnormalized clustering is only consistent under strong additional assumptions which are not always satisfied in real data we conclude that our analysis provides strong evidence for the superiority of normalized spectral clustering

manifold regularization a geometric framework for learning from labeled and unlabeled examples we propose a family of learning algorithms based on a new form of regularization that allows us to exploit the geometry of the marginal distribution we focus on a semi supervised framework that incorporates labeled and unlabeled data in a general purpose learner some transductive graph learning algorithms and standard methods including support vector machines and regularized least squares can be obtained as special cases we use properties of reproducing kernel hilbert spaces to prove new representer theorems that provide theoretical basis for the algorithms as a result in contrast to purely graph based approaches we obtain a natural out of sample extension to novel examples and so are able to handle both transductive and truly semi supervised settings we present experimental evidence suggesting that our semi supervised algorithms are able to use unlabeled data effectively finally we have a brief discussion of unsupervised and fully supervised learning within our general framework

generalization error bounds in semi supervised classification under the cluster assumption we consider semi supervised classification when part of the available data is unlabeled these unlabeled data can be useful for the classification problem when we make an assumption relating the behavior of the regression function to that of the marginal distribution seeger 2000 proposed the well known cluster assumption as a reasonable one we propose a mathematical formulation of this assumption and a method based on density level sets estimation that takes advantage of it to achieve fast rates of convergence both in the number of unlabeled examples and the number of labeled examples

minimum density hyperplanes associating distinct groups of objects clusters with contiguous regions of high probability density high density clusters is central to many statistical and machine learning approaches to the classification of unlabelled data we propose a novel hyperplane classifier for clustering and semi supervised classification which is motivated by this objective the proposed minimum density hyperplane minimises the integral of the empirical probability density function along it thereby avoiding intersection with high density clusters we show that the minimum density and the maximum margin hyperplanes are asymptotically equivalent thus linking this approach to maximum margin clustering and semi supervised support vector classifiers we propose a projection pursuit formulation of the associated optimisation problem which allows us to nd minimum density hyperplanes efficiently in practice and evaluate its performance on a range of benchmark data sets the proposed approach is found to be very competitive with state of the art methods for clustering and semi supervised classification

unified framework for representing and ranking in the database retrieval and nearest neighbor classification tasks the two basic problems are to represent the query and database objects and to learn the ranking scores of the database objects to the query many studies have been conducted for the representation learning and the ranking score learning problems however they are always learned independently from each other in this paper we argue that there are some inner relationships between the representation and ranking of database objects and try to investigate their relationships by learning them in a unified way to this end we proposed the unified framework for representation and ranking ur2 of objects for the database retrieval and nearest neighbor classification tasks the learning of representation parameter and the ranking scores are modeled within one single unified objective function the objective function is optimized alternately with regard to representation parameter and the ranking scores based on the optimization results iterative algorithms are developed to learn the representation parameter and the ranking scores on a unified way moreover with two different formulas of representation feature selection and subspace learning we give two versions of ur2 the proposed algorithms are tested on two challenging tasks mri image based brain tumor retrieval and nearest neighbor classification based protein identification the experiments show the advantage of the proposed unified framework over the state of the art independent representation and ranking methods highlightsobject representation and ranking are two basic problems for database retrieval they are always studies separately we proposed the first unified framework for both representation and ranking experimental results show its advantage over independent representation or ranking methods

spectral embedded clustering a framework for in sample and out of sample spectral clustering spectral clustering sc methods have been successfully applied to many real world applications the success of these sc methods is largely based on the manifold assumption namely that two nearby data points in the high density region of a low dimensional data manifold have the same cluster label however such an assumption might not always hold on high dimensional data when the data do not exhibit a clear low dimensional manifold structure e g high dimensional and sparse data the clustering performance of sc will be degraded and become even worse than k means clustering in this paper motivated by the observation that the true cluster assignment matrix for high dimensional data can be always embedded in a linear space spanned by the data we propose the spectral embedded clustering sec framework in which a linearity regularization is explicitly added into the objective function of sc methods more importantly the proposed sec framework can naturally deal with out of sample data we also present a new laplacian matrix constructed from a local regression of each pattern and incorporate it into our sec framework to capture both local and global discriminative information for clustering comprehensive experiments on eight real world high dimensional datasets demonstrate the effectiveness and advantages of our sec framework over existing sc methods and k means based clustering methods our sec framework significantly outperforms sc using the nystrom algorithm on unseen data

dirac operator on a disk with global boundary conditions we compute the functional determinant for a dirac operator in the presence of an abelian gauge field on a bidimensional disk under global boundary conditions of the type introduced by atiyah patodi singer we also discuss the connection between our result and the index theorem

antisymmetric tensor fields on spheres functional determinants and non local counterterms the hodge de rham laplacian on spheres acting on antisymmetric tensor fields is considered explicit expressions for the spectrum are derived in a quite direct way confirming previous results associated functional determinants and the heat kernel expansion are evaluated using this method new non local counterterms in the quantum effective action are obtained which can be expressed in terms of betti numbers

discriminative semi supervised feature selection via manifold regularization feature selection has attracted a huge amount of interest in both research and application communities of data mining we consider the problem of semi supervised feature selection where we are given a small amount of labeled examples and a large amount of unlabeled examples since a small number of labeled samples are usually insufficient for identifying the relevant features the critical problem arising from semi supervised feature selection is how to take advantage of the information underneath the unlabeled data to address this problem we propose a novel discriminative semi supervised feature selection method based on the idea of manifold regularization the proposed approach selects features through maximizing the classification margin between different classes and simultaneously exploiting the geometry of the probability distribution that generates both labeled and unlabeled data in comparison with previous semi supervised feature selection algorithms our proposed semi supervised feature selection method is an embedded feature selection method and is able to find more discriminative features we formulate the proposed feature selection method into a convex concave optimization problem where the saddle point corresponds to the optimal solution to find the optimal solution the level method a fairly recent optimization method is employed we also present a theoretic proof of the convergence rate for the application of the level method to our problem empirical evaluation on several benchmark data sets demonstrates the effectiveness of the proposed semi supervised feature selection method

fisher regularized support vector machine this paper proposes fisher regularized support vector machine fishersvm fishersvm is a graph based supervised learning method fishersvm has two regularization terms maximum margin and fisher regularizations fishersvm aims to maximize the margin and minimize the within class scatter support vector machine svm and fisher discriminant analysis fda are two commonly used methods in machine learning and pattern recognition a combined method of the linear svm and fda called svm lda linear discriminant analysis has been proposed only for the linear case this paper generalizes this combined method to the nonlinear case from the view of regularization a fisher regularization is defined and incorporated into svm to obtain a fisher regularized support vector machine fishersvm in fishersvm there are two regularization terms the maximum margin regularization and fisher regularization which allow fishersvm to maximize the classification margin and minimize the within class scatter roughly speaking fishersvm can approximatively fulfill the fisher criterion and obtain good statistical separability this paper also discusses the connections of fishersvm to graph based regularization methods and the mathematical programming method to kernel fisher discriminant analysis kfda it shows that fishersvm can be thought of as a graph based supervised learning method or a robust kfda experimental results on artificial and real world data show that fishersvm has a promising generalization performance

local linear regression on manifolds and its geometric interpretation high dimensional data analysis has been an active area and the main focus areas have been variable selection and dimension reduction in practice it occurs often that the variables are located on an unknown lower dimensional nonlinear manifold under this manifold assumption one purpose of this article is regression and gradient estimation on the manifold and another is developing a new tool for manifold learning as regards the first aim we suggest directly reducing the dimensionality to the intrinsic dimension d of the manifold and performing the popular local linear regression llr on a tangent plane estimate an immediate consequence is a dramatic reduction in the computational time when the ambient space dimension p d we provide rigorous theoretical justification of the convergence of the proposed regression and gradient estimators by carefully analyzing the curvature boundary and nonuniform sampling effects we propose a bandwidth selector that can handle heteroscedastic errors with re

determinants of dirac operators with local boundary conditions we study functional determinants for dirac operators on manifolds with boundary we give for local boundary conditions an explicit formula relating these determinants to the corresponding green s functions we finally apply this result to the case of a bidimensional disk under baglike conditions

flexible manifold embedding a framework for semi supervised and unsupervised dimension reduction we propose a unified manifold learning framework for semi supervised and unsupervised dimension reduction by employing a simple but effective linear regression function to map the new data points for semi supervised dimension reduction we aim to find the optimal prediction labels f for all the training samples x the linear regression function h x and the regression residue f0 f h x simultaneously our new objective function integrates two terms related to label fitness and manifold smoothness as well as a flexible penalty term defined on the residue f0 our semi supervised learning framework referred to as flexible manifold embedding fme can effectively utilize label information from labeled data as well as a manifold structure from both labeled and unlabeled data by modeling the mismatch between h x and f we show that fme relaxes the hard linear constraint f h x in manifold regularization mr making it better cope with the data sampled from a nonlinear manifold in addition we propose a simplified version referred to as fme u for unsupervised dimension reduction we also show that our proposed framework provides a unified view to explain and understand many semi supervised supervised and unsupervised dimension reduction techniques comprehensive experiments on several benchmark databases demonstrate the significant improvement over existing dimension reduction algorithms

a functional analysis approach to subband system approximation and identification the subband system identification method consists of identifying a linear system in the time frequency domain this technique can be also used to approximate a linear system in the same domain in both cases it has the advantage of having a very high numerical efficiency however analyzing such a technique is not trivial and the best setup for subband system approximation and identification is not clear in this paper we propose a functional analysis setting to the analysis of the subband technique which leads us to some results on both subband system approximation and identification concerning system approximation we provide an analytical expression to calculate the optimal subband approximation of a given full band system when the quality of the approximation is measured by the power of the output error signal assuming a white input signal we also provide a tight approximation error bound for a given subband configuration which applies in the case where the unknown system to be approximated is known to be the finite impulse response fir of a given order concerning system identification we provide a novel identification strategy that consists of identifying a low quality subband model and use it to build the required model for either subband or delayless reconstruction this identification strategy reduces the computational complexity of the identification process and yields significantly smaller asymptotic residual errors when compared with the existing methods

image interpolation via graph based bayesian label propagation in this paper we propose a novel image interpolation algorithm via graph based bayesian label propagation the basic idea is to first create a graph with known and unknown pixels as vertices and with edge weights encoding the similarity between vertices then the problem of interpolation converts to how to effectively propagate the label information from known points to unknown ones this process can be posed as a bayesian inference in which we try to combine the principles of local adaptation and global consistency to obtain accurate and robust estimation specially our algorithm first constructs a set of local interpolation models which predict the intensity labels of all image samples and a loss term will be minimized to keep the predicted labels of the available low resolution lr samples sufficiently close to the original ones then all of the losses evaluated in local neighborhoods are accumulated together to measure the global consistency on all samples moreover a graph laplacian based manifold regularization term is incorporated to penalize the global smoothness of intensity labels such smoothing can alleviate the insufficient training of the local models and make them more robust finally we construct a unified objective function to combine together the global loss of the locally linear regression square error of prediction bias on the available lr samples and the manifold regularization term it can be solved with a closed form solution as a convex optimization problem experimental results demonstrate that the proposed method achieves competitive performance with the state of the art image interpolation algorithms

hessian eigenmaps locally linear embedding techniques for high dimensional data we describe a method for recovering the underlying parametrization of scattered data m i lying on a manifold m embedded in high dimensional euclidean space the method hessian based locally linear embedding derives from a conceptual framework of local isometry in which the manifold m viewed as a riemannian submanifold of the ambient euclidean space r n is locally isometric to an open connected subset of euclidean space r d because does not have to be convex this framework is able to handle a significantly wider class of situations than the original isomap algorithm the theoretical framework revolves around a quadratic form h f m h f m f dm defined on functions f m r here hf denotes the hessian of f and h f averages the frobenius norm of the hessian over m to define the hessian we use orthogonal coordinates on the tangent planes of m the key observation is that if m truly is locally isometric to an open connected subset of r d then h f has a d 1 dimensional null space consisting of the constant functions and a d dimensional space of functions spanned by the original isometric coordinates hence the isometric coordinates can be recovered up to a linear isometry our method may be viewed as a modification of locally linear embedding and our theoretical framework as a modification of the laplacian eigenmaps framework where we substitute a quadratic form based on the hessian in place of one based on the laplacian

on data integration problems with manifolds this article focuses on data integration problems where the predictor variables for some response variable partition into known subsets this type of data is often referred to as multi view data a

network unfolding map by vertex edge dynamics modeling the emergence of collective dynamics in neural networks is a mechanism of the animal and human brain for information processing in this paper we develop a computational technique using distributed processing elements in a complex network which are called particles to solve semisupervised learning problems three actions govern the particles dynamics generation walking and absorption labeled vertices generate new particles that compete against rival particles for edge domination active particles randomly walk in the network until they are absorbed by either a rival vertex or an edge currently dominated by rival particles the result from the model evolution consists of sets of edges arranged by the label dominance each set tends to form a connected subnetwork to represent a data class although the intrinsic dynamics of the model is a stochastic one we prove that there exists a deterministic version with largely reduced computational complexity specifically with linear growth furthermore the edge domination process corresponds to an unfolding map in such way that edges stretch and shrink according to the vertex edge dynamics consequently the unfolding effect summarizes the relevant relationships between vertices and the uncovered data classes the proposed model captures important details of connectivity patterns over the vertex edge dynamics evolution in contrast to the previous approaches which focused on only vertex or only edge dynamics computer simulations reveal that the new model can identify nonlinear features in both real and artificial data including boundaries between distinct classes and overlapping structures of data

joint harmonic functions and their supervised connections the cluster assumption had a significant impact on the reasoning behind semi supervised classification methods in graph based learning the literature includes numerous applications where harmonic functions provided estimates that conformed to data satisfying this well known assumption but the relationship between this assumption and harmonic functions is not as well understood theoretically we investigate these matters from the perspective of supervised kernel classification and provide concrete answers to two fundamental questions i under what conditions do semi supervised harmonic approaches satisfy this assumption ii if such an assumption is satisfied then why precisely would an observation sacrifice its own supervised estimate in favor of the cluster first a harmonic function is guaranteed to assign labels to data in harmony with the cluster assumption if a specific condition on the boundary of the harmonic function is satisfied second it is shown that any harmonic function estimate within the interior is a probability weighted average of supervised estimates where the weight is focused on supervised kernel estimates near labeled cases we demonstrate that the uniqueness criterion for harmonic estimators is sensitive when the graph is sparse or the size of the boundary is relatively small this sets the stage for a third contribution a new regularized joint harmonic function for semi supervised learning based on a joint optimization criterion mathematical properties of this estimator such as its uniqueness even when the graph is sparse or the size of the boundary is relatively small are proven a main selling point is its ability to operate in circumstances where the cluster assumption may not be fully satisfied on real data by compromising between the purely harmonic and purely supervised estimators the competitive stature of the new regularized joint harmonic approach is established

new semi supervised classification method based on modified cluster assumption the cluster assumption which assumes that similar instances should share the same label is a basic assumption in semi supervised classification learning and has been found very useful in many successful semi supervised classification methods it is rarely noticed that when the cluster assumption is adopted there is an implicit assumption that every instance should have a crisp class label assignment in real applications however there are cases where it is difficult to tell that an instance definitely belongs to one class and does not belong to other neighboring classes in such cases it is more adequate to assume that similar instances should share similar label memberships rather than sharing a crisp label assignment here label memberships can be represented as a vector where each element corresponds to a class and the value at the element expresses the likelihood of the concerned instance belonging to the class by adopting this modified cluster assumption in this paper we propose a new semi supervised classification method that is semi supervised classification based on class membership ssccm specifically we try to solve the decision function and adequate label memberships for instances simultaneously and constrain that an instance and its local weighted mean lwm share the same label membership vector where the lwm is a robust image of the instance constructed by calculating the weighted mean of its neighboring instances we formulate the problem in a unified objective function for the labeled unlabeled data and their lwms based on the square loss function and take an alternating iterative strategy to solve it in which each step generates a closed form solution and the convergence is guaranteed the solution will provide both the decision function and the label membership function for classification their classification results can verify each other and the reliability of semi supervised classification learning might be enhanced by checking the consistency between those two predictions experiments show that ssccm obtains encouraging results compared to state of the art semi supervised classification methods

a recursive method for the approximation of lti systems using subband processing using the subband technique an lti system can be implemented by the composition of an analysis filterbank followed by a transfer matrix subband model and a synthesis filterbank the advantage of this approach is that it offers a good tradeoff between latency and computational complexity in this paper we propose an optimization method for approximating an lti system using the subband technique the proposed method includes optimal allocation of parameters from different fir entries of the subband model while keeping constant the total number of parameters for a better utilization of the available coefficients the optimization is done in a weighted least squares sense considering either linear or logarithmic amplitude scale simulation results demonstrate the advantages of the proposed method when compared with classical implementation approaches using pole zero transfer functions or segmented fft algorithms

ergodic properties for multirate linear systems stochastic analysis of a multirate linear system typically requires the signals in the system to possess certain ergodic properties among them ergodicity in the mean and ergodicity in the correlation are the most commonly used ones we show that multirate operations and time variant linear filtering can destroy these ergodic properties motivated by this fact we introduce the notions of strong ergodicity in the mean and strong ergodicity in the correlation we show that these properties are preserved under a number of operations namely downsampling upsampling addition and uniformly stable linear time variant filtering we also show that white random processes with uniformly bounded second moments are strongly ergodic in the mean and that mutually independent random processes with uniformly bounded fourth moments are jointly strongly ergodic in the correlation the main implication of these results is that if a multirate linear system is driven by white independent random processes with uniformly bounded second fourth moments then every signal in the system is strongly ergodic in the mean correlation and therefore ergodic in the mean correlation an application of these results is also discussed

asymptotic properties of subband identification the purpose of the paper is to study the asymptotic properties i e strong convergence and asymptotic convergence rate of the subband identification method in every subband and in the overall method the study of strong convergence aims to answer the question whether the best possible model is retrieved on the limit with probability one the study of the asymptotic convergence rate aims to give an expression that quantifies how fast the model approaches the best possible value as the number of samples goes to infinity to do this we need to generalize existing results for fullband identification in the process of doing so we come up with a new notion of ergodicity which we call strong ergodicity strongly ergodic signals not only satisfy the assumptions required for our analysis but also enjoy an interesting property which is that strong ergodicity is invariant under a number of transformations in particular the subband components of a strongly ergodic signal are guaranteed to be strongly ergodic therefore ergodic which is not true for an ergodic signal in general

semi supervised learning with nuclear norm regularization integrating new knowledge sources into various learning tasks to improve their performance has recently become an interesting topic in this paper we propose a novel semi supervised learning ssl approach called semi supervised learning with nuclear norm regularization ssl nnr which can simultaneously handle both sparse labeled data and additional pairwise constraints together with unlabeled data specifically we first construct a unified ssl framework to combine the manifold assumption and the pairwise constraints assumption for classification tasks then we provide a modified fixed point continuous algorithm to learn a low rank kernel matrix that takes advantage of laplacian spectral regularization finally we develop a two stage optimization strategy and present a semi supervised classification algorithm with enhanced spectral kernel esk moreover we also present a theoretical analysis of the proposed esk algorithm and derive an easy approach to extend it to out of sample data experimental results on a variety of synthetic and real world data sets demonstrate the effectiveness of the proposed esk algorithm

data visualization and dimensionality reduction using kernel maps with a reference point in this paper a new kernel based method for data visualization and dimensionality reduction is proposed a reference point is considered corresponding to additional constraints taken in the problem formulation in contrast with the class of kernel eigenmap methods the solution coordinates in the low dimensional space is characterized by a linear system instead of an eigenvalue problem the kernel maps with a reference point are generated from a least squares support vector machine ls svm core part that is extended with an additional regularization term for preserving local mutual distances together with reference point constraints the kernel maps possess primal and dual model representations and provide out of sample extensions e g for validation based tuning the method is illustrated on toy problems and real life data sets

density sensitive semisupervised inference semisupervised methods are techniques for using labeled data x 1 y 1 ldots x n y n together with unlabeled data x n 1 ldots x n to make predictions these methods invoke some assumptions that link the marginal distribution p x of x to the regression function f x for example it is common to assume that f is very smooth over high density regions of p x many of the methods are ad hoc and have been shown to work in specific examples but are lacking a theoretical foundation we provide a minimax framework for analyzing semisupervised methods in particular we study methods based on metrics that are sensitive to the distribution p x our model includes a parameter alpha that controls the strength of the semisupervised assumption we then use the data to adapt to alpha

semi supervised domain adaptation on manifolds in real life problems the following semi supervised domain adaptation scenario is often encountered we have full access to some source data which is usually very large the target data distribution is under certain unknown transformation of the source data distribution meanwhile only a small fraction of the target instances come with labels the goal is to learn a prediction model by incorporating information from the source domain that is able to generalize well on the target test instances we consider an explicit form of transformation functions and especially linear transformations that maps examples from the source to the target domain and we argue that by proper preprocessing of the data from both source and target domains the feasible transformation functions can be characterized by a set of rotation matrices this naturally leads to an optimization formulation under the special orthogonal group constraints we present an iterative coordinate descent solver that is able to jointly learn the transformation as well as the model parameters while the geodesic update ensures the manifold constraints are always satisfied our framework is sufficiently general to work with a variety of loss functions and prediction problems empirical evaluations on synthetic and real world experiments demonstrate the competitive performance of our method with respect to the state of the art

large margin semi supervised learning in classification semi supervised learning occurs when a large amount of unlabeled data is available with only a small number of labeled data in such a situation how to enhance predictability of classification through unlabeled data is the focus in this article we introduce a novel large margin semi supervised learning methodology using grouping information from unlabeled data together with the concept of margins in a form of regularization controlling the interplay between labeled and unlabeled data based on this methodology we develop two specific machines involving support vector machines and learning denoted as ssvm and spsi through difference convex programming in addition we estimate the generalization error using both labeled and unlabeled data for tuning regularizers finally our theoretical and numerical analyses indicate that the proposed methodology achieves the desired objective of delivering high performance in generalization particularly against some strong performers

zeta function determinant of the laplace operator on the d dimensional ball we present a direct approach for the calculation of functional determinants of the laplace operator on balls dirichlet and robin boundary conditions are considered using this approach formulas for any value of the dimension d of the ball can be obtained quite easily explicit results are presented here for dimensionsd 2 3 4 5 and 6

laplacian eigenmaps for dimensionality reduction and data representation one of the central problems in machine learning and pattern recognition is to develop appropriate representations for complex data we consider the problem of constructing a representation for data lying on a low dimensional manifold embedded in a high dimensional space drawing on the correspondence between the graph laplacian the laplace beltrami operator on the manifold and the connections to the heat equation we propose a geometrically motivated algorithm for representing the high dimensional data the algorithm provides a computationally efficient approach to nonlinear dimensionality reduction that has locality preserving properties and a natural connection to clustering some potential applications and illustrative examples are discussed

from local similarities to global coding a framework for coding applications feature coding has received great attention in recent years as a building block of many image processing algorithms in particular the importance of the locality assumption in coding approaches has been studied in many previous works we review this assumption and claim that using the similarity of data points to a more global set of anchor points does not necessarily weaken the coding method as long as the underlying structure of the anchor points is considered we propose to capture the underlying structure by assuming a random walker over the anchor points we also show that our method is a fast approximation to the diffusion map kernel experiments on various data sets show that with a knowledge of the underlying structure of anchor points different state of the art coding algorithms may boost their performance in different learning tasks by utilizing the proposed method

graph based semi supervised learning by mixed label propagation with a soft constraint abstract in recent years various graph based algorithms have been proposed for semi supervised learning where labeled and unlabeled examples are regarded as vertices in a weighted graph and similarity between examples is encoded by the weight of edges however most of these methods cannot be used to deal with dissimilarity or negative similarity in this paper we propose a mixed label propagation model with a single soft constraint which can effectively handle positive similarity and negative similarity simultaneously as well as allow the labeled data to be relabeled specifically the soft mixed label propagation model is a fractional quadratic programming problem with a single quadratic constraint and we apply the global optimal algorithm 1 for solving it yielding an global optimal solution in a computational effort of o n 3 log 1 numerical comparisons with several existing methods for common test datasets and a class of collaborative filtering problems verify the effectiveness of the method

performance analysis for subband identification the so called subband identification method has been introduced recently as an alternative method for identification of finite impulse response systems with large tap sizes it is known that this method can be more numerically efficient than the classical system identification method however no results are available to quantify its advantages this paper offers a rigorous study of the performance of the subband method more precisely we aim to compare the performance of the subband identification method with the classical fullband identification method the comparison is done in terms of the asymptotic residual error asymptotic convergence rate and computational cost when the identification is carried out using the prediction error method and the optimization is done using the least squares method it is shown that by properly choosing the filterbanks the number of parameters in each subband the number of subbands and the downsampling factor the two identification methods can have compatible asymptotic residual errors and convergence rate however for applications where a high order model is required the subband method is more numerically efficient we study two types of subband identification schemes one using critical sampling and another one using oversampling the former is simpler to use and easier to understand whereas the latter involves more design problems but offers further computational savings

support vector machine with manifold regularization and partially labeling privacy protection a novel support vector machine with manifold regularization and partially labeling privacy protection termed as svm mr plpp is proposed for semi supervised learning ssl scenarios where only few labeled data and the class proportion of unlabeled data due to privacy protection concerns are available it integrates manifold regularization and privacy protection regularization into the laplacian support vector machine lapsvm to improve the classification accuracy privacy protection here refers to use only the class proportion of data in order to circumvent the high computational burden of the matrix inversion operation involved in svm mr plpp its scalable version called ssvm mr plpp is further developed by introducing intermediate decision variables into the original regularization framework so that the computational burden of the corresponding transformed kernel in ssvm mr plpp can be greatly reduced making it highly scalable to large datasets the experimental results on numerous datasets show the effectiveness of the proposed classifiers

manifold regularization and semi supervised learning some theoretical analyses manifold regularization belkin et al 2006 is a geometrically motivated framework for machine learning within which several semi supervised algorithms have been constructed here we try to provide some theoretical understanding of this approach our main result is to expose the natural structure of a class of problems on which manifold regularization methods are helpful we show that for such problems no supervised learner can learn effectively on the other hand a manifold based learner that knows the manifold or learns it from unlabeled examples can learn with relatively few labeled examples our analysis follows a minimax style with an emphasis on finite sample results in terms of n the number of labeled examples these results allow us to properly interpret manifold regularization and related spectral and geometric algorithms in terms of their potential use in semi supervised learning

a variance minimization criterion to feature selection using laplacian regularization in many information processing tasks one is often confronted with very high dimensional data feature selection techniques are designed to find the meaningful feature subset of the original features which can facilitate clustering classification and retrieval in this paper we consider the feature selection problem in unsupervised learning scenarios which is particularly difficult due to the absence of class labels that would guide the search for relevant information based on laplacian regularized least squares which finds a smooth function on the data manifold and minimizes the empirical loss we propose two novel feature selection algorithms which aim to minimize the expected prediction error of the regularized regression model specifically we select those features such that the size of the parameter covariance matrix of the regularized regression model is minimized motivated from experimental design we use trace and determinant operators to measure the size of the covariance matrix efficient computational schemes are also introduced to solve the corresponding optimization problems extensive experimental results over various real life data sets have demonstrated the superiority of the proposed algorithms

constrained nonnegative matrix factorization for image representation nonnegative matrix factorization nmf is a popular technique for finding parts based linear representations of nonnegative data it has been successfully applied in a wide range of applications such as pattern recognition information retrieval and computer vision however nmf is essentially an unsupervised method and cannot make use of label information in this paper we propose a novel semi supervised matrix decomposition method called constrained nonnegative matrix factorization cnmf which incorporates the label information as additional constraints specifically we show how explicitly combining label information improves the discriminating power of the resulting matrix decomposition we explore the proposed cnmf method with two cost function formulations and provide the corresponding update solutions for the optimization problems empirical experiments demonstrate the effectiveness of our novel algorithm in comparison to the state of the art approaches through a set of evaluations based on real world applications

multiclass data segmentation using diffuse interface methods on graphs we present two graph based algorithms for multiclass segmentation of high dimensional data on graphs the algorithms use a diffuse interface model based on the ginzburg landau functional related to total variation and graph cuts a multiclass extension is introduced using the gibbs simplex with the functional s double well potential modified to handle the multiclass case the first algorithm minimizes the functional using a convex splitting numerical scheme the second algorithm uses a graph adaptation of the classical numerical merriman bence osher mbo scheme which alternates between diffusion and thresholding we demonstrate the performance of both algorithms experimentally on synthetic data image labeling and several benchmark data sets such as mnist coil and webkb we also make use of fast numerical solvers for finding the eigenvectors and eigenvalues of the graph laplacian and take advantage of the sparsity of the matrix experiments indicate that the results are competitive with or better than the current state of the art in multiclass graph based segmentation algorithms for high dimensional data

asymptotic properties of black box identification of transfer functions the problem of estimating the transfer function of a linear stochastic system is considered the transfer function is parametrized as a black box and no given order is chosen a priori this means that the model orders may increase to infinity when the number of observed data tends to infinity the consistency and convergence properties of the resulting transfer function estimates are investigated asymptotic expressions for the variances and distributions of these estimates are also derived for the case that the model orders increase it is shown that the variance of the transfer function estimate at a certain frequency is asymptotically given by the noise to signal ratio at that frequency mulliplied by the model order to number of data points ratio