a new approach to collaborative filtering operator estimation with spectral regularization we present a general approach for collaborative filtering cf using spectral regularization to learn linear operators mapping a set of users to a set of possibly desired objects in particular several recent low rank type matrix completion methods for cf are shown to be special cases of our proposed framework unlike existing regularization based cf our approach can be used to incorporate additional information such as attributes of the users objects a feature currently lacking in existing regularization based cf approaches using popular and well known kernel methods we provide novel representer theorems that we use to develop new estimation methods we then provide learning algorithms based on low rank decompositions and test them on a standard cf data set the experiments indicate the advantages of generalizing the existing regularization based cf methods to incorporate related information about users and objects finally we show that certain multi task learning methods can be also seen as special cases of our proposed approach

fast svm training using approximate extreme points applications of non linear kernel support vector machines svms to large data sets is seriously hampered by its excessive training time we propose a modification called the approximate extreme points support vector machine aesvm that is aimed at overcoming this burden our approach relies on conducting the svm optimization over a carefully selected subset called the representative set of the training data set we present analytical results that indicate the similarity of aesvm and svm solutions a linear time algorithm based on convex hulls and extreme points is used to compute the representative set in kernel space extensive computational experiments on nine data sets compared aesvm to libsvm chang and lin 2011 cvm tsang et al 2005 bvm tsang et al 2007 lasvm bordes et al 2005 svmperf joachims and yu 2009 and the random features method rahimi and recht 2007 our aesvm implementation was found to train much faster than the other methods while its classification accuracy was similar to that of libsvm in all cases in particular for a seizure detection data set aesvm training was almost 500 times faster than libsvm and lasvm and 20 times faster than cvm and bvm additionally aesvm also gave competitively fast classification times

modified cholesky factorizations in interior point algorithms for linear programming we investigate a modified cholesky algorithm typical of those used in most interior point codes for linear programming cholesky based interior point codes are popular for three reasons their implementation requires only minimal changes to standard sparse cholesky algorithms allowing us to take full advantage of software written by specialists in that area they tend to be more efficient than competing approaches that use alternative factorizations and they perform robustly on most practical problems yielding good interior point steps even when the coefficient matrix of the main linear system to be solved for the step components is ill conditioned we investigate this surprisingly robust performance by using analytical tools from matrix perturbation theory and error analysis illustrating our results with computational experiments finally we point out the potential limitations of this approach

sparse robust estimation and kalman smoothing with nonsmooth log concave densities modeling computation and theory we introduce a new class of quadratic support qs functions many of which already play a crucial role in a variety of applications including machine learning robust statistical inference sparsity promotion and inverse problems such as kalman smoothing well known examples of qs penalties include the l2 huber l1 and vapnik losses we build on a dual representation for qs functions using it to characterize conditions necessary to interpret these functions as negative logs of true probability densities this interpretation establishes the foundation for statistical modeling with both known and new qs loss functions and enables construction of non smooth multivariate distributions with specified means and variances from simple scalar building blocks the main contribution of this paper is a flexible statistical modeling framework for a variety of learning applications together with a toolbox of efficient numerical methods for estimation in particular a broad subclass of qs loss functions known as piecewise linear quadratic plq penalties has a dual representation that can be exploited to design interior point ip methods ip methods solve nonsmooth optimization problems by working directly with smooth systems of equations characterizing their optimality we provide several numerical examples along with a code that can be used to solve general plq problems the efficiency of the ip approach depends on the structure of particular applications we consider the class of dynamic inverse problems using kalman smoothing this class comprises a wide variety of applications where the aim is to reconstruct the state of a dynamical system with known process and measurement models starting from noisy output samples in the classical case gaussian errors are assumed both in the process and measurement models for such problems we show that the extended framework allows arbitrary plq densities to be used and that the proposed ip approach solves the generalized kalman smoothing problem while maintaining the linear complexity in the size of the time series just as in the gaussian case this extends the computational efficiency of the mayne fraser and rauch tung striebel algorithms to a much broader nonsmooth setting and includes many recently proposed robust and sparse smoothers as special cases

training a support vector machine in the primal most literature on support vector machines svms concentrates on the dual optimization problem in this letter we point out that the primal problem can also be solved efficiently for both linear and nonlinear svms and that there is no reason for ignoring this possibility on the contrary from the primal point of view new families of algorithms for large scale svm training can be investigated

assessing granger non causality using nonparametric measure of conditional independence in recent years granger causality has become a popular method in a variety of research areas including engineering neuroscience and economics however despite its simplicity and wide applicability the linear granger causality is an insufficient tool for analyzing exotic stochastic processes such as processes involving non linear dynamics or processes involving causality in higher order statistics in order to analyze such processes more reliably a different approach toward granger causality has become increasingly popular this new approach employs conditional independence as a tool to discover granger non causality without any assumption on the underlying stochastic process this paper discusses the concept of discovering granger non causality using measures of conditional independence and proposes a novel measure of conditional independence in brief the proposed approach estimates the conditional distribution function through a kernel based least square regression approach this paper also explores the strengths and weaknesses of the proposed method compared to other available methods and provides a detailed comparison of these methods using a variety of synthetic data sets

mixing linear svms for nonlinear classification in this paper we address the problem of combining linear support vector machines svms for classification of large scale nonlinear datasets the motivation is to exploit both the efficiency of linear svms lsvms in learning and prediction and the power of nonlinear svms in classification to this end we develop a lsvm mixture model that exploits a divide and conquer strategy by partitioning the feature space into subregions of linearly separable datapoints and learning a lsvm for each of these regions we do this implicitly by deriving a generative model over the joint data and label distributions consequently we can impose priors on the mixing coefficients and do implicit model selection in a top down manner during the parameter estimation process this guarantees the sparsity of the learned model experimental results show that the proposed method can achieve the efficiency of lsvms in the prediction phase while still providing a classification performance comparable to nonlinear svms

hash kernels for structured data we propose hashing to facilitate efficient kernels this generalizes previous work using sampling and we show a principled way to compute the kernel matrix for data streams and sparse feature spaces moreover we give deviation bounds from the exact kernel matrix this has applications to estimation on strings and graphs

generalization performance of regularization networks and support vector machines via entropy numbers of compact operators we derive new bounds for the generalization error of kernel machines such as support vector machines and related regularization networks by obtaining new bounds on their covering numbers the proofs make use of a viewpoint that is apparently novel in the field of statistical learning theory the hypothesis class is described in terms of a linear operator mapping from a possibly infinite dimensional unit ball in feature space into a finite dimensional space the covering numbers of the class are then determined via the entropy numbers of the operator these numbers which characterize the degree of compactness of the operator can be bounded in terms of the eigenvalues of an integral operator induced by the kernel function used by the machine as a consequence we are able to theoretically explain the effect of the choice of kernel function on the generalization performance of support vector machines

kernel based nonlinear blind source separation we propose ktdsep a kernel based algorithm for nonlinear blind source separation bss it combines complementary research fields kernel feature spaces and bss using temporal information this yields an efficient algorithm for nonlinear bss with invertible nonlinearity key assumptions are that the kernel feature space is chosen rich enough to approximate the nonlinearity and that signals of interest contain temporal information both assumptions are fulfilled for a wide set of real world applications the algorithm works as follows first the data are implicitly mapped to a high possibly infinite dimensional kernel feature space in practice however the data form a smaller submanifold in feature space even smaller than the number of training data points a fact that has already been used by for example reduced set techniques for support vector machines we propose to adapt to this effective dimension as a preprocessing step and to construct an orthonormal basis of this submanifold the latter dimension reduction step is essential for making the subsequent application of bss methods computationally and numerically tractable in the reduced space we use a bss algorithm that is based on second order temporal decorrelation finally we propose a selection procedure to obtain the original sources from the extracted nonlinear components automatically experiments demonstrate the excellent performance and efficiency of our ktdsep algorithm for several problems of nonlinear bss and for more than two sources

divide and conquer kernel ridge regression a distributed algorithm with minimax optimal rates we study a decomposition based scalable approach to kernel ridge regression and show that it achieves minimax optimal convergence rates under relatively mild conditions the method is simple to describe it randomly partitions a dataset of size n into m subsets of equal size computes an independent kernel ridge regression estimator for each subset using a careful choice of the regularization parameter then averages the local solutions into a global predictor this partitioning leads to a substantial reduction in computation time versus the standard approach of performing kernel ridge regression on all n samples our two main theorems establish that despite the computational speed up statistical optimality is retained as long as m is not too large the partition based estimator achieves the statistical minimax rate over all estimators using the set of n samples as concrete examples our theory guarantees that the number of subsets m may grow nearly linearly for finite rank or gaussian kernels and polynomially in n for sobolev spaces which in turn allows for substantial reductions in computational cost we conclude with experiments on both simulated data and a music prediction task that complement our theoretical results exhibiting the computational and statistical benefits of our approach

tpmsvm a novel twin parametric margin support vector machine for pattern recognition a novel twin parametric margin support vector machine tpmsvm for classification is proposed in this paper this tpmsvm in the spirit of the twin support vector machine twsvm determines indirectly the separating hyperplane through a pair of nonparallel parametric margin hyperplanes solved by two smaller sized support vector machine svm type problems similar to the parametric margin email protected support vector machine email protected svm this tpmsvm is suitable for many cases especially when the data has heteroscedastic error structure that is the noise strongly depends on the input value but there is an advantage in the learning speed compared with the email protected svm the experimental results on several artificial and benchmark datasets indicate that the tpmsvm not only obtains fast learning speed but also shows good generalization

maximum margin clustering made practical motivated by the success of large margin methods in supervised learning maximum margin clustering mmc is a recent approach that aims at extending large margin methods to unsupervised learning however its optimization problem is nonconvex and existing mmc methods all rely on reformulating and relaxing the nonconvex optimization problem as semidefinite programs sdp though sdp is convex and standard solvers are available they are computationally very expensive and only small data sets can be handled to make mmc more practical we avoid sdp relaxations and propose in this paper an efficient approach that performs alternating optimization directly on the original nonconvex problem a key step to avoid premature convergence in the resultant iterative procedure is to change the loss function from the hinge loss to the laplacian square loss so that overconfident predictions are penalized experiments on a number of synthetic and real world data sets demonstrate that the proposed approach is more accurate much faster hundreds to tens of thousands of times faster and can handle data sets that are hundreds of times larger than the largest data set reported in the mmc literature

on the equivalence of the trefftz method and method of fundamental solutions for laplace and biharmonic equations in this paper it is proved that the two approaches known in the literature as the method of fundamental solutions mfs and the trefftz method are mathematically equivalent in spite of their essentially minor and apparent differences in formulation in deriving the equivalence of the trefftz method and the mfs for the laplace and biharmonic problems it is interesting to find that the complete set in the trefftz method for the laplace and biharmonic problems are embedded in the degenerate kernels of the mfs the degenerate scale appears using the mfs when the geometrical matrix is singular the occurring mechanism of the degenerate scale in the mfs is also studied by using circulant the comparison of accuracy and efficiency of the two methods was addressed

a product form cholesky factorization method for handling dense columns in interior point methods for linear programming cholesky factorization has become the method of choice for solving the symmetric system of linear equations arising in interior point methods ipms for linear programming lp its main advantages being numerical stability and efficiency for sparse systems however in the presence of dense columns there is dramatic loss of efficiency a typical approach to remedy this problem is to apply the sherman morrison woodbury smw update formula to the dense part this approach while being very efficient is not numerically stable here we propose using product form cholesky factorization to handle dense columns the proposed approach is essentially as stable as the original cholesky factorization and nearly as efficient as the smw approach we demonstrate these properties both theoretically and computationally a key part of our theoretical analysis is the proof that the elements of the cholesky factors of the matrices that arise in ipms for lp are uniformly bounded as the duality gap converges to zero

the sum over paths covariance kernel a novel covariance measure between nodes of a directed graph this work introduces a link based covariance measure between the nodes of a weighted directed graph where a cost is associated with each arc to this end a probability distribution on the usually infinite countable set of paths through the graph is defined by minimizing the total expected cost between all pairs of nodes while fixing the total relative entropy spread in the graph this results in a boltzmann distribution on the set of paths such that long high cost paths occur with a low probability while short low cost paths occur with a high probability the sum over paths sop covariance measure between nodes is then defined according to this probability distribution two nodes are considered as highly correlated if they often co occur together on the same preferably short paths the resulting covariance matrix between nodes say n nodes in total is a gram matrix and therefore defines a valid kernel on the graph it is obtained by inverting an n times n matrix depending on the costs assigned to the arcs in the same spirit a betweenness score is also defined measuring the expected number of times a node occurs on a path the proposed measures could be used for various graph mining tasks such as computing betweenness centrality semi supervised classification of nodes visualization etc as shown in section 7

low rank kernel learning with bregman matrix divergences in this paper we study low rank matrix nearness problems with a focus on learning low rank positive semidefinite kernel matrices for machine learning applications we propose efficient algorithms that scale linearly in the number of data points and quadratically in the rank of the input matrix existing algorithms for learning kernel matrices often scale poorly with running times that are cubic in the number of data points we employ bregman matrix divergences as the measures of nearness these divergences are natural for learning low rank kernels since they preserve rank as well as positive semidefiniteness special cases of our framework yield faster algorithms for various existing learning problems and experimental results demonstrate that our algorithms can effectively learn both low rank and full rank kernel matrices

low rank decomposition meets kernel learning a generalized nystrom method abstract low rank matrix decomposition and kernel learning are two useful techniques in building advanced learning systems low rank decomposition can greatly reduce the computational cost of manipulating large kernel matrices however existing approaches are mostly unsupervised and do not incorporate side information such as class labels making the decomposition less effective for a specific learning task on the other hand kernel learning techniques aim at constructing kernel matrices whose structure is well aligned with the learning target which improves the generalization performance of kernel methods however most kernel learning approaches are computationally very expensive to obtain the advantages of both techniques and address their limitations in this paper we propose a novel kernel low rank decomposition formulation called the generalized nystrom method our approach inherits the linear time and space complexity via matrix decomposition while at the same time fully exploits partial label information in computing task dependent decomposition in addition the resultant low rank factors can generalize to arbitrary new samples rendering great flexibility in inductive learning scenarios we further extend the algorithm to a multiple kernel learning setup the experimental results on semi supervised classification demonstrate the usefulness of the proposed method

sparse spectral clustering method based on the incomplete cholesky decomposition a novel sparse spectral clustering method using linear algebra techniques is proposed spectral clustering methods solve an eigenvalue problem containing a graph laplacian the proposed method exploits the structure of the laplacian to construct an approximation not in terms of a low rank approximation but in terms of capturing the structure of the matrix with this approximation the size of the eigenvalue problem can be reduced to obtain the indicator vectors from the eigenvectors the method proposed by zha et al 2002 26 which computes a pivoted lq factorization of the eigenvector matrix is adapted this formulation also gives the possibility to extend the method to out of sample points

using the sherman morrison woodbury inversion formula for a fast solution of tridiagonal block toeplitz systems abstract a fast numerical algorithm for solving systems of linear equations with tridiagonal block toeplitz matrices is presented the algorithm is based on a preliminary factorization of the generating quadratic matrix polynomial associated with the toeplitz matrix followed by the sherman morrison woodbury inversion formula and solution of two bidiagonal and one diagonal block toeplitz systems tight estimates of the condition numbers are provided for the matrix system and the main matrix systems generated during the preliminary factorization the emphasis is put on rigorous stability analysis to rounding errors of the sherman morrison woodbury inversion numerical experiments are provided to illustrate the theory

generalized core vector machines kernel methods such as the support vector machine svm are often formulated as quadratic programming qp problems however given m training patterns a naive implementation of the qp solver takes o m 3 training time and at least o m2 space hence scaling up these qps is a major stumbling block in applying kernel methods on very large data sets and a replacement of the naive method for finding the qp solutions is highly desirable recently by using approximation algorithms for the minimum enclosing ball meb problem we proposed the core vector machine cvm algorithm that is much faster and can handle much larger data sets than existing svm implementations however the cvm can only be used with certain kernel functions and kernel methods for example the very popular support vector regression svr cannot be used with the cvm in this paper we introduce the center constrained meb problem and subsequently extend the cvm algorithm the generalized cvm algorithm can now be used with any linear nonlinear kernel and can also be applied to kernel methods such as svr and the ranking svm moreover like the original cvm its asymptotic time complexity is again linear in m and its space complexity is independent of m experiments show that the generalized cvm has comparable performance with state of the art svm and svr implementations but is faster and produces fewer support vectors on very large data sets

efficient svm training using low rank kernel representations svm training is a convex optimization problem which scales with the training set size rather than the feature space dimension while this is usually considered to be a desired quality in large scale problems it may cause training to be impractical the common techniques to handle this difficulty basically build a solution by solving a sequence of small scale subproblems our current effort is concentrated on the rank of the kernel matrix as a source for further enhancement of the training procedure we first show that for a low rank kernel matrix it is possible to design a better interior point method ipm in terms of storage requirements as well as computational complexity we then suggest an efficient use of a known factorization technique to approximate a given kernel matrix by a low rank matrix which in turn will be used to feed the optimizer finally we derive an upper bound on the change in the objective function value based on the approximation error and the number of active constraints support vectors this bound is general in the sense that it holds regardless of the approximation method

decomposition methods for linear support vector machines in this letter we show that decomposition methods with alpha seeding are extremely useful for solving a sequence of linear support vector machines svms with more data than attributes this strategy is motivated by keerthi and lin 2003 who proved that for an svm with data not linearly separable after c is large enough the dual solutions have the same free and bounded components we explain why a direct use of decomposition methods for linear svms is sometimes very slow and then analyze why alpha seeding is much more effective for linear than nonlinear svms we also conduct comparisons with other methods that are efficient for linear svms and demonstrate the effectiveness of alpha seeding techniques in model selection

a classification of pseudospherical surface equations of type ut uxxx g u ux uxx the equations of type ut uxxx g u ux uxx which describe pseudospherical surfaces are characterized a new class of such equations is obtained this class together with the korteweg de vries kdv equation the modified korteweg de vries mkdv equation and the linear equation completes the classification