the analysis of decomposition methods for support vector machines the support vector machine svm is a promising technique for pattern recognition it requires the solution of a large dense quadratic programming problem traditional optimization methods cannot be directly applied due to memory restrictions up to now very few methods can handle the memory problem and an important one is the decomposition method however there is no convergence proof so far we connect this method to projected gradient methods and provide theoretical proofs for a version of decomposition methods an extension to bound constrained formulation of svm is also provided we then show that this convergence proof is valid for general decomposition methods if their working set selection meets a simple requirement

a fast iterative nearest point algorithm for support vector machine classifier design in this paper we give a new fast iterative algorithm for support vector machine svm classifier design the basic problem treated is one that does not allow classification violations the problem is converted to a problem of computing the nearest point between two convex polytopes the suitability of two classical nearest point algorithms due to gilbert and mitchell et al is studied ideas from both these algorithms are combined and modified to derive our fast algorithm for problems which require classification violations to be allowed the violations are quadratically penalized and an idea due to cortes and vapnik and friess is used to convert it to a problem in which there are no classification violations comparative computational evaluation of our algorithm against powerful svm methods such as platt s sequential minimal optimization shows that our algorithm is very competitive

training ν support vector classifiers theory and algorithms the support vector machine svm for classification proposed by scholkopf smola williamson and bartlett 2000 has the advantage of using a parameter on controlling the number of support vectors in this article we investigate the relation between svm and c svm in detail we show that in general they are two different problems with the same optimal solution set hence we may expect that many numerical aspects of solving them are similar however compared to regular c svm the formulation of svm is more complicated so up to now there have been no effective methods for solving large scale svm we propose a decomposition method for svm that is competitive with existing methods for c svm we also discuss the behavior of svm by some numerical experiments

dc algorithm for extended robust support vector machine nonconvex variants of support vector machines svms have been developed for various purposes for example robust svms attain robustness to outliers by using a nonconvex loss function while extended svm e svm extends the range of the hyperparameter by introducing a nonconvex constraint here we consider an extended robust support vector machine er svm a robust variant of e svm er svm combines two types of nonconvexity from robust svms and e svm because of the two nonconvexities the existing algorithm we proposed needs to be divided into two parts depending on whether the hyperparameter value is in the extended range or not the algorithm also heuristically solves the nonconvex problem in the extended range in this letter we propose a new efficient algorithm for er svm the algorithm deals with two types of nonconvexity while never entailing more computations than either e svm or robust svm and it finds a critical point of er svm furthermore we show that er svm includes the existing robus

multicategory support vector machines two category support vector machines svm have been very popular in the machine learning community for classification problems solving multicategory problems by a series of binary classifiers is quite common in the svm paradigm however this approach may fail under various circumstances we propose the multicategory support vector machine msvm which extends the binary svm to the multicategory case and has good theoretical properties the proposed method provides a unifying framework when there are either equal or unequal misclassification costs as a tuning criterion for the msvm an approximate leave one out cross validation function called generalized approximate cross validation is derived analogous to the binary case the effectiveness of the msvm is demonstrated through the applications to cancer classification using microarray data and cloud classification with satellite radiance profiles

nonparallel hyperplane support vector machine for binary classification problems in this paper we propose a nonparallel hyperplane support vector machine nhsvm for binary classification problems our proposed nhsvm is formulated by clustering the training points according to the similarity between classes it constructs two nonparallel hyperplanes simultaneously by solving a single quadratic programming problem and is consistent between its predicting and training processes an essential difference that distinguishes it from other nonparallel svms this proposed nhsvm has been analyzed theoretically and implemented experimentally the results of experiments conducted using it on both artificial and publicly available benchmark datasets confirm its feasibility and efficacy especially for cross planes datasets and datasets with heteroscedastic noise

rush hour is psapce complete or why you should generously tip parking lot attendants rush hour is a children s game that consists of a grid board several cars that are restricted to move either vertically or horizontally but not both a special target car and a single exit on the perimeter of the grid the goal of the game is to find a sequence of legal moves that allows the target car to exit the grid we consider a slightly generalized version of the game that uses an nn grid and assume that we can place the single exit and target car at any location we choose on initialization of the game in this work we show that deciding if the target car can legally exit the grid is pspace complete our constructive proof uses a lazy form of dual rail reversible logic such that movement of output cars can only occur if logical combinations of input cars can also move emulating this logic only requires three types of devices two switches and one crossover thus our proof technique can be easily generalized to other games and planning problems in which the same three primitive devices can be constructed

twin support vector machine for clustering the twin support vector machine twsvm is one of the powerful classification methods in this brief a twsvm type clustering method called twin support vector clustering twsvc is proposed our twsvc includes both linear and nonlinear versions it determines k cluster center planes by solving a series of quadratic programming problems to make twsvc more efficient and stable an initialization algorithm based on the nearest neighbor graph is also suggested the experimental results on several benchmark data sets have shown a comparable performance of our twsvc

on the optimal parameter choice for spl nu support vector machines we determine the asymptotically optimal choice of the parameter spl nu for classifiers of spl nu support vector machine spl nu svm type which has been introduced by scholkopf et al 2000 it turns out that spl nu should be a close upper estimate of twice the optimal bayes risk provided that the classifier uses a so called universal kernel such as the gaussian rbf kernel moreover several experiments show that this result can be used to implement some modified cross validation procedures which improve standard cross validation for spl nu svms

symmetry reductions and exact solutions of a class of nonlinear heat equations abstract classical and nonclassical symmetries of the nonlinear heat equation ut uxx f u are considered the method of differential grobner bases is used both to find the conditions on f u under which symmetries other than the trivial spatial and temporal translational symmetries exist and to solve the determining equations for the infinitesimals a catalogue of symmetry reductions is given including some new reductions for the linear heat equation and a catalogue of exact solutions of the nonlinear heat equation for cubic f u in terms of the roots of f u 0

improvements to platt s smo algorithm for svm classifier design this article points out an important source of inefficiency in platt s sequential minimal optimization smo algorithm that is caused by the use of a single threshold value using clues from the kkt conditions for the dual problem two threshold parameters are employed to derive modifications of smo these modified algorithms perform significantly faster than the original smo on all benchmark data sets tried

on minimum distribution discrepancy support vector machine for domain adaptation domain adaptation learning dal is a novel and effective technique to address pattern classification problems where the prior information for training is unavailable or insufficient its effectiveness depends on the discrepancy between the two distributions that respectively generate the training data for the source domain and the testing data for the target domain however dal may not work so well when only the distribution mean discrepancy between source and target domains is considered and minimized in this paper we first construct a generalized projected maximum distribution discrepancy gpmdd metric for dal on reproducing kernel hilbert space rkhs based domain distributions by simultaneously considering both the projected maximum distribution mean and the projected maximum distribution scatter discrepancy between the source and the target domain in the sequel based on both the structure risk and the gpmdd minimization principle we propose a novel domain adaptation kernelized support vector machine daksvm with respect to the classical svm and its two extensions called ls daksvm and m daksvm with respect to the least square svm and the v svm respectively moreover our theoretical analysis justified that the proposed gpmdd metric could effectively measure the consistency not only between the rkhs embedding domain distributions but also between the scatter information of source and target domains hence the proposed methods are distinctive in that the more consistency between the scatter information of source and target domains can be achieved by tuning the kernel bandwidth the better the convergence of gpmdd metric minimization is and thus improving the scalability and generalization capability of the proposed methods for dal experimental results on artificial and real world problems indicate that the performance of the proposed methods is superior to or at least comparable with existing benchmarking methods

a study on reduced support vector machines recently the reduced support vector machine rsvm was proposed as an alternate of the standard svm motivated by resolving the difficulty on handling large data sets using svm with nonlinear kernels it preselects a subset of data as support vectors and solves a smaller optimization problem however several issues of its practical use have not been fully discussed yet for example we do not know if it possesses comparable generalization ability as the standard svm in addition we would like to see for how large problems rsvm outperforms svm on training time in this paper we show that the rsvm formulation is already in a form of linear svm and discuss four rsvm implementations experiments indicate that in general the test accuracy of rsvm are a little lower than that of the standard svm in addition for problems with up to tens of thousands of data if the percentage of support vectors is not high existing implementations for svm is quite competitive on the training time thus from this empirical study rsvm will be mainly useful for either larger problems or those with many support vectors experiments in this paper also serve as comparisons of 1 different implementations for linear svm and 2 standard svm using linear and quadratic cost functions

a ga based model selection for smooth twin parametric margin support vector machine the recently proposed twin parametric margin support vector machine denoted by tpmsvm gains good generalization and is suitable for many noise cases however in the tpmsvm it solves two dual quadratic programming problems qpps moreover compared with support vector machine svm tpmsvm has at least four regularization parameters that need regulating which affects its practical applications in this paper we increase the efficiency of tpmsvm from two aspects first by introducing a quadratic function we directly optimize a pair of qpps of tpmsvm in the primal space called stpmsvm for short compared with solving two dual qpps in the tpmsvm stpmsvm can obviously improve the training speed without loss of generalization second a genetic algorithm ga based model selection for stpmsvm in the primal space is suggested the ga based stpmsvm can not only select the parameters efficiently but also provide discriminative feature selection computational results on several synthetic as well as benchmark datasets confirm the great improvements on the training process of our ga based stpmsvm

multi label core vector machine with a zero label abstract multi label core vector machine rank cvm is an efficient and effective algorithm for multi label classification but there still exist two aspects to be improved reducing training and testing computational costs further and detecting relevant labels effectively in this paper we extend rank cvm via adding a zero label to construct its variant with a zero label i e rank cvmz which is formulated as the same quadratic programming form with a unit simplex constraint and non negative ones as rank cvm and then is solved by frank wolfe method efficiently attractively our rank cvmz has fewer variables to be solved than rank cvm which speeds up training procedure dramatically further the relevant labels are effectively detected by the zero label experimental results on 12 benchmark data sets demonstrate that our method achieves a competitive performance compared with six existing multi label algorithms according to six indicative instance based measures moreover on the average our rank cvmz runs 83 times faster and has slightly fewer support vectors than its origin rank cvm

asymptotic behaviors of support vector machines with gaussian kernel support vector machines svms with the gaussian rbf kernel have been popular for practical use model selection in this class of svms involves two hyperparameters the penalty parameter c and the kernel width this letter analyzes the behavior of the svm classifier when these hyperparameters take very small or very large values our results help in understanding the hyperparameter space that leads to an efficient heuristic method of searching for hyperparameter values with small generalization errors the analysis also indicates that if complete model selection using the gaussian kernel has been conducted there is no need to consider linear svm

formulations of support vector machines a note from an optimization point of view in this article we discuss issues about formulations of support vector machines svm from an optimization point of view first svms map training data into a higher maybe infinite dimensional space currently primal and dual formulations of svm are derived in the finite dimensional space and readily extend to the infinite dimensional space we rigorously discuss the primal dual relation in the infinite dimensional spaces second svm formulations contain penalty terms which are different from unconstrained penalty functions in optimization traditionally unconstrained penalty functions approximate a constrained problem as the penalty parameter increases we are interested in similar properties for svm formulations for two of the most popular svm formulations we show that one enjoys properties of exact penalty functions but the other is only like traditional penalty functions which converge when the penalty parameter goes to infinity

remarks on convergence of the matrix splitting algorithm for the symmetric linear complementarity problem in this paper it is shown how to remove a restriction on the perturbation vectors h i imposed by mangasarian and luo and tseng in their respective convergence analyses of the matrix splitting algorithms

successive overrelaxation for support vector machines successive overrelaxation sor for symmetric linear complementarity problems and quadratic programs is used to train a support vector machine svm for discriminating between the elements of two massive datasets each with millions of points because sor handles one point at a time similar to platt s sequential minimal optimization smo algorithm 1999 which handles two constraints at a time and joachims svm sup light 1998 which handles a small number of points at a time sor can process very large datasets that need not reside in memory the algorithm converges linearly to a solution encouraging numerical results are presented on datasets with up to 10 000 000 points such massive discrimination problems cannot be processed by conventional linear or quadratic programming methods and to our knowledge have not been solved by other methods on smaller problems sor was faster than svm sup light and comparable or faster than smo

feasibility and finite convergence analysis for accurate on line nu support vector machine the support vector machine svm for classification has the advantage of using a parameter on controlling the number of support vectors and margin errors recently an interesting accurate on line algorithm accurate on line svm algorithm aonsvm is proposed for training svm aonsvm can be viewed as a special case of parametric quadratic programming techniques it is demonstrated that aonsvm avoids the infeasible updating path as far as possible and successfully converges to the optimal solution based on experimental analysis however because of the differences between aonsvm and classical parametric quadratic programming techniques there is no theoretical justification for these conclusions in this paper we prove the feasibility and finite convergence of aonsvm under two assumptions the main results of feasibility analysis include 1 the inverses of the two key matrices in aonsvm always exist 2 the rules for updating the two key inverse matrices are reliable 3 the variable can control the adjustment of the sum of all the weights efficiently and 4 a sample cannot migrate back and forth in successive adjustment steps among the set of margin support vectors the set of error support vectors and the set of the remaining vectors moreover the analyses of aonsvm also provide the proofs of the feasibility and finite convergence for accurate on line c svm learning directly

improvements on twin support vector machines for classification problems the generalized eigenvalue proximal support vector machine gepsvm and twin support vector machine twsvm are regarded as milestones in the development of the powerful svms as they use the nonparallel hyperplane classifiers in this brief we propose an improved version named twin bounded support vector machines tbsvm based on twsvm the significant advantage of our tbsvm over twsvm is that the structural risk minimization principle is implemented by introducing the regularization term this embodies the marrow of statistical learning theory so this modification can improve the performance of classification in addition the successive overrelaxation technique is used to solve the optimization problems to speed up the training procedure experimental results show the effectiveness of our method in both computation time and classification accuracy and therefore confirm the above conclusion further

a matrix free smoothing algorithm for large scale support vector machines this paper is concerned with the iterative solution of sequences of karush kuhn tucker kkt systems arising from smoothing newton type algorithms applied to support vector machines svms every perturbed newton step is crucial for the efficiency of the overall algorithm however the kkt systems based on smoothing reformulation become increasingly ill conditioned as the smoothing parameter approaches to zero by exploiting preconditioning techniques we propose a matrix free smoothing algorithm for solving large scale support vector machines numerical results and comparisons are given to demonstrate the effectiveness and speed of the algorithm

semi automatic computer aided lesion detection in dental x rays using variational level set a semi automatic lesion detection framework is proposed to detect areas of lesions from periapical dental x rays using level set method in this framework first a new proposed competitive coupled level set method is used to segment the image into three pathologically meaningful regions using two coupled level set functions tailored for the dental clinical setting a two stage clinical segmentation acceleration scheme is used the method uses a trained support vector machine svm classifier to provide an initial contour for two coupled level sets then based on the segmentation results an analysis scheme is applied firstly the scheme builds an uncertainty map from which those areas with radiolucent will be automatically emphasized by a proposed color emphasis scheme those radiolucent in the teeth or jaw usually suggested possible lesions secondly the scheme employs a method based on the average intensity profile to isolate the teeth and locate two types of lesions periapical lesion pl and bifurcation lesion bl experimental results show that our proposed segmentation method is able to segment the image into pathological meaningful regions for further analysis our proposed framework is able to automatically provide direct visual cues for the lesion detection and when given the orientation of the teeth it is able to automatically locate the pl and bl with a seriousness level marked for further dental diagnosis when used in the clinical setting the framework enables dentist to improve interpretation and to focus their attention on critical areas

a general soft method for learning svm classifiers with l 1 norm penalty based on the geometric interpretation of support vector machines svms this paper presents a general technique that allows almost all the existing l 2 norm penalty based geometric algorithms including gilbert s algorithm schlesinger kozinec s sk algorithm and mitchell dem yanov malozemov s mdm algorithm to be softened to achieve the corresponding learning l 1 svm classifiers intrinsically the resulting soft algorithms are to find e optimal nearest points between two soft convex hulls theoretical analysis has indicated that our proposed soft algorithms are essentially generalizations of the corresponding existing hard algorithms and consequently they have the same properties of convergence and almost the identical cost of computation as a specific example the problem of solving n svms by the proposed soft mdm algorithm is investigated and the corresponding solution procedure is specified and analyzed to validate the general soft technique several real classification experiments are conducted with the proposed l 1 norm based mdm algorithms and numerical results have demonstrated that their performance is competitive to that of the corresponding l 2 norm based algorithms such as sk and mdm algorithms

training hard margin support vector machines using greedy stagewise algorithm hard margin support vector machines hm svms suffer from getting overfitting in the presence of noise soft margin svms deal with this problem by introducing a regularization term and obtain a state of the art performance however this disposal leads to a relatively high computational cost in this paper an alternative method greedy stagewise algorithm for svms named gs svms is presented to cope with the overfitting of hm svms without employing the regularization term the most attractive property of gs svms is that its computational complexity in the worst case only scales quadratically with the size of training samples experiments on the large data sets with up to 400 000 training samples demonstrate that gs svms can be faster than libsvm 2 83 without sacrificing the accuracy finally we employ statistical learning theory to analyze the empirical results which shows that the success of gs svms lies in that its early stopping rule can act as an implicit regularization term

interior point methods for massive support vector machines we investigate the use of interior point methods for solving quadratic programming problems with a small number of linear constraints where the quadratic term consists of a low rank update to a positive semidefinite matrix several formulations of the support vector machine fit into this category an interesting feature of these particular problems is the volume of data which can lead to quadratic programs with between 10 and 100 million variables and if written explicitly a dense q matrix our code is based on ooqp an object oriented interior point code with the linear algebra specialized for the support vector machine application for the targeted massive problems all of the data is stored out of core and we overlap computation and input output to reduce overhead results are reported for several linear support vector machine formulations demonstrating that the method is reliable and scalable

progressive iterative approximation for regularized least square bivariate b spline surface fitting abstract recently the use of progressive iterative approximation pia to fit data points has received a deal of attention benefitting from its simplicity flexibility and generality in this paper we present a novel progressive iterative approximation for regularized least square bivariate b spline surface fitting rlspia rlspia extends the pia property of univariate ntp normalized totally positive bases to linear dependent non tensor product bivariate b spline bases which leads to a lower order fitting result than common tensor product b spline surface during each iteration the weights for generating fairing updating surface are obtained by solving an energy minimization problem with box constraints iteratively furthermore an accelerating term is introduced to speed up the convergence rate of rlspia which is comparable favourably with the theoretical optimal one several examples are provided to illustrate the efficiency and effectiveness of the proposed method

lagrangian support vector machines an implicit lagrangian for the dual of a simple reformulation of the standard quadratic program of a linear support vector machine is proposed this leads to the minimization of an unconstrained differentiable convex function in a space of dimensionality equal to the number of classified points this problem is solvable by an extremely simple linearly convergent lagrangian support vector machine lsvm algorithm lsvm requires the inversion at the outset of a single matrix of the order of the much smaller dimensionality of the original input space plus one the full algorithm is given in this paper in 11 lines of matlab code without any special optimization tools such as linear or quadratic programming solvers this lsvm code can be used as is to solve classification problems with millions of points for example 2 million points in 10 dimensional input space were classified by a linear surface in 82 minutes on a pentium iii 500 mhz notebook with 384 megabytes of memory and additional swap space and in 7 minutes on a 250 mhz ultrasparc ii processor with 2 gigabytes of memory other standard classification test problems were also solved nonlinear kernel classification can also be solved by lsvm although it does not scale up to very large problems it can handle any positive semidefinite kernel and is guaranteed to converge a short matlab code is also given for nonlinear kernels and tested on a number of problems

general polynomial time decomposition algorithms we present a general decomposition algorithm that is uniformly applicable to every suitably normalized instance of convex quadratic optimization and efficiently approaches an optimal solution the number of iterations required to be within e of optimality grows linearly with 1 e and quadratically with the number m of variables the working set selection can be performed in polynomial time if we restrict our considerations to instances of convex quadratic optimization with at most k0 equality constraints for some fixed constant k0 plus some so called box constraints conditions that hold for most variants of svm optimization the working set is found in linear time our analysis builds on a generalization of the concept of rate certifying pairs that was introduced by hush and scovel in order to extend their results to arbitrary instances of convex quadratic optimization we introduce the general notion of a rate certifying q set we improve on the results by hush and scovel 2003 in several ways first our result holds for convex quadratic optimization whereas the results by hush and scovel are specialized to svm optimization second we achieve a higher rate of convergence even for the special case of svm optimization despite the generality of our approach third our analysis is technically simpler we prove furthermore that the strategy for working set selection which is based on rate certifying sets coincides with a strategy which is based on a so called sparse witness of sub optimality viewed from this perspective our main result improves on convergence results by list and simon 2004 and simon 2004 by providing convergence rates and by holding under more general conditions

reduced support vector machines a statistical theory in dealing with large data sets the reduced support vector machine rsvm was proposed for the practical objective to overcome some computational difficulties as well as to reduce the model complexity in this paper we study the rsvm from the viewpoint of sampling design its robustness and the spectral analysis of the reduced kernel we consider the nonlinear separating surface as a mixture of kernels instead of a full model the rsvm uses a reduced mixture with kernels sampled from certain candidate set our main results center on two major themes one is the robustness of the random subset mixture model the other is the spectral analysis of the reduced kernel the robustness is judged by a few criteria as follows 1 model variation measure 2 model bias deviation between the reduced model and the full model and 3 test power in distinguishing the reduced model from the full one for the spectral analysis we compare the eigenstructures of the full kernel matrix and the approximation kernel matrix the approximation kernels are generated by uniform random subsets the small discrepancies between them indicate that the approximation kernels can retain most of the relevant information for learning tasks in the full kernel we focus on some statistical theory of the reduced set method mainly in the context of the rsvm the use of a uniform random subset is not limited to the rsvm this approach can act as a supplemental algorithm on top of a basic optimization algorithm wherein the actual optimization takes place on the subset approximated data the statistical properties discussed in this paper are still valid

feature space interpretation of svms with indefinite kernels kernel methods are becoming increasingly popular for various kinds of machine learning tasks the most famous being the support vector machine svm for classification the svm is well understood when using conditionally positive definite cpd kernel functions however in practice non cpd kernels arise and demand application in svm the procedure of plugging these indefinite kernels in svm often yields good empirical classification results however they are hard to interpret due to missing geometrical and theoretical understanding in this paper we provide a step toward the comprehension of svm classifiers in these situations we give a geometric interpretation of svm with indefinite kernel functions we show that such svm are optimal hyperplane classifiers not by margin maximization but by minimization of distances between convex hulls in pseudo euclidean spaces by this we obtain a sound framework and motivation for indefinite svm this interpretation is the basis for further theoretical analysis e g investigating uniqueness and for the derivation of practical guidelines like characterizing the suitability of indefinite svm

convergence of iterates of an inexact matrix splitting algorithm for the symmetric monotone linear complementarity problem convergence of iterates is established for a symmetric regular matrix splitting algorithm for the solution of the symmetric monotone linear complementarity problem where the subproblems are solved inexactly the notable iterate convergence recently established by luo and tseng for exact subproblem solution is extended here to inexact subproblem solution for a symmetric matrix splitting a principal application of the present result is to iterate convergence for the inexact block jacobi method for which pang and yang established convergence of a subsequence of the iterates

new support vector algorithms we propose a new class of support vector algorithms for regression and classification in these algorithms a parameter lets one effectively control the number of support vectors while this can be useful in its own right the parameterization has the additional benefit of enabling us to eliminate one of the other free parameters of the algorithm the accuracy parameter epsilon in the regression case and the regularization constant c in the classification case we describe the algorithms give some theoretical results concerning the meaning and the choice of and report experimental results

a unified classification model based on robust optimization a wide variety of machine learning algorithms such as the support vector machine svm minimax probability machine mpm and fisher discriminant analysis fda exist for binary classification the purpose of this letter is to provide a unified classification model that includes these models through a robust optimization approach this unified model has several benefits one is that the extensions and improvements intended for svms become applicable to mpm and fda and vice versa for example we can obtain nonconvex variants of mpm and fda by mimicking perez cruz weston hermann and scholkopf s 2003 extension from convex i svm to nonconvex ei svm another benefit is to provide theoretical results concerning these learning methods at once by dealing with the unified model we give a statistical interpretation of the unified classification model and prove that the model is a good approximation for the worst case minimization of an expected loss with respect to the uncertain probability distribution we also propose a nonconvex optimization algorithm that can be applied to nonconvex variants of existing learning methods and show promising numerical results

a one layer recurrent neural network with a discontinuous hard limiting activation function for quadratic programming in this paper a one layer recurrent neural network with a discontinuous hard limiting activation function is proposed for quadratic programming this neural network is capable of solving a large class of quadratic programming problems the state variables of the neural network are proven to be globally stable and the output variables are proven to be convergent to optimal solutions as long as the objective function is strictly convex on a set defined by the equality constraints in addition a sequential quadratic programming approach based on the proposed recurrent neural network is developed for general nonlinear programming simulation results on numerical examples and support vector machine svm learning show the effectiveness and performance of the neural network

the immersed interface method for elliptic equations with discontinuous coefficients and singular sources the authors develop finite difference methods for elliptic equations of the form nabla cdot beta x nabla u x kappa x u x f x in a region omega in one or two space dimension

efficient learning of label ranking by soft projections onto polyhedra we discuss the problem of learning to rank labels from a real valued feedback associated with each label we cast the feedback as a preferences graph where the nodes of the graph are the labels and edges express preferences over labels we tackle the learning problem by defining a loss function for comparing a predicted graph with a feedback graph this loss is materialized by decomposing the feedback graph into bipartite sub graphs we then adopt the maximum margin framework which leads to a quadratic optimization problem with linear constraints while the size of the problem grows quadratically with the number of the nodes in the feedback graph we derive a problem of a significantly smaller size and prove that it attains the same minimum we then describe an efficient algorithm called sopopo for solving the reduced problem by employing a soft projection onto the polyhedron defined by a reduced set of constraints we also describe and analyze a wrapper procedure for batch learning when multiple graphs are provided for training we conclude with a set of experiments which show significant improvements in run time over a state of the art interior point algorithm

global convergence of decomposition learning methods for support vector machines decomposition methods are well known techniques for solving quadratic programming qp problems arising in support vector machines svms in each iteration of a decomposition method a small number of variables are selected and a qp problem with only the selected variables is solved since large matrix computations are not required decomposition methods are applicable to large qp problems in this paper we will make a rigorous analysis of the global convergence of general decomposition methods for svms we first introduce a relaxed version of the optimality condition for the qp problems and then prove that a decomposition method reaches a solution satisfying this relaxed optimality condition within a finite number of iterations under a very mild condition on how to select variables

weighted mahalanobis distance kernels for support vector machines the support vector machine svm has been demonstrated to be a very effective classifier in many applications but its performance is still limited as the data distribution information is underutilized in determining the decision hyperplane most of the existing kernels employed in nonlinear svms measure the similarity between a pair of pattern images based on the euclidean inner product or the euclidean distance of corresponding input patterns which ignores data distribution tendency and makes the svm essentially a ldquolocalrdquo classifier in this paper we provide a step toward a paradigm of kernels by incorporating data specific knowledge into existing kernels we first find the data structure for each class adaptively in the input space via agglomerative hierarchical clustering ahc and then construct the weighted mahalanobis distance wmd kernels using the detected data distribution information in wmd kernels the similarity between two pattern images is determined not only by the mahalanobis distance md between their corresponding input patterns but also by the sizes of the clusters they reside in although wmd kernels are not guaranteed to be positive definite pd or conditionally positive definite cpd satisfactory classification results can still be achieved because regularizers in svms with wmd kernels are empirically positive in pseudo euclidean pe spaces experimental results on both synthetic and real world data sets show the effectiveness of ldquopluggingrdquo data structure into existing kernels

symmetries periodic plane waves and blow up of l ω systems abstract nonclassical symmetries of one dimensional reaction diffusion equations of the type have been studied the functional forms of and for which the system admits nonclassical symmetries have been determined and the corresponding reduced systems have been obtained some of these reduced systems admit symmetries which lead to further reductions among the several classes of exact solutions that have been obtained asymptotically periodic plane waves appear as similarity solutions of systems we also have obtained a family of solutions that exhibit a blow up process

the computational complexity of motion planning in this paper we show that a generalization of a popular motion planning puzzle called lunar lockout is computationally intractable in particular we show that the problem is pspace complete we begin with a review of np completeness and polynomial time reductions introduce the class pspace and motivate the significance of pspace complete problems afterwards we prove that determining whether a given instance of a generalized lunar lockout puzzle is solvable is pspace complete

group analysis of a hyperbolic lane emden system in this paper we carry out a complete noether and lie group classification of the radial form of a coupled system of hyperbolic equations from the noether symmetries we establish the corresponding conserved vectors we also determine constraints that the nonlinearities should satisfy in order for the scaling symmetries to be noetherian this led us to a critical hyperbola for the systems under consideration an explicit solution is also obtained for a particular choice of the parameters

leave one out bounds for support vector regression model selection minimizing bounds of leave one out errors is an important and efficient approach for support vector machine svm model selection past research focuses on their use for classification but not regression in this letter we derive various leave one out bounds for support vector regression svr and discuss the difference from those for classification experiments demonstrate that the proposed bounds are competitive with bayesian svr for parameter selection we also discuss the differentiability of leave one out bounds

from additional symmetries to linearization of virasoro symmetries abstract we construct the additional symmetries and derive the adler shiota van moerbeke formula for the two component bkp hierarchy considered as certain reductions of the two component bkp hierarchy the drinfeld sokolov hierarchies of type d are proved to possess symmetries written as the linear action of a series of virasoro operators on the tau function it results in that the drinfeld sokolov hierarchies of type d coincide with dubrovin and zhang s hierarchies associated to the frobenius manifolds for coxeter groups of type d and that every solution of such a hierarchy together with the string equation is annihilated by certain combinations of the virasoro operators and the time derivations of the hierarchy

regularization path for nu support vector classification the v support vector classification v svc proposed by scholkopf has the advantage of using a regularization parameter v for controlling the number of support vectors and margin errors however compared to c svc its formulation is more complicated and to date there are no effective methods for computing its regularization path in this paper we propose a new regularization path algorithm which is designed on the basis of a modified formulation of v svc and traces the solution path with respect to the parameter v through theoretical analysis and confirmatory experiments we show that our algorithm can avoid the infeasible updating path under several assumptions i e assumptions 1 and 2 and fit the entire solution path in a finite number of steps when the regularization path of v svc is available a novel approach proposed by yang and ong can be applied to obtain the global optimal solution of common validation functions for v svc and the computation for the whole process is minimal numerical experiments show that it is more efficient than various kinds of grid search methods for selecting the optimal regularization parameter v

tuning support vector machines for minimax and neyman pearson classification this paper studies the training of support vector machine svm classifiers with respect to the minimax and neyman pearson criteria in principle these criteria can be optimized in a straightforward way using a cost sensitive svm in practice however because these criteria require especially accurate error estimation standard techniques for tuning svm parameters such as cross validation can lead to poor classifier performance to address this issue we first prove that the usual cost sensitive svm here called the 2c svm is equivalent to another formulation called the 2 nu svm we then exploit a characterization of the 2 nu svm parameter space to develop a simple yet powerful approach to error estimation based on smoothing in an extensive experimental study we demonstrate that smoothing significantly improves the accuracy of cross validation error estimates leading to dramatic performance gains furthermore we propose coordinate descent strategies that offer significant gains in computational efficiency with little to no loss in performance

application of lie point symmetries to the resolution of an interface problem in a generalized fisher equation abstract in this work we study a generalized reaction diffusion fisher equation using equivalence transformations and lie symmetries reaction diffusion equations have been widely used for modeling the growth of tumors brain gliomas in particular and for modeling biological invasions more recently these models have been used to depict and explain various nonlinear physical chemical and biological phenomena finding analytical solutions describing the pass through white and gray matter in brain can be useful to describe the dynamics of glioma thus we find analytical solutions for a model of tumor growth at its interface

local minimax learning of functions with best finite sample estimation error bounds applications to ridge and lasso regression boosting tree learning kernel machines and inverse problems optimal local estimation is formulated in the minimax sense for inverse problems and nonlinear regression this theory provides best mean squared finite sample error bounds for some popular statistical learning algorithms and also for several optimal improvements of other existing learning algorithms such as smoothing splines and kernel regularization the bounds and improved algorithms are not based on asymptotics or bayesian assumptions and are truly local for each query not depending on cross validating estimates at other queries to optimize modeling parameters results are given for optimal local learning of approximately linear functions with side information context using real algebraic geometry in particular finite sample error bounds are given for ridge regression and for a local version of lasso regression the new regression methods require only quadratic programming with linear or quadratic inequality constraints for implementation greedy additive expansions are then combined with local minimax learning via a change in metric an optimal strategy is presented for fusing the local minimax estimators of a class of experts providing optimal finite sample prediction error bounds from random forests local minimax learning is extended to kernel machines best local prediction error bounds for finite samples are given for tikhonov regularization the geometry of reproducing kernel hilbert space is used to derive improved estimators with finite sample mean squared error mse bounds for class membership probability in two class pattern classification problems a purely local cross validation free algorithm is proposed which uses fisher information with these bounds to determine best local kernel shape in vector machine learning finally a locally quadratic solution to the finite fourier moments problem is presented after reading the first three sections the reader may proceed directly to any of the subsequent applications sections

training v support vector regression theory and algorithms we discuss the relation between e support vector regression e svr and support vector regression svr in particular we focus on properties that are different from those of c support vector classification c svc and support vector classification svc we then discuss some issues that do not occur in the case of classification the possible range of e and the scaling of target values a practical decomposition method for svr is implemented and computational experiments are conducted we show some interesting numerical observations specific to regression

linear convergence rate for the mdm algorithm for the nearest point problem in this paper we will prove a linear convergence rate for the extension of the mitchell dem yanov and malozemov mdm algorithm for solving the nearest point problem npp while linear convergence proofs for the related but different smo method intended for svm training require that the kernel matrix be positive definite no such assumption is needed in npp for mdm moreover we will also show linear convergence for the sequence of mdm vectors to the unique solution vector w of npp and for a quantity that measures the gap in the karush kuhn tucker conditions furthermore even if there might be several multiplier representations for w we will show that any mdm generated multiplier sequence converges linearly to an optimal multiplier this linear convergence is shown to be optimal and it is also numerically illustrated over six datasets we will follow an approach that relies on a geometric point of view that yields a simple path to the proofs highlightslinear convergence rate for the mdm algorithm is proved no particular assumption is needed for the proof a geometric and simple approach is followed this may pave the way for proving similar results for the smo algorithm used to train svms

a study on smo type decomposition methods for support vector machines decomposition methods are currently one of the major methods for training support vector machines they vary mainly according to different working set selections existing implementations and analysis usually consider some specific selection rules this paper studies sequential minimal optimization type decomposition methods under a general and flexible way of choosing the two element working set the main results include 1 a simple asymptotic convergence proof 2 a general explanation of the shrinking and caching techniques and 3 the linear convergence of the methods extensions to some support vector machine variants are also discussed

support vector machinery for infinite ensemble learning ensemble learning algorithms such as boosting can achieve better performance by averaging over the predictions of some base hypotheses nevertheless most existing algorithms are limited to combining only a finite number of hypotheses and the generated ensemble is usually sparse thus it is not clear whether we should construct an ensemble classifier with a larger or even an infinite number of hypotheses in addition constructing an infinite ensemble itself is a challenging task in this paper we formulate an infinite ensemble learning framework based on the support vector machine svm the framework can output an infinite and nonsparse ensemble through embedding infinitely many hypotheses into an svm kernel we use the framework to derive two novel kernels the stump kernel and the perceptron kernel the stump kernel embodies infinitely many decision stumps and the perceptron kernel embodies infinitely many perceptrons we also show that the laplacian radial basis function kernel embodies infinitely many decision trees and can thus be explained through infinite ensemble learning experimental results show that svm with these kernels is superior to boosting with the same base hypothesis set in addition svm with the stump kernel or the perceptron kernel performs similarly to svm with the gaussian radial basis function kernel but enjoys the benefit of faster parameter selection these properties make the novel kernels favorable choices in practice

generalized pete s pike is pspace complete in this paper we study the computational complexity of a generalized version of the logic puzzle pete s pike by thinkfun 14 it is shown that generalized pete s pike is pspace complete moreover we construct an infinite family of initial configurations with the property that the length of a shortest solution is superpolynomial in the size of the game

qp algorithms with guaranteed accuracy and run time for support vector machines we describe polynomial time algorithms that produce approximate solutions with guaranteed accuracy for a class of qp problems that are used in the design of support vector machine classifiers these algorithms employ a two stage process where the first stage produces an approximate solution to a dual qp problem and the second stage maps this approximate dual solution to an approximate primal solution for the second stage we describe an o n log n algorithm that maps an approximate dual solution with accuracy 2 2km 1 2 8 1 2 2 ep2 to an approximate primal solution with accuracy ep where n is the number of data samples kn is the maximum kernel value over the data and 0 is the svm regularization parameter for the first stage we present new results for decomposition algorithms and describe new decomposition algorithms with guaranteed accuracy and run time in particular for rate certifying decomposition algorithms we establish the optimality of 1 n 1 in addition we extend the recent 1 n 1 algorithm of simon 2004 to form two new composite algorithms that also achieve the 1 n 1 iteration bound of list and simon 2005 but yield faster run times in practice we also exploit the rate certifying property of these algorithms to produce new stopping rules that are computationally efficient and that guarantee a specified accuracy for the approximate dual solution furthermore for the dual qp problem corresponding to the standard classification problem we describe operational conditions for which the simon and composite algorithms possess an upper bound of o n on the number of iterations for this same problem we also describe general conditions for which a matching lower bound exists for any decomposition algorithm that uses working sets of size 2 for the simon and composite algorithms we also establish an o n2 bound on the overall run time for the first stage combining the first and second stages gives an overall run time of o n2 ck 1 where ck is an upper bound on the computation to perform a kernel evaluation pseudocode is presented for a complete algorithm that inputs an accuracy ep and produces an approximate solution that satisfies this accuracy in low order polynomial time experiments are included to illustrate the new stopping rules and to compare the simon and composite decomposition algorithms

snowman is pspace complete abstract sokoban is one of the most studied combinatorial puzzle game in the literature its computational complexity was first shown to be pspace complete in 1997 a new proof of this result was obtained by hearn and demaine 2005 8 by introducing the nondeterministic constraint logic ncl problem since then ncl has been used to prove the pspace completeness of several other puzzles including a few sokoban variants by many authors in this paper we show that snowman a new sokoban like puzzle game released in 2015 is pspace complete by reduction from ncl

successive overrelaxation for laplacian support vector machine semisupervised learning ssl problem which makes use of both a large amount of cheap unlabeled data and a few unlabeled data for training in the last few years has attracted amounts of attention in machine learning and data mining exploiting the manifold regularization mr belkin et al proposed a new semisupervised classification algorithm laplacian support vector machines lapsvms and have shown the state of the art performance in ssl field to further improve the lapsvms we proposed a fast laplacian svm flapsvm solver for classification compared with the standard lapsvm our method has several improved advantages as follows 1 flapsvm does not need to deal with the extra matrix and burden the computations related to the variable switching which make it more suitable for large scale problems 2 flapsvm s dual problem has the same elegant formulation as that of standard svms this means that the kernel trick can be applied directly into the optimization model and 3 flapsvm can be effectively solved by successive overrelaxation technology which converges linearly to a solution and can process very large data sets that need not reside in memory in practice combining the strategies of random scheduling of subproblem and two stopping conditions the computing speed of flapsvm is rigidly quicker to that of lapsvm and it is a valid alternative to plapsvm

a formal analysis of stopping criteria of decomposition methods for support vector machines in a previous paper the author 2001 proved the convergence of a commonly used decomposition method for support vector machines svms however there is no theoretical justification about its stopping criterion which is based on the gap of the violation of the optimality condition it is essential to have the gap asymptotically approach zero so we are sure that existing implementations stop in a finite number of iterations after reaching a specified tolerance here we prove this result and illustrate it by two extensions spl nu svm and a multiclass svm by crammer and singer 2001 a further result shows that in final iterations of the decomposition method only a particular set of variables are still being modified this supports the use of the shrinking and caching techniques in some existing implementations finally we prove the asymptotic convergence of a decomposition method for this multiclass svm discussions on the difference between this convergence proof and the one in another paper by lin are also included

soft margin multiple kernel learning multiple kernel learning mkl has been proposed for kernel methods by learning the optimal kernel from a set of predefined base kernels however the traditional l 1 rm mkl method often achieves worse results than the simplest method using the average of base kernels i e average kernel in some practical applications in order to improve the effectiveness of mkl this paper presents a novel soft margin perspective for mkl specifically we introduce an additional slack variable called kernel slack variable to each quadratic constraint of mkl which corresponds to one support vector machine model using a single base kernel we first show that l 1 rm mkl can be deemed as hard margin mkl and then we propose a novel soft margin framework for mkl three commonly used loss functions including the hinge loss the square hinge loss and the square loss can be readily incorporated into this framework leading to the new soft margin mkl objective functions many existing mkl methods can be shown as special cases under our soft margin framework for example the hinge loss soft margin mkl leads to a new box constraint for kernel combination coefficients using different hyper parameter values for this formulation we can inherently bridge the method using average kernel l 1 rm mkl and the hinge loss soft margin mkl the square hinge loss soft margin mkl unifies the family of elastic net constraint regularizer based approaches and the square loss soft margin mkl incorporates l 2 rm mkl naturally moreover we also develop efficient algorithms for solving both the hinge loss and square hinge loss soft margin mkl comprehensive experimental studies for various mkl algorithms on several benchmark data sets and two real world applications including video action recognition and event recognition demonstrate that our proposed algorithms can efficiently achieve an effective yet sparse solution for mkl

an effective bayesian neural network classifier with a comparison study to support vector machine we propose a new bayesian neural network classifier different from that commonly used in several respects including the likelihood function prior specification and network structure under regularity conditions we show that the decision boundary determined by the new classifier will converge to the true one we also propose a systematic implementation for the new classifier in our implementation the tune of connection weights the selection of hidden units and the selection of input variables are unified by sampling from the joint posterior distribution of the network structure and connection weights the numerical results show that the new classifier consistently outperforms the commonly used bayesian neural network classifier and the support vector machine in terms of generalization performance the reason for the inferiority of the commonly used bayesian neural network classifier and the support vector machine is discussed at length

fuzzy kernel perceptron a new learning method the fuzzy kernel perceptron fkp in which the fuzzy perceptron fp and the mercer kernels are incorporated is proposed in this paper the proposed method first maps the input data into a high dimensional feature space using some implicit mapping functions then the fp is adopted to find a linear separating hyperplane in the high dimensional feature space compared with the fp the fkp is more suitable for solving the linearly nonseparable problems in addition it is also more efficient than the kernel perceptron kp experimental results show that the fkp has better classification performance than fp kp and the support vector machine

sparse lssvm in primal using cholesky factorization for large scale problems for support vector machine svm learning least squares svm lssvm derived by duality lssvm d lssvm is a widely used model because it has an explicit solution one obvious limitation of the model is that the solution lacks sparseness which limits it from training large scale problems efficiently in this paper we derive an equivalent lssvm model in primal space lssvm p lssvm by the representer theorem and prove that p lssvm can be solved exactly at some sparse solutions for problems with low rank kernel matrices two algorithms are proposed for finding the sparse approximate solution of p lssvm by cholesky factorization one is based on the decomposition of the kernel matrix k as p p top with the best low rank matrix p approximately by pivoting cholesky factorization the other is based on solving p lssvm by approximating the cholesky factorization of the hessian matrix with rank one update scheme for linear learning problems theoretical analysis and experimental results support that p lssvm can give the sparsest solutions in all svm learners experimental results on some large scale nonlinear training problems show that our algorithms based on p lssvm can converge to acceptable test accuracies at very sparse solutions with a sparsity level 1 and even as little as 0 01 hence our algorithms are a better choice for large scale training problems