MASSACHUSETTS INSTITUTE OF TECHNOLOGY ARTIFICIAL INTELLIGENCE LABORATORY and CENTER FOR BIOLOGICAL AND COMPUTATIONAL LEARNING DEPARTMENT OF BRAIN AND COGNITIVE SCIENCES

Support Vector Machines SVMs perform pattern recognition between two point classes by nding a decision surface determined by certain points of the training set termed Support Vectors SV This surface which in some feature space of possibly in nite dimension can be regarded as a hyperplane is obtained from the solution of a problem of quadratic programming that depends on a regularization parameter In this paper we study some mathematical properties of support vectors and show that the decision surface can be written as the sum of two orthogonal terms the rst depending only on themargin vectors which are SVs lying on the margin the second proportional to the regularization parameter For almost all values of the parameter this enables us to predict how the decision surface varies for small parameter changes In the special but important case of feature space of nite dimensionm we also show that there are at most m margin vectors and observe that m SVs are usually su cient to fully determine the decision surface For relatively small m this latter result leads to a consistent reduction of the SV number Copyright c Massachusetts Institute of Technology This report describes research done within the Department of Physics University of Genoa and in collaboration with the Center for Biological and Computational Learning at MIT Introduction Support Vector Machines SVMs have been recently introduced as a new technique for solving pattern recognition problems Cortes and Vapnik Blanz et al Scholkopf et al Osuna Freund and Girosi According to the theory od SVMs Vapnik Vapnik while traditional techniques for pattern recognition are based on the minimization of the empirical risk that is on the attempt to optimize the performance on the training set SVMs minimize the structural risk that is the probability of misclassifying yet to be seen patterns for a xed but unknown probability distribution of the data This new induction principle which is equivalent to minimize an upper bound on the generalization error relies on the theory of uniform convergence in probability Vapnik What makes SVMs attractive is a the ability to condense the information contained in the training set and b the use of families of decision surfaces of relatively low VC dimension Vapnik and Chervonenkis In the linear separable case the key idea of a SVM can be explained in plain words Given a training set S which contains points of either of two classes a SVM separates the classes through a hyperplane determined by certain points of S termed support vectors In the separable case this hyperplane maximizes the margin or twice the minimum distance of either class from the hyperplane and all the support vectors lie at the same minimum distance from the hyperplane and are thus termed margin vectors In real cases the two classes may not be separable and both the hyperplane and the support vectors are obtained from the solution of a problem of constrained optimization The solution is a trade o between the largest margin and the lowest number of errors trade o controlled by a regularization parameter The aim of this paper is to gain a better understanding of the nature of support vectors and how the regularization parameter determines the decision surface in both the linear and nonlinear case We thus investigate some mathematical properties of support vectors and characterize the dependence of the decision surface on the changes of the regularization parameter The analysis is rst carried out in the simpler linear case and then extended to include nonlinear decision surfaces The paper is organized as follows We rst review the theory of SVMs in section and then present our analysis in section Finally we summarize the conclusions of our work in section Theoretical overview In this section we recall the basics of the theory of SVM Vapnik Cortes and Vapnik in both the linear and nonlinear case We start with the simple case of linearly separable sets Optimal separating hyperplane In what follows we assume we are given a set S of points xi IR with i N Each point xi belongs to either of two classes and thus is given a label yi f g The goal is to establish the equation of a hyperplane that divides S leaving all the points of the same class on the same side while maximizing the minimum distance between either of the two classes and the hyperplane To this purpose we need some preliminary de nitions De nition The set S is linearly separable if there exist w IR and b IR such that w xi b if yi w xi b if yi In more compact notation the two inequalities can be rewritten yi w xi b for i N The pair w b de nes a hyperplane of equation w x b named separating hyperplane see gure a If we denote with w the norm of w the signed distance di of a point xi from the separating hyperplane w b is given by di w xi b w Combining inequality and equation for all xi S we have yidi w Therefore w is the lower bound on the distance between the points xi and the separating hyperplane w b