Random Projection Features and Generalized Additive Models

We propose to learn generalized additive models for classification which represents the classifier using a su m of piecewise linear functions and show that a recently propose d fast linear SVM training method (Pegasos) can be adapted to train such models with the same convergence rates. To be able to learn functions on combination of dimensions, we explorethe use of random projection features which learns a classifier o n data projected using an arbitrary matrix. In our experiment s we find that : (i) The piecewise linear consistently outperforms linear on various datasets (ii) Random projection features perfor m even better and are close to the best results (using RBF kernels) w hile potentially being much faster to train on large datasets. I. I NTRODUCTION Non parametric methods like kernel support vector machines (SVMs) are attractive because they can approximate any decision boundary arbitrarily well given enough traini ng data. Though non-linear kernel SVMs are more versatile, the y do not scale well with training data and take significantly longer to train on large datasets compared to linear SVMs. In addition it has to remember a potentially significant fracti on of the data (support vectors) to represent the classifier. At classification time the kernel SVMs have to compute the kerne l function between an unseen feature vector and all the suppor t vectors which can be very expensive. This makes linear SVMs very attractive for classification and there has been a recen t surge of efficient for training linear SVMs including [5], [1 0], [3]. One possible way to leverage on these developments is to construct linear embeddings of the data which represent the underlying Reproducing Kernel Hilbert Space (RKHS). This is non trivial as for some kernels like radial basis kernels, the RKHS is infinite dimensional. The work of Rahimi [8] shows that many shift invariant kernels k(x − y), can approximated withinǫ by embedding into a hilbert space with D = O(dǫ−2 log 1 ǫ2 ). For various datasets, they observe that their method outperforms the state-of-the-art kernel meth ods. In our work we explore a different approach based on the generalized additive model (GAM) framework introduced by Hastie [2]. First notice that the linear SVM is an additive model which predicts the output based on: Y = β0 + β1x1 + β2x2 + . + βmxm (1) where, sign(Y) is the class label and xi are the features. In contrast, the GAM replaces the parameter terms βixi with functionsf(xi) Y = β0 + f(x1) + f(x2) + . + f(xm) (2) These functionsfi are arbitrary and often nonparametric, thus providing the potential for better fits to data than othe r methods. Overfitting can be an issue if the class of these func tions is large, without suitable regularization. We model t h se functionsfi using piecewise linear functions of k segments. This particular choice has attractive properties which all ow for efficient discriminative learning algorithms and is str ictly more general than a linear SVM. This classifier is similar to a classifier which maps each dimension into k dimensional space but with significant differences, which corresponds t o a different regularization. We show that a recently propose d stochastic gradient descent (pegasos [10]) can be adapted t o train such models, with similar cost per iteration compared to the training a linear SVM on the input data, and convergence rates. In our experiments on the Ratsch Dataset [9] the piecewise linear classifier consistently outperforms the linear SVM b ut is still not close to the best results using rbf kernel on some datasets. Even though the classifier can learn arbitrar y functions in each dimension it still remains a additive mode l over the initial features and thereby cannot learn function s f dependent features. To this end we train models on random linear projections of the data. Random features have been recently shown to have attractive sparseness properties an d we can leverage on those. Notice that a linear SVM trained on linear random projection essentially has the same represen tation power as the original features as the final classifier is s till linear in the input features. The piecewise linear classifie r on the other hand gains representation power with the addition of projected features. We experiment with two schemes of generating the random matrix to project the data, and show that the piecewise linear functions trained on those perfor m better than one learned on the original features and as well a s the rbf kernel SVMs. A. Linear and Piecewise Linear In this section we introduce our classifier and use it with the L1 loss (hinge loss) similar to the one used with SVMs. We first begin with a review of linear support vector machines for classification. Given labeled training data of the form {(yi,xi)} N i=1, with yi ∈ {−1, +1}, xi ∈ R , the C-SVM formulation [? ] finds a hyperplane which best separates the