nils-herrmann/clean_scirepeval huggingface dataset:
Kernel methods for Multi-labelled classification and Categ orical regression problems

This report presents a SVM like learning system to handle multi-label problems. Such problems arise naturally in bio-informatics. Consider for instance the MIPS Yeast genome database in , it is formed by around 3,300 genes associated to their functional classes. One gene can have many classes, and different genes do not belong to the same number of functional categories. Such a problem can not be solved directly with classical approaches and it is generally decomposed into many two-class problems. The binary decomposition has been done partially by different researchers  on the Yeast dataset but it does not provide a satisfactory answer. We explore in this report a new direct approach. It is based on a large margin ranking system that shares a lot of common properties with Support Vector Machines. We tested it on a toy problem and on real datasets with positive results. We also present a new method to do feature selection with multi-labelled datasets. Our method is based on the multiplicative update rule defined in .
Adaptive Margin Support Vector Machines

In this chapter we present a new learning algorithm, Leave{One{Out (LOO{) SVMs and its generalization Adaptive Margin (AM{) SVMs, inspired by a recent upper bound on the leave{one{out error proved for kernel classiiers by Jaakkola and Haussler. The new approach minimizes the expression given by the bound in an attempt to minimize the leave{one{out error. This gives a convex optimization problem which constructs a sparse linear classiier in feature space using the kernel technique. As such the algorithm possesses many of the same properties as SVMs and Linear Programming (LP{) SVMs. These former techniques are based on the minimization of a regularized margin loss, where the margin is treated equivalently for each training pattern. We propose a minimization problem such that adaptive margins for each training pattern are utilized. Furthermore, we give bounds on the generalization error of the approach which justiies its robustness against outliers. We show experimentally that the generalization error of AM{SVMs is comparable to SVMs and LP{SVMs on benchmark datasets from the UCI repository.
Vicinal Risk Minimization

The Vicinal Risk Minimization principle establishes a bridge between generative models and methods derived from the Structural Risk Minimization Principle such as Support Vector Machines or Statistical Regularization. We explain how VRM provides a framework which integrates a number of existing algorithms, such as Parzen windows, Support Vector Machines, Ridge Regression, Constrained Logistic Classifiers and Tangent-Prop. We then show how the approach implies new algorithms for solving problems usually associated with generative models. New algorithms are described for dealing with pattern recognition problems with very different pattern distributions and dealing with unlabeled data. Preliminary empirical results are presented.
Support vector regression with ANOVA decomposition kernels

Support Vector Machines using ANOVA Decomposition Kernels (SVAD) [Vapng] are a way of imposing a structure on multi-dimensional kernels which are generated as the tensor product of one-dimensional kernels. This gives more accurate control over the capacity of the learning machine (VCdimension). SVAD uses ideas from ANOVA decomposition methods and extends them to generate kernels which directly implement these ideas. SVAD is used with spline kernels and results show that SVAD performs better than the respective non ANOVA decomposition kernel. The Boston housing data set from UCI has been tested on Bagging [Bre94] and Support Vector methods before [DBK97] and these results are compared to the SVAD method.
Support vector machines for multi-class pattern recognition

The solution of binary classi cation problems using support vector machines (SVMs) is well developed, but multi-class problems with more than two classes have typically been solved by combining independently produced binary classi ers. We propose a formulation of the SVM that enables a multi-class pattern recognition problem to be solved in a single optimisation. We also propose a similar generalization of linear programming machines. We report experiments using bench-mark datasets in which these two methods achieve a reduction in the number of support vectors and kernel calculations needed. 1. k-Class Pattern Recognition The k-class pattern recognition problem is to construct a decision function given ` iid (independent and identically distributed) samples (points) of an unknown function, typically with noise: (x1; y1); : : : ; (x`; y`) (1) where xi; i = 1; : : : ; ` is a vector of length d and yi 2 f1; : : : ; kg represents the class of the sample. A natural loss function is the number of mistakes made. 2. Solving k-Class Problems with Binary SVMs For the binary pattern recognition problem (case k = 2), the support vector approach has been well developed [3, 5]. The classical approach to solving k-class pattern recognition problems is to consider the problem as a collection of binary classi cation problems. In the one-versus-rest method one constructs k classi ers, one for each class. The n classi er constructs a hyperplane between class n and the k 1 other classes. A particular point is assigned to the class for which the distance from the margin, in the positive direction (i.e. in the direction in which class \one" lies rather than class \rest"), is maximal. This method has been used widely in ESANN'1999 proceedings European Symposium on Artificial Neural Networks Bruges (Belgium), 21-23 April 1999, D-Facto public., ISBN 2-600049-9-X, pp. 219-224
Extensions to the support vector method

The Support Vector Machine (SVM) is a new technique for solving various function estimation problems. We refer to function estimation as learning, and a technique for estimating the unknown function from data as a learning machine. To construct a learning machine one requires four components: a domain (a learning problem with associated loss function), an induction principle, a set of decision functions, and nally an algorithm that implements all three other components. The present work is devoted to the extension of the Support Vector learning machine. Our objective is to show one can consider using the Support Vector approach in various contexts to solve various problems (in di erent domains, with di erent loss functions, induction principles and sets of decision functions). We argue that the Support Vector approach is a family of related algorithms that share some important common concepts and methodology { the approach is conceptually elegant and well supported by the theory. The material is organized as follows. We start with an introduction to the framework of statistical learning theory (Chapter 1). Chapter 2 describes the Support Vector approach and reviews achievements in the domains of pattern recognition, regression estimation and the solution of linear operator equations. The remainder of the work describes novel extensions to the Support Vector approach. Chapter 3 describes the extension of SVMs to the domain of multiclass pattern recognition, and Chapter 4 the extension of SVMs to the domain of density estimation. Chapter 5 introduces SVMs for pattern recognition which attempt to minimize leave{one{out error. Chapter 6 introduces SVMs with dictionaries of kernels for pattern recognition, regression estimation and density estimation. Chapter 7 introduces a method of transduction for function estimation. Finally, Chapter 8 concludes.
Adaptive margin support vector machines for classification

We propose a learning algorithm for classification learning based on the support vector machine (SVM) approach. Existing approaches for constructing SVMs are based on minimization of a regularized margin loss where the margin is treated equivalently for each training pattern. We propose a reformulation of the minimization problem such that adaptive margins for each training pattern are utilized, which we call the adaptive margin (AM-) SVM. We give bounds on the generalization error of AM-SVMs which justify their robustness against outliers, and show experimentally that the generalization error of AM-SVMs is comparable to classical SVMs on benchmark datasets from the UCI repository.
Leave-One-Out Support Vector Machines

We present a new learning algorithm for pattern recognition inspired by a recent upper bound on leave-one-out error [Jaakkola and Haussler, 1999] proved for Support Vector Machines (SVMs) [Vapnik, 1995; 1998]. The new approach directly minimizes the expression given by the bound in an attempt to minimize leave-one-out error. This gives a convex optimization problem which constructs a sparse linear classifier in feature space using the kernel technique. As such the algorithm possesses many of the same properties as SVMs. The main novelty of the algorithm is that apart from the choice of kernel, it is parameterless - the selection of the number of training errors is inherent in the algorithm and not chosen by an extra free parameter as in SVMs. First experiments using the method on benchmark datasets from the UCI repository show results similar to SVMs which have been tuned to have the best choice of parameter.
Support Vector Machines for Multi-class Pattern Recognition 1. K-class Pattern Recognition 2. Solving K-class Problems with Binary Svms

The solution of binary classiication problems using support vector machines (SVMs) is well developed, but multi-class problems with more than two classes have typically been solved by combining independently produced binary classiiers. We propose a formulation of the SVM that enables a multi-class pattern recognition problem to be solved in a single optimisation. We also propose a similar generalization of linear programming machines. We report experiments using benchmark datasets in which these two methods achieve a reduction in the number of support vectors and kernel calculations needed. The k-class pattern recognition problem is to construct a decision function givenìid (independent and identically distributed) samples (points) of an unknown function, typically with noise: class of the sample. A natural loss function is the number of mistakes made. For the binary pattern recognition problem (case k = 2), the support vector approach has been well developed 3, 5]. The classical approach to solving k-class pattern recognition problems is to consider the problem as a collection of binary classiication problems. In the one-versus-rest method one constructs k classiiers, one for each class. The n th classiier constructs a hyperplane between class n and the k ? 1 other classes. A particular point is assigned to the class for which the distance from the margin, in the positive direction (i.e. in the direction in which class \one" lies rather than class \rest"), is maximal. This method has been used widely in
Transductive Inference for Estimating Values of Functions

We introduce an algorithm for estimating the values of a function at a set of test points xl+1,..., xl+m given a set of training points (x1, Y1),...,(xl, yl) without estimating (as an intermediate step) the regression function. We demonstrate that this direct (transductive) way for estimating values of the regression (or classification in pattern recognition) can be more accurate than the traditional one based on two steps, first estimating the function and then calculating the values of this function at the points of interest.
Support vector density estimation

We describe how the SV technique of solving linear operator equations can be applied to the problem of density estimation and how this method makes use of a special type of problem-speciic regularization. We present a new optimization procedure and set of kernels that guarantee the estimate to be a density (be non-negative everywhere and have an integral of 1). We introduce a dictionary of kernel functions to nd approximations using kernels of diierent widths adaptively. A method of SV regression using square loss is introduced and it is shown how this technique is useful for density estimation. Finally, a way of compressing density estimates from classical kernel based methods is described, and all these algorithms are compared to classical kernel density estimates (Parzen's windows). 1.1 The density estimation problem We wish to approximate the density function p(x) from data where the corresponding distribution function is F(x) = P(X x) = Z x ?1 p(t)dt: (If not speciied otherwise our densities are with respect to the usual Lebesgue measure.) Finding the required density means solving the linear operator equation 1 Z 1 ?1 (x ? t)p(t)dt = F(x); (1.1) 1.
Statistical Convergence of Kernel CCA

While kernel canonical correlation analysis (kernel CCA) has been applied in many problems, the asymptotic convergence of the functions estimated from a finite sample to the true functions has not yet been established. This paper gives a rigorous proof of the statistical convergence of kernel CCA and a related method (NOCCO), which provides a theoretical justification for these methods. The result also gives a sufficient condition on the decay of the regularization coefficient in the methods to ensure convergence.
Multivariate Regression via Stiefel Constraints

We introduce a new framework for regression between multi-dimensional spaces. Standard methods for solving this problem typically reduce the problem to one-dimensional regression by choosing features in the input and/or output spaces. These methods, which include PLS (partial least squares), KDE (kernel dependency estimation), and PCR (principal component regression), select features based on different a-priori judgments as to their relevance. Moreover, loss function and constraints are chosen not primarily on statistical grounds, but to simplify the resulting optimisation. By contrast, in our approach the feature construction and the regression estimation are performed jointly, directly minimizing a loss function that we specify, subject to a rank constraint. A major advantage of this approach is that the loss is no longer chosen according to the algorithmic requirements, but can be tailored to the characteristics of the task at hand; the features will then be optimal with respect to this objective. Our approach also allows for the possibility of using a regularizer in the optimization.
Multivariate Regression with Stiefel Constraints

We introduce a new framework for regression between multi-dimensional spaces. Standard methods for solving this problem typically reduce the problem to one-dimensional regression by choosing features in the input and/or output spaces. These methods, which include PLS (partial least squares), KDE (kernel dependency estimation), and PCR (principal component regression), select features based on different a-priori judgments as to their relevance. Moreover, loss function and constraints are chosen not primarily on statistical grounds, but to simplify the resulting optimisation. By contrast, in our approach the feature construction and the regression estimation are performed jointly, directly minimizing a loss function that we specify, subject to a rank constraint. A major advantage of this approach is that the loss is no longer chosen according to the algorithmic requirements, but can be tailored to the characteristics of the task at hand; the features will then be optimal with respect to this objective. Our approach also allows for the possibility of using a regularizer in the optimization. Finally, by processing the observations sequentially, our algorithm is able to work on large scale problems.
Multivariate Regression via Stiefel Manifold Constraints

We introduce a learning technique for regression between high-dimensional spaces. Standard methods typically reduce this task to many one-dimensional problems, with each output dimension considered independently. By contrast, in our approach the feature construction and the regression estimation are performed jointly, directly minimizing a loss function that we specify, subject to a rank constraint. A major advantage of this approach is that the loss is no longer chosen according to the algorithmic requirements, but can be tailored to the characteristics of the task at hand; the features will then be optimal with respect to this objective, and dependence between the outputs can be exploited.
Behaviour and Convergence of the Constrained Covariance

We discuss reproducing kernel Hilbert space (RKHS)-based measures of statistical dependence, with emphasis on constrained covariance (COCO), a novel criterion to test dependence of random variables. We show that COCO is a test for independence if and only if the associated RKHSs are universal. That said, no independence test exists that can distinguish dependent and independent random variables in all circumstances. Dependent random variables can result in a COCO which is arbitrarily close to zero when the source densities are highly non-smooth, which can make dependence hard to detect empirically. All current kernel-based independence tests share this behaviour. Finally, we demonstrate exponential convergence between the population and empirical COCO, which implies that COCO does not suffer from slow learning rates when used as a dependence test.
On-line one-class support vector machines. An application to signal segmentation

We describe an efficient algorithm to sequentially update a density support estimate obtained using one-class support vector machines. The solution provided is an exact solution, which proves to be far more computationally attractive than a batch approach. This deterministic technique is applied to the problem of audio signal segmentation, with simulations demonstrating the computational performance gain on toy data sets, and the accuracy of the segmentation on audio signals.
Nonstationary signal classification using support vector machines

We demonstrate the use of support vector (SV) techniques for the binary classification of nonstationary sinusoidal signals with quadratic phase. We briefly describe the theory underpinning SV classification, and introduce Cohen's group time-frequency representation, which is used to process the nonstationary signals so as to define the classifier input space. We show that the SV classifier outperforms alternative classification methods on this processed data.
Support vector regression for black-box system identification

We demonstrate the use of support vector regression (SVR) techniques for black-box system identification. These methods derive from statistical learning theory, and are of great theoretical and practical interest. We describe the theory underpinning SVR, and compare support vector methods with other approaches using radial basis networks. Finally, we apply SVR to modeling the behaviour of a hydraulic robot arm, and show that SVR improves on previously published results.
Bound on the Leave-One-Out Error for -SVMs

A bound on the leave-one-out error for -support vector (SV) machine binary classifiers is presented. This bound is based on the ge ometrical concept of thespan, which was introduced in the context of bounding the leave-one-out error for C-SV machine binary classifiers. It is shown that the bound presented herein requires less restrictive a ssumptions than theC-SV bound, and that the prediction of generalisation perfor mance is improved, in particular for noisy data sets.
DESIGN OF CHIMES TO PRODUCE CONSONANT, NON-HARMONIC SCALES

Thin, cylindrical metal chimes can be used to play in a variety of scales requiring non-harmonic spectra The speclrum Il'quired for the chimes to have collSOIllllltintervals in a given non-harmonic ocale can be detertnined acoord ingtoamethoddevelopedby Sethares,based on the theory of dissonance proposed by Plomp andLevelt First order perturbation theory was appHedto eontrol the frequencies of the first six modes ofa thin, cylindrical metal chime by perturbations to the chime radius. The resultant chime profile was then refined using finite eiemellt methods. Twa alwninium chimes manufactured with this optimised profile demonstrated superior musical peIformance compared with unperturbed thin cylinders.
Intrinsic Dimensionality Estimation of Submanifolds in R

We present a new method to estimate the intrinsic dimensionality of a submanifold M in R from random samples. The method is based on the convergence rates of a certain U -statistic on the manifold. We solve at least partially the question of the choice of the scale of the data. Moreover the proposed method is easy to implement, can handle large data sets and performs very well even for small sample sizes. We compare the proposed method to two standard estimators on several artificial as well as real data sets.
Geometrical aspects of statistical learning theory

Geometry plays an important role in modern statistical learning theory, and many different aspects of geometry can be found in this fast developing field. This thesis addresses some of these aspects. A large part of this work will be concerned with so called manifold methods, which have recently attracted a lot of interest. The key point is that for a lot of real-world data sets it is natural to assume that the data lies on a low-dimensional submanifold of a potentially high-dimensional Euclidean space. We develop a rigorous and quite general framework for the estimation and approximation of some geometric structures and other quantities of this submanifold, using certain corresponding structures on neighborhood graphs built from random samples of that submanifold. Another part of this thesis deals with the generalization of the maximal margin principle to arbitrary metric spaces. This generalization follows quite naturally by changing the viewpoint on the well-known support vector machines (SVM). It can be shown that the SVM can be seen as an algorithm which applies the maximum margin principle to a subclass of metric spaces. The motivation to consider the generalization to arbitrary metric spaces arose by the observation that in practice the condition for the applicability of the SVM is rather difficult to check for a given metric. Nevertheless one would like to apply the successful maximum margin principle even in cases where the SVM cannot be applied. The last part deals with the specific construction of so called Hilbertian metrics and positive definite kernels on probability measures. We consider several ways of building such metrics and kernels. The emphasis lies on the incorporation of different desired properties of the metric and kernel. Such metrics and kernels have a wide applicability in so called kernel methods since probability measures occur as inputs in various situations.
Hilbertian Metrics and Positive Definite Kernels on Probability Measures

We investigate the problem of defining Hilbertian metrics resp. positive definite kernels on probability measures, continuing the work in . This type of kernels has shown very good results in text classification and has a wide range of possible applications. In this paper we extend the two-parameter family of Hilbertian metrics of Topsoe such that it now includes all commonly used Hilbertian metrics on probability measures. This allows us to do model selection among these metrics in an elegant and unified way. Second we investigate further our approach to incorporate similarity information of the probability space into the kernel. The analysis provides a better understanding of these kernels and gives in some cases a more efficient way to compute them. Finally we compare all proposed kernels in two text and two image classification problems.