nils-herrmann/clean_scirepeval huggingface dataset:
Duality, Geometry, and Support Vector Regression

We develop an intuitive geometric framework for support vector regression (SVR). By examining when ∊-tubes exist, we show that SVR can be regarded as a classification problem in the dual space. Hard and soft ∊-tubes are constructed by separating the convex or reduced convex hulls respectively of the training data with the response variable shifted up and down by ∊. A novel SVR model is proposed based on choosing the max-margin plane between the two shifted datasets. Maximizing the margin corresponds to shrinking the effective ∊-tube. In the proposed approach the effects of the choices of all parameters become clear geometrically.
Bagging neural network sensitivity analysis for feature reduction for in-silico drug design

This paper illustrates a new approach to sensitivity analysis for feature selection using multiple ensemble neural networks in a bootstrapping mode with bagging. This methodology is applied to in-silico drug design with QSAR (quantitative structural activity relationship), which is notoriously challenging for machine learning because typically there are on the order of 300-1000 dependent features, often for as few as 50-100 data points. For an HIV dataset with 160 wavelets descriptors, the number of relevant features was reduced to 35, and the resulting predictive neural network model gave better results than with the full feature set.
Optimization Approaches to Semi-Supervised Learning

We examine mathematical models for semi-supervised support vector machines (S3VM). Given a training set of labeled data and a working set of unlabeled data, S3VM constructs a support vector machine using both the training and working sets. We use S3VM to solve the transductive inference problem posed by Vapnik. In transduction, the task is to estimate the value of a classification function at the given points in the working set. This contrasts with inductive inference which estimates the classification function at all possible values. We propose a general S3VM model that minimizes both the misclassification error and the function capacity based on all the available data. Depending on how poorly-estimated unlabeled data are penalized, different mathematical models result. We examine several practical algorithms for solving these model. The first approach utilizes the S3VM model for 1-norm linear support vector machines converted to a mixed-integer program (MIP). A global solution of the MIP is found using a commercial integer programming solver. The second approach uses a nonconvex quadratic program. Variations of block-coordinate-descent algorithms are used to find local solutions of this problem. Using this MIP within a local learning algorithm produced the best results. Our experimental study on these statistical learning methods indicates that incorporating working data can improve generalization.
Duality and Geometry in SVM Classifiers

We develop an intuitive geometric interpretation of the standard support vector machine (SVM) for classification of both linearly separable and inseparable data and provide a rigorous derivation of the concepts behind the geometry. For the separable case finding the maximum margin between the two sets is equivalent to finding the closest points in the smallest convex sets that contain each class (the convex hulls). We now extend this argument to the inseparable case by using a reduced convex hull reduced away from outliers. We prove that solving the reduced convex hull formulation is exactly equivalent to solving the standard inseparable SVM for appropriate choices of parameters. Some additional advantages of the new formulation are that the effect of the choice of parameters becomes geometrically clear and that the formulation may be solved by fast nearest point algorithms. By changing norms these arguments hold for both the standard 2-norm and 1-norm SVM.
Support vector machines: hype or hallelujah?

Support Vector Machines (SVMs) and related kernel methods have become increasingly popular tools for data mining tasks such as classification, regression, and novelty detection. The goal of this tutorial is to provide an intuitive explanation of SVMs from a geometric perspective. The classification problem is used to investigate the basic concepts behind SVMs and to examine their strengths and weaknesses from a data mining perspective. While this overview is not comprehensive, it does provide resources for those interested in further exploring SVMs.
Chapter 1 OPTIMIZATION APPROACHES TO SEMI-SUPER- VISED LEARNING

We examine mathematical models for semi-supervised support vector machines (SVM). Given a training set of labeled data and a working set of unlabeled data, SVM constructs a support vector machine using both the training and working sets. We use SVM to solve the transductive inference problem posed by Vapnik. In transduction, the task is to estimate the value of a classification function at the given points in the working set. This contrasts with inductive inference which estimates the classification function at all possible values. We propose a general SVM model that minimizes both the misclassification error and the function capacity based on all the available data. Depending on how poorly-estimated unlabeled data are penalized, different mathematical models result. We examine several practical algorithms for solving these model. The first approach utilizes the SVM model for 1-norm linear support vector machines converted to a mixedinteger program (MIP). A global solution of the MIP is found using a commerical integer programming solver. The second approach uses a noncovex quadratic program. Variations of block-coordinate-descent algorithms are used to find local solutions of this problem. Using this MIP within a local learning algorithm produced the best results. Our experimental study on these statistical learning methods indicates that incorporating working data can improve generalization.
A Linear Programming Approach to Novelty Detection

Novelty detection involves modeling the normal behaviour of a system hence enabling detection of any divergence from normality. It has potential applications in many areas such as detection of machine damage or highlighting abnormal features in medical data. One approach is to build a hypothesis estimating the support of the normal data i.e. constructing a function which is positive in the region where the data is located and negative elsewhere. Recently kernel methods have been proposed for estimating the support of a distribution and they have performed well in practice - training involves solution of a quadratic programming problem. In this paper we propose a simpler kernel method for estimating the support based on linear programming. The method is easy to implement and can learn large datasets rapidly. We demonstrate the method on medical and fault detection datasets.
Constrained K-Means Clustering

We consider practical methods for adding constraints to the K-Means clustering algorithm in order to avoid local solutions with empty clusters or clusters having very few points. We often observe this phenomena when applying K-Means to datasets where the number of dimensions is n 10 and the number of desired clusters is k 20. We propose explicitly adding k constraints to the underlying clustering optimization problem requiring that each cluster have at least a minimum number of points in it. We then investigate the resulting cluster assignment step. Preliminary numerical tests on real datasets indicate the constrained approach is less prone to poor local solutions, producing a better summary of the underlying data. Contrained K-Means Clustering 1
Multicategory Classification by Support Vector Machines

We examine the problem of how to discriminate between objects of three or more classes. Specifically, we investigate how two-class discrimination methods can be extended to the multiclass case. We show how the linear programming (LP) approaches based on the work of Mangasarian and quadratic programming (QP) approaches based on Vapnik's Support Vector Machine (SVM) can be combined to yield two new approaches to the multiclass problem. In LP multiclass discrimination, a single linear program is used to construct a piecewise-linear classification function. In our proposed multiclass SVM method, a single quadratic program is used to construct a piecewise-nonlinear classification function. Each piece of this function can take the form of a polynomial, a radial basis function, or even a neural network. For the k > 2-class problems, the SVM method as originally proposed required the construction of a two-class SVM to separate each class from the remaining classes. Similarily, k two-class linear programs can be used for the multiclass problem. We performed an empirical study of the original LP method, the proposed k LP method, the proposed single QP method and the original k QP methods. We discuss the advantages and disadvantages of each approach.
Density-based indexing for approximate nearest-neighbor queries

We consider the problem of performing nearest-neighbor queries e ciently over large high-dimensional databases. Assuming that a full database scan to determine the nearest neighbor entries is not acceptable, we study the possibility of constructing an index structure over the database. It is well-accepted that traditional database indexing algorithms fail for high-dimensional data (say d > 10 or 20 depending on the scheme). Some arguments have advocated that nearest-neighbor queries do not even make sense for high-dimensional data since the ratio of maximum and minimum distance goes to 1 as dimensionality increases. We show that these arguments are based on over-restrictive assumptions, and that in the general case it is meaningful and possible to perform such queries. We present an approach for deriving a multidimensional index to support approximate nearestneighbor queries over large databases. Our approach, called DBIN, scales to high-dimensional databases by exploiting statistical properties of the data. The approach is based on statistically modeling the density of the content of the data table. DBIN uses the density model to derive a single index over the data table and requires physically re-writing data in a new table sorted by the newly created index (i.e. create what is known as a clustered-index in the database literature). The indexing scheme produces a mapping between a query point (a data record) and an ordering on the clustered index values. Data is then scanned according to the index until the probability that the nearest-neighbor has been found exceeds some threshold. We present theoretical and empirical justi cation for DBIN. The scheme supports a family of distance functions which includes the traditional Euclidean distance measure. Microsoft Research Technical Report MSR-TR-98-58 Revised: February 28, 1999  Usama Fayyad (http://research.microsoft.com/~fayyad) address: Microsoft Research One Microsoft Way Redmond, WA 98008, USA phone: +1-425-703-1528 fax: +1-425-936-7329 e-mail: fayyad@microsoft.com This work was performed while the author was visiting Microsoft Research This work was performed while the author was on sabbatical at Microsoft Research
Semi-Supervised Clustering Using Genetic Algorithms

A semi-supervised clustering algorithm is proposed that combines the benefits of supervised and unsupervised learning methods. The approach allows unlabeled data with no known class to be used to improve classification accuracy. The objective function of an unsupervised technique, e.g. K-means clustering, is modified to minimize both the cluster dispersion of the input attributes and a measure of cluster impurity based on the class labels. Minimizing the cluster dispersion of the examples is a form of capacity control to prevent overfitting. For the the output labels, impurity measures from decision tree algorithms such as the Gini index can be used. A genetic algorithm optimizes the objective function to produce clusters. Experimental results show that using class information improves the generalization ability compared to unsupervised methods based only on the input attributes.
On support vector decision trees for database marketing

We introduce a support vector decision tree method for customer targeting in the framework of large databases (database marketing). The goal is to provide a tool to identify the best customers based on historical data. This tool is then used to forecast the best potential customers among a pool of prospects. We begin by regressively constructing a decision tree. Each decision consists of a linear combination of independent attributes. A linear program motivated by the support vector machine method from Vapnik's statistical learning theory is used to construct each decision. This linear program automatically selects the relevant subset of attributes for each decision. Each customer is scored based on the decision tree. A gain chart table is used to verify the goodness-of-fit of the targeting, to determine the likely prospects and the expected utility or profit. Successful results are given for three industrial problems.
Large margin classification methods for data mining

In this dissertation, we study large margin classification methods for data mining. First, we examine a support vector machine (SVM) approach for classifying spam e-mail. We address issues like preprocessing of messages, data representation, and training and evaluating SVM. Our SVM based e-mail filters are very accurate, effective and robust. Second, a new extension of SVM, the logical support vector machine, is proposed. The logical SVMs construct learning machines in a special logical feature space, and can handle categorical and numerical data uniformally. The logical SVMs are simple to train and may improve the classification accuracy of noisy data. Third, we examine algorithms for large margin perceptron decision trees (PDT) for induction. In particular, we demonstrate how incorporating margin maximization into an PDT algorithm can dramatically improve classification performance. An experimental study confirms the theoretical results that enlarging margins in PDT improves generalization. Next we examine how these inductive PDT algorithms can be extended to transduction. Transduction uses both labeled training data and unlabeled test data for learning, and infers the labels of unlabeled test data directly without inducing a classifier first. Our transduction algorithm is significantly better than its non-margin-based induction counterpart, but only shows slight advantage over its large margin induction counterpart. Finally, we propose the support vector decision tree method (SVDT), a linear SVM-based large margin PDT algorithm. The SVDT approach is then applied to three large database marketing applications. In all three cases, simple decision trees are constructed that use only a very small fraction of the features. An effective method for producing gainscharts based on SVDT is developed as well. The SVDT algorithm is very suitable for solving large data mining applications. Overall our study shows that enlarging margins in classification is a very effective strategy for inductive learning. This result is true across the many types of methodologies investigated. The advantage of transduction is less clear, so further study is required.
A support vector machine approach to decision trees

Key ideas from statistical learning theory and support vector machines are generalized to decision trees. A support vector machine is used for each decision in the tree. The "optimal" decision tree is characterized, and both a primal and dual space formulation for constructing the tree are proposed. The result is a method for generating logically simple decision trees with multivariate linear, nonlinear or linear decisions. By varying the kernel function used, the decisions may consist of linear threshold units, polynomials, sigmoidal neural networks, or radial basis function networks. The preliminary results indicate that the method produces simple trees that generalize well with respect to other decision tree algorithms and single support vector machines.
Feature minimization within decision trees

Decision trees for classification can be constructed using mathematical programming. Within decision tree algorithms, the feature minimization problem is to construct accurate decisions using as few features or attributes within each decision as possible. Feature minimization is an important aspect of data mining since it helps identify what attributes are important and helps produce accurate and interpretable decision trees. In feature minimization with bounded accuracy, we minimize the number of features using a given misclassification error tolerance. This problem can be formulated as a parametric bilinear program and is shown to be NP-complete. A parametric FrankWolfe method is used to solve the bilinear subproblems. The resulting minimization algorithm produces more compact, accurate, and interpretable trees. This procedure can be applied to many different error functions. Formulations and results for two error functions are given. One method, FM RLP-P, dramatically reduced the number of features of one dataset from 147 to 2 while maintaining an 83.6% testing accuracy. Computational results compare favorably with the standard univariate decision tree method, C4.5, as well as with linear programming methods of tree construction.
Semi-Supervised Support Vector Machines

We introduce a semi-supervised support vector machine (S3VM) method. Given a training set of labeled data and a working set of unlabeled data, S3VM constructs a support vector machine using both the training and working sets. We use S3VM to solve the transduction problem using overall risk minimization (ORM) posed by Vapnik. The transduction problem is to estimate the value of a classification function at the given points in the working set. This contrasts with the standard inductive learning problem of estimating the classification function at all possible values and then using the fixed function to deduce the classes of the working set data. We propose a general S3VM model that minimizes both the misclassification error and the function capacity based on all the available data. We show how the S3VM model for 1-norm linear support vector machines can be converted to a mixed-integer program and then solved exactly using integer programming. Results of S3VM and the standard 1-norm support vector machine approach are compared on ten data sets. Our computational results support the statistical learning theory results showing that incorporating working data improves generalization when insufficient training information is available. In every case, S3VM either improved or showed no significant difference in generalization compared to the traditional approach.
A hybrid of tabu search and local descent algorithms with applications in artificial intelligence

We introduce the idea of combining a fast local descent method with with a robust tabu search method. A descent algorithm will stop when it reaches a local minimum. To move beyond this local minimum and search for better solutions, we have developed a hybrid algorithm. Tabu search is used to move away from a local minimum encountered by a descent algorithm and into a new search space. We alternate between the two search methods until some stopping criteria is met. Two different hybrid algorithms are introduced. 
The first hybrid algorithm, HEPTS, is a combination of a Frank-Wolfe type local descent algorithm and a tabu search on the extreme points of a polyhedral region. HEPTS can be applied to any problem with linear constraints for which an extreme point solution is desired. There is no restriction on what type of objective function can be posed; nonlinear, non-differentiable, or even non-continuous. We develop three new error functions for minimizing the error of a decision tree. The decision tree can be formulated as an optimization problem with linear constraints for which an extreme point solution is desired. Our algorithm is then applied to the decision tree problem. 
Our second hybrid algorithm, HMITS, is a combination of a block coordinate type local descent algorithm and a tabu search on binary variables. HMITS can be applied to a large class of mixed integer optimization problems. We propose a new formulation for a decision tree in which each decision is a support vector machine. Three benefits of support vector machines are maximal margins, mapping of data to a higher dimension to construct nonlinear discriminants, and problem size depending only on the number of training data, not the dimension of that data. These last two benefits are gained only by considering the dual problem. We then apply HMITS to three-decision trees of support vector machines.
A Parametric Optimization Method for Machine Learning

The classification problem of constructing a plane to separate the members of two sets can be formulated as a parametric bilinear program. This approach was originally created to minimize the number of points misclassified. However, a novel interpretation of the algorithm is that the subproblems represent alternative error functions of the misclassified points. Each subproblem identifies a specified number of outliers and minimizes the magnitude of the errors on the remaining points. A tuning set is used to select the best result among the subproblems. A parametric Frank-Wolfe method was used to solve the bilinear subproblems. Computational results on a number of datasets indicate that the results compare very favorably with linear programming and heuristic search approaches. The algorithm can be used as part of a decision tree algorithm to create nonlinear classifiers.
The nonhelical tail domain of keratin 14 promotes filament bundling and enhances the mechanical properties of keratin intermediate filaments in vitro

Keratin filaments arise from the copolymerization of type I and II sequences, and form a pancytoplasmic network that provides vital mechanical support to epithelial cells. Keratins 5 and 14 are expressed as a pair in basal cells of stratified epithelia, where they occur as bundled arrays of filaments. In vitro, bundles of K5–K14 filaments can be induced in the absence of cross-linkers, and exhibit enhanced resistance to mechanical strain. This property is not exhibited by copolymers of K5 and tailless K14, in which the nonhelical tail domain has been removed, or copolymers of K5 and K19, a type I keratin featuring a short tail domain. The purified K14 tail domain binds keratin filaments in vitro with specificity (kD ∼2 μM). When transiently expressed in cultured cells, the K14 tail domain associates with endogenous keratin filaments. Utilization of the K14 tail domain as a bait in a yeast two-hybrid screen pulls out type I keratin sequences from a skin cDNA library. These data suggest that the tail domain of K14 contributes to the ability of K5–K14 filaments to self-organize into large bundles showing enhanced mechanical resilience in vitro.