nils-herrmann/clean_scirepeval huggingface dataset:
A support vector method for multivariate performance measures

This paper presents a Support Vector Method for optimizing multivariate nonlinear performance measures like the F1-score. Taking a multivariate prediction approach, we give an algorithm with which such multivariate SVMs can be trained in polynomial time for large classes of potentially non-linear performance measures, in particular ROCArea and all measures that can be computed from the contingency table. The conventional classification SVM arises as a special case of our method.
Error bounds for correlation clustering

This paper presents a learning theoretical analysis of correlation clustering (Bansal et al., 2002). In particular, we give bounds on the error with which correlation clustering recovers the correct partition in a planted partition model (Condon & Karp, 2001; McSherry, 2001). Using these bounds, we analyze how the accuracy of correlation clustering scales with the number of clusters and the sparsity of the graph. We also propose a statistical test that analyzes the significance of the clustering found by correlation clustering.
Query chains: learning to rank from implicit feedback

This paper presents a novel approach for using clickthrough data to learn ranked retrieval functions for web search results. We observe that users searching the web often perform a sequence, or chain, of queries with a similar information need. Using query chains, we generate new types of preference judgments from search engine logs, thus taking advantage of user intelligence in reformulating queries. To validate our method we perform a controlled user study comparing generated preference judgments to explicit relevance judgments. We also implemented a real-world search engine to test our approach, using a modified ranking SVM to learn an improved ranking function from preference data. Our results demonstrate significant improvements in the ranking given by the search engine. The learned rankings outperform both a static ranking function, as well as one trained without considering query chains.
Citation Classification And Its Applications

AbstractCitation analysis has been used to study various aspects of scholarly communication. In general, these studies have not differentiated among the multiple reasons for citations. However, authors cite other works for a number of reasons including demonstrating knowledge of the field, establishing the placement of the citing work in the field, comparing and criticizing other works, and paying homage to seminal work by pioneers in the field. In this paper, we present a number of applications in which distinguishing among authors' motivations for citations might be useful and present a machine learning approach to automatically classifying citations according to these motivations. Our approach to citation classification makes use of the structure and the argumentative nature of the scientific papers. We present the results of experiments we ran on papers in the computer science field. The results are encouraging and give us hope that we can use our citation classifier in analyzing large corpora of scientific papers.
Supervised clustering with support vector machines

Supervised clustering is the problem of training a clustering algorithm to produce desirable clusterings: given sets of items and complete clusterings over these sets, we learn how to cluster future sets of items. Example applications include noun-phrase coreference clustering, and clustering news articles by whether they refer to the same topic. In this paper we present an SVM algorithm that trains a clustering algorithm by adapting the item-pair similarity measure. The algorithm may optimize a variety of different clustering functions to a variety of clustering performance measures. We empirically evaluate the algorithm for noun-phrase and news article clustering.
Error Bounds for Correlation Clustering (DRAFT)

This paper presents a learning theoretical analysis of correlation clustering (Bansal et al., 2002). In particular, we give bounds on the error with which correlation clustering recovers the correct partition in a planted partition model (McSherry, 2001). Using these bounds, we analyze how the accuracy of correlation clustering scales with the number of clusters and the sparsity of the graph. We also propose a statistical test that analyzes the significance of the clustering found by correlation clustering.
Eye-tracking analysis of user behavior in WWW search

We investigate how users interact with the results page of a WWW search engine using eye-tracking. The goal is to gain insight into how users browse the presented abstracts and how they select links for further exploration. Such understanding is valuable for improved interface design, as well as for more accurate interpretations of implicit feedback (e.g. clickthrough) for machine learning. The following presents initial results, focusing on the amount of time spent viewing the presented abstracts, the total number of abstract viewed, as well as measures of how thoroughly searchers evaluate their results set.
Guest Editors' Introduction to the Special Issue on Automated Text Categorization

Information Retrieval (IR), broadly intended as the discipline concerned with computerized access to data with poorly understood semantics, has grown in the last decade from a relatively small academic specialty to a richly articulated field at the forefront of computer science. The most conspicuous trend of the '90s within IR, aside from the emergence of Web search engines, has been the proliferation of a number of subtasks that depart from the mainstream " text search " paradigm, and tackle information access and use from a larger perspective. Tasks such as automated text categorization, information filtering (or routing), information extraction , text mining, question answering, text summarization, and topic (or event) detection and tracking, are no longer the " next frontier " of IR, but have become its pulsating heart. Central to the development of these novel (or relatively novel) IR tasks is the application of techniques from machine learning (ML). The key idea is that there is a lot of knowledge out there that can be brought to bear on information access tasks, but this knowledge hardly ever comes in the form of a systematized knowledge base, since such knowledge bases have to be manually built, and for most applications are thus unavailable or too costly to develop. Rather, knowledge manifests itself in the data, and has to be extracted from it; the shift of focus is thus from an " intensional " to an " extensional " notion of semantics, whereby the semantics of a concept is no more a declarative description of it or a set of rules for recognizing its instances, but is the set of contexts (e.g. documents) in which it is instantiated. In this setting, the goal of ML techniques is to help structure data in various respects. Probably the best known among the disciplines that lie at the crossroads of IR and ML is Automated Text Categorization (ATC), the task of building software tools capable of classifying text documents under one or more of a set of predefined categories or subject codes. ATC dates back to the early '60s, when it was mainly viewed as a means to alleviate the task of indexing scientific literature by controlled vocabulary terms. However, it was only in the early '90s that ATC fully flourished, under the pressure caused by the increased availability of ever larger numbers of text documents in digital form and by the ensuing need …
Transductive Learning via Spectral Graph Partitioning

We present a new method for transductive learning, which can be seen as a transductive version of the k nearest-neighbor classifier. Unlike for many other transductive learning methods, the training problem has a meaningful relaxation that can be solved globally optimally using spectral methods. We propose an algorithm that robustly achieves good generalization performance and that can be trained efficiently. A key advantage of the algorithm is that it does not require additional heuristics to avoid unbalanced splits. Furthermore, we show a connection to transductive Support Vector Machines, and that an effective Co-Training algorithm arises as a special case.
Learning a Distance Metric from Relative Comparisons

This paper presents a method for learning a distance metric from relative comparison such as "A is closer to B than A is to C". Taking a Support Vector Machine (SVM) approach, we develop an algorithm that provides a flexible way of describing qualitative training data as a set of constraints. We show that such constraints lead to a convex quadratic programming problem that can be solved by adapting standard methods for SVM training. We empirically evaluate the performance and the modelling flexibility of the algorithm on a collection of text documents.
SIGIR 2003 workshop report: implicit measures of user interests and preferences

The goal of the workshop on Implicit Measures was to explore how various implicit measures of user interest can be used in information retrieval and filtering applications.  In most information retrieval or filtering applications, it is difficult to get explicit feedback from users about the relevance of the results, the appropriateness of the presentation, and more generally about the quality of their experience. Yet explicit judgments are assumed by researchers for many activities like the tuning and selection of ranking algorithms, information combination, user modeling, information presentation, etc. This workshop explored how implicit measures of user interest (such as dwell time, click through, and user activities like annotation, printing, and purchasing) can be used to develop predictive models for a variety of purposes. In the context of information retrieval such models could be used to: improve ranking and relevance assessment (e.g., the extent to which implicit measures can be used to evaluate the quality of systems, ranking algorithms and recommendations, or as input to relevance feedback algorithms); personalize search, filtering or presentation; personalize considering both individual and aggregate data; adaptive link generation or web site design; etc. An examination of theoretical issues such as modeling approaches (Bayesian techniques and other predictive models), gold standards for user behavior (e.g., relevance judgments, purchases), combining implicit and explicit preferences, and biases introduced by reliance on implicit measures were also encouraged. Since this was the first workshop on the topic at SIGIR, we encouraged participation from people with different backgrounds and perspectives including theoretical modeling, experimental analysis, and applications development. Applications from information retrieval, collaborative filtering, e-commerce, user modeling, and human-computer interaction were encouraged. Participation from both academia and industry was encouraged. As an outcome of the workshop we hoped to identify key theoretical modeling issues, systematize engineering principles and best practices, and spark new research directions. The workshop was held on August 1, 2003 in Toronto, Canada, immediately following the SIGIR 2003 conference. The workshop drew about 35 participants, representing a nice mix of academia and industry, and information retrieval and learning.  To encourage interaction among participants, we limited the number and length of the talks to 15 minutes. This was followed by a 5 minute discussion by one of the organizers and additional open discussion among participants.  This seemed to work well as a way to encourage interaction. We also had poster presentations available during the lunch break, which worked well, although only two posters were presented.
Evaluating Retrieval Performance Using Clickthrough Data

This paper proposes a new method for evaluating the quality of retrieval functions. Unlike traditional methods that require relevance judgments by experts or explicit user feedback, it is based entirely on clickthrough data. This is a key advantage, since clickthrough data can be collected at very low cost and without overhead for the user. Taking an approach from experiment design, the paper proposes an experiment setup that generates unbiased feedback about the relative quality of two search results without explicit user feedback. A theoretical analysis shows that the method gives the same results as evaluation with traditional relevance judgments under mild assumptions. An empirical analysis verifies that the assumptions are indeed justified and that the new method leads to conclusive results in a WWW retrieval study.
Transductive Text Classification

For many practical uses of text classification, it is crucial that the learner be able to generalize well using little training data. A news-filtering service, for example, requiring a hundred days’ worth of training data is unlikely to please even the most patient users. The work presented in the following tackles the problem of learning from small training samples by taking a transductive [Vapnik, 1998], instead of an inductive approach. In the inductive setting the learner tries to induce a decision function which has a low error rate on the whole distribution of examples for the particular learning task. Often, this setting is unnecessarily complex. In many situations we do not care about the particular decision function, but rather that we classify a given set of examples (i.e. a test set) with as few errors as possible. This is the goal of transductive inference.
Learning to classify text using support vector machines - methods, theory and algorithms

Based on ideas from Support Vector Machines (SVMs), Learning To Classify Text Using Support Vector Machines presents a new approach to generating text classifiers from examples. The approach combines high performance and efficiency with theoretical understanding and improved robustness. In particular, it is highly effective without greedy heuristic components. The SVM approach is computationally efficient in training and classification, and it comes with a learning theory that can guide real-world applications.  a complete and detailed description of the SVM approach to learning text classifiers, including training algorithms, transductive text classification, efficient performance estimation, and a statistical learning model of text classification. In addition, it includes an overview of the field of text classification, making it self-contained even for newcomers to the field. This book gives a concise introduction to SVMs for pattern recognition, and it includes a detailed description of how to formulate text-classification tasks for machine learning.
Unbiased Evaluation of Retrieval Quality using Clickthrough Data

This paper proposes a new method for evaluating the quality of retrieval functions. Unlike traditional methods that require relevance judgements by experts or explicit user feedback, it is based entirely on clickthrough data. This is a key advantage, since clickthrough data can be collected at very low cost and without overhead for the user. Taking an approach from experiment design, the paper proposes an experiment setup that generates unbiased feedback about the relative quality of two search results without explicit user feedback. A theoretical analysis shows that the method gives the same results as evaluation with traditional relevance judgements under mild statistical assumptions. An empirical analysis verifies that the assumptions are indeed justified and that the new method leads to conclusive results in a WWW retrieval study.
SVM Light: Support Vector Machine

Overview SVM light is an implementation of Support Vector Machines (SVMs) in C. The main features of the program are the following: fast optimization algorithm working set selection based on steepest feasible descent "shrinking" heuristic caching of kernel evaluations use of folding in the linear case solves classification and regression problems. For multivariate and structured outputs use SVM struct. solves ranking problems (e. g. learning retrieval functions in STRIVER search engine). computes XiAlpha-estimates of the error rate, the precision, and the recall efficiently computes Leave-One-Out estimates of the error rate, the precision, and the recall includes algorithm for approximately training large transductive SVMs (TSVMs) (see also Spectral Graph Transducer) can train SVMs with cost models and example dependent costs allows restarts from specified vector of dual variables handles many thousands of support vectors handles several hundred-thousands of training examples supports standard kernel functions and lets you define your own uses sparse vector representation SVM struct : SVM learning for multivariate and structured outputs like trees, sequences, and sets (available here).
Integrating Kernel Methods into a Knowledge-Based Approach to Evidence-Based Medicine

Operational protocols are a valuable means for quality control. However, developing operational protocols is a highly complex and costly task. We present an integrated approach involving both intelligent data analysis and knowledge acquisition from experts that supports the development and validation of operational protocols. The aim is to lower development cost through the use of machine learning and at the same time ensure high quality standards for the protocol through empirical validation. We demonstrate our approach of integrating expert knowledge with data driven techniques based on our effort to develop an operational protocol for the hemodynamic system.
Comparison between various regression depth methods and the support vector machine to approximate the minimum number of missclassifications

SummaryThe minimum number of misclassifications achievable with affine hyperplanes on a given set of labeled points is a key quantity in both statistics and computational learning theory. However, determining this quantity exactly is NP-hard, c. f. Höffgen, Simon and van Horn (1995). Hence, there is a need to find reasonable approximation procedures. This paper introduces two new approaches to approximating the minimum number of misclassifications achievable with affine hyperplanes. Both approaches are modifications of the regression depth method proposed by Rousseeuw and Hubert (1999) for linear regression models. Our algorithms are compared to the existing regression depth algorithm (cf. Christmann and Rousseeuw, 1999) for various data sets. We also used a support vector machine approach, as proposed by Vapnik (1998), as a reference method.
The enigmatics of affect

Affective computation generally focuses on the informatics of affect: structuring, formalizing, and representing emotion as informational units. We propose instead an enigmatics of affect, a critical technical practice that respects the rich and undefinable complexities of human affective experience. Our interactive installation, the Influencing Machine, allows users to explore a dynamic landscape of emotionally expressive sound and child-like drawings, using a tangible, intuitive input device that supports open-ended engagement. The Influencing Machine bridges the subjective experience of the user and the necessary objective rationality of the underlying code. It functions as a cultural probe, reflecting and challenging users to reflect on the cultural meaning of affective computation.
Classification Based on the Support Vector Machine, Regression Depth, and Discriminant Analysis

The minimum number of misclassifications achievable with affine hyperplanes on a given set of labeled points is a key quantity in both statistics and computational learning theory. We compare the modern approaches the regression depth method and the support vector machine with discrimimant analysis. Summarizing, the regression depth method using currently available algorithms yields often better classifications results for small to moderate data sets, say for sample sizes less than 1000 and dimension up to 10, whereas the support vector machine is often more appropriate for larger or higher dimensional data mining problems.
Support Vector Machines

This chapter gives a short introduction to support vector machines, the basic learning method used, extended, and analyzed for text classification throughout this work. Support vector machines [Cortes and Vapnik, 1995][Vapnik, 1998] were developed by Vapnik et al. based on the Structural Risk Minimization principle [Vapnik, 1982] from statistical learning theory. The idea of structural risk minimization is to find a hypothesis h from a hypothesis space H for which one can guarantee the lowest probability of error Err(h) for a given training sample S 
 
$$({\vec x_1},{y_1}), \ldots ,({\vec x_n},{y_n}) {\vec x_i} \in {\Re ^N},{y_i} \in \{ - 1, + 1\}$$ 
 
(3.1) 
 
of n examples. The following upper bound connects the true error of a hypothesis h with the error Err train (h) of h on the training set and the complexity of h [Vapnik, 1998] (see also Section 1.1).
Efficient Performance Estimators for SVMs

Predicting the generalization performance of a learner is one of the central goals of learning theory. The previous chapter approached this question based on an intensional description of the learning task. However, such a model is necessarily coarse, since it operates on a high level of abstraction. Training data can give more details about a learning task than an intensional model with only a few parameters. This chapter explores the problem of predicting the generalization performance of an SVM after training data becomes available.
Optimizing search engines using clickthrough data

This paper presents an approach to automatically optimizing the retrieval quality of search engines using clickthrough data. Intuitively, a good information retrieval system should present relevant documents high in the ranking, with less relevant documents following below. While previous approaches to learning retrieval functions from examples exist, they typically require training data generated from relevance judgments by experts. This makes them difficult and expensive to apply. The goal of this paper is to develop a method that utilizes clickthrough data for training, namely the query-log of the search engine in connection with the log of links the users clicked on in the presented ranking. Such clickthrough data is available in abundance and can be recorded at very low cost. Taking a Support Vector Machine (SVM) approach, this paper presents a method for learning retrieval functions. From a theoretical perspective, this method is shown to be well-founded in a risk minimization framework. Furthermore, it is shown to be feasible even for large sets of queries and features. The theoretical results are verified in a controlled experiment. It shows that the method can effectively adapt the retrieval function of a meta-search engine to a particular group of users, outperforming Google in terms of retrieval quality after only a couple of hundred training examples.
Training Inductive Support Vector Machines

Training a support vector machine (SVM) leads to a quadratic optimization problem with bound constraints and one linear equality constraint. Despite the fact that this type of problem is well understood in principle, there are many issues to be considered in designing an SVM learner. In particular, for large learning tasks with many training examples, off-the-shelf optimization techniques for general quadratic programs such as Newton, Quasi Newton, etc., quickly become intractable in their memory and time requirements.
A Statistical Learning Model of Text Classification for SVMs

There are at least two ways to motivate why a particular learning method is suitable for a particular learning task. Since ultimately one is interested in the performance of the method, one way is through comparative studies. Chapters 6 and 7 present such studies and show that SVMs deliver state-of-the-art classification performance. However, success on benchmarks is a brittle justification for a learning algorithm and gives only limited insight. Therefore, this dissertation takes a different approach. It introduces support vector machines for learning text classifiers from a theoretical perspective.