nils-herrmann/clean_scirepeval huggingface dataset:
Compression Schemes for VC Classes

A compression scheme of size k for the given class F (of indicators) consists of a compression function and a reconstruction function. For any sample σ labelled by some function f in the class, the compression function produces a subsample τ ⊆ σ of size at most k which then can be mapped by the reconstruction function to a hypothesis (not necessarily in the class) which agrees with the original function f on the original sample σ.
Ju n 20 05 Efficient Multiclass Implementations of L 1-Regularized Maximum Entropy

This paper discusses the application of L1-regularized max i um entropy modeling or SL1-Max[9] to multiclass categorization problems. A new modification t the SL1-Max fast sequential learning algorithm is proposed to handle conditionaldistributions. Furthermore, unlike most previous studies , the present research goes beyond a single type of conditiona l distribution. It describes and compares a variety of modeling assumptions about the class distributi on (independent or exclusive) and various types of joint or conditional distributions. It results in a new me thodology for combining binary regularized classifiers to achieve multiclass categorization. In this c ontext, Maximum Entropy can be considered as a generic and efficient regularized classification tool th at matches or outperforms the state-of-the art represented by AdaBoost and SVMs.
Efficient Multiclass Implementations of L1-Regularized Maximum Entropy

This paper discusses the application of L1-regularized maximum entropy modeling or SL1-Max  to multiclass categorization problems. A new modification to the SL1-Max fast sequential learning algorithm is proposed to handle conditional distributions. Furthermore, unlike most previous studies, the present research goes beyond a single type of conditional distribution. It describes and compares a variety of modeling assumptions about the class distribution (independent or exclusive) and various types of joint or conditional distributions. It results in a new methodology for combining binary regularized classifiers to achieve multiclass categorization. In this context, Maximum Entropy can be considered as a generic and efficient regularized classification tool that matches or outperforms the state-of-the art represented by AdaBoost and SVMs.
Analyzing Margins in Boosting

While the success of boosting or voting methods has been evident from experimental data [11], questions about why boosting does not overfit on training data remain. One idea about the effectiveness of boosting was given by Schapire et al. in which they presented an explanation of why the test error of a boosting classifier does not increase with its size by looking at margins. They showed that boosting increases the margins of training examples effectively and that this increase in margins is related to the auspicious performance of AdaBoost . Breiman invented ArcGv, an algorithm that increases margins even more aggressively than AdaBoost, yet does not perform better. He claimed this contradicts Schapire’s margins explanation, since all else being equal a higher margins distribution did not in this case indicate better results . In this paper, we experimentally examine Breiman’s results by comparing the margins and performance of AdaBoost and ArcGv. We also analyze our approaches and show that our results partially contradict Breiman’s findings. We also explore some other voting algorithms’ performance and present some studies only indirectly related to this problem. We also present possibilities for further research and experimentation in this area, which it is our intention to continue pursuing.
Correcting sample selection bias in maximum entropy density estimation

We study the problem of maximum entropy density estimation in the presence of known sample selection bias. We propose three bias correction approaches. The first one takes advantage of unbiased sufficient statistics which can be obtained from biased samples. The second one estimates the biased distribution and then factors the bias out. The third one approximates the second by only using samples from the sampling distribution. We provide guarantees for the first two approaches and evaluate the performance of all three approaches in synthetic experiments and on real data from species habitat modeling, where maxent has been successfully applied and where sample selection bias is a significant problem.
Adding dense, weighted connections to WordNet

WORDNET, a ubiquitous tool for natural language processing, suffers from sparsity of connections between its component concepts (synsets). Through the use of human annotators, a subset of the connections between 1000 hand-chosen synsets was assigned a value of “evocation” representing how much the first concept brings to mind the second. These data, along with existing similarity measures, constitute the basis of a method for predicting evocation between previously unrated pairs. Long Article Topic Areas: Extending WORDNET Author of Record: Jordan Boyd-Graber, jbg@princeton.edu Under consideration for other conferences (specify)?None Adding Dense, Weighted Connections to WORDNET
Convergence and Consistency of Regularized Boosting Algorithms with Stationary B-Mixing Observations

We study the statistical convergence and consistency of regularized Boosting methods, where the samples are not independent and identically distributed (i.i.d.) but come from empirical processes of stationary β-mixing sequences. Utilizing a technique that constructs a sequence of independent blocks close in distribution to the original samples, we prove the consistency of the composite classifiers resulting from a regularization achieved by restricting the 1-norm of the base classifiers' weights. When compared to the i.i.d. case, the nature of sampling manifests in the consistency result only through generalization of the original condition on the growth of the regularization parameter.
Margin-Based Ranking Meets Boosting in the Middle

We present several results related to ranking. We give a general margin-based bound for ranking based on the L∞ covering number of the hypothesis space. Our bound suggests that algorithms that maximize the ranking margin generalize well. 
 
We then describe a new algorithm, Smooth Margin Ranking, that precisely converges to a maximum ranking-margin solution. The algorithm is a modification of RankBoost, analogous to Approximate Coordinate Ascent Boosting. 
 
We also prove a remarkable property of AdaBoost: under very natural conditions, AdaBoost maximizes the exponentiated loss associated with the AUC and achieves the same AUC as RankBoost. This explains the empirical observations made by Cortes and Mohri, and Caruana and Niculescu-Mizil, about the excellent performance of AdaBoost as a ranking algorithm, as measured by the AUC.
Drifting Games

We introduce and study a general, abstract game played between two players called the shepherd and the adversary. The game is played in a series of rounds using a finite set of “chips” which are moved about in ℝn. On each round, the shepherd assigns a desired direction of movement and an importance weight to each of the chips. The adversary then moves the chips in any way that need only be weakly correlated with the desired directions assigned by the shepherd. The shepherd's goal is to cause the chips to be moved to low-loss positions, where the loss of each chip at its final position is measured by a given loss function.We present a shepherd algorithm for this game and prove an upper bound on its performance. We also prove a lower bound showing that the algorithm is essentially optimal for a large number of chips. We discuss computational methods for efficiently implementing our algorithm.We show that our general drifting-game algorithm subsumes some well studied boosting and on-line learning algorithms whose analyses follow as easy corollaries of our general result.
Boosting Based on a Smooth Margin

We study two boosting algorithms, Coordinate Ascent Boosting and Approximate Coordinate Ascent Boosting, which are explicitly designed to produce maximum margins. To derive these algorithms, we introduce a smooth approximation of the margin that one can maximize in order to produce a maximum margin classifier. Our first algorithm is simply coordinate ascent on this function, involving a line search at each step. We then make a simple approximation of this line search to reveal our second algorithm. These algorithms are proven to asymptotically achieve maximum margins, and we provide two convergence rate calculations. The second calculation yields a faster rate of convergence than the first, although the first gives a more explicit (still fast) rate. These algorithms are very similar to AdaBoost in that they are based on coordinate ascent, easy to implement, and empirically tend to converge faster than other boosting algorithms. Finally, we attempt to understand AdaBoost in terms of our smooth margin, focusing on cases where AdaBoost exhibits cyclic behavior.
Toward efficient agnostic learning

In this paper we initiate an investigation of generalizations of the Probably Approximately Correct (PAC) learning model that attempt to significantly weaken the target function assumptions. The ultimate goal in this direction is informally termedagnostic learning, in which we make virtually no assumptions on the target function. The name derives from the fact that as designers of learning algorithms, we give up the belief that Nature (as represented by the target function) has a simple or succinct explanation. We give a number of positive and negative results that provide an initial outline of the possibilities for agnostic learning. Our results include hardness results for the most obvious generalization of the PAC model to an agnostic setting, an efficient and general agnostic learning method based on dynamic programming, relationships between loss functions for agnostic learning, and an algorithm for a learning problem that involves hidden variables.
Generalization bounds for averaged classifiers

We study a simple learning algorithm for binary classification. Instead of predicting with the best hypothesis in the hypothesis class, that is, the hypothesis that minimizes the training error, our algorithm predicts with a weighted average of all hypotheses, weighted exponentially with respect to their training error. We show that the prediction of this algorithm is much more stable than the prediction of an algorithm that predicts with the best hypothesis. By allowing the algorithm to abstain from predicting on some examples, we show that the predictions it makes when it does not abstain are very reliable. Finally, we show that the probability that the algorithm abstains is comparable to the generalization error of the best hypothesis in the class.
A Discussion of: "Process Consistency for AdaBoost" by Wenxin Jiang "On the Bayes-risk consistency of regularized boosting methods" by G´ abor Lugosi and Nicolas Vayatis "Statistical Behavior and Consistency of Classification Methods based on Convex Risk Minimization" by Tong Zhang

The notion of a boosting algorithm was originally introduced by Valiant in the context of the “probably approximately correct” (PAC) model of learnability [19]. In this context boosting is a method for provably improving the accuracy of any “weak” classification learning algorithm. The first boosting algorithm was invented by Schapire [16] and the second one by Freund . These two algorithms were introduced for a specific theoretical purpose. However, since the introduction of AdaBoost , quite a number of perspectives on boosting have emerged. For instance, AdaBoost can be understood as a method for maximizing the “margins” or “confidences” of the training examples [17]; as a technique for playing repeated matrix games [4, 6]; as a linear or convex programming method ; as a functional gradient-descent technique [8, 13, 14, 3]; as a technique for Bregman-distance optimization in a broader framework that includes logistic regression [1, 10, 12]; and finally as a stepwise model-fitting method for minimization of the exponential loss function, an approximation of the negative log binomial likelihood [7]. The current papers add to this list of perspectives, giving a view of boosting that is very different from its original interpretation and analysis as an algorithm for improving the accuracy of a weak learner. These many different points of view add to the richness of the theory of boosting, and are enormously helpful in the practical design of new or better algorithms for machine learning and statistical inference. Originally, boosting algorithms were designed expressly for classification. The goal in this setting is to accurately predict the classification of a new example. Either the prediction is correct, or it is not. There is no attempt made to estimate the conditional probability of each class. In practice, this sometimes is not enough since we may want to have some sense of how likely our prediction is to be correct, or we may want to incorporate numbers that look like probabilities into a larger system. Later, Friedman, Hastie and Tibshirani [7] showed that AdaBoost can in fact be used to estimate such probabilities, arguing that AdaBoost approximates a form of logistic regression. They and others  subsequently modified AdaBoost to explicitly minimize the loss function associated with logistic regression, with the intention of computing such estimated probabilities. In one of the current papers, Zhang vastly generalizes this approach showing that conditional probability estimates P{y|x} can be obtained when minimizing any smooth convex loss function,
The Dynamics of AdaBoost: Cyclic Behavior and Convergence of Margins

In order to study the convergence properties of the AdaBoost algorithm, we reduce AdaBoost to a nonlinear iterated map and study the evolution of its weight vectors. This dynamical systems approach allows us to understand AdaBoost's convergence properties completely in certain cases; for these cases we find stable cycles, allowing us to explicitly solve for AdaBoost's output.Using this unusual technique, we are able to show that AdaBoost does not always converge to a maximum margin combined classifier, answering an open question. In addition, we show that "non-optimal" AdaBoost (where the weak learning algorithm does not necessarily choose the best weak classifier at each iteration) may fail to converge to a maximum margin classifier, even if "optimal" AdaBoost produces a maximum margin. Also, we show that if AdaBoost cycles, it cycles among "support vectors", i.e., examples that achieve the same smallest margin.
On the Worst-Case Analysis of Temporal-Difference Learning Algorithms

We study the behavior of a family of learning algorithms based on Sutton‘s method of temporal differences. In our on-line learning framework, learning takes place in a sequence of trials, and the goal of the learning algorithm is to estimate a discounted sum of all the reinforcements that will be received in the future. In this setting, we are able to prove general upper bounds on the performance of a slightly modified version of Sutton‘s so-called TD((gl) algorithm. These bounds are stated in terms of the performance of the best linear predictor on the given training sequence, and are proved without making any statistical assumptions of any kind about the process producing the learner‘s observed training sequence. We also prove lower bounds on the performance of any algorithm for this learning problem, and give a similar analysis of the closely related problem of learning to predict in a model in which the learner must produce predictions for a whole batch of observations before receiving reinforcement.
A maximum entropy approach to species distribution modeling

We study the problem of modeling species geographic distributions, a critical problem in conservation biology. We propose the use of maximum-entropy techniques for this problem, specifically, sequential-update algorithms that can handle a very large number of features. We describe experiments comparing maxent with a standard distribution-modeling tool, called GARP, on a dataset containing observation data for North American breeding birds. We also study how well maxent performs as a function of the number of training examples and training time, analyze the use of regularization to avoid overfitting when the number of examples is small, and explore the interpretability of models constructed using maxent.
Performance Guarantees for Regularized Maximum Entropy Density Estimation

We consider the problem of estimating an unknown probability distribution from samples using the principle of maximum entropy (maxent). To alleviate overfitting with a very large number of features, we propose applying the maxent principle with relaxed constraints on the expectations of the features. By convex duality, this turns out to be equivalent to finding the Gibbs distribution minimizing a regularized version of the empirical log loss. We prove non-asymptotic bounds showing that, with respect to the true underlying distribution, this relaxed version of maxent produces density estimates that are almost as good as the best possible. These bounds are in terms of the deviation of the feature empirical averages relative to their true expectations, a number that can be bounded using standard uniform-convergence techniques. In particular, this leads to bounds that drop quickly with the number of samples, and that depend very moderately on the number or complexity of the features. We also derive and prove convergence for both sequential-update and parallel-update algorithms. Finally, we briefly describe experiments on data relevant to the modeling of species geographical distributions.
Boosting, margins, and dynamics

Boosting algorithms are currently used for a wide variety of statistical learning tasks, such as document classification and optical character recognition. Freund and Schapire's 1997 “AdaBoost” algorithm is the most popular boosting algorithm, due to its excellent performance on many types of data and fast rate of convergence. 
A strong indicator of the generalization ability of a boosted classifier is the value of the margin it achieves; large margin classifiers often achieve a low probability of error on test data. Thus, it is important to study the convergence properties of boosting algorithms, namely, whether they converge to maximum margin solutions. In empirical studies, AdaBoost seems to achieve maximum margins, and in theoretical studies, AdaBoost has been proven to achieve large margins; thus the margin theory somewhat accounts for AdaBoost's remarkable ability to generalize. The question we address in this thesis is whether AdaBoost provably converges to a maximum margin solution. 
In order to understand AdaBoost's convergence properties, we look toward its dynamics. We simplify AdaBoost to reveal a nonlinear iterated map and analyze its behavior in specific cases. We find stable cycles for these cases, which can explicitly be used to solve for AdaBoost's output. Thus, we find the key to answering the question of whether AdaBoost always maximizes the margin—a set of examples in which AdaBoost's convergence can be completely understood. 
By considering low-dimensional cases, we show that AdaBoost does not always converge to a maximum margin solution, answering the question stated above. Our results suggest that AdaBoost is a much richer system than previously thought; the margin theory only partially explains its strong generalization ability. This study also provides a method of analyzing AdaBoost's convergence of margins via its beautiful cyclic dynamics. 
Since AdaBoost does not necessarily maximize the margin, and since the margin is a strong indicator of an algorithm's generalization ability, we introduce two coordinate boosting algorithms that maximize the margin. Both algorithms perform well empirically and are as easy to implement as AdaBoost. We prove that both algorithms possess a polynomial-time convergence rate.
Decision-Theoretic Bidding Based on Learned Density Models in Simultaneous, Interacting Auctions

Auctions are becoming an increasingly popular method for transacting business, especially over the Internet. This article presents a general approach to building autonomous bidding agents to bid in multiple simultaneous auctions for interacting goods. A core component of our approach learns a model of the empirical price dynamics based on past data and uses the model to analytically calculate, to the greatest extent possible, optimal bids. We introduce a new and general boosting-based algorithm for conditional density estimation problems of this kind, i.e., supervised learning problems in which the goal is to estimate the entire conditional distribution of the real-valued label. This approach is fully implemented as ATTac-2001, a top-scoring agent in the second Trading Agent Competition (TAC-01). We present experiments demonstrating the effectiveness of our boosting-based price predictor relative to several reasonable alternatives.
On the Dynamics of Boosting

In order to understand AdaBoost's dynamics, especially its ability to maximize margins, we derive an associated simplified nonlinear iterated map and analyze its behavior in low-dimensional cases. We find stable cycles for these cases, which can explicitly be used to solve for AdaBoost's output. By considering AdaBoost as a dynamical system, we are able to prove Ratsch and Warmuth's conjecture that AdaBoost may fail to converge to a maximal-margin combined classifier when given a 'non-optimal' weak learning algorithm. AdaBoost is known to be a coordinate descent method, but other known algorithms that explicitly aim to maximize the margin (such as AdaBoost* and arc-gv) are not. We consider a differentiable function for which coordinate ascent will yield a maximum margin solution. We then make a simple approximation to derive a new boosting algorithm whose updates are slightly more aggressive than those of arc-gv.
Active learning for spoken language understanding

We describe active learning methods for reducing the labeling effort in a statistical call classification system. Active learning aims to minimize the number of labeled utterances by automatically selecting for labeling the utterances that are likely to be most informative. The first method, inspired by certainty-based active learning, selects the examples that the classifier is least confident about. The second method, inspired by committee-based active learning, selects the examples that multiple classifiers do not agree on. We have evaluated these active learning methods using a call classification system used for AT&T customer care. Our results indicate that it is possible to reduce human labeling effort at least by a factor of two.
The Boosting Approach to Machine Learning An Overview

Boosting is a general method for improving the accuracy of any given learning algorithm. Focusing primarily on the AdaBoost algorithm, this chapter overviews some of the recent work on boosting including analyses of AdaBoost’s training error and generalization error; boosting’s connection to game theory and linear programming; the relationship between boosting and logistic regression; extensions of AdaBoost for multiclass classification problems; methods of incorporating human knowledge into boosting; and experimental and applied work using boosting.