nils-herrmann/clean_scirepeval huggingface dataset:
Finite-size effects in Bayesian model selection and generalization

We show that in supervised learning from a supplied data set Bayesian model selection, based on the evidence, does not optimize generalization performance even for a learnable linear problem. This is demonstrated by examining the finite size effects in hyperparameter assignment from the evidence procedure and the resultant generalization performance. Our approach demonstrates the weakness of average case and asymptotic analyses. Using simulations we corroborate our analytic results and examine an alternative model selection criterion, namely cross-validation. This numerical study shows that the cross-validation hyperparameter estimates correlate more strongly than those of the evidence with optimal performance. However, we show that for a sufficiently large input dimension the evidence procedure could provide a reliable alternative to the more computationally expensive cross-validation.
Learning with Noise and Regularizers in Multilayer Neural Networks

We study the effect of noise and regularization in an on-line gradient-descent learning scenario for a general two-layer student network with an arbitrary number of hidden units. Training examples are randomly drawn input vectors labeled by a two-layer teacher network with an arbitrary number of hidden units; the examples are corrupted by Gaussian noise affecting either the output or the model itself. We examine the effect of both types of noise and that of weight-decay regularization on the dynamical evolution of the order parameters and the generalization error in various phases of the learning process.
Adaptive Back-Propagation in On-Line Learning of Multilayer Networks

An adaptive back-propagation algorithm is studied and compared with gradient descent (standard back-propagation) for on-line learning in two-layer neural networks with an arbitrary number of hidden units. Within a statistical mechanics framework, both numerical studies and a rigorous analysis show that the adaptive back-propagation method results in faster training by breaking the symmetry between hidden units more efficiently and by providing faster convergence to optimal generalization than gradient descent.
Learning and Generalization in Radial Basis Function Networks

The two-layer radial basis function network, with fixed centers of the basis functions, is analyzed within a stochastic training paradigm. Various definitions of generalization error are considered, and two such definitions are employed in deriving generic learning curves and generalization properties, both with and without a weight decay term. The generalization error is shown analytically to be related to the evidence and, via the evidence, to the prediction error and free energy. The generalization behavior is explored; the generic learning curve is found to be inversely proportional to the number of training pairs presented. Optimization of training is considered by minimizing the generalization error with respect to the free parameters of the training algorithms. Finally, the effect of the joint activations between hidden-layer units is examined and shown to speed training.
A statistical mechanical analysis of a Bayesian inference scheme for an unrealizable rule

Within a Bayesian framework we consider a system that learns from examples. In particular, using a statistical mechanical formalism, we calculate the evidence and two performance measures, namely the generalization error and the consistency measure, for a linear perceptron trained and tested on a set of examples generated by a nonlinear teacher. The teacher is said to be unrealizable because the student can never model it without error. In fact, our model allows us to interpolate between the known linear case and an unrealizable, nonlinear, case. A comparison of the hyperparameters which maximize the evidence with those that optimize the performance measures reveals that, when the student and teacher are fundamentally mismatched, the evidence procedure is a misleading guide to optimizing the performance measures considered.
Ranking-based evaluation of regression models

We suggest the use of ranking-based evaluation measures for regression models, as a complement to the commonly used residual-based evaluation. We argue that in some cases, such as the case study we present, ranking can be the main underlying goal in building a regression model, and ranking performance is the correct evaluation metric. However, even when ranking is not the contextually correct performance metric, the measures we explore still have significant advantages: They are robust against extreme outliers in the evaluation set; and they are interpretable. The two measures we consider correspond closely to non-parametric correlation coefficients commonly used in data analysis (Spearman's ρ and Kendall's τ); and they both have interesting graphical representations, which, similarly to ROC curves, offer useful various model performance views, in addition to a one-number summary in the area under the curve. An interesting extension which we explore is to evaluate models on their performance in “partially” ranking the data, which we argue can better represent the utility of the model in many cases. We illustrate our methods on a case study of evaluating IT Wallet size estimation models for IBM's customers.
Wallet Estimation Models

The wallet of a customer is defined as the total amount this customer can spend in a specific product category. This is a vital piece of information for planning marketing and sales efforts. We discuss the important problem of customer wallet estimation, while emphasizing the use of predictive modeling technologies to generate useful estimates, and minimizing reliance on primary research. We suggest several customer wallet definitions and corresponding evaluation approaches. Our main contribution is in presenting several new predictive modeling approaches which allow us to estimate customer wallets, despite the fact that these are typically unobserved. We present empirical results on the success of these modeling approaches, using a dataset of IBM customers.
Roc Confidence Bands: An Empirical Study

This paper is about constructing confidence bands around an ROCcurve such that (1 - \delta)% of the ROC curves traced by data setsof size r will fall completely within the bands. We introduce tothe machine learning community three methods from the medicalfield that are applicable to generate such bands. We then evaluatethese methods on the simple case of A'Â'A Â¢A'Â A Â¬A'Â A ÂSbinormalA'Â'A Â¢A'Â A Â¬A'Â A Â? distributionsA'Â'A Â¢A'Â A Â¬A'Â A Â the scores for positive and the score for negative instances aredrawn from two normal distributions. We show that none of themethods generate appropriate bands and investigate two types ofvariances problems. We show that widening the bands does notproduce the proper bandwidths but that fitting a normal distributionto the observed drawn samples and drawing samples fromthis distribution (parametric bootstrap) does generate bands thatare much closer to the desired coverage although still not perfect.We tested the original methods as well as parametric bootstrap onthe covertype data set from the UCI ML-repority. The originalmethods perform the same as in the synthetic case, whereas theparametric bootstrap technique did not yield the expected results.This is primarily due to not being able to generate a good fit forthe score distributions. Whether it is possible to fit well-behavingparametric distribution to learned models is an open question weleave to the machine learning community to answer.
ROC Confidence Bands : An Empirical Study 0

This paper is about constructing confidence bands around an R OC curve such that (1 − δ)% of the ROC curves traced by data sets of sizer will fall completely within the bands. We introduce to the machine learning community three methods from the medic al field that are applicable to generate such bands. We then eval uate these methods on the simple case of “binormal” distribution s— the scores for positive and the score for negative instances ar drawn from two normal distributions. We show that none of the methods generate appropriate bands and investigate two typ es of variances problems. We show that widening the bands does not produce the proper bandwidths but that fitting a normal distr ibution to the observed drawn samples and drawing samples from this distribution (parametric bootstrap) does generate ba nds that are much closer to the desired coverage although still not pe rfect. We tested the original methods as well as parametric bootstr ap on the covertype data set from the UCI ML-repority. The origina l methods perform the same as in the synthetic case, whereas th parametric bootstrap technique did not yield the expected r esults. This is primarily due to not being able to generate a good fit fo r the score distributions. Whether it is possible to fit well-b ehaving parametric distribution to learned models is an open questi on we leave to the machine learning community to answer.
Pointwise ROC Confidence Bounds: An Empirical Evaluation

This paper is about constructing and evaluating pointwise confidence bounds on an ROC curve. We describe four confidencebound methods, two from the medical field and two used previously in machine learning research. We evaluate whether the bounds indeed contain the relevant operating point on the “true” ROC curve with a confidence of 1−δ. We then evaluate pointwise confidence bounds on the region where the future performance of a model is expected to lie. For evaluation we use a synthetic world representing “binormal” distributions–the classification scores for positive and negative instances are drawn from (separate) normal distributions. For the “true-curve” bounds, all methods are sensitive to how well the distributions are separated, which corresponds directly to the area under the ROC curve. One method produces bounds that are universally too loose, another universally too tight, and the remaining two are close to the desired containment although containment breaks down at the extremes of the ROC curve. As would be expected, all methods fail when used to contain “future” ROC curves. Widening the bounds to account for the increased uncertainty yields identical qualitative results to the “true-curve” evaluation. We conclude by recommending a simple, very efficient method (vertical averaging) for large sample sizes and a more computationally expensive method (kernel estimation) for small sample sizes.
Ranking-based evaluation of regression models

We suggest the use of ranking-based evaluation measures for regression models, as a complement to the commonly used residual-based evaluation. We argue that in some cases, such as the case study we present, ranking can be the main underlying goal in building a regression model, and ranking performance is the correct evaluation metric. However, even when ranking is not the contextually correct performance metric, the measures we explore still have significant advantages: They are robust against extreme outliers in the evaluation set; and they are interpretable. The two measures we consider correspond closely to non-parametric correlation coefficients commonly used in data analysis (Spearman's p and Kendall's r); and they both have interesting graphical representations, which, similarly to ROC curves, offer useful "partial" model performance views, in addition to a one-number summary in the area under the curve. We illustrate our methods on a case study of evaluating IT wallet size estimation models for IBM's customers.
Sparsity and smoothness via the fused lasso

The lasso penalizes a least squares regression by the sum of the absolute values (L1‐norm) of the coefficients. The form of this penalty encourages sparse solutions (with many coefficients equal to 0). We propose the ‘fused lasso’, a generalization that is designed for problems with features that can be ordered in some meaningful way. The fused lasso penalizes the L1‐norm of both the coefficients and their successive differences. Thus it encourages sparsity of the coefficients and also sparsity of their differences—i.e. local constancy of the coefficient profile. The fused lasso is especially useful when the number of features p is much greater than N, the sample size. The technique is also extended to the ‘hinge’ loss function that underlies the support vector classifier. We illustrate the methods on examples from protein mass spectroscopy and gene expression data.
Robust boosting and its relation to bagging

Several authors have suggested viewing boosting as a gradient descent search for a good fit in function space. At each iteration observations are re-weighted using the gradient of the underlying loss function. We present an approach of weight decay for observation weights which is equivalent to "robustifying" the underlying loss function. At the extreme end of decay this approach converges to Bagging, which can be viewed as boosting with a linear underlying loss function. We illustrate the practical usefulness of weight decay for improving prediction performance and present an equivalence between one form of weight decay and "Huberizing" --- a statistical method for making loss functions more robust.
ROC confidence bands: an empirical evaluation

This paper is about constructing confidence bands around ROC curves. We first introduce to the machine learning community three band-generating methods from the medical field, and evaluate how well they perform. Such confidence bands represent the region where the "true" ROC curve is expected to reside, with the designated confidence level. To assess the containment of the bands we begin with a synthetic world where we know the true ROC curve---specifically, where the class-conditional model scores are normally distributed. The only method that attains reasonable containment out-of-the-box produces non-parametric, "fixed-width" bands (FWBs). Next we move to a context more appropriate for machine learning evaluations: bands that with a certain confidence level will bound the performance of the model on future data. We introduce a correction to account for the larger uncertainty, and the widened FWBs continue to have reasonable containment. Finally, we assess the bands on 10 relatively large benchmark data sets. We conclude by recommending these FWBs, noting that being non-parametric they are especially attractive for machine learning studies, where the score distributions (1) clearly are not normal, and (2) even for the same data set vary substantially from learning method to learning method.
Following Curved Regularized Optimization Solution Paths

Regularization plays a central role in the analysis of modern data, where non-regularized fitting is likely to lead to over-fitted models, useless for both prediction and interpretation. We consider the design of incremental algorithms which follow paths of regularized solutions, as the regularization varies. These approaches often result in methods which are both efficient and highly flexible. We suggest a general path-following algorithm based on second-order approximations, prove that under mild conditions it remains "very close" to the path of optimal solutions and illustrate it with examples.
A Method for Inferring Label Sampling Mechanisms in Semi-Supervised Learning

We consider the situation in semi-supervised learning, where the "label sampling" mechanism stochastically depends on the true response (as well as potentially on the features). We suggest a method of moments for estimating this stochastic dependence using the unlabeled data. This is potentially useful for two distinct purposes: a. As an input to a supervised learning procedure which can be used to "de-bias" its results using labeled data only and b. As a potentially interesting learning task in itself. We present several examples to illustrate the practical usefulness of our method.
Model selection via the AUC

We present a statistical analysis of the AUC as an evaluation criterion for classification scoring models. First, we consider significance tests for the difference between AUC scores of two algorithms on the same test set. We derive exact moments under simplifying assumptions and use them to examine approximate practical methods from the literature. We then compare AUC to empirical misclassification error when the prediction goal is to minimize future error rate. We show that the AUC may be preferable to empirical error even in this case and discuss the tradeoff between approximation error and estimation error underlying this phenomenon.
Discussion of "Least angle regression" by Efron et al

1.  We congratulate the authors on their excellent work. The paper combines elegant theory and useful practical results in an intriguing manner. The LAR–Lasso–boosting relationship opens the door for new insights on existing methods' underlying statistical mechanisms and for the development of new and promising methodology. Two issues in particular have captured our attention, as their implications go beyond the squared error loss case presented in this paper, into wider statistical domains: robust fitting, classification, machine learning and more. We concentrate our discussion on these two results and their extensions.
Customer Lifetime Value Models for Decision Support

We present and discuss the important business problem of estimating the effect of marketing activities on the Lifetime Value of a customer in the Telecommunications industry. We discuss the components of this problem, in particular customer value and length of service (or tenure) modeling, and present a novel segment-based approach, motivated by the segment-level view marketing analysts usually employ. We describe in detail how we build on this approach to estimate the effects of retention campaigns on Lifetime Value, and also discuss its application in other situations. Our solution has been successfully implemented by the Business Insight (BI) Professional Services.
Corrected proof of the result of ‘A prediction error property of the Lasso estimator and its generalization’ by Huang (2003)

The Lasso achieves variance reduction and variable selection by solving an ℓ1‐regularized least squares problem. Huang (2003) claims that ‘there always exists an interval of regularization parameter values such that the corresponding mean squared prediction error for the Lasso estimator is smaller than for the ordinary least square estimator’. This result is correct. However, its proof in Huang (2003) is not. This paper presents a corrected proof of the claim, which exposes and uses some interesting fundamental properties of the Lasso.
Boosting as a Regularized Path to a Maximum Margin Classifier

In this paper we study boosting methods from a new perspective. We build on recent work by Efron et al. to show that boosting approximately (and in some cases exactly) minimizes its loss criterion with an l1 constraint on the coefficient vector. This helps understand the success of boosting with early stopping as regularized fitting of the loss criterion. For the two most commonly used criteria (exponential and binomial log-likelihood), we further show that as the constraint is relaxed---or equivalently as the boosting iterations proceed---the solution converges (in the separable case) to an "l1-optimal" separating hyper-plane. We prove that this l1-optimal separating hyper-plane has the property of maximizing the minimal l1-margin of the training data, as defined in the boosting literature. An interesting fundamental similarity between boosting and kernel support vector machines emerges, as both can be described as methods for regularized optimization in high-dimensional predictor space, using a computational trick to make the calculation practical, and converging to margin-maximizing solutions. While this statement describes SVMs exactly, it applies to boosting only approximately.
1-norm Support Vector Machines

The standard 2-norm SVM is known for its good performance in two-class classification. In this paper, we consider the 1-norm SVM. We argue that the 1-norm SVM may have some advantage over the standard 2-norm SVM, especially when there are redundant noise features. We also propose an efficient algorithm that computes the whole solution path of the 1-norm SVM, hence facilitates adaptive selection of the tuning parameter for the 1-norm SVM.
Boosting and Support Vector Machines as Optimal Separators

In this paper we study boosting methods from a new perspective. We build on recent work by Efron et al. to show that boosting approximately (and in some cases exactly) minimizes its loss criterion with an L1 constraint. For the two most commonly used loss criteria (exponential and logistic log-likelihood), we further show that as the constraint diminishes, or equivalently as the boosting iterations proceed, the solution converges in the separable case to an “L1-optimal” separating hyper-plane. This “L1-optimal” separating hyper-plane has the property of maximizing the minimal margin of the training data, as de£ned in the boosting literature. We illustrate through examples the regularized and asymptotic behavior of the solutions to the classifcation problem with both loss criteria.
Integrating customer value considerations into predictive modeling

The success of prediction models for business purposes should not be measured by their accuracy only. Their evaluation should also take into account the higher importance of precise prediction for "valuable" customers. We illustrate this idea through the example of churn modelling in telecommunications, where it is obviously much more important to identify potential churn among valuable customers. We discuss, both theoretically and empirically, the optimal use of "customer value" data in the model training, model evaluation and scoring stages. Our main conclusion is that a nontrivial approach of using "decayed" value-weights for training is usually preferable to the two obvious approaches of either using nondecayed customer values as weights or ignoring them.
Discussion of Boosting Papers

We congratulate the authors for their interesting papers on boosting and related topics. Jiang deals with the asymptotic consistency of Adaboost. Lugosi and Vayatis study the convex optimization of loss functions associated with boosting. Zhang studies the loss functions themselves. Their results imply that boosting-like methods can reasonably be expected to converge to Bayes classifiers under sufficient regularity conditions (such as the requirement that trees with at least p+ 1 terminal nodes are used, where p is the number of variables in the model). An interesting feature of their results is that whenever data-based optimization is performed, some form of regularization is needed in order to attain consistency. In the case of AdaBoost this is achieved by stopping the boosting procedure early, whereas in the case of convex loss optimization, it is achieved by constraining the L1 norm of the coefficient vector. These results re-iterate, from this new perspective, the critical importance of regularization for building useful prediction models in high-dimensional space. This is also the theme of the remainder of our discussion. Since the publication of the AdaBoost procedure by Freund and Schapire in 1996, there has been a flurry of papers seeking to answer the question: why does boosting work? Since AdaBoost has been generalized in different ways by different authors, the question might be better posed as; what are the aspects of boosting that are the key to its good performance?
�-norm Support Vector Machines

The standard -norm SVM is known for its good performance in twoclass classification. In this paper, we consider the -norm SVM. We argue that the -norm SVM may have some advantage over the standard -norm SVM, especially when there are redundant noise features. We also propose an efficient algorithm that computes the whole solution path of the -norm SVM, hence facilitates adaptive selection of the tuning parameter for the -norm SVM.