nils-herrmann/clean_scirepeval huggingface dataset:
Active Learning for Parzen Window Classifier

The problem of active learning is approached in this paper by minimizing directly an estimate of the expected test error. The main difficulty in this “optimal” strategy is that output probabilities need to be estimated accurately. We suggest here different methods for estimating those efficiently. In this context, the Parzen window classifier is considered because it is both simple and probabilistic. The analysis of experimental results highlights that regularization is a key ingredient for this strategy.
Implicit surface modelling as an eigenvalue problem

We discuss the problem of fitting an implicit shape model to a set of points sampled from a co-dimension one manifold of arbitrary topology. The method solves a non-convex optimisation problem in the embedding function that defines the implicit by way of its zero level set. By assuming that the solution is a mixture of radial basis functions of varying widths we attain the globally optimal solution by way of an equivalent eigenvalue problem, without using or constructing as an intermediate step the normal vectors of the manifold at each data point. We demonstrate the system on two and three dimensional data, with examples of missing data interpolation and set operations on the resultant shapes.
An Analysis of the Anti-learning Phenomenon for the Class Symmetric Polyhedron

This paper deals with an unusual phenomenon where most machine learning algorithms yield good performance on the training set but systematically worse than random performance on the test set. This has been observed so far for some natural data sets and demonstrated for some synthetic data sets when the classification rule is learned from a small set of training samples drawn from some high dimensional space. The initial analysis presented in this paper shows that anti-learning is a property of data sets and is quite distinct from over-fitting of a training data. Moreover, the analysis leads to a specification of some machine learning procedures which can overcome anti-learning and generate machines able to classify training and test data consistently.
Semi-Supervised Classification by Low Density Separation

We believe that the cluster assumption is key to successful semi-supervised learning. Based on this, we propose three semi-supervised algorithms: 1. deriving graph-based distances that emphazise low density regions between clusters, followed by training a standard SVM; 2. optimizing the Transductive SVM objective function, which places the decision boundary in low density regions, by gradient descent; 3. combining the first two to make maximum use of the cluster assumption. We compare with state of the art algorithms and demonstrate superior accuracy for the latter two methods.
Choosing Multiple Parameters for Support Vector Machines

The problem of automatically tuning multiple parameters for pattern recognition Support Vector Machines (SVMs) is considered. This is done by minimizing some estimates of the generalization error of SVMs using a gradient descent algorithm over the set of parameters. Usual methods for choosing parameters, based on exhaustive search become intractable as soon as the number of parameters exceeds two. Some experimental results assess the feasibility of our approach for a large number of parameters (more than 100) and demonstrate an improvement of generalization performance.
Feature selection for support vector machines by means of genetic algorithm

The problem of feature selection is a difficult combinatorial task in machine learning and of high practical relevance, e.g. in bioinformatics. genetic algorithms (GAs) offer a natural way to solve this problem. In this paper, we present a special genetic algorithm, which especially takes into account the existing bounds on the generalization error for support vector machines (SVMs). This new approach is compared to the traditional method of performing cross-validation and to other existing algorithms for feature selection.
Incorporating invariances in nonlinear Support Vector Machines

The choice of an SVM kernel corresponds to the choice of a representation of the data in a feature space and, to improve performance, it should therefore incorporate prior knowledge such as known transformation invariances. We propose a technique which extends earlier work and aims at incorporating invariances in nonlinear kernels. We show on a digit recognition task that the proposed approach is superior to the Virtual Support Vector method, which previously had been the method of choice.
Bounds on Error Expectation for Support Vector Machines

We introduce the concept of span of support vectors (SV) and show that the generalization ability of support vector machines (SVM) depends on this new geometrical concept. We prove that the value of the span is always smaller (and can be much smaller) than the diameter of the smallest sphere containing the support vectors, used in previous bounds (Vapnik, 1998). We also demonstate experimentally that the prediction of the test error given by the span is very accurate and has direct application in model selection (choice of the optimal parameters of the SVM).
A Non-asymptotic Theory for Model Selection

Model selection is a classical topic in statistics. The idea of selecting a model via penalizing a log-likelihood type criterion goes back to the early seventies with the pioneering works of Mallows and Akaike. One can find many consistency results in the literature for such criteria. These results are asymptotic in the sense that one deals with a given number of models and the number of observations tends to infinity. We shall give an overview of a non asymtotic theory for model selection which has emerged during these last ten years. In various contexts of function estimation it is possible to design penalized log-likelihood type criteria with penalty terms depending not only on the number of parameters defining each model (as for the classical criteria) but also on the ”complexity” of the whole collection of models to be considered. The performance of such a criterion is analyzed via non asymptotic risk bounds for the corresponding penalized estimator which express that it performs almost as well as if the ”best model” (i.e. with minimal risk) were known. For practical relevance of these methods, it is desirable to get a precise expression of the penalty terms involved in the penalized criteria on which they are based. This is why this approach heavily relies on concentration inequalities, the prototype being Talagrand’s inequality for empirical processes. Our purpose will be to give an account of the theory and discuss some selected applications.
Moment inequalities for functions of independent random variables

A general method for obtaining moment inequalities for functions of independent random variables is presented. It is a generalization of the entropy method which has been used to derive concentration inequalities for such functions [Boucheron, Lugosi and Massart  (2003) 1583-1614], and is based on a generalized tensorization inequality due to Latala and Oleszkiewicz [ (2000) 147-168]. The new inequalities prove to be a versatile tool in a wide range of applications. We illustrate the power of the method by showing how it can be used to effortlessly re-derive classical inequalities including Rosenthal and Kahane-Khinchine-type inequalities for sums of independent random variables, moment inequalities for suprema of empirical processes and moment inequalities for Rademacher chaos and U-statistics. Some of these corollaries are apparently new. In particular, we generalize Talagrand's exponential inequality for Rademacher chaos of order 2 to any order. We also discuss applications for other complex functions of independent random variables, such as suprema of Boolean polynomials which include, as special cases, subgraph counting problems in random graphs.
Risk bounds for Statistical Learning ( Running Title : Risk bounds for Statistical Learning )

We propose a general theorem providing upper bounds for the risk of an empirical risk minimizer (ERM).We essentially focus on the binary classication framework. We extend Tsybakovs analysis of the risk of an ERM under margin type conditions by using concentration inequalities for conveniently weighted empirical processes. This allows us to deal with other ways of measuring the sizeof a class of classiers than entropy with bracketing as in Tsybakovs work. In particular we derive new risk bounds for the ERM when the classication rules belong to some VC-class under margin conditions and discuss the optimality of those bounds in a minimax sense.
DISCUSSION OF "LEAST ANGLE REGRESSION" BY EFRON ET AL.

BY EFRONET AL.By Jean-Michel Loubes and Pascal MassartUniversit´e Paris-SudThe issue of model selection has drawn the attention of both applied andtheoretical statisticians for a long time. Indeed, there has been an enor-mous range of contribution in model selection proposals, including work byAkaike (1973), Mallows (1973), Foster and George (1994), Birg´e and Mas-sart (2001a) and Abramovich, Benjamini, Donoho and Johnstone (2000).Over the last decade, modern computer-driven methods have been devel-oped such as All Subsets, Forward Selection, Forward Stagewise or Lasso.Such methods are useful in the setting of the standard linear model, wherewe observe noisy data and wish to predict the response variable using onlya few covariates, since they provide automatically linear models that ﬁt thedata. The procedure described in this paper is, on the one hand, numeri-cally very eﬃcient and, on the other hand, very general, since, with slightmodiﬁcations, it enables us to recover the estimates given by the Lasso andStagewise.1. Estimation procedure. The “LARS” method is based on a recursiveprocedure selecting, at each step, the covariates having largest absolute cor-relation with the response y. In the case of an orthogonal design, the esti-mates can then be viewed as an l
Concentration inequalities using the entropy method

We investigate a new methodology, worked out by Ledoux and Massart, to prove concentration-of-measure inequalities. The method is based on certain modified logarithmic Sobolev inequalities. We provide some very simple and general ready-to-use inequalities. One of these inequalities may be considered as an exponential version of the Efron--Stein inequality. The main purpose of this paper is to point out the simplicity and the generality of the approach. We show how the new method can recover many of Talagrand's revolutionary inequalities and provide new applications in a variety of problems including Rademacher averages, Rademacher chaos, the number of certain small subgraphs in a random graph, and the minimum of the empirical risk in some statistical estimation problems.
Gaussian model selection

Our purpose in this paper is to provide a general approach to model selection via penalization for Gaussian regression and to develop our point of view about this subject. The advantage and importance of model selection come from the fact that it provides a suitable approach to many different types of problems, starting from model selection per se (among a family of parametric models, which one is more suitable for the data at hand), which includes for instance variable selection in regression models, to nonparametric estimation, for which it provides a very powerful tool that allows adaptation under quite general circumstances. Our approach to model selection also provides a natural connection between the parametric and nonparametric points of view and copes naturally with the fact that a model is not necessarily true. The method is based on the penalization of a least squares criterion which can be viewed as a generalization of Mallows’Cp. A large part of our efforts will be put on choosing properly the list of models and the penalty function for various estimation problems like classical variable selection or adaptive estimation for various types of lp-bodies.
4 From Model Selection to Adaptive Estimation

Many different model selection information criteria can be found in the literature in various contexts including regression and density estimation. There is a huge amount of literature concerning this subject and we shall, in this paper, content ourselves to cite only a few typical references in order to illustrate our presentation. Let us just mention AIC, Cp or CL, BIC and MDL criteria proposed by Akaike (1973), Mallows (1973), Schwarz (1978), and Rissanen (1978) respectively. These methods propose to select among a given collection of parametric models that model which minimizes an empirical loss (typically squared error or minus log-likelihood) plus some penalty term which is proportional to the dimension of the model. From one criterion to another the penalty functions differ by factors of logn, where n represents the number of observations. The reasons for choosing one penalty rather than another come either from information theory or Bayesian asymptotic computations or approximate evaluations of the risk on specific families of models. Many efforts were made to understand in what circumstances these criteria allow to identify the right model asymptotically (see Li (1987) for instance). Much less is known about the performances of the estimators provided by these methods from a nonparametric point of view. Let us consider the particular context of density estimation in L for instance. By a nonparametric point of view, we mean that the unknown density does not necessarily belong to any of the given models and that the best model should approximately realize the best trade-off between the risk of estimation within the model and the distance of the unknown density to the model. When the models have good approximation properties (following Grenander (1981) these models will be called sieves), an adequate choice of the penalty can produce adaptive estimators in the sense that they estimate a density of unknown
An Adaptive Compression Algorithm in Besov Spaces

Given a function f on [0,1] and a wavelet-type expansion of f , we introduce a new algorithm providing an approximation 
$\tilde f of f with a prescribed number D of nonzero coefficients in its expansion. This algorithm depends only on the number of coefficients to be kept and not on any smoothness assumption on f . Nevertheless it provides the optimal rate D-α of approximation with respect to the Lq -norm when f belongs to some Besov space Bαp,∈fty whenever α>(1/p-1/q)+ . These results extend to more general expansions including splines and piecewise polynomials and to multivariate functions. Moreover, this construction allows us to compute easily the metric entropy of Besov balls.
Some applications of concentration inequalities to statistics

Nous presentons quelques applications d'inegalites de concentration a la resolution de problemes de selection de modeles en statistique. Nous etudions en detail deux exemples pour lesquels cette approche s'avere particulirement fructueuse. Nous considerons tout d'abord le classique mais delicat probleme du choix d'un bon histogramme. Nous presentons un extrait de travail de Castellan sur la question, mettant en evidence que la structure meme des inegalites de concentration de Talagrand pour des processus empiriques influence directement la construction d'un critere de selection de type Akaike modifie. Nous presentons egalement un nouveau theoreme de selection de modeles bien adapte a la resolution de problemes d'apprentissage. Ce resultat permet de reinterpreter et d'ameliorer la methode dite de minimisation structurelle du risque due a Vapnik.
About the constants in Talagrand's concentration inequalities for empirical processes

We propose some explicit values for the constants involved in the exponential concentration inequalities for empirical processes which are due to Talagrand. It has been shown by Ledoux that deviation inequalities for empirical processes could be obtained by iteration of logarithmic Sobolev type inequalities. Our approach follows closely that of Ledoux. The improvements that we get with respect to Ledoux’s work are based on refinements of his entropy inequalities and computations.
Adaptive estimation of a quadratic functional by model selection

We consider the problem of estimating ∥s∥ 2 when s belongs to some separable Hilbert space and one observes the Gaussian process Y(t) = (s, t) + σ L(t), for all t ∈ H, where L is some Gaussian isonormal process. This framework allows us in particular to consider the classical Gaussian sequence model for which H = l 2 (N * ) and L(t) = Σ λ≥1 t λ e λ , where (e λ ) λ≥1 is a sequence of i.i.d. standard normal variables. Our approach consists in considering some at most countable families of finite-dimensional linear subspaces of H (the models) and then using model selection via some conveniently penalized least squares criterion to build new estimators of ∥s∥ 2 . We prove a general nonasymptotic risk bound which allows us to show that such penalized estimators are adaptive on a variety of collections of sets for the parameter s, depending on the family of models from which they are built. In particular, in the context of the Gaussian sequence model, a convenient choice of the family of models allows defining estimators which are adaptive over collections of hyperrectangles, ellipsoids, l p -bodies or Besov bodies. We take special care to describe the conditions under which the penalized estimator is efficient when the level of noise σ tends to zero. Our construction is an alternative to the one by Efroimovich and Low for hyperrectangles and provides new results otherwise.
Risk bounds for model selection via penalization

Performance bounds for criteria for model selection are developed using recent theory for sieves. The model selection criteria are based on an empirical loss or contrast function with an added penalty term motivated by empirical process theory and roughly proportional to the number of parameters needed to describe the model divided by the number of observations. Most of our examples involve density or regression estimation settings and we focus on the problem of estimating the unknown density or regression function. We show that the quadratic risk of the minimum penalized empirical contrast estimator is bounded by an index of the accuracy of the sieve. This accuracy index quantifies the trade-off among the candidate models between the approximation error and parameter dimension relative to sample size.If we choose a list of models which exhibit good approximation properties with respect to different classes of smoothness, the estimator can be simultaneously minimax rate optimal in each of those classes. This is what is usually called adaptation. The type of classes of smoothness in which one gets adaptation depends heavily on the list of models. If too many models are involved in order to get accurate approximation of many wide classes of functions simultaneously, it may happen that the estimator is only approximately adaptive (typically up to a slowly varying function of the sample size).We shall provide various illustrations of our method such as penalized maximum likelihood, projection or least squares estimation. The models will involve commonly used finite dimensional expansions such as piecewise polynomials with fixed or variable knots, trigonometric polynomials, wavelets, neural nets and related nonlinear expansions defined by superposition of ridge functions.
From Model Selection to Adaptive Estimation

Many different model selection information criteria can be found in the literature in various contexts including regression and density estimation. There is a huge amount of literature concerning this subject and we shall, in this paper, content ourselves to cite only a few typical references in order to illustrate our presentation. Let us just mention AIC, C p , or C L , BIC and MDL criteria proposed by Akaike (1973), Mallows (1973), Schwarz (1978), and Rissanen (1978) respectively. These methods propose to select among a given collection of parametric models that model which minimizes an empirical loss (typically squared error or minus log-likelihood) plus some penalty term which is proportional to the dimension of the model. From one criterion to another the penalty functions differ by factors of log n, where n represents the number of observations.
Invariance principles for absolutely regular empirical processes

Soit (ξ i ) i ∈ Z une suite strictement stationnaire de variables aleatoires de loi marginale P. Soit Z n = n −1/2 ∑ 1 n (δ ξi − P) la mesure empirique normalisee et centree. On suppose que la suite (ξ i ) i ∈ Z est β-melangeante, de coefficients de β-melange β n en serie sommable. On lui associe la fonction de melange β(.) definie par β(t) = β [t] si t ≥ 1 et β(t) = 1 sinon. Pour toute fonction numerique f, on note Q f la fonction de quantile de |f (ξ 0 )|. Nous definissons une nouvelle forme pour f par ... (formule)... ou β −1 designe l'inverse cadlag de la fonction decroissante β(.). Cette norme coincide avec la forme usuelle de L 2 (P) dans le cas independant. On note alors L 2,β (P) l'espace des fonctions numeriques de forme ∥.∥ 2,β finie. Soit F une classe de fonctions de L 2,β (P). Dans un article recent, les auteurs ont montre la convergence vers un vecteur gaussien des marginales de {Z n (f): f ∈ F} de dimension finie. Dans cet article, nous demontrons un principe d'invariance fonctionnel au sens de Donsker pour {Z n (f): f ∈ F} sous une condition d'integrabilite de l'entropie avec crochets de F par rapport a la norme ∥.∥ 2,β . Ce theoreme generalise celui d'Ossiander (1987) pour des variables independantes
Estimation of Integral Functionals of a Density

Let φ be a smooth function of k + 2 variables. We shall investigate in this paper the rates of convergence of estimators of T(f)= fφ(f(x), f'(x),..., f (k) (x), x) dx when f belongs to some class of densities of smoothness s. We prove that, when δ ≥ 2k + 1/4, one can define an estimator T n of T(f), based on n i.i.d. observations of density f on the real line, which converges at the semiparametric rate 1/√n. On the other hand, when s < 2k + 1/4, T(f) cannot be estimated at a rate faster than n -γ with γ = 4(s-k)/[4s + 1]. We shall also provide some extensions to the multidimensional case. Those results extend previous works of Levit, of Bickel and Ritov and of Donoho and Nussbaum on estimation of quadratic functionals.
Initializing subspace constrained Gaussian mixture models

A recent series of papers [1, 2, 3, 4] introduced subspace constrained Gaussian mixture models (SCGMM) and showed that SCGMM can very efficiently approximate full covariance Gaussian mixture models (FCGMM); a significant reduction in the number of parameters is achieved with little loss in the accuracy of the model. SCGMM were arrived at as a sequence of generalizations of diagonal covariance GMM. As an artifact of this process the initialization of SCGMM parameters in that work is complex, i.e., relies on best parameter settings of less general models. This paper overcomes this problem by showing how an FCGMM can be used to give a simple and direct initialization of an SCGMM. The initialization scheme is powerful enough that as the number of parameters in an SCGMM approaches that of an FCGMM (i.e., large SCGMM) further training of the SCGMM is unnecessary.
Phaselets of framelets

Phaselets are a set of dyadic wavelets that are related in a particular way such that the associated redundant wavelet transform is nearly shift-invariant. Framelets are a set of functions that generalize the notion of a single dyadic wavelet in the sense that dyadic dilates and translates of these functions form a frame in L/sup 2/(IR). This paper generalizes the notion of phaselets to framelets. Sets of framelets that only differ in their Fourier transform phase are constructed such that the resulting redundant wavelet transform is approximately shift invariant. Explicit constructions of phaselets are given for frames with two and three framelet generators. The results in this paper generalize the construction of Hilbert transform pairs of framelets.