nils-herrmann/clean_scirepeval huggingface dataset:
Model Selection for Small Sample Regression

Model selection is an important ingredient of many machine learning algorithms, in particular when the sample size in small, in order to strike the right trade-off between overfitting and underfitting. Previous classical results for linear regression are based on an asymptotic analysis. We present a new penalization method for performing model selection for regression that is appropriate even for small samples. Our penalization is based on an accurate estimator of the ratio of the expected training error and the expected generalization error, in terms of the expected eigenvalues of the input covariance matrix.
Guest Introduction: Special Issue on New Methods for Model Selection and Model Combination

A classical challenge in fitting models to data is managing the complexity of the models to simultaneously avoid under-fitting and over-fitting the data; fulfilling the goal of producing models that generalize well to unseen data. The papers in this special issue present recent developments in model-complexity control for supervised learning. These thirteen papers represent three areas of significant current research on this subject: model selection (explicitly choosing model complexity), sparse models (reducing complexity by enforcing sparse representations), and model combination (combining multiple models to improve generalization).
Unsupervised Sense Disambiguation Using Bilingual Probabilistic Models

We describe two probabilistic models for unsupervised word-sense disambiguation using parallel corpora. The first model, which we call the Sense model, builds on the work of Diab and Resnik (2002) that uses both parallel text and a sense inventory for the target language, and recasts their approach in a probabilistic framework. The second model, which we call the Concept model, is a hierarchical model that uses a concept latent variable to relate different language specific sense labels. We show that both models improve performance on the word sense disambiguation task over previous unsupervised approaches, with the Concept model showing the largest improvement. Furthermore, in learning the Concept model, as a by-product, we learn a sense inventory for the parallel language.
Locally Weighted Full Covariance Gaussian Density Estimation

We describe an interesting application of the principle of local learning to density estimation. Locally weighted fitting of a Gaussian with a regularized full covariance matrix yields a density estimator which displays improved behavior in the case where much of the probability mass is concentrated along a low dimensional manifold. While the proposed estimator is not guaranteed to integrate to 1 with a finite sample size, we prove asymptotic convergence to the true density. Experimental results illustrating the advantages of this estimator over classic non-parametric estimators are presented. Nous decrivons une application du principe d'apprentissage local a l'estimation de densite. Le lissage pondere localement d'une gaussienne utilisant une matrice de covariance pleine et regularisee conduit a un estimateur de densite ayant un comportement ameliore lorsque la masse de probabilite est concentree le long d'une variete de basse dimension. Meme si l'estimateur propose n'est pas garanti d'integrer a 1 sur un ensemble de donnees fini, nous prouvons la convergence asymptotique de la vraie densite. Les resultats experimentaux illustrant les avantages de cet estimateur sur les estimateurs non parametriques classiques sont presentes.
Markovian Models for Sequential

Hidden Markov Models HMMs are statistical models of sequential data that have been used successfully in many machine learning applications especially for speech recognition Further more in the last few years many new and promising probabilistic models related to HMMs have been proposed We rst summarize the basics of HMMs and then review several recent related learning algorithms and extensions of HMMs including in particular hybrids of HMMs with arti cial neural networks Input Output HMMs which are conditional HMMs using neu ral networks to compute probabilities weighted transducers variable length Markov models and Markov switching state space models Finally we discuss some of the challenges of future research in this very active area
Inference for the Generalization Error

In order to compare learning algorithms, experimental results reported in the machine learning literature often use statistical tests of significance to support the claim that a new learning algorithm generalizes better. Such tests should take into account the variability due to the choice of training set and not only that due to the test examples, as is often the case. This could lead to gross underestimation of the variance of the cross-validation estimator, and to the wrong conclusion that the new algorithm is significantly better when it is not. We perform a theoretical investigation of the variance of a variant of the cross-validation estimator of the generalization error that takes into account the variability due to the randomness of the training set as well as test examples. Our analysis shows that all the variance estimators that are based only on the results of the cross-validation experiment must be biased. This analysis allows us to propose new estimators of this variance. We show, via simulations, that tests of hypothesis about the generalization error using those new variance estimators have better properties than tests involving variance estimators currently in use and listed in Dietterich (1998). In particular, the new tests have correct size and good power. That is, the new tests do not reject the null hypothesis too often when the hypothesis is true, but they tend to frequently reject the null hypothesis when the latter is false.
Kernel Matching Pursuit

Matching Pursuit algorithms learn a function that is a weighted sum of basis functions, by sequentially appending functions to an initially empty basis, to approximate a target function in the least-squares sense. We show how matching pursuit can be extended to use non-squared error loss functions, and how it can be used to build kernel-based solutions to machine learning problems, while keeping control of the sparsity of the solution. We present a version of the algorithm that makes an optimal choice of both the next basis and the weights of all the previously chosen bases. Finally, links to boosting algorithms and RBF training procedures, as well as an extensive experimental comparison with SVMs for classification are given, showing comparable results with typically much sparser models.
Discovering Shared Structure in Manifold Learning

We claim and present arguments to the effect that a large class of manifold learning algorithms that are essentially local will suffer from at least four generic problems associated with (1) noise in the data, (2) curvature of the manifold, (3) dimensionality of the manifold, and (4) the presence of many manifolds with little data per manifold. This analysis suggests non-local manifold learning algorithms which attempt to discover shared structure in the tangent planes at different positions. A criterion for such an algorithm is proposed and experiments estimating a tangent plane prediction function are presented. The function has parameters that are shared across space rather than estimated based on the local neighborhood, as in current non-parametric manifold learning algorithms. The results show clearly the advantages of this approach with respect to local manifold learning algorithms.
Learning from Partial Labels with Minimum Entropy

This paper introduces the minimum entropy regularizer for learning from partial labels. This learning problem encompasses the semi-supervised setting, where a decision rule is to be learned from labeled and unlabeled examples. The minimum entropy regularizer applies to diagnosis models, i.e. models of the posterior probabilities of classes. It is shown to include other approaches to the semi-supervised problem as particular or limiting cases. A series of experiments illustrates that the proposed criterion provides solutions taking advantage of unlabeled examples when the latter convey information. Even when the data are sampled from the distribution class spanned by a generative model, the proposed approach improves over the estimated generative model when the number of features is of the order of sample size. The performances are definitely in favor of minimum entropy when the generative model is slightly misspecified. Finally, the robustness of the learning scheme is demonstrated: in situations where unlabeled examples do not convey information, minimum entropy returns a solution discarding unlabeled examples and performs as well as supervised learning. Cet article introduit le regularisateur a entropie minimum pour l'apprentissage d'etiquettes partielles. Ce probleme d'apprentissage incorpore le cadre non supervise, ou une regle de decision doit etre apprise a partir d'exemples etiquetes et non etiquetes. Le regularisateur a entropie minimum s'applique aux modeles de diagnostics, c'est-a-dire aux modeles des probabilites posterieures de classes. Nous montrons comment inclure d'autres approches comme un cas particulier ou limite du probleme semi-supervise. Une serie d'experiences montrent que le critere propose fournit des solutions utilisant les exemples non etiquetes lorsque ces dernieres sont instructives. Meme lorsque les donnees sont echantillonnees a partir de la classe de distribution balayee par un modele generatif, l'approche mentionnee ameliore le modele generatif estime lorsque le nombre de caracteristiques est de l'ordre de la taille de l'echantillon. Les performances avantagent certainement l'entropie minimum lorsque le modele generatif est legerement mal specifie. Finalement, la robustesse de ce cadre d'apprentissage est demontre : lors de situations ou les exemples non etiquetes n'apportent aucune information, l'entropie minimum retourne une solution rejetant les exemples non etiquetes et est aussi performante que l'apprentissage supervise.
Semi-supervised Learning by Entropy Minimization

We consider the semi-supervised learning problem, where a decision rule is to be learned from labeled and unlabeled data. In this framework, we motivate minimum entropy regularization, which enables to incorporate unlabeled data in the standard supervised learning. Our approach includes other approaches to the semi-supervised problem as particular or limiting cases. A series of experiments illustrates that the proposed solution benefits from unlabeled data. The method challenges mixture models when the data are sampled from the distribution class spanned by the generative model. The performances are definitely in favor of minimum entropy regularization when generative models are misspecified, and the weighting of unlabeled data provides robustness to the violation of the "cluster assumption". Finally, we also illustrate that the method can also be far superior to manifold learning in high dimension spaces.
Approche statistique pour le repérage de mots informatifs dans les textes oraux

Nous présentons les résultats de l’approche statistique que nous avons développée pour le repérage de mots informatifs à partir de textes oraux. Ce travail fait partie d’un projet lancé par le département de la défense canadienne pour le développement d’un système d’extraction d’information dans le domaine de la Recherche et Sauvetage maritime (SAR). Il s’agit de trouver et annoter les mots pertinents avec des étiquettes sémantiques qui sont les concepts d’une ontologie du domaine (SAR). Notre méthode combine deux types d’information : les vecteurs de similarité générés grâce à l’ontologie du domaine et le dictionnaire-thésaurus Wordsmyth ; le contexte d’énonciation représenté par le thème. L’évaluation est effectuée en comparant la sortie du système avec les réponses de formulaires d’extraction d’information prédéfinis. Les résultats obtenus sur les textes oraux sont comparables à ceux obtenus dans le cadre de MUC7 pour des textes écrits.
Repérage de mots informatifs dans les textes conversationnels

Nous presentons les resultats d'une approche d'etiquetage semantique developpee pour le reperage de mots informatifs a partir de textes conversationnels. Ce travail entre dans le cadre du developpement d'un systeme d'extraction d'information dans le domaine de la recherche et sauvetage maritime. Il s'agit de detecter et d'annoter les mots pertinents avec des etiquettes semantiques correspondant aux concepts d'une ontologie du domaine. Notre methode combine une approche symbolique basee sur un automate a etats finis et une approche statistique exploitant deux types d'information: les vecteurs de scores de similarite et le contexte discursif represente par le theme. Le F-score obtenu sur des transcriptions manuelles de conversations telephoniques dans le domaine de la recherche et sauvetage maritime est de 82,2 %.
Learning Eigenfunctions Links Spectral Embedding and Kernel PCA

In this letter, we show a direct relation between spectral embedding methods and kernel principal components analysis and how both are special cases of a more general learning problem: learning the principal eigenfunctions of an operator defined from a kernel and the unknown data-generating density. Whereas spectral embedding methods provided only coordinates for the training points, the analysis justifies a simple extension to out-of-sample examples (the Nystrm formula) for multidimensional scaling (MDS), spectral clustering, Laplacian eigenmaps, locally linear embedding (LLE), and Isomap. The analysis provides, for all such spectral embedding methods, the definition of a loss function, whose empirical average is minimized by the traditional algorithms. The asymptotic expected value of that loss defines a generalization performance and clarifies what these algorithms are trying to learn. Experiments with LLE, Isomap, spectral clustering, and MDS show that this out-of-sample embedding formula generalizes well, with a level of error comparable to the effect of small perturbations of the training set on the embedding.
Non-Local Manifold Tangent Learning

We claim and present arguments to the effect that a large class of manifold learning algorithms that are essentially local and can be framed as kernel learning algorithms will suffer from the curse of dimensionality, at the dimension of the true underlying manifold. This observation suggests to explore non-local manifold learning algorithms which attempt to discover shared structure in the tangent planes at different positions. A criterion for such an algorithm is proposed and experiments estimating a tangent plane prediction function are presented, showing its advantages with respect to local manifold learning algorithms: it is able to generalize very far from training data (on learning handwritten character image rotations), where a local non-parametric method fails.
Locally Linear Embedding for dimensionality reduction in QSAR

Current practice in Quantitative Structure Activity Relationship (QSAR) methods usually involves generating a great number of chemical descriptors and then cutting them back with variable selection techniques. Variable selection is an effective method to reduce the dimensionality but may discard some valuable information. This paper introduces Locally Linear Embedding (LLE), a local non-linear dimensionality reduction technique, that can statistically discover a low-dimensional representation of the chemical data. LLE is shown to create more stable representations than other non-linear dimensionality reduction algorithms, and to be capable of capturing non-linearity in chemical data.
Markovian Models for Sequential Data

Hidden Markov Models HMMs are statistical models of sequential data that have been used successfully in many machine learning applications especially for speech recognition Further more in the last few years many new and promising probabilistic models related to HMMs have been proposed We rst summarize the basics of HMMs and then review several recent related learning algorithms and extensions of HMMs including in particular hybrids of HMMs with arti cial neural networks Input Output HMMs which are conditional HMMs using neu ral networks to compute probabilities weighted transducers variable length Markov models and Markov switching state space models Finally we discuss some of the challenges of future research in this very active area
Brain Inspired Reinforcement Learning

Successful application of reinforcement learning algorithms often involves considerable hand-crafting of the necessary non-linear features to reduce the complexity of the value functions and hence to promote convergence of the algorithm. In contrast, the human brain readily and autonomously finds the complex features when provided with sufficient training. Recent work in machine learning and neurophysiology has demonstrated the role of the basal ganglia and the frontal cortex in mammalian reinforcement learning. This paper develops and explores new reinforcement learning algorithms inspired by neurological evidence that provides potential new approaches to the feature construction problem. The algorithms are compared and evaluated on the Acrobot task.
Estimation de densité conditionnelle lorsque l'hypothèse de normalité est insatisfaisante

We aim at modelling fat-tailed densities whose distributions are unknown but are potentially asymmetric. In this context, the standard normality assumption is not appropriate.In order to make as few distributional assumptions as possible, we use a non-parametric algorithm to model the center of the distribution. Density modelling becomes more difficult as we move further in the tail of the distribution since very few observations fall in the upper tail area. Hence we decide to use the generalized Pareto distribution (GPD) to model the tails of the distribution. The GPD can approximate finite, exponential or subexponential tails. The estimation of the parameters of the GPD is based solely on the extreme observations. An observation is defined as being extreme if it is greater than a given threshold. The main difficulty with GPD modelling is to determine an appropriate threshold. Nous cherchons a modeliser des densites dont la distribution est inconnue mais qui est asymetrique et presente des queues lourdes. Dans ce contexte, l'hypothese de normalite n'est pas appropriee. Afin de maintenir au minimum le nombre d'hypotheses distributionnelles, nous utilisons une methode non parametrique pour modeliser le centre de la distribution. La modelisation est plus difficile dans les queues de la distribution puisque peu d'observations s'y trouvent. Nous nous proposons donc d'utiliser la Pareto generalisee (GPD) pour modeliser les queues de la distribution. La GPD permet d'approximer tous les types de queues de distributions (qu'elles soient finies, exponentielles ou sous-exponentielles). L'estimation des parametres de la GPD est uniquement basee sur les observations extremes. Une observation est definie comme etant extreme si elle depasse un seuil donne. La principale difficulte de la modelisation avec la GPD reside dans le choix d'un seuil adequat.
Learning Eigenfunctions of Similarity: Linking Spectral Clustering and Kernel PCA

In this paper, we show a direct equivalence between spectral clustering and kernel PCA, and how both are special cases of a more general learning problem, that of learning the principal eigenfunctions of a kernel, when the functions are from a Hilbert space whose inner product is dened with respect to a density model. This suggests a new approach to unsupervised learning in which abstractions (such as manifolds and clusters) that represent the main features of the data density are extracted. Abstractions discovered at one level can be used to build higher-level abstractions. This paper also discusses how these abstractions can be used to recover a quantitative model of the input density, which is at least useful for evaluative and comparative purposes.
No Unbiased Estimator of the Variance of K-Fold Cross-Validation

Most machine learning researchers perform quantitative experiments to estimate generalization error and compare the performance of different algorithms (in particular, their proposed algorithm). In order to be able to draw statistically convincing conclusions, it is important to estimate the uncertainty of such estimates. This paper studies the very commonly used K-fold cross-validation estimator of generalization performance. The main theorem shows that there exists no universal (valid under all distributions) unbiased estimator of the variance of K-fold cross-validation. The analysis that accompanies this result is based on the eigen-decomposition of the covariance matrix of errors, which has only three different eigenvalues corresponding to three degrees of freedom of the matrix and three components of the total variance. This analysis helps to better understand the nature of the problem and how it can make naive estimators (that don't take into account the error correlations due to the overlap between training and test sets) grossly underestimate variance. This is confirmed by numerical experiments in which the three components of the variance are compared when the difficulty of the learning problem and the number of folds are varied.
Extensions to Metric-Based Model Selection

Metric-based methods have recently been introduced for model selection and regularization, often yielding very significant improvements over the alternatives tried (including cross-validation). All these methods require unlabeled data over which to compare functions and detect gross differences in behavior away from the training points. We introduce three new extensions of the metric model selection methods and apply them to feature selection. The first extension takes advantage of the particular case of time-series data in which the task involves prediction with a horizon h. The idea is to use at t the h unlabeled examples that precede t for model selection. The second extension takes advantage of the different error distributions of cross-validation and the metric methods: cross-validation tends to have a larger variance and is unbiased. A hybrid combining the two model selection methods is rarely beaten by any of the two methods. The third extension deals with the case when unlabeled data is not available at all, using an estimated input density. Experiments are described to study these extensions in the context of capacity control and feature subset selection.
Spectral Clustering and Kernel PCA are Learning Eigenfunctions

In this paper, we show a direct equivalence between spectral clustering and kernel PCA, and how both are special cases of a more general learning problem, that of learning the principal eigenfunctions of a kernel, when the functions are from a Hilbert space whose inner product is defined with respect to a density model. This defines a natural mapping for new data points, for methods that only provided an embedding, such as spectral clustering and Laplacian eigenmaps. The analysis also suggests new approaches to unsupervised learning in which abstractions such as manifolds and clusters that represent the main features of the data density are extracted. Dans cet article, on montre une equivalence directe entre la classification spectrale et l'ACP a noyau, et on montre que les deux sont des cas particuliers d'un probleme plus general, celui d'apprendre les fonctions propres d'un noyau. Ces fonctions fournissent une base pour un espace de Hilbert dont le produit scalaire est defini par rapport a la densite des donnees. Les fonctions propres definissent une transformation de coordonnees naturelles pour de nouveaux points, alors que des methodes comme la classification spectrale et les 'Laplacian eigenmaps' ne fournissaient un systeme de coordonnees que pour les exemples d'apprentissage. Cette analyse suggere aussi de nouvelles approches a l'apprentissage non-supervise dans lesquelles on extrait des abstractions qui resument la densite des donnees, telles que des varietes et des classes naturelles.