nils-herrmann/clean_scirepeval huggingface dataset:
Quick Training of Probabilistic Neural Nets by Importance Sampling

Our previous work on statistical language modeling introduced the use of probabilistic feedforward neural networks to help dealing with the curse of dimensionality. Training this model by maximum likelihood however requires for each example to perform as many network passes as there are words in the vocabulary. Inspired by the contrastive divergence model, we propose and evaluate sampling-based methods which require network passes only for the observed “positive example” and a few sampled negative example words. A very significant speed-up is obtained with an adaptive importance sampling.
Stochastic Gradient Descent on a Portfolio Management Training Criterion Using the IPA Gradient Estimator

In this paper, we set the basis for learning a multitype assets portfolio management technique relying on no assumptions over the distributions of the financial data. The neural network based model tries to capture patterns in the evolution of the market. Furthermore, the model allows a stochastic perturbation in the asset pricing from the network to avoid local maxima in the decision space. Under those settings, we prove that our investment decision is a Markovian decision process which is Lipschitz continuous almost surely in its parameters. Therefore, the IPA gradient estimator, obtained here by the classical backpropagation algorithm, can be used in a gradient descent procedure to converge to a local maximum of our learning criterion, the Sharpe ratio. Dans cet article, nous jetons les bases pour l'apprentissage d'une strategie de gestion d'un portefeuille de biens, de natures variees, et ne s'appuyant sur aucune supposition quant aux distributions des donnees financieres. Ce modele, base sur l'utilisation d'un reseau de neurones, tente de capturer les tendances du marche. De plus, le modele permet l'introduction d'un bruit stochastique au niveau des prix prevus par le reseau afin d'eviter les maxima locaux dans l'espace de decision. Dans ces conditions, nous demontrons que notre strategie d'investissement suit un processus de decision markovien qui est presque surement lipchitzien en ses parametres. Ainsi, l'estimateur du gradient IPA, obtenu ici par la methode classique de retropropagation, peut etre utilise pour approcher, par une descente de gradient, un maximum local de notre critere d'apprentissage, le Sharpe ratio.
Out-of-Sample Extensions for LLE, Isomap, MDS, Eigenmaps, and Spectral Clustering

Several unsupervised learning algorithms based on an eigendecomposition provide either an embedding or a clustering only for given training points, with no straightforward extension for out-of-sample examples short of recomputing eigenvectors. This paper provides a unified framework for extending Local Linear Embedding (LLE), Isomap, Laplacian Eigenmaps, Multi-Dimensional Scaling (for dimensionality reduction) as well as for Spectral Clustering. This framework is based on seeing these algorithms as learning eigenfunctions of a data-dependent kernel. Numerical experiments show that the generalizations performed have a level of error comparable to the variability of the embedding algorithms due to the choice of training data.
Comment améliorer la capacité de généralisation des algorithmes d'apprentissage pour la prise de décisions financières

This report presents and proposes several methods to improve the capacity of generalization of the learning algorithms in a context of financial decision-making. These methods, overall, aim at controlling the capacity of the learning algorithms in order to limit the problem of the over-training, which is one of most pernicious in finance because of the high levels of noise met in practice. We propose some tracks of research in order to improve the algorithms and results already obtained. Ce rapport presente et propose plusieurs methodes pour ameliorer la capacite de generalisation des algorithmes d'apprentissage dans un contexte de prise de decisions financieres. Globalement, ces methodes visent a controler la capacite des algorithmes d'apprentissage en vue de limiter le probleme du sur-apprentissage, qui est l'un des plus pernicieux en finance a cause des niveaux de bruit eleves rencontres en pratique. Nous proposons quelques pistes de recherches afin d'ameliorer les algorithmes et resultats deja obtenus.
Scaling Large Learning Problems with Hard Parallel Mixtures

A challenge for statistical learning is to deal with large data sets, e.g. in data mining. The training time of ordinary Support Vector Machines is at least quadratic, which raises a serious research challenge if we want to deal with data sets of millions of examples. We propose a "hard parallelizable mixture" methodology which yields significantly reduced training time through modularization and parallelization: the training data is iteratively partitioned by a "gater" model in such a way that it becomes easy to learn an "expert" model separately in each region of the partition. A probabilistic extension and the use of a set of generative models allows representing the gater so that all pieces of the model are locally trained. For SVMs, time complexity appears empirically to local growth linearly with the number of examples, while generalization performance can be enhanced. For the probabilistic version of the algorithm, the iterative algorithm probably goes down in a cost function that is an upper bound on the negative log-likelihood.
Statistical Learning Algorithms Applied to Automobile Insurance Ratemaking

following sections are included:IntroductionConcepts of Statistical Learning TheoryHypothesis Testing: an ExampleParameter Optimization: an ExampleMathematical ObjectivesThe Precision CriterionThe Fairness CriterionMethodologyModelsConstant ModelLinear ModelTable-Based MethodsGreedy Multiplicative ModelGeneralized Linear ModelCHAID Decision TreesCombination of CHAID and Linear ModelOrdinary Neural NetworkHow Can Neural Networks Represent Nonlinear Interactions?Softplus Neural NetworkRegression Support Vector MachineMixture ModelsExperimental ResultsMean-Squared Error ComparisonsEvaluating Model FairnessComparison with Current PremiumsApplication to Risk Sharing Pool FacilitiesConclusionAppendixReferences
Bias learning, knowledge sharing

Biasing properly the hypothesis space of a learner has been shown to improve generalization performance. Methods for achieving this goal have been proposed, that range from designing and introducing a bias into a learner to automatically learning the bias. Multitask learning methods fall into the latter category. When several related tasks derived from the same domain are available, these methods use the domain-related knowledge coded in the training examples of all the tasks as a source of bias. We extend some of the ideas presented in this field and describe a new approach that identifies a family of hypotheses, represented by a manifold in hypothesis space, that embodies domain-related knowledge. This family is learned using training examples sampled from a group of related tasks. Learning models trained on these tasks are only allowed to select hypotheses that belong to the family. We show that the new approach encompasses a large variety of families which can be learned. A statistical analysis on a class of related tasks is performed that shows significantly improved performances when using this approach.
Estimators of Variance for K-Fold Cross-Validation

1 Motivations In machine learning, the standard measure of accuracy for models is the prediction error (PE), i.e. the expected loss on future examples. We consider here the i.i.d. regression or classification setups, where future examples are assumed to be independently sampled from the distribution that generated the training set. When the data distribution is unknown, PE cannot be computed. Therefore, it is estimated to select a particular model within a learning algorithm and to predict its performance. In experiments aiming at comparing learning algorithms, the expected prediction error EPE (where the expectation is taken over training data sets) may be considered as a more relevant criterion . If the amount of data is large enough, PE can be estimated by the mean error over a hold-out test set. The usual variance estimates for means of independent samples can then be computed to derive error bars on the estimated prediction error, and to assess the statistical significance of differences between models. Note however that these procedures do not aim at estimating the variance w.r.t. estimates of EPE. The hold out technique makes an inefficient use of data which forbids its application to small sample sizes. In this situation, one resorts to computer intensive resampling methods such as cross-validation or bootstrap to estimate EPE. We focus on estimating the variance of EPE, the estimate provided by K-fold cross-validation. While it is known that cross-validation provides an unbiased estimate of EPE, it is also known that its variance may be very large . An accurate estimate of variance should provide means to build confidence intervals on EPE, and to derive statistics to test the significance of observed differences between algorithms .
Extracting Hidden Sense Probabilities from Bitexts

We propose a probabilistic model that is inspired by Diab & Resnik’s algorithm to extract disambiguation information from aligned bilingual texts. Like Diab & Resnik’s, the proposed model uses WordNet and the fact that word ambiguities are not always the same in the two languages. The generative model introduces a dependency between two translated words through a common ancestor in WordNet’s ontology.  Diab & Resnik’s algorithm it does not suppose that the translation in the source language has a single meaning.
On Out-of-Sample Statistics for Time-Series

This paper studies an out-of-sample statistic for time-series prediction that is analogous to the widely used R2 in-sample statistic. We propose and study methods to estimate the variance of this out-of-sample statistic. We suggest that the out-of-sample statistic is more robust to distributional and asymptotic assumptions behind many tests for in-sample statistics. Furthermore we argue that it may be more important in some cases to choose a model that generalizes as well as possible rather than choose the parameters that are closest to the true parameters. Comparative experiments are performed on a financial time-series (daily and monthly returns of the TSE300 index). The experiments are performed for varying prediction horizons and we study the relation between predictibility (out-of-sample R2), variability of the out-of-sample R2 statistic, and the prediction horizon. Cet article etudie une statistique hors-echantillon pour la prediction de series temporelles qui est analogue a la tres utilisee statistique R2 de l'ensemble d'entrainement (in-sample). Nous proposons et etudions une methode qui estime la variance de cette statistique hors-echantillon. Nous suggerons que la statistique hors-echantillon est plus robuste aux hypotheses distributionnelles et asymptotiques pour plusieurs tests faits pour les statistiques sur l'ensemble d'entrainement (in-sample). De plus, nous affirmons qu'il peut etre plus important, dans certains cas, de choisir un modele qui generalise le mieux possible plutot que de choisir les parametres qui sont le plus proches des vrais parametres. Des experiences comparatives furent realisees sur des series financieres (rendements journaliers et mensuels de l'indice du TSE300). Les experiences realisees pour plusieurs horizons de predictions, et nous etudions la relation entre la predictibilite (hors-echantillon), la variabilite de la statistique R2 hors-echantillon, et l'horizon de prediction.
Forecasting Non-Stationary Volatility with Hyper-Parameters

We consider sequential data that is sampled from an unknown process, so that the data are not necessarily iid. We develop a measure of generalization for such data and we consider a recently proposed approach to optimizing hyper-parameters, based on the computation of the gradient of a model selection criterion with respect to hyper-parameters. Hyper-parameters are used to give varying weights in the historical data sequence. The approach is successfully applied to modeling the volatility of Canadian stock returns one month ahead. Nous considerons des donnees sequentielles echantillonnees a partir d'un processus inconnu, donc les donnees ne sont pas necessairement iid. Nous developpons une mesure de generalisation pour de telles donnees et nous considerons une approche recemment proposee pour optimiser les hyper-parametres qui est basee sur le calcul du gradient d'un critere de selection de modele par rapport a ces hyper-parametres. Les hyper-parametres sont utilises pour donner differents poids dans la sequence de donnees historiques. Notre approche est appliquee avec succes a la modelisation de la volatilite des rendements d'actions canadiennes sur un horizon de un mois.
On Out-of-Sample Statistics for Financial Time-Series

This paper studies an out-of-sample statistic for time-series prediction that is analogous to the widely used R in-sample statistic. We propose and study methods to estimate the variance of this out-of-sample statistic. We suggest that the out-of-sample statistic is more robust to distributional and asymptotic assumptions behind many tests for insample statistics. Furthermore we argue that it may be more important in some cases to choose a model that generalizes as well as possible rather than choose the parameters that are closest to the true parameters. Comparative experiments are performed on artificial data as well as on a financial time-series (daily and monthly returns of the TSE300 index). The experiments are performed for varying prediction horizons and we study the relation between predictibility (out-of-sample R), variability of the outof-sample R statistic, and the prediction horizon. In particular, we find that very different conclusions would be obtained when testing against the null hypothesis of no dependency rather than testing against the null hypothesis that the proposed model does not generalize better than a naive forecast.
A Parallel Mixture of SVMs for Very Large Scale Problems

Support vector machines (SVMs) are the state-of-the-art models for many classification problems, but they suffer from the complexity of their training algorithm, which is at least quadratic with respect to the number of examples. Hence, it is hopeless to try to solve real-life problems having more than a few hundred thousand examples with SVMs. This article proposes a new mixture of SVMs that can be easily implemented in parallel and where each SVM is trained on a small subset of the whole data set. Experiments on a large benchmark data set (Forest) yielded significant time improvement (time complexity appears empirically to locally grow linearly with the number of examples). In addition, and surprisingly, a significant improvement in generalization was observed.
Étude du biais dans le prix des options

The price of an option should reflect the average value that a buyer receives for it, and also a risk premium. This report describes an empirical study for analysing these factors as a graphical and quantitative manner. The analysis focuses on the average difference between the price option and its present average value at maturity (the bias), and tries to detect some temporal regularities in the pattern of this bias. We found some very surprising almost-periodic patterns for the bias, in particular for the long-time maturities (not so clearly for the puts), as studied by spectral analysis. Le prix d'une option devrait refleter la valeur moyenne que l'acheteur en recoit ainsi qu'une prime de risque. Ce rapport decrit une etude empirique pour analyser ces facteurs de maniere graphique et quantitative. L'analyse se concentre sur la difference moyenne entre le prix de l'option et sa valeur actualisee moyenne a maturite (le "biais""), et tente de cerner des regularites temporelles dans les patrons de cette difference. On y decouvre de surprenants patrons quasi-periodiques de ces variations, en particulier pour les calls de maturite elevee (moins clairement pour les puts), qui sont etudies avec une analyse spectrale."
Manifold Parzen Windows

The similarity between objects is a fundamental element of many learning algorithms. Most non-parametric methods take this similarity to be fixed, but much recent work has shown the advantages of learning it, in particular to exploit the local invariances in the data or to capture the possibly non-linear manifold on which most of the data lies. We propose a new non-parametric kernel density estimation method which captures the local structure of an underlying manifold through the leading eigenvectors of regularized local covariance matrices. Experiments in density estimation show significant improvements with respect to Parzen density estimators. The density estimators can also be used within Bayes classifiers, yielding classification rates similar to SVMs and much superior to the Parzen classifier.
Metric-based model selection for time-series forecasting

Metric-based methods, which use unlabeled data to detect gross differences in behavior away from the training points, have recently been introduced for model selection, often yielding very significant improvements over alternatives (including cross-validation). We introduce extensions that take advantage of the particular case of time-series data in which the task involves prediction with a horizon h. The ideas are: (i) to use at t the h unlabeled examples that precede t for model selection, and (ii) take advantage of the different error distributions of cross-validation and the metric methods. Experimental results establish the effectiveness of these extensions in the context of feature subset selection.
Valorisation d'Options par Optimisation du Sharpe Ratio

Prior work on option pricing falls mostly in two categories: it either relies on strong distributional or economical assumptions, or it tries to mimic the Black-Scholes formula through statistical models, trained to fit today's market price based on information available today. The work presented here is closer to the second category but its objective is different: predict the future value of the option, and establish its current value based on a trading scenario. This work thus innovates in two ways: first it proposes an empirical and hypothesis-free method to compare different option pricing systems (by having trade against each other or against the market), second it uses this criterion to train a non-parametric statistical model (here based on neural networks) to estimate a price for the option that maximizes the expected utility when trading against the market. Note that the price will depend on the utility function and current portfolio (i.e. current risks) of the trading agent. Preliminary experiments are presented on the S&P 500 options. Les travaux precedents sur la valorisation des options entraient en gros dans deux categories : ou bien ils etaient bases sur de fortes hypotheses distributionnelles ou economiques, ou bien ils essayaient d'imiter la formule de Black-Scholes par des modeles statistiques entraines a approximer les prix de marche quotidiens a l'aide d'information disponible le jour meme. Le travail presente ici se rapproche plus de la deuxieme categorie mais son objectif est different : predire les prix futurs d'une option, et etablir sa valeur courante a l'aide d'un scenario de transactions. Ce travail innove donc de deux facons : premierement, il propose une methode empirique et sans hypothese pour comparer differents systemes de valorisation d'options (en transigeant contre lui-meme ou contre le marche) et deuxiemement, il utilise ce critere pour entrainer un modele statistique non-parametrique (utilisant dans ce cas-ci des reseaux de neurones) pour estimer un prix pour l'option qui maximise l'utilite esperee lorsque l'on transige contre le marche. A noter que les prix dependront de la fonction d'utilite ainsi que du portefeuille (i.e. des risques courants) de la personne qui transige. Des resultats preliminaires sur des options d'achat du S&P 500 sont presentes.
Robust Regression with Asymmetric Heavy-Tail Noise Distributions

In the presence of a heavy-tail noise distribution, regression becomes much more difficult. Traditional robust regression methods assume that the noise distribution is symmetric, and they downweight the influence of so-called outliers. When the noise distribution is asymmetric, these methods yield biased regression estimators. Motivated by data-mining problems for the insurance industry, we propose a new approach to robust regression tailored to deal with asymmetric noise distribution. The main idea is to learn most of the parameters of the model using conditional quantile estimators (which are biased but robust estimators of the regression) and to learn a few remaining parameters to combine and correct these estimators, to minimize the average squared error in an unbiased way. Theoretical analysis and experiments show the clear advantages of the approach. Results are on artificial data as well as insurance data, using both linear and neural network predictors.
The Challenge of Non-Linear Regression on Large Datasets with Asymmetric Heavy Tails

Regression becomes unstable under a heavy-tail error distribution due to dominant effects ofutliers. Traditional robust estimators are helpful under symmetricerror, by reducing the effect of outliers equally from both sides of the distribution. Under asymmetricerror, however, those estimators are biasedbecause the outliers appear only one side of the distribution. Motivated by datamining problems for the insurance industry, we propose in this paper a new approach to robust regression that is tailored to deal with asymmetric error. The main idea is to estimate a majority of the parameters using quantile regression(which may be biased but robust), and to estimate the few remaining using least squares estimator to correct the biases. Theoretical analysis shows conditions when the conditional expectations can be recovered from the conditional quantiles and what can be gained asymptotically with the proposed algorithm. Experiments confirm the clear advantages of the approach. Results are on synthetic data as well as real insurance data, using both linear and neural-network predictors.
Scaling Large Learning Problems with Hard Parallel Mixtures

A challenge for statistical learning is to deal with large data sets, e.g. in data mining. Popular learning algorithms such as Support Vector Machines have training time at least quadratic in the number of examples: they are hopeless to solve problems with a million examples. We propose a "hard parallelizable mixture" methodology which yields significantly reduced training time through modularization and parallelization: the training data is iteratively partitioned by a "gater" model in such a way that it becomes easy to learn an "expert" model separately in each region of the partition. Ap robabilistic extension and the use of a set of generative models allows representing the gater so that all pieces of the model are locally trained. For SVMs, time complexity appears empirically to locally grow linearly with the number of examples, while generalization performance can be enhanced. For the probabilistic version of the algorithm, the iterative algorithm provably goes down in a cost function that is an upper bound on the negative log-likelihood.