text: 344,253,1367839174469029899,370409954,Gael Varoquaux,"New preprint: Accounting for Variance in Machine Learning Benchmarks <LINK> Lead by @bouthilx and @Mila_Quebec friends We show that ML benchmarks contain multiple sources of uncontrolled variation, not only inits. We propose procedure for reliable conclusion 1/8 Data split and hyper-parameter selection (even with fancy hyper-parameter optimization) appear as the leading source of arbitrary variations in ML benchmarks, beyond random weight init. These must be sampled to give empirical evidence on algorithm comparison that generalize 2/8 <LINK> Even in deep-learning benchmarks, performed on large datasets, the variance of the observed performance is limited by the set of the test set 3/8 <LINK> Altogether, these variances are not small compared to observed improvements in the literature. Hence, there is a risk that published findings may be due to chance, for instance finding better hyper-parameters from one algorithm than another 4/8 <LINK> Sampling the variance of hyper-parameter tuning is very costly, eg in deep learning. We measure it in clean (and costly) experiments, but also study imperfect estimators. We show that it is best to sample all sources of variation to minimize error on estimation of performance 5/8 <LINK> Variance of performance results must be accounted for to conclude on whether or not there is evidence that an algorithm is an improvement. To avoid accepting trivial differences, we use to use ""non-inferiority"", or Neyman-Pearson, tests, used in clinical trials 6/8 <LINK> Our recommendations (based on 5 case studies, deep learning & classical ML): • Randomize as many sources of variations as possible (weight inits, data order, data splitting) • Multiple validation splits • P(A &gt; B) &gt; 0.75 (comparing improvement to variance) 7/8 This work relies on extensive experiments, with multiple datasets and ML pipelines, including many hyper-parameter optimization of deep learning pipelines. Thanks to an amazing teams, comprising @bouthilx @AssyaTrofimov @EdwardRaffML and many more <LINK> 8/8",https://arxiv.org/abs/2103.03098,"Strong empirical evidence that one machine-learning algorithm A outperforms another one B ideally calls for multiple trials optimizing the learning pipeline over sources of variation such as data sampling, data augmentation, parameter initialization, and hyperparameters choices. This is prohibitively expensive, and corners are cut to reach conclusions. We model the whole benchmarking process, revealing that variance due to data sampling, parameter initialization and hyperparameter choice impact markedly the results. We analyze the predominant comparison methods used today in the light of this variance. We show a counter-intuitive result that adding more sources of variation to an imperfect estimator approaches better the ideal estimator at a 51 times reduction in compute cost. Building on these results, we study the error rate of detecting improvements, on five different deep-learning tasks/architectures. This study leads us to propose recommendations for performance comparisons. ",Accounting for Variance in Machine Learning Benchmarks,8,"['New preprint: Accounting for Variance in Machine Learning Benchmarks\n<LINK>\nLead by @bouthilx and @Mila_Quebec friends\n\nWe show that ML benchmarks contain multiple sources of uncontrolled variation, not only inits. We propose procedure for reliable conclusion 1/8', 'Data split and hyper-parameter selection (even with fancy hyper-parameter optimization) appear as the leading source of arbitrary variations in ML benchmarks, beyond random weight init.\n\nThese must be sampled to give empirical evidence on algorithm comparison that generalize 2/8 https://t.co/kE8VemNrPY', 'Even in deep-learning benchmarks, performed on large datasets, the variance of the observed performance is limited by the set of the test set 3/8 https://t.co/PAvhOpJ86O', 'Altogether, these variances are not small compared to observed improvements in the literature. Hence, there is a risk that published findings may be due to chance, for instance finding better hyper-parameters from one algorithm than another 4/8 https://t.co/qwCdfCmPj5', 'Sampling the variance of hyper-parameter tuning is very costly, eg in deep learning. We measure it in clean (and costly) experiments, but also study imperfect estimators.\nWe show that it is best to sample all sources of variation to minimize error on estimation of performance 5/8 https://t.co/uOsyCzM8C9', 'Variance of performance results must be accounted for to conclude on whether or not there is evidence that an algorithm is an improvement.\n\nTo avoid accepting trivial differences, we use to use ""non-inferiority"", or Neyman-Pearson, tests, used in clinical trials 6/8 https://t.co/5CsC13AsRb', 'Our recommendations (based on 5 case studies, deep learning &amp; classical ML):\n• Randomize  as  many  sources  of  variations  as  possible (weight inits, data order, data splitting)\n• Multiple validation splits\n• P(A &gt; B) &gt; 0.75 (comparing improvement to variance)\n\n7/8', 'This work relies on extensive experiments, with multiple datasets and ML pipelines, including many hyper-parameter optimization of deep learning pipelines.\n\nThanks to an amazing teams, comprising @bouthilx @AssyaTrofimov @EdwardRaffML and many more\n\nhttps://t.co/StDk21LCTI\n8/8']",21,03,2040