A computer-implemented method at a machine learning system comprising: at a processor, receiving, for each of a plurality of options to which resources of the machine learning system may be allocated, a plurality of samples describing behavior of the option, the number of samples based at least in part on an automatically determined error tolerance; for each option, calculating a score using the samples, the score comprising a bounded difference statistic; for each option, calculating a confidence interval of the score using the samples by calculating the score for that option, plus or minus a constant times the square root of, an estimated variance of the score times, a logarithm of two divided by the error tolerance; selecting one or more of the options on the basis of the confidence intervals and the scores, selecting the one or more options on the basis of the confidence intervals including: determining a highest scoring option of the plurality of options; comparing the confidence interval of the highest scoring option with the confidence interval of a different option of the plurality of options; and excluding the different option from further consideration when the confidence interval of the highest scoring option and the confidence interval of the different option do not overlap; and allocating resources of the machine learning system to the selected one or more options. at a processor, receiving, for each of a plurality of options to which resources of the machine learning system may be allocated, a plurality of samples describing behavior of the option, the number of samples based at least in part on an automatically determined error tolerance; for each option, calculating a score using the samples, the score comprising a bounded difference statistic; for each option, calculating a confidence interval of the score using the samples by calculating the score for that option, plus or minus a constant times the square root of, an estimated variance of the score times, a logarithm of two divided by the error tolerance; selecting one or more of the options on the basis of the confidence intervals and the scores, selecting the one or more options on the basis of the confidence intervals including: determining a highest scoring option of the plurality of options; comparing the confidence interval of the highest scoring option with the confidence interval of a different option of the plurality of options; and excluding the different option from further consideration when the confidence interval of the highest scoring option and the confidence interval of the different option do not overlap; and allocating resources of the machine learning system to the selected one or more options. determining a highest scoring option of the plurality of options; comparing the confidence interval of the highest scoring option with the confidence interval of a different option of the plurality of options; and excluding the different option from further consideration when the confidence interval of the highest scoring option and the confidence interval of the different option do not overlap; and allocating resources of the machine learning system to the selected one or more options.