Parametric and Non-Parametric Confidence Interval Estimation for Machine Learning in 3 Lines of Code | by Giovanni Ciampi | Jan, 2021 | Medium | Towards Data Science

[Edit 11/01/2021] Added better definition and adjusted the example in section 1 according to helpful suggestion by AlexMurphy.
Confidence intervals are a great tool to provide useful information when reporting results of Machine Learning models. They are an easy and straightforward way of assessing, among other things, the reliability of a model.
In this post, I am going to give a brief introduction to confidence intervals and to present confidence_interval_estimator_ML, a tool that will allow you to estimate confidence intervals both parametrically and non-parametrically for your machine learning models in few lines of code.
Here is how this post is structured:
Introduction to Confidence Intervals with Examples.Hidden Assumptions About Confidence Intervals.What are Parametric Confidence Intervals.What are Non-Parametric Confidence Intervals.How to Use the confidence_interval_estimator_ML to Compute Parametric and Non-Parametric Confidence Intervals.
Introduction to Confidence Intervals with Examples.
Hidden Assumptions About Confidence Intervals.
What are Parametric Confidence Intervals.
What are Non-Parametric Confidence Intervals.
How to Use the confidence_interval_estimator_ML to Compute Parametric and Non-Parametric Confidence Intervals.
Paraphrasing Wikipedia, confidence intervals indicate a range of plausible values for an unknown parameter p, with an associated degree of confidence indicating the degree of belief that the true p is contained that range.
In the context of machine learning, confidence intervals are often estimated with respect to the performances of a model, and in particular with respect to the accuracy of a classification model.
Given an experiment that aims at measuring a parameter p (in our case the accuracy of a model) a confidence interval I with an associated degree of confidence C% can be interpreted as follows: if the experiment is repeated N times, in C% of the outcomes p will be in the interval I.
Although it might be tempting to do so, the degree of confidence cannot be technically interpreted as a probability, as discussed in the next section.
Taking as an example the KERMIT paper presented at EMNLP2020, the authors state that their model has an accuracy of 85.91(±0.03) % on the Yelp Polarity dataset with a 95% degree of confidence.
From this result, by re-running the model training and evaluation from scratch on the Yelp Polarity dataset 100 times (and changing the random seeds every time), we can expect that 95 times the accuracy will be in the range [85.88, 85.94]. The fact that the degree of confidence is high (95%) and the interval is relatively narrow indicates that in the given conditions, the KERMIT model is pretty robust!
When computing statistics there always are some assumptions that tend to be hidden but can have a great deal of importance on the outcome. In the case of confidence intervals, in particular, in the context of machine learning, there are two main assumptions that, in my opinion, can be of crucial importance: the normality of data and the interpretation of the confidence degree.
As regards the normality of data, it is assumed that the data sample was drawn from a normal distribution, and that the variance is therefore independent of the mean. This assumption can be problematic as the distribution of the data is often unknown.
Now suppose that you want to estimate the confidence interval for the accuracy of model M on dataset D, you not only have to know the distribution of D, but it also has to be that this distribution is a Gaussian. Luckily, the Central Limit Theorem comes in handy, establishing that if the sample size is large enough, the data distribution tends to a normal distribution. Here the definition of large is dependent on several factors, but 30 is generally considered a good threshold for the size of the sample of data to be used for the evaluation of the model. In general though, as often in data science, the more data, the better.
As regards the interpretation of the confidence degree, it is often implicitly assumed that the confidence degree represents the probability that a the true value p will fall in the confidence interval I. This is actually wrong, as the confidence degree refers to the reliability of the estimation procedure, as extensively discussed in this Wikipedia section. Anyway, from a practical standpoint, the confidence interval may be understood as an estimate of plausible values for the population parameter.
Parametric estimation is a way to estimate the confidence intervals in closed form, from even a single result of your model. The formula to compute confidence intervals through parametric estimation is the following:
where A is the accuracy of your model, N indicates the size of the sample on which A was computed and factor is a constant determined by the confidence degree and sample size. The aim of factor is to determine how wide the interval is. Generally speaking, a higher degree of confidence will lead to wider intervals. The source of this constant depends upon the sample size, and will be specified shortly.
The value R represents the range of the interval, the interval will be therefore represented by I = [A-R, B+R].
As mentioned in point 2, parametric estimation assumes that the data sample was drawn from a normal distribution. Even though you don’t know the distribution of your data, if the sample size is bigger than 30 this is not a problem thanks to the central limit theorem. In this case, the factor parameter will be the z-score associated with the desired confidence degree.
If your sample size is smaller than 30, then it is common practice to assume that the data was drawn from a t-student distribution with sample_size-1 degrees of freedom. Without going into too much detail, this assumption leads to wider confidence intervals, as the small sample size provides additional uncertainty. In this case, the factor parameter will be the t-score associated with the t-student distribution having N-1 degrees of freedom and the specified confidence degree.
Non-Parametric confidence interval estimation is a way of estimating the confidence interval empirically. This method goes under several other names: Bootstrap Confidence Interval Estimation, Monte Carlo Confidence Interval Estimation, or simply Empirical Confidence Interval Estimation, but they all indicate the same thing.
To estimate non-parametric confidence intervals for the accuracy of model M on dataset D, the model is trained and tested multiple times on samples of D. This is the algorithm to perform the estimation:
The dataset D is sampled N times to obtain N datasets Di. This sampling is usually performed with replacement.Each sample Di is split in two separate sets Di_train and Di_test.For each Di, an instance Mi of the model is generated. Every instance Mi is evaluated on the corresponding data sample by performing training on Di_train and testing on Di_test. Let’s denote the list of test accuracies by A.The list of test accuracies A is sorted.Let’s denote the desired confidence-level by C; if we want the c.i. for 95% degree of confidence, C will be set to 95. Let’s denote by alpha the value 100-C. If the desired confidence level is C=95, alpha=100–95=5.The lower bound of the confidence interval is given by the alpha/2-th percentile of A, while the upper bound of the confidence interval is given by the C+(alpha/2)-th percentile.We can say that the confidence interval of model M on dataset D is [lower_bound, upper_bound] with a confidence degree of C%.
The dataset D is sampled N times to obtain N datasets Di. This sampling is usually performed with replacement.
Each sample Di is split in two separate sets Di_train and Di_test.
For each Di, an instance Mi of the model is generated. Every instance Mi is evaluated on the corresponding data sample by performing training on Di_train and testing on Di_test. Let’s denote the list of test accuracies by A.
The list of test accuracies A is sorted.
Let’s denote the desired confidence-level by C; if we want the c.i. for 95% degree of confidence, C will be set to 95. Let’s denote by alpha the value 100-C. If the desired confidence level is C=95, alpha=100–95=5.
The lower bound of the confidence interval is given by the alpha/2-th percentile of A, while the upper bound of the confidence interval is given by the C+(alpha/2)-th percentile.
We can say that the confidence interval of model M on dataset D is [lower_bound, upper_bound] with a confidence degree of C%.
Intuitively, we want to take lower_bound and upper_bound so that they cover C% of the elements of A, as it is shown in the following illustration.
This way of computing confidence intervals challenges both of the assumptions we discussed in point 2 of this post. That is why, when computing confidence intervals non parametrically, it is of crucial importance to have both a large dataset and to use a large number of iterations to perform the estimate.
At this point you should have a clear idea of how parametric and non-parametric estimation of confidence intervals work, so I will introduce confidence_interval_estimator_ML, a Python tool that will make the estimation process very simple and straightforward.
It is easy to install this tool through pip in your command line:
> pip install confidence_interval_estimator_ML
Now it is pretty straightforward to perform the estimation: for the parametric case, you just need to import the library and run one line of code, as shown in the following example:
In the presented example, the sample size used is bigger than 30, that is why the function specifies that it is assumed that the data is sampled from a normal distribution.
The non-parametric case is slightly more complex, as you may need to specify a number of parameters for the estimation you intend to perform. Anyway, this is accomplished within two lines of code:
The first line performs the first three steps of the algorithm presented at point 4 (it trains and validates instances of the model for a specified number of iterations), while the second line performs the remaining four points (computes the bounds of the confidence interval).
The first function, get_accuracy_samples takes several input parameters:
get_classifier_instance is a function that returns a new instance of the model by calling it. The returned model must provide fit and predict methods. A new instance of the model will be created at each iteration.
model_params_dict is an optional parameter that allows you to specify parameters to be passed to the model constructor.
X and y represent respectively the input and target of the model.
sample_ratio indicates the size of the sample to be drawn at each iteration from (X, y). As an example, if (X, y) has 1000 data-points and sample_ratio is set to 0.70, at each iteration a dataset of 700 data-points will be sampled from (X, y).
train_ratio specifies what ratio of the sampled dataset will be used for the training and testing of the model at each iteration. Following the example of the previous point, if (X, y) has 1000 data-points and sample_ratio=0.70, then at each iteration a dataset of 700 data-points will be sampled. If train_ratio=0.80, 560 data-points will be used to train the model while the remaining 140 will be used for the evaluation (to compute the test accuracy).
The following code provides a more detailed example of the usage of the module.
These methods provide other parameters and use cases that are not discussed here for the sake of brevity. If you want to explore all the possibilities offered by the confidence_interval_estimator_ML module, or you simply need more details to perform your estimates, you can check this Google Colab notebook, or simply the GitHub repo providing the code and the documentation.
https://ocw.mit.edu/courses/mathematics/18-05-introduction-to-probability-and-statistics-spring-2014/readings/MIT18_05S14_Reading24.pdfhttps://sphweb.bumc.bu.edu/otlt/MPH-Modules/PH717-QuantCore/PH717-Module6-RandomError/PH717-Module6-RandomError11.htmlhttps://machinelearningmastery.com/report-classifier-performance-confidence-intervals/https://machinelearningmastery.com/confidence-intervals-for-machine-learning/#:~:text=Much%20of%20machine%20learning%20involves,the%20uncertainty%20of%20an%20estimate.&text=That%20a%20confidence%20interval%20is,estimate%20of%20a%20population%20parameter.http://www2.stat.duke.edu/~ar182/rr/examples-gallery/BootstrapConfidenceIntervals.html
https://ocw.mit.edu/courses/mathematics/18-05-introduction-to-probability-and-statistics-spring-2014/readings/MIT18_05S14_Reading24.pdf
https://sphweb.bumc.bu.edu/otlt/MPH-Modules/PH717-QuantCore/PH717-Module6-RandomError/PH717-Module6-RandomError11.html
https://machinelearningmastery.com/report-classifier-performance-confidence-intervals/
https://machinelearningmastery.com/confidence-intervals-for-machine-learning/#:~:text=Much%20of%20machine%20learning%20involves,the%20uncertainty%20of%20an%20estimate.&text=That%20a%20confidence%20interval%20is,estimate%20of%20a%20population%20parameter.
http://www2.stat.duke.edu/~ar182/rr/examples-gallery/BootstrapConfidenceIntervals.html
If you liked this article, please don’t hesitate to connect on Linkedin!