Package: macest
MACEst (Model Agnostic Confidence Estimator)What is MACEst?MACEst is a confidence estimator that can be used alongside any model (regression or
classification) which uses previously seen data (i.e. any supervised learning model) to produce a
point prediction.In the regression case, MACEst produces aconfidence intervalabout the point prediction, e.g.
"the point prediction is 10 and I am 90% confident that the prediction lies between 8 and 12."In Classification MACEst produces aconfidence scorefor the point prediction. e.g. the point
prediction is class 0 and I am 90% sure that the prediction is correct.MACEst produces well-calibrated confidence estimates, i.e. 90% confidence means that you will on
average be correct 90% of the time.
It is also aware of the model limitations i.e. when a model is being asked to predict a point which
it does not have the necessary knowledge (data) to predict confidently.
In these cases MACEst is able to incorporate the (epistemic) uncertainty due to this and return a
very low confidence prediction (in regression this means a large prediction interval).Why use MACEst ?Machine learning has become an integral part of many of the tools that are used every day.
There has been a huge amount of progress on improving the global accuracy of machine learning
models but calculating how likely a single prediction is to be correct has seen considerably less
progress.Most algorithms will still produce a prediction, even if this is in a part of the feature space the
algorithm has no information about.
This could be because the feature vector is unlike anything seen during training, or because the
feature vector falls in a part of the feature space where there is a large amount of uncertainty
such as if the border between two classes overlaps.
In cases like this the prediction may well be meaningless.
In most models, it is impossible to distinguish this sort of meaningless prediction from a sensible
prediction.
MACEst addresses this situation by providing an additional confidence estimate.In some areas such as Finance, Infrastructure, or Healthcare, making a single bad prediction can
have major consequences.
It is important in these situations that a model is able to understand how likely any prediction it
makes is to be correct before acting upon it.
It is often even more important in these situations that any modelknows what it doesn't knowso
that it will not blindly make bad predictions.Summary of the MethodologyTL;DRMACEst produces confidence estimates for a given point x by considering two factors:How accurate is the model when predicting previously seen points that aresimilarto x?
Less confident if the model is less accurate in the region close to x.Howsimilaris x to the points that we have seen previously?
Less confident if x is notsimilarto the data used to train the model.Longer ExplanationMACEst seeks to provide reliable confidence estimates for both regression and classification.
It draws from ideas present in trust scores, conformal learning, Gaussian processes, and Bayesian
modelling.The general idea is that confidence is a local quantity.
Even when the model is accurate globally, there are likely still some predictions about which it
should not be very confident.
Similarly, if the model is not accurate globally, there may still be some predictions for which the
model can be very confident about.To model this local confidence for a given prediction on a point x, we define the local
neighbourhood by finding the k nearest neighbours to x.
We then attempt to directly model the two causes of uncertainty, these are:Aleatoric Uncertainty: Even with lots of (possibly infinite) data there will be some
variance/noise in the predictions.
Our local approximation to this will be to define a local accuracy estimate. i.e. for the k nearest
neighbours how accurate were the predictions?Epistemic Uncertainty: The model can only know relationships learnt from the training data.
If the model has not seen any data point similar to x then it does not have as much knowledge about
points like x, therefore the confidence estimate should be lower.
MACEst estimates this by calculating howsimilarx is to the k nearest (most similar) points
that it has previously seen.We define a simple parametric function of these two quantities and calibrate this function so that
our confidence estimates approximate the empirical accuracy, i.e. 90% confident -> 90% correct on
average.
By directly modelling these two effects, MACEst estimates are able to encapsulate the local
variance accurately whilst also being aware of when the model is being asked to predict a point
that is very different to what it has been trained on.
This will make it robust to problems such as overconfident extrapolations and out of sample
predictions.ExampleIf a model has been trained to classify images of cats and dogs, and we want to predict an image of
a poodle, we find the k most poodle-like cats and the k most poodle-like dogs.
We then calculate how accurate the model was on these sets of images, and how similar the poodle is
to each of these k cats and k dogs. We combine these two to produce a confidence estimate for each
class.As the poodle-like cats will likely be strange cats, they will be harder to classify and the
accuracy will be lower for these than the poodle-like dogs this combined with the fact that image
will be considerably more similar to poodle-like dogs the confidence of the dog prediction will be
high.If we now try to classify an image of a horse, we find that the new image is verydissimilarto
both cats and dogs, so the similarity term dominates and the model will return an approximately
uniform distribution, this can be interpreted as MACEst saying "I don't know what this is because
I've never seen an image of a horse!".Getting StartedTo install MACEst run the following cmd:pipinstallmacestOr addmacestto your project'srequirements.txtfile as a dependency.Software PrerequisitesTo import and use MACEst we recommend Python version >=3.6.8.Basic UsageBelow shows examples of using MACEst for classification and regression.
For more examples, and advanced usage, please see the examplenotebooks.ClassificationTo use MACEst for a classification task, the following example can be used:importnumpyasnpfrommacest.classificationimportmodelsascl_modfromsklearn.ensembleimportRandomForestClassifierfromsklearnimportdatasetsfromsklearn.model_selectionimporttrain_test_splitX,y=datasets.make_circles(n_samples=2*10**4,noise=0.4,factor=0.001)X_pp_train,X_conf_train,y_pp_train,y_conf_train=train_test_split(X,y,test_size=0.66,random_state=10)X_conf_train,X_cal,y_conf_train,y_cal=train_test_split(X_conf_train,y_conf_train,test_size=0.5,random_state=0)X_cal,X_test,y_cal,y_test,=train_test_split(X_cal,y_cal,test_size=0.5,random_state=0)point_pred_model=RandomForestClassifier(random_state=0,n_estimators=800,n_jobs=-1)point_pred_model.fit(X_pp_train,y_pp_train)macest_model=cl_mod.ModelWithConfidence(point_pred_model,X_conf_train,y_conf_train)macest_model.fit(X_cal,y_cal)conf_preds=macest_model.predict_confidence_of_point_prediction(X_test)RegressionTo use MACEst for a regression task, the following example can be used:importnumpyasnpfrommacest.regressionimportmodelsasreg_modfromsklearn.linear_modelimportLinearRegressionfromsklearn.model_selectionimporttrain_test_splitX=np.linspace(0,1,10**3)y=np.zeros(10**3)y=2*X*np.sin(2*X)**2+np.random.normal(0,1,len(X))X_pp_train,X_conf_train,y_pp_train,y_conf_train=train_test_split(X,y,test_size=0.66,random_state=0)X_conf_train,X_cal,y_conf_train,y_cal=train_test_split(X_conf_train,y_conf_train,test_size=0.5,random_state=1)X_cal,X_test,y_cal,y_test,=train_test_split(X_cal,y_cal,test_size=0.5,random_state=1)point_pred_model=LinearRegression()point_pred_model.fit(X_pp_train[:,None],y_pp_train)preds=point_pred_model.predict(X_conf_train[:,None])test_error=abs(preds-y_conf_train)y_conf_train_var=np.var(train_error)macest_model=reg_mod.ModelWithPredictionInterval(point_pred_model,X_conf_train[:,None],test_error)macest_model.fit(X_cal[:,None],y_cal)conf_preds=confidence_model.predict_interval(X_test,conf_level=90)MACEst with sparse data (see notebooks for more details)importscipyfromscipy.sparseimportcsr_matrixfromscipy.sparseimportrandomassp_randfromsklearn.model_selectionimporttrain_test_splitfromsklearn.ensembleimportRandomForestClassifierfrommacest.classificationimportmodelsasclmodimportnmslibn_rows=10**3n_cols=5*10**3X=csr_matrix(sp_rand(n_rows,n_cols))y=np.random.randint(0,2,n_rows)X_pp_train,X_conf_train,y_pp_train,y_conf_train=train_test_split(X,y,test_size=0.66,random_state=10)X_conf_train,X_cal,y_conf_train,y_cal=train_test_split(X_conf_train,y_conf_train,test_size=0.5,random_state=0)X_cal,X_test,y_cal,y_test,=train_test_split(X_cal,y_cal,test_size=0.5,random_state=0)model=RandomForestClassifier(random_state=0,n_estimators=800,n_jobs=-1)model.fit(csr_matrix(X_pp_train),y_pp_train)param_bounds=clmod.SearchBounds(alpha_bounds=(0,500),k_bounds=(5,15))neighbour_search_params=clmod.HnswGraphArgs(query_args=dict(ef=1100),init_args=dict(method="hnsw",space="cosinesimil_sparse",data_type=nmslib.DataType.SPARSE_VECTOR))macest_model=clmod.ModelWithConfidence(model,X_conf_train,y_conf_train,search_method_args=neighbour_search_params)macest_model.fit(X_cal,y_cal)macest_point_prediction_conf=macest_model.predict_confidence_of_point_prediction(X_test)ContributingSee theCONTRIBUTING.mdfile for information about contributing to MACEst.LicenseCopyright (c) 2021, Oracle and/or its affiliates. All rights reserved.This library is licensed under Universal Permissive License (UPL) 1.0 as shown athttps://oss.oracle.com/licenses/uplSeeLICENSE.txtfor more details.