Package: uaml
Uncertainty-aware machine learningPython package for uncertainty-aware classification built on top of Scikit-learn.Descriptionuamlis a Python package for uncertainty-aware machine learning based on probabilistic ensembles and the Jensen–Shannon divergence. Currently, it is built on top of Scikit-learn and supports all probabilistic base classifiers.InstallationClone this repositorytfmortie/uamland runpip install . -r requirements.txtor install by means ofpip install uaml.ExampleThe uncertainty-aware classifier is provided throughuaml.multiclass.UAClassifier. Below we show a minimal working and more elaborate example.Basic usageWe start by importing some packages that we will need throughout the example:fromsklearn.datasetsimportmake_moonsfromsklearn.linear_modelimportLogisticRegressionfromsklearn.model_selectionimporttrain_test_split# some example dataX,y=make_moons(n_samples=100,noise=0.1,random_state=0)X_train,X_test,y_train,y_test=train_test_split(X,y,test_size=.4,random_state=42)Creating an uncertainty-aware classifier, withLogisticRegressionas underlying probabilistic model, is done as follows:fromuaml.multiclassimportUAClassifier# use LogisticRegression as base (probabilistic) estimatorest=LogisticRegression(solver="liblinear")# construct and fit an uncertainty-aware classifier with 500 estimators and parallelize over 5 coresclf=UAClassifier(est,ensemble_size=500,train_ratio=0.5,n_jobs=5)UAClassifierfollows the Scikit-learn API, as illustrated below:# fit our classifierclf.fit(X_train,y_train)# obtain predictions by means of majority votingpreds=clf.predict(X_test,avg=True)# obtain probabilitiesprobs=clf.predict_proba(X_test,avg=True)Finally, let's calculate aleatoric and epistemic uncertainty:ua,ue=clf.get_uncertainty(X_test)VisualisationIn a next example, let's see how aleatoric and epistemic uncertainty evaluate in the feature space of the "two moons" dataset for different classifiers:importnumpyasnpimportmatplotlibmatplotlib.use('Agg')importmatplotlib.pyplotaspltfromuaml.multiclassimportUAClassifierfromsklearn.model_selectionimporttrain_test_splitfromsklearn.preprocessingimportStandardScalerfromsklearn.datasetsimportmake_moonsfromsklearn.neural_networkimportMLPClassifierfromsklearn.neighborsimportKNeighborsClassifierfromsklearn.svmimportSVCfromsklearn.treeimportDecisionTreeClassifierfromsklearn.discriminant_analysisimportQuadraticDiscriminantAnalysisfromsklearn.datasetsimportmake_moonsfromsklearn.treeimportDecisionTreeClassifier# different estimators for UAClassifierclassifiers={"5-NN":KNeighborsClassifier(5),"Linear SVM":SVC(kernel="linear",C=0.025,probability=True),"RBF SVM":SVC(gamma=1,C=1,probability=True),"Decision Tree":DecisionTreeClassifier(max_depth=5),"Simple Neural Network":MLPClassifier(alpha=1,max_iter=1000),"QDA":QuadraticDiscriminantAnalysis()}# create datasetX,y=make_moons(n_samples=100,noise=0.1,random_state=0)X=StandardScaler().fit_transform(X)X_train,X_test,y_train,y_test=\train_test_split(X,y,test_size=.4,random_state=42)x_min,x_max=X[:,0].min()-1,X[:,0].max()+1y_min,y_max=X[:,1].min()-1,X[:,1].max()+1xx,yy=np.meshgrid(np.arange(x_min,x_max,0.02),np.arange(y_min,y_max,0.02))# create plotcm=plt.cm.viridisfig,ax=plt.subplots(len(classifiers),3,figsize=(10,10))fori,clfinenumerate(classifiers.keys()):# fit classifiers and obtain predictions and uncertainty estimatesmodel=classifiers[clf]clf=UAClassifier(model,500,0.8,n_jobs=5,verbose=1)clf.fit(X_train,y_train)Zp=clf.predict(np.c_[xx.ravel(),yy.ravel()],avg=True)Za,Ze=clf.get_uncertainty(np.c_[xx.ravel(),yy.ravel()])# construct contour plotZp=Zp.reshape(xx.shape)Za=Za.reshape(xx.shape)Ze=Ze.reshape(xx.shape)ax[i,0].contourf(xx,yy,Zp,cmap=cm,alpha=.8)ifi==0:ax[i,0].set_title("Prediction")# prediction plot# plot the training pointsax[i,0].scatter(X_train[:,0],X_train[:,1],c=y_train,cmap=cm)# plot the testing pointsax[i,0].scatter(X_test[:,0],X_test[:,1],c=y_test,cmap=cm,alpha=0.6)ax[i,0].set_xlim(xx.min(),xx.max())ax[i,0].set_ylim(yy.min(),yy.max())# aleatoric uncertainty plotax[i,1].contourf(xx,yy,Za,cmap=cm,alpha=.8)ifi==0:ax[i,1].set_title("Aleatoric uncertainty")# plot the training pointsax[i,1].scatter(X_train[:,0],X_train[:,1],c=y_train,cmap=cm)# plot the testing pointsax[i,1].scatter(X_test[:,0],X_test[:,1],c=y_test,cmap=cm,alpha=0.6)ax[i,1].set_xlim(xx.min(),xx.max())ax[i,1].set_ylim(yy.min(),yy.max())# epistemic uncertainty plotax[i,2].contourf(xx,yy,Ze,cmap=cm,alpha=.8)ifi==0:ax[i,2].set_title("Epistemic uncertainty")# plot the training pointsax[i,2].scatter(X_train[:,0],X_train[:,1],c=y_train,cmap=cm)# plot the testing pointsax[i,2].scatter(X_test[:,0],X_test[:,1],c=y_test,cmap=cm,alpha=0.6)ax[i,2].set_xlim(xx.min(),xx.max())ax[i,2].set_ylim(yy.min(),yy.max())ReferencesAleatoric and epistemic uncertainty in machine learning: an introduction to concepts and methods, Hüllermeier et al., Machine learning (2021)