Package: smartsvm
SmartSVMSmartSVM is a Python package that implements the methods fromFast
Meta-Learning for Adaptive Hierarchical Classifier DesignbyGerrit J.J. van den BurgandAlfred O. Hero. The package contains functions for
estimating the Bayes error rate (BER) using the Henze-Penrose divergence and a
hierarchical classifier called SmartSVM. See the Usage documentation below for
more details.InstallationSmartSVM is available on PyPI and can be installed easily with:pipinstallsmartsvmUsageIn the paper the main focus is on the accurate Bayes error estimates and the
hierarchical classifier SmartSVM. These will therefore be of most interest to
users of the SmartSVM package. Below we briefly explain how to use these
functions.CitingIf you use this package in your research, please cite the paper using the
following BibTex entry:@article{van2017fast,
  title={Fast Meta-Learning for Adaptive Hierarchical Classifier Design},
  author={Gerrit J.J. van den Burg and Alfred O. Hero},
  journal={arXiv preprint arXiv:1711.03512},
  archiveprefix={arXiv},
  year={2017},
  eprint={1711.03512},
  url={https://arxiv.org/abs/1711.03512},
  primaryclass={cs.LG}
}Bayes error estimatesError estimation is implemented three functions:hp_estimatefor the Henze-Penrose estimator of the Bayes error rate.
This can be used as:>>>importnumpyasnp>>>fromsmartsvmimporthp_estimate>>>X1=np.random.multivariate_normal([-1,0],[[1,0],[0,1]],100)>>>X2=np.random.multivariate_normal([1,0],[[1,0],[0,1]],100)>>>hp_estimate(X1,X2)# with normalization>>>hp_estimate(X1,X2,normalize=False)# without normalizationcompute_error_graphandcompute_ovr_errorrespectively compute the
complete weighted graph of pairwise BER estimates or the One-vs-Rest BER for
each class. They have a similar interface:>>>importnumpyasnp>>>fromsmartsvmimportcompute_error_graph,compute_ovr_error>>>fromsklearn.datasetsimportload_digits>>>digits=load_digits(5)>>>n_samples=len(digits.images)>>>X=digits.images.reshape((n_samples,-1))>>>y=digits.target>>>G=compute_error_graph(X,y,n_jobs=2,normalize=True)>>>d=compute_ovr_error(X,y,normalize=True)SmartSVM ClassifierSmartSVM is an adaptive hierarchical classifier which constructs a
classification hierarchy based on the Henze-Penrose estimates of the Bayes
error between each pair of classes. The classifier is build on top of
Scikit-Learn and can be used in the exact same way as other sklearn
classifiers:>>>importnumpyasnp>>>fromsmartsvmimportSmartSVM>>>fromsklearn.datasetsimportload_digits>>>digits=load_digits(10)>>>n_samples=len(digits.images)>>>X=digits.images.reshape((n_samples,-1))>>>y=digits.target>>>clf=SmartSVM()>>>clf.fit(X,y)>>>clf.predict(X)By default, the SmartSVM classifier uses the Linear Support Vector Machine
(LinearSVC) as the underlying binary classifier for each binary subproblem
in the hierarchy.  This can easily be changed with thebinary_clfparameter to the class constructor, for instance:>>>fromsklearn.treeimportDecisionTreeClassifier>>>clf=SmartSVM(binary_clf=DecisionTreeClassifier)>>>clf.fit(X,y)>>>clf._get_binary()DecisionTreeClassifier(class_weight=None,criterion='gini',max_depth=None,max_features=None,max_leaf_nodes=None,min_impurity_decrease=0.0,min_impurity_split=None,min_samples_leaf=1,min_samples_split=2,min_weight_fraction_leaf=0.0,presort=False,random_state=None,splitter='best')You may optionally add parameters for the classifier through theclf_paramsparameter. This should be a dict with the parameters to the
binary classifier, as follows:>>>clf=SmartSVM(binary_clf=DecisionTreeClassifier,clf_params={'criterion':'entropy'})>>>clf.fit(X,y)>>>clf._get_binary()DecisionTreeClassifier(class_weight=None,criterion='entropy',max_depth=None,max_features=None,max_leaf_nodes=None,min_impurity_decrease=0.0,min_impurity_split=None,min_samples_leaf=1,min_samples_split=2,min_weight_fraction_leaf=0.0,presort=False,random_state=None,splitter='best')Finally, it’s possible to retrieve probability estimates for the classes if
the underlying classifier supports thepredict_probamethod:>>>fromsklearn.svmimportSVC>>>clf=SmartSVM(binary_clf=SVC,clf_params={"probabilities":True})>>>clf.fit(X,y)>>>prob=clf.predict_proba(X)>>>importpandasaspd>>>df=pd.DataFrame(prob)>>>df012...09.999997e-011.716831e-182.677824e-13...11.000000e-079.956408e-011.035589e-09...22.595652e-051.452011e-029.722321e-01...For more information about parameters to SmartSVM, see the API documentationhere.Known LimitationsThe Henze-Penrose estimator of the Bayes error rate is based on construction
of the Euclidean minimal spanning tree. The current algorithm for this in the
SmartSVM package uses an adaptation ofWhitney’s algorithm. This is not the fastest way to
construct a minimal spanning tree. TheFast Euclidean Minimal Spanning Tree
algorithm by March et al., would be
a faster option but this makes it more difficult to construct orthogonal MSTs.
Incorporating this algorithm into the SmartSVM package is considered a topic
for future work.ReferencesThe main reference for this package is:G.J.J. van den Burg and A.O. Hero - Fast Meta-Learning for Adaptive
Hierarchical Classifier Design (2017).The theory of the Henze-Penrose estimator is developed in:V. Berisha, A. Wisler, A.O. Hero, A. Spanias - Empirically Estimable
Classification Bounds Based on a Nonparametric Divergence Measure (2016).V. Berisha, A.O. Hero -  Empirical Non-Parametric Estimation of the Fisher
Information (2015).