Package: classically
CLASSICALLYClassifier Comparison and Feature Analysis Tools in PythonA python package for the evaluation of classification performance.InstallationClassicallyis published on PyPi so you can install it withpip.>>> python -m pip install classicallyApplicationsCritical Difference DiagramA main application is the comparison of categorized paired metric data, e.g. accuracy results of
different classification methods in machine learning.
ThereforeClassicallyimplements thecritical difference diagram(described in[1]).ExampleImagine that we have five different classification methods tested on 14 different datasets.
Every classifiers returns an accuracy result on each test set in the corresponding dataset.
We collect the results in a table like this:ClassifierA0.600.810.620.190.930.540.530.410.210.970.320.820.380.75B0.330.680.430.230.900.430.320.200.220.860.210.820.410.73C0.250.640.400.100.850.390.310.190.180.900.230.780.430.71D0.640.840.600.260.950.600.360.370.190.950.440.840.410.84E0.370.680.470.180.880.370.270.250.240.790.250.830.360.64We load this table in anumpy arrayof shape(5, 14)and call the functionclassically.critical_difference_diagram. The resulting plot can be seen below.Markings on this number line represent the average ranks of one classifier based on his accuracy
over all datasets. The lowest rank corresponds to the highest accuracy. Classifiers are connected
by a horizontal line if they do not have a significant difference. This significance is based on
post-hoc Wilcoxon signed rank tests for each pair of classifiers.Therefore, it seems like classifier D is the best choice for the overall classification task.
It works best on the 14 chosen datasets, altough it's not the best classfier for every single
dataset on its own. But we can also see, that there is no significant (alpha=0.05) difference in
the accuracy results of classifier D and A. If D would be much more computationally expensive than
A, then we should consider choosing A as the better classifier.Scatter MatrixFor an in-depth comparison of the classifiers on single datasets a special type of scatter matrix
that is designed to compare multiple categories of data can be found inClassically.ExampleFor a more elaborate decision in theexample abovewe could directly compare the best three
classifiers A, B and D using the functionclassically.scatter_comparison.Points above the diagonal line represent datasets that are better classified by the method in the
upper left corner. A horizontal and vertical line indicates the mean accuracy of the corresponding
classifier. A solid line marks the higher mean.
A choice can now be easily made for the comparison of classifier A and B as well as B and D.
We also see that D is better than A in mean accuracy but that A has a big advantage on one dataset
that is well beyond the diagonal line for five percent difference.
The datasets could now be further analyzed by, for example, looking at the number of training and
test instances. An option for setting the opacity value of points in the scatterplots accordingly
is available.Feature ScoreEvaluating the importance of features in data can be very helpful in reducing the dimensionality of
the feature space. While principal component analysis transforms original features into new ones,
it can also be used to creating a ranking of those features.Classicallyis able to compute the
feature score for a given dataset.ExampleWe can analyze the importance of the four features in the well-known IRIS dataset by using the
methodclassically.plot_feature_score.The plot shows the normalized feature score. The 'most important' feature based on that score is
'petal length (cm)'. All other features then have a relatively low score, e.g. sepal length's score
is about 80% lower. The red markings show the accuracy results of a ridge classifier where only the
firstnfeatures (in descending score order) are used (ngets incremented with each step on the
x-axis).References[1]Demšar, Janez (2006).
"Statistical comparisons of classifiers over multiple data sets."
The Journal of Machine learning research 7, 1-30.